Email triage that survives a real inbox

Every demo of AI email handling classifies 500 clean test messages and declares victory. Then it meets a real inbox: the supplier who puts three separate requests in one email, the customer who replies to a six-month-old thread with a new problem, the forwarded message where the actual ask is buried four levels deep.

Benchmarks back this up. Bryo AI tested 1,565 business emails across 10 categories in September 2026 and got 96.4% accuracy with Jev, 98.5% with Gemini 3.8 Flash. Aid4Mail benchmarked 11 models on a 2,000-email corpus; every one scored above 94% F1, with the top five between 98.8% and 99.6%. Those numbers hold up under test conditions.

Production is not test conditions.

Where classification breaks

That Bryo result includes a detail most summaries skip: emails where the model scored below 70% confidence were wrong nearly half the time. Their most confident 85.5% of predictions had zero errors. So 96% is the average, and errors cluster in the emails you most need classified correctly: ambiguous requests, multi-topic messages, and anything outside the training distribution.

One enterprise study using FLAN-T5-Large found 90.4% overall accuracy but only 25.3% precision on technical emails. It was good at identifying what was not technical and poor at catching what was. Brief troubleshooting confirmations and implicit references to infrastructure fell through because the model relied on surface keywords rather than context.

For an Australian business, failure patterns matter more than accuracy numbers. A misclassified invoice delays payment. A misclassified support request loses a customer. A misclassified RFQ misses a quoting window. Each error type costs differently, and a single accuracy score hides that.

What a production pipeline actually looks like

Azizov’s manufacturing case study (2026) describes the pattern that works: shadow mode first, then routing. His system ran for two weeks observing real traffic without acting. Staff validated classification rules against their own judgment. Only then did it go live.

That two-week observation period is the part most implementations skip. It also catches the edge cases: the supplier who uses “urgent” in every subject line, the internal forwarding chain where the original sender is not the person making the request, the automated receipt that looks like a customer query.

The pipeline has three layers. First, classification: assign each email to a category and a confidence score. Second, routing: high-confidence messages get acted on automatically (forwarded to the right person and logged in the CRM). Third, escalation: everything below the confidence threshold goes to a person, with the model’s best guess attached so the human does not start from scratch.

Bryo’s data supports this. At 99% confidence, there were zero errors on their test set. Below 70%, nearly half were wrong. A production system that auto-routes everything above 90% confidence and escalates the rest captures most of the speed benefit while keeping the error rate close to zero.

What it costs

Running costs are not the barrier. Bryo reported $0.08 to $1.79 per 1,000 emails depending on model. At 500 emails a day, that is $1.50 to $54 per month. Real cost sits in the build: connecting mailboxes, defining categories, tuning confidence thresholds, and integrating with whatever system receives the classified output.

A self-hosted n8n workflow handling email classification via API costs roughly $20 per month for compute. Build time is two to four weeks for a business with clear category definitions and a technical person who can wire the IMAP connector. Without that person, a contractor build runs $2,000 to $5,000 depending on mailbox count and downstream complexity.

Compare that to a manual process. One person spending 90 minutes a day sorting email burns 30 hours per month. At $40 per hour, that is $1,200 monthly. Automation pays for itself in two to five months.

Where the human stays

The human stays in three places. First, on every email below the confidence threshold. Second, on any email that triggers a consequential action (sending a quote, issuing a refund, escalating a complaint). Third, on the weekly review of classification accuracy, because email patterns shift: new suppliers arrive, customer language changes, automated systems get replaced.

Email triage is not a solved problem. It is a problem where the tools now handle the boring 80%, freeing a person to spend real attention on the 20% that matters.

Sources

• Bryo AI email classification benchmark, 1,565 emails, September 2026. Nikhil Mudholkar, CTO.
• Aid4Mail AI model benchmark, 11 models, 2,000-email corpus, 2026.
• Azizov, S. “AI Email Routing for Manufacturing,” Inbox Router case study, 2026.
• MDPI Applied Sciences, “Enterprise E-Mail Classification Using Instruction-Following Large Language Models,” 2026.
• InterRisk email automation PoC, deepsense.ai, 2026.