Victoria — reliable email-triage agent
A rebuild of an n8n email-triage flow that hallucinated. Deterministic code now handles the rules; the LLM only makes the fuzzy judgement call, with a strict schema and a human review queue behind it.
The challenge
The existing n8n flow packed everything into one ~500-line prompt: exact pattern matching (“body contains X → ignore”), strict business rules (which addresses to CC, when to send an SMS) and fuzzy judgement (“is this really an incident or just a slowdown?”) — then asked the model to return it all as free- form JSON. Asked to recite 500 lines of rules verbatim, the model dropped some and invented others: wrong priorities, malformed JSON, values outside the allowed set.
Approach
Separate the engine (deterministic, reliable code) from the business context (rules, projects, contacts — editable without a redeploy), and give the LLM only the part it’s actually good at.
- LangGraph state machine — explicit steps, not one big prompt.
- Deterministic nodes — ignore filters, monitor-alert detection and project detection are plain Python matching against variations stored in the database. Zero LLM calls, zero chance of hallucination on those.
- One LLM node — semantic judgement only (category, status, confidence, reasoning), via tool-calling on Claude or forced structured output on Ollama, then Pydantic-validated. It can no longer emit malformed JSON or an out-of-range value.
- Code derives the rest — priority, CC whitelist and SMS eligibility come from the validated decision, never directly from the model.
- Review queue — anything unclassifiable or below the confidence threshold goes to a manual queue with no automatic action.
/adminUI — projects, keywords, contacts, templates, the active LLM provider and the confidence threshold all live in the DB and are edited by non- technical staff through a small web interface.- Shadow mode — classifies and logs every email to an audit log without sending or inserting anything, until the Outlook, FileMaker and SMS integrations are switched on.
Result
The triage that used to misfire now runs on rails: deterministic cases are decided by code, the model’s output is schema-checked, and every decision is logged and reversible. The business rules can change from the admin screen without a deploy.
Add classification accuracy and volume once it’s out of shadow mode.
Want an outcome like this?
No more malformed output or invented values: the model can't choose a priority, a CC address or an SMS on its own — code derives all of that from a validated decision.
Book a callBurgerBase — a self-hosted backend for Base44 apps