Know what your automations did last night, measure whether the AI steps are right, and change a prompt without breaking what already worked.
Module 12 ~50 min read + lab No codePrerequisites: Module 11: Security and governance, and the gated triage flow from Module 10.
A traditional automation either works or throws an error. An AI step is different: it can run perfectly and still be wrong, confidently, every time. Nobody gets an alert when a model starts labelling refund requests as sales enquiries. The only way to know is to look, on purpose, with a method.
This module gives you that method in three layers: logs (what happened), evaluation (was it right), and improvement (change it safely). The habit is borrowed from software operations, where it is called observability. You will do it in spreadsheets.
For each workflow, can you answer within five minutes: how many runs yesterday, how many failed, how much did it cost, and how often was the AI step wrong? If not, this module fixes that.
Every automation platform keeps a run history. n8n calls it Executions: one entry per run, with the input and output of every node, the time it took, and where it failed. Zapier has Zap history, Make has the scenario execution log, Power Automate has run history. Open it daily for the first two weeks of any new workflow, then weekly.
The platform log is not enough for AI steps, because it does not summarise. Add your own AI log: one row per AI call, appended to a sheet by the workflow itself.
Keep the log in the same workbook as the approval log from Module 10. One workbook per workflow, one tab per log, is a pattern you can maintain.
Things fail for boring reasons: a model endpoint is busy, an email server times out, a sheet is locked. Design for it once and the workflow becomes reliable.
| Technique | What it does | Where to set it |
|---|---|---|
| Retry with backoff | Try again after 5 s, then 15 s, then 45 s. Cures most transient errors. | Node settings: "retry on fail", max tries 3, wait between tries |
| Fallback path | If the primary model fails, use a second model or a simpler rule ("if subject contains invoice, category = billing"). | Error output of the node, routed to an alternative branch |
| Dead-letter / human queue | Anything that still fails is written to a "needs human" sheet or channel with the full context, and a person handles it. | Error workflow (n8n) or a catch branch that appends to the queue and notifies |
| Idempotency | Running the same input twice does not create two tickets or two emails. Check for an existing record before creating one. | A lookup step before every create step |
| Alerting | A message when the failure rate passes a threshold, not on every failure. | A scheduled workflow that counts errors in the last hour |
Never retry an action that may already have succeeded, such as "send email", without checking first. Retry the read, then decide again.
A golden set is a small collection of real inputs with the correct answer written by a person. Twenty examples is enough to start; fifty is comfortable. For the triage flow, that is twenty real (redacted) emails, each with the category a competent colleague would give it.
To evaluate, run every golden input through the AI step and compare. For categories the score is simply accuracy: correct divided by total. For free text such as a drafted reply, use a rubric: three or four yes/no criteria a person can judge in ten seconds.
Scoring fifty drafts by hand takes an hour. You can ask a second model to apply the rubric instead: give it the customer email, the draft and the four criteria, and ask for a score with a one-line reason per criterion. This is LLM-as-judge. It is fast and consistent, and it is good enough to catch big regressions. It is not a replacement for people on anything that matters, because the judge can share the blind spots of the model it is judging. The rule: a person scores the golden set; the judge scores everything else, and a person checks a sample of the judge's scores each month.
Every prompt edit and every model upgrade is a change that can make some cases better and others worse. Software teams call an unnoticed "worse" a regression. The golden set is your regression test.
Sum the cost column of the AI log per day and per workflow. Three numbers to watch: cost per run (rising means longer prompts or more loops), runs per day (rising means adoption, or a runaway trigger), and cost per correct outcome (cost divided by accuracy, the number that lets you compare a cheap-but-sloppy model with an expensive accurate one). Set a monthly budget per workflow and an alert at 80%.
A single tab with a pivot or four formulas is enough: runs per day, failures per day, cost per day, and the latest golden-set score. Chart the first three as lines over the last 30 days. Anyone who opens the sheet should understand the health of the workflow in ten seconds.
An automation nobody trusts is switched off within a month. Adoption is a design task, not a launch email.
Before go-live, know exactly how to stop the workflow in under a minute (deactivate in n8n, turn off the Zap, disable the flow) and who is allowed to do it, including out of hours. Write it in the policy from Module 11 and test it once. An automation you cannot stop quickly is not under control.
You will measure the triage flow, change it, and measure again. You need the gated triage workflow, twenty real or realistic support emails (redacted), a colleague for ten minutes, and a workbook with three tabs: Golden set, AI log, Dashboard.
Paste twenty emails into the Golden set tab with columns: id, email text, my label. Ask a colleague to label the same twenty in a separate column without seeing yours. Resolve disagreements together and record the agreed label. Mark any case you both found hard.
After the classify step in the workflow, add a Google Sheets append-row node writing to AI log with the columns from Section 2. Use the model node's token counts for cost. Run one test email and check the row appears.
Run the twenty golden emails through the workflow (a manual trigger with the sheet as input is easiest). Copy the model's labels next to the agreed labels and compute accuracy with a COUNTIF. Record the score and date in a Changelog tab.
Edit the classification prompt in one way: add clearer category definitions, or one example of each category (not from the golden set). Do not change the model at the same time.
Rerun the twenty. Compare item by item: which cases improved, which got worse. Keep or revert the change and write the decision in the changelog.
On the model node set retry on fail (3 tries, 10 s apart). Route the node's error output to a branch that appends the email to a Needs human tab and posts one message to your channel.
In the Dashboard tab compute runs per day, failures per day and cost per day from the AI log, plus the latest golden-set score. Add one line chart. Write the kill-switch instruction at the top of the tab: who deactivates the workflow and how.
The workbook with golden set, AI log, changelog (two scored runs) and dashboard, plus a screenshot of the failure branch. Save as M12-monitoring.
Pick one answer per question, then check your score. These mirror the style of the final exam.
Answer in your own words first, then open the model answer.
Timestamp, workflow, run id, step, model, redacted input, output, confidence or category, tokens, cost, latency, human decision. Confidence enables escalation of uncertain cases and lets you evaluate whether the model knows when it is unsure.
Record the current golden-set score; make one change; rerun and compare per item; keep only if at least as good and nothing important broke, then note it in the changelog.
The documented way to stop the workflow in under a minute. Before go-live it must be written down, assigned to named people including out of hours, and tested once.
Further reading: Monitoring and observability (DevOps Lab, Module 11) · n8n: error handling and recovery strategies
