Sujata Karan
← Back to the Lab
Campaign QA Agent

Campaign QA Agent: catching campaign reporting errors before they become insights.

An AI-assisted n8n workflow that takes a raw campaign performance export, runs it through five parallel rules-based checks, hands the flagged rows to an LLM for plain-language explanations, and queues everything for human sign-off. Walked through end to end below.

Built in n8n GPT-4o narrative layer Human-in-the-loop Evaluated against ground truth
Why this exists

Automate the mechanical checks. Keep judgment with people.

Campaign reports are often reviewed under deadline pressure. Small data issues can be missed, then treated as real performance changes. This agent is designed to catch those issues before analysis starts, explain them clearly, and route them to a human rather than acting on them automatically.

The problem
  • Manual QA is repetitive and easy to skip
  • Small reporting errors can distort interpretation
  • Raw error codes are not useful to every reviewer
The solution
  • Run deterministic checks on every row
  • Use AI to explain, not to validate raw numbers
  • Send every finding to a review queue
Design principle
  • Deterministic where possible
  • AI where useful
  • Human judgment where consequential
The pipeline

One input, five checks, one narrative pass, one queue.

Every campaign report goes through the same shape: structured checks first, language model second, human last. The rules catch what's mechanically wrong; the model explains what that means; a person still signs off before anything is treated as fact.

Input
Google Sheets
Campaign performance export lands in a source sheet.
Trigger
Read & split rows
Each row is prepared for parallel checking.
Check 1
Math consistency
Do spend, clicks, and conversions reconcile?
Check 2
Zero / null anomaly
Flags missing or zeroed metrics.
Check 3
Duplicate row
Catches double-counted campaign rows.
Check 4
Anomaly swing
Flags metrics that jump outside normal range.
Check 5
Date freshness
Flags stale or out-of-window reporting dates.
Merge
Merge node
Combines the five check outputs into one summary per row.
Narrative
LLM chain (GPT-4o, temp 0.2)
Turns flagged rows into plain-language explanations with confidence scores.
Output
Parse to items
Clean JSON array, no markdown or prose wrapping to clean up.
Write
Append to review queue
campaign_qa_review_queue Google Sheet, one row per flag.
Gate
Human review
A person confirms or dismisses each flag before it counts as a finding.
Eval
Automated scoring
Flags are scored against a ground-truth set to track accuracy over time.
My role

Designed and built end to end.

This is a portfolio build. I designed the workflow architecture, implemented the QA logic, defined the LLM narrative step, structured the human review process, and set up the evaluation approach.

Workflow
  • n8n architecture and orchestration
  • Parallel QA checks and merge logic
  • Google Sheets input and review queue
AI layer
  • Prompt and narrative design
  • Structured JSON output
  • Confidence-scored explanations
Trust & evaluation
  • Human-in-the-loop review design
  • Ground-truth evaluation set
  • Accuracy tracking over time
What a flag actually looks like

From a raw row to a reviewable finding.

A sanitized, illustrative example of the shape the Campaign QA Agent produces — a raw campaign row on the left, what the pipeline generates from it on the right.

Raw row · source sheet
campaign
EMEA_Q3_LinkedIn_Sponsored
date
2026-07-14
spend
€4,200.00
impressions
118,400
clicks
0
conversions
12
Check 1 · Math consistency Check 2 · Zero anomaly
Generated finding · review queue
"Clicks are reported as zero for this row, but 12 conversions are recorded against it. Conversions without clicks are only plausible under view-through attribution, which this report doesn't track. Recommend confirming the click data pulled correctly before this row is used in ROAS calculations."
Confidence 0.91
Status Pending review

Illustrative example built to match the structure the Campaign QA Agent processes, not a live production row.

Walkthrough

How it works, stage by stage

Same shape as everything else in the logbook: the problem, the approach, and what changed because of it.

STAGE 01

The problem it solves

Problem

Campaign performance reports get skimmed under time pressure, so the same handful of errors (misaligned dates, duplicated rows, a metric that quietly went to zero) slip through and get treated as real trends.

Approach

Push the mechanical checking to rules, not a model. Five deterministic checks run in parallel over every row before any AI is involved, so the flags are reproducible and explainable on their own.

Google Sheets n8n Rules-based checks
STAGE 02

Where the LLM comes in

Approach

The five check outputs merge into one summary per row, then pass through a Basic LLM Chain node (GPT-4o, temperature 0.2) that writes a plain-language explanation for each flag and attaches a confidence score, rather than a raw pass/fail code a reader has to interpret.

Result

A clean JSON array of flagged items with explanations and confidence scores, parsed directly with no markdown or prose wrapping to strip out.

GPT-4o Basic LLM Chain Structured output
STAGE 03

Human-in-the-loop, by design

Approach

The flagged, explained items are written to an campaign_qa_review_queue sheet rather than acted on automatically. A human confirms or dismisses each one before it becomes a finding anyone reports on.

Why it matters

The agent's job is to surface and explain, not decide. That boundary is what makes the output usable in a real reporting workflow, where a wrong flag treated as fact is more costly than a missed one.

Review queue Human sign-off AI governance
STAGE 04

Measuring whether it's actually right

Approach

Flagged items are scored automatically against a ground-truth set, so accuracy is tracked as a number over time rather than assumed from a handful of spot checks.

Status

Full pipeline is built and running end to end: rules checks, LLM narrative layer, human review gate, and automated eval scoring.

Eval scoring Ground truth
Evaluation

How well does it work?

The pipeline is evaluated against a labelled ground-truth set so performance can be measured rather than assumed. The fields below are intentionally left as placeholders until the latest evaluation run is ready to publish.

Test set
  • Total rows: [add result]
  • Known issue rows: [add result]
  • Correctly detected: [add result]
Quality
  • Precision: [add result]
  • Recall: [add result]
  • False positives: [add result]
What I track
  • Detection accuracy by rule
  • False-positive patterns
  • Reviewer confirmations and dismissals
Guardrails

Why this counts as AI Trust work, not just automation

The interesting part isn't that a workflow runs on its own. It's what it's allowed to decide, and what it isn't.

Deterministic first
  • Rules run before the model does
  • Every flag traces to a specific check
  • No LLM judgment on raw numbers
Explainable output
  • Plain-language reason per flag
  • Confidence score attached
  • Clean structured JSON, not prose to parse
Human authority
  • Nothing auto-actions on a flag
  • Review queue gates every finding
  • Accuracy tracked against ground truth
More from the lab

Two more pieces are on the way.

An automation case study with before/after time saved, and a lightweight AI governance toolkit, both built the same way: real workflow, real constraints, human judgment kept in the loop.