Case study · Decize · 2025–2026

How I made Decize's AI debriefs trustworthy

By Samir Maikap, Senior AI Product Engineer · November 2026 · 5 min read

Decize is a decision ledger for teams. When a decision plays out, an AI writes the debrief. This is how I measured whether that debrief was telling the truth, found where it wasn't, and took faithful lines from 91.6% to 100%.

The product

Decize captures team decisions from the web app, email and Slack, helps the team frame and forecast them, follows up on outcomes, and runs a review. All of its AI work runs through one Supabase edge function with 13 modes. Each call goes to Gemini first, and any failure moves it to Claude.

The risk

A debrief has two parts: what happened, and what to do next time. If the model adds a line nobody wrote, the team stops trusting the whole ledger. "Sounds right" was not good enough. Every line had to be traceable to something a person actually said.

Extractive by design

I made the debrief extractive. Every line must be a word-for-word phrase from its source, and code checks this and drops any line that isn't. A made-up sentence can't reach the screen, even if the model writes one.

Measuring it with blind judges

• I ran the production function locally, three times per case, so a lucky run couldn't hide a bad prompt.
• Three blind LLM judges, each reading through a different lens, scored every line the model showed. A line passed with 2 of 3 votes, and a tie-break judge settled splits.
• I built a stress set of 57 cases covering 19 ways a debrief can go wrong, plus a 27-case check of the test rig itself.

The first run scored 391 of 427 lines faithful (91.6%). That cleared the original 90% gate, but not the bar I wanted for a feature people would act on. So before changing anything, I set a new bar of 97% on every set.

Two rules fixed it

Prompt version 28 added two rules:

1. A mixed answer keeps both sides. If someone says part went well and part didn't, the debrief can't keep only the half that fits a neat story.
2. Plans, advice to others and sarcasm never count as "Next time". Only a real lesson from the person who lived it does.

Proving it on fresh data

A fix tuned against one set can just memorise that set, so I wrote a new 36-case holdout after the fix. Results:

• Stress set: 363 of 363 lines faithful.
• Fresh holdout: 221 of 221. The old prompt scored 89.6% on the same cases.
• Rig check: 27 of 27.

What the numbers do and don't say

The scores count the lines the model chose to show. After the fix it leaves out "What happened" more often (shown in 120 of 171 stress runs). That's the trade-off I chose: fewer lines, but every line shown is faithful. The prompt rules are pinned by tests in CI. The live eval runs are still a manual gate, and moving them into CI is the next step.

the boring part is the point

Stack

• Next.js
• TypeScript
• Supabase Edge Functions
• Deno
• Gemini
• Claude
• node:test