No. 01 · Case study

Verity

An agent that answers questions about contracts by quoting the clause word for word, and refuses when it has no evidence.

Code: matteogallo-ai/verity

In brief

What it does: It answers questions about a set of contracts by quoting the exact clause, and says “I don’t know” when the documents do not contain the answer.
What is measured: On test questions never seen during tuning: whether it refuses the unanswerable questions, and whether its answers make anything up.
Main defect: It answered questions that had no answer in the documents. What I did about it: I published it, with the three questions involved named in the results.

17/20
unanswerable questions correctly refused
0/31
answers containing a fabrication, according to a judge checked against my own ratings
3/20
unanswerable questions it answered anyway

The problem

In three sentences, why this project exists.

Assistants that answer over company documents do so with confidence, and are wrong just often enough to be dangerous. On a contract, a fluent but wrong answer costs more than an “I don’t know”. Verity answers only by quoting the clause word for word, refuses otherwise, and measures whether it refuses rightly.

What I built

The path of a question, from finding the passages to the answer or the refusal: it shows where a refusal is decided.

The refusal is decided in order: no quote, then the checks written in code, then the confidence rating and its threshold. The judge, a second model that rates the answers, is not on this path: it evaluates the runs afterwards.

Verity is a RAG agent: it first looks for the useful passages in the documents, then drafts its answer from them. It keeps the exact position of each passage in its document and returns either an answer whose every quote is found as is in the source, or a refusal with its cause. The demo on the home page replays answers from a real run.

Measures

What its answers and refusals are worth, in figures, with their uncertainty and a comparison with a simpler system.

The figures come from the reference run, on test questions never seen during tuning. On the questions whose documents do not contain the answer, Verity refuses 17/20 of them, but still answers 3/20. The judge, a second model in charge of rating the answers, finds a fabrication in 0/31 answers. This judge agrees with my own ratings, made without seeing its own, at κ 0.71: that is a Cohen’s kappa, a measure of agreement that removes the share due to chance, from zero for chance to one for perfect agreement. This figure mixes real claims and deliberately planted ones; on the real ones alone, it drops to κ 0.48, a fragile figure.

Compared with a naive RAG, the same principle without any check, on tuning questions and on runs the budget stopped before the end, Verity does not do measurably better: it handles 16/21 questions that have an answer, against 21/21, and on so few questions the gap cannot be told apart from chance. This is the published result, not the favourable one.

Measures in detail

A proportion carries its 95% confidence interval (CI): the range where the true value probably lies, wide when the sample is small.

Refusal threshold of the run: 0.00.

Reference run, test questions

Measure	Value	95% CI	Reading
Refusal recall	17/20	63% – 95%	Questions with no answer in the documents, refused.
Refusal precision	17/22	56% – 90%	Refusals that concerned a truly unanswerable question; the others come from the quote checks.
Unanswerable questions answered	3/20	5% – 37%	The main defect: an answer given where the documents contain none.
Hallucination rate (judge), answerable questions answered	0/31	0% – 12%	According to the judge checked below. Zero observed is not zero guaranteed: hence the interval.
Judge–blind human agreement, Cohen's kappa (60 claims)	κ 0.71	Kappa is not a proportion: the source file gives it no interval.	Agreement of the judge with my blind ratings, real and planted claims.
Judge–blind human agreement, Cohen's kappa (40 real claims)	κ 0.48	Kappa is not a proportion: the source file gives it no interval.	Fragile: the human judges almost no real claim unsupported, so a single disagreement moves it a lot.
Perturbed claims the judge finds unsupported (upper bound)	16/20	58% – 92%	Upper bound: a planted claim counts as detected as soon as one of its parts is rejected.

Against a naive RAG (tuning questions, incomplete runs)

Measure	Verity	95% CI	Naive RAG	95% CI	Reading
Answerable questions handled	16/21	54% – 90%	21/21	84% – 100%	Not distinguishable according to the comparison file. Verity refuses more, with no measured gain in return.
Fabrication rate (judge)	0/16	0% – 20%	0/21	0% – 16%	No fabrication measured on either side: nothing to separate them.
Unanswerable questions it answered	0/2	0% – 66%	2/2	34% – 100%	The advantage expected of Verity, but on so few questions that it cannot be told apart from chance.

The budget stopped both runs before the end; the comparison covers the questions both of them finished (comparison file).

What did not work, and what I did

Two defects the figures alone do not show, and what I did about them.

The confidence threshold decides nothing

Verity gives each answer a confidence rating and refuses below a threshold. By default the code refuses below 0.55. The reference run ran at 0.00: the threshold was chosen on the tuning questions by a rule fixed in advance, and no threshold did better than another.

The threshold decides nothing on this data.
docs/results.md

Every refusal comes from a check written in code or from the rating model itself; none is due to the threshold. The confidence rating is the model’s self-assessment, and it does not match its actual error rate. What I did: publish it. The actual threshold stands next to the code’s, and a refusal based on reliable confidence is on the roadmap instead of being claimed.

An exact quote of a badly extracted text

Premium Support shortens the Priority 1 response time to thirty (30) minutes, twentyfour [sic, PDF extraction defect] hours a day, seven days a week.
Quote from the answer to q-010, reference run

In the order form PDF, “twenty-” ends a line and “four” starts the next one; text extraction glued the two together without the hyphen. The “word for word” check accepted it, since the quote faithfully copies the faulty passage: it proves the quote is in the extracted text, not that the extracted text is faithful to the document.

An audit proposed a rule: refuse any answer with a number that is not in its quotes. Measured on the published runs before switching it on, it refused correct answers, for computed numbers, numbers taken from the question, identifiers, and here a duration written in digits in the answer and in words in the faulty quote. It is not switched on (measurement).

Three decisions

Three architecture choices, each written in a decision record (ADR) with its reasons and consequences: enough to judge the reasoning, not only the result.

A refusal is an answer in its own right, and it is measured
The evaluation set deliberately contains unanswerable questions, and the correctness of refusals is among the main measures. A change that makes it answer more often by answering wrongly is a regression. The record promised a “calibrated” refusal, that is a confidence that matches the actual error rate; the measurement contradicted it, and the record says so at the top.
ADR 0003
We reproduce the run file, not the model’s output
No call fixes the model’s randomness. A real run is recorded once, with the code version, the models and the detail of each question, and is never rewritten; every published figure is recomputed from that file. Two runs of the same version may differ: that is the model’s randomness, reported as such.
ADR 0007
Instruction files read locally, without PromptLang
An earlier record planned to compile the model’s instructions with PromptLang, my other project; that integration was never built. Rather than keep the promise, I wrote down the actual state: a small tested file reader, an error if a variable is missing, untrusted content marked as data. The dependency would have served a story, not a need.
ADR 0005

How it is built

For a technical reader: the stack, the tools and the models of the reference run.

Python, FastAPI, Postgres with pgvector (or in-memory storage), local models to compare the meaning of texts and rank the passages, Anthropic and OpenAI clients behind a fallback router, Next.js interface. Tooling: uv, ruff, strict mypy, pytest, GitHub Actions.

Reference run: agent claude-sonnet-4-6, judge claude-opus-5-5, meaning comparison BAAI/bge-small-en-v1.5. Detailed architecture: docs/architecture.md.

What comes next

What is still missing, according to the project itself.

A published run of the current version: the reference runs predate the model’s current instructions.
A refusal based on reliable confidence: the confidence error measured on the test set does not allow it.
A judge from another model family, and a second human annotator.
Reading web pages: the code exists, no test exercises it.

Roadmap · what the figures do not say

In brief

The problem

What I built

Measures

Reference run, test questions

Against a naive RAG (tuning questions, incomplete runs)

What did not work, and what I did

The confidence threshold decides nothing

An exact quote of a badly extracted text

Three decisions

A refusal is an answer in its own right, and it is measured

We reproduce the run file, not the model’s output

Instruction files read locally, without PromptLang

How it is built

What comes next