No. 03 · Case study
Praxis
A tool that turns a business question into an analysis note (a briefing) written by AI models, whose sources are checked by code and whose drafting is reviewed by a model tasked with finding the flaw.
In brief
- What it does
- From a business question, it searches the web, checks in code that every fact has an acceptable source, then writes an analysis note that itself says what it could not source.
- What is measured
- How many real, paid runs end in a note, what a note costs, and how many notes pass the source check.
- Main defect
- No real note passes the source check. What I did about it: the note is delivered anyway, with a section written by the code that lists every unsourced item; the defect is published, not hidden.
6/13Proof: praxis/docs/SYSTEM_CARD.md
real runs that reached the end, out of the attempts
$3.29–$4.75Proof: praxis/README.md
cost of a completed note, from the cheapest to the most expensive
completed notes flagged as insufficiently sourced
The problem
In two sentences, why this project exists.
An analysis note written by an AI model looks sourced even when it is not. Praxis writes a note in a chosen format (preparation note, memo, position paper) and tells the reader, in the text itself, what rests on a verifiable source and what does not.
What I built
The stages of a note, from the question to the delivered file, and the two possible outcomes.
The instructions given to the models are .prompt files read by PromptLang. The section of unsourced items is written by the code from the check’s report: no model can soften it.
Measures
What the real, paid runs give, and what is not measured at all.
Of the real runs, 6/13Proof: praxis/docs/SYSTEM_CARD.md reached the end; a completed note cost $3.29–$4.75Proof: praxis/README.md. Every completed note is flagged as insufficiently sourced: 6/6Proof: praxis/README.md. The flag is raised as soon as a single item blocks.
No point of comparison
Nothing in the repository compares Praxis with a simpler approach, a single model call or drafting without search. The only score published before rated simulated notes, written by hand for the tests: it measured those examples, not a model, and it was withdrawn (changelog). The quality of the notes themselves is reviewed for no question; only the first-hand sources of one question were reviewed by a human (real runs).
Measures in detail
A proportion carries its 95% confidence interval (CI): the range where the true value probably lies, wide when the sample is small.
Real, paid runs
| Measure | Value | 95% CI | Reading |
|---|---|---|---|
| Completed notes, out of the attempts | 6/13Proof: praxis/docs/SYSTEM_CARD.md | 23% – 71% | The other attempts failed: source check too strict, text before the structured response, time limit exceeded, truncated response, section refused. |
| Notes flagged as insufficiently sourced | 6/6Proof: praxis/README.md | The repository publishes no interval for this count. | The main defect: every completed note is flagged. The flag is raised as soon as a single item blocks. |
| Cost of a completed note | $3.29–$4.75Proof: praxis/README.md | The range of the completed runs, from the cheapest to the most expensive: not an estimate. | Cost recorded with every completed run. |
What did not work, and what I did
Three real failures, their cause when it is known, and what each one changed.
Declared done without a single real note
Every stage and every format was built and tested on hand-written examples before the first real run. The tests could only prove that the stages fitted together, the test bench measured those examples, and the first real runs all failed. What I did: an independent audit, then fixes without any paid call (general instruction in its proper place, model chosen per stage, tolerant reading of the structured response, time limits per stage, resuming where the run had stopped, cost measurement), then real runs published, failures included (fix plan).
The strict check threw all the work away
In strict mode, a single item without a source cancelled every stage already done. No trace of those runs was kept: the cause is read in the instruction and the code, not observed. The search instruction required a consultation date that the model cannot know, which left only one compliant answer: “missing source”. Praxis now dates its consultations itself, which removes the probable cause without guaranteeing a source for every item, and the note is delivered with its gaps written into it (ADR 0002).
A section refused for a web address in its text
A summary section was refused, then refused again on the retry, this time for a www. address in the text, which document readers turn into a link. The fix (#44) replaces that instruction with numbered references to the sources, checked by the code. No paid retry was run to “verify” it: the fix is not proven on a real run.
Three decisions
Three architecture choices, each written in a decision record (ADR) with its reasons and consequences: enough to judge the reasoning, not only the result.
Read the structured response in code, repairing nothing
Anthropic’s interface refuses responses in an imposed format when the web search cites its sources, and refuses to force a tool call on the models used. For the stages with search, the code keeps the text after the last tool call and extracts from it the largest complete and readable structured response (a JSON object), repairing nothing; the stages without search impose their format. In every case, the code validates each field.
Deliver with a flag rather than give up
On the command line and in the test bench, a failed source check is recorded, the run continues, the report is flagged and the code adds a section that lists every blocking item. The note is complete, but says in its own text what it could not source. Used as a library, Praxis still gives up by default: changing the errors a public function can raise would break the code of those who call it.
Which sources to trust
A company’s website is accepted for that company’s facts only, provided that the mapping between organisations and websites is explicit, reviewed and versioned. After a human review, the rule never again accepts a first-hand source for the likelihood or impact of a risk. The second part, the specialist press, is not implemented.
How it is built
For a technical reader: the stack, the models used and the test tools.
TypeScript on Bun. Anthropic’s Messages programming interface (API), with its web search tool: claude-opus-5-5 for scoping, options and the adversarial review,claude-sonnet-5-5 for search, actors, risks and drafting. @promptlang/core from npm, pinned by its hash; pdfkit for the PDF, the Word file written by hand. Testsbun test, fast-check, Stryker. Architecture: docs/architecture.md.
What comes next
What is missing, according to the project’s roadmap.
- A real run that ends reliably, and measured: cost, volume of text processed, duration, rated sample.
- The quality of real notes rated by a judge model, validated against human ratings, with its margin of error.
- The age of a source measured on the date of the page itself.
- A web search that filters its results along the way, evaluated against the base tool.