No. 02 · Case study

PromptLang

A language for writing the instructions given to AI models (prompts) as code: typed, compiled to TypeScript or Python, versioned.

Code: matteogallo-ai/promptlang

In brief

What it does: It turns a prompt into a checked file, compiled to TypeScript or Python code, and flags when a new version breaks the code of those who use it.
What is measured: How solid the tests of the language reader and of the version check are, and how often an automatic reviewer, the AI linter, is right when it points at writing defects in prompts.
Main defect: Most of the AI linter’s flags are wrong, barely better than flagging everything. What I did about it: I measured it against that point of comparison, published it, and the linter is presented as an aid to the human reviewer, not as a check.

96.13%
of the errors deliberately introduced into the language reader and the version check, caught by the tests
44/45
designed defects found by the AI linter; flagging everything finds them all
12/40
AI linter flags judged correct in human review

The problem

In three sentences, why this project exists.

Prompts live as scraps of text scattered through the code: nothing checks their inputs, nothing tests them, nothing versions them. When a prompt breaks in production, a customer is the one who tells you. PromptLang turns them into files that are checked, compiled, tested and versioned like code.

What I built

What becomes of a prompt file, from reading it to the three uses that come out of it.

The file once read serves three independent uses. The generated code checks every model response against the declared output type; the version check compares the prompt with its previous version; the review flags writing defects.

A .prompt file declares what the prompt receives, what it must return and the model it targets. The compiler derives a typed function from it. The version check compares this contract with the previous version in Git and refuses a change that breaks the calling code without a major version number. The AI linter reads the prompt’s text before the human review. The@promptlang/core package is published on npm, the JavaScript package registry, and Praxis uses it to read its own prompts.

Measures

What the tests and the AI linter are worth, measured against a very simple point of comparison: flagging everything.

To measure the tests, small errors are deliberately introduced into the code (a reversed condition, a changed value) and we check whether the tests fail: they catch 96.13%, on the language reader and the version check only.

On test prompts with known defects, the AI linter finds 44/45 of them. But a linter that flags everything, on every prompt, finds them all: 45/45. The difference lies in correctness, and the gap is modest: 44/217 of the linter’s flags point at a real defect, against 45/360 for flagging everything, and in human review only 12/40 are correct.

Measures in detail

A proportion carries its 95% confidence interval (CI): the range where the true value probably lies, wide when the sample is small.

Tests, AI linter, audit

Measure	Value	95% CI	Reading
Errors introduced on purpose, caught by the tests	96.13%	A count of all the errors introduced into the covered code, not a sample.	Local run, outside continuous integration, on the language reader and the version check only: neither the compiler nor the execution.
AI linter precision (human review)	12/40	18% – 46%	The main defect: in human review, most flags are wrong. Not distinguishable from the bound already published.
Findings of the audit by a context-free agent	23	A count, not an estimate.	Audit of the repository by an AI agent that did not know the project: defects, and one finding about context. Every defect is fixed or refuted in writing.

AI linter against “flag everything” (test prompts)

Measure	AI linter	95% CI	Flag everything	95% CI	Reading
Correctness of the flags	44/217	15% – 27%	45/360	9% – 17%	The gain over a linter that flags every category on every prompt is modest.
Defects found	44/45	88% – 100%	45/45	92% – 100%	Flagging everything finds everything: this figure alone does not tell the linter apart from the point of comparison.

Correctness on the test prompts counts the flags that land on a designed defect; human review reads flags one by one. Both say the same thing: the AI linter helps the reviewer, it does not replace them (system card).

What did not work, and what I did

What an uncompromising audit found, and what became of each finding.

I had the repository audited by an AI agent that knew nothing about the project, tasked with looking for the flaw. Its verdict: the promises of the project’s front page (the README) went beyond what the code actually did. What I did about it, finding by finding, is in the audit follow-up.

An AI linter declared shipped, with no point of comparison

That page presented the linter as shipped although its correctness was low and nothing said what it was worth. A linter that flags everything finds every designed defect: finding many proved almost nothing. I published its correctness next to that point of comparison (#29), then had flags reviewed by hand; the linter is described as an aid to the reviewer, not as a check.

A typed output never checked

The generated code promised a model response matching a type, without checking it. It now checks it, in TypeScript as in Python, and raises an error otherwise; both languages run on the same responses and must agree (#27). The fix started with a failing test.

Tests measured where the risk was not

Measuring the tests with introduced errors covers only the language reader and the version check. The audit found defects outside that scope (compiler, execution, command line, text analysis, spending cap), and one in the version check itself: renaming an exported type was classed as a mere fix (a patch), the smallest version change. That was a choice written in the decision record, and no introduced error reveals a badly chosen rule. The report now lists what it does not cover (report), and that renaming requires a major version (#25).

Three decisions

Three architecture choices, each written in a decision record (ADR) with its reasons and consequences: enough to judge the reasoning, not only the result.

What counts as a prompt’s contract
What the prompt receives (names, types, optional values), what it returns and the model it targets, in a form that ignores the order of parameters and of the values of a closed list. An amendment from the audit adds the names of exported types: renaming a type that the calling code imports requires a major version.
ADR 0002
How the version check finds the previous version
The previous version comes from Git: the reference given, otherwise origin/main, otherwise the last version label set in Git (a tag); the current version is the one of the files on disk, so the check runs before a commit as well as in continuous integration. The level required is that of the most serious change, and a file unreadable in the previous version is an error, never a new file.
ADR 0003
The approved cost is a spending cap
Every paid run shows its estimated cost and asks for approval; that estimate becomes a cap. Before every call, the client reserves the call’s maximum cost; a call that would exceed the cap is not sent. The record also writes what the cap does not guarantee: a text more costly to process than expected exceeds the estimate.
ADR 0004

How it is built

For a technical reader: the stack, the tools and the AI linter’s model.

Strict TypeScript on Bun. Tests with bun test, randomly generated tests (fast-check) on the language reader and the version check, errors introduced with Stryker, Biome, knip. Execution with several providers: Anthropic, OpenAI, Ollama or a supplied client. The AI linter calls claude-haiku-5-5. Architecture: docs/architecture.md.

What comes next

What is missing, according to the project’s roadmap; none of it is scheduled.

Running the tests written in the prompt files: they are read, not run.
Measuring the tests of the compiler and of the execution with introduced errors, outside the scope measured today.
An AI linter with providers other than Claude, each with its own measurement.
Streaming the response as it is written, and editor support (highlighting, completion).

Roadmap