Results

In one sentence

This page shows what the text-to-SQL adapter measured on an exam of databases it never trained on, read straight from the run’s results file, and keeps those numbers apart from the simulation numbers used elsewhere in these docs.

In plain words

A report card is only useful if you know what test it came from. Here the test is a set of questions about databases the student has never opened. The score is “how often does the student’s query return the same rows as the teacher’s query”.

Every score comes with a range, like “somewhere between 64% and 82%”. A small exam gives a wide range. That is not a flaw in the report. It is the honest size of the doubt.

Unseen databasesthe examWhole databases, some from industries never in training.

Free Colab T4where the adapter trainedNot on Nebius GPUs. The teacher ran on Token Factory.

CPU onlyto recompute every numberFrom the raw answers, with one command.

The adapter’s results

Status: Pending: the run's results file is not in yet

Results will appear here when the training run finishes

Nothing on this page is a measured result yet. When the run is done, its results.json is copied into the site and this panel turns into the numbers below. Nothing is typed by hand.

• Exam size, and how many questions were measured
• Base model and adapter: how often each matches the teacher, with a 95% range
• The same split by difficulty and by industry seen or never seen in training
• Coverage and precision when the adapter only answers if it is confident
• Estimated cost against the teacher alone (an estimate: token counts are measured, prices are inputs)
• SHA-256 prefixes of the files behind it, and the command that recomputes everything

How it really works

1. A run ends with a results.json. measure_adapter.py report reads the raw answers and writes every number into that file, with the SHA-256 of each input file.

2. The file is copied into the docs site. The site reads src/data/sql-results.json when it is built, so no number on this page is typed by hand. If the file says the run is pending, the panel above says so.

3. verify recomputes it. Anyone with the raw answers, the exam and the exam databases can run measure_adapter.py verify and get OK or a FAIL: line. See Proof you can run.

flowchart TB
  RAW[("raw.jsonl<br/>teacher, base and<br/>adapter answers")] ==> REP["measure_adapter.py report<br/>on a CPU"]
  GOLD[("golden.jsonl, dbs.json<br/>the exam and<br/>its databases")] ==> REP
  REP ==> RES[("results.json<br/>numbers and the<br/>inputs' SHA-256")]
  RES ==>|"copied in"| PAGE["This page,<br/>at build time"]
  RAW -.-> VER{{"measure_adapter.py verify:<br/>every number recomputed<br/>and equal?"}}
  GOLD -.-> VER
  RES -.-> VER
  VER -->|"yes"| OK["OK"]
  VER -->|"no"| BAD["FAIL: names the file<br/>or the numbers"]
  class RAW,GOLD,RES store
  class VER gate
  class OK ok
  class BAD danger

Reading this diagram. The thick arrows make the numbers and the dotted ones check them, and both start from the same three files. The page is fed from the results file by a build step (it is copied to docs-site/src/data/sql-results.json), not by hand. The check does not trust the results file at all: it recomputes everything from the raw answers and compares.

• A file
• The check
• Makes the numbers
• Checks them

How to read the numbers

• Match with the teacher is execution match: the student’s query and the teacher’s query return the same rows on the same database. It is not “the answer is right”.
• The 95% range is a Wilson interval. If the exam were drawn again many times, about 19 of 20 such ranges would contain the true rate. With a small exam it is wide, and two bars whose ranges overlap should not be called different.
• Base model against adapter is the point of the work. Both answer the same questions with the same prompt and the same limits. The gap is what the adapter added.
• The split by difficulty and by industry shows where the adapter helps. Each cell has few questions, so read the ranges before the bars. “Industry never seen in training” is the fairest test of whether it learned SQL or memorised.
• Coverage and precision when it answers describe the confidence gate. Coverage is the share of questions the student answers on its own. Precision is how often those answers match the teacher. A higher threshold raises precision and lowers coverage.
• The estimated cost is an estimate: token counts are measured, the prices are inputs (Nemotron 3 Ultra at $1 and $3 per million tokens in and out, and a student price you set), and no student endpoint was run, so no GPU-hour bill is in it. See Cost and savings.
• The data is synthetic. The teacher wrote the databases and the questions. These numbers say how well the adapter copies the teacher on queries that run, not how well it serves your real users.

What this page cannot tell you

It cannot tell you how the adapter does on your schema, in another SQL dialect, or on questions a person would phrase badly. Run your own held-out questions through measure_adapter.py to find out.

Simulation numbers are a different thing

Elsewhere in these docs you will see numbers such as “69% saved per token”. Those come from mock models in the simulator (python -m sim.simulate). They show that the loop works: the student starts at zero share of traffic, takes over as nights pass, a corrupted version is rolled back, and a launch day sends new questions to the teacher. They say nothing about how good a real model is.

	The adapter results above	The simulation
Models	A real student and a real teacher	Mock models with scripted behaviour
Data	Synthetic databases, every SQL executed	Synthetic support tickets
Measures	Execution match on unseen databases	Student share, escalations, audit agreement, cost
Reproduce	measure_adapter.py verify	python -m sim.simulate --days 10 --requests 2000 --bad-deploy-version 3 (seed 7)
Shown as	Measured	Always labelled simulation

Simulation. The reference run’s headline figures are on the cost page: student share 0% on day 1 and 98% on day 3, a corrupted version 3 rolled back on day 4, and a per-token saving of 69% over ten days that turns into a loss by GPU-hours at that volume. These are simulation outputs.

flowchart TB
  subgraph REAL["Measured on real models"]
    A1["Exam of unseen<br/>databases"] --> A2["Execution match,<br/>base vs adapter"]
    A2 --> A3["Coverage and<br/>precision of the gate"]
  end
  subgraph SIM["Simulation with mock models"]
    B1["Synthetic tickets,<br/>10 days"] --> B2["Loop behaviour: nights,<br/>release, rollback"]
    B2 --> B3["Cost in a pinned<br/>reference run"]
  end
  A3 --> Q{{"Is the adapter good?"}}
  B3 --> Q2{{"Does the system<br/>behave as designed?"}}
  class A1,A2,A3 student
  class B1,B2,B3 outline
  class REAL studentZone
  class SIM outlineZone

Reading this diagram. The two boxes answer two different questions. The measured side tells you about model quality and is the only side that can support a claim about the adapter. The simulation side tests the machinery around the model and never claims model quality. Mixing them would make the simulation’s tidy percentages look like a result.

• The adapter, measured on real models
• Simulation, mock models

For engineers

What	Where
The panel	docs-site/src/components/SqlResults.astro, reads docs-site/src/data/sql-results.json
Layout test data	docs-site/src/data/sql-results.sample.json (made up, never shown unless copied over, and then bannered SAMPLE DATA)
Feed a real run	cp run/out/results.json docs-site/src/data/sql-results.json, then rebuild
The numbers’ code	measure_adapter.py: agreement, wilson, calibrate, build_report
Simulation	python -m sim.simulate, make reference; see the repository’s docs/TESTING.md

Common questions

The panel says results are pending. Is something broken?

No. It means the training run has not produced a results.json yet. Nothing on the page is a stand-in for it.

Why not just show one accuracy number?

One number hides the sample size and the slices. The range shows how much to trust it, and the slice by unseen industry shows whether it generalises.

Why is the exam only partly used for scoring?

Half of the exam rows set the confidence threshold and the other half grade it. The comparison between the base model and the adapter is also scored on the graded half, so both are measured on the same questions.

Can I check these numbers without a GPU?

Yes. Scoring and verify run on a CPU from the raw answers. Only generating the raw answers needs a GPU.

Are the simulation numbers a forecast?

No. They come from mock models and a pinned seed. Use them to understand how the loop behaves, not to predict savings.

Related

• A text-to-SQL adapter you can check
• Proof you can run
• Cost and savings
• Evidence you can check