Testing

pip install -e ".[dev]"
pytest -q                     # 1412 tests (test_core, test_tasks, test_registry, test_nightschool, test_release, test_system, test_hardening, test_resilience, test_scout, test_contract, test_openai_compat, test_traceability, test_live, test_cost, test_stats, test_provenance, test_receipts, test_privacy, test_erasure, test_licensing, test_feedback, test_docs_claims, test_perf, test_security, test_observability, test_readiness, test_idempotency, test_offpath, test_limits, test_admin, test_calibration, test_mock_world, test_structured, test_tools), ~120 s
pytest -q --runslow           # also the full-size Monte Carlo runs marked slow (as in CI)
pytest -q --cov=ratchet --cov=sim --cov-report=term   # with coverage (as in CI)
python -m sim.simulate --days 10 --requests 2000 --bad-deploy-version 3
python -m sim.simulate --workload research --days 8 --requests 1500   # Scout's triage + reader calls
python -m sim.simulate --bad-deploy-version 3 --compare-release      # Wilson vs the anytime-valid gate, same seed
python -m sim.simulate --calibration ltt --paired-gate              # certified thresholds and the paired gate
python -m sim.simulate --receipts --teacher-reasoning-mean 420      # savings receipts against the true cost
python -m sim.simulate --workload schema --days 8 --requests 1500     # structured outputs (the generic mock world)
python scripts/regen_sim_reference.py --check        # does the pinned reference simulation still match? (writes nothing)
python scripts/reference_check.py                    # do tickets, mixed and research still reproduce v0.4.0? (make reference, CI)

File	Tests
tests/test_core.py	66 (44 functions; one has 19 cases, one has 5; 1 slow: skipped without --runslow)
tests/test_tasks.py	81 (35 functions; one has 28 cases, one has 20)
tests/test_registry.py	29
tests/test_nightschool.py	29 (25 functions; one has 5 cases)
tests/test_release.py	37 (28 functions; one has 8 cases, two have 2)
tests/test_system.py	19
tests/test_hardening.py	104 (70 functions; one has 11 cases, two have 8, one has 6, one has 4, two have 2)
tests/test_resilience.py	87 (49 functions; one has 20 cases, two have 7, one has 6, one has 3)
tests/test_scout.py	40
tests/test_contract.py	56 (45 functions; one has 9 cases, one has 4)
tests/test_openai_compat.py	144 (37 functions; one has 87 cases, one has 22)
tests/test_live.py	59 (3 of them strict xfails for NFR-60)
tests/test_cost.py	47 (29 functions; one has 11 cases, one has 9)
tests/test_stats.py	19 (2 of them slow: skipped without --runslow)
tests/test_provenance.py	26
tests/test_receipts.py	43 (32 functions; one has 8 cases, one has 5)
tests/test_privacy.py	26
tests/test_erasure.py	54 (36 functions; one has 13 cases, one has 4, three have 2)
tests/test_licensing.py	10
tests/test_feedback.py	30
tests/test_perf.py	22
tests/test_security.py	11
tests/test_observability.py	28
tests/test_readiness.py	51 (20 functions; one has 31 cases, one 2)
tests/test_idempotency.py	25
tests/test_offpath.py	31
tests/test_limits.py	27
tests/test_admin.py	38 (26 functions; one has 13 cases)
tests/test_calibration.py	26 (24 functions; one has 3 cases)
tests/test_mock_world.py	21
tests/test_structured.py	45
tests/test_tools.py	26 (20 functions; one has 7 cases)
Total	1412

There are three layers:

1. Unit tests (tests/test_core.py, tests/test_tasks.py, tests/test_nightschool.py, tests/test_release.py, tests/test_stats.py, tests/test_provenance.py, tests/test_receipts.py, tests/test_calibration.py): the pure logic of each module (the exact statistics and the Merkle trees against hand-computed values, brute force and seeded Monte Carlo), and every Night School and release-gate rule tested directly against a small ledger with recording fakes (tests/fakes.py).
2. System tests (tests/test_system.py, tests/test_resilience.py, tests/test_scout.py, tests/test_openai_compat.py, tests/test_live.py, tests/test_cost.py, tests/test_privacy.py, tests/test_erasure.py, tests/test_licensing.py, tests/test_feedback.py, tests/test_perf.py, tests/test_security.py, tests/test_observability.py, tests/test_readiness.py, tests/test_idempotency.py, tests/test_offpath.py, tests/test_limits.py, tests/test_admin.py, tests/test_mock_world.py, tests/test_structured.py): the whole loop over simulated days, the gateway API (including OpenAI request validation and the real OpenAI SDK in-process), the HTTP clients, Scout end to end, the live-mode and operational entry points run offline (Live-path tests), the hot path’s caches, journal and time bounds (Hot-path tests), who may call each route (tests/test_security.py), request ids, failures, logs and metrics (tests/test_observability.py), the startup configuration check, liveness and readiness (tests/test_readiness.py), retry de-duplication (tests/test_idempotency.py), and audits and shadows run after the response (tests/test_offpath.py), and the spend and rate limits (tests/test_limits.py). One of them pins the reference simulation exactly (below).
3. Simulation (sim/): a reproducible offline world with a mock teacher, mock students that learn, synthetic traffic, drift and fault injection. It is used for demos, and to find design bugs before spending on GPUs.

Across the first two layers, the contract tests (tests/test_contract.py) pin what clients, the dashboard and the license rules depend on, so a rewrite of Ratchet.handle or gateway.chat cannot change it unnoticed (Contract tests).

Shared fixtures (gw, gw_env, day1_check, reference_check, reference_run) and the slow marker (--runslow) are in tests/conftest.py; make_openai_client(app) (the OpenAI SDK wired to the gateway in-process) is in tests/sdk_bridge.py; the recording provider, trainer and evaluator, FakeClock, the live-mode doubles (FakeOpenAIServer, RecordingRun, AppTransport) and the ledger builders (_version, _audit, _shadow, _examples) are in tests/fakes.py. tests/CLAUDE.md lists them.

Every test names the requirements it proves with @pytest.mark.req.

None of these touch Token Factory or a GPU. Live behaviour is checked with scripts/day1_check.py and the steps in DEPLOYMENT.md; tests/test_live.py runs the same code paths offline first.

What each test covers

tests/test_core.py

Test	Covers
test_task_agree_ignores_free_text_and_reasoning	Task.agree compares only the task’s fields (category and priority as labels), case-insensitively, and ignores <think> blocks and code fences
test_invalid_json_never_agrees	Two invalid answers never count as agreeing
test_strip_reasoning_cases (19 cases)	FR-2.7: strip_reasoning removes a full <think> pair, everything before a lone </think> (also before a fence, which it keeps), an unterminated <think> that opens what is left with what follows ('', also after whitespace, a pair or a lone </think>), but keeps one after answer text (content, such as a reply that mentions the tag), tags in upper or mixed case, nested and repeated blocks (the last </think> wins); text without tags is unchanged apart from surrounding whitespace; None is ''; stripping twice changes nothing
test_clean_answer_unwraps_fences_only_for_structured_tasks	FR-2.7, FR-2.13: for ticket-triage, one fence around the whole answer (json, JSON, none, single-line, closing fence on the last line) is unwrapped after reasoning is stripped; a free-text task keeps it. Kept as is: a fence whose body is not JSON, two fences, prose around a fence, a four-backtick fence, a body nested too deeply to parse
test_answers_that_mention_think_tags_keep_their_text	FR-2.7, FR-2.13: a structured answer that is one JSON object quoting <think> or </think> in a string is kept whole, bare, fenced, after a reasoning pair or after a lone </think>; its canonical labels and parse_json value are right, and cleaning it twice changes nothing; a free-text answer that mentions <think> keeps it; reasoning cut off while holding an object or quoting a tag is still <invalid> (review finding)
test_empty_answers_are_invalid_for_every_task	FR-2.13: an answer that is empty after cleaning (whitespace, unterminated or reasoning-only) is <invalid> for free-text and structured tasks, never agrees, and three of them score 0 with valid false
test_unanimous_confident_samples_score_high	3 identical samples at p=0.95 give agreement 1.0 and confidence above 0.9
test_disagreement_lowers_confidence	2-of-3 majority gives agreement 2/3, confidence below 0.85, and the majority answer is chosen
test_all_invalid_scores_zero	All invalid samples give confidence 0
test_length_samples_are_invalid_in_score	FR-2.13: samples with finish_reason length, content_filter, abort, unknown or another value but stop and tool_calls get no vote even when their text parses (three of them: confidence 0, valid false); with two cut-off samples and one complete one, the complete one is the answer at agreement 1/3; tool_calls votes as usual
test_calibrate_threshold_hits_target_precision	The calibrated threshold actually reaches the target precision, including ties (the regression test for bug 5 below)
test_closed_teachers_are_refused (5 cases)	gpt-5.6-sol, openai/gpt-4o, claude-opus-5-5, anthropic/claude-haiku, gemini-3.1-pro all raise LicenseError
test_open_teacher_allowed_and_unknown_refused	Ultra maps to OpenMDW-1.1; an unknown model is refused
test_examples_dedupe_and_count_and_stable_split	Duplicate requests are stored once with seen=2; the split is the same in a fresh ledger
test_bill_of_materials_lists_licenses	BOM groups by license and has a 64-character fingerprint
test_wilson_bounds	19/20 interval contains 0.95; n=0 gives (0, 1)
test_cold_start_goes_to_canary_and_bad_candidates_rejected	Golden precision 0.80 is rejected; with no live version, a good candidate goes to canary at 10%
test_regression_vs_live_is_rejected_and_good_one_shadows	A candidate worse than live on coverage is rejected; an equal-or-better one goes to shadow; live is untouched
test_canary_advances_then_goes_live	12 clean audits move canary 10 → 50; with no new audits it holds at 50 (per-stage counting); 12 more promote it to live and retire the old one
test_canary_rolls_back_on_strong_evidence	20 audits at 50% precision cause a rollback; live is unchanged; a rollback event is written
test_audit_counts_and_served_counts	FR-16.1: Ledger.audit_counts over a version’s life and after a request id counts only student-served audited rows (a teacher-served audited row is not one; an audit without a verdict counts against the version); version_live_quality agrees; served_counts includes a stand-in (degraded) answer and leaves teacher-served rows out
test_gen_params_from_a_validated_request	FR-1.13: GenParams.from_request takes every generation field of a validated request (max_completion_tokens as max_tokens, stop as a tuple, an unknown field in extra) and to_body sends them by OpenAI’s names, never user, safety_identifier, store or metadata; empty tools and logit_bias count as not sent; only keeps the object when nothing is dropped; an extra key never replaces a standard field or sets model, n or messages
test_param_allowlists_parse_and_refuse_what_ratchet_never_forwards	FR-1.13, INV-1 (and FR-1.14: the default teacher list includes logprobs and top_logprobs; FR-1.41: and the tool-call fields, so it is every standard field): empty is the default, none nothing, all the standard fields, all,top_k adds a vendor field, max_completion_tokens means max_tokens, names match in any case; model, messages, n, stream, user, safety_identifier, store, metadata, service_tier, functions and a name that is not an identifier raise, and so does a near miss of a standard field (max_token, stops, temprature, tool: “did you mean”), while vendor fields (top_k, min_p, min_tokens, repetition_penalty, max_new_tokens, guided_json) pass; the default lists leave out tool calls and the teacher’s logprobs
test_gen_policy_gives_each_role_its_parameters_and_caps	FR-1.13, NFR-3: the student gets the shaping fields capped at 1,024 and never the client’s temperature or logprobs; the serving teacher gets every standard field by default, tools included (logprobs only when sent; all of them when allowed; a vendor field only when listed), at temperature 0 when none was sent, capped at 8,192; audits and arbitrations run at 0 without CHECK_DROPS, keeping limit, stops, format, tools and effort; the headroom adds to the teacher’s limit under the cap and visible_limit says when an answer can exceed the client’s; a field outside an allowlist is never sent but the caps apply; 0 means no cap; bad caps raise
test_stored_conditions_join_the_hash_only_when_set	FR-1.13, FR-4.3: no conditions, sampling fields only, or the default text format hash like v0.4; response_format and stop each give another hash; a tuple and a list of the same stops are one example; GenParams.from_stored reads stored conditions back and leaves out values of the wrong shape
test_v04_example_hash_unchanged	FR-1.13, FR-4.3: literals computed with the v0.4 code (a Unicode ticket and a plain request) still come out of example_hash, and add_example with only a seed stores that hash with no conditions
test_redact_json_redacts_string_values_and_keeps_keys	FR-1.13, INV-12: emails in a stop string and a tool description become [EMAIL] (counted), keys and numbers stay, the input is not changed, and 5,000 levels of nesting are fine
test_served_logprobs_spell_the_served_text_and_drop_reasoning	FR-1.14: providers.served_logprobs returns the tokens as they are when they spell the served text, in OpenAI’s shape, without changing the provider’s list; drops <think> reasoning, a fence and the whitespace clean_answer strips (trimming a token only by whitespace, bytes recomputed); returns only the answer’s tokens when a server lists reasoning tokens its content leaves out; None when a token holds both the answer and a tag, when reasoning was removed from the middle, or when the tokens do not spell the answer (U+FFFD parts), and when the model sent none; [] for an empty answer
test_served_logprobs_check_every_entry_and_keep_the_clients_top_logprobs	FR-1.14: top_logprobs cut to the client’s count (none for 0 or unset), malformed alternatives left out; a malformed entry (no logprob, a string, a boolean, NaN, +Infinity, no token, an unpaired surrogate) makes the list None; -Infinity is -9999.0 (MIN_LOGPROB); the server’s bytes are kept when they are bytes; the fakes’ token_entries spell their text
test_majority_gives_n_agreeing_choices_with_the_best_sample_first	FR-1.14, FR-2.2: confidence.majority gives the best scored sample first, then the other agreeing samples by token probability, never the minority; None when fewer than n agree, a cut-off or invalid sample never fills a choice, and a repeated object counts twice
test_gen_plan_for_n_choices	FR-1.14, FR-1.13: GenPolicy.plan(params, n): the serving teacher at 0.7 (CHOICES_TEMPERATURE) for n > 1 without a client temperature that reaches it (also when the allowlist leaves temperature out), the client’s otherwise; audits at 0; the student’s sampling unchanged; the teacher’s logprobs only when asked, never in an audit; a bad n is a ValueError
test_certificate_frontier_and_per_task_from_held_out_points	FR-16.3, INV-6: 62 held-out points at threshold 0.85: 52 served, 2 errors, lcb = cp_lower(50, 52, 0.05) (0.884), certified at 0.85 and not at 0.9; serving nothing gives 0 and ltt_trace only for Learn-then-Test; the frontier’s ten rows at delta / 10 with coverage never rising with the threshold; per-task counts and bounds; the outcome vector S/W/E
test_paired_counts_on_hand_built_vectors	FR-16.3, FR-6.30, INV-4: ten hand-built rows give exact discordances, flips (an escalated row the candidate answers wrong is a negative flip), the served flip rate 2/5 with its Wilson interval and both McNemar p-values; 4 rows only live serves give 0.0625; bad vectors raise
test_recheck_recomputes_every_number_from_the_counts	FR-16.3: a JSON round trip of a full eval rechecks clean (1e-12 of float noise is not a mismatch); an edited lcb, errors (also against the test vector and the per-task sums), target, frontier row, uncorrected frontier deltas, reordered rows, a paired p-value, impossible counts, a paired n or flip count the candidate’s test vector rules out, a string count and a bad vector each give their line
test_recheck_follows_the_ltt_trace	FR-16.3: Learn-then-Test’s trace rechecks; a threshold below the lowest one it kept and a forged p-value are named
test_certificate_status_and_section	FR-16.3: section of an eval with and without a certificate; the status follows the latest certificate, falls back to the signed one, is no certificate without one and suspended under an open drift alarm
test_versions_filter_by_state_in_order	FR-16.3: Ledger.versions(states) selects in SQL in id order (a generator, an empty list), version(vid) reads one row, live and by_state the latest
test_replay_hajek_estimate_and_cost_on_a_hand_computed_example	FR-16.5: six requests worked out by hand: Hajek precision 1/3 at the live threshold (weights 1 / label_p, an unaudited served answer has no label), 3/7 and 3/8 below it with the escalations’ labels, Kish n_eff 1.8, the Wilson interval at it, cost 0.14 against a 0.158 baseline (0.126 at audit rate 0, which leaves precision alone), an escalation served at 0.35 audited at the version’s mean logged rate, errors per 1,000, and at(rows) (each request at its logged threshold) equal to the replay at 0.6; finite JSON
test_replay_coverage_is_exact_and_the_frontier_never_gains_coverage	FR-16.5: coverage at eight thresholds equals the count of requests at or above them, never counting a request with no valid answer or one that could not fill its n choices; the 71-row frontier never gains coverage and its rows equal at to the bit; grid counts in whole steps and refuses bad grids; a threshold of 0, above 1.01, NaN or a boolean is refused
test_replay_without_positivity_gives_no_precision_above_the_live_threshold	FR-16.5: with audit rate 0 every threshold that serves a request gives a null precision with the positivity reason, while coverage and cost stay known; nothing served and no labels give their own reasons
test_replay_precision_is_unbiased_under_the_designed_labelling	FR-16.5: 2,000 requests with known correctness, logged at threshold 0.6 (labelled with probability 1 below, 0.1 above), 120 seeded replicates of the labelling draw: the mean replayed precision at 0.4 and 0.8 is within 3 standard errors and 0.01 of the truth, while the unweighted mean of the labels is more than 4 points off
test_replay_precision_is_unbiased_under_the_designed_labelling_full_size	The same with 500 replicates (slow)
test_replay_suggests_the_lowest_passing_grid_threshold_or_none	FR-16.5: suggest scans CERT_GRID down from 0.95 and returns 0.80, the last threshold whose interval’s lower bound reaches 0.97, with the advisory text; five rows at the top make it None although lower thresholds would pass (fixed sequence); money_left_on_table against 0.9 (nothing when every row was audited as logged, the savings and dollars at a 10% audit rate) and None when the current threshold is already lower; a target of 1 is refused
test_ratio_estimate_recovers_the_cost_ratio	FR-16.5: stats.ratio_estimate recovers 1.3 within 3 standard errors on 400 noisy pairs, and replay.ratio from audited served answers weighted by 1 / label_p at mixed audit rates (escalations left out), with its interval; an unaudited served answer is priced at R times its estimate; insufficient (R 1.0) below 30 audited answers

tests/test_tasks.py

Field comparators and task specs (FR-2.14). Pure logic on compare.FieldSpec and tasks.Task, plus small ledgers and RecordingProvider for what the cascade, the golden eval and the dataset do with an answer the spec refuses.

Test	Covers
test_exact_is_the_v04_rule	exact: strings stripped and lower-cased; 1 and 1.0, true and 1, lists in another order differ
test_label_normalises_and_checks_its_vocabulary	label: NFKC, case, trailing sentence punctuation and separators; a value outside values is invalid; c# is not c
test_label_aliases	P1 is urgent, SEV-2 is high; an alias outside the vocabulary is refused
test_bool_reads_words_and_json	yes/no/y/n/1/0 and JSON booleans; maybe and 2 invalid
test_number_reads_money_separators_and_tolerances	1,234.50 == 1234.5 with abs_tol 0.01; 1.234,50 € with decimal=","; (5) == -5; relative tolerance; percent as number or fraction
test_parse_number_cases (20 cases)	Thousands groupings (also Indian, apostrophe, space), US$, ISO codes, exponents, signs; ambiguous 1,5, lower-case codes, units, NaN, infinities, 1_000, double signs refused
test_number_refuses_non_finite_and_non_numbers	Strings, booleans, lists, 1e999, NaN and an integer too large for a float are invalid
test_ordinal_tolerance	low and medium agree with tolerance 1, low and high do not; an unknown label is invalid; order is required
test_text_modes	loose with suffixes (Acme Corp. == ACME corp == Acme, Inc. == Acme,Inc., a two-token suffix), punctuation as a word break (hello-world), apostrophes and initials joined (O'Brien, U.S.A.), a suffix that is the whole name kept, exact, token_set
test_date_formats	Six written forms == 2026-10-03; month first by default, day_first; the local day of a datetime; UTC datetimes; impossible dates invalid
test_language_codes_and_names	English == en == eng == en-US, Deutsch, 日本語, locale tags with scripts, regions and variants; unknown languages invalid; values restricts; prose that starts with a code (It is English, no idea, in progress) and tags led by a 639-2 or retired code are invalid, so It is English. and It is French. do not agree
test_set_order_case_and_jaccard	Order, case and duplicates ignored; Jaccard 0.8 boundary; empty_values; split with number elements; a string without split is invalid
test_records_unordered_keyed_and_ordered	Unkeyed and keyed pairing across order, ordered=True, a changed field or length, duplicate keys and a missing sub-field invalid
test_records_without_a_key_pair_in_any_order	Records with a tolerance pair by a maximum matching: the listed order never decides ([0, 1] and [1, 0] both agree with [0.5, 2]), a 50-record chain with one pairing, labels and nested records grouping first
test_ranking_top_k	Top-3 in order and as a set; an id ranked twice is invalid
test_root_label_text_format	A text task’s $ label: Positive. == positive, reasoning removed; other paths refused
test_top_level_array_with_dollar_path	A $ set over a top-level array (fenced, after a sentence, after reasoning), a $ number, dot paths into nested objects
test_empty_json_is_invalid_when_fields_required	{} is invalid for ticket-triage (score 0), but still valid for a compare_keys task, as in v0.4
test_out_of_vocab_label_invalid	INV-16: check reasons (invalid field category, invalid field priority, invalid answer, empty answer); every answer the mock world writes is valid
test_nullable_field_accepts_null	Required, nullable and optional fields and how null and missing agree
test_bad_specs_fail_when_defined (28 cases)	Reserved and unknown types, bad paths, options on the wrong type (values, tolerance, abs_tol, mode, k, min_jaccard, day_first), duplicate labels, an alias that is a label of the vocabulary or names two labels, tolerances on set elements, record keys, the value, field and format limits, compare_keys with . or $, policy keys and ranges, tools
test_records_nest_at_most_four_deep_and_hold_at_most_200	Depth 4 allowed and 5 refused; 200 records valid and 201 invalid; the 200 count across nesting
test_spec_hash_covers_the_rules_not_the_name	Same rules, same hash, whatever the name or source; a vocabulary or pinned changes it; dataclasses.replace recomputes it, also on a compare_keys task; policy is read-only
test_legacy_compare_keys_canonical_byte_identical	FR-2.6: a compare_keys task’s canonical and agree equal v0.4’s on 25 answers with fences, <think>, nulls, 1 vs 1.0, nested values, NaN and deep nesting
test_pairwise_score_equals_counter_score_for_exact_tasks	FR-2.2: on 600 seeded random sample sets (invalid and cut-off samples, ties, missing logprobs), the pairwise score equals v0.4’s Counter score: the same answer object, confidence, agreement and token probability; majority’s choices agree
test_builtin_task_agreement_decisions_unchanged	On mock answers for the three built-in tasks, the typed specs decide agreement, confidence and the best sample exactly as v0.4’s exact keys
test_tolerance_non_transitive	0, 0.6, 1.2 with abs_tol 1: the middle sample, agreement 1.0, even at a lower token probability; majority; two groups tie and the first wins; with four samples the served sample’s agreement is its own
test_score_never_raises_when_a_key_does_not_match_itself	INV-7: a comparator that is not reflexive gives confidence 0 instead of raising on the request path
test_default_compares_json_canonically	Key order, spacing (inside strings too, as in v0.4), case, NFKC and 1 vs 1.0 agree when both are JSON; list order and true vs 1 do not; the text rule otherwise; deep nesting compared as text; a json task without fields
test_comparators_are_linear_on_adversarial_answers	Long labels, suffix runs, token sets, numbers, dates, languages and sets normalise at once; 200 records pair reversed, shifted within a tolerance, and with ten tolerant fields in the worst order, all under 2 s
test_comparators_never_raise_on_junk	INV-7: 3,000 seeded junk values (odd strings, NaN, huge integers, nested lists and objects) never make a comparator raise, and every valid value matches itself
test_invalid_teacher_answer_served_but_not_stored	FR-2.13, INV-16: an out-of-vocabulary or incomplete teacher answer is served, not stored, skip_example names the field; an invalid student answer is never served; store=False stores nothing without a skip
test_invalid_audit_not_counted	INV-16: an audit whose teacher answer the spec refuses is audited=0, not stored
test_cascade_eval_skips_invalid_golden_refs	INV-6: rows with invalid references are not sampled and are counted in golden_invalid_refs; the halves alternate over the scored rows
test_dataset_rows_skip_invalid	INV-2, INV-16: train rows stored under a looser spec are left out and counted (skipped_invalid, also in manifest.json and in the night’s trained or skip event); golden rows stay

tests/test_registry.py

The task registry (FR-2.15): specs from RATCHET_TASKS_FILE and the admin API, the admin key, operator-only pinning, the unknown-task policy, client allow-lists, pinned rows left out of Night School, and spec changes. Gateway tests use gw_env (mock) or a live Runtime on a file database with the teacher behind FakeOpenAIServer; unit tests use taskspec, tasks and a small ledger. The registry is module state, so tests/conftest.py resets it after every test.

Test	Covers
test_tasks_file_loads_and_registers	FR-1.3, FR-2.9: a file’s tasks (one extending and adjusting the built-in default, one pinned) listed by GET /ratchet/tasks with their source and spec; a request distilled under the file task, the pinned one answered by the teacher and not stored
test_tasks_file_in_yaml	The same in YAML (skipped without PyYAML); clients checked against the API key names; a syntax error with its line; bare yes/no read as booleans refused
test_tasks_file_unknown_key_fails_startup_with_path	A bad file stops Runtime and python -m ratchet.live with every error and its path (tasks[0].feilds); a missing file, broken JSON and NaN refused
test_yaml_without_pyyaml_message	A .yaml file without PyYAML is an error that says so; JSON needs nothing extra
test_spec_errors_list_every_problem_with_its_path	One document with 35 mistakes (unknown keys and options, options for another type, bad paths, ranges, reserved types and formats, policy, match, routing, clients, names, duplicates, extends cycles and missing bases, records nested too deep) gives exactly 35 errors, in file order
test_spec_parser_never_raises_anything_but_spec_error	3,000 seeded junk specs (wrong types, unhashable values, unknown types, nesting) in parse_spec and parse_document give SpecError or a task, never another exception
test_specs_round_trip_and_every_option_is_checked	Every FieldSpec option has a check; a spec with every type, and every built-in task, survives document and parse_spec with the same hash; description, clients and match leave the hash alone, field order does not; the ledger keeps the order
test_extends_copies_and_merges	extends within a file (forward too), from a built-in task, a file adjusting a built-in task of the same name, a copy of that, through the API on the task’s own name; fields, policy and match merge, null removes, name and description are not inherited
test_registry_precedence_and_sources	A file task replaces a built-in one; the API never replaces a built-in or file task; an auto task never shadows a registered one; adhoc and built-in tasks cannot be registered; removal and restore; names_for and allow-lists; an adhoc task is never pinned
test_registry_reads_never_see_a_half_made_registry	Lock-free reads under 4 writer threads registering, removing and resetting
test_registry_cap	MAX_TASKS refuses a new name but not a replacement; a file with too many tasks; 409 for the mock cap and the registry cap
test_put_task_requires_admin_key	INV-13, FR-1.7: 401 without or with a wrong admin key, 403 with an API key, for PUT, DELETE and validate; reads stay open; only the admin’s name reaches the ledger and the events (the database dump holds no key)
test_admin_keys_are_their_own	INV-13: a key in both RATCHET_ADMIN_KEYS and RATCHET_API_KEYS stops startup; errors name the right variable
test_admin_disabled_in_live_without_keys	Live mode without admin keys: 403 admin API disabled for every write; mock mode: open, as mock-admin
test_put_builtin_task_forbidden	INV-13: 403 for a built-in or file task (PUT and DELETE), 404 for an unknown one; validate says the same
test_client_cannot_pin_via_header_metadata_model_or_schema	INV-10, FR-2.9: pin-like headers, metadata, model and a response_format with x-ratchet never pin; the tasks are distilled; nor can a header unpin research-write; Task refuses a pinned adhoc or auto task
test_task_lifecycle_through_the_admin_api	Create, an identical PUT (changed: false, no event), traffic, a changed spec (task_changed with both hashes), a metadata-only change, recalibrating, every 422 (bad JSON, NaN, an unpaired surrogate, another name, not an object, a bad option, 65 fields, a bad URL name), delete (examples kept), the name then following the unknown-task policy and listed as unregistered
test_api_task_persists_across_runtime_restart	A live Runtime on a file database: an API task registered again after a restart with the same hash and updated_by; a change logs both hashes; a deleted task stays gone, its examples kept
test_stored_task_that_no_longer_parses_is_skipped	A stored task that no longer parses, one whose name the file now has, and a garbled row are skipped with task_skipped events; the gateway starts
test_validate_endpoint_has_no_side_effects	One spec and a document, valid and not; a document may adjust a built-in task, a single spec may not; the registry, the events and the tasks table are unchanged
test_concurrent_admin_writes_keep_the_registry_and_the_table_in_step	8 threads creating, changing and deleting tasks: the registry, the table and the event counts agree
test_unknown_tasks_teacher_policy_stores_nothing	FR-1.3, INV-10: teacher answers an unregistered task with the teacher only (route unknown_task, no student call even with a live and a shadow version, nothing stored); registered tasks still distil; a typo stops startup; the same through the gateway
test_unknown_tasks_reject_policy_400	FR-1.3: reject is a 400 unknown_task listing the registered tasks (param for metadata), logs nothing and uses no quota; direct callers get the same; at most 20 names listed
test_clients_allow_list_403	INV-13: a clients name that is not an API key is a 422; another key gets 403 task_not_allowed before the quota; the named key is served; rejection messages list only tasks the key may use
test_pinning_task_excludes_existing_examples_from_dataset_and_eval	INV-10, FR-2.9, INV-2: pinning a task with data leaves its rows out of dataset.rows, the manifest (excluded_pinned), cascade_eval and run_night’s trainer and evaluator; the eval records spec_hashes; unpinning brings them back
test_spec_change_forces_teacher_until_recalibrated	FR-2.3, INV-4: a changed spec makes the threshold 1.01 for that task only; the next night (even without enough data to train) re-scores the live version and records the new hash (rescored), and the student serves again; nothing to re-score the night after; a failed re-score keeps the task on the teacher (rescore_failed)
test_fixture_paths_resolve_inside_the_fixtures_dir	resolve_fixture refuses .., absolute paths, empty and NUL paths and a symlink out of RATCHET_FIXTURES_DIR

tests/test_nightschool.py

The invariant tests for Night School (INV-1 to INV-6). Each runs run_night or cascade_eval on a small ledger with a RecordingTrainer and a RecordingEvaluator, so it can assert on exactly which rows reached the trainer and which golden rows were scored.

The last six check that the student trains on the prompt it is served (FR-8.42): the training file’s rows, the vendored Nemotron 3 Nano chat template rendered offline with jinja2 (tests/fixtures/nano4b_chat_template.jinja), train.py’s checks and loss setup, and the pinned training image.

Test	Covers
test_training_targets_are_teacher_answers_only	INV-2: the teacher answers JSON that is valid for both tasks used (INV-16 would skip anything else); a student that answers STUDENT-ONLY with high confidence serves 200 requests with audit_rate=1 and a shadow present, then 200 more without audits where a disagreeing shadow (SHADOW-ONLY) makes the teacher arbitrate and unparseable answers escalate. All 400 stored examples and dataset rows hold exactly the teacher’s answer to their own request, and neither marker appears anywhere
test_add_example_called_only_from_allowed_callers	INV-2, INV-10: an AST scan of ratchet/ finds every use of add_example (a call, a bound method passed on, or getattr(..., "add_example")); each must sit in a function in ALLOWED_ADD_EXAMPLE_CALLERS (today only Ratchet._store_example), and every listed caller must still use it (so moving a call site replaces its entry rather than adding one beside it)
test_add_example_guard_sees_indirect_uses	The scanner itself (no requirement marker: it tests the tool, not INV-2), on a throwaway package: a direct call, getattr inside a nested function and a bound method in a module-level lambda are found and named; a docstring that mentions add_example is not
test_golden_rows_never_reach_the_trainer_except_frequent	INV-3: the trainer gets the train split plus the golden requests seen at least FREQUENT (3) times; the evaluator gets the other golden rows; the two never overlap; a golden request seen twice stays held out; the stored split never changes
test_every_variant_of_a_prompt_shares_its_split_and_count	INV-3, INV-6, FR-4.3, FR-1.13: 200 prompts, each stored bare and under three sets of conditions: every prompt’s four examples share one split (both splits occur), and the bare example’s split is the one it gets alone (v0.4’s); a request seen once bare and twice in JSON mode is frequent, so both variants move to training together
test_examples_cut_at_a_stop_are_held_out_of_training	INV-3, FR-4.1, FR-5.1: train.py ignores stored conditions, so a row the client’s stop cut is left out of the train rows (bare and JSON-mode rows stay); as a golden row it is kept, with its stop
test_closed_teacher_rows_are_dropped_at_dataset_build	INV-1: rows from openai/gpt-4o and an unregistered teacher in the ledger never reach dataset.rows, train.jsonl, golden.jsonl, the trainer, the evaluator or the adapter’s BOM
test_night_skips_with_reason (5 cases)	FR-4.4, INV-5: 49 new examples, 40 new since the last version, no golden set, a shadow in flight and a canary in flight each skip with their reason and a skip event; the trainer and evaluator are never called (not even to re-score the champion) and no version changes
test_retrain_from_base_on_cumulative_data	FR-4.2: three nights (the first with exactly 50 new examples). Each candidate passes and is released to live through the gate, so nights 2 and 3 train while the previous night’s adapter is the champion; every night still trains from the base model (each TrainCall.base_model and Version.base_model), never the live adapter. Each night’s rows contain every earlier night’s rows, and their counts (50, 100, 175) equal each version’s train_examples
test_cascade_eval_measures_on_held_out_half	INV-6: with right answers on even golden rows and wrong ones on odd rows at the same confidence, precision is 0.0 (graded on the odd half only), test_n is half of golden_n and all report keys are present; the mirror image calibrates on wrong answers and serves nothing (threshold 1.01)
test_cascade_eval_passes_row_params_to_sample	FR-5.1, FR-1.13, INV-6: the sampler gets each golden row’s stored conditions, None for a row without them
test_cascade_eval_scores_a_refused_row_as_not_served	FR-5.1, INV-7, INV-6: three of six golden rows the sampler refuses with a 400 score as confidence 0 and wrong (refused_n 3, agreement 0.5, coverage 2/3 of the odd rows); a 401, 404, 429 or 503 still raises
test_golden_eval_runs_under_production_conditions	FR-5.1, FR-1.13, FR-4.1: an example stored with a stop and JSON mode comes back from dataset.rows with them; the router (with RATCHET_STUDENT_MAX_TOKENS=256), LiveEvaluator and MockFleet send the student exactly the same arguments for those rows
test_champion_rescored_on_same_golden_set	INV-4: the live version is evaluated first, on the same golden list as the candidate; its stored eval now holds the re-checked precision and coverage, with rechecked_day = the ledger’s day; the candidate, which would regress against the stale eval, is admitted to shadow against the fresh one; the candidate is trained from the base model, not from the champion (FR-4.2)
test_cascade_eval_certifies_on_the_held_out_half	FR-16.3, INV-6, FR-5.2: 80 rows of two tasks with seeded confidences: the threshold equals the pre-change point rule exactly, the certificate’s counts are the odd rows’ at it (its precision the eval’s own), lcb ≤ precision, the test vector has test_n letters, the frontier’s coverage never rises with the threshold, per-task counts add up, recheck is clean; another target and delta change only the certificate
test_cascade_eval_ltt_picks_a_grid_threshold	FR-16.3, INV-6: 120 right calibration rows certify every grid threshold (served at 0.5, lcb 0.05^(1/120)); 10 rows certify nothing (1.01, one trace step); confident right rows and unsure wrong ones stop at a grid threshold above the unsure ones
test_cascade_eval_refuses_bad_certification_settings	An unknown calibration, a target of 1 and a delta of 0 or True raise
test_run_night_pairs_the_candidate_with_the_rescored_live_version	FR-16.3, FR-6.30, INV-4, INV-14: the candidate’s eval and its signed BOM carry paired against the live version (the champion’s re-score carries none); vectors of different lengths, a missing vector and empty vectors give none, and so does a cold start
test_paired_gate_admits_what_the_unpaired_rule_rejects_on_noise	FR-6.30, FR-16.3, INV-4: the reference run’s v2 (70 of 74 rows against live’s 74, simulation): the unpaired rule rejects it, the paired gate admits it with serves -4 of 74 vs live v1 (McNemar p=0.06)
test_training_rows_carry_reasoning_off	FR-8.42, INV-2, FR-4.1: every row of train.jsonl (from dataset.build and dataset.write_training, which the live trainer uses) has prompt (the stored messages), completion (the teacher’s stored answer as the one assistant message), chat_template_kwargs == {"enable_thinking": false} and no messages, keeps params and provenance, and no row contains <think> or </think>; train.check_rows accepts them; dataset.TRAIN_TEMPLATE_KWARGS (read-only), train.TEMPLATE_KWARGS and the endpoint’s DEFAULT_TEMPLATE_KWARGS are equal; answers with a literal tag are left out of training and counted (skipped_think_tag), stay golden rows, and the night’s event says how many
test_training_prompt_is_a_prefix_and_matches_serving	FR-8.42, FR-5.1 (skipped without jinja2, which the dev extra installs): three requests (one turn, a system prompt, a conversation with an assistant turn) escalate through the router and are stored; the vendored Nano 4B template (sha256 checked), compiled as transformers compiles it, renders each training row’s prompt ending in <|im_start|>assistant\n<think></think>, prompt + answer as exactly that prompt + the answer + <|im_end|>\n, and the prompt equal to the one rendered for the router’s student call with the endpoint’s default arguments. Without the argument the prompt ends in <think>\n and is not a prefix (the bug), and train.check_prefix raises TemplateMismatch naming the row; so does an answer with a think tag. Messages filled with None keys (as datasets hands them to TRL) render the same prompt; a written prompt that differs, or a different row count, raises
test_train_args_have_no_assistant_only_loss	FR-8.42, FR-4.5: an AST scan of train.py finds one SFTConfig with completion_only_loss=True and no assistant_only_loss (or False), main calling check_rows, check_prefix, columns and SFTTrainer, and no Unsloth import; train.columns gives TRL only prompt, completion and chat_template_kwargs
test_dockerfile_is_pinned	FR-8.42: deploy/Dockerfile.train has no || true and trl==1.16.0; every package of every pip install carries ==, and together they equal deploy/train-versions.txt (no Unsloth); every step after torch reads that file as constraints; the kernel import check comes after the Mamba install; python-is-python3 is installed; train.LIBRARIES names exactly the pinned packages; the train extra in pyproject.toml agrees with the list
test_train_records_library_versions	FR-8.42, FR-4.5, INV-14: train.versions records Python, CUDA and each pinned package (None when missing, from an injected lookup) in LIBRARIES order; bom.EXTRA_KEYS has no section for them, so versions.json stays beside the adapter; ratchet_manifest.json records the template arguments
test_train_refuses_rows_that_would_train_another_prompt	FR-8.42: train.check_rows refuses, naming the row and never its text, an old messages row, missing or other template arguments, a completion that is not one assistant message with text, an answer with a think tag, an empty or malformed prompt, provenance without a license, a row that is not an object, and an empty file

tests/test_release.py

Test	Covers
test_gate_rejects_low_coverage	Golden coverage one point under min_coverage is rejected; an eval without coverage is rejected (fails closed); exactly min_coverage passes
test_gate_tolerance_boundary (2 cases)	For precision and for coverage: a candidate exactly tolerance worse than live is admitted to shadow; a hair more is rejected as a regression
test_admit_retires_older_inflight	A rejected candidate leaves the shadow and canary in flight alone; an admitted one retires both (traffic 0, one retired event each) and is the only version in flight
test_shadow_promotes_and_both_rejections	99 would-serve answers hold; at 100, precision exactly 95% and live’s coverage it moves to canary 10%. The count is would-serve answers, not shadowed requests: 150 requests with 75 would-serve answers hold, 200 with 100 promote. 90% precision is rejected; 50% coverage against a live student at 100% is rejected, but passes against a live student that also serves 50%
test_shadow_coverage_is_not_compared_with_requests_the_live_student_never_answered	FR-6.4, FR-2.11 (regression): rows where the live student never answered (route student_breaker or student_error) stay in the shadow’s n and coverage but not in live_coverage, so a shadow serving 50% is rejected against a live student that served all it answered; the same pattern with the live student unsure (route confidence) is live coverage of 50% and promotes
test_shadow_coverage_counts_an_n_choices_escalation_as_live_served	FR-6.4, FR-1.14: a request the live student was confident about but escalated because fewer than n samples agreed (route n_choices) counts as served in live_coverage, so a shadow serving 50% is rejected against a live student that would have served them all
test_shadow_gate_counts_unarbitrated_disagreements_against_the_candidate	FR-6.4, FR-2.10: 100 would-serve answers whose arbitrated ones all agree, but 10 of which the teacher could not arbitrate (shadow_agree NULL), give precision 90%, not 100%, and are rejected with “(10 unarbitrated counted as disagreements)”; 2 of 100 give 98% and promote, and the event says so; 50 rows without a shadow verdict (shadow_serve NULL) change neither n, coverage, the evidence count nor precision
test_canary_holds_without_strong_evidence	18 of 20 audits (90%, under the 92% floor, Wilson upper bound above it) neither roll back nor advance, over several ticks; 20 more agreeing audits move it to 50%. The rollback boundary (FR-6.6): 16 of 20 (Wilson upper bound 91.9%, just under the floor) rolls back to live, 17 of 20 (94.8%) only holds
test_rollback_without_live_returns_traffic_to_teacher	A cold-start canary at 50% audited precision is rolled back with “traffic back to the teacher”; afterwards 50 requests are all answered by the teacher and the student is never called
test_shadow_arbitration_calls_teacher_once_and_stores_example	A shadow that would serve a different answer than the live student triggers exactly one teacher call, whose answer is stored as the example, while the user keeps the live answer; agreeing or unsure shadows trigger none; with an audit, the audit is the reference and no second call is made; a shadow whose endpoint is down (raise_on) leaves the request untouched, with no shadow record and no extra cost (INV-7). Reported cost includes the shadow’s samples, the arbitration and the audit (NFR-3)
test_canary_traffic_split	Ratchet._pick_version with a seeded RNG: over 4,000 picks at 10% and at 50%, the canary’s share is inside the 99% Wilson interval; 0% never picks it, 100% always does, and at cold start the rest of the traffic gets None (the teacher)
test_tick_runs_each_step_once_in_order_with_the_live_version_it_started_with	tick calls _tick_drift, _tick_live, _tick_shadow, _tick_canary and _tick_watchers once each, in that order, the middle three with the live version read before any step (a canary promoted in the tick retires that one); the drift and watcher steps write nothing yet
test_policy_refuses_values_out_of_range (8 cases)	ReleasePolicy refuses a live_floor of 1 or 0, a precision above 1, a negative coverage or audit count, a fractional or boolean count and an infinite tolerance, naming the variable
test_sequential_canary_rolls_back_after_two_failures	FR-16.1, US-6: with sequential=1 and min_audits=15, no audit is no decision, 0 of 1 agreeing (e-value 4.22) holds, 0 of 2 (29.8 >= 20) rolls back to live with the exact event text (counts, e-value, threshold, alpha, 95% CS [0.001, 0.931]); the Wilson rule holds on the same two audits
test_sequential_rollback_at_cold_start_returns_traffic_to_the_teacher	FR-16.1: a cold-start canary rolled back by the e-process sends traffic to the teacher
test_sequential_rollback_counts_every_audit_of_the_canarys_life	FR-16.1: rollback is one e-process over the canary’s whole life: 40 agreeing audits at 10% still count at 50%, so 2 failures (which alone would reach 29.8) hold and it takes 11 (40/51, e-value 23.2)
test_sequential_promotion_needs_evidence	FR-16.1, FR-6.5: with promote_evidence=1, 20 agreeing audits (2.57) hold and 60 (27.6) advance to 50% with the old event text as a prefix; the count restarts per stage, 55 agreeing audits (19.996) hold and 56 (21.3) promote to live; the point rule promotes on 20
test_futility_retires_inconclusive_canary (2 cases)	FR-16.1: 45/50 under the point rule and 55/60 under evidence-based promotion: one audit short of max_stage_audits holds, at the cap the canary is retired with an inconclusive event (counts and confidence sequence), live untouched, no rollback
test_futility_counts_audits_not_time	FR-16.1: a canary that serves 500 answers without an audit holds
test_sequential_policy_refuses_settings_that_decide_nothing	FR-16.1: promote_evidence without sequential, a cap below min_audits, a cap of 50 for evidence-based promotion (“needs 56 audits that all agree”) and the default cap at a 0.99 floor (“needs 450”) are refused; 56, no cap, a cap equal to min_audits and a cap without the sequential rule are fine; alphas 0 and 1 and switches of 2 are refused, naming the variable
test_live_alert_uses_the_e_process	FR-16.2, FR-6.7: the default alert counts audits since the version went live (5 failures as a canary do not count), holds at 0/2 (29.8) and fires once at 0/3 (268 >= 100) with the exact text; a version above the floor is never alerted on over 400 audits; alert_eprocess=0 keeps the Wilson rule (nothing before 20 audits)
test_evidence_is_recomputed_from_the_counts	FR-16.1, FR-16.2: ReleaseGate.evidence equals the stats functions of the life and stage counts, key by key; a live version’s counts; finite JSON for 3,000 failures (e-value capped at e^690, a vanishing one still a number); no audits give 1 and [0, 1]
test_canary_audit_rate_focuses_audits_on_the_canary	FR-16.1, FR-6.5: with RATCHET_CANARY_AUDIT_RATE=1 and an audit rate of 0, every canary answer is audited and no live one, label_p logs 1.0 and 0.0; exactly two random draws per request with and without it (a counting RNG), and the same requests reach the canary
test_label_p_is_the_designed_labelling_probability	FR-16.1: label_p is NULL at cold start, 1.0 for an escalation the student answered, and the audit rate (0.25) for a served answer
test_stats_versions_carry_the_gates_evidence	FR-16.1, FR-16.2: /ratchet/stats gives the canary and the live version evidence equal to ReleaseGate.evidence, null for a retired one; the mock gateway reads RATCHET_SEQUENTIAL and RATCHET_MIN_AUDITS (15 when unset)
test_min_certified_precision_rejects_a_weak_certificate	FR-6.31, FR-16.3: 46/48 (bound 87.5%) is rejected at a 90% floor with the exact reason and admitted at 85% or off; a stored lcb of 0.99 is ignored (recomputed), no or a broken certificate fails the floor, and the precision floor still comes first
test_certified_versions_say_so_in_their_gate_events	FR-16.3: the cold-start canary event and the shadow event append ; certified >= 95% (lcb 96.4%, delta 0.05) (or 96.5% at another target and delta) for a certified candidate, and nothing for an uncertified one or a claim its counts do not back
test_paired_gate_rejects_only_large_significant_regressions	FR-6.30, FR-16.3: 2 of 63 rows within noise (p 0.42) are rejected by the unpaired rule and admitted by the paired gate with its event; a record against another version or with unreadable counts falls back; 10 rows lost (p 0.001) and 8 rows newly wrong of 100 (p 0.004) reject with exact reasons, the same 8 of 1,000 do not; 6 negative flips of 60 reject at max_nfr 0.02 (lower bound 4.7%) and pass at 0.05 or off

tests/test_system.py

Test	Covers
test_loop_learns_and_saves_money	6 days × 1,500 requests with no drift: day 1 is 0% student; final day is above 80% student and above 60% savings; a version reaches live; daily audited agreement never falls below 85%
test_bad_deploy_is_rolled_back_and_users_return_to_live	Fault-injects student-v3; v3 ends rolled_back and v1 stays live (skips if v3 was rejected before deploy for that seed). Shares the module fixture bad_deploy (the run under each canary rule) with the next test
test_sequential_gate_reduces_bad_deploy_exposure	FR-16.1, FR-9.4: the same corrupted v3 under the anytime-valid rule is rolled back too, after serving no more answers than under Wilson; each run’s exposure, rollbacks and served agree; the rollback event carries an e-value and a confidence sequence (simulation, mock models)
test_every_version_proves_its_golden_rows_were_never_trained_on	INV-3, INV-6, FR-16.4: over the six-day corrupted-deploy run (the module fixture bad_deploy_runs, which bad_deploy reads), every version’s committed leaves rebuild the training and golden roots its BOM signs, the sets are disjoint (brute force), and a held-out certificate built from the leaves verifies with proof.verify_heldout from the signed roots alone (simulation, mock models)
test_replay_reproduces_each_versions_logged_coverage_and_audits	FR-16.5, FR-6.5: over the same run, for every version, the replay at each request’s logged threshold serves exactly the requests the student served, each group of requests logged at one threshold reproduces its coverage at that threshold, and the replayed precision equals the audited precision the gate counted (audit_counts), inside its interval; nothing is left out (simulation, mock models)
test_label_p_and_student_agree_follow_the_design	FR-16.5, FR-6.4, INV-15: on every row of the run, label_p is 0.10 on served answers and 1.0 on escalations; student_agree equals audit_agree on audited rows, is set on every escalation with a valid student answer, and is never set on a served row that only a shadow arbitration took to the teacher (some exist); baseline_est_usd is always set
test_reference_simulation_unchanged	The pinned reference run equals tests/fixtures/sim_reference_seed7.json exactly: the pinned inputs (a flag added later is not compared), daily numbers, release events and final versions (below)
test_reference_workloads_reproduce_v040	the tickets workload of tests/fixtures/reference.json (captured from v0.4.0) matches the session’s reference run under scripts/reference_check.py’s rules, and every input the fixture pins for all three workloads is still the simulator’s default
test_reference_check_runs_with_per_task_calibration_off	reference_check.capture runs each workload with RATCHET_PER_TASK_CALIBRATION=0 (LEGACY_ENV), whatever the caller’s shell says, and puts the caller’s RATCHET_* variables back afterwards
test_reference_check_rules	reference_check.compare ignores float noise below a relative 1e-9, new day and totals keys, advisory event kinds and text appended to an event, and reports one line for a moved number, a removed key or day, reworded event text, an extra event, a changed version or a changed input default
test_provenance_only_lists_distillable_teachers	Every BOM source is Ultra with OpenMDW-1.1
test_gateway_openai_shape_and_demo_flow	Mock gateway: / (landing page) returns 200; /ratchet/demo/day works; /v1/chat/completions returns a valid chat.completion with the right category; stats count 401 requests; break-next and reset work
test_replay_endpoints_on_the_mock_gateway	FR-16.5, NFR-11: on the mock gateway, /ratchet/replay is a 400 without a live version; after two simulated days the canary’s replay names its state, simulation (mock models) and per_token, has actual next to the counterfactual at the current threshold (and at 0.4 with audit rate 0, with more coverage), the calibrated baseline and finite JSON; the frontier has 71 rows from 0.30 and an advisory suggestion; bad thresholds, rates and days are 400s, an unknown version 404
test_gateway_rejects_bad_requests	Empty messages returns 400; a stream: true request the cascade refuses (the strict mock, RATCHET_MOCK_STRICT=1, does not know it) gets a normal HTTP 400 before any stream starts
test_openai_compat_provider_parses_logprobs_and_usage	OpenAICompatProvider against an httpx.MockTransport: sends logprobs and n, sends the bearer header, parses logprobs per choice, splits completion tokens across choices (evenly here, because one choice has no logprobs), computes mean_prob
test_every_version_carries_a_certificate_in_its_eval_and_bom	FR-16.3, INV-14: 3 simulated days: every version’s eval and signed manifest.eval carry a certificate with lcb ≤ precision, ten frontier rows, per-task bounds and a test vector of test_n letters, all clean under recheck; a version never re-scored has its signed eval
test_certificates_move_no_simulation_number	FR-16.3, FR-9.2: the same run with the certificate code taken out of the golden eval has the same days, totals, versions, thresholds and decisions; events only gain appended text
test_ltt_thresholds_come_from_the_fixed_grid	FR-16.3: with --calibration ltt --cert-target 0.9 every version’s threshold is a grid value or 1.01 and its trace starts at 0.95

tests/test_hardening.py

Regression tests for the issues in Issues found while documenting: test_old_database_file_is_migrated, test_auto_day_counts_days_from_first_request, test_student_failure_falls_back_to_teacher, test_fallback_model_uses_default_threshold, test_teacher_failure_returns_502, test_unknown_tasks_do_not_grow_registry, test_live_alert_fires_once_per_version (for the e-process alert, the default since FR-16.2, and the Wilson rule), test_policy_from_env. test_anytime_valid_settings_parse checks the anytime-valid gate’s variables (FR-16.1, FR-16.2): RATCHET_SEQUENTIAL, RATCHET_ALPHA_ROLLBACK and RATCHET_ALERT_EPROCESS parse, from_env(defaults=) fills only unset fields, bad values name their variable, and RATCHET_CANARY_AUDIT_RATE is None unset, 0.5 when set, and stops Ratchet at 0, 1.5, -0.1 or NaN (and Settings at a non-number). test_task_names_match_in_full covers a bug found while building request validation: a task name with a trailing newline passed the ^...$ check. test_live_night_checks_the_student_endpoint_before_paid_work covers the issue found while testing the live entry points (Live-path tests). test_stored_threshold_serves_what_the_eval_measured covers a bug found while building certified precision: the golden eval stored its threshold rounded to 4 decimals, which could round up past the confidence it was calibrated at, so a version served less than its eval measured (nothing at all when every row had one confidence, as behind a constant-logprob endpoint). The answer-hygiene tests (v0.5, FR-2.7, FR-2.13, INV-16) cover what a code reading found before any live run: reasoning before a lone </think> was served and stored, an unterminated <think> let the reasoning’s JSON count as the answer, structured answers were stored inside code fences, a teacher answer cut off by the token limit (or empty, or not JSON for a structured task) became a training target and an audit, JSON nested too deeply crashed the request instead of escalating, and some long answers took seconds to clean.

The v0.4 hardening tests (privacy, API keys, streaming, signed BOMs) are in the same file. The gw_env fixture (in tests/conftest.py) reloads the gateway in mock mode with extra environment variables and reloads it again with the defaults afterwards, so settings such as API keys never leak into other tests.

Test	Covers
test_redact_finds_each_kind_and_checks_checksums	Every label is found in one text (two phones, ten placeholders); a card number that fails Luhn is left alone
test_redact_leaves_ordinary_text_alone	Order numbers, dates, times, prices, version strings, page counts, URLs and an ISBN are not touched
test_redact_messages_handles_content_parts	OpenAI content-part lists are redacted part by part; non-text parts and the caller’s copy are untouched
test_redact_messages_covers_names_refusals_and_call_arguments	name, an assistant’s refusal, tool-call and legacy function-call arguments are redacted and counted; a malformed tool call and the caller’s copy are untouched
test_redaction_is_a_no_op_on_the_mock_world	2,000 requests from the mixed stream and 800 each from schema and tools, their stored conditions and their teacher answers (text and tool-call arguments) are unchanged by redaction, so mock lookups and simulation numbers cannot shift
test_examples_are_stored_redacted_but_the_teacher_sees_the_original	The teacher receives the email and phone; the stored example has [EMAIL] and [PHONE] in messages and completion, redactions == 3, and privacy_stats agrees; with redact_pii=False the example is stored verbatim
test_keyring_parses_and_identifies	Named and bare keys, lookup, short keys, bad names and duplicate names raise; bearer() reads Authorization and X-API-Key
test_api_keys_protect_v1_and_name_the_client	No key or a wrong key gets 401 on /v1/chat/completions and /v1/models; Bearer and X-API-Key both work; clients in stats lists both names and never the keys; Scout stays public
test_daily_quota_per_key	Quota 2: remaining-requests header counts down, the third call gets 429 with remaining 0, and another key is unaffected
test_streaming_matches_the_non_streaming_answer	stream: true returns text/event-stream, a role chunk first, content chunks that join into exactly the non-streaming answer, a final chunk with finish_reason: "stop" and the ratchet block, then [DONE]
test_signed_bom_per_adapter_verifies_and_detects_tampering	With a signing key, a simulated night stores a BOM that matches the schema, lists only distillable sources and verifies with the public key; editing it breaks the SHA-256, re-hashing still fails the signature, and another key fails; unknown versions get 404
test_bom_is_stored_unsigned_without_a_key_and_cli_verifies	Without a key, a simulated night stores an unsigned BOM whose dataset hash matches the version; the CLI verify and keygen commands work
test_bad_signing_key_fails_at_startup	A key that is not base64, or not 32 bytes, raises
test_v03_database_gets_v04_columns	A v0.3 database file gets client, redactions and bom on open
test_v04_database_gets_the_route_column	A v0.4 database file with requests in it gets requests.route on open; stats_by_task reports the old rows as unrecorded next to a new cold_start row
test_v04_database_gets_the_examples_params_column	FR-1.13, FR-4.1: a v0.4 examples table gains params; its old row keeps its v0.4 hash and no conditions, a repeat bumps it, the same messages under JSON mode are a new example, and dataset.rows returns both
test_healthz_stays_public_when_keys_are_on	/healthz answers 200 while /v1/models needs a key
test_redact_preview_endpoint	POST /ratchet/redact returns the redacted text and counts; over-long or missing text gets 400
test_live_night_checks_the_student_endpoint_before_paid_work	Without RATCHET_STUDENT_BASE, NebiusTrainer.train raises before the sync, the training job or any file write; reload with adapters to load raises the same clear error (and with none, returns an empty list)
test_task_names_match_in_full	tasks.valid_name matches the whole name: a trailing newline, 65 characters, an empty string, a space or a tab is not a name, and tasks.get falls back to default for each
test_teacher_reasoning_and_fences_are_not_stored_or_served (2 cases)	FR-2.7, FR-2.13, INV-2, INV-16: a teacher answer with a <think> block, or with only a closing </think>, and a fenced JSON answer is served and stored as the bare JSON, with stored = 1 and no skip
test_served_text_is_clean_on_the_student_and_pinned_paths	FR-2.7, INV-10: a confident student’s fenced answer with reasoning is served as bare JSON; a pinned (research-plan) and a teacher-only answer lose the reasoning but keep their fence (free text), store nothing and are no skip
test_truncated_teacher_answer_is_not_stored	FR-2.13, INV-16: through the real OpenAICompatProvider and a FakeOpenAIServer that reports finish_reason: "length", a teacher answer whose visible part parses is served on a structured and a free-text task, but nothing is stored, both rows have stored = 0, and one skip_example event names the task and finish_reason length
test_empty_teacher_answer_not_stored (4 cases)	FR-2.13, INV-16: an empty, whitespace, unterminated-reasoning or reasoning-only answer is served as '' on both kinds of task, not stored, stored = 0, empty answer
test_invalid_json_teacher_answer_for_ticket_triage_not_stored	FR-2.13, INV-16: prose, cut JSON and a JSON list are served but not stored for ticket-triage (invalid answer), while the same text is stored for the free-text default task
test_redaction_that_breaks_a_structured_answer_is_not_stored	FR-2.13, INV-16, INV-12: a card-like number outside quotes becomes [CARD] and breaks the JSON, so the answer is served but not stored (invalid after redaction); with redaction off it is stored as is
test_unusable_teacher_answer_is_not_an_audit	FR-2.13, INV-16: an audit whose teacher answer was cut off gives audited = 0, audit_agree NULL, stored = 0, no audit for the version, and a cost that still includes the teacher’s call; a complete answer is an audit and an example again
test_invalid_shadow_reference_leaves_shadow_agree_null	FR-2.13, INV-16, FR-6.3: a shadow disagreeing with the live student, arbitrated with prose, gets shadow_agree NULL and one unarbitrated row, nothing stored; a valid arbitration settles the next one and is stored; an escalation whose teacher answer was cut off leaves shadow_agree NULL too
test_unusable_audit_answer_is_no_shadow_reference	FR-2.13, INV-16, FR-6.3: with a cut-off audit answer, a shadow that agrees with the served answer gets shadow_agree = 1 (no arbitration, precision 1.0), as without the audit; a disagreeing shadow is still arbitrated, and the cut-off ruling leaves NULL (review finding)
test_answers_that_quote_think_tags_are_served_and_stored_whole	FR-2.7, FR-2.13, INV-16: a triage reply that mentions a stray <think> tag is served and stored whole with no skip; a confident student quoting </think> in its reply is served; a free-text answer that mentions <think> is served and stored whole (review finding)
test_skip_example_events_are_throttled_per_reason	FR-2.13, INV-16, INV-12: 200 cut-off answers from 8 threads write exactly 3 events (1st, 10th, 100th); empty answer and an unnamed finish reason (other) are counted apart; no event quotes the answer; every row has stored = 0
test_stored_column_marks_requests_that_stored_an_example	FR-2.13, FR-3.4: an escalation and its duplicate (seen 2) both have stored = 1; pinned and an unaudited student answer have 0; an audited one has 1
test_v05_database_gets_the_stored_column	A ledger from before answer hygiene gains requests.stored; its old row keeps NULL and a new escalation records 1
test_answers_nested_too_deeply_are_invalid_not_a_crash	FR-2.2, FR-2.7: parse_json returns None instead of raising RecursionError on deeply nested JSON, so such a student answer is invalid and the request escalates (route confidence) instead of failing with a 500
test_answer_cleaning_is_linear_on_adversarial_answers	FR-2.7, NFR-1: 20,000 unclosed <think> tags, 100,000 spaces, 100,000 {, a long space run before a fence, and stray closing tags followed by unclosed ones are each cleaned and judged invalid in well under 2 s (seconds each before, from regexes that rescanned every position); a fenced object after a sentence still parses
test_provider_parses_finish_reason_reasoning_tokens_and_per_choice_tokens	FR-2.13, FR-1.11, NFR-7: per choice, the text is content only (never reasoning_content or reasoning), finish_reason is read, output tokens follow the logprob counts and add up to usage.completion_tokens (in proportion when the counts do not, evenly with the remainder first when a choice has no logprobs, the counts as they are without usage), and reasoning tokens are split evenly; a single teacher choice gets the whole bill
test_finish_reason_values_are_read_safely (11 cases)	FR-2.13, FR-1.11: a missing, null, empty or blank finish_reason is stop; a word is lower-cased and trimmed; a number, a phrase or more than 40 characters is unknown
test_logprobs_too_large_for_a_float_are_ignored_not_a_crash	FR-1.14, FR-1.11 (regression): a token logprob of 10^400, which json.loads accepts as an integer, made math.isfinite raise OverflowError while parsing (a 500 once the teacher returns logprobs); now it is left out of the confidence’s logprobs and the served logprobs are None, and a normal value is served
test_provider_sends_params_and_its_explicit_arguments_win	FR-1.13, FR-1.11: OpenAICompatProvider.complete(params=...) adds the fields to the body; the explicit temperature, n, logprobs (off also drops top_logprobs) and response_format win; without params the body is v0.4’s, and an unknown keyword argument is ignored; allow narrows what is sent
test_answers_under_label_altering_sampling_are_served_not_stored (6 cases)	FR-1.13, INV-2, FR-4.1, threat model T2: a client’s logit_bias of +100, a presence penalty of 2, a negative frequency penalty, temperature 2, a vendor field (guided_choice) or all three at once reach the serving teacher and its answer is returned, but nothing is stored (stored 0, one skip_example event naming the reason); neutral sampling (temperature 1, top_p, a seed, zero penalties) stores; an audit, which drops these fields, stores its answer
test_v04_file_gains_the_evidence_schema	a ledger file with v0.4.0’s schema and one row per table gains every new column (old rows: contested and checked_epoch 0, the rest NULL), the eight evidence tables, the append-only triggers and the new indexes, twice without change; its old data reads as before and new rows take the columns
test_every_index_exists_on_a_new_ledger	every INDEXES entry exists on a fresh ledger (a misspelled column would make _migrate skip it everywhere), and the gate’s audit count uses requests_version_audited
test_committed_leaves_and_sealed_receipts_are_append_only	INV-17: version_leaves and receipts accept new rows and refuse UPDATE, DELETE, INSERT OR REPLACE, INSERT OR IGNORE and a duplicate INSERT (also on a migrated v0.4 file), a NULL key column and a split other than train or golden, leaving the rows as they were
test_reserved_settings_stop_startup_until_their_feature_lands	each of the 9 reserved settings set away from its default stops Settings.check_reserved(), Ratchet and python -m ratchet.live (before the ledger is opened), naming the variable and the feature; the defaults spelled another way pass; a non-number names its variable
test_reserved_release_policy_fields_stop_startup	each of the 5 reserved ReleasePolicy fields set through its variable stops from_env; the defaults, also set explicitly, pass
test_policy_from_env_names_a_bad_value (8 cases)	a non-number, a fractional or negative count, a live_floor of 1 or 0, NaN and a negative tolerance each stop from_env with the variable named
test_certified_precision_settings_parse	FR-16.3, FR-6.30, FR-6.31: RATCHET_CALIBRATION (any case; empty is point), RATCHET_CERT_TARGET and RATCHET_CERT_DELTA reach Settings.cert_policy() and the mock fleet; bad values stop cert_policy, the mock Runtime and python -m ratchet.live before a ledger is opened; the four ReleasePolicy fields parse and refuse values out of range
test_stored_threshold_serves_what_the_eval_measured	FR-5.2, INV-6, FR-16.3 (regression): every golden row at confidence 0.98049 measured 100% coverage, but the threshold stored rounded to 0.9805 served nothing; stored exactly, the router serves every row the eval counted
test_replay_migrates_an_old_file_and_counts_the_rows_it_cannot_replay	FR-16.5: a v0.4.0 file migrates on open; its rows with a confidence are left out of the replay as before_logging and those without one counted as no_student_answer; rows the gateway writes afterwards are replayed at their logged threshold and audit rate; since_day leaves the old day out; two rows edited outside Ratchet (a probability of 1.5, a negative cost) are malformed and the report stays finite JSON
test_replay_endpoint_refuses_to_mix_versions	FR-16.5: version twice, a list, a word, a negative, a fraction, or superscript and circled digits (², ①, 1², which str.isdigit accepts and int refuses) is a 400 on both replay endpoints, never a 500, saying versions are never pooled; an unknown version is 404; without version the live one is read
test_pinned_tasks_never_appear_in_replay_rows	FR-16.5, INV-10: a pinned task’s requests have no replay row, and a task pinned after its traffic was logged is left out as pinned_now
test_replay_reads_a_long_history_in_chunks_up_to_the_id_it_started_at	FR-16.5: with REPLAY_CHUNK at 2, Ledger.replay_rows returns exactly the rows of one read; another thread logs a request between each of the four chunks (the lock is free), and those requests are left out of this read and in the next
test_replay_endpoints_run_one_at_a_time	FR-16.5: while a replay holds rt.replay_lock, both replay endpoints answer 429 with Retry-After: 1; afterwards 200
test_replay_leaves_rows_held_by_a_changed_spec_unservable	FR-16.5, FR-2.15: rows logged at SERVE_NOTHING while the task’s spec is changed for the version are never served at any replayed threshold, rows logged before the change are, and after a re-score under the new spec the held rows are servable again
test_replay_current_threshold_is_the_one_the_router_uses	FR-16.5: the replay’s current_threshold is the version’s own in mock mode and the configured default (0.7, not the version’s 0.9) for the hosted fallback student in live mode without a student endpoint, on both endpoints
test_receipt_verifiers_never_raise_on_nested_junk	FR-15.5 (8 cases): a receipt, or the previous receipt, whose period is a string, a number, a list or {day: true}, or whose merkle is a list or a string: verify_chain (both ways round), verify_proof, verify_rows, recheck, check_manifest and verify_receipt answer FAIL (they raised AttributeError before), and python -m ratchet.receipt verify with --prev, --proof and --rows exits 1 instead of printing a traceback
test_receipt_verifiers_never_raise_on_seeded_nested_mutations	FR-15.5: 300 seeded trials, each replacing up to three fields anywhere in a receipt, the previous receipt, a proof or the leaves with junk of another type (or removing them): no verifier raises (72 raises in 2,000 trials before the fix)
test_a_stand_in_that_loses_money_lowers_the_fee	FR-15.5: a fallback teacher’s answer that cost $0.0002 more than its baseline adds -200,000 nano-USD to savings.stand_in_loss_nusd, the billable savings and the net bound, and lowers the fee to a quarter of the lower net; a cheaper fallback answer and a degraded answer add nothing (gains are never billed); the billable set is unchanged and the sealed receipt rechecks from its leaves
test_mock_reasoning_mean_out_of_range_stops_live_startup_too	FR-15.5: RATCHET_MOCK_TEACHER_REASONING_MEAN of -5 stops a live Runtime, and 1e9 stops python -m ratchet.live, before the ledger is opened (it was checked in mock mode only); 420 is accepted

tests/test_resilience.py

Upstream failures: around a request the student already answered (FR-2.10, INV-15), the provider’s retries and the client statuses they end in (FR-1.11, FR-1.8, INV-7), the student circuit breaker with automatic adapter reloads (FR-2.11, NFR-2), and the teacher’s breaker, degraded mode and the fallback teacher (FR-2.12); the student falling back to the teacher is in test_hardening.py. Breaker tests run on a FakeClock (Ratchet(clock=...), Runtime(clock=...)), so no test waits out a cooldown. The failing teacher puts its URL, with a query token, in each message, and every test checks the events, errors and responses never repeat it. Retry tests use the local Upstream fake (a scripted endpoint behind httpx.MockTransport that records each attempt’s timeouts) and inject sleep, clock and rng, so nothing really waits.

Test	Covers
test_audit_failure_still_serves_the_student (3 cases)	With audit_rate=1, a teacher that times out, answers 503 or returns an unreadable body: the confident student’s answer is served, the row has audited=0 and audit_agree NULL, nothing is billed for the failed call, the baseline is the no-audit estimate, nothing is stored, one audit_error event (ticket-triage: ReadTimeout, HTTPStatusError 503, ValueError; never the URL), and health() counts one audit error
test_audit_errors_are_counted_and_throttled_and_escalation_still_fails	With the teacher’s breaker off (teacher_breaker_failures=0), 12 failed audits: 12 counted, events on the 1st and 10th only, no row counted as audited. The teacher comes back: audits and their training examples resume. An unsure student with the teacher down still raises (INV-7, an error status at the gateway) and is not counted as an optional failure
test_failure_counters_are_exact_under_threads	8 threads, 200 failed audits (the teacher’s breaker off): every request served by the student, exactly 200 counted, exactly three events (1st, 10th, 100th)
test_shadow_arbitration_failure_is_silent	The shadow disagrees with the live student and the teacher times out only on that request: the user gets the live answer, the row keeps shadow_serve=1 with shadow_agree NULL, no teacher cost and no example, one shadow_error (v2 arbitration: ReadTimeout); shadow_quality counts the row in n and coverage and as a disagreement (precision 1.0 → 0.5, unarbitrated 1)
test_shadow_sampling_error_counted	With the breaker off (student_breaker_failures=0), a shadow endpoint that times out on 10 requests: all served by the live student at the samples’ cost, no shadow record and no teacher call, events on the 1st and 10th, shadow_errors 10
test_optional_teacher_failures_through_the_gateway	Live Runtime with the real HTTP provider, its retry policy and a Token Factory that times out: audited requests (plain and streamed) are 200 with the student’s answer, and the shadow’s disagreement is not sent to the teacher again after the audit failed (one teacher call, two HTTP attempts, per request); unaudited, the failed arbitration is still a 200; the events read UpstreamError 504 upstream_timeout; an escalation is a 504 teacher_timeout. Live mode runs the audit and the shadow after the response, so the test waits for the sidecar before each check
test_student_error_event_never_quotes_the_endpoint	INV-12: live Runtime with a student endpoint whose 400 quotes the request (Invalid content: 'My SSN is 123-45-6789, email jane@corp.com') and its URL, as vLLM’s validation errors can: the teacher answers (route student_error), and the student_error event, in the ledger and in the public /ratchet/stats, reads v1: UpstreamError 400, with no request text, message or URL
test_retry_backs_off_with_full_jitter_under_the_cap	503, 503, 200: three attempts, waits uniform(0, min(cap, base·2^i)) that grow and stay under the cap; a long outage waits at the cap and returns the last UpstreamError (still an httpx.HTTPError); jitter’s low end waits 0
test_retry_after_is_honoured	A 429 with Retry-After: 2 waits exactly 2 s; retry-after-ms wins over Retry-After; a wait over the 8 s cap is not waited for: one attempt, and the error carries the wait
test_parse_retry_after_forms	Seconds, fractions, milliseconds, an HTTP date, case-insensitive names, a bad ms value falling back; empty, zero, negative, infinite, NaN, past and malformed values are ignored
test_client_errors_are_not_retried (6 cases)	400, 401, 403, 404, 413 and 422: one attempt, no wait
test_deadline_stops_retries	With a 5 s deadline and 1 s attempts, a wait that would end past it is not started (three attempts), each attempt’s connect and read timeouts shrink to the time left (5, 3.5, 1.5 s), and a server wait counts too; with no policy, one attempt with the configured timeouts
test_read_timeout_is_retried_exactly_once	A read timeout twice gives up after two attempts with 504 upstream_timeout (the cause kept, no URL); a 503 first does not use up that retry; connection failures before the model saw the request are retried up to max_attempts; write timeouts and read or write errors are not retried
test_bad_upstream_bodies_are_502_and_not_retried (7 cases)	Empty, missing or null choices, a choice that is not an object, a body that is not an object or not JSON, JSON nested too deeply for the parser (a RecursionError): 502 bad_upstream_response after one attempt (before: an IndexError, so a 500)
test_malformed_optional_fields_count_as_absent	Text parts joined, junk logprobs (strings, booleans, NaN) skipped, bad usage counted as 0, a missing model replaced by the requested one
test_upstream_error_bodies_are_parsed_without_the_url (7 cases)	OpenAI’s {"error": {...}}, vLLM’s legacy body (integer code dropped), an error string, FastAPI’s detail (string and validation list, with param), HTML (the reason phrase), and a long message with a control character and a lone surrogate (cleaned, clipped to 300); never the URL or the key; raise_for_status()’s error reads the same
test_upstream_errors_map_to_client_statuses (20 cases)	Live Runtime at cold start, the teacher behind MockTransport: upstream 400 context_length_exceeded → 400 with the teacher’s message, param and code; 413 → 400 request_too_large; 429 → 429 with Retry-After 7 (after two 7 s waits), 1 when none was sent, 60 for a 600 s wait (not retried); 500 and a connection error → 502 teacher_unavailable after three attempts; a 503 with Retry-After 2 → 502 with Retry-After 2 after two 2 s waits, with 30 or 600 → 502 with Retry-After 30 or 60 after one attempt (a wait over the 8 s cap goes to the client), a 504 with Retry-After 20 → 504 with it; a read timeout → 504 teacher_timeout after two; 401 and 403 → 502 upstream_auth_error; 404 → 502 upstream_not_found; 402 (also with a Retry-After, which is then not passed on) and a 301 redirect → 502 with X-Should-Retry: false; empty choices and JSON nested too deeply → 502, not 500. X-Should-Retry: false exactly where a retry cannot help; nothing logged, no gateway_error
test_live_runtime_passes_timeouts_and_retries	The teacher gets its read timeout and retry policy (and the injected sleep), the student its own timeouts and no retries; the timeouts arrive on each request; the hosted fallback is a separate client with the student’s settings; the variables reach Settings; zero, negative or NaN values stop live mode at startup
test_quota_429_says_when_it_resets	At 23:00 UTC: X-RateLimit-Reset-Requests: 1h0m0s on the 200 and the 429; the 429 is insufficient_quota / daily_quota_exceeded with Retry-After: 3600 and X-Should-Retry: false; 90.5 s before midnight rounds up to 91 s and 1m31s; refusals are not counted; _duration formats
test_sdk_sends_an_exhausted_quota_once	The real OpenAI SDK with its default two retries sends the quota-exhausted request once and raises RateLimitError with the type and code
test_sdk_sends_an_upstream_client_error_once	The SDK raises BadRequestError with the teacher’s context_length_exceeded and param after one request, and InternalServerError 502 upstream_auth_error without retrying
test_scout_error_event_names_the_upstream_status	A failed Scout run says a model or search call failed: upstream 503 service_unavailable (or upstream 502 upstream_unreachable, upstream 504 upstream_timeout), never the URL
test_breaker_opens_probes_backs_off_and_closes	Breaker on a fake clock: 4 failures and a success stay closed, 5 in a row open it (open_until 30 s on); exactly one probe after the wait; failed probes wait 60, 120, 240, 300, 300 s (at 300 s only the 1st probe’s transitions are reported); a successful probe closes it and resets the wait; a released probe and a lost one (no report within the wait) let the next caller probe; a late result from another thread in half_open, and results while open, change nothing; threshold 0 never opens; a failing listener is swallowed; bad arguments and bad RATCHET_STUDENT_BREAKER_* settings raise ValueError when the cascade is built
test_half_open_lets_exactly_one_probe_through	25 rounds of 16 threads released by a barrier at the end of the cooldown: exactly one allow() is True each time
test_a_probe_slower_than_the_cooldown_still_counts_and_backs_the_breaker_off	Regression (review of FR-2.11): a probe held past the cooldown on a hung endpoint used to be replaced and its failure dropped, so the breaker stayed half-open for the whole outage. Now a lost probe’s failure reopens the breaker and doubles the wait (and its success closes it), and with probe_timeout_s nobody else probes while the probe can still be running. A 900 s hung-endpoint run (a request a second, calls held 15, 17 or 60 s, cooldowns 5 and 30 s) backs off to 300 s every time, with one probe in flight at most when probe_timeout_s is set; a bad probe_timeout_s raises
test_router_never_replaces_a_probe_that_can_still_be_running	The router gives a breaker the student call’s longest time (OpenAICompatProvider.longest_call_s(): pool, connect, write and read timeouts, 32 s by default, plus 5 s): with a 5 s cooldown, requests 5, 10 and 36.9 s into a probe that hangs on a MockTransport student skip the student (route student_breaker), and the probe’s timeout reopens the breaker for 10 s
test_circuit_events_at_the_longest_wait_are_throttled	120 failed probes at 300 s report only the 1st, 10th and 100th pair of circuit transitions (each says which), every probe still counts in failures, closing is always reported, and the count starts again after the breaker closes
test_endpoint_failures_are_told_from_request_errors	Timeouts, connection and protocol errors, 5xx, 408, 429 and an unreadable answer count as the endpoint down; 400, 401, 404 and AdapterMissing as the endpoint answering; a KeyError or ValueError as neither. CircuitOpen is a 503 circuit_open
test_open_student_breaker_sends_requests_straight_to_the_teacher	A student that times out: 5 student_errors and student: closed->open after 5 failures (ReadTimeout); the 6th request never reaches the student (call counter), is served by the teacher with route student_breaker, escalated=1, confidence NULL and only the teacher’s cost, and is not another student error; a failed probe after 30 s waits 60 s; once the student is back the probe serves and closes it; the five circuit events in order; the teacher’s breaker stays closed
test_shadow_breaker_is_its_own	A shadow that times out opens the shadow breaker after 5 calls; the live student serves all 10 requests and its breaker stays closed; skipped shadow calls are not counted as errors and leave no shadow record
test_breakers_are_per_endpoint_and_role	Live and canary on one endpoint share the student breaker (five failures between them open it for both); breakers are keyed by base_url (two clients of one URL share one), by class without one (a new provider object per call keeps one breaker), and by role; names student, student-2, student-3, shadow (and the teacher’s own teacher)
test_lost_adapter_is_reported_and_does_not_open_the_breaker	vLLM’s 404 The model `student-v1` does not exist.: 10 requests escalate, on_adapter_missing is called each time with the version, the breaker stays closed, the student_error detail says AdapterMissing 404 adapter_missing; a 404 that names no adapter or another one, and the hosted fallback model’s 404, call nothing; a failing or missing callback never fails the request; a shadow’s lost adapter is reported too and leaves the shadow breaker closed
test_errors_without_an_http_status_say_nothing_to_the_breaker	10 KeyErrors from the student leave the breaker closed with no failures; a half-open probe that ends in a KeyError is handed back and the next request probes (and closes it); a 200 with no choices is a 502 bad_upstream_response that counts, so five of them open the breaker
test_adapter_reloader_rate_limits_and_reports	AdapterReloader on a fake clock: one queued reload per adapter per 60 s, adapter_reloaded and adapter_reload_failed (HTTPStatusError 503, never the URL) events; the thread survives a failing ledger write; nothing is queued after close(); adapter_reloader refuses to build one without RATCHET_STUDENT_BASE
test_live_gateway_reloads_a_lost_adapter_once_a_minute	Live Runtime with FakeOpenAIServer(loaded=set()) as a restarted vLLM: 10 requests in 50 s are 200s from the teacher, one load_lora_adapter call with the adapter’s path and the endpoint token, the breaker closed, rows routed student_error, adapter_reload_failed while loading fails; a minute later the next 404 reloads it and the student serves again. Mock mode and the hosted fallback have no reloader
test_demo_student_outage_opens_and_closes_the_breaker	Mock gateway on a fake clock: seconds clamped to 5 and 300; a 60 s outage gives 5 student_errors, then student_breaker, a failed probe at 30 s, student_breaker again, and at 91 s a probe served by the student; the five circuit events and the fault event; /ratchet/stats routes add up to the task’s requests
test_every_request_row_has_a_route	Cold start, a pinned task, a teacher-only request, a confident student, one that is confident but cannot fill 4 choices, an unsure student, the fallback teacher for a down teacher, a degraded answer with both teachers down, a failing student, an open breaker and the student turned off by an operator give cold_start, pinned, pinned, student, n_choices, confidence, fallback_teacher, degraded, student_error, student_breaker, student_disabled, and a tool request and a part only the teacher reads give tools and multimodal (FR-1.41): every route in router.ROUTES, no NULL route, stats_by_task routes per task, most frequent first, and stats_by_day’s degraded and fallback_teacher counts
test_open_teacher_breaker_answers_503_at_once_with_retry_after	FR-2.12: live Runtime at cold start with a Token Factory that answers 503: five requests are 502 teacher_unavailable after three attempts each and open the teacher’s breaker (teacher: closed->open after 5 failures (UpstreamError 503)); the 6th is a 503 teacher_circuit_open with Retry-After: 30, no X-Should-Retry and no teacher call; 12.5 s later the OpenAI SDK raises InternalServerError 503 with Retry-After: 18; nothing is logged; after the cooldown the probe finds the teacher back and closes the breaker. _map_upstream rounds the wait up and clamps it to 1 to 60 s; a student’s CircuitOpen, or an upstream’s own teacher_circuit_open code, stays a 502
test_open_teacher_breaker_skips_audits_and_arbitration_without_counting_them	Five escalations that time out open the breaker; the 6th raises CircuitOpen (503, retry_after 30) with no call and no row; then a confident student is served with the audit and the shadow’s arbitration skipped (no teacher call, rows audited=0, shadow_agree NULL), no audit_error or shadow_error, only the circuit event; after the cooldown the next audit is the probe, succeeds and closes the breaker
test_teacher_breaker_counts_only_teacher_outages	Seven 400 context_length_exceeded answers leave the breaker closed; four timeouts, then a tool-call answer (a success for the breaker) and a ValueError (no count), then four timeouts: still closed with 4 failures; teacher_breaker_failures=0 never opens; bad RATCHET_TEACHER_BREAKER_* values stop startup; Breaker.retry_in() is None while closed or half-open, the time left while open, 0 once it is up
test_serve_student_mode_answers_degraded_with_a_flag_and_a_header	FR-2.12: live Runtime with RATCHET_ON_TEACHER_FAILURE=serve_student, a student below its 0.99 threshold and a Token Factory that answers 503: a 200 with the student’s answer, ratchet.degraded: "teacher_unavailable", X-Ratchet-Degraded and X-Ratchet-Served-By: student; the row is route degraded, audited=0, escalated=1, stored=0, no shadow record (the shadow student is not called), billed for the samples only; a degraded event (default: UpstreamError 503); streamed the same; after the breaker opens, degraded answers come with no teacher call; no audits for the live version, no shadow evidence; /ratchet/stats counts them per day and per route; a confident answer has degraded: null and no header
test_error_mode_is_the_default_and_pinned_requests_never_degrade	FR-2.12: by default an unsure student with the teacher down is still a 502 with no degraded header and no row; with serve_student, a pinned task (research-plan) is a 502 five times, then a 503 once the breaker is open, and the student is never called
test_degraded_mode_never_hides_a_client_error_or_serves_an_invalid_answer	With serve_student, an unavailable teacher degrades (also with a client max_tokens), but these still raise: a request with tools (teacher-only since FR-1.41), the teacher-only path, a 400 context_length_exceeded, a non-HTTP error, a student whose samples are all invalid, a student that failed or whose breaker is open, a teacher tool-call answer and a cold start. Serve_Student with spaces is accepted, empty means error, other values stop startup
test_degraded_answers_are_counted_exactly_under_threads	8 threads, 200 degraded answers: every row route degraded with audited and stored 0, health()["degraded_answers"] exactly 200, three degraded events (1st, 10th, 100th)
test_n_choices_when_the_teacher_is_down	FR-2.12, FR-1.14: a degraded answer for n = 2 serves two agreeing student samples (usage of both); with only 2 of 3 agreeing, n = 3 gets the teacher’s error while n = 1 still degrades; the fallback teacher writes n choices at 0.7 and stores nothing by default
test_fallback_teacher_answers_when_the_primary_is_unavailable	FR-2.12: with the primary down, the fallback answers an escalation (route fallback_teacher, served_by teacher, the fallback’s model id, its own price, the primary’s price as the baseline, its own fingerprint), not stored, no shadow call, one fallback_teacher event; a 400 about the request is not sent to the fallback; five failures open the primary’s breaker and the fallback then answers without a primary call; a pinned task gets the fallback, priced lower than its baseline; an audit is skipped, never sent to the fallback; with both down the primary’s CircuitOpen is raised; no fallback-served row has a shadow record
test_stored_fallback_answers_carry_the_fallback_license	INV-1, INV-2, FR-3.3: with store_fallback_teacher, the fallback’s answer is stored with its own model and license (NVIDIA Nemotron Open Model License) and the primary’s with OpenMDW-1.1, and the BOM lists both; by default nothing is stored, and without its own provider the fallback model is asked on the teacher’s client; RequestMeta(storable=False) stores nothing, with no skip_example event
test_live_runtime_wires_the_fallback_teacher	FR-2.12, INV-1: RATCHET_TEACHER_FALLBACK_BASE gives the fallback its own client (the teacher’s key, retries and timeouts); at cold start with Token Factory’s host down the response’s model is the fallback’s, after three primary attempts and one fallback call, both with the teacher’s key; without it, the fallback model is asked on the teacher’s client; breakers teacher and fallback_teacher; the six variables reach Settings; RATCHET_TEACHER_FALLBACK_MODEL=gpt-4o stops the live Runtime
test_a_golden_row_the_student_refuses_never_stops_night_school	FR-5.1, INV-7, FR-4.4: 60 examples, half stored with tools; a student behind MockTransport that answers 400 to tools as vLLM without a tool parser does; under an allowlist that forwards tools, run_night trains and reports refused_n equal to the golden rows with tools; the default allowlist sends no tools; the judge’s refused candidate does not pass (pass rate 0.5); a 401 still stops the eval

tests/test_scout.py

The gw fixture (in tests/conftest.py) reloads the gateway in mock mode with RATCHET_MOCK_WARMUP_DAYS=0. The guard tests use FakeClock.

Test	Covers
test_examples_match_the_research_world	research.EXAMPLES equals the UI’s EXAMPLE_QUESTIONS; every planned query returns simulated results
test_scout_run_emits_the_documented_events	A run emits start … done; steps have exactly the documented keys; planner first, writer last, all five agents present; one reader step per relevant triage; llm_calls excludes searcher steps; cost_usd equals the sum of the steps
test_pinned_tasks_go_to_teacher_and_are_never_training_data	research-plan and research-write are pinned, source-triage is not; after a run the only stored example tasks are source-triage and page-read; stats_by_task shows 0% student share for the planner and one writer request
test_student_learns_scout_calls_over_simulated_days	After 4 simulated days of mixed traffic a version is live, and a Scout run has at least 60% student calls and more than 30% savings
test_research_workload_simulation_learns	--workload research, 6 days × 1,200: day 1 is 0% student; the last day is above 80% student and above 60% savings
test_gateway_scout_routes	/, /scout, /dashboard return 200; /scout/examples in mock mode; /scout/run returns events ending in done; non-example and blank questions get 400; /scout/stream streams start … done with HTTP 200, and a bad question gives HTTP 200 with a single error event; /scout/status; tasks in /ratchet/stats; /ratchet/demo/day?workload=research works and an unknown workload gets 400
test_rate_limit_returns_429	With a guard of 2 runs per hour, the third /scout/run gets 429
test_guard_budget_and_hourly_window	Per-IP hourly limit, the window rolling over after an hour, the budget refusing new runs once spent, and a new UTC day resetting it (fake clock)
test_tavily_search_sends_bearer_and_parses_results	TavilySearch against httpx.MockTransport: bearer header, search_depth: basic, max_results; results without a URL are skipped; a missing title falls back to the URL; snippet ≤ 500 and excerpt ≤ 1,500 characters with whitespace collapsed; an empty key raises
test_scout_survives_bad_json_from_models	When every model call returns non-JSON, nothing is judged relevant, nothing is read, and the writer’s raw text becomes the answer
test_guard_caps_runs_per_day	max_runs_per_day=2 refuses the third run of the day and resets the next UTC day
test_demo_controls_are_serialised_and_reset_has_a_cooldown	A demo action while another holds _demo_lock gets 409; a second reset within 30 s gets 429
test_failed_run_returns_502_and_its_spend_still_counts	When the writer’s call fails, /scout/run returns 502 and the planner, triage and reader spend is still added to the budget
test_stats_mark_pinned_tasks	/ratchet/stats tasks reports pinned for the planner and writer tasks only
test_react_app_is_served_with_lite_fallback	With the committed build, /, /scout and /dashboard return the React shell and /assets/* is served; /lite* return the hand-written pages; without a build / falls back to the lite page

tests/test_contract.py

The contract tests (Contract tests). Gateway tests use gw_env; cascade tests build a small ledger with _version and script the models with RecordingProvider, with every setting they depend on passed explicitly.

Test	Covers
test_v1_models_lists_ratchet	FR-1.2: GET /v1/models returns a list with a ratchet model; with API keys on it needs a key (401) and answers the same with one
test_task_from_metadata_and_header_precedence	FR-1.3: the ledger’s task column is default with neither, the metadata task alone, the header alone, and the header when both are set (both orders); other metadata keys are ignored
test_missing_or_malformed_messages (9 cases)	FR-1.5: missing, null, empty, string, object and number messages, an unknown role, a number as content and null user content get 400 and log nothing (the last three were 500s until v0.5)
test_messages_items_must_be_objects_400	FR-1.5: ["hi"], [null], [1] and a nested list as messages get 400 and log nothing
test_bad_metadata_is_400	FR-1.3: metadata as a string, a list, or with a non-string ratchet_task gets 400 and logs nothing
test_text_content_parts_answered_in_mock	FR-1.1: a demo ticket sent as OpenAI text content parts gets the same answer as the string form and the same stored example (seen 2)
test_bom_endpoint_counts_by_split_and_fingerprint_changes	FR-3.5, US-7: /ratchet/bom counts per split equal the ledger’s; every source names the configured teacher and its license; examples_sha256 is the documented SHA-256 over the sorted example hashes, stable between reads and equal to the stats block; a repeated request changes nothing, a new one changes the fingerprint and adds one
test_cost_report_contract	FR-7.20, FR-15.3: every key in tests/fixtures/cost_keys.json (what the dashboard’s Economics panel reads from GET /ratchet/cost), with its JSON type, on a fresh gateway (an empty report: no requests, hours or days) and after two simulated days and nights plus a training job left open for 13 hours (open intervals and warnings have items, each checked); the fixture still holds every key, with its type, it had when the panel shipped (COST_MINIMUM); each view’s breakdown adds up to its total and savings are the baseline less the total; each day’s dollars, as the panel derives them, match the report’s daily shares; the /ratchet/stats cost summary matches the report
test_economics_panel_helpers	FR-7.20: runs web/tests/economics-model.test.mjs under Node’s test runner (skipped without Node 22.6+ or web/node_modules; CI’s web job runs it with npm test): every gridline label on the break-even meter and the daily savings chart, over a sweep of values, prints the value it sits at (a 2.5 step labels 2.5 and 7.5, not 3 and 8; 0 is never −$0.00); both ends of the meter carry a tick and its log caption names the real ratio (100× over more than 6 decades); the empty state needs a report whose counts are present and zero, so one with only views and recommendation still shows them; a truncated daily list without since_day labels the window “Through day N”; idle time is described “in hours” or “on days” as the report counted it (idle_granularity)
test_task_registry_contract	FR-7.21, FR-2.15: every key in tests/fixtures/tasks_keys.json (what the dashboard’s Task registry panel reads from GET /ratchet/tasks, GET /ratchet/tasks/{name}, POST /ratchet/tasks/validate, PUT (200 and 422) and DELETE), with its JSON type, on a fresh gateway and after a request naming an unregistered task, a refused and a saved API task (records, a set with an object item, a policy); the fixture keeps the keys this item named; every field spec at any depth names a comparator type the panel labels; the editor’s SPEC_KEYS (read from registry-model.ts) are taskspec’s without the name, and putting them back changes nothing; extends on a built-in task copies its fields and spec hash; a built-in task is 403; a 404 for a name that is gone
test_resilience_panel_contract	FR-7.22, FR-2.11, FR-2.12, INV-19: CIRCUIT_RE, OUTAGE_RE and WAIT_RE, read from the dashboard’s source and compiled by Python, match every event of the outage drill (on a FakeClock) and of a teacher outage driven to the 300 s wait; the state the panel derives (each breaker’s latest circuit event) equals Ratchet.health() at every step, except one probe behind at the longest wait, as the panel’s footnote says; the parsed waits are the breaker’s; the drill’s fault time plus its length is the response’s until; the fallback counts the panel shows come from routes
test_dashboard_admin_key_stays_in_memory	FR-7.21, INV-13, INV-18: a scan of web/src: the theme is the only thing in browser storage, no file that holds the admin key writes a URL, and api.ts sends the key only through adminHeaders (a Bearer header)
test_story_simulation_numbers_match_reference_run	FR-7.23, NFR-11, INV-20: every simulation number the story page prints (web/src/lib/sim.ts) is the reference run’s at the precision its literal is written with: the honest bill’s SIM_COST_REPORT (both cost views, the break-even, the crossover, the recommendation and its explanation word for word, read as strict JSON up to its // end marker), “Watch it learn”’s ten days and totals, the command the page says reproduces them (parsed into the same simulator inputs) and the GPU-hour line that command prints (which web/tests/story-model.test.mjs pins simulateLine to); the four recommendations the chapter explains are cost.RECOMMENDATIONS in order, and its “idle more than half” is IDLE_SHARE. Uses the session’s reference_run
test_story_page_helpers	FR-7.23, FR-7.24: runs web/tests/story-model.test.mjs under Node’s test runner (skipped without Node 22.6+ or web/node_modules): the reference report and a live one render through the same billOf; a field the gateway did not send stays unknown, never zero; the break-even and crossover formulas hold for the rounded numbers the page prints them with (the divisor to three significant digits), and the formula divides the endpoint rate the report recorded (endpoint_usd_per_hour), not the configured price; idle time is described in the report’s buckets (idle_granularity); the three bills share one dollar scale and a tiny part keeps a sliver; the meter’s rows and log scale; the four recommendations with the report’s marked; the simulator’s GPU-hour line rebuilt exactly; the trust chapter’s redaction list and clipped event text
test_story_trust_claims_match_code	FR-7.24, INV-14, INV-21: the story’s “Built to be checked” chapter claims only what the code does: every file it names exists; its redaction labels are redact.PATTERNS in order, and redaction is on by default as it says; its gate numbers (precision, regression tolerance, audits, floor) are ReleasePolicy’s defaults; the invariants it cites (4, 6, 12, 14) say what it says; every route it names is served; its BOM commands work as printed (a signed BOM fetched from a mock gateway with RATCHET_BOM_SIGNING_KEY verifies offline with the key from /ratchet/bom/public-key and prints the page’s line; an edited one prints the page’s failure); the BOM carries the Clopper–Pearson certificate the v0.5 note mentions and python -m ratchet.stats check passes on it; CI runs the traceability command it prints; its link lands on the dashboard’s #redactor; it names v0.5 work only where its foundations exist, and no evidence tour before ratchet/tour.py exists
test_who_model_helpers	FR-7.40: runs web/tests/who-model.test.mjs under Node’s test runner (skipped without Node 22.6+ or web/node_modules): a student call whose model is student-v3 is “Nano · yours” with its adapter version in the tooltip; one whose model is the gateway’s fallback_student (case and spaces ignored) is “Nano 30B · hosted”, its tooltip naming the threshold and “Not covered by our adapter’s precision certificate”, the window sentence only when student_serving.window is set; the teacher is “Ultra · Token Factory”; a tool is “Tavily” or “Search”; every text matches the label table; the badge context whoContext builds from /ratchet/stats or /scout/status (an older gateway’s reads as before); the dashboard header’s serving line for each kind; no precision figure while hosted_fallback serves; Scout counts hosted calls and a run they answered (or, serving hosted, a run with no student answer) stops saying “your model”, while a run our adapter answered keeps it
test_student_serving_kinds_match_the_ui	FR-7.40: the StudentServingKind union in web/src/lib/api.ts lists exactly STUDENT_SERVING_KINDS, and /ratchet/stats reports one of them with fallback_student equal to the setting the live resolver uses
test_stats_contract	FR-7.1 to FR-7.4, FR-7.7, US-9: every key in tests/fixtures/stats_keys.json, with its JSON type, on a fresh gateway (lists empty) and after two simulated days and nights plus a keyed request (every list has items, each item checked); the fixture still holds the keys this item named; totals, days, tasks and clients add up to the same requests and dollars, and the savings are positive; each task’s routes add up to its requests and use only known routes
test_demo_endpoints_are_403_in_live_mode	INV-8: in live mode POST /ratchet/demo/day (also with bad arguments), break-next, student-outage and reset, and GET /ratchet/demo/sample-example, get 403 and change nothing in the ledger
test_student_sampled_once_with_production_params	FR-2.1, NFR-1: one student call per request with n = Settings.samples (3 and 5), temperature 0.7 and logprobs on, to the version’s adapter with the request as sent; an escalated or audited request adds exactly one teacher call
test_confidence_formula_exact	FR-2.2: two of three samples agreeing, the best majority sample at p = 0.9, give 0.4 × 0.9 + 0.6 × 2/3 (not the minority’s 0.99, not the weaker majority sample’s 0.6), from score() and from the router, which logs the same number
test_confidence_without_logprobs_is_agreement	FR-2.2: without token logprobs the confidence is the agreement: 2/3 for a split, exactly 1.0 for a unanimous student whose endpoint ignores logprobs=True
test_threshold_boundary_serves_at_equality_and_escalates_below	FR-2.3, FR-2.4, US-3: for a calibrated threshold and for the global default, a confidence equal to the threshold is served by the student (no teacher call, nothing stored) and one a float step below escalates (teacher answers once, its answer stored, the version still logged as the one that tried)
test_audit_rate_zero_and_one	FR-2.5, US-4: rate 0 never calls the teacher on served answers; rate 1 audits every one, logs each verdict, stores each answer and never changes what the user gets, and version_live_quality reads the result; an escalation is never counted as an audit
test_audit_rate_is_respected_statistically	FR-2.5, US-4: at rate 0.2 over 2,000 served requests (seeded), the audited share is inside the 99% Wilson interval, with one teacher call and one stored example per audit
test_cost_includes_audit_shadow_and_arbitration	NFR-3: cost and all-teacher baseline to the token in 11 cases (served, audited, escalated, student endpoint down, cold start, pinned task, shadow agreeing, unsure and arbitrated, an audit or an escalation as the shadow’s reference), with distinct token counts per model; the ledger row equals the result
test_replay_columns_are_logged_without_extra_calls	FR-16.5, FR-2.10: on 24 paths (served, audited, an audit or escalation that disagrees or is invalid, an invalid student answer, the student down, cold start, pinned, shadow agreeing, only an arbitration, an audit or escalation as the shadow’s reference, degraded, the fallback teacher, and escalations under a client’s temperature 0, logprobs, temperature 1, seed or logit_bias, for two choices, and an audit under temperature 1) the logged threshold, label_p, student_agree and baseline_est_usd are as designed and the teacher and student calls are exactly as many as before; student_agree is never set from an arbitration, a stand-in, an invalid answer, or an escalation the teacher answered under other conditions than an audit’s (a sampled or client-steered answer: temperature above 0, a seed, logit_bias, n choices; logprobs alone keep the label); without a teacher answer baseline_est_usd is the baseline to the bit; for n choices the formula counts the choices served, or the student’s answer once per choice
test_usage_is_the_served_answer_and_the_breakdown_every_call	FR-1.12, NFR-3: on the 11 paths of the cost test, Result’s usage is the served answer’s (the chosen student sample’s 30 + 7 tokens, or the teacher’s 120 + 45 with its 40 reasoning and 100 cached tokens), usage_breakdown lists exactly the calls made (student samples, served, audit, shadow, arbitration; None for the others), cost_usd equals the price of the breakdown to 1e-12, and the fingerprint is the live version’s or the teacher’s
test_usage_estimates_missing_counts_and_keeps_details_inside_totals	FR-1.12: a teacher that reports no usage gets prompt tokens estimated from the messages as JSON (non-ASCII kept) and completion tokens from the answer as served (reasoning stripped), while the breakdown keeps its zeros; an empty answer has 0 completion tokens; reasoning and cached tokens above their totals are cut to them
test_concurrent_requests_keep_the_ledger_consistent	NFR-6: 8 threads × 40 requests through one Ratchet: no request lost or logged twice, one teacher call and one example per audit, each row’s cost matches its own path, totals equal the sum
test_each_result_names_its_ledger_row	Result.request_id is the id of the row log_request wrote, on the student, pinned, teacher-only and cold-start paths, and for each of 100 requests from 4 threads
test_each_call_gets_the_parameters_its_role_allows	FR-1.13, FR-2.1, FR-2.5, FR-2.9, INV-1: with one client parameter set, the confident student gets n 3, 0.7, logprobs and its shaping fields, and the audit temperature 0 with the limit, stop and format; an escalation, a cold start, a pinned task and a teacher-only request send the teacher everything, logprobs included (FR-1.14), at the client’s temperature; a shadow samples like the live student and its arbitration is a check; without parameters both caps alone; the model is always the configured teacher; a Task object works as task
test_examples_store_the_conditions_the_teacher_answered_under	FR-1.13, FR-4.3, INV-2, INV-12: an escalation stores JSON mode and its stop (an email in it redacted and counted) as params, hashed with them; the same messages without conditions are a second example, and sampling fields that keep the teacher’s own distribution (temperature 0.8, top_p, a seed, a zero penalty) bump it; the completion is the teacher’s; an audit stores its own (check) conditions
test_reasoning_headroom_marks_answers_over_the_clients_budget	FR-1.13, FR-2.13, INV-16: with 600 tokens of headroom the teacher gets 620 for a client’s 20; an answer whose visible part is longer is served whole with finish_reason length, not stored, and a skip_example event names it; a short one after long reasoning is stored; an over-budget audit is not an audit; without headroom the teacher’s own stop stands
test_a_request_that_is_not_storable_stores_nothing	FR-1.13, FR-2.13, INV-2: RequestMeta(storable=False) on an audited student answer and an escalation stores and bumps nothing, still logs the audit’s verdict, and writes no skip_example event
test_bad_generation_settings_stop_startup	FR-1.13: a forbidden or malformed allowlist entry, or a negative cap or headroom, makes Ratchet(...) raise; Settings.gen_policy() parses a valid list
test_n_choices_are_scored_as_calibrated_and_choice_zero_is_the_one_choice_answer	FR-1.14, FR-2.1, FR-2.2, NFR-3: with scripted samples, n = 1 to 4 draws max(3, n) samples and reports and logs the one-choice confidence; served choices are the best scored sample then the agreeing ones by token probability, with usage (prompt once, every choice), breakdown, cost and baseline to the token; a confident student that cannot fill n escalates with route n_choices, the teacher writes n choices at 0.7, cost and baseline price the whole call, and only choice 0 is stored; an unsure student escalates on confidence
test_teacher_writes_n_choices_and_only_choice_zero_is_stored	FR-1.14, FR-4.3, INV-2, INV-10, FR-2.5: an escalation gets n choices at 0.7 or the client’s temperature, and its example is bumped once per request; cold start, pinned and teacher-only requests get n choices, pinned ones stored never; a teacher returning fewer choices gives fewer; an audit of an n = 2 student answer is one answer at temperature 0 without logprobs, judged against choice 0, with a baseline of one audit-sized answer per choice; the shadow samples 3 while the live student draws 4, and its disagreement with choice 0 is arbitrated
test_logprobs_come_back_only_when_asked_and_match_the_score	FR-1.14, FR-2.2: choice 0’s token logprobs spell its text and their mean probability is Scored.token_prob, with the client’s top_logprobs alternatives (the student got top_logprobs); none without logprobs, the student sampled as before; each of n choices carries its own; the teacher gets logprobs and top_logprobs only when asked, an audit never; a teacher that sends none gives None
test_empty_json_answer_scores_zero	FR-2.2, FR-2.6, FR-2.14: three {} answers to ticket-triage score 0 and {} never agrees with {} (a strict xfail until field comparators made the built-in fields required)
test_o_series_prefixes_refused (4 cases)	FR-3.2, US-8, INV-1: o1-preview, o3-mini, o4-mini and openai/o3-mini are refused under the OpenAI terms
test_closed_teacher_fails_at_startup	FR-3.2, INV-1: Ratchet(...) raises for gpt-4o and for an unknown teacher; the gateway’s Runtime refuses to start with gpt-4o in live and mock mode
test_closed_fallback_teacher_fails_at_startup	FR-3.2, INV-1, FR-2.12: Ratchet(...) raises LicenseError for a fallback teacher that is closed (gpt-4o, openai/gpt-4o, o3-mini, a Claude and a Gemini model) or unknown; the gateway’s Runtime refuses to start with it in live and mock mode; a distillable fallback is licensed up front (Ratchet.licenses)
test_examples_record_license_url	FR-3.3: for two registered teachers, the stored example has the teacher, license name and URL from the registry, the request hash, a split, and the BOM lists exactly that source
test_configured_models_are_licensed_and_day1_checked	INV-9 (offline proxy): every priced model in config.py except student is in licenses.REGISTRY and in WANT in scripts/day1_check.py, and the default teacher, judge and fallback student are priced

tests/test_openai_compat.py

OpenAI compatibility of /v1/chat/completions (FR-1.5, FR-1.10, FR-1.12, US-1): request validation, message normalisation, the error envelope, the limits, the rule that a client mistake is never a 500, and responses and stream chunks that pass the OpenAI SDK’s own models, with usage and system_fingerprint. Gateway tests reload the mock gateway with every setting they depend on pinned; the SDK tests drive the real openai package through tests/sdk_bridge.py (skipped if it is not installed).

Test	Covers
test_bad_request_gets_an_openai_error (22 cases)	["hi"] as messages, metadata: "x", a missing role, role robot, temperature 3, n 0 and 5 (above RATCHET_MAX_N), five stop strings, top_logprobs without logprobs, json_schema without a name, bad JSON (incl. NaN, 100,000 nested brackets, bad UTF-8, an empty body, an unpaired UTF-16 surrogate escaped or as UTF-8 bytes), a non-object body, a body over 2 MiB (413), 120,001 characters of input, an image part, and a bad field with stream: true: each gets the expected status, code and param in the envelope (with detail repeating the message), as JSON, with nothing logged
test_no_client_mistake_gives_500	Every value of a request that uses every field (developer role, text parts, tool calls, metadata, json_schema, tools) swapped for one of eight kinds of junk, or deleted, each judged by validate_chat_request first: one it refuses gets that exact code and param from the gateway; one it accepts gets 200 from a live-mode gateway with the teacher behind MockTransport and from the mock gateway (the generic mock world answers it). Never a 500; refused requests are never logged
test_unpaired_surrogate_is_a_400_in_both_modes	An unpaired surrogate in a system message sent with a demo ticket, in name, in a text part or in a key is 400 invalid_json with its JSON path in mock mode, and in live mode the teacher is never called; a valid escaped pair (an emoji) reaches the teacher as one character
test_unknown_v1_path_and_wrong_method	Unknown paths under /v1 and /openai get 404 not_found_error / unknown_url with Unknown request URL: METHOD path.; GET /v1/chat/completions gets 405 with Allow: POST; other paths keep {"detail": "Not Found"}
test_unhandled_error_is_a_500_envelope_and_an_event	A crash in the cascade gives 500 api_error without the exception’s message; the ledger gets gateway_error events (class name only) at the 1st and 10th crash; a crash outside /v1 keeps the plain-text 500 and is counted but not written (11th); when the ledger itself fails at the 100th, the 500 still goes out
test_mock_keyerror_is_400_and_live_keyerror_is_500	FR-9.7: with RATCHET_MOCK_STRICT=1 an unknown request, a request without a user message and a Scout writer request without JSON facts (also facts nested too deeply to parse) get 400 mock mode only answers demo requests and log nothing; without it the generic mock world answers all four with a 200; in live mode a KeyError from the cascade is a 500
test_framework_validation_errors_become_400_on_v1_only	FastAPI’s validation errors are 400 envelopes with param and code on /v1, and stay 422 {"detail": [...]} elsewhere
test_ratchet_routes_keep_detail	/ratchet/redact (400, and 413 over a 1,000-byte limit), /ratchet/versions/999/bom, /ratchet/demo/day and /scout/run errors keep {"detail": ...}
test_checks_run_in_order_and_refusals_cost_nothing	With a key and a quota of 1: no key and a bad body is 401 (invalid_api_key, WWW-Authenticate kept); a bad body with a key is 400 and uses no quota; the next good request is 200; the one after is 429 rate_limit_error with Retry-After and X-Should-Retry: false; a bad body at the quota is still 400; one ledger row
test_body_limit_declared_and_streamed	A declared Content-Length over the limit is 413; a chunked body driven straight into the ASGI app is refused at the chunk that passes the limit, with the rest left unread; outside /v1 the 413 is {"detail"}; a chunked body under the limit is answered; a client that disconnects halfway through its body gets a 400, not a 500; 0 turns the limit off
test_message_and_input_limits_come_from_settings	RATCHET_MAX_MESSAGES and RATCHET_MAX_INPUT_CHARS (text parts added up) give 400; 0 turns both off (a 301-message, 200,000-character request is answered)
test_developer_role_reaches_the_teacher_as_system	A developer message reaches the teacher, and the stored example, as system; RATCHET_MERGE_SYSTEM=1 merges the leading system messages with a blank line
test_dropped_keys_reach_neither_the_teacher_nor_the_ledger	FR-1.10, INV-12, FR-1.41: keys a message’s role does not use (a vendor field, annotations, a tool call’s index) never reach the teacher; a conversation with tool calls and results is a tool request, so its tool call’s arguments reach the teacher as sent and nothing is stored (route tools, uses_tools 1); redaction of stored tool turns is in tests/test_tools.py
test_text_parts_ticket_matches_the_string_form	A demo ticket sent as text parts gets the same message back as the string form, and one stored example, seen twice, under the string form’s hash
test_multimodal_parts_go_to_the_teacher_only	With RATCHET_TEACHER_MODALITIES=text,image_url and a live student, an image request reaches the teacher with its parts as sent, never the student, stores nothing and logs one teacher row; a text request on the same gateway is still served by the student; audio stays refused; the mock teacher answers the image request from its text part, and nothing is stored
test_generation_parameters_reach_teacher_student_and_audit	FR-1.13, FR-2.1, FR-2.5, INV-1: a live gateway, both models behind MockTransport: for max_completion_tokens 50, a stop, seed 7, temperature 0.9, model: gpt-4o and a vendor top_k, the escalated teacher’s body has max_tokens 50, the stop, the seed and 0.9 under the configured model, the student’s has 0.7, n 3, logprobs, the stop, the seed and 50, and a forced audit’s has 0 with the same stop and limit and no seed; top_k reaches the teacher only with RATCHET_TEACHER_PARAMS=all,top_k (the audit runs after the response in live mode, so the test waits for the sidecar first)
test_allowlists_and_token_caps_through_the_gateway	FR-1.13, NFR-3: max_tokens 10⁶ is sent as 8,192 to the teacher and 1,024 to the student; without logit_bias in RATCHET_TEACHER_PARAMS it is not forwarded; RATCHET_STUDENT_PARAMS=none leaves the student its cap; RATCHET_TEACHER_MAX_TOKENS=0 sends the client’s value; a request without parameters still carries both caps; a forbidden allowlist entry or a negative cap stops Runtime
test_a_cut_off_teacher_answer_reaches_the_client_as_length	FR-1.13, FR-2.13, FR-1.6: a teacher answer with finish_reason length is returned with length (and only the stream’s last chunk carries it), and is not stored; an upstream value clients do not know is returned as stop
test_finish_reason_is_tool_calls_only_with_tool_calls	FR-1.13, FR-1.1, FR-1.6, FR-1.41: a request with tools goes to the teacher alone with tools and tool_choice (a student that would serve anything is never called, nothing stored); a prose answer ends with stop, and so does a teacher that says tool_calls but sends none, streamed or not; a teacher tool call to a request without tools (default or RATCHET_TEACHER_PARAMS=all) is a 502 tool_calls_unsupported with X-Should-Retry: false, also when streaming, and stores nothing, while the same call to a request with tools comes back as message.tool_calls with finish_reason: "tool_calls"
test_mock_models_honour_max_tokens_and_stop	FR-1.13, FR-2.13, FR-9.2: in mock mode max_completion_tokens 3 cuts the teacher’s answer to 12 characters with length and stores nothing; a stop string ends it with stop and stores it with that condition; the real OpenAI SDK reads length
test_openai_sdk_round_trip_and_errors	The real OpenAI SDK: a ChatCompletion with the ratchet block, the same answer for text parts and the developer role, and BadRequestError with code and param for temperature 3 and for too much input
test_openai_sdk_auth_and_quota_errors	The SDK raises AuthenticationError with code invalid_api_key, and RateLimitError for the daily quota without retrying it (two requests sent with max_retries=2)
test_responses_are_openai_chat_completions_with_usage	FR-1.1, FR-1.12, US-1: on a live gateway (both models behind MockTransport), a student-served and a teacher-served response pass the OpenAI SDK’s ChatCompletion model with exactly OpenAI’s top-level fields, service_tier default, a fp_ fingerprint, message.refusal null, annotations empty and logprobs null; usage is one student sample’s tokens (10 + 5, not the 3 samples’ 15), or the teacher’s 40 + 320 with 300 reasoning and 16 cached tokens; the breakdown lists the student’s call and the served one; a cut-off teacher answer is length and still validates; the real SDK reads the same usage and fingerprint
test_mock_teacher_reports_its_reasoning_tokens	FR-1.12, FR-9.2, FR-2.9: with RATCHET_TEACHER_REASONING_TOKENS=120 the mock teacher’s response reports 120 reasoning tokens inside completion_tokens, total_tokens is their sum and positive, the fingerprint is the teacher’s; a pinned Scout planner request has usage from its one call and a breakdown with only served
test_system_fingerprint_follows_the_serving_configuration	FR-1.12: equal for answers of one version (and the documented hash of adapter, version, training-set hash and threshold); another after a promotion; the old one again after a rollback; another when the threshold changes; the teacher’s own, stable, also in every stream chunk
test_stream_chunks_validate_and_include_usage_ends_with_the_usage	FR-1.6, FR-1.12, US-1: student- and teacher-served, every chunk passes ChatCompletionChunk and carries the id, model, fingerprint, tier and null logprobs; with include_usage every chunk has usage: null and the last data chunk before [DONE] has no choices, the non-streamed usage and the ratchet block; without it (absent, false, {}) no chunk has usage and the block is on the finish chunk; stream_options without a stream changes nothing; the real SDK’s last chunk has the usage
test_provider_reads_cached_prompt_tokens	FR-1.12, FR-1.11: usage.prompt_tokens_details.cached_tokens reaches every choice of a call; a string, negative or boolean count, or details that are not an object, count as 0
test_n_choices_from_a_trained_mock_student	FR-1.14, FR-1.12, FR-1.6, US-1: the mock fleet trained on 40 demo tickets; n = 2 with logprobs gives two choices with the same labels, usage that adds up both (also in the breakdown), a valid ChatCompletion, each choice’s tokens spelling its content with one alternative each, and choice 0’s mean token probability equal to Scored.token_prob of the same seeded samples; streamed, choice 0’s role, content and finish chunks come before choice 1’s, every chunk validates as a ChatCompletionChunk, content and logprobs per index add up to the non-streamed ones, the role and finish chunks carry none, and include_usage ends with the same usage; without logprobs both are null; the real OpenAI SDK reads both choices and their logprobs
test_n_choices_escalated_over_http	FR-1.14, FR-1.13, INV-2: escalated, the teacher behind MockTransport receives n 2 at temperature 0.7 without logprobs, the student n 3, the response has both choices (usage 12) and one example is stored; with logprobs, top_logprobs 2 and temperature 0.3 the teacher gets them all and each choice spells its content, the example bumped once; n 4 draws 4 student samples; a teacher-only image request gets n too; a reasoning teacher whose tokens include <think> text returns only the answer’s tokens and nothing of the reasoning; a teacher that returns no logprobs gives {"content": null, "refusal": null} per choice, also in every streamed content chunk, and the response and chunks still validate
test_max_n_comes_from_settings	FR-1.14, FR-1.10: RATCHET_MAX_N=1 refuses n 2 (400 unsupported_value, nothing logged), 6 accepts 5 (five choices from the mock teacher), the default is 4, and 0, -1, 129, 2.5 or True stop Limits and the Runtime with a ValueError naming the variable
test_chat_request_fields	ChatRequest: max_completion_tokens wins over max_tokens, a string stop becomes a list, null fields are unset, unknown fields are kept in extra; conversations with tool calls, tool answers, legacy function calls and assistant messages dumped from a response (unused fields null) pass; valid forms of every optional field come back as sent; a non-string metadata key from a Python caller is refused
test_validation_rules (87 cases)	One case per rule in API.md: the body, messages, roles, content and parts, tool messages, assistant tool calls in full (id, type, function.name, function.arguments), legacy function_call arguments, refusal, every sampling range, n, stop, logit_bias, logprobs, metadata (17 keys, long keys and values, task names incl. a trailing newline), response_format and json_schema (name, type, depth 17, over 64 KB, strict), tools (129, custom type, names, parameters, over 64 KB), tool_choice, the legacy function fields, the string and boolean fields and stream_options.include_usage: each gives a 400 with the documented code and param
test_limits_and_their_off_switch	Limits: too many messages, too many characters in a string, in text parts, in a refusal part, and with name, refusal, tool_call_id or tool and function call arguments counted; keys a role does not use are not counted; exactly at the limit passes; 0 means no limit; max_n (4 by default)
test_parse_body_and_error_helpers	Strict JSON (Infinity, NaN, bad UTF-8 refused); a valid surrogate pair is one character (bytes, text, UTF-16), and an unpaired surrogate (escaped, as bytes, in a key, CESU-8) is refused with its JSON path in an error that is itself valid UTF-8; the status-to-type table for plain HTTP errors (401 invalid_api_key, 403, 404 unknown_url only for the router’s own 404, 409, 413, 429, 5xx api_error); headers kept; FastAPI validation errors turned into a JSON path
test_normalize_rewrites_only_what_it_must	developer → system (other keys kept), text parts joined by newlines, a refusal part becomes text, keys a role does not use dropped (assistant audio and annotations, vendor fields, tool_calls on a user message), tool and function calls kept to their documented fields, plain messages returned as the same objects, the caller’s list never changed, and merging leading system messages
test_non_text_parts_need_a_teacher_modality	image_url, input_audio and file parts, and a refusal part from the user, are refused with their exact param; with image_url allowed the message comes back as sent and multimodal is set; parse_modalities
test_normalize_is_identity_on_the_mock_world	Every mock ticket (every template, drift too) and every Scout request in the research world comes back as the same objects, with or without merging, and passes validation with its task, so example hashes, golden splits and the reference simulation cannot move

tests/test_live.py

The live-path tests (Live-path tests). Each runs offline: model endpoints are FakeOpenAIServer behind httpx.MockTransport, commands go to RecordingRun, and the Scout CLI reaches the gateway through AppTransport.

Test	Covers
test_nebius_trainer_writes_syncs_trains_and_hot_loads	FR-4.6, FR-8.2, US-10: commands in order (RATCHET_SYNC_CMD, then bash deploy/train_job.sh student-v1, each with check=True); night/train.jsonl holds exactly the rows as prompt-completion training rows with the template arguments (dataset.training_record, FR-8.42) and is on disk before the sync; then POST {root}/v1/load_lora_adapter with lora_name, lora_path under /data/adapters and the endpoint’s bearer token. {version} and {data} are substituted, RATCHET_ADAPTER_ROOT moves the path, no sync command means no sync, a failed job stops before loading, and an empty student_base raises. deploy/train_job.sh mounts the bucket at /data, reads /data/night/train.jsonl and writes /data/adapters/${VERSION}, the paths the trainer uses
test_load_adapter_tolerates_already_loaded_and_raises_otherwise	FR-8.2: the vLLM root is the base URL without /v1 (with or without a trailing slash, or no /v1 at all); no token, no header; a 400 saying “already” (any case) is tolerated; any other 400, a 401, a 404 (runtime LoRA loading off) and a 500 raise
test_bom_copies_never_leave_the_data_folder	FR-16.4: an adapter name that is a path (../../escaped, an s3:// URL, ..) gets no copy beside weights (a message says so) but still its boms/student-vN.json, which is still uploaded; nothing is written outside RATCHET_DATA_DIR; a version without a BOM writes nothing
test_reload_adapters_loads_live_canary_shadow	NFR-2: only live, canary and shadow adapters are reloaded, in version order, from RATCHET_ADAPTER_ROOT, with the bearer token; one already loaded is skipped over; nothing to reload makes no request; a 503 raises
test_live_main_tick_reload_night_offline	FR-8.4, FR-3.6, FR-15.2, FR-16.4: python -m ratchet.live tick, reload and night against a ledger file with 60 train and 6 golden examples. The night records its training job as a closed training_job compute interval at the default GPU rate, linked to the version; a night that skips records none. night builds the student and judge clients with the configured bases and keys, the judge with the teacher’s timeouts and retries and the student with its own timeouts and the teacher’s retries (no teacher stands behind it in the eval), runs the training command, hot-loads student-v1, samples it on every golden row as in production, asks the judge in JSON mode, admits it to canary and writes data/boms/student-v1.json, equal to the stored BOM and signed with RATCHET_BOM_SIGNING_KEY, and a copy in data/adapters/student-v1/ratchet_bom.json, which a second RATCHET_SYNC_CMD run uploads (the first, before training, saw none); the BOM’s training commitment covers the 60 rows and their leaves are in the ledger. reload then loads the canary; no command means night, which skips while the release is in flight (no sync, no BOM); the next trained night whose upload fails keeps both copies locally, keeps the night’s BOM, and exits non-zero naming RATCHET_SYNC_CMD; an unknown command exits with a message
test_live_runtime_wiring	FR-8.1, FR-8.3, FR-1.8, INV-7: the gateway’s live Runtime with transports, over HTTP. A confident student answers from the adapter named by the version, with the endpoint token and STUDENT_SAMPLING; an unsure one escalates to the teacher with the Token Factory key and model, and the answer is stored. A 500 from the student falls back to the teacher (student_error event); a 500 from the teacher is a 502 teacher_unavailable after two retries (waits injected). Without student_base, the hosted Nano on Token Factory answers under the global threshold, not the version’s. The mock warm-up never runs in live mode, and an unknown transport key is refused
test_stats_student_serving_kinds	FR-7.40, FR-8.3: /ratchet/stats student_serving per mode: mock (with the mock fleet’s student-vN adapter, never the fallback id); live without RATCHET_STUDENT_BASE, none on a cold start, then hosted_fallback with the fallback model, the global threshold and certified: false; live with it, endpoint naming a cold start’s canary, then the live version’s adapter, its calibrated threshold and its certificate. Reading the stats calls no model
test_hosted_fallback_is_never_certified	FR-7.40, FR-8.3: a live version whose eval carries a certifying certificate is not certified while the hosted fallback serves, and its threshold is Settings.confidence_threshold (what Ratchet.threshold holds the fallback to); behind an endpoint it is certified with the version’s threshold, and not once its certificate stops certifying
test_hosted_fallback_response_names_the_fallback_model	FR-7.40, FR-8.3: a live chat request the hosted fallback answers comes back with ratchet.served_by == "student" and model equal to the fallback id, the same id /ratchet/stats and /scout/status report as fallback_student; a Scout step built from the same result carries it
test_scout_status_has_fallback_student	FR-7.40: /scout/status keeps the guard’s keys and adds fallback_student and the same student_serving as /ratchet/stats
test_live_evaluator_uses_production_sampling	FR-5.1, FR-5.5: one cascade call per golden row with n = samples, exactly STUDENT_SAMPLING (read-only) and the student’s token cap as params (rows without stored conditions), the judge’s candidates under the same params; the judge grades free-text rows only, in JSON mode, with request, reference and candidate; its pass rate is reported; a judge answer that is not a clear {"score": 1} (not JSON, a list, a string or 2, no score) scores 0, while fences and reasoning around valid JSON are fine; no judge or no free-text rows means no judge calls
test_production_and_both_evaluators_share_student_sampling	FR-5.1, FR-2.1: Ratchet._sample, LiveEvaluator.evaluate and MockFleet.evaluate send the same sampling arguments, {"n": 3, **STUDENT_SAMPLING, "params": GenParams(max_tokens=1024)}
test_train_parse_args_defaults_and_row_contract	FR-4.5: train.py defaults (epochs, learning rate, rank, sequence length, batch, Mamba projections in the targets), required --data and --out, the same base model in config.py, train_job.sh and student_endpoint.sh; rows written by Night School (dataset.write_training) load back unchanged (blank lines skipped), pass train.check_rows, have the teacher’s stored answer as their completion and carry teacher and license; train.manifest (saved as ratchet_manifest.json) names the base model, the licenses and the chat-template arguments
test_scout_cli_over_http	FR-10.6, FR-1.7: python -m ratchet.scout against the mock gateway in-process, with an API key: every agent prints a line, --max-results reaches search, citations and the cost summary print, every call is logged under the key’s name with its task header (one planner and one writer request); the key also comes from RATCHET_API_KEY; a wrong key exits 1 with the 401 on stderr
test_sim_cli_writes_json_and_table	FR-9.4, FR-15.3: python -m sim.simulate (2 days × 200, mixed, --drift-day 0) prints one table row per day, the totals, the GPU-hour line with its recommendation (labelled simulation), every event and a line saying the run is a simulation with mock models, and writes the JSON results with the documented keys, cost included (48 endpoint hours), and exposure, rollbacks and served
test_sim_cli_compares_release_rules	FR-9.4, FR-16.1: --compare-release (with --promote-evidence for the sequential run and --canary-audit-rate) writes both rules’ results with the same requests and prints the comparison header; compare_table gives one row per rule for each version either rule rolled back (“-” for audits and day where that rule did not); --promote-evidence without --release sequential, a canary audit rate of 0 or 1.5 and an unknown rule are usage errors (exit 2)
test_mock_warmup_trains_a_student	FR-9.6: RATCHET_MOCK_WARMUP_DAYS=1 and RATCHET_MOCK_WARMUP_REQUESTS=300 give day 1, 300 requests of mixed traffic, a student-v1 in canary with a BOM and a warmup event, also in /ratchet/stats; the demo reset warms up again on a fresh ledger
test_trust_proxy_uses_forwarded_ip_only_when_enabled	FR-10.5: with RATCHET_TRUST_PROXY=1 the guard sees the first X-Forwarded-For address on /scout/run and /scout/stream (the peer address when the header is missing or empty); with it off the header is ignored; with it on, two forwarded clients each get their own hourly limit
test_day1_check_reports_missing_models_and_logprobs	FR-8.5, INV-9: scripts/day1_check.py against a fake Token Factory: no key exits 2 without a request; a missing model is reported (with the NVIDIA models that exist) and never called; missing logprobs and unparseable JSON are reported, and each model’s finish_reason is printed (INV-16 stores only stop answers); an HTTP 500 fails the check (exit 1); every model request has the bearer key, logprobs, JSON mode and max_tokens 400; then the teacher gets max_tokens 8,192 with a stop, a seed, a temperature and JSON mode, and a 400 there fails the check; then the json_schema probe (SCHEMA_FORMAT), whose fenced answer is checked as the gateway checks it; all present exits 0 with “All good.”
test_day1_check_probes_json_schema	FR-8.5, FR-1.40: the json_schema probe against a fake teacher: an answer inside the schema is [ok] (the body sends SCHEMA_FORMAT); a 400 refusing the format, an answer outside the schema ($.category) or one that is not JSON is a [warn] that still passes; a 500 or a body without an answer is [FAIL]
test_deploy_scripts_parse_and_dry_run	FR-11.1, FR-11.2: bash -n on every deploy/**/*.sh; DRY_RUN=1 bash deploy/vm/create_vm.sh prints the subnet, disk and instance nebius commands with the project id and <cloud-init.yaml> in place of the user data, never the private-repo token; a given subnet skips the lookup; a missing project, missing key file, reserved user name or bad preset stops before any command; backup.sh exits 0 in mock mode. Skipped where bash is missing
test_sim_cli_certificate_flags	FR-9.4, FR-16.3, FR-6.30, FR-6.31: --calibration ltt --cert-target 0.9 --cert-delta 0.1 reach every night’s certificate and the table’s certificates block (simulation); --paired-gate and --min-certified reach the gate’s policy; bad values exit 2
test_sim_db_and_the_replay_cli	FR-16.5, FR-9.4: python -m sim.simulate --db writes the run’s ledger to a new file and refuses an existing one and --compare-release (exit 2); python -m ratchet.replay --db prints one table per version (never pooled), labelled as simulation, with its suggestion, and --json with --audit-rate; a missing file, a bad sweep, target, audit rate, version or day range exit 2 with error: and create nothing
test_train_job_uses_job_create	FR-4.6 (W0): with a fake nebius on PATH (FakeNebius), deploy/train_job.sh student-v3 makes exactly one call, ai job create (never the interactive ai create), with --name ratchet-student-v3, the image, --platform gpu-l40s-a --preset 1gpu-8vcpu-32gb, --volume <bucket>:/data, --disk-size 200Gi, --timeout 2h and --container-command bash; --args splits (as a shell would) into -c and the training command with the dataset and adapter paths ratchet.live uses; --subnet-id and --registry-secret appear only when set; DRY_RUN=1 adds --dry-run and returns without polling; DRY_RUN=true, a version or base model that would break out of bash -c '...' and a missing image stop before any call; a refused job fails the script. Skipped where bash is missing
test_endpoint_stop_and_start_use_the_id	FR-8.2, FR-15.2 (W0): stop and start call ai endpoint list --format json, then ai endpoint stop|start --id <id from tests/fixtures/nebius_endpoint_list.json>; start is the default; RATCHET_ENDPOINT_ID skips the lookup; with no endpoint of that name start creates one and stop exits 1; nothing to do in the state already reached; status prints <id> <state> (by name, or by RATCHET_ENDPOINT_ID); DRY_RUN=1 prints start and stop instead of running them; a start or stop that nebius refuses fails the script. A stub ratchet.cost shows exactly which uptime records were made (up after create or start, down after stop, none in a dry run or after a refused call). Skipped where bash is missing
test_endpoint_lookup_never_guesses	FR-8.2 (W0): create refuses an existing name; output that is not JSON, items that is not a list, two endpoints with the name, an id that is empty or has a space, a name missing from a list with more pages, or a failing list stop start (and stop and create where it matters) before anything else runs; an empty list printed as nothing, {}, {"items": null}, [] and the like means start creates; STOPPING and DELETING refuse, a lower-case, numeric, missing or ERROR state lets Nebius decide; an unknown action or DRY_RUN value and a missing Python stop first. Skipped where bash is missing
test_endpoint_args_carry_the_model_card_flags	FR-8.2, NFR-5 (W0): create’s flags (image vllm/vllm-openai:v0.18.0-cu130, platform, preset, port 8000, --shm-size 16Gi, the bucket volume, token auth, runtime LoRA) and its vLLM arguments in order, split as the container does: the model card’s --mamba_ssm_cache_dtype float32, --max-model-len 8192, --max-num-seqs 64 and --default-chat-template-kwargs with JSON that parses to {"enable_thinking": false}; the token never printed; RATCHET_VLLM_IMAGE, RATCHET_SERVE_MAX_LEN, RATCHET_SERVE_MAX_SEQS, RATCHET_REASONING_PARSER_PLUGIN (adds --reasoning-parser nano_v3), RATCHET_SUBNET_ID and an empty RATCHET_SERVE_TEMPLATE_KWARGS take effect; DRY_RUN=1 adds --dry-run and records nothing; values that would break the one --args string or that vLLM would refuse after the GPU is up (not a whole number, not a JSON object, a relative plugin path, a single quote) and a missing token or bucket stop before create. Skipped where bash is missing
test_deploy_scripts_never_say_cost_nothing	FR-15.2 (W0): neither deploy script, deploy/CLAUDE.md nor docs/DEPLOYMENT.md says a stopped endpoint costs nothing (the mounted bucket is still billed); the endpoint script says so
test_deploy_variables_are_listed_and_documented	NFR-10 (W0): every RATCHET_* variable the two Nebius scripts read is read by config.py or listed in config.DEPLOY_ENV (part of KNOWN_ENV, which the traceability check holds to a row in CONFIGURATION.md), with no stale or duplicate name; all of them and DRY_RUN are documented
test_liverun_plan_refuses_over_budget	FR-8.20: python -m ratchet.liverun plan --max-usd 0.01 exits 3 and makes no call at all; the plan lists requests per release stage at the configured canary audit rate, 0.5 and 1.0, the audit rates as set, the gate profile and whether it is in force, the golden-size line and a USD row per round; at the default rates the 10% canary stage needs 7,398 requests against 737 at 0.5; plan_stage_requests equals a brute-force scan with binom_sf
test_liverun_refuses_mock_gateway_and_wrong_ledger	FR-8.20, INV-8: a mock gateway exits 2 after /healthz alone (no demo route); RATCHET_DB on another file than the gateway’s exits 2 after the one probe; a gateway without RATCHET_API_KEYS exits 2 naming liverun:<key>; no RATCHET_LIVERUN_KEY, a training command not on PATH or a script that is not there, and no open endpoint interval exit 2 before any paid call
test_liverun_end_to_end_offline	FR-8.20, FR-8.21, INV-8: the module run (liverun_world: a live Runtime on Token Factory and vLLM fakes, 3 rounds of 100 tickets, nights in process, a 50% control sample, the untuned arm): exit 0, two versions trained through live.main’s seams (canary, then shadow), only /healthz and /v1/chat/completions requested, every request with its liverun-<run>-r<k>-<i> id and the harness’s key, results.json with the documented keys and live.* values each with a source, the three liverun_* events, the models as listed, no key or base URL in the settings, client and server-side latency, the cost total as its parts, LIVE_RESULTS.md with claim anchors
test_liverun_rounds_are_keyed_by_client_and_id_range	FR-8.20: a request with another key between two batches of round 2 falls inside round 2’s id range, is counted as other_clients and is left out of its counts
test_liverun_writes_no_text_and_no_key	FR-8.20, INV-12, INV-13: no file in the run directory and nothing on stdout holds a key, a base URL or any 20-character piece of a request or answer the fakes saw; the liverun_* events neither
test_liverun_control_arm_is_never_stored	FR-8.21, INV-2, INV-10: the ledger has one row per cascade request (none for the control calls); the same rounds with --control-share 0 on a fresh world store exactly as many training and golden examples
test_liverun_control_call_matches_the_serving_teacher	FR-8.21: for escalated requests the control arm’s body equals the gateway’s teacher call (model, messages, max_tokens cap, temperature, no logprobs, n 1), with the Token Factory key
test_liverun_control_ratio_and_reasoning_tokens	FR-8.21: control reasoning tokens average 420 beside the formula’s 300; ratio_student equals stats.ratio_estimate on the pairs read back from requests.ndjson.gz and is above 1; teacher-served pairs give 1; the measured reduction and its interval; nine student-served pairs are insufficient and leave the value out, ten give the exact ratio
test_liverun_accuracy_against_template_labels	FR-8.21: with a teacher wrong on hard templates and invalid on one, cascade accuracy is 100% on easy and below on hard, invalid answers count as wrong, category and priority are reported apart, the control arm’s accuracy too, labelled “template labels”; a cold start without nights has no student pairs, so no measured baseline
test_liverun_usd_cap_stops_between_batches	FR-8.20: a teacher reporting 200,000 output tokens per call stops the run with max_usd (exit 1) after one batch, spend within the cap plus one batch, the control arm’s own cap skipping the rest of its sample, results and state complete
test_liverun_server_cap_and_rate_limit_responses	FR-8.20, INV-7: a 429 rate_limit_exceeded waits its Retry-After and retries once; a 429 budget_exceeded stops with server_cap; a 401 with auth; three 503 teacher_circuit_open in a row with teacher_down; failures above 10% of a batch with gateway_errors
test_liverun_night_failure_stops_with_partial_results	FR-8.20: a failing training command records night_failed (the class, never its output), exit 1, round 1’s results written and the training interval closed; resumed once fixed, the night runs again and trains before round 2, the failure kept on record; --on-night-failure continue runs round 2 on the teacher
test_liverun_until_settled_extends_the_round	FR-8.20, INV-5: a canary still in flight after a round’s requests gets extension batches until it settles; --max-extend 0 records settled: false, and the next night skips by itself (“still in canary”)
test_liverun_resume_continues_the_stream	FR-8.20: a host that goes away after round 1’s second batch stops the run; run --resume (with the base URL again: it is never written) sends every remaining index with the same id and template as an uninterrupted run, never one index twice, and only the lost batch is missing
test_window_percentiles_filter_by_route_client_and_ids	FR-8.20: Ledger.last_ids, client_window (one client’s rows in an id range, other_clients and no_client), events_since, and the per-route nearest-rank percentiles of liverun.window_stats on a hand-built ledger
test_liverun_report_bill_publish_and_verify	FR-8.21: report --bill-usd 12.34 --bill-note ... sets live.bill.nebius_usd, redacts the note and shows the bill next to the break-even line; bad amounts and notes exit 2; verify exits 0, and 1 after one edited number or a missing source; --publish refuses a stopped run without --allow-partial and a smoke run always
test_live_targets_stay_out_of_test_and_ci	FR-8.20: the Makefile’s test and lint recipes and the CI workflows never mention live-smoke, live-run or ratchet.liverun
test_liverun_untuned_arm_paired	FR-8.21: the hosted Nano (unsure on hard templates) covers less than the tuned adapter on the same golden rows; arms.untuned.paired equals certificate.paired of the two test vectors; the arm’s calls went straight to Token Factory
test_liverun_scout_phase	FR-8.21, INV-11: --scout-runs 1 with Tavily on a MockTransport runs Scout over /v1 with the harness’s key, one request id per agent call, its totals and Tavily credits recorded; no key or question text written; without TAVILY_API_KEY it is refused before any call
test_liverun_file_workload	FR-8.21: a JSONL workload with three mislabelled rows scores 9 of 12 against the file’s labels (priority 12 of 12), sends every row once in a pass, and a bad row names its line and field, never its text
test_liverun_gate_profile_and_the_arms_key	FR-8.20, FR-8.21, INV-13: run without the planned gate profile exits 2 before any call; --gate-profile as-set runs and records which settings were in force; NEBIUS_API_KEY_LIVERUN carries the arms’ calls while the gateway keeps NEBIUS_API_KEY, and is never written
test_liverun_control_budget_is_reserved_across_workers	FR-8.21: eight workers reserving $1 against a $3 control cap at once: exactly three win
test_liverun_repeat_run_and_report	FR-8.21: smoke --repeat-of runs the same plan and seed under a new run id (same texts, same order) and compares every value with the first run’s; report <first> --repeat <repeat> --publish attaches the comparison and copies the repeat to evidence/live/repeat/, both verify, an edited repeat table fails verify; a smoke repeats only a smoke, a repeat of the wrong run is refused, --resume with --repeat-of exits 2

The React app itself is checked with npx tsc --noEmit, npm test (pure helpers in web/tests/*.test.mjs under Node’s test runner, such as the Economics panel’s axis labels and scales and the story page’s honest bill) and a production build in CI (web job), and by Playwright screenshots during development (light, dark, 1440 and 390 px).

During development the v0.4 features were also checked end to end against a running gateway with the official OpenAI Python SDK (401 on a wrong key, streaming text identical to the non-streaming answer, the quota cutting one key off at its limit while another kept working, a signed BOM verifying with python -m ratchet.bom verify and failing once edited), and the dashboard’s Trust and privacy panel with Playwright in light and dark mode at desktop and phone widths. The dashboard’s Economics panel (FR-7.20) was checked the same way at 1360 and 375 px against the mock gateway, and against crafted /ratchet/cost bodies routed in the browser (self-host with 90 days, scale to zero with warnings and an open job, not enough data in live mode, an empty report, a 404 that hides the panel, a 500 and a malformed body that fall back to the /ratchet/stats summary), with no horizontal scroll on phones; a refresh that fails keeps the last report with a notice, and a demo day updates it. The Task registry and Resilience panels (FR-7.21, FR-7.22) were checked the same way at 1360 and 375 px: the outage drill run end to end in the browser (the countdown, the breaker opening during a day of traffic, the probe closing it the next day, the release log’s breaker entries); the task editor’s check, a refused spec with each error, bad JSON caught before sending, save, a no-change save, a copy of a built-in task and delete against the mock gateway; and a gateway with RATCHET_ADMIN_KEYS, where a save is a 401 without the key and recorded as ops with it, with nothing written to browser storage or cookies; no horizontal scroll on phones. The story page’s honest bill and Built to be checked chapters (FR-7.23, FR-7.24) were checked with Playwright in light and dark mode at 1360 and 375 px (full-page screenshots, no horizontal scroll), against the mock gateway (its own cost report, labelled Simulation (mock models)) and with every /ratchet/* call failing (the reference run, labelled Simulation); the source toggle, reduced motion (every bar and marker drawn at its final place), and the link to the dashboard’s redactor, which lands on it both after an in-app navigation and on a direct load of /dashboard#redactor, also while panels above it are still loading.

tests/test_cost.py

Utilization-aware cost accounting (FR-15.1 to FR-15.4, NFR-11). Hand-built ledgers run on the synthetic clock (day N starts at N × 86400 s) and every number is worked out in the test; the simulated days assert signs and trends only. A module fixture runs five simulated days of 1,500 tickets on a mock Runtime once (about 4 s).

Test	Covers
test_split_is_exact_on_every_cost_path	FR-15.1, NFR-3: the 11 cost paths of the contract test, each row’s student part (live and shadow samples), teacher part (escalation, audit, arbitration, pinned), cost_usd equal to their sum and to the contract test’s exact values, and the student tokens
test_pinned_and_teacher_only_requests_are_all_teacher_cost	FR-15.1, INV-10: a pinned task and a teacher-only request: no student call, every cent the teacher’s, zero student tokens
test_split_holds_on_every_row_of_simulated_days	FR-15.1, FR-15.2: 7,500 simulated requests with canaries, shadows, audits and arbitrations: every row’s parts add up (within 1e-12), one 24-hour endpoint interval per day and one training interval per trained version (600 s + 0.05 s per example from 22:00)
test_overlapping_and_nested_intervals_are_not_double_counted	FR-15.2: nested, overlapping and separate intervals merge into hours once; a pricier overlapping record bills only its stretch at its rate; clipping to a window; empty records count nothing; the per-day split adds up
test_open_interval_counts_to_now_and_to_the_window	FR-15.2: an open interval runs to now, never past the window, never negative; listed in open_intervals
test_endpoint_start_is_idempotent_and_stop_closes_it	FR-15.2: a second up (also from a second connection to the file) returns the open interval; an end before the start is refused and closes nothing; down while down is None; jobs get an interval per start and close by id with their version and note
test_concurrent_endpoint_starts_keep_one_open_interval	FR-15.2, NFR-6: eight connections to one file starting the endpoint at once leave one interval (only the write transaction can serialise them; it fails without BEGIN IMMEDIATE)
test_bad_compute_records_are_refused	FR-15.2: an unknown kind, a negative or NaN rate, True or negative GPUs, an infinite time, a rate above $1,000 per GPU-hour or too large for a float, more than 1,024 GPUs, a closed record that ends before it starts (11 cases)
test_training_interval_is_closed_when_train_raises	FR-15.2, FR-4.2: run_night(train_gpu=...) closes the interval at the failure time with a note and re-raises; a successful night records the job’s duration and version; without a rate it records nothing
test_report_math_on_a_hand_built_ledger	FR-15.3, FR-15.4, NFR-11: every total, both views (a negative GPU-hour saving, not clamped), s̄, break-even, the hosted-fallback price and crossover, observed volume, utilization, idle hours, the hosted_fallback recommendation, the daily rows, the headline following RATCHET_STUDENT_COST_MODEL, and the window arguments
test_recommendation_for_each_branch	FR-15.4: self_host (“Self-hosting pays.” with positive GPU-hour savings), hosted_fallback, scale_to_zero (idle days, and below break-even with a pricier hosted fallback), insufficient_data (no uptime; uptime but no student answer), an empty ledger with no division by zero or NaN
test_explanation_between_break_even_and_the_hosted_crossover	FR-15.4, NFR-11: above break-even and below the crossover the recommendation is hosted_fallback and the explanation says self-hosting beats the teacher with the GPU-hour savings, the hosted fallback’s crossover and the equal-coverage caveat, never “does not pay” or “below the break-even”; the same with the endpoint idle most hours (scale_to_zero)
test_break_even_uses_the_recorded_rate_and_never_says_pays_at_a_loss	FR-15.4, NFR-11: break-even and the crossover at the rate the intervals recorded (2 GPUs against a setting of 1); an endpoint above break-even with a training job that leaves the window at a loss is kept without the words “Self-hosting pays”
test_live_ledger_days_count_from_the_first_request	FR-15.3: a live ledger’s days start at its first request; uptime before it is day 0; idle time in hours (49 of 51); a one-day window takes that day’s hours; a reopened file reports the same days
test_live_ledger_counts_observed_volume_and_idle_time_by_hour	FR-15.3, FR-15.4: an endpoint up for one hour is credited with that hour’s answers only, the later ones are priced per token as the hosted fallback’s (with a warning), the daily row matches the totals, and a 24/7 endpoint with one request a day is told to scale to zero
test_last_days_before_the_first_live_request_keep_recorded_uptime	FR-15.3: before the first request, days=1 and days=7 keep the 5 hours the endpoint has run; the first request then starts day 1
test_simulated_day_leaves_an_endpoint_opened_through_the_api_alone	FR-15.2, INV-8: a simulated day writes its own closed interval, so an endpoint a visitor marked up after the day’s simulated hours stays open (no error), and an overlapping one is counted once
test_student_answers_without_recorded_uptime_are_priced_per_token	FR-15.3, NFR-11: student spend on a day with no recorded uptime stays in the GPU-hour view at its per-token price, stays out of the observed volume, and gets a warning
test_report_warnings_and_daily_cap	FR-15.3: a training job open for 30 hours is named with the command that closes it; the daily breakdown keeps the latest MAX_DAILY days while the totals keep every day
test_mock_demo_with_a_24h_endpoint_does_not_pay	FR-15.3, FR-15.4, NFR-11: five simulated days of 1,500 requests: per-token savings positive, GPU-hour savings negative, never self_host, observed below break-even; /ratchet/stats cost and totals.cost_model; ?days=1; 400 and 422 for bad days
test_simulated_endpoint_hours_follow_the_setting	FR-15.2, FR-9.4: RATCHET_MOCK_ENDPOINT_HOURS=8 records 8-hour days without events; 0 records none and the report says insufficient_data
test_post_endpoint_records_uptime_with_an_admin_key	FR-15.2, INV-13: 401 without a key, 403 for an API key, 400 for each bad body (12, including rates above $1,000, 1e308 and a 401-digit integer, which was a 500), up with rate, GPUs and note, up again unchanged, down, down again; endpoint events with the admin’s name and no key anywhere
test_post_endpoint_is_open_in_mock_capped_and_off_in_live	FR-15.2, INV-8: open in mock mode as mock-admin on the synthetic clock, the mock cap (409, never for down), 403 in live mode without admin keys, and the report public in live mode
test_cli_records_and_reports	FR-15.2, FR-15.3: endpoint up (with --at and --gpus), up again, down with an ISO time, report and report --json (the second up adds nothing), --days 1 meaning today, close training_job; exit 2 for a missing ledger (never created), a bad or future --at, an end before the start, a rate above the cap or too large for a float, too many GPUs, --days 0 and a bad setting
test_cli_runs_as_a_module_with_the_environment	FR-15.3: python -m ratchet.cost report in a subprocess with RATCHET_DB
test_deploy_script_records_uptime_and_never_fails_on_it	FR-15.2, FR-8.2: deploy/student_endpoint.sh start (creating the endpoint, since the fake nebius lists none) and stop (by the id the list gives) with FakeNebius record the endpoint in a real ledger through the CLI; with no ledger the deploy still exits 0 and says so; an unknown action exits 2. Skipped where bash is missing
test_v04_database_gets_the_cost_columns_and_table	FR-15.1, FR-15.2: a v0.4 file gains the split and token columns and compute_intervals; its old rows count as teacher spend (split_unknown_requests, a warning) and give no hosted-fallback price
test_bad_cost_settings_stop_startup_naming_the_variable	FR-15.3, NFR-11: a negative, non-numeric, NaN or 1e308 rate, fractional, negative or 4,096 GPUs, an unknown cost model and 25 endpoint hours a day each stop the runtime with the variable’s name (9 cases)
test_cost_settings_defaults_and_spellings	NFR-11: the defaults, a cost model in any case with spaces, RATCHET_PRICES_VERIFIED=1 changing the label, an empty model meaning per_token, and --at parsing

tests/test_stats.py

The exact statistics of ratchet/stats.py. Monte Carlo checks are seeded, counts-only and sized to run in under a second; the two marked slow repeat them at full size.

Test	Covers
test_binomial_tails_match_brute_force	binom_sf, binom_cdf and binom_logpmf equal sums of math.comb terms for every k of small n, edge probabilities 0 and 1 and 60 random cases at n = 120
test_binomial_tails_keep_tiny_values_and_refuse_bad_input	0.5^100 and 0.1^200 to relative 1e-12 and 1e-9 (summed directly, not as 1 minus almost 1); bad counts and probabilities raise
test_clopper_pearson_bounds	46/48 → 0.8746, 129/130 → 0.9640, 127/128 → 0.9635; s/s → delta^(1/s); for 50 random (s, n) the bound is where P(X ≥ s) = delta, from below, and the upper bound mirrors it
test_precision_pvalue_planner_and_mcnemar	required_rows: 0.95 → 59, 93, 124, 153 rows for 0 to 3 errors; 0.97 → 99, 157, 208, 257; 0.90 → 29, 46, 61, 76; the p-value falls as agreement rises; McNemar 4 vs 0 is 0.0625
test_ltt_fixed_sequence_stops_at_the_first_failure	10 correct points certify nothing (1.01); 80 correct at 0.97 and 40 wrong at 0.72 certify 0.75 and stop at 0.70 with its trace; grid order does not matter; bad grids raise
test_ltt_controls_the_error_rate	Confidences U(0,1) with P(correct) = c: over 300 seeded draws of 400 rows, the chosen threshold has true precision below the 0.80 target in at most 5% (+3 SE) of draws
test_e_value_oracles_and_confidence_sequence_oracles	e(0/2 below 0.92) = 29.77, e(0/1) = 4.22, e(20/20) = 0.2006, e(7/17) = 2.59e5, e(60/60 above) = 27.6, e(83/83 above) = 129.7, within 1%; CS(0/2) = (0.001, 0.931) and CS(35/38) = (0.710, 0.992) within 0.002
test_alt_grid_components_are_supermartingales	For p0 in 0.5, 0.8, 0.92, 0.97 and every true rate on the null side, each alternative’s one-trial mean factor is at most 1 (the validity argument, checked exactly); weights sum to 1, also near the edges
test_e_values_edges_and_input_checks	No trials: e = 1; a huge count stays finite (capped at exp(690)); the confidence sequence narrows with data; bad sides, probabilities and steps raise; the cached confidence sequence equals a fresh scan and still refuses bad counts (a bool, a list, a float, s > n)
test_e_process_false_rollback_rate_and_wilson_peeking	At a true rate on the 0.92 floor, checked after every audit (120 runs of 200): the e-process rolls back at most 5% (+3 SE); Wilson’s bound checked the same way does more than 10% (about 23%, simulation)
test_confidence_sequence_covers_the_truth_at_every_step	At p = 0.95, the true rate leaves the 95% confidence sequence at any step in at most 5% (+3 SE) of 120 runs
test_e_process_and_confidence_sequence_error_rates_full (slow)	The two checks above at 300 runs of 300 audits, at 0.92, 0.95 and 0.80
test_clopper_pearson_coverage_full (slow)	P(cp_lower ≤ p) ≥ 0.95 (−3 SE) for (20, 0.90), (50, 0.95), (100, 0.97) and (130, 0.99), 2,000 draws each
test_wilson_lives_in_stats_and_release_still_exports_it	release.wilson is stats.wilson; 18/20 → (0.699, 0.972)
test_kish_and_hajek	Kish’s n_eff of equal and unequal weights; the Hajek mean of a hand-computed example; empty or zero weights give None; bad input raises
test_ratio_estimate	R = 1.3 recovered from 400 synthetic pairs; the ratio and standard error equal the textbook formulas; equal weights change nothing; weighted case by hand; too few pairs or no x are insufficient (ratio 1.0, se None)
test_ratio_estimate_interval_covers_the_truth	R ± 1.96 se covers 1.3 in at least 90% of 200 seeded samples of 150 rows
test_certificate_on_the_held_out_half_holds_at_its_rate	FR-16.3: confidences U(0,1) with P(correct) = c, the threshold chosen on 200 calibration points by the point rule and certified at 0.80 on 200 fresh ones: over 300 seeded draws a certificate claims more than the truth in at most 5% (+3 SE), and some draws do certify
test_stats_cli_plans_bounds_and_tests	FR-16.3: plan prints 0 errors: 59 rows, 1: 93, 2: 124, 3: 153, bound 0.9635 and 0.8746, mcnemar 0.0625 and 1; impossible counts and a target of 1.5 exit 2; python -m ratchet.stats in a subprocess

tests/test_provenance.py

Merkle trees and the BOM v2 scaffold, and committed training sets.

Test	Covers
test_keyset_roots	The empty set’s root is 32 zero bytes, one key’s root is its leaf hash, hex and bytes agree, and the root depends on the set, not the order
test_keyset_proves_members_and_non_members	1,000 random members verify included (with their value) and 1,000 random non-members absent
test_keyset_agrees_with_a_brute_force_set	Sizes 0 to 40 over keys that share long prefixes: every key of a 120-key universe verifies included exactly when it is in the set
test_tampered_proofs_never_prove_inclusion	Flipping a byte of any sibling or of the terminal value makes a proof invalid; another key or an absent key never reads as included; an inclusion proof never reads as absent
test_malformed_and_overlong_proofs_are_invalid	A terminal leaf without the key’s prefix, 257 siblings, missing fields, bad hex and wrong lengths are invalid, never an error; duplicate or short keys raise when building
test_keyset_builds_and_proves_quickly	2,500 keys built and 200 proofs in well under 0.5 s
test_log_tree_matches_rfc_vectors	RFC 9162: the empty tree is sha256(‘’), the leaf hash of ‘’ is 6e340b9c…, the Certificate Transparency 8-leaf vector’s root is 5dc9da79…, and every prefix matches an independent implementation
test_every_inclusion_path_verifies_and_alterations_fail	Sizes 1 to 64: every audit path verifies (bytes and hex); an altered leaf or index, another tree’s root, a size the path cannot fit or an extra node fails; malformed input is False
test_bom_v2_manifest_sections	Schema v2 with every v1 field, redactor, training.origins; commitments only when given (ratchet.smt/v1, size checked against the rows); EXTRA_KEYS merged; unknown sections, a wrong size and a bad root refused
test_bom_verify_accepts_v1_and_v2_and_refuses_unknown_schemas	Signed v2 and v1 documents verify with the public key, an unsigned v1 document by its hash; an unknown schema and junk are refused
test_rows_report_their_origin	dataset.rows reports examples.origin, or live when it is not recorded
test_night_writes_a_v2_bom_with_its_redaction_setting	run_night(redact_pii=False) writes a verifiable v2 BOM with redactor.redact_pii false and the origins of its 60 rows
test_mock_gateway_bom_records_the_redaction_setting	A mock Runtime with redact_pii=False writes it into its first adapter’s BOM
test_certificate_endpoint_and_stats_check_on_a_signed_bom	FR-16.3, INV-14: on a mock gateway with a signing key, GET /ratchet/versions/1/certificate returns every key, the signed certificate, frontier and per-task block, the BOM’s hash, the gate’s evidence and three assumptions; 404 before training and for an unknown version; a re-score changes current and the status, not training; a version from before v0.4 (no BOM) and v0.5 (no certificate) answers no certificate; python -m ratchet.stats check passes the BOM, fails it after errors is edited (hash and numbers), after a re-hash (numbers), without the key’s signature, and passes a BOM from before v0.5
test_commitment_binds_each_examples_label_teacher_license_and_redactions	FR-16.4: dataset.commitment keys are the example hashes in order; each value is the SHA-256 of a canonical JSON written out independently in the test (non-ASCII kept); a different label, teacher, license or redaction count moves the root; the order of rows does not; the empty set is EMPTY; a duplicate key raises
test_version_leaves_are_written_once_and_found_by_key	FR-16.4, INV-17: set_version_leaves writes and reads back in key order, an empty split writes nothing, versions_containing finds versions by key and split; a second write to a split, an unknown version, a duplicate key, uppercase or short hex, a null value and an unknown split are refused with nothing written; SQL UPDATE is refused by the trigger
test_night_commits_the_exact_rows_it_trained_and_graded_on	FR-16.4, INV-3, INV-14: two nights; each BOM’s training and golden commitments equal dataset.commitment of exactly the rows the trainer and the evaluator got, size equal to examples and to the stored leaves; every golden key is absent from the signed training root; an example from night 1 is in both versions’ training sets (cumulative retraining), one from night 2 only in the second
test_a_failed_training_job_commits_nothing	A training job that raises leaves no version and no leaves
test_request_key_is_the_stored_hash	FR-16.4, INV-12: for a plain request and one with a developer message, text parts, PII, stop, response_format and a temperature, with redaction on and off, proof.request_key and resolve_key give exactly the hash Ratchet._store_example stored
test_resolve_key_validates_like_the_gateway	A request without a task is default, normalised like the gateway’s (developer role, text parts), metadata.ratchet_task and a body task (which wins); a key in any case; ten bad bodies refused with their reason; a part only the teacher reads refused because it is never stored
test_proofs_and_the_held_out_certificate_through_the_gateway	FR-16.4, INV-3, INV-6, INV-14: two mock days with a signing key: a v2 BOM whose commitments’ sizes equal examples; the sample train example (prompt only) is included with its stored value and the signed root, size and BOM hash, the sample golden example absent from training and included in the golden set, by request or by key; both verify with proof.verify_proof; the default sample is the live version’s (404 before there is one); the held-out certificate verifies with the exact success line; the CLI exits 0 for valid proofs and certificates and 1 for an unexpected result, another key’s signature, a tampered sibling, a file that lies about its result, another root, BOM, version or split, an edited golden leaf, a missing proof and a file that is not JSON (2 for a missing file); 404 for an unknown version and commitment not available without a BOM, 400 for a bad split or body
test_an_edited_leaf_fails_against_the_signed_root	FR-16.4, INV-17: with the trigger dropped and one leaf edited, the proof and held-out endpoints answer 500 (twice) and write one integrity event; a proof built from the edited leaves verifies against its own root but fails against the BOM’s (root mismatch, invalid proof)
test_a_v1_bom_still_verifies_and_has_nothing_to_prove_against	FR-16.4, INV-14: a signed v1 BOM verifies; verify and heldout exit 1 with no set commitment; bom.commitment is None for junk, another alg and a boolean size
test_verifiers_never_raise_on_malformed_files	FR-16.4: 14 malformed proofs (non-list siblings among them, the regression) and 11 malformed certificates each give FAIL: lines and no exception; a value in another hex case still verifies
test_proof_endpoints_need_an_admin_key_in_live_mode	FR-16.4, INV-8, INV-13: in live mode the proof and held-out endpoints are 401 without a key and 403 for an API key, admitted for an admin key (404: nothing trained), 403 admin API disabled without admin keys; the sample endpoint is 403 (mock-only) even for an admin
test_cli_key_prints_the_stored_key	FR-16.4: python -m ratchet.proof key prints request_key for a messages file (with --no-redact too) and for a request body with metadata.ratchet_task and stop; junk, a missing file, a string and empty messages exit 2

tests/test_privacy.py

Data governance: storage opt-outs and pseudonymous end users (FR-13.1, FR-13.2, NFR-4). Its invariants are INV-22 (opt-outs only reduce storage) and INV-23 (end-user identifiers are never stored or logged); their proving tests carry those markers.

Test	Covers
test_store_false_escalation_calls_the_teacher_and_stores_nothing	FR-13.1, INV-2: RequestMeta(storable=False) on an escalation: the teacher answers, Result.stored false, store_opt_out request, the row stored 0 with its reason; the storable request stores, an opted-out duplicate (reason key) does not bump it; no skip_example event
test_store_false_still_audits_and_arbitrates_without_storing	FR-13.1, INV-15: with audit_rate=1 the audit still records audit_agree 1 and nothing is stored; a shadow that disagrees is still arbitrated (shadow_agree 0) and the arbitration is not stored, while the same disagreement on a storable request stores it
test_zero_retention_stores_nothing_for_any_caller	FR-13.1, NFR-4: store_examples=False stores nothing for Scout’s call shape (no meta), a request with an end user, or an audited student answer (still audited); every row global, stored 0
test_zero_retention_mode_night_skips	FR-13.1: run_night(store_examples=False) skips with storage disabled before the trainer or the evaluator is called, although 60 stored examples would train; the default trains; the mock Runtime’s simulated day under RATCHET_STORE_EXAMPLES=0 serves 100 requests, stores nothing and skips
test_live_night_skips_under_zero_retention	FR-13.1: python -m ratchet.live night with RATCHET_STORE_EXAMPLES=0 prints storage disabled, runs no training command and creates no version
test_no_store_tasks_from_the_setting_and_the_spec	FR-13.1: RATCHET_NO_STORE_TASKS (spaces trimmed, unregistered names allowed) and a spec’s store: false give reason task; other tasks are stored; a space-separated list or an over-long name stops startup
test_examples_record_client	FR-13.2, FR-3.3: an example keeps its first client and end user; a duplicate from another client adds one example_clients row, its own duplicates, internal calls and opted-out requests add none; an example first stored without a client records the first client that sends it again
test_no_train_end_users_are_never_stored_even_when_registered_mid_request	FR-13.2: a registered end user’s escalation is served and not stored (end_user), another end user’s is; a registration made while the request waits for the teacher still keeps it out (the store-time check) and is reported like one found at the start (store_opt_out end_user in the result and the row, counted in not_stored_requests), without changing the caller’s RequestMeta
test_pinned_and_teacher_only_requests_record_the_end_user_and_opt_out	FR-13.1, INV-10: pinned and teacher-only rows record the end user’s hash and the opt-out, and store nothing
test_opt_outs_hold_under_concurrent_requests	FR-13.1: 80 requests from 8 threads, three quarters opted out three ways (request, key, no-train end user): exactly the 24 storable ones are stored, and every row’s stored matches its store_opt_out
test_store_false_over_http_escalation	FR-13.1: store: false over HTTP on an escalation: the student and the teacher are called once each, ratchet.stored false and store_opt_out request, nothing stored; store: true stores
test_store_header_metadata_and_key_opt_outs	FR-13.1: X-Ratchet-Store none, 0 and FALSE, metadata.ratchet_store "false" and "0", and a key with store=0 in RATCHET_KEY_LIMITS (also with store: true, X-Ratchet-Store: 1 and ratchet_store: "true") store nothing; explicit opt-ins store; any one opt-out wins; an unknown header or metadata value is a 400 before anything is logged; /ratchet/stats names the key and counts by reason; an empty X-Ratchet-Store is the default (stored), not a 400
test_end_user_is_an_hmac_and_the_raw_value_is_stored_nowhere	FR-13.2, INV-12, INV-13, NFR-4: with redaction off, user and safety_identifier (which wins) become 16-hex HMACs scoped per key in requests and examples; an empty user is none; a 309-character user is accepted and hashed whole (not as its first 128 characters); a non-string safety_identifier is a 400 that does not quote it; the raw values, a no-train value and the keys appear in no text value of any table; events carry no hash; clients[].end_users and privacy.no_train_end_users count
test_end_user_hash_survives_reopening_the_file_ledger	FR-13.2: the generated key survives reopening the file (same hash, no event); another file gives another hash; RATCHET_END_USER_KEY gives the same hash in any file; switching keys writes one end_user_key event without the key; a short key or a corrupted stored key raises
test_after_no_train_registration_escalations_are_not_stored	FR-13.2: POST /v1/ratchet/no-train with an API key registers the scoped hash without returning it (created true, then false); that client’s end user is no longer stored, another client’s same identifier is; 403 for another client, 200 for its own name; an admin registers once for every client (scope *), or for a named client (400 for an unknown one); a client cannot remove an admin’s registration (kept), removes its own (removed, and storage resumes); an admin’s removal for one client leaves the every-client registration (kept), its removal without a client clears it; absent; an admin registration over a client’s own protects it; no_train events name who and never the end user or its hash; 401 without or with a wrong key
test_no_train_requests_are_validated	FR-13.2: missing, non-string and empty end_user, an unknown field (unknown_parameter), a non-string client, a non-object body and bad JSON are 400s in OpenAI’s envelope; a 5,000-character end_user is accepted and never echoed; a keyless caller cannot name a client (403), registers in the keyless scope and gets created and status null either way; the mock cap (409) counts new registrations only, and a refused batch writes nothing
test_a_no_train_registration_holds_whichever_identifier_a_request_sends	FR-13.2: registered by user, a request that sends user and safety_identifier is still not stored; registered by safety_identifier, likewise; an admin’s every-client registration by user holds for a request with both; another client’s same values are stored; rows hold the safety_identifier hash and no alias
test_no_train_registrations_across_key_renames_and_keys_added_later	FR-13.2: a live file ledger across five startups: keyless registrations become unmatched once keys are on; an admin’s every-client registration covers a key added later and a renamed one; a client’s registration lapses when its key is renamed, and a no_train_unmatched event (names and counts only, once per change) and privacy.no_train_unmatched say so; the name back restores it; an admin’s removal without a client reaches the unmatched keyless scope
test_removing_a_registration_needs_a_key_in_live_mode	FR-13.2, NFR-5: a live gateway without API keys registers for anyone but removes only for an admin key (403 otherwise)
test_zero_retention_still_rescores_a_task_whose_spec_changed	FR-13.1, FR-2.15: under store_examples=False the night skips (storage disabled, no training) but re-scores the live version on the stored golden rows of a task whose spec changed (new spec_hashes, a rescored event, no example written); nothing to re-score the next night; no re-score while a release is in flight
test_every_add_example_caller_honours_storage_opt_outs	FR-13.1: every function in INV-2’s ALLOWED_ADD_EXAMPLE_CALLERS calls store_opt_out before add_example, so a writer added at a merge (a backfill) must honour opt-outs
test_metadata_is_echoed_and_the_ratchet_block_says_whether_it_was_stored	FR-13.1, FR-7.7: metadata echoed in the response, on the stream’s ratchet chunk (the finish chunk, or the usage chunk with include_usage) and nowhere else, absent when not sent, never stored
test_privacy_stats_report_opt_outs_and_clients	FR-13.1, FR-13.2, FR-7.7: /ratchet/stats privacy keys on a fresh gateway (zero counts, no_store_tasks from the setting and specs, no_store_clients), not_stored_by_reason after traffic, clients[].end_users and stored_share (3 of 5), and store_examples false and end_user_key env when set
test_key_limits_parse_and_refuse_typos	FR-13.1, INV-13: RATCHET_KEY_LIMITS parsing (spaces, empty fields, several keys); an unknown name (not quoted, in case it is a key), an unknown field, a bad store, a key or field twice and malformed entries raise; with keys off every name is unknown
test_storage_settings_refuse_typos	FR-13.1, FR-13.2: RATCHET_STORE_EXAMPLES accepts 1/0, true/false, yes/no, on/off in any case (empty is the default) and refuses anything else; a non-bool in code stops Ratchet; a short RATCHET_END_USER_KEY and RATCHET_KEY_LIMITS without API keys stop the gateway
test_ledger_from_before_opt_outs_gains_the_columns_and_tables	FR-13.2: a file from before this change gains requests.end_user and store_opt_out, examples.client and end_user, example_clients, no_train and the (end_user, ts) index (used by the query planner), twice without change; old rows read NULL

tests/test_erasure.py

Retention, erasure and export (FR-13.3 to FR-13.5, FR-4.4, NFR-4). Its invariants are INV-24 (an erased example is gone for good and leaves hashes only) and INV-25 (request retention never removes a release’s evidence, also across the stages of a canary); their proving tests carry those markers. Time is injected (now=, aged ts and last_seen_ts, a FakeClock for the gateway’s schedule).

Test	Covers
test_erase_a_client_purges_its_examples_and_shared_ones_by_policy	FR-13.3, NFR-4 (delete, keep): erasing acme purges its own examples and leaves globex’s; under delete an example both sent is purged, under keep it stays with globex promoted to first sender (no end user) and acme unlinked, and a second erasure finds nothing; the erasures row and the erase event
test_erasure_records_hold_hashes_and_no_text	FR-13.3, NFR-4: an erasure by client, end user and text purges one example and unlinks one request row; no table holds the text, the raw end user or the email in the reason (stored as [EMAIL]); the scope holds the HMAC and the SHA-256 of contains; erasure_items and tombstones hold the hash; events hold neither hash
test_dry_run_reports_and_writes_nothing	FR-13.3: a dry run returns the counts and the affected version and changes no table or event
test_affected_versions_are_reported_and_their_signed_boms_stay_unchanged	FR-13.3, INV-14: two signed nights; erasing an early train row, an early golden row and a later train row reports v1 with 1 and v2 with 2 (tainted, status pending); both BOMs are unchanged and verify; a golden-only erasure reports none (complete)
test_affected_versions_use_committed_training_sets_when_a_version_has_them	FR-13.3: a version with version_leaves is counted by membership, one without by time; leaves win over the time rule
test_bom_fingerprint_covers_kept_examples_and_counts_purged_ones	FR-13.3, FR-3.5: /ratchet/bom changes its fingerprint (the SHA-256 over the kept hashes), counts purged_examples and leaves the purged row out of sources; /ratchet/stats privacy counts
test_an_erased_hash_is_never_stored_again_and_a_retention_purged_one_is_restored	FR-13.3, FR-13.4: add_example refuses an erased hash from another client (None; nothing bumped; suppressed 1); a duplicate of a retention-purged one restores the text with its own teacher and license, clears the purge and records the client; dataset.rows sees it again
test_the_cascade_skips_a_tombstoned_answer_as_erased	FR-13.3, INV-16: after an erasure the same request is answered by the teacher and logged stored 0 with a skip_example erased event
test_erasure_makes_a_retention_tombstone_final	FR-13.3: a retention tombstone keeps its end user, an erasure by that end user makes it final (reason erasure, end user cleared, purge time kept) and refused
test_contains_matches_the_stored_text_literally	FR-13.3: %, _, quotes and backslashes are literal (look-alike rows do not match); ASCII case folds, É does not; the answer and the conditions are searched too
test_erase_refuses_a_bad_scope_without_quoting_it	FR-13.3 (13 cases): no selector, bad hashes (format, a string, a non-string, none), blank contains, a bad client name, a blank reason or actor, an end user without the key, empty or not a string, a bad shared; never quotes the end user; nothing recorded
test_erasure_holds_under_concurrent_stores	FR-13.3: three threads storing duplicates while 150 examples are erased: no error, every row erased, none revived
test_admin_erase_endpoint_needs_an_admin_key_and_is_a_dry_run_by_default	FR-13.3: 401 without a key, 403 for an API key; a dry run by default; eight bad bodies are 400s that never quote the end user; an end user without a client is erased in every key’s scope and the keyless one; events
test_a_client_erases_its_own_end_users_examples	FR-13.3: DELETE /v1/ratchet/examples erases the caller’s end user’s examples and unlinks its request rows, not another client’s same identifier; 401 without a key; an end_user in the URL is a 400 that does not echo it; 403 for another client’s scope; dry run; bad bodies; the actor and default reason; an admin erases across clients
test_keyless_erasure_callers_learn_nothing_and_admins_reach_removed_keys	FR-13.3: with API keys off, DELETE /v1/ratchet/examples answers a keyless caller with null counts for an identifier with data and one without (dry run and real); an admin’s erasure of an end user for every client, by POST and by DELETE, reaches a key no longer configured
test_retention_purges_examples_not_seen_within_the_window_golden_included	FR-13.4, NFR-4: train and golden rows not seen for 30 days lose their text (reason retention), a recently seen duplicate keeps its row, one purge event; a second run purges nothing
test_retention_purges_examples_a_chunk_per_transaction	FR-13.4: with chunks of 2, seven expired examples are purged in four transactions, each under the ledger lock and the lock free before each; the rows, the result and the one purge event equal a single pass; a chunk of 0 is refused
test_request_rollup_keeps_every_stat_identical	FR-13.4 (chunks of 20,000 and 7): folding half the rows (a day split across the cut), then the rest, leaves stats_by_day, stats_by_task, stats_by_client (but end_users), totals, the opt-out counts and cost_by_day (also a day window) equal within 1e-9
test_fold_dry_run_counts_and_writes_nothing	FR-13.4: a dry fold counts and writes nothing
test_retention_never_folds_rows_of_a_release_in_flight	FR-13.4, INV-25: a canary’s rows older than the window stay (its audited precision unchanged) until it is promoted
test_retention_keeps_every_stage_of_a_canary_for_the_anytime_valid_rule	FR-13.4, FR-16.1, INV-25: a canary’s rows from its 10% stage (two disagreements) stay after it moved to 50%, so ReleaseGate.evidence under the anytime-valid rule (12 audits, 10 agree; 4 in the stage) is the same before and after a fold; only the live version’s older rows go, and the canary’s go once it is promoted
test_an_erased_committed_example_keeps_its_leaf_and_is_never_sampled	FR-13.3, FR-16.4, INV-17, INV-24: erasing the example behind a version’s first committed leaf leaves the leaves unchanged, and Ledger.leaf_example (the mock demo’s sample) moves to the next leaf that still has its text, and to None once every one is purged
test_day_numbers_survive_folding_the_rows_that_defined_day_one	FR-13.4: a live file ledger keeps its day numbers and day 1’s start after folding the oldest rows, and after folding all of them
test_run_night_applies_retention_before_reading_the_ledger	FR-13.4, FR-4.4: run_night(retention=) purges expired rows before the dataset is built: the trainer gets 60 of 70
test_live_gateway_purges_hourly_off_the_request_path	FR-13.4: a live Runtime purges on its first tick on a background thread, not again within the hour, again after it; a failure is a purge_failed event with the class only; mock mode and zero windows never purge
test_bad_retention_settings_stop_startup	FR-13.4 (4 cases): a negative or too large window, a bool, or an unknown RATCHET_ERASURE_SHARED stop Settings.retention() and the Runtime; spaces and case are forgiven, empty is the default
test_retention_days_must_be_whole_numbers_in_the_environment	A non-number in RATCHET_REQUEST_RETENTION_DAYS stops startup naming it
test_live_purge_cli	FR-13.4: python -m ratchet.live purge with --dry-run, --client, --older-than-days, a bad window and no windows; a bad --client or a window past 36,500 days exits with a message, not a traceback
test_run_night_trains_again_after_deletions_once_50_new_examples_arrive	FR-4.4, FR-13.3, FR-13.4 (erasure, retention): after 60 of v1’s 100 rows are purged, 49 new rows skip (only 49 new examples) and 50 train on 90 rows without the purged ones (v0.4’s rule said -11); then three nights with no new data do not train (only 0 new examples; measured against v1’s purges they retrained every night), a deletion of 10 of v2’s rows is not new data either, and 50 new rows train on 130
test_a_purged_variant_still_counts_toward_its_requests_frequency	INV-3, FR-13.4: a golden prompt seen three times over a bare and a stop variant trains; purging the stop variant keeps its prompt key and its count, so the bare variant stays in training
test_export_round_trips_with_provenance_and_holds_no_keys	FR-13.5, NFR-4, INV-13: every kind streams NDJSON as an attachment; examples equal the ledger’s kept rows with their provenance and clients; requests have no end user; versions, erasures (items) and events; no API key, admin key, end-user key, end-user hash or raw end user in any line
test_export_filters_and_refusals	FR-13.5: split, task, client and since (ISO date, epoch, date-time) filters; 401 and 403; an unknown kind, a filter the kind does not take, a bad since, split or client name are 400s
test_export_pages_through_large_tables	FR-13.5: pages of 5 give all 23 rows in order, one line each
test_admin_cli_erases_and_exports	FR-13.3, FR-13.5: python -m ratchet.admin erase (a dry run, then --apply with the end user from stdin across every scope), its errors, and export to a file and to stdout, with a refused filter
test_admin_cli_erases_end_users_only_with_the_gateways_key_and_never_writes_otherwise	FR-13.3: python -m ratchet.admin erase --end-user in a shell without the gateway’s RATCHET_END_USER_KEY, with another key or a short one is refused (dry run and --apply) and the ledger’s meta, events and erasures stay unchanged; with the right key and no RATCHET_API_KEYS it finds the end user in every scope the ledger holds (a removed key’s too) and logs no key change; a gateway’s generated key is used from the ledger; a ledger that never hashed an end user is refused; a free-text --actor is refused
test_a_ledger_from_before_retention_gains_its_columns_and_tables	FR-13.3, FR-13.4: a file with the old erasures table gains the new columns and tables twice without change; last_seen_ts is backfilled from ts, and retention works on it

tests/test_licensing.py

Licenses that travel with the adapter and configured teachers (FR-3.50 to FR-3.53, NFR-7, NFR-9, INV-1, INV-14). It extends invariants 1 (configuration cannot relabel a known closed teacher) and 14 (the NOTICE is part of the signed BOM); the proving tests carry INV-1 and INV-14. The live night runs through live.main’s seams (run=, http=, provider_factory=), offline.

Test	Covers
test_write_notices_lists_base_and_teacher_licenses	FR-3.51, NFR-9, INV-2: notices.sources (URL, origin, distill_ok and count per teacher); NOTICE.md names the base model’s license and URL, each teacher with its license, URL and count, “(configured)” for a configured one, teacher outputs only and the Ratchet version; the same bytes whatever the order; a warning for an unknown base model; no sources; cells that cannot break the table; train.write_metadata writes ratchet_manifest.json and NOTICE.md from the container’s built-in registry
test_base_model_licenses_cover_the_configured_base_models	FR-3.50: the default student base model, the hosted fallback and the 30B checkpoints map to the NVIDIA Nemotron Open Model License, in any case; a local path, an unlisted id or a bare name do not; base_license’s warning
test_bom_v2_has_base_model_license_and_v1_still_verifies	FR-3.50, FR-3.51, FR-3.6, NFR-9, INV-14: base_model_license, each source’s license_origin, notices with the exact text and its SHA-256; a warning, not an error, for an unknown base; the signature covers the notice; v1 and early v2 documents still verify and get a rendered NOTICE; run_night writes it
test_live_night_puts_bom_and_notice_in_adapter_folder	FR-3.51, FR-3.53, NFR-9, US-10, FR-3.6: python -m ratchet.live night syncs, trains, writes data/adapters/student-v1/ratchet_bom.json (the stored BOM’s bytes, also in data/boms/) and the NOTICE.md the BOM signed, then syncs again (the second sync sees both files); it installs RATCHET_TEACHER_LICENSES itself, so the configured teacher is in the BOM and NOTICE; a night that does not train writes and syncs nothing
test_live_night_reports_a_failed_second_sync	FR-3.51: a failed second sync leaves the files written, writes a sync_failed event with the class only, prints the report and exits non-zero; without RATCHET_SYNC_CMD the files are written and nothing runs after training
test_export_bundle	FR-3.52, NFR-7, US-10, INV-14: export copies the adapter folder (nested files too), writes the verified BOM and the NOTICE it signed over the job’s copy, prints the BOM’s sha256 and returns a summary; python -m ratchet.bom verify --notice accepts the exported NOTICE.md and refuses an edited or missing one; refuses a non-empty folder or a file, a folder inside the adapter folder or anywhere in RATCHET_DATA_DIR, an unknown version, a version without a BOM, a tampered BOM, an adapter name outside the adapters folder, a missing --version, writing nothing; without local weights it writes the licenses and says where the weights are; adapter_files counts the adapter’s own files, not the job’s NOTICE.md; python -m ratchet.live export prints only the export’s lines
test_export_after_a_live_night_says_the_weights_are_in_the_bucket	FR-3.52, FR-3.51, NFR-9, US-10: after a live night with an upload-only sync the adapter folder holds only the BOM and NOTICE.md publish_bom wrote; export (and python -m ratchet.live export) then counts no adapter files, reports no weights and says where in the bucket they are, never that it exported the adapter; a folder with a stray file but no weights still gets the hint; once adapter_config.json and the weights are there, it ships them
test_config_teacher_license_allowed_and_closed_prefix_still_refused	FR-3.53, INV-1, FR-3.1, FR-3.2: configured teachers are licensed with origin config and canonical license names; the built-in registry is a read-only copy source; seventeen closed-family ids claimed as Apache-2.0 are refused (azure-gpt-4o, chatgpt-4o-latest, anthropic/next-model, openai/gpt-oss-120b, my_gemini_mirror, davinci-002, o5-mini…) and so are router aliases (openrouter/auto, .../auto:floor), while open ids pass the family check and an unrecognised closed id (x-ai/grok-4) is shown to pass it too; five license names outside the allowlist are refused; a built-in teacher cannot be relabelled; a closed id forced into the registry still raises; after a reset the teacher is unknown again
test_config_teacher_end_to_end	FR-3.53, INV-1, FR-3.3: a configured teacher is refused without the file and licensed with it; its answer is stored with its license and URL; Night School trains on its rows and the BOM and NOTICE name it as configured; a cascade built without the file publishes the built-in registry, and the rows leave training
test_bad_teacher_license_file_fails_at_startup	FR-3.53, INV-1: bad JSON, NaN, not a list, too many entries, and twelve entry errors in one file, each listed with its place; a missing file; the registry left as it was; Ratchet, the gateway’s Runtime in both modes, python -m ratchet.live and python -m ratchet.admin stop before opening the ledger; the fixed file starts

tests/test_feedback.py

Ground truth from people: request ids, verdicts and corrections, human labels as a provenance source, labelled-data import and golden-only rows (FR-13.30, FR-13.31, FR-3.54, FR-4.50, INV-1, INV-2, INV-3, INV-10). It amends invariants 1, 2, 3, 10 and 16 (human is never a teacher; human labels are targets, student outputs never; golden-only rows; no human label for a pinned task; the second writer of examples) and adds INV-26 (human verdicts never promote or roll back a version); the proving tests carry those markers. Gateway tests drive a live version whose student and teacher are RecordingProviders.

Test	Covers
test_response_has_request_id	FR-13.30: ratchet.request_id (req_ + 32 hex) on every response, the completion id with the same digits, stored in requests.request_id; a stream’s chunks carry it; internal calls have none; parse_request_id takes either form and nothing else
test_feedback_only_on_own_requests	FR-13.30, INV-13: another key’s request is the same 404 as an unknown one, without the id quoted; 401 without a key; one row per request, a later verdict replacing it; no text or key in the table
test_correct_verdict_never_creates_example	FR-13.30, INV-2: a correct verdict on a student answer stores nothing and calls no model; a correct verdict carrying the student’s text is a 400; task agreement in /ratchet/stats
test_corrected_feedback_stored_with_human_provenance_redacted_and_verified	FR-13.30, FR-3.54, INV-2, INV-3, INV-12: a correction stored as human, customer-provided labels, verified, origin feedback, with its client; messages and label redacted and counted; the teacher answer’s hash and its split; Night School reads it
test_human_overrides_teacher_on_same_hash	FR-3.54, FR-13.30: a correction replaces the teacher’s stored answer without counting the request twice; a later teacher answer only bumps seen; a JSON-mode request’s correction needs its conditions to replace that variant, and other conditions are a 409
test_teacher_model_human_refused_at_startup	FR-3.54, INV-1: human in any case as the teacher or fallback teacher stops Ratchet; never in the registry; a RATCHET_TEACHER_LICENSES entry named Human is refused; lookup_source
test_feedback_body_is_checked	FR-13.30: eleven bad bodies, each a 400 with its param and code (missing fields, a bad verdict or id, an unknown key, a non-string or invalid correction, messages without a correction or a correction without messages, an image part); nothing recorded
test_correction_storage_rules_still_apply	FR-13.30, FR-13.1, INV-10: the request’s own opt-out, the key’s store=0, a no-train end user registered after the request (the identifiers required and checked), a pinned task, RATCHET_FEEDBACK_CORRECTIONS=0 and zero retention each keep the correction out with the reason, the verdict kept
test_correction_needs_every_end_user_identifier	FR-13.30, FR-13.2: a request that sent user and safety_identifier logs both keyed hashes; a correction with one of them is a 400 (end_user_required), a wrong second one end_user_mismatch; with both, a no-train registration made with user (the client’s, or an admin’s for every client) keeps it out; one identifier on the request needs one; the second hash is never exported and an erasure by either identifier clears both
test_correction_conflicts_and_fixes	FR-13.30: the same request’s correction again replaces its own label (seen unchanged); other messages and a correct verdict after a stored correction are 409s
test_label_precedence_human_teacher_import	FR-3.54, FR-13.30, FR-13.31: teacher < correction < import; another client’s correction never replaces one; a later import replaces an import; a teacher import never replaces a label; a retention-purged human label comes back as the teacher’s; an erased example refuses a correction
test_store_label_is_atomic_across_processes	FR-3.54, FR-13.31: a second connection on the same file cannot write between store_label’s read and write (Ledger.atomic), so the import’s label stands and the teacher’s answer only bumps seen (from 0: the import was no occurrence); a refused golden-only row leaves no transaction open; an error rolls back
test_human_precision_alerts_and_never_moves_a_release	FR-13.30: one human_alert once min_audits verdicts give anytime-valid evidence that precision is below the floor, never again (meta row); the version stays live and the gate writes nothing; teacher-served answers count toward task agreement only
test_human_alert_is_anytime_valid	FR-13.30: the alert fires exactly when the mixture e-value against ‘precision >= live_floor’ reaches 1 / ALERT_ALPHA, checked after every verdict; 14 correct of 20 (Wilson upper bound under 0.92) does not alert
test_feedback_outlives_folded_request_rows	FR-13.30, FR-13.4: after request retention folds the row, the version’s human precision and the task’s agreement keep the verdict, and a new verdict on that request is a 404
test_concurrent_verdicts_keep_one_row_and_one_label	FR-13.30: eight identical corrections at once: one feedback row, one example, seen 1
test_old_ledger_file_gains_feedback_schema	an old file gains request_id, end_user_secondary, verified, golden_only, the feedback table and its indexes
test_import_openai_finetune_jsonl	FR-13.31, FR-3.54, INV-12: OpenAI rows, a multi-turn row with weights and content parts (earlier turns kept, address redacted), in-file duplicates and conflicts, six kinds of invalid rows with line numbers, tools hashing like a live request; imported rows start at seen 0; the import event (counts, attestation, SHA-256, no row text); importing again counts nothing twice
test_import_refuses_closed_model_source	FR-13.31, INV-1: twelve closed-family ids and a router alias refused with the clause quoted, also by the CLI before reading the file; unknown teachers, teacher:human and malformed sources; an open teacher’s rows keep its license, unverified
test_import_requires_attest_for_human	FR-13.31: no, empty, multi-line or overlong attestation refused
test_import_refuses_pinned_task	FR-13.31, INV-10, FR-13.1: pinned, no-store (spec and setting), zero retention and unregistered tasks refused before anything is stored; rows of a task pinned later leave Night School; store_human_example refuses a pinned task
test_golden_only_rows_never_trained	FR-4.50, FR-13.31, INV-3, INV-6: golden-only rows whose hash says train stay golden, past FREQUENT and with every variant of the prompt; prompts already trained on are refused (trained); a real run_night trains on none and its BOM counts golden.human_labels
test_labels_that_are_no_occurrence_never_reach_frequent	FR-13.31, FR-13.30, INV-3: an imported variant of a twice-seen golden prompt (and a non-bumping correction) starts at seen 0 and leaves the prompt golden; the third real request makes it frequent; an imported prompt needs FREQUENT real requests; the export reports 0
test_golden_only_check_reads_the_ledger_at_each_row	FR-4.50, FR-13.31, INV-3: a train-split row the gateway stores while an import runs keeps that golden-only row out (trained); the check is per row, in the transaction that writes it
test_bom_lists_human_source	FR-3.54, FR-13.31, INV-14: sources lists teachers only, label_sources the human labels by origin, training.targets and the NOTICE’s Human labels section; an adapter without them gets the earlier NOTICE text; the ledger’s /ratchet/bom lists them apart
test_import_ratchet_train_jsonl_roundtrip	FR-13.31: Ratchet’s train.jsonl imports back with the same hashes; a teacher’s rows cannot come in as human labels or under another task
test_import_cli_dry_run_and_apply	FR-13.31: python -m ratchet.importer dry run (refused for a ledger file that does not exist, which it does not create; an older file read without being migrated, byte for byte), apply (rows attributed to --client, one event), a missing file, a bad --client, a line over the body limit skipped
test_counts_as_train_reads_human_labels	a human training row counts toward the new-data count, a golden-only row never
test_export_carries_label_flags	the NDJSON export carries verified, golden_only and the provenance

tests/test_receipts.py

Verified savings receipts (FR-15.5). Synthetic populations are seeded; simulation runs are a few days of mock traffic and assert trends.

Test	Covers
test_leaves_hold_only_metadata_and_the_key_name	INV-12, INV-13: through a gateway with an API key, a request’s leaf has exactly LEAF_KEYS, whole numbers and strings only, the key’s name; neither the receipt, the leaves nor the proof holds the request’s text, its answer or the key; no float anywhere in the manifest
test_leaf_turns_amounts_into_whole_nano_usd	leaf rounds USD to nano-USD (half to even), label_p to parts per billion and ts to milliseconds; NaN and missing amounts are null; '' clients are null; leaf_bytes is canonical JSON
test_ratio_estimator_recovers_the_ratio_and_its_lower_bound_covers	y = 1.3 x (1 + 0.1 z) audited at 20%: the mean R over 200 seeded populations is within 0.01 of 1.3, and the one-sided 95% lower bound covers the true total in at least 180 of 200 (about 95% in 2,000)
test_weights_follow_label_p_and_the_finite_population_correction	Answers audited at two rates are weighted 1 / label_p (R equals the hand-computed Horvitz-Thompson ratio, n_eff below n); with every answer audited the variance is 0 and the point is the exact total
test_insufficient_tasks_are_not_billed_and_the_fee_is_a_share_of_the_lower_bound	A task under min_audits is insufficient_audits and only counts at its recorded baseline; the billable set is the teacher-served requests and the calibrated task’s served answers; margin = ⌈1.645 se⌉; fee = ⌊share × net⌋ (share as a float or a string); spend outside requests comes off; a negative bound or no calibrated task bills 0 (formula); bad shares, minimums and spend raise
test_stand_in_answers_are_never_billed	UNVERIFIED_ROUTES equals the router’s STAND_IN_ROUTES; a degraded answer and a fallback teacher’s answer change neither the calibration nor the bill, and count at their recorded baseline
test_unlogged_formula_estimates_are_never_calibrated	Served answers without baseline_est_usd (before v0.5) make the task unlogged: no point, no bill
test_integer_sums_are_exact_over_100k_rows	100,000 synthetic rows: the totals equal the sum of the rounded amounts exactly, within half a nano-USD per row of the float sum
test_seal_is_idempotent_and_never_updates	INV-17: a second seal and seal_due return the stored document and issue no INSERT, UPDATE, DELETE or REPLACE (set_trace_callback); add_receipt refuses an existing (period, client)
test_period_rows_are_read_in_chunks_with_the_lock_free_between_them	period_rows reads RECEIPT_CHUNK rows per locked statement: another thread logs a request between chunks; the chunks equal the one read, in id order, a client’s rows only for a client; receipt_row reads one row
test_concurrent_sealing_writes_each_receipt_once	8 threads sealing at once write each (day, client) receipt once with no error; a second connection to the same file that loses the race keeps the first writer’s document
test_a_day_is_sealed_only_once_it_is_over	The current day raises PeriodOpen; a day without requests has no receipt; on a live ledger a day is closed 5 minutes after it ended, not before
test_signature_verifies_and_edits_break_it_and_the_chain	Three chained days: prev_sha256 links each to the one before; the signature verifies with the public key and fails with another; an edited total fails, also re-hashed; an earlier receipt with a lowered fee, or the wrong day, breaks the chain check
test_receipts_are_chained_per_client_and_hold_only_that_clients_ids	Two clients and anonymous traffic over two days: each client’s receipt holds only its ids, rechecks from its own leaves and chains to its own previous receipt; the gateway’s covers every request; GPU-view notes
test_spend_outside_requests_is_netted_and_shared_by_request_count	A backfill spend row comes off the gateway’s bound in full and off a client’s by its share of the day’s requests
test_recheck_recomputes_everything_from_the_leaves	recheck passes on the stored leaves; a changed amount (root and totals), reversed order, a missing leaf and a fee beyond the tolerance are caught; an estimated amount within 2 nano-USD is not
test_verifiers_never_raise_on_malformed_input	8 junk documents: verify_receipt, verify_chain, verify_proof, verify_rows and check_manifest return failures, never an exception
test_gateway_returns_the_request_id_and_its_proof_verifies	A signed mock gateway: X-Ratchet-Request-Id and ratchet.row_id (also on the last stream chunk; ratchet.request_id is the feedback id); the day is 409 until it ends, then sealed by the demo day; the listing, the signature, the GPU view; each request’s proof verifies with merkle.verify_inclusion and verify_proof; unknown ids 404, bad periods 400; the leaves recheck; the next day chains to it
test_api_keys_read_only_their_own_receipts	With API and admin keys: client B gets 403 on client A’s receipt, proof and leaves, 404 for A’s request in its own receipt; no key or a wrong key 401; an admin reads the gateway’s and every client’s receipt, and the listing per caller
test_live_mode_without_keys_serves_no_receipts	Live mode without keys: 403; the mock demo without keys is open
test_rows_changed_after_sealing_are_caught	A row slipped into a sealed day: proof and leaves answer 500 (twice) with one integrity event, and the receipt is unchanged; a row edited after its tree was cached is caught too (the proof reads its row again)
test_request_retention_never_breaks_a_receipt	FR-15.5, FR-13.4: a signed mock gateway; day 0 is sealed and a proof served (its tree cached); a fold takes day 0’s rows and one of day 1’s: day 0’s receipt is unchanged and verifies, its proof (from the cache too) and leaves answer 410 folded by request retention, never the integrity 500; day 1, folded before it was sealed, is never sealed from the row left (410, one receipt_skipped event however often it is asked for) and day 2 chains to day 0
test_live_purge_seals_every_ended_day_before_it_folds	FR-15.5, FR-13.4: python -m ratchet.live purge with a 2-day request window on a live ledger seals days 1 and 2 (the client’s and the gateway’s receipts, 2 and 1 leaves) before it folds their 3 rows; no receipt_skipped
test_bad_receipt_settings_stop_the_gateway	5 cases: a take rate above 1 or NaN, a minimum of 1 audit, a negative or huge mock reasoning mean stop the gateway with a message naming the variable
test_receipt_settings_are_read_and_checked	Defaults 0.25 and 30 and no mock mean; the variables read and no longer reserved; share_text
test_mock_teacher_reasoning_varies_around_its_mean_deterministically	The default bills exactly 300; with mean 420 and cv 0.3 the same request always costs the same, the mean is 395 to 445 over 400 requests, out_tokens follow; cv 0 bills the mean
test_sealing_moves_no_simulation_number	--receipts on, off and with seal_due monkeypatched out: identical days, totals, versions, nights, events and serving; with the formula right (300 tokens) the bound held and R is about 1
test_receipts_measure_and_correct_a_teacher_that_reasons_more	--teacher-reasoning-mean 420 (simulation): each calibrated day’s ticket-triage R is above 1.2, the formula’s baseline is below 95% of the true cost and the calibrated one within 10% of it, a fee is billed on the bound, and the bound is at or below the truth on all but one day at most
test_cli_verifies_a_bundle_and_fails_after_a_one_digit_edit	python -m ratchet.receipt verify with the key, the previous receipt, a proof and the leaves: four OK lines, exit 0; one digit of the fee changed: exit 1 (sha256 does not match); not JSON: 1; a missing file: 2
test_cli_module_entry_point	The same as a subprocess (python -m ratchet.receipt), with an unchecked chain reported
test_offline_seal_of_a_simulated_ledger	python -m ratchet.receipt seal --db on a sim.simulate --db ledger: every logged day sealed and labelled simulation, a day after the run 2, nothing left to seal 0, a missing file and --client without --period 2
test_live_seal_command_seals_every_day_that_ended	python -m ratchet.live seal on a live ledger seals every client’s and the gateway’s receipts of the days that ended (not today’s), then says nothing is left; labelled live traffic

tests/test_perf.py

The hot path (FR-12.8, NFR-1). See Hot-path tests for how the timing tests stay stable.

Test	Covers
test_file_ledger_uses_wal_and_memory_does_not	A file ledger: WAL, synchronous=NORMAL, 5 s busy timeout, a read-only second connection that sees each commit at once and refuses writes; :memory: has none; an old rollback-journal file is switched on open and gains wall_ms
test_sync_full_keeps_the_rollback_journal_and_bad_values_stop_startup	RATCHET_DB_SYNC=FULL (any case) takes a WAL file back to the rollback journal with synchronous=FULL and no second connection; anything else raises naming the variable; config.SYNC_MODES matches the ledger’s
test_full_cannot_take_a_file_out_of_wal_while_another_process_holds_it	FULL on a file another connection holds in WAL stops with an error naming the setting; NORMAL alongside is fine
test_gate_queries_search_their_indexes	EXPLAIN QUERY PLAN of the ledger’s own query constants: the audit tally searches requests_version_audited from the last id, the shadow verdicts requests_shadow_version, the routing rebuild versions_state, a day’s percentiles requests_day; none scans requests
test_version_cache_sees_another_connections_writes	Two ledgers on one file (gateway and cron tick): a new live version, canary steps, a rollback and a new eval written by one are seen by the other on its next read
test_version_cache_invalidates_on_state_change	Every version write (create, eval, state, traffic, BOM, and SQL on ledger.db that bypasses the writers) is seen on the next read; each read is a fresh copy, so changing one changes nothing cached; version(id) agrees
test_routing_reads_no_version_rows_until_a_version_changes	With 31 versions, ten routing rounds run only PRAGMA data_version; the next version write makes the next read rebuild once
test_live_quality_tally_matches_a_full_count	version_live_quality equals one COUNT(audit_agree)/AVG over the stage after new audits (read above the last id), audits without a verdict (pending off-path audits, counted in neither number), other versions’ and teacher rows, a new stage, a deleted row, and rows another connection commits
test_bom_digest_is_cached_until_examples_change	The BOM digest is recomputed after an insert, a rewritten or deleted hash, or another connection’s insert, not after a duplicate; never cached against an uncommitted write; always equal to a fresh computation
test_events_last_reads_only_the_newest	events(last=n) is events()[-n:]; 0 gives none; a bad n raises
test_stats_reads_skip_the_writer_lock	On a WAL file every stats query returns while another thread holds the writer’s lock
test_tick_throttle_runs_the_gate_once_per_interval	With a 5 s interval on a fake clock: one gate tick per interval, none while another thread ticks (without waiting), force and simulated days always tick, a failing tick is not retried on every request, interval 0 ticks every call
test_request_path_ticks_at_most_once_per_interval	Through /v1/chat/completions: three requests within the interval share one tick; the next one after it ticks
test_snapshot_cache_ttl_and_single_flight	SnapshotCache: one value per TTL; during a recompute other callers get the previous value at once, and the very first callers wait; a failed recompute raises and keeps the old value; TTL 0 computes every time
test_stats_ttl_returns_the_cached_payload	/ratchet/stats with a 2 s TTL returns the same payload within it despite a request, the new one after it, and a fresh one after a demo reset
test_interval_settings_have_mode_defaults_and_bounds	5 s and 2 s in live mode, 0 in mock; 0 to 3600 s; negative, too long, NaN, infinite, bool and unparsable values raise naming the variable, also at gateway startup
test_latency_percentiles	Nearest-rank p50 and p95 of wall_ms, exact, by day and served_by; q 0 and 1 are the minimum and maximum, 0.07 of 20 is rank 2; old rows without wall_ms count toward nothing; None without rows; bad quantiles raise
test_stats_by_day_reports_latency_percentiles_and_sorts_only_changed_days	stats_by_day’s p50_student_ms and p95_ms equal latency_percentiles; a second read sorts nothing, a new request sorts only its day, a deleted row every day
test_wall_ms_logged_and_at_least_latency	Over OpenAICompatProvider and a fake server, cold start, student-served, escalated, pinned and teacher-only rows each have wall_ms >= latency_ms > 0
test_wall_ms_is_the_timer_around_the_whole_cascade	With Ratchet(timer=...), wall_ms is exactly the two timer reads apart, on handle, a pinned task and teacher_only; latency_ms is the provider’s
test_handle_overhead_does_not_grow_with_versions	Zero-latency models: the mean request time with 365 retired versions is at most twice the one-version mean plus 0.5 ms (v0.4’s lookups took about 25 times as long)
test_perf_guard_200k_requests	A WAL ledger with 200,000 requests and a live, canary and shadow version: a cold tick under 250 ms, a warm tick under 20 ms (median), a request under 5 ms of its own time, the tally still exact

tests/test_security.py

Endpoint access, scrubbed public stats, the OpenAPI switch, the live-mode auth guard and security headers (FR-12.7, NFR-5, INV-13). A private live gateway runs offline through Runtime(transports=) with three kinds of keys.

Test	Covers
test_every_route_has_an_access_level	Every route the app serves has a level in access.ROUTES or access.PREFIXES, and the Access column of docs/API.md shows the same one; HEAD reads as GET, the prefixes cover routes other packages add (/ratchet/requests/..., /v1/ratchet/no-train), /livez, /readyz, /metrics and /ratchet/kpis are pre-classified, and an unknown route is not guessed
test_private_live_dashboard_needs_an_admin_or_dashboard_key	Every dashboard and BOM route of a private live gateway: 401 with WWW-Authenticate: Bearer without a key or with a wrong one, 403 for an API key, 200 for an admin or dashboard key (Bearer or X-API-Key); health, the BOM public key, the pages and Scout stay open; a dashboard key cannot write tasks (403) or call /v1/* (401); keyed readers see client names and no key reaches the payload or the database; RATCHET_PUBLIC_BOM=1 opens only the BOM endpoints
test_private_dashboard_without_reader_keys_is_closed_to_everyone	No admin or dashboard keys: every read is 403 naming RATCHET_DASHBOARD_KEYS, also with an API key
test_mock_dashboard_is_public_by_default	Mock mode: the dashboard API is open with or without admin and API keys, and unscrubbed; RATCHET_PUBLIC_DASHBOARD=0 closes it like a live one
test_public_live_stats_are_scrubbed_for_anonymous_readers	A public live dashboard: anonymous readers get client-1, client-2 by sorted name (the same pseudonyms in task allow-lists), a v0.4 ConnectError: ... event cut to the class, URLs as <url>, no client name, address or key; admin and dashboard readers get the full payload from the same cached one (never changed by a scrubbed copy); RATCHET_SCRUB_STATS 1 and 0
test_scrub_helpers	scrub_detail on v0.4 and current event formats (today’s class-and-status details are kept), pseudonyms, scrub_stats leaving its input unchanged, scrub_task
test_live_mode_without_api_keys_does_not_start	A live Runtime without API keys raises before the ledger file exists; allow_unauthenticated lets it start; mock needs no keys; the same from the environment as uvicorn loads it
test_each_key_has_one_role	A dashboard key that is an API or admin key, a short one, or an admin key that is an API key stops startup naming the variable
test_access_switches_resolve_by_mode_and_refuse_typos	Mock and live defaults; explicit 0 and 1; set but empty is the default; yes, true, 2, on stop startup naming the variable; a CSP with a line break is refused; Access.scrubs for every case
test_openapi_is_off_in_live_mode	Live: /openapi.json, /docs and /redoc are 404, back with RATCHET_OPENAPI=1; mock serves the schema, 0 removes it
test_security_headers_on_every_response	nosniff and no-referrer on a page, health, stats, a chat completion, a stream, a 400, a 404, a 413 from the body limit, Scout, and both 500s (sent outside the middleware); no CSP by default; RATCHET_CSP adds it

tests/test_observability.py

Request ids, the failures table, structured logs, Prometheus metrics, trace forwarding and version-transition events (FR-1.60, FR-12.4, NFR-6, NFR-3, INV-12, INV-13). Gateways run in mock mode or as private live gateways through Runtime(transports=) with FakeOpenAIServer and httpx.MockTransport teachers; the cascade runs on RecordingProviders. parse_metrics is a small parser of the text format 0.0.4 that fails on any malformed line or duplicate series.

Test	Covers
test_request_id_echoed_and_client_id_stored	x-request-id (req_ + 32 hex) and openai-processing-ms on a completion; id is chatcmpl- + the same digits and ratchet.request_id the id; a valid X-Request-Id or X-Client-Request-Id lands in requests.client_request_id; spaces, 129 characters, ; and non-ASCII bytes are ignored without an error; every page and 404 gets its own id, never the client’s
test_streamed_answer_carries_the_request_id	A stream’s header, every chunk’s id and the last chunk’s ratchet.request_id
test_error_responses_carry_x_request_id_and_envelope_request_id	A 400, 401, 413 and 500 carry the header and request_id beside OpenAI’s untouched four-field error; the 400 writes no failures row, the 401 an auth row (HTTPException 401 invalid_api_key, no client), the 413 a gateway row, the 500 a gateway row naming the class only; a non-/v1 404 keeps {"detail"} with the header
test_request_context_parsing	new_context: a padded client id, a valid traceparent and tracestate; the second header when the first is invalid; upper case, all-zero ids, version ff, extra fields on version 00 and a short parent id refused; a later version’s extra fields accepted; a 600-character tracestate dropped; lookup_hex; no headers outside a request
test_traceparent_forwarded_to_the_teacher_with_a_new_span	Each teacher attempt (a 503, then the retry) carries the trace id, flags and tracestate with its own new span id, never the client’s; a malformed or missing traceparent sends none
test_request_lookup_own_request_and_another_clients	GET /v1/ratchet/requests/{id} by either id form: the own request’s metadata (no text); another key’s request and an unknown id are the same 404 request_not_found; malformed ids 404; no key 401; a quota 429 is found by its failures row alone; GET /ratchet/requests/{id} reads any request
test_teacher_failure_row_carries_the_student_cost	An escalation whose teacher fails: no requests row, one teacher row with the student’s sample cost, status 502, the client, task, both ids and UpstreamError 502 teacher_down; totals counts it in failed_cost_usd, saved_usd and savings_pct
test_teacher_502_through_the_gateway_writes_one_row_with_the_student_cost	The same through a live gateway: exactly one row (the router’s), no URL or message in error, and ratchet_cost_usd_total, ratchet_failed_cost_usd_total and ratchet_failures_total{stage="teacher"} rise by it
test_quota_429_writes_a_quota_row_and_counts_a_rejection	A quota 429: a quota row with the client, task and id (APIError 429 daily_quota_exceeded, cost 0), one quota rejection, still no requests row
test_teacher_429_is_a_teacher_failure_not_a_refusal	A teacher 429 answered as a 429: one teacher row and no gateway rate row; no rejection counted; ratchet_requests_total{status="429"} once
test_a_bug_on_the_teacher_path_is_a_gateway_failure_not_the_teachers	A RuntimeError in _store_example after the student was sampled: a 500 and one gateway row (500, RuntimeError, the samples’ cost), no teacher failure counted, the failed spend in the metrics and totals; with a throttle of 0 a row that carries spend is still written; in process, on handle and teacher_only, the router writes gateway rows with no status
test_gateway_failure_row_exists_before_the_response_starts	The app called as raw ASGI: when a wrong key’s 401 starts, its auth row is already in the ledger under the response’s x-request-id
test_swallowed_failures_get_rows_and_breaker_refusals_are_sampled	30 requests against a failing student: 5 student rows for the errors (502), 3 for 25 breaker refusals (503), ratchet_failures_total counts all 30, every request answered; a timed-out audit: an audit row (504, ReadTimeout, cost 0) and the student’s answer served
test_stats_errors_last_hour_and_failed_cost	errors.last_hour by stage leaves out a two-hour-old row; totals.failed_cost_usd; log_failure refuses an unknown stage or column
test_gateway_failure_rows_are_throttled_per_stage	FailureThrottle per stage and minute; five wrong keys write two rows but count five rejections and five failures
test_json_log_lines_carry_no_content_and_no_keys	A live gateway at INFO: no marker from a message or an answer, card number, API key, teacher key or endpoint host in any line; the chat lines have every expected key, the ids, the client name and trace id; the 502’s line has served_by null; one upstream_error warning (role, class, status); a mirrored event cut to its exception class
test_log_settings_formats_and_handlers	log_settings defaults by mode, case-insensitive values, typos stop startup; config and obs agree on the values; configure never doubles a handler; the text format quotes values with spaces or quotes; JSON lines; levels filter
test_event_listeners_see_every_event_and_cannot_fail_it	A listener added twice runs once and gets the stored row; a failing listener does not stop the event
test_every_version_transition_writes_an_event	Cold start to canary to live, shadow to canary to live over the old live, a candidate replaced by a newer one, a rolled-back canary and a rejected candidate: all 14 moves (9 kinds of transition) go through transition or promote_live (set_version raises if the gate calls it) and each writes, inside the same call, an event naming the version, one of them live_retired
test_metrics_parse_count_and_bound_their_labels	/metrics parses, has # TYPE for every family and the 0.0.4 content type; a request counts under its task, an ad-hoc task as other (its name never a label), a 400 as none/400 and an invalid_request rejection; an example counted by split; cost and baseline rise; ratchet_info; histogram buckets cumulative, ending at +Inf equal to _count
test_cascade_metrics_for_upstreams_audits_and_examples	Upstream ok, timeout and circuit_open outcomes, audits by result, escalations, stored examples and upstream latency observations counted exactly; task_label
test_metrics_401_in_live_mode_without_a_key_or_token	Live: no key or a wrong token 401, an API key 403; the metrics token, a dashboard key and an admin key 200 with key names in the labels; the token reads nothing else; without reader keys the token alone opens it, and without the token too it is 403
test_metrics_token_is_checked_at_startup	A short token, or one that is also an API, admin or dashboard key, stops startup
test_public_live_metrics_scrub_client_labels	A public live dashboard: anonymous /metrics and request lookups show client-2, never a client name; a keyed reader sees globex
test_scrape_time_gauges	Version traffic for the live and canary versions, the student breaker open after a demo outage and the teacher’s closed, today’s spend
test_metrics_module_format	Label and help escaping, a counter refusing to go down or take the wrong label count, duplicate names refused, bad buckets refused, Gauge.replace dropping old series, NaN observations ignored, cumulative buckets, number() formatting, relabelled series added together, client_label
test_a_failure_never_starts_the_live_ledgers_day_one	On a fresh live (auto_day) file a 401 leaves the origin unset (day 0); day 1 starts at the first request 20 h later; a later failure is filed under day 2; after reopening the file the origin is the same and the next request is still day 1
test_old_ledger_gains_request_ids_and_the_failures_table	An old file gains request_id, client_request_id and the failures table; the lookup uses requests_request_id; the stage list; spend_since adds answered and failed spend per client since a time

tests/test_readiness.py

The startup configuration check, the config check CLI, liveness and readiness (FR-12.30, NFR-35, NFR-2, INV-1). Live mode runs offline through Runtime(transports=), FakeOpenAIServer and httpx.MockTransport; every test clears the shell’s RATCHET_* variables first.

Test	Covers
test_bad_setting_stops_startup_naming_the_variable (29 cases)	Each bad value (an unparsable or out-of-range number, a bad mode, a non-http(s) URL, an empty Token Factory key, no API keys, a short admin key, release policy rules across fields, an unwritable or missing RATCHET_DB, a bad timeout, an unknown teacher failure mode, a bad 0/1 switch, a closed or unknown teacher, judge or fallback teacher) stops a live Runtime with a ConfigError (a ValueError) naming the variable, before the ledger file exists; with the variable fixed it starts
test_every_error_is_listed_at_once	Six bad variables give one error listing all six (the unparsable one first, as load() read it); a refused teacher makes it a LicenseError too; outside load() a bad variable still raises at once with var, value and expected; load() keeps the default and collects the issue
test_readers_that_check_several_variables_report_each_one	Sixteen bad variables across the release policy, the cost and generation policies, the logs, a breaker and the reserved settings, each reader of which stops at its first problem: one ConfigError names all sixteen; ReleasePolicy.env_issues alone; config.defaults() ignores the environment
test_config_check_and_startup_agree_on_every_variable	In mock and live mode, every variable Ratchet reads set to garbage and to -1: python -m ratchet.config check exits 2 exactly when Runtime refuses to start, and a refusal is a ConfigError before the ledger file exists (a bad RATCHET_TASKS_FILE is the loader’s SpecError)
test_misspelled_variable_warns_with_a_suggestion_and_strict_mode_stops	RATCHET_AUDIT_RAT warns with “did you mean RATCHET_AUDIT_RATE?”, an unrelated name without a suggestion, the VM’s own variables not at all; the warning is kept and logged without the value; RATCHET_STRICT_CONFIG=1 makes it fatal, a bad strict value is an error, strict and clean starts
test_warnings_for_settings_that_do_nothing	A mock-only variable in live mode (not at its default), a 0/1 switch spelled otherwise, a student endpoint without a token, a live audit rate under 1%, an unpriced model, an empty key for a teacher off Nebius, an unpriced hosted fallback student: warnings, never errors, none in mock mode
test_release_policy_rules_across_fields	ReleasePolicy.validate: the default and the edges are clean; a floor above the bar, 0 audits, 0 shadow answers and a tolerance over 0.2 are errors naming their variable, all four listed together; such policies can still be built
test_config_check_cli_exit_codes	python -m ratchet.config check: 0 clean, 1 warnings, 2 errors (errors listed first, counts in the last line), 2 for warnings under strict; --json; usage errors; in its own process in live mode it reports every error and creates no ledger file
test_config_check_cli_loads_the_tasks_file	The CLI reports a bad RATCHET_TASKS_FILE (every spec error), accepts a good one, refuses clients that name no API key, and reports a missing file
test_known_env_covers_every_variable_the_deploy_kit_sets	Every RATCHET_* name in the deploy kit, the Dockerfiles, the Makefile, the scripts and the simulator is known, so the VM’s .env never warns; mock-only and loose switches are in KNOWN_ENV; DEPLOY_ENV does not overlap it
test_update_script_checks_the_config_and_waits_for_readyz	deploy/vm/update.sh runs the check before docker compose up -d and waits for /readyz after it; the Docker health check stays on /healthz
test_healthz_answers_with_every_worker_thread_busy	/healthz, /livez and /readyz are coroutines and answer while AnyIO’s thread limiter is fully borrowed
test_readyz_in_mock_mode_checks_the_ledger_only	Mock mode: ledger and config (with the warning’s message) only, the flags, Cache-Control: no-store
test_readyz_503_when_the_ledger_cannot_be_written	A read-only connection and a closed file ledger are 503 with read_only and closed; a lock held past the probe’s wait is 200 degraded (lock_timeout, busy_runs 1) and 503 when the next probe still finds it held, ready once released; a probe that outlasts its caller (timeout) counts as busy too; results are cached for 15 s on the runtime’s clock; no failures rows
test_readyz_reports_missing_adapters_and_queues_their_reload	Teacher /models 200 and a student /models without the canary’s student-v3: 200 degraded, missing_adapters=['student-v3'], the reload sent to the endpoint; each key sent to its own endpoint; no URL or key in the answer; one probe per TTL, the student’s failed result for 15 s; ready once the adapter is listed; an ok student result reused for 5 minutes
test_readyz_follows_the_teacher	An open teacher breaker is 503, unless a fallback teacher or a standing-in student keeps it 200 degraded; a 401 model list is 503; a 500, an unreadable list, a list without the teacher model and a connection error are degraded (the error named by class, no URL); an open student breaker is degraded; without a student endpoint the hosted fallback is reported
test_readyz_recovers_from_an_open_teacher_breaker_without_traffic	An open teacher breaker is 503 while it waits; past its wait, with no request at all for up to a day, 200 degraded with probe_due while /models answers, 503 while the endpoint is unreachable; half-open is degraded; a failed probe is 503 for the doubled wait, a successful one ready; a fallback teacher’s due probe counts on its own endpoint, and on the teacher’s only when the endpoint answers
test_hung_endpoints_never_make_the_ledger_look_unwritable	Teacher and student /models that hang hold both HTTP probe threads; the ledger’s own thread still answers, 200 degraded, also once its result is stale; the hung runs are shared, not started again
test_probe_cache_shares_runs_times_out_and_names_classes	ProbeCache: one run per name per TTL shared by concurrent callers, a caller’s timeout that leaves the run going (and keeps its result), a raising probe named by class only, clear()
test_ledger_error_codes_and_model_lists	ledger_error’s codes; list_models reads an OpenAI model list, sends no Authorization without a key and calls a data that is not a list unreadable

tests/test_idempotency.py

Retry de-duplication (FR-1.70, INV-3, NFR-6): Idempotency-Key answers, the SDK’s retry count and the ledger’s seen window. The gateway tests run the mock gateway with a RecordingProvider teacher (cold start: the teacher answers and the example is stored); TTLs and windows run on a FakeClock (Runtime(clock=)); threads meet at known points by polling IdemCache.stats()["waiting"] or a BlockingTeacher’s events, never by sleeping for a guess. Metrics are process-wide, so the tests compare deltas.

Test	Covers
test_keys_retry_counts_and_fingerprints_parse_strictly	check_key (1 to 255 printable ASCII, trimmed), retry_count (a plain count, else 0), fingerprint (key order ignored; the task header and the body count)
test_idem_cache_keeps_expires_evicts_and_releases	Replays, KeyReused, keys per client, a second finish ignored, the TTL on the cache’s clock, a failure releasing the key, least-recently-used eviction that never drops a running entry, the byte budget (an oversized answer not kept), the sweep, bad arguments, enabled
test_idem_cache_byte_budget_counts_each_entry_not_only_its_body	Each kept answer counts its body plus ENTRY_OVERHEAD (2 KiB), so tiny answers stop at the byte budget whatever max_entries allows; an answer whose body fits but whose entry does not is not kept
test_idem_cache_waiters_share_the_answer_or_take_over_after_a_failure	A waiter gets the owner’s answer even with TTL 0; another body refused while the first runs; after a failure one waiter takes over and the other gets its answer; a wait that runs out is InFlight
test_waiters_per_key_are_bounded	Past max_waiters (default MAX_WAITERS_PER_KEY, 4) a duplicate is InFlight at once; another key is not affected; the slots are given back
test_seen_window_counts_once_per_window_per_sender	SeenWindow: once per window per sender, a skipped repeat not restarting it, the oldest pair forgotten first
test_add_example_skips_retries_and_repeats_within_the_window	Ledger.add_example(count_seen=, sender=): no window counts every repeat; count_seen=False never counts and closes the refused INSERT’s transaction; the window per sender (‘’ included), internal traffic (None) always counting, a repeat after the window, a new example starting its window; ratchet_seen_skipped_total
test_concurrent_retries_count_once	Eight threads repeating one example for one client count once; eight clients count once each
test_old_ledger_gains_retry_count	An old file gains requests.retry_count (old rows 0)
test_same_idempotency_key_replays_without_a_cascade	One ledger row and one teacher call; the replay’s body byte-identical (same completion id), Idempotent-Replayed: true, the answer’s headers but no rate-limit headers, its own x-request-id; served after the daily quota is used up while a new key gets 429; another client’s key space; outcome counts
test_concurrent_duplicates_call_the_teacher_once	A duplicate that arrives while the teacher is blocked waits and gets the same answer (joined); one teacher call, one row
test_key_reused_with_another_body_or_malformed_is_a_400	Another body or task header: 400 idempotency_key_reused with X-Should-Retry: false; malformed keys 400 invalid_idempotency_key; bad JSON never claims its key; ratchet_rejections_total{reason="idempotency"}
test_a_body_too_deep_to_fingerprint_is_a_400_and_claims_nothing	Regression: a fingerprint that hits RecursionError is a 400 invalid_json, not a 500, with no key claimed, no teacher call and no failures row; around the real nesting limit (found by bisection without a key) every request with a key is a 400
test_in_flight_wait_is_bounded_by_the_teacher_deadline	After RATCHET_TEACHER_DEADLINE_S the duplicate gets 409 idempotency_key_in_use with Retry-After: 1; once the first finished, the key replays
test_idempotency_ttl_expiry_runs_again	On a fake clock: a replay at 599 s, a second cascade at 601 s
test_streamed_answer_is_replayed_as_the_same_events	A stream with include_usage replayed as the same events and headers; the same key without stream is reused
test_failed_first_request_releases_the_key	A 502 is never kept: the retry with the key runs and answers
test_idempotency_off_ignores_the_header	RATCHET_IDEMPOTENCY_CACHE=0: every request runs and the header is not checked
test_replay_log_line_says_it_replayed	The chat log line’s retry_count and idempotent_replay; the replay’s replay_of names the request that ran, its cost_usd is 0; the key never logged
test_sdk_retries_do_not_drain_the_golden_set	Three escalated retries with x-stainless-retry-count: 1 leave a golden example at seen 1 and in dataset.rows(ledger, "golden"), rows recording the count; a keyless client’s repeats within 300 s count once; another client’s counts
test_the_same_request_minutes_later_still_counts	Repeats ten minutes apart (fake clock) count, and the third moves the example to training (FREQUENT)
test_openai_sdk_retry_after_a_server_error_is_not_a_repeat	The real OpenAI SDK’s retry of a request that failed after its example was stored sends x-stainless-retry-count: 1, and seen stays 1
test_internal_traffic_and_mock_mode_count_every_repeat	Mock mode leaves the window off by default; calls outside a gateway request (no meta, or a meta whose request is not in flight) count every repeat; an explicit retry count is still honoured
test_retry_settings_are_checked_at_startup	Mode defaults (300 s live, 0 mock) and each out-of-range value as one error naming its variable
test_bad_retry_setting_stops_the_gateway	Unparsable and negative values stop a Runtime with one ConfigError naming both

tests/test_offpath.py

Off-path quality checks (FR-12.31, NFR-1, NFR-3, INV-15): the sidecar, Ledger.update_request and the pending rows the gate ignores, and the router’s audit and shadow after the response. Threads meet at known points through events (a GatedProvider teacher that answers only once released and says when a call started; a gated FakeOpenAIServer for the gateway), and every wait is bounded (WAIT_S), never a guess; SideCar.drain waits for the jobs. Metrics are process-wide, so the tests compare deltas.

Test	Covers
test_sidecar_without_workers_runs_nothing	SideCar(0) takes no places (and does not call itself full), drains and closes at once
test_sidecar_is_bounded_never_blocks_and_keeps_places_for_audits	Refusals return at once; shadow work stops before the audits’ places, an audit takes the last one; a cancelled place is free again; a slot ends once
test_sidecar_audit_places_scale_with_the_queue	audit_places for queues of 1, 2, 10 and 1000 places and reserve=0; bad arguments
test_sidecar_bounds_the_request_text_its_jobs_hold	max_bytes with its share kept for audits: a reservation past it is refused (and counted as full), a job larger than the whole budget is never taken, a running job holds its bytes, and a cancelled, finished, abandoned or refused job gives them back; job_bytes counts wide characters’ storage
test_sidecar_jobs_run_in_the_submitting_context_and_failures_are_counted	A job sees the submitter’s context variables; a job that raises is counted and logged by its class, never its message
test_sidecar_close_drains_then_gives_up_and_refuses_late_jobs	close waits, gives up on a queued job (returned by tag), refuses a job handed over after it, lets a running job finish; a second close gives up on nothing more
test_update_request_writes_each_verdict_once_and_adds_costs_call_by_call	A verdict written only while pending, stored only raised, costs summed in order to the same float as Python’s, a missing row, refused columns and values that write nothing
test_update_request_does_not_invalidate_the_hot_path_caches	Finishing a row leaves the requests generation alone; any other rewrite moves it
test_live_quality_ignores_pending_audits_until_their_verdict	Pending audits count in neither number; a verdict counts once written, a failed audit is forgotten; the day’s audit agreement; a fresh tally reads the same
test_shadow_quality_ignores_pending_shadow_rows	A pending shadow row counts once its verdict is written
test_an_audited_answer_returns_before_the_teachers_audit_ends	With the teacher blocked, the answer returns with overhead_pending, the serving cost and no audit in the breakdown, the row pending; once released, the row has the verdict, the audit’s cost and baseline, stored, and the gate counts it
test_offpath_checks_leave_the_rows_an_inline_run_writes	60 scripted requests inline and off the path (audits that agree, disagree, fail or measure nothing; shadows that agree, disagree and are arbitrated, fail to sample or meet a failed audit; escalations): every request row (costs to the last bit), example, failure and event, the failure counters, version_live_quality and shadow_quality are equal, and so is what ratchet_cost_usd_total and ratchet_baseline_usd_total count (once per cent: the ledger’s totals)
test_a_drawn_audit_runs_inline_when_the_queue_is_full	With every place held, the drawn audit runs before the response, counted per version, with a sidecar_inline event
test_a_long_request_past_the_byte_budget_audits_on_the_request_path	A request whose text would pass the byte budget audits before the response; a short one still goes off the path; both rows end audited
test_a_shadow_queued_after_an_inline_audit_keeps_the_audit_as_its_reference	The audit runs inline (queue full), a place frees up and the shadow goes off the path: judged against the counted audit’s answer without an arbitration, and after a failed audit not arbitrated (INV-15); one teacher call either way
test_a_shadow_without_a_place_is_skipped_and_counted_but_rides_with_an_audit	A shadow without a place leaves no shadow record, counted per version with a sidecar_dropped event; with an audit drawn it rides in the audit’s reserved place and gets its verdict
test_offpath_failures_follow_the_optional_call_rules	A failed off-path audit: audited 0, no arbitration, the disagreement unarbitrated, audit_errors, an audit_error event and a failures row naming the request; a failed shadow sample: no shadow record, through the shadow’s breaker
test_an_offpath_arbitration_gives_up_sooner_than_a_served_answer	The off-path arbitration passes deadline_s 60; the inline one passes none
test_a_provider_deadline_shortens_its_retries_never_lengthens_them	OpenAICompatProvider.complete(deadline_s=) on a fake clock: four attempts by default, two within 3 s, four with a longer deadline
test_requests_without_meta_keep_their_checks_on_the_request_path	No meta (Scout, the simulator) or offpath=False: the checks run inline and cost_usd is the row’s
test_a_ledger_error_gives_the_place_back	A failing log_request cancels the reserved place
test_a_sidecar_that_stops_meanwhile_hands_the_job_back_to_the_request	A job refused after shutdown began runs before handle returns
test_close_gives_up_on_queued_jobs_and_counts_them_per_version	Jobs given up at shutdown: counted per version (audits and shadows), one sidecar_dropped event, rows left pending; the running job finishes; afterwards checks run inline and a closed sidecar is not counted as full
test_concurrent_requests_all_finish_their_checks	Eight threads, 120 requests, four workers: every job done, no row pending, costs split consistently, the gate’s count equal to the audited rows
test_sidecar_settings_defaults_and_bounds	Mode defaults (4 live, 0 mock), every out-of-range value an error naming its variable (also through Settings.validate), an unparsable one collected by load
test_mock_mode_and_the_simulator_keep_every_check_on_the_request_path	A mock-mode Ratchet has no workers; a live one has four, starts no thread before its first job, and keeps calls without meta inline
test_the_gateway_answers_first_and_its_shutdown_drains_the_queue	A live gateway with a blocked teacher answers with ratchet.overhead_pending; /ratchet/stats sidecar and the /metrics gauge; leaving the TestClient runs the lifespan, whose drain completes the audit; the chat log line has overhead_pending and its cost plus the offpath line’s is the row’s
test_a_shutdown_past_the_drain_leaves_rows_pending_and_says_so	RATCHET_SIDECAR_DRAIN_S=0: the queued job is given up with a sidecar_dropped event, its row stays pending and uncounted; the running one finishes
test_mock_gateway_answers_have_no_pending_overhead	Mock mode: no overhead_pending, an idle sidecar block
test_a_demo_reset_stops_the_old_sidecar	Runtime.reset closes the old cascade’s sidecar and builds a new one
test_offpath_job_logs_one_line_without_content	One offpath log line naming the row, its verdict and the cost it added, never the request’s text; ratchet_sidecar_jobs_total{outcome="done"}

tests/test_limits.py

Spend and rate limits (FR-12.2, FR-12.3, NFR-3, NFR-5, INV-11, INV-13). Gateways run as live gateways through Runtime(transports=) with a FakeOpenAIServer teacher whose answers cost a known amount (10 prompt and 5 completion tokens at the teacher’s price; cold start, so the teacher answers every request), or in mock mode; the spend meter runs on its own with an injected wall clock (FakeClock), the rate limits on the runtime’s monotonic FakeClock. Concurrency tests block a teacher on a threading.Event and send the second request from the main thread while the first waits in a background thread, both through one TestClient; the stalled-body test calls the ASGI app directly under anyio.run, with a receive that waits on an event between body chunks. Live Settings pass every limit explicitly (_live), because live mode’s per-IP default would refuse a burst from one test address.

Test	Covers
test_key_daily_usd_cap_refuses_once_spent_and_other_keys_go_on	A cap of 2.5 requests’ worth: three 200s with X-Ratchet-Spend-Today-USD and X-Ratchet-Budget-Remaining-USD counting, the third overshooting by its own cost; the fourth a 429 insufficient_quota / budget_exceeded with remaining 0, Retry-After, X-Ratchet-Budget-Reset, X-Should-Retry: false, a budget failures row and a budget rejection; another key without a cap gets a 200 and no spend headers; no requests row for the refusal
test_monthly_cap_resets_on_the_first_and_names_the_later_reset	On Oct 30 a spent daily and monthly cap: Retry-After runs to Nov 1 (the later reset); still refused on the 31st, a new day; answered on the 1st
test_spend_survives_a_restart_on_the_same_ledger	A second Runtime on the same file reads today’s requests, dollars and failed spend (a failures row) per key and for the gateway, and refuses the spent dollar cap and the used request quota
test_failed_requests_spend_counts_against_the_key	An unsure student, then a teacher 502: the samples’ cost counts against the key’s dollars (no request counted), at once and after a restart
test_utc_rollover_resets_the_counters_and_a_clock_moved_back_reads_the_ledger	23:59 on Oct 31, then 00:01 on Nov 1: day, month and gateway counters start from zero; a new day within the month keeps the month; a clock moved back to Oct 31 reads that day’s gateway rows from the ledger, leaving out a row without a gateway request id; month_bounds
test_meter_ignores_bad_costs_and_counts_spend_without_a_request	Negative, NaN and infinite costs count as 0 (the requests still count); add(requests=0) adds dollars only
test_budget_alerts_fire_once_per_share_also_after_a_restart	50, 80 and 100% of a key’s $5.00 daily cap and 50% of the gateway’s $10.00, each once and in order (two shares at once when one request crosses both); a restarted meter on the same ledger repeats none; one started with a lower gateway budget writes the three shares it already passed; the next UTC day starts over; money
test_alerts_name_a_tiny_cap_exactly_and_read_it_back	money writes caps of $0.0000001 and $0.0000375 with every digit (never $0. or an exponent); a restarted meter reads those alerts back and writes none again
test_an_alert_that_cannot_be_written_is_tried_again	A failing event write is retried on the next add, never raised
test_quota_path_issues_no_count_query	With a sqlite trace callback: SpendMeter.usage issues no statement, and a chat request under a daily quota and dollar budget issues no COUNT( of any kind
test_student_only_serves_a_confident_student_and_refuses_what_needs_the_teacher	The first request’s audit spends the gateway’s budget; then a confident student answer is a 200 with no audit and no shadow (route student, audited 0, no shadow version, one student call); an escalation is a 429 gateway_budget_exceeded that names no amount, with one budget failures row carrying the samples’ cost; a pinned task and a request with tools get the 429 without any model call; a budget event at 100%
test_block_mode_refuses_every_chat_request_until_midnight	Past the gateway’s budget every key’s chat request is a 429 gateway_budget_exceeded; /v1/models still answers; /ratchet/stats budget block
test_a_scrubbed_public_read_names_no_key_and_no_cap	A public live dashboard read anonymously after a key’s cap and the gateway’s budget were reached: every budget event reads client-1 reached N% of its daily budget or the gateway reached ... with no amount, no key name or $ anywhere, daily_cap_usd, budget.daily_usd and remaining_usd null; a dashboard key reads the events and caps as written; public_alert on a detail that is not an alert and on a name that is no longer a key
test_scout_spend_stays_under_its_own_guard	A Scout run spends money that neither the meter nor a fresh one counts; /v1 spends the budget, the next chat request is refused, Scout still runs
test_key_limits_override_the_defaults_and_bad_specs_stop_startup	A per-key daily_requests override against the default; parse_key_limits with spaces and every field; 22 malformed specs, each naming RATCHET_KEY_LIMITS and never echoing a pasted key (acme:<key> in RATCHET_API_KEYS’ syntax, a key as a field, a value or a task list), and a 400-digit dollar cap that would read as infinite; a Runtime naming an unknown key does not start
test_task_scope_is_403_and_lists_only_the_scoped_tasks	A pinned task and default outside the scope are 403 task_not_allowed; an unknown-task 400 lists only the scope’s tasks; the refusals used no quota; another key sends the pinned task
test_store_zero_keeps_a_keys_requests_out_of_the_training_set	store=0: the key’s teacher answer is served with stored 0; another key’s is stored
test_rpm_bucket_refuses_past_the_burst_and_refills_on_the_clock	60 a minute: ten 200s with the remaining count falling to 0; the 11th a 429 rate_limit_exceeded with Retry-After: 1, reset 10s, no X-Should-Retry, a rate row and rejection; another key unaffected and without rate headers; one more a second later; the burst back a minute later
test_daily_quota_headers_win_over_the_rate_headers	A key with both: the 200’s X-RateLimit-*-Requests describe the daily quota
test_concurrency_one_refuses_a_second_request_while_the_first_runs	concurrency=1: while the first request waits on the teacher, the key’s second is a 429 with Retry-After: 1; another key is answered; after the first ends the slot is free again
test_max_inflight_answers_503_instead_of_queueing	RATCHET_MAX_INFLIGHT=1: a second request is a 503 overloaded with Retry-After: 1, an overloaded rejection and a rate row; the in-flight count returns to 0
test_a_stalled_body_holds_no_in_flight_slot	Straight into the ASGI app with RATCHET_MAX_INFLIGHT=1 and concurrency=1 for one key: two requests send headers and 5 bytes of chunked body, then stall; the in-flight counts stay 0 and a whole request from each key is a 200; the stalled bodies, released one at a time, are 200s too (the code before the fix refused the second stalled request at once)
test_per_ip_rate_on_a_gateway_without_keys	Live default 120, mock 0; an open gateway with 6 a minute: the second request from one address (the first X-Forwarded-For hop) is a 429 with Retry-After: 10 that does not echo the address; another address is answered; ten seconds later the first is again; with API keys the per-IP rate is off
test_token_bucket_concurrency_and_limiter_units	TokenBucket (burst, refill, a clock moved back), burst, RateLimiter’s LRU bound and a new bucket for a new rate, Concurrency, Limiter.rate (takes a token, holds no slot; refused when empty) and Limiter.hold (the in-flight cap, then the key’s; a refusal gives back the slot it took)
test_limit_settings_are_checked_at_startup	limit_policy reads the mode in any case and the shares sorted, once; thirteen bad values (a dollar default past a billion and a NaN gateway budget among them) each name their variable
test_stats_report_each_keys_spend_and_cap_and_the_security_switches	clients[].spent_today_usd and daily_cap_usd, the three security keys and the budget block; key names only, never keys
test_router_refuses_teacher_work_under_student_only_without_a_gateway	RequestMeta(student_only=True) on Ratchet itself: a pinned task raises BudgetExceeded (pinned) before any call and writes nothing; an unsure student raises it (escalation) after its samples with a budget row (429, the samples’ cost)

tests/test_admin.py

The admin API and CLI and compare-and-set release moves (FR-12.1, FR-12.9, INV-5, INV-8, INV-10, INV-13). HTTP tests run live gateways through Runtime(transports=) with FakeOpenAIServer teacher and student; the races run two Ledger instances on one file, as the gateway’s tick and the cron tick do, and make the other process act between one gate’s reads and its write (_first_call_runs), then repeat the race with real threads.

Test	Covers
test_transition_moves_a_version_only_from_where_the_caller_saw_it	A stale share or state moves nothing and writes no event; the right one moves, writes its event and starts a new stage; the same decision twice is refused; a move into shadow or canary while another version is in flight is refused; a NULL share reads as 0; a move to live, an unknown state and a share past 100 raise
test_promote_live_retires_the_old_live_version_in_the_same_transaction	A caller that saw no live version promotes nothing; the promotion retires the old live version, live_retired before live; the same promotion again writes nothing; promoting a live version raises
test_a_transaction_commits_its_moves_and_events_together_or_not_at_all	An exception inside rolls back the move and its event, the listener hears nothing and the routing snapshot built inside is dropped; nested transactions join; listeners hear events after the commit; a writer that commits by itself inside raises
test_two_ticks_on_one_ledger_file_advance_a_canary_once	Gate B has read a ready canary when gate A advances it: one canary 50% event, 50% on both connections; six threaded races, one advance each
test_a_promotion_race_leaves_exactly_one_live_version	Two gates promoting one canary: one live version, one live and one live_retired event; an admin (CLI, another connection) promoting while the gate is about to: one live version, no gate events, one admin_actions row
test_admit_leaves_a_version_an_admin_moved_alone	A candidate an admin promoted to canary while it was scored stays a canary, with a conflict event
test_every_admin_route_needs_an_admin_key	All 16 /ratchet/admin/* routes have the admin level; no key 401, an API key and a dashboard key 403, nothing changed; live mode without admin keys 403 admin API disabled on each; an admin key does not open the mock-only demo controls (INV-8); no key in the ledger (INV-13)
test_canary_rollback_over_http_writes_the_move_an_audit_row_and_an_event	The answer’s before and after; the canary rolled_back, live untouched; the admin_actions row (actor api:ops, reason, before, after, the response’s x-request-id); the admin event; a second rollback 409 and nothing written; the audit log and its limit bounds
test_promote_to_live_over_http_retires_the_old_live_version_atomically	The next canary stage by default, an unchanged share answered changed: false with no row, then live with both versions in before and after and exactly one live; the event; GET /ratchet/admin/versions with golden numbers and stage audits
test_refused_promotions (13 cases)	A rejected version to live or canary without force, a rolled-back version or a candidate without force, live to canary, shadow to live (409); a share of 100 or a string, a share for live, an unknown to, a non-boolean force or load_adapter, an unknown key (400); nothing changed or recorded
test_the_transition_table	Forced returns of a rejected, rolled-back, retired and candidate version to canary, refused while another is in flight; no eval, never; shadow to canary by default 10%, then 50%, then the last stage refused; rollback of live refused with a hint; retiring live; a retired version cannot be retired; 404 and 400 for a bad id
test_a_reason_is_required_checked_and_redacted	Missing, short, blank, long, two-line and non-string reasons and malformed actors (one that looks like a key) refused with nothing changed; an e-mail address in a reason stored as [EMAIL], nowhere in the ledger
test_load_adapter_before_a_live_promotion	A failing /v1/load_lora_adapter is a 502 naming the class and status, nothing changed or recorded; a working one is called with the adapter’s name, path and the endpoint key before the move; 409 without a student endpoint; ignored in mock mode
test_student_disabled_sends_every_request_to_the_teacher_and_stores_as_usual	With a live version and a shadow: route student_disabled, no student or shadow resolution, the teacher’s answer stored, no version on the row; switched back on, the student serves
test_student_kill_switch_over_http	disable answers before and after, the next request is the teacher’s with no student call, a second disable writes nothing, controls and the by-task routes in /ratchet/stats, enable, GET /ratchet/admin/flags with who set it
test_a_runtime_pin_keeps_a_task_on_the_teacher_and_out_of_training	A pinned task (by name and as a Task object) takes route pinned with no student call and nothing stored (INV-10); its stored rows leave the training set (excluded_pinned); its spec_hash is unchanged; a spec-pinned task’s pin is a no-op and its unpin 409; unknown and invalid names; unpinned, the registered task is back and so are its rows
test_task_pins_over_http_show_in_stats_and_the_task_list	POST .../pin sends the next request to the teacher; controls.pinned_tasks; runtime_pinned true with the spec’s pinned false; 404 for an unregistered name; DELETE restores the student
test_put_audit_rate_changes_behaviour_without_a_restart	Rate 0, then 1 (every answer audited), then null (the configured rate again); four bad values 400; audit_rate required; 0 during a canary 409, accepted with force; with audits off a new canary needs force
test_training_paused_skips_the_night_before_anything_else	A paused night trains and evaluates nothing (reason training paused, a skip event) though there is plenty of data; resumed, it trains
test_frozen_releases_hold_every_promotion_but_never_a_rollback_or_an_alert	Frozen: a cold start shadows the teacher, a ready shadow is held, then promoted on its evidence once unfrozen; a good canary does not advance; a bad canary is rolled back and the live alert fires while frozen; an audit rate of 0 holds a ready shadow too
test_a_frozen_gate_in_a_corrupt_deploy_simulation	sim.simulate.run on a ledger file, freezing from another connection when v1 becomes a canary (simulation): deployed corrupted, v1 is still rolled back and nothing is promoted after the freeze; deployed sound, it stays at 10% instead of advancing, and the nights skip
test_the_tick_endpoint_runs_the_gate_and_records_who_ran_it	A forced tick rolls a bad canary back and answers before, after and moved; the rollback then the admin event; a tick without a body; both recorded, the second without a reason
test_cli_changes_reach_a_running_ledger_through_data_version	On one file, a running ledger with cached switches and routing sees the CLI’s flag, rollback and pin at once; audit-log --json with actor cli:<user> and no request id; versions output; exit 1 for a refused move, 2 for bad values (2, nan, a short reason); default; a missing ledger is refused and not created
test_bad_flag_rows_written_by_hand_never_fail_a_request	Unparseable and unknown runtime_flags rows read as the defaults; set_flag refuses an unknown name and NaN; admin_actions refuses a boolean limit
test_requests_from_many_threads_while_an_operator_flips_switches	Four threads, 100 requests, while the kill switch and a pin flip ten times: no error, every route student, student_disabled or pinned, 100 rows
test_admin_tables_exist_on_an_old_ledger_file	An old file gains admin_actions, runtime_flags and task_overrides on open

tests/test_calibration.py

Per-task calibration when several tasks share one adapter (FR-14.1). Scripted golden rows set each answer’s confidence and correctness exactly; the simulation numbers come from mock models.

Test	Covers
test_cascade_eval_per_task_thresholds	FR-14.1, FR-5.2, INV-6: an easy and a noisy task on one adapter. Pooled, the global threshold (0.85) serves the noisy task’s wrong answers (97% overall); the records give the easy task 0.85 and the noisy one 0.95 (precision 1.0, coverage 0.85), with their row counts, agreement, spec hashes and calibrated. When nothing reaches its target the noisy task gets 1.01; a policy.target_precision of 0.8 gives it 0.85 and a new spec hash
test_single_task_eval_identical_to_global (3 cases)	FR-14.1, FR-5.2, FR-5.3: with one task, its record holds exactly the global threshold, precision, coverage, row counts and agreement, for min_task_n 1, 20 and 1000 (calibrated true, true, false)
test_task_with_few_calibration_rows_is_never_below_global	FR-14.1, FR-5.2: with fewer than min_task_n calibration rows a task takes the higher of its own threshold and the global one (calibrated: false), the global one with no calibration row, 1.01 with no held-out row; enough rows make it its own
test_task_without_test_rows_is_teacher_only	FR-14.1, INV-6: a task whose one golden row calibrates (none held out) gets 1.01, and so does a task whose references are all invalid; the router escalates it without a student call (route task_disabled) while serving the other task
test_router_uses_task_threshold	FR-14.1, FR-2.3: at confidence 0.9 task A (0.7) is served, B (0.95) escalates and C (no record) uses the global 0.8; without a task the global threshold, for the fallback model RATCHET_CONFIDENCE_THRESHOLD; each task of a version has its own fingerprint
test_disabled_task_skips_the_student	FR-14.1, FR-2.3, FR-2.12: a task the gate disabled on the serving version goes to the teacher without a student call (route task_disabled, no student cost, confidence null); with the teacher down and RATCHET_ON_TEACHER_FAILURE=serve_student another task degrades but the disabled one gets the teacher’s error; the shadow still runs; the hosted untuned fallback model ignores the records; a failing resolver leaves the task held back
test_stale_spec_hash_threshold_ignored	FR-14.1, FR-2.3, FR-2.15: a record under an old spec hash gives 1.01 (neither its own nor the global threshold), and stale_tasks names it; a current record wins over a stale spec_hashes entry
test_versions_without_task_eval_unchanged	FR-14.1, FR-2.3: a version without records keeps its one threshold for every task; spec_hashes alone still sends a changed task to the teacher; the gate checks only the global numbers
test_threshold_reads_malformed_records_safely	FR-14.1, FR-2.3, INV-7: records that are not mappings are ignored; a non-numeric or NaN threshold, a missing hash or a disabled record give 1.01; a global threshold that is not a number serves nothing instead of failing the request
test_admit_disables_regressed_task_but_ships_version	FR-14.1, FR-6.1: 48 of 50 right (96%) is above the 95% floor but more than 2 points behind live’s 100%: the task is disabled (threshold 1.01, the reason, a task_disabled event) and the version shadows; below the floor (70%) it is disabled with nothing live
test_per_task_gate_holds_each_task_to_its_floor	FR-14.1, FR-6.1: the global gate’s point-estimate rule on each task’s own rows: 36 of 40, 9 of 10, 18 of 20 and 91 of 100 right all fail a 95% floor next to a task at 100% (a Wilson upper bound passed them all), and a cold start ships the good task alone; exactly 95% passes; a task that answers none of its held-out rows fails as unmeasured
test_admit_reject_mode	FR-14.1, FR-6.1: with task_mode="reject" the same candidate is rejected, naming the task, and nothing is disabled
test_per_task_gate_needs_evidence	FR-14.1, FR-6.1: no failure under min_task_test_n rows, for a dip within the tolerance (49 of 50 against live’s 100%), for a task already at 1.01, or against a live record that serves nothing, has another spec or too few rows; 97% passes the 95% floor but not a policy floor of 99%
test_admit_rejects_a_version_that_would_serve_too_little	FR-14.1, FR-6.1, INV-5: disabling a task that leaves 12.5% of the held-out rows answered, or per-task thresholds that answer none, reject the candidate instead of leaving it in shadow or canary
test_disabled_task_survives_rescore	FR-14.1, INV-4: the champion re-check overwrites the eval but keeps the disabled tasks disabled, also one the re-score has no record for
test_per_task_live_alert_once	FR-14.1, FR-6.7, NFR-8: 12 of 20 audits on one task of the live version raise one alert, not repeated by later ticks or a new gate; has_event compares a prefix (a _ in a task name is not a wildcard); the grouped query runs again only after a new audit; the canary gets its own alert
test_per_task_alert_is_written_once_across_processes	FR-14.1, FR-6.7: two Ledger connections on one file; the second checks for the alert while the first is writing it and waits on the write lock, then finds it: one alert, never two (Ledger.event_once: BEGIN IMMEDIATE, a check and the insert, COMMIT); another task or kind is another event; a detail outside its prefix is refused
test_per_task_alert_leaves_single_task_versions_to_the_version_alert	FR-14.1, FR-6.7: a version whose audits are all one task’s gets only the version-level alert
test_stats_by_task_has_audit_agreement	FR-14.1, FR-7.7: audits and audit_agreement per task; threshold, precision, coverage, calibrated and disabled from the live version’s records (the global threshold for a task without one, 1.01 for a disabled task or a changed spec, null for a pinned task or without a live version)
test_bom_eval_carries_task_records	FR-14.1, INV-14: a night’s BOM carries the eval’s per-task records, and spec_hashes matches them
test_task_calibration_settings	FR-14.1: RATCHET_MIN_TASK_GOLDEN (a whole number, 1 or more, reaching LiveEvaluator), RATCHET_TASK_MODE (case-insensitive, empty is the default, a bad value names the variable) and RATCHET_MIN_TASK_TEST_N
test_per_task_calibration_can_be_switched_off	FR-14.1: RATCHET_PER_TASK_CALIBRATION=0 makes cascade_eval write no records and leaves the pooled numbers as they are; the simulator, the mock gateway and python -m ratchet.live night pass it on; only 0 switches it off
test_ledger_merges_a_task_record_atomically	FR-14.1: merge_task_eval merges into one record, keeps the others, and rolls back on an unknown version
test_mixed_workload_calibrates_each_task	FR-14.1, FR-9.5, US-4: a 6-day mixed simulation (mock models) with RATCHET_PER_TASK_CALIBRATION=1: every trained version has a record for ticket-triage, source-triage and page-read under their current specs, and the student still takes over (over 80% on the last day) at audited agreement of 90% or more

tests/test_mock_world.py

The generic mock world (FR-9.7): every request the ticket and research worlds do not know gets a deterministic answer from mock models. Schemas are checked with a small validator in the test file (the subset the mock fills), since jsonschema is not a dependency; when it is installed, one test checks the same instances with it too. The simulation numbers are trends from mock models.

Test	Covers
test_lookup_tries_tickets_then_research_then_the_generic_world	FR-9.7: a demo ticket keeps its world whatever it is sent with (a schema too), a research request as well; hello, a system-only request, an image-only request and an empty list get a generic item that answers, and each raises KeyError under strict (RATCHET_MOCK_STRICT); a schema or tools pick the schema or tools sub-world
test_generic_items_are_deterministic_and_keyed_by_what_a_model_sees	FR-9.7: the same request gives the same template, labels and answer whatever task it is sent under; another system prompt gives another template; 60 texts spread over all four framings
test_free_text_world_restates_the_prompt	FR-9.7: [mock teacher] You asked: "hello". ...; a wrong framing differs; a long prompt quotes 12 words and ...; a 1,000,000-character prompt in under 0.5 s; text parts joined; json_object gives {"answer": ...}
test_instance_from_schema_fills_the_structured_outputs_subset	FR-9.7: 200 seeds of a schema with an enum, a boolean, bounded and exclusive (draft 4 and draft 6) integers and numbers, multipleOf, formats (date, date-time, UUID), min and max lengths, a nullable type, anyOf with null, const, arrays with prefixItems, minItems and uniqueItems, allOf, local $ref: every instance validates, the same seed gives the same instance, and enums, booleans and integers are drawn
test_instance_from_schema_ends_and_never_raises_on_hostile_schemas	FR-9.7: a recursive $ref ends within MAX_DEPTH, a $ref loop and 50 × 50 × 50 nested arrays end, bounds of ±1.7e308 and 10^400 give finite JSON, all in under 1 s; 1,500 seeded junk schemas, also as a json_schema and as tool parameters (with tool_choice required), never raise, and every label space keeps a wrong value to draw
test_instance_from_schema_agrees_with_jsonschema	FR-9.7: with jsonschema installed (skipped otherwise), the subset and the workload schemas validate under Draft 2020-12 for 100 seeds each
test_schema_answers_stay_valid_for_any_labels_and_wrong_ones_change_enums_first	FR-9.7: 300 schema-workload items: the teacher’s answer and a guessing student’s both validate; labels list enums first; every schema task appears; a corrupted deploy’s three samples agree on enum values that are all wrong; the mock teacher writes the item’s answer
test_mock_teacher_calls_the_catalogue_with_arguments_read_from_the_prompt	FR-9.7: lookup_order with order_id 48213, no content, finish tool_calls, a stable call_ id; refund with the amount and a reason for each wording (damaged, duplicate_charge, not_received, wrong_item, changed_mind, a $ amount too); a plain reply without an order number; after the tool’s result a final text answer restating it (inside {"answer": ...} for json_object)
test_tool_choice_and_client_tools	FR-9.7: a forced tool_choice wins, none gives free text, required calls a tool even without an order number; a client’s own tool gets arguments that validate against its schema; an integer order_id gets an integer
test_mock_student_tool_calls_improve_with_training	FR-9.7, FR-9.2: on 300 tools-workload requests a student trained on 200 examples of the template makes the teacher’s call (name and arguments) over 75% of the time, an untrained one under 60%; a tool-call sample lists no content tokens
test_mock_fleet_trains_and_evaluates_on_generic_and_ticket_rows	FR-9.7, FR-9.3: a ticket row, a free-text row, a schema row and a tool row with their stored conditions, a system-only row and a content-parts row train without raising (also with strict), every template is counted and the golden eval scores all six; a strict fleet’s live student raises KeyError for hello and still answers a ticket; a lenient shadow student answers hello on staging
test_strict_teacher_and_unreadable_scout_writer_requests	FR-9.7: a Scout writer request without readable facts gets a generic answer, and KeyError from the strict teacher, as hello does
test_mock_strict_setting	FR-9.7: RATCHET_MOCK_STRICT 1 and 0
test_schema_workload_student_takes_over	FR-9.7, FR-9.4: --workload schema, 4 days × 300 (seed 7, 5 audits and 40 shadow answers to move a release): no student on day 1, over 50% on day 4 and above day 2, audited agreement of 90% or more on the last day; every stored example is one of the three schema tasks and keeps its json_schema condition
test_tools_workload_is_answered_by_the_teacher_alone	FR-9.7, FR-1.41: --workload tools, 3 days × 300: every request goes to the teacher alone (route tools, uses_tools 1, nothing stored), the student share stays 0 and stats_by_task counts 900 tool_requests for order-support
test_demo_day_runs_the_schema_and_tools_workloads	FR-9.7, INV-8: gateway.WORKLOADS is sim.world.WORKLOADS; /ratchet/demo/day?workload=schema and tools answer 200 with simulation: true, their tasks appear in /ratchet/stats, and an unknown workload is a 400
test_sdk_gets_answers_and_structured_outputs_from_the_mock_gateway	FR-9.7, FR-1.1, US-1: through the OpenAI SDK, hello gets a 200 restating it, and client.chat.completions.parse(response_format=Claim) (a Pydantic model with a Literal, a bounded integer, a date, a nested list and an optional field) returns a parsed Claim whose raw content validates against the schema the SDK sent
test_a_mock_tool_call_passes_through	FR-9.7, FR-1.41: with the default allowlist the mock teacher gets the tools and its call comes back as message.tool_calls (null content, finish_reason: "tool_calls"), the same call the mock teacher makes directly; with tools kept from the teacher (RATCHET_TEACHER_PARAMS=max_tokens,stop) it answers in prose; teacher-only and nothing stored either way

tests/test_structured.py

Structured outputs (FR-1.40, FR-14.2): the JSON Schema check (ratchet/jsonschema_lite.py), fields derived from a schema and auto tasks (ratchet/schema_tasks.py), the contract a response_format binds to a request’s task, one repair of a teacher answer that breaks it, the json_object fallback, and what clients see. Models are RecordingProviders or FakeOpenAIServers behind httpx.MockTransport.

Test	Covers
test_validator_checks_required_enum_and_additional_properties	FR-1.40: required, enum and additionalProperties: false, as schema paths that never quote a value or a key from the answer (a secret placed in both is absent); max_errors; an additionalProperties schema checked under .*; a key that is not an identifier written $["a-b"]
test_validator_types_and_nullable	FR-1.40: type lists with null, JSON’s integer (1.0 is one, true is not), NaN and infinity refused, an unknown type name ignored, anyOf with null
test_validator_resolves_refs_and_guards_against_loops	FR-1.40: a recursive $defs tree, a $ref loop that reaches no value, an unresolved or remote $ref (an error, never valid), ~1 escapes, #
test_validator_combinators_const_items_and_bounds	FR-1.40: oneOf (an integer matches both integer and number), allOf, const and enum with JSON equality, items, minItems/maxItems, maxLength, exclusive and inclusive bounds on a 10^400 integer; pattern, format and multipleOf ignored
test_validator_tuples_and_pattern_properties	FR-1.40: prefixItems with and without items (also items: false), the old list form ignored; additionalProperties: false not enforced next to patternProperties
test_validator_never_raises_and_is_bounded	FR-1.40: MAX_DEPTH on a recursive schema; an exploding anyOf over 200 items stops at MAX_STEPS in under 2 s as a mismatch; junk schemas never raise
test_json_equal_is_json_equality	FR-1.40: 1 == 1.0, true != 1, nested values
test_derive_takes_enums_and_ignores_free_text	FR-14.2, FR-2.14: the spec’s example gives category and priority labels and ignores reply; required follows the schema; no object root, or nothing comparable, gives no fields
test_derive_resolves_refs_nullable_nested_records_and_formats	FR-14.2, FR-2.14: a Pydantic-shaped schema: bool, nullable date, date-time, exact text for email, number with its relative tolerance, records through a $ref (free text left out), sets of labels, booleans and dates, a nullable parent’s children not required, an integer enum with null; consts, integer lists, bad names and two-way anyOf left out; the task accepts what the schema accepts; key order changes nothing
test_an_enum_of_numbers_takes_what_the_schema_takes	FR-14.2, FR-1.40: an enum of numbers (and an array of one) is an exact number, so 2.0 meets the field as it meets the schema; the schema still decides the allowed values; a mixed enum stays a label
test_derive_stops_at_four_levels_and_sixty_four_fields	FR-14.2: four levels deep, 64 fields wide, a records field counted with its sub-fields
test_x_ratchet_annotations_include_exclude_and_are_stripped	FR-14.2: ignore, text, ordered (a ranking, ordered records), abs_tol; unknown hints ignored; strip_annotations and forwardable remove every x-ratchet key at any depth but keep a property named that way, and copy nothing when there is none
test_derived_tasks_round_trip_through_a_spec	FR-14.2: an auto task (source auto, never pinned) reads back from taskspec.document unchanged, for the test schemas, every mock-world schema and an annotated one; a pinned auto task is refused
test_auto_task_name_is_stable_across_key_order	FR-14.2: auto-<slug>-<16 hex> (at most 62 characters), the same for a reordered schema, another for another schema, 40 slug characters at most, schema without a name
test_contracts_and_bound_tasks	FR-1.40: contract_for (cached per schema), Task.bound (same spec_hash, equal, not registrable), check reasons, clean_answer serving the JSON inside reasoning, a sentence and a fence, an array root
test_for_row_grades_a_stored_row_under_its_stored_format	FR-1.40, INV-6: tasks.for_row from a dict, JSON text, none and bad JSON, and on a ledger row
test_auto_task_is_registered_persisted_and_never_pinned	FR-14.2, FR-1.40: the first request registers and stores the auto task (updated_by null, the document and hash), one task_registered event, the example under its name with the client’s format; a second request only looks it up
test_a_registered_task_beats_the_schema	FR-1.3, FR-14.2: ticket-triage named with a schema stays itself, the schema forwarded and still checked (an answer outside it is served with schema_valid false and not stored)
test_a_name_the_client_makes_up_becomes_an_auto_task_under_distill_only	FR-14.2, FR-2.15: an unregistered name becomes an auto task under distill (a later schema with other fields gets its own auto task and an auto_task_collision event), stays unknown under teacher (route unknown_task, nothing stored); no name gives the auto task under any policy
test_a_client_cannot_choose_the_fields_another_clients_answers_are_compared_on	FR-14.2: a client naming another client’s computable auto name first with a weaker schema gets its own schema’s task; the victim’s auto task compares the refund; naming it later with another schema is a collision; a made-up name held by a weaker schema is not used for a stronger one
test_an_auto_name_held_by_other_fields_stores_nothing	FR-14.2, INV-16: an auto name registered with other fields gives an unregistered task that stores nothing, an auto_task_collision event, and a 400 under reject
test_a_schema_without_comparable_fields_follows_the_unknown_task_policy	FR-14.2, FR-2.15: a free-text schema is not registered; distill stores it, teacher routes it unknown_task, reject raises UnknownTask
test_auto_tasks_stop_at_the_cap_and_say_so_once	FR-14.2: past RATCHET_MAX_AUTO_TASKS new schemas get unregistered tasks; one auto_task_cap event, also across two processes on one ledger; 0 turns them off (default or the named task, held to the schema, never refused for it); a bad cap stops startup
test_an_auto_task_whose_schema_holds_personal_data_is_not_stored	FR-14.2, INV-12: an enum of email addresses never reaches the tasks table; an auto_task_skipped event
test_auto_tasks_come_back_at_startup_and_night_school_grades_under_the_schema	FR-14.2, INV-6, INV-16: taskspec.install registers a stored auto task again; dataset.rows and cascade_eval hold each row to its stored schema (the student sampled under it), so an answer whose fields agree but which lacks a required free-text key gets no vote and is no training target (skipped_invalid)
test_concurrent_requests_register_one_auto_task	FR-14.2: 8 threads on a new schema register one task, store one row and write one event
test_invalid_student_samples_get_no_vote_and_escalate	FR-1.40, INV-16: confident samples outside the enum have confidence 0 and escalate; matching ones are served; json_object refuses a non-object
test_json_object_answers_are_served_as_the_json_they_hold	FR-1.40: json_object serves and stores the object inside reasoning and a sentence, as written; text with no JSON is served as written with schema_valid false and not stored
test_an_answer_is_checked_against_its_schema_once_per_request	FR-1.40: three samples, an audit and the served choice with one text run one schema check (Contract.problem’s memo)
test_schema_mismatches_share_one_skip_counter	FR-1.40, INV-12: schema paths from clients count toward one throttle key; the 1st and 10th events each name their path, never the answer
test_a_repaired_teacher_answer_is_served_and_stored_with_the_original_messages	FR-1.40, INV-2, INV-16: the repair call carries the request, the first answer without its reasoning and a user turn naming the error; the same parameters; one example with the original messages and the repaired answer; both calls in cost_usd, the replaced one in usage_breakdown.repair, the baseline one call
test_a_teacher_answer_invalid_twice_is_served_but_never_stored	FR-1.40, INV-16: served with schema_valid false, not stored, a skip_example naming the path; a cut-off answer and n = 2 are not repaired
test_repair_can_be_switched_off_and_a_failed_repair_serves_the_first_answer	FR-1.40, INV-15: RATCHET_SCHEMA_REPAIR=0; a repair call that raises serves the first answer, counts repair_errors and writes schema_repair_error with the class only
test_an_audit_answer_is_repaired_too	FR-1.40: an audit’s invalid answer is repaired, counted as an audit and stored
test_arbitration_and_pinned_answers_are_repaired_and_priced	FR-1.40, INV-10, FR-15.1: an arbitration’s repair settles the shadow disagreement and is priced; a pinned task’s answer is repaired and priced (baseline one call) and never stored
test_after_a_repair_the_baseline_usage_and_latency_count_the_request_itself	FR-1.40, FR-15.1, FR-1.12: with a teacher whose prompt tokens grow with what it is sent, the baseline, usage.prompt_tokens and cached_tokens count the request’s own prompt on the escalation, pinned and audit paths; cost_usd and the breakdown pay both calls; latency_ms adds them
test_a_cut_off_or_empty_repair_answer_keeps_the_first_answer	FR-1.40, INV-16: a repair answer that was cut off, empty or only reasoning keeps the first answer and its finish_reason; nothing stored; the repair call is paid for
test_a_teacher_that_refuses_json_schema_gets_json_object_and_the_schema	FR-1.40, FR-1.11: a 400 naming response_format is retried once as json_object with the schema in the system message (added when there is none); the example keeps the client’s format and messages; the schema goes straight to json_object afterwards, one event; another 400 is not retried
test_refused_client_schemas_never_turn_json_schema_off_for_the_others	FR-1.40, FR-1.11: refused client schemas go as json_object one by one; after JSON_SCHEMA_STRIKES the endpoint is probed once with schema_tasks.PROBE_FORMAT, takes it, and json_schema stays on for every other schema; no second probe; an inconclusive probe (503) is asked again
test_json_schema_is_turned_off_for_an_endpoint_that_refuses_the_probe_too	FR-1.40, FR-1.11: an endpoint that refuses every json_schema refuses the probe too: supports_json_schema False, every schema as json_object; two events, not repeated by a second process; another endpoint unaffected
test_annotations_never_reach_a_model_or_the_ledger	FR-14.2, FR-1.13: the teacher’s request body and the stored conditions hold the schema without x-ratchet keys; the annotation decided the fields; the name hashes the annotated schema
test_fenced_teacher_json_is_served_as_bare_json	FR-1.40, FR-1.12: a live gateway serves bare JSON from a reasoning-and-fence answer (the SDK’s ChatCompletion validates it), streamed too, with task and schema_valid in the ratchet block
test_responses_name_the_task_and_stats_name_the_compared_keys	FR-1.40, FR-14.2, FR-2.15: ratchet.task and schema_valid for plain, named and schema requests; compare_keys and compare_source in /ratchet/stats; the auto task in /ratchet/tasks (updated_by null), deletable, a built-in one not; under reject a free-text schema is a 400 with param response_format
test_the_sdk_parse_helper_works_end_to_end_in_mock_mode	FR-1.40, FR-9.7: client.chat.completions.parse(response_format=Claim) against the mock gateway parses into the Pydantic model; the auto task compares covered, decision and parts
test_a_cut_off_answer_is_length_so_the_sdk_raises_its_length_error	FR-1.40: a cut-off teacher answer is finish_reason: "length", so .parse() raises LengthFinishReasonError; nothing stored

tests/test_tools.py

Tool and function calling (FR-1.41): what a tool request is, the legacy fields, the provider reading tool calls, the router answering tool requests with the teacher alone and storing nothing, the response and stream shapes the OpenAI SDK reads, and the multimodal route. Teachers are small providers in the file, RecordingProviders, or FakeOpenAIServer / ToolCallingServer behind httpx.MockTransport; nothing calls a real API.

Test	Covers
test_uses_tools_names_every_kind_of_tool_request	FR-1.41: tools, functions, an assistant tool_calls or function_call, a tool or function message, and tool_choice: "none" with tools are tool requests; empty tools or functions, a tool_choice or parallel_tool_calls with nothing to call, and empty tool_calls / null function_call are not
test_tool_params_map_the_legacy_fields	FR-1.41: functions become tools, function_call the tool choice (auto, none, {name}); legacy only when no current field is sent; a legacy request asks for one call (parallel_tool_calls: false unless set); both forms merge (the request’s tools first, duplicates by name dropped); nothing about calling is sent without a tool; GenParams.from_request carries the mapped fields and never the legacy names
test_legacy_history_pairs_calls_and_results	FR-1.41: an assistant function_call becomes a tool_calls entry with a stable call_ id and the function reply a tool message with that id; an orphan reply and plain messages stay as they are; the caller’s list is untouched; nothing to change returns the same list
test_provider_reads_tool_calls_in_openais_shape	FR-1.41, FR-1.11: vLLM’s call with index (dropped, its id kept) and an OpenAI call; object arguments JSON-encoded (non-ASCII kept), missing arguments {}, a missing type a function, a missing or empty id call_ and 24 hex digits (distinct); text beside the calls; a legacy function_call answer as one call; [] and null as no calls
test_a_tool_call_a_client_could_not_use_is_a_bad_upstream_response	FR-1.41, FR-1.11, INV-7: 7 cases (not an array, an entry not an object, a custom type, no function, no name, a blank name, one bad call of two): each a 502 bad_upstream_response
test_tool_call_text_that_is_not_valid_unicode_is_refused	FR-1.41, FR-1.11: an unpaired surrogate in a name or in the arguments is refused at once, not after the teacher was paid for and the response failed to encode
test_a_tool_request_is_answered_by_the_teacher_alone_and_never_stored	FR-1.41, INV-2, INV-10: route tools, the calls as made, finish_reason tool_calls, schema_valid null; the student never sampled and add_example never called; the configured teacher sent the tools, the choice and the parallel flag; tool_choice: "none" takes the same path with a prose answer; n = 2 choices; tool_requests per task
test_tool_history_without_tools_is_teacher_only	FR-1.41, INV-10: tool calls, a tool result, a legacy call and a function message without tools each go to the teacher alone with nothing stored; the same task without them is distilled
test_tool_routes_keep_the_operators_reason_first	FR-1.41, INV-10, FR-2.15: a pinned task and an unregistered one under teacher keep their route (tool calls returned); teacher_only gives tools or the route passed; a task with tools: distill is still teacher-only; a pinned, generic or multimodal teacher-only request without tools whose teacher makes a call is a 502 tool_calls_unsupported; tool_requests per task; without tools the generic path gives pinned
test_a_tool_call_is_not_held_to_the_response_format	FR-1.41, FR-1.40: a tool call with a json_schema is not repaired and schema_valid is null; a prose answer to the same request is checked
test_tool_requests_use_the_fallback_teacher_and_map_teacher_failures	FR-1.41, FR-2.12, INV-1, INV-7: an unavailable teacher’s tool request answered by the fallback teacher (route fallback_teacher, nothing stored, uses_tools 1); without one the error reaches the gateway
test_a_tool_request_gets_no_audit_and_no_shadow	FR-1.41, INV-2, INV-10: at audit rate 1 with a shadow candidate a tool request makes one teacher call (audited 0, no shadow version on the row); the same request without tools runs both
test_usage_counts_a_tool_calls_tokens	FR-1.41, FR-1.12: with no completion count from the teacher, the estimate includes the calls’ JSON
test_redact_messages_covers_tool_arguments_and_results	FR-1.41, FR-4.7, INV-12: emails in tool call and function call arguments and in tool and function results (strings and text parts) become [EMAIL], five counted; the caller’s messages untouched
test_tool_calls_pass_through_the_gateway	FR-1.41, FR-1.1, INV-1, INV-2, INV-10: a live gateway whose teacher (MockTransport) makes two calls: the SDK’s ChatCompletion validates the response, the calls and ids are identical, content null, finish_reason tool_calls; the teacher got the tools, choice and parallel flag under the configured model although the client sent gpt-4o; the student endpoint got nothing; nothing stored; route tools in /ratchet/stats; a teacher 400 about the request is the client’s 400
test_a_malformed_teacher_tool_call_is_a_502_and_nothing_is_stored	FR-1.41, FR-1.11, INV-7, INV-16: a call without a function name makes the response a 502 teacher_unavailable naming bad_upstream_response; the student never asked, nothing stored
test_legacy_function_calling_round_trips	FR-1.41, FR-1.1: functions and function_call reach the teacher as tools, tool_choice and parallel_tool_calls: false, a legacy history as tool turns; the answer is message.function_call with finish_reason function_call, streamed as one function_call delta (each chunk validated); a teacher’s second call is dropped in the legacy form; with tools too the current form (both calls); a server that answers the old way gets the current form for a current request
test_streamed_tool_calls_rebuild_with_the_sdks_accumulator	FR-1.41, FR-1.6, FR-1.12: the role chunk, the text beside the calls, one delta per call with the whole arguments, the finish chunk; every chunk validated and ChatCompletionStreamState rebuilds the calls of the non-streamed response; the usage chunk equals its usage; calls alone give a null-content role chunk and no content chunk
test_multimodal_requests_are_teacher_only_with_their_own_route	FR-1.41, FR-1.10, INV-12: with RATCHET_TEACHER_MODALITIES=text,image_url an image request skips the student, stores nothing and is logged with route multimodal (a pinned task keeps pinned); no image bytes anywhere in the ledger dump; an image with tools counts as a tool request
test_the_openai_sdk_runs_a_tool_loop_against_the_mock_gateway	FR-1.41, FR-9.7, US-1: the real SDK against the mock gateway: lookup_order with the order number, the tool result sent back, the final answer restating it, .stream() rebuilding the same call (stable mock ids), the legacy functions argument getting function_call; nothing stored; four tools routes (simulation, mock models)

Live-path tests

Live mode, the training job, the day-1 check, the Scout CLI and the deploy scripts would otherwise run for the first time on paid Nebius resources. tests/test_live.py runs each of them offline first, through seams that are default arguments, so production code passes nothing and talks to the real services:

Seam	Where	Test double
transport=	OpenAICompatProvider(base_url, api_key, timeout, retry, transport, sleep, clock=, rng=)	FakeOpenAIServer(...).transport (an httpx.MockTransport); sleep=, clock= and rng= replace the waits between retries
transports=	Runtime(settings, transports={"teacher": ..., "student": ...}), live mode only (other keys raise)	the same; later items drive live mode offline through it
client= / http=	load_adapter, NebiusTrainer, reload_adapters, live.main, http_caller, the Scout CLI’s main(argv, client), day1_check.main(client)	an httpx.Client on a fake server, or on AppTransport(gateway.app)
run=	NebiusTrainer, live.main (in place of subprocess.run)	RecordingRun
PATH	deploy/train_job.sh, deploy/student_endpoint.sh (the nebius they call)	FakeNebius: logs each call’s argv, prints tests/fixtures/nebius_endpoint_list.json for ai endpoint list; a stub ratchet.cost in the working directory logs the uptime records
provider_factory=	live.main (builds the student and judge clients)	a factory returning providers on fake servers
argv=	sim.simulate.main(argv), the Scout CLI	a list

The student’s sampling parameters are one read-only constant, confidence.STUDENT_SAMPLING (temperature 0.7, logprobs on), used by Ratchet._sample, LiveEvaluator.evaluate and MockFleet.evaluate, so the golden eval samples exactly as production does (FR-5.1) and a test fails if one of them drifts. The judge’s free-text candidate is still sampled greedily (TASK-FREE-TEXT-SIMILARITY changes that).

The live-run harness (python -m ratchet.liverun) is tested the same way: LiveRunWorld builds a live Runtime on Token Factory and vLLM fakes with its ledger in a file, and the harness gets its seams as default arguments (client= an httpx.Client on AppTransport(gateway.app) that records every path and header, provider_factory= over the fakes, run= a RecordingRun, http=, tavily=, clock= a FakeClock, timer= and sleep=). The module fixture liverun_world runs one 3-round run with nights in process once and most of the tests read its run directory. Tests that need the gateway’s seeded audit draws in a fixed order pass --concurrency 1.

These tests prove the wiring, the order of operations and the failure handling. They do not prove that the real services accept the requests: model ids, nebius CLI flags, vLLM’s runtime LoRA loading and GPU training stay untested (live) until the first-run checks and a live run pass.

Found while writing them: without RATCHET_STUDENT_BASE, NebiusTrainer ran the sync and the paid training job before noticing there was no endpoint to load the adapter into. It now checks first, and python -m ratchet.live reload gives the same clear error instead of an httpx URL error (test_live_night_checks_the_student_endpoint_before_paid_work in test_hardening.py).

Hot-path tests

tests/test_perf.py measures time, which a busy CI machine can stretch. Its timing tests stay stable by design: models answer with zero latency, so only Ratchet’s own work is timed; each bound is a ratio between two runs on the same machine (365 versions against one) or a ceiling several times above what a laptop measures (a warm tick takes about 3 ms against 20, a request about 0.3 ms against 5); each figure is the best of three runs or a median; and the 200,000 rows go in with one executemany, so the whole file runs in about 3 s. The structural claims (which index a query searches, how many sorts a stats read does, that routing reads no version row) are checked exactly, with EXPLAIN QUERY PLAN and the connection’s trace callback, so they do not depend on timing at all.

Contract tests

tests/test_contract.py was written before the v0.5 rewrites of Ratchet.handle and gateway.chat (request validation and normalisation, field comparators), as their regression net. Both have landed (tests/test_openai_compat.py, tests/test_tasks.py), and their five strict xfails now pass as ordinary tests. Each test pins behaviour that someone outside the module depends on: an OpenAI client (status codes, /v1/models, task selection), the dashboard (/ratchet/stats), an auditor (/ratchet/bom, license fields), or the operator’s bill (cost to the token).

Rules:

• The stats keys only grow. tests/fixtures/stats_keys.json lists every key the dashboard reads from /ratchet/stats (it mirrors Stats in web/src/lib/api.ts) with its JSON type; a [] key is a list whose every item is checked. A change may add keys to the payload and to the fixture. It must not remove or retype one, so the React dashboard and other clients keep working. The test also fails if the fixture loses one of the keys this contract started with.
• The cost report keys only grow. tests/fixtures/cost_keys.json does the same for GET /ratchet/cost and the dashboard’s Economics panel (it mirrors CostReport in web/src/lib/api.ts), checked by test_cost_report_contract.
• The task API keys only grow. tests/fixtures/tasks_keys.json does the same for the task API and the dashboard’s Task registry panel (it mirrors TaskRegistry, TaskEntry, FieldDoc and the write results in web/src/lib/api.ts), checked by test_task_registry_contract. The Resilience panel’s event grammar is not a fixture: the test reads it from the panel’s source, so the event text and the parser change together.
• Known bugs are strict xfails that name the fixing item, a v0.5 work item (none today: API-VALIDATE-NORMALIZE removed its four and TASK-COMPARATORS its one). When the fix lands the test passes, strict mode turns that into a failure, and the fix removes the marker in the same change. Write the mark literally (@pytest.mark.xfail(strict=True, reason=...), or marks=pytest.mark.xfail(...) in a pytest.param): the traceability scan reads marks from the source, so a mark held in a variable would make an xfail count as proof.
• Cost assertions are token-exact: each model gets distinct token counts, and the expected cost is built from Settings.price(...), so a term billed with the wrong tokens, price or multiplicity fails.
• The dashboard’s rendering is still checked by hand (web/CLAUDE.md); test_stats_contract proves the data it renders. FR-7.6 (light and dark mode, phone width) stays a manual check.

Simulation methodology

sim/simulate.py runs --days days. Each day sends --requests requests from the chosen --workload through Ratchet.handle(), calls ReleaseGate.tick() ten times, then runs run_night(). It prints a day table and the event log, and writes JSON (--out, default sim_results.json).

Flag	Default	Meaning
--days	8	Days to simulate
--requests	2000	Requests per day
--threshold	0.80	Fallback threshold before calibration
--audit-rate	0.10	Higher than live (0.05) so canaries move in a few days
--min-shadow	100	ReleasePolicy.min_shadow
--min-audits	15	ReleasePolicy.min_audits (live default 20)
--min-new	50	run_night(min_new_examples=…)
--drift-day	5	Product launch day (0 = never)
--bad-deploy-version	0	Fault injection: this version is served corrupted once in canary/live
--workload	tickets	tickets (support tickets), research (Scout’s triage and reader calls), mixed (half each), schema (structured outputs) or tools (support requests with the mock tool catalogue). With tickets the night trains on ticket-triage only; otherwise on every task
--seed	7	Seed for traffic and routing
--out	sim_results.json	Results file

Assumptions (these drive the numbers; they are not measurements)

Traffic (sim/tickets.py):

• 20 templates across 6 categories (billing, technical, account, shipping, refund, security) and 4 priorities. 6 of them are marked hard.
• Names, products and order numbers vary, so some templates produce many distinct texts and others very few (frequent repeats).
• From --drift-day, 20% of traffic comes from 3 new “Ratchet Watch” templates the student has never seen (1 of them hard).

Research traffic (sim/research.py, --workload research):

• 8 questions about well-known ML ideas, each phrased 4 ways, each with a pool of 72 simulated search results (12 templates × 6 variants). 60% of requests are source-triage calls, 40% page-read calls.
• Each template has fixed labels. 6 of the 12 are hard (for example a good forum answer next to an off-topic one, or a broad survey paper).
• From --drift-day, 20% of traffic comes from questions 7 and 8, which the student has not seen.
• mixed alternates between the ticket and research streams at random (50/50, its own seed), so both keep their own sequences.

Schema and tools traffic (sim/generic.py, --workload schema, --workload tools):

• schema: requests with response_format json_schema over three schemas, each its own task (review-rating: an enum, a 1 to 5 integer, a boolean; ticket-route: two enums and a boolean; meeting-extract: two enums, an integer and an array of enums). From --drift-day, 20% of traffic is a fourth schema the student has not seen (warranty-claim, with nested records through $ref). Prompts combine products, opinions and a four-digit reference, so most texts are distinct.
• tools: support requests sent with the mock catalogue (lookup_order, refund) under order-support: 40% look-ups, 45% refunds with an amount and a reason, 15% questions no tool answers; new wordings from --drift-day.
• The right answer is drawn from sha256 of the request’s text, not read from it (the mock teacher is right by definition). A schema’s labels are its enum, boolean and number fields; a template is a schema name (or a tool set, or a system prompt), and a schema with more than three labels is hard.
• A tool request is answered by the teacher alone and never stored (FR-1.41), so the tools workload never trains the student: its share stays 0 and it saves nothing (until tool-call distillation).

Mock teacher:

• Always right on the labels.
• Bills approx_tokens(prompt) input plus answer and 300 reasoning output tokens.
• 2.4 s latency.

Mock student skill for a ticket of template t:

• If its exact text was in training: skill = 0.985.
• Otherwise: skill = max(0.35, CEIL × (1 − e^(−n/6))), where n is the number of distinct training examples of t, and CEIL = 0.985 (easy) or 0.86 (hard).
• Whether the student “knows” a ticket is fixed per (version, text).

Sample behaviour:

• Knows: each sample is right with probability 0.97; token probability 0.80–0.97.
• Doesn’t know, confidently wrong (15% of unknown cases): all samples give the same wrong labels with probability 0.62–0.92. This deliberately overlaps with correct answers, so calibration cannot separate them perfectly. That is the case audits and shadow exist for.
• Doesn’t know, otherwise: random labels, probability 0.35–0.65.

Research items use the same skill model, per result template. Scout’s pinned planner and writer prompts are answered by the mock teacher only.

Fault injection:

• The corrupted version is fine on the staging copy, which golden eval and shadow use.
• Once it reaches canary or live, its “fragile” slices become confidently wrong: billing, refund and security tickets; paper and docs triage results; date reads; in the generic world refund calls, schema answers whose first enum takes its first value, and free text framed as steps. This models pushing the wrong adapter file to production.

Costs: Ultra $1/$3 per 1M tokens. The student uses the $0.06/$0.24 proxy, with 3 samples billed.

Typical result (seed 7, 10 days, 2,000/day, drift day 5, bad deploy v3)

This run is the pinned reference (below), so these numbers cannot drift silently.

day  student  escal. audit ok     $/1k all-Ultra  saved
  1       0%      0%        -    1.093     1.093     0%
  2      44%      1%     100%    0.675     1.093    38%
  3      98%      2%     100%    0.160     1.093    85%
  4      98%      2%      93%    0.160     1.093    85%   v3 canary → rolled back (41% audited precision)
  5      86%     14%      97%    0.272     1.093    75%   product launch
  6      88%     12%      92%    0.246     1.093    77%
  7      88%     12%      97%    0.255     1.093    77%
  8      95%      5%      99%    0.200     1.093    82%
  9      98%      2%      97%    0.162     1.093    85%
 10      98%      2%      99%    0.147     1.093    87%
total saved 69%

Seed robustness (10 days, 2,000/day, drift day 5, bad deploy v3)

Seed	Final student share	Final-day savings	Total savings	Min daily audit agreement	Bad v3 outcome
1	98%	85%	69%	97%	rejected before deploy
2	98%	86%	66%	96%	rejected before deploy
3	98%	87%	67%	93%	rolled back
11	98%	85%	66%	92%	rolled back

“Total” includes the cold-start days, when all traffic goes to the teacher.

The ticket numbers above are the same as in v0.1. Adding the research world did not change the random draws for tickets (sim/CLAUDE.md).

Workloads: research and mixed

Simulation, mock models. 8 days × 1,500 requests/day, drift day 5, no fault injection (selected days shown). Both workloads have several tasks on one adapter, so each task has its own threshold and its own golden check (per-task calibration, on by default):

python -m sim.simulate --workload research --days 8 --requests 1500     (seed 7)
day  student  escal. audit ok     $/1k all-Ultra  saved
  1       0%      0%        -    1.133     1.133     0%
  2      15%      3%     100%    0.987     1.131    13%
  3      94%      6%      99%    0.205     1.132    82%
  4      93%      7%      99%    0.218     1.132    81%
  5      93%      7%      98%    0.197     1.132    83%
  8      92%      8%      98%    0.211     1.132    81%
total: 12000 requests, $5.11 vs $13.58 all-Ultra -> saved 62%

python -m sim.simulate --workload mixed --days 8 --requests 1500        (seed 7)
day  student  escal. audit ok     $/1k all-Ultra  saved
  1       0%      0%        -    1.112     1.112     0%
  3      19%      2%     100%    0.932     1.112    16%
  4      92%      8%      98%    0.226     1.111    80%
  5      86%     14%      96%    0.265     1.111    76%
  8      94%      6%      97%    0.202     1.112    82%
total: 12000 requests, $6.48 vs $13.34 all-Ultra -> saved 51%

Seeds 1, 2, 3, 7, 11 (same settings):

Workload	Final student share	Final-day savings	Total savings	Lowest daily audit agreement	First version live
research	65–97%	56–86%	19–65%	97–98%	day 2–6
mixed	90–96%	78–85%	47–60%	96–98%	day 2–3
research, RATCHET_PER_TASK_CALIBRATION=0	96–98%	85–86%	57–65%	96–98%	day 2–3
mixed, RATCHET_PER_TASK_CALIBRATION=0	92–98%	79–86%	54–63%	93–96%	day 2–3

On mixed traffic the first student comes a night later with seed 7 (v1 was rejected at 94.3% golden precision), so total savings are lower. With one pooled threshold and the pooled gate (RATCHET_PER_TASK_CALIBRATION=0, v0.4’s behaviour) the student serves more at a lower worst day of audited agreement. Per task, each task’s threshold and its golden check rest on a few dozen held-out rows, and the check is a point estimate, as the global gate’s is: a good task dips under its 95% floor, or more than 2 points under live’s 100% on it, now and then, and is disabled on that version (research, seed 11: the first version’s page-read at 18 of 19 right; its canary, answering one task at 10% of traffic, then took until day 6 to collect its audits). A version with a disabled task that live still answers usually loses the shadow’s coverage comparison too, so it is rejected a day later (research, seed 1, nights 3 to 5).

Workloads: schema and tools

Simulation, mock models. 8 days × 1,500 requests/day, drift day 5, no fault injection. Both come from the generic mock world; their right answers are drawn from each request’s text, so these numbers show the loop on structured outputs, not how well a model reads a review.

python -m sim.simulate --workload schema --days 8 --requests 1500       (seed 7)
day  student  escal. audit ok     $/1k all-Ultra  saved
  1       0%      0%        -    1.020     1.020     0%
  2      15%      4%     100%    0.893     1.020    12%
  3      77%     19%     100%    0.334     1.020    67%
  4      85%     15%      97%    0.248     1.020    76%
  5      80%     20%      96%    0.312     1.023    70%
  8      93%      7%      97%    0.193     1.023    81%
total: 12000 requests, $5.33 vs $12.26 all-Ultra -> saved 57%

python -m sim.simulate --workload tools --days 8 --requests 1500        (seed 7)
day  student  escal. audit ok     $/1k all-Ultra  saved
  1       0%      0%        -    1.068     1.068     0%
  2       0%      0%        -    1.069     1.069     0%
  8       0%      0%        -    1.069     1.069     0%
total: 12000 requests, $12.83 vs $12.83 all-Ultra -> saved 0%

Seeds 1, 2, 3, 7, 11 (same settings):

Workload	Final student share	Final-day savings	Total savings	Lowest daily audit agreement	First version live
schema	70–93%	60–81%	40–57%	92–97%	day 3
tools	0%	0%	0%	no audits	never (nothing stored)
schema, RATCHET_PER_TASK_CALIBRATION=0	86–94%	75–82%	55–61%	92–96%	day 2–3

On schema the drift schema (warranty-claim, five or more labels, hard) holds the share near 80% for two nights from day 5. Its three tasks share one adapter, so per-task calibration (on by default) applies, and with some seeds a task disabled on a few dozen held-out rows lowers the share, as on the research workload above; tools has one task and is the same either way. On tools every request offers tools, so the mock teacher answers it alone with its tool calls and nothing is stored (FR-1.41): no night trains, the student never serves and the run saves nothing. It shows what the gateway does with tool requests until tool-call distillation; before FR-1.41 both models answered these prompts in prose with the tools stripped, and that prose was learned within two nights.

Bugs the simulation found (and the fixes)

Running the loop end to end exposed design flaws that unit tests would not have caught. Each fix is in the code now.

#	Symptom in the simulation	Root cause	Fix
1	Every new candidate rejected as “regresses vs live” for days	Candidate measured on today’s golden set, live measured on its own (older, easier) golden set from the day it was trained	Champion vs challenger: run_night re-evaluates the live version on the same golden set first and overwrites its eval (rechecked_day)
2	Every candidate rejected in shadow at ~70% “agreement”	Shadow judged on all traffic, including requests it would have escalated; and an agreement with the live student was the only reference	Shadow runs the full cascade: records shadow_serve and judges precision only on what it would serve, plus coverage vs live. Disagreements with the live student are arbitrated by the teacher
3	Student never learned the most common questions	The hash-based golden split held some frequently repeated requests out of training forever	Requests seen at least FREQUENT = 3 times always go to training and leave the golden set
4	Canary never finished; promotions stalled	Each night’s new candidate replaced the version in canary	run_night skips training while a shadow or canary is in flight, which also saves GPU time
5	Threshold promised 95% precision but delivered 94%	calibrate_threshold checked precision before including all points tied at the same confidence	Walk down in groups of equal confidence; return inf (serve nothing) if no threshold qualifies; cascade_eval caps it at 1.01
6	Golden coverage 94% vs real coverage 80%	Deduplication meant the golden set over-represented rare phrasings, and the threshold was calibrated and measured on the same rows	seen count stored per example; calibrate on one half of the golden set, measure on the other; frequent requests handled by fix 3

Fixes 1, 3, 4 and 6 also have direct tests now: test_champion_rescored_on_same_golden_set, test_golden_rows_never_reach_the_trainer_except_frequent, test_night_skips_with_reason and test_cascade_eval_measures_on_held_out_half; fix 2 is covered by test_shadow_promotes_and_both_rejections and test_shadow_arbitration_calls_teacher_once_and_stores_example.

Two more problems were fixed earlier in development. A fixed global threshold let about 10% wrong answers through, which led to per-version calibration. A point-estimate rollback rule would have rolled back healthy canaries about 19% of the time at n=20, which led to the Wilson upper bound.

Issues found while documenting, and their fixes

Writing these docs turned up ten gaps in v0.1. Each one is fixed in v0.1.1 and covered by tests/test_hardening.py.

#	Issue	Fix	Test
1	Live gateway never advanced Ledger.day, so every request landed in one bucket	Ledger(auto_day=True) in live mode: day 1 starts at the first logged request, one day per 24 hours	test_auto_day_counts_days_from_first_request
2	A student HTTP error became a 500	Student errors fall back to the teacher and log a student_error event (1st, 10th, 100th, 1000th). A teacher error returns 502	test_student_failure_falls_back_to_teacher, test_teacher_failure_returns_502
3	Runtime-loaded LoRA adapters are lost when the endpoint restarts	New command python -m ratchet.live reload re-loads live, canary and shadow adapters	test_reload_adapters_loads_live_canary_shadow (in test_live.py, against a fake vLLM); on a real endpoint, see RUNBOOK.md
4	The untuned fallback model was judged with the tuned version’s threshold	Ratchet.threshold(version, model) uses the global default for RATCHET_FALLBACK_STUDENT	test_fallback_model_uses_default_threshold
5	alert events repeated on every tick	One alert per live version	test_live_alert_fires_once_per_version
6	No schema migration for existing database files	Ledger._migrate() adds missing columns on open (MIGRATIONS table in ledger.py)	test_old_database_file_is_migrated
7	Stale docstring and column comment	Fixed	n/a
8	Unknown X-Ratchet-Task values were stored forever in tasks.REGISTRY	Unknown names get a temporary whole-text task; names must match [A-Za-z0-9_.-]{1,64}	test_unknown_tasks_do_not_grow_registry
9	ReleasePolicy could only be changed in code	ReleasePolicy.from_env() reads RATCHET_<FIELD> (e.g. RATCHET_MIN_AUDITS)	test_policy_from_env
10	LoRA targets might miss Nemotron-H Mamba layers	train.py --targets (default adds in_proj,out_proj); confirm names on day 1	The default: test_train_parse_args_defaults_and_row_contract; the module names: manual (GPU)

Still open: RATCHET_SYNC_CMD and RATCHET_TRAIN_CMD run without a shell, so pipes and && do not work; wrap them in a script.

The pinned reference simulation

tests/fixtures/sim_reference_seed7.json holds one run of the simulator with every input pinned (seed 7, 10 days × 2,000 requests, drift day 5, corrupted v3, mock models; the same run as make sim): the inputs, each day’s requests, student share, escalation rate, savings and audited agreement rounded to 4 decimal places, the ordered Night School and release events as (day, kind, version), and each version’s final state, traffic and training-set size. test_reference_simulation_unchanged runs it once per session (the session-scoped reference_run fixture, about 7 s) and compares exactly. The inputs compared are the ones the fixture pins: a simulator flag added later, whose default keeps the old behaviour, does not force a regeneration, while a changed or removed pinned input fails the test.

It and the reference workloads below are the deliberate exceptions to “assert on trends, not numbers”. Refactors and new features that claim to leave the legacy behaviour untouched prove it by keeping this test green. When a change is meant to move the simulation, run python scripts/regen_sim_reference.py, review the JSON diff (one day or event per line), commit it with the change and add a CHANGELOG line saying why; never edit the file by hand. --check compares without writing, with the same rule as the test.

The run unsets every RATCHET_* variable while it runs, so settings in your shell cannot move it, and it depends only on seeded random.Random streams, SHA-256-derived seeds and SQLite arithmetic. When the fixture was made it was identical on CPython 3.10, 3.11, 3.12 and 3.13 (Linux, the CI matrix) and under different PYTHONHASHSEED values. If it fails on one platform only, compare the first differing day before regenerating anything. The numbers are simulation output, not measurements.

The reference workloads

scripts/reference_check.py pins three workloads, captured from v0.4.0 (commit 1f9b18c) in tests/fixtures/reference.json: tickets-seed7 (the run above), mixed-seed1 and research-seed7 (8 days × 1,500 requests). All three run with RATCHET_PER_TASK_CALIBRATION=0 (LEGACY_ENV in the script), because per-task calibration, on by default, changes what the multi-task runs serve by design; so the fixture is still v0.4.0’s capture, byte for byte. The tickets run has one task, so it gives the same output with the switch on (the session’s reference run, which the pytest check uses). It pins more than the reference simulation does: every input, every v0.4 key of each day’s stats at full precision (cost_per_1k, avg_latency_ms, …), the totals, each version’s final state, training-set size and dataset hash, and the event log with its text. Every later feature defaults to v0.4’s behaviour, so every change passes it: make reference (about 25 s) and a CI step run all three; the pytest suite checks the tickets workload against the session’s reference run at no extra cost (test_reference_workloads_reproduce_v040).

The rules: floats agree within a relative 1e-9 (an index may add the same numbers in another order); days and totals may gain keys; events compare as (day, kind, version) in order, and each pinned text must be a prefix of the fresh one, so a feature may append evidence to a message but never change it; the advisory kinds later features add (drift, teacher_drift, teacher_model_changed, warning, student_healed) are left out; a flag added later is not compared, a changed default is. When a change is meant to move these runs: python scripts/reference_check.py --write --captured-from "<release or commit>", review the diff, and add a CHANGELOG line that says why. The numbers are simulation output from mock models, not measurements.

Adding tests

• Pure logic goes in test_core.py with small hand-built ledgers (Ledger(":memory:")); Night School rules in test_nightschool.py; release-gate and router release behaviour in test_release.py. Set up states with the builders in tests/fakes.py (_version, _audit, _shadow, _examples) and fake models with RecordingProvider, RecordingTrainer and RecordingEvaluator, rather than new local helpers.
• A change to what clients or the dashboard see (a status code, a /ratchet/stats key, cost accounting) goes in test_contract.py. New stats keys also go in tests/fixtures/stats_keys.json, new cost report keys the dashboard reads in tests/fixtures/cost_keys.json, and new task API keys it reads in tests/fixtures/tasks_keys.json; existing ones stay. A change to the text of a circuit event or the outage drill’s fault event also changes the dashboard’s CIRCUIT_RE / OUTAGE_RE / WAIT_RE (test_resilience_panel_contract).
• Scout, the research world and the guard go in test_scout.py. Use the gw fixture (mock gateway, no warm-up) and fake clocks for the guard.
• Live mode, CLIs and deploy scripts go in test_live.py. Pass the seams (Live-path tests) instead of monkeypatching httpx or subprocess, and set every Settings value and environment variable the test depends on.
• Loop behaviour goes in test_system.py through sim(...) with fixed seeds. Assert on trends (rising share, falling escalations), not exact numbers, so tuning the mock does not break tests. The exception is the pinned reference run; reuse the reference_run fixture instead of starting another 10-day simulation.
• A new Ledger.add_example call under ratchet/ needs a reviewed edit to ALLOWED_ADD_EXAMPLE_CALLERS in test_nightschool.py.
• Never call real APIs in tests. Use httpx.MockTransport as in test_openai_compat_provider_parses_logprobs_and_usage.
• Mark every new test with the requirement ids it proves (@pytest.mark.req), keep parametrize lists literal so the suite can be counted statically, and update the test counts above.