Command line

Every Ratchet command and Make target, with when to use it and its help text captured from the code.

Every command line tool in the repository, with a line on when to use it and its help text. The help text is captured from the code by python docs-site/scripts/export-reference.py, which runs each tool with --help (or reads its module docstring when it has no parser), so it matches the code it was exported from. Run commands from the repository root after make install.

Make targets

From the Makefile. On Windows, run the command in the second column from WSL or Git Bash, or set the variables the PowerShell way ($env:RATCHET_MODE="mock").

Target	Runs	What it is for
`make install`	`pip install -e ".[dev]" ruff`	Install the package with its dev dependencies and ruff.
`make test`	`pytest -q`	Run the test suite (no network, no GPU).
`make lint`	`ruff check .` `ruff format --check .` `python scripts/reqtrace.py --check`	Lint, check formatting and check the requirements matrix. Run it with `make test` before every commit.
`make reference`	`python scripts/reference_check.py`	Three pinned workloads (tickets, mixed, research) must still reproduce v0.4.0’s behaviour (about 25 s; CI runs it).
`make fmt`	`ruff format .` `ruff check --fix .`	Format the code and apply safe lint fixes.
`make sim`	`python -m sim.simulate --days 10 --requests 2000 --bad-deploy-version 3`	The 10-day simulation with a corrupted deploy of v3 (mock models).
`make demo`	`RATCHET_MODE=mock RATCHET_MOCK_WARMUP_DAYS=4 uvicorn ratchet.gateway:app --port 8000`	The mock gateway with a 4-day warm-up on `http://localhost:8000`: `/`, `/scout` and `/dashboard`.
`make web`	`cd web && npm ci && npm run build`	React app (web/) -> ratchet/web/dist, which the gateway serves at /, /scout and /dashboard.
`make web-dev`	`cd web && npm ci && npm run dev`	Hot-reload UI on :5173 against a gateway on :8000 (run `make demo` in another terminal).
`make day1`	`python scripts/day1_check.py`	Live Token Factory checks before live mode (needs `NEBIUS_API_KEY`; costs cents).
`make live-plan`	`python -m ratchet.liverun plan --rounds $(ROUNDS) --requests $(REQUESTS) --max-usd $(MAX_USD) --workload $(WORKLOAD)`	Live-run harness: plan a run (requests per release stage, golden size, dollars per round). Free.
`make live-smoke`	`python -m ratchet.liverun smoke --base-url $(BASE_URL)`	Live-run harness: 20 requests and a few control calls, no nights. Costs cents.
`make live-run`	`python -m ratchet.liverun run --base-url $(BASE_URL) --rounds $(ROUNDS) --requests $(REQUESTS) --max-usd $(MAX_USD) --workload $(WORKLOAD)`	Live-run harness: the measured run, capped at `MAX_USD`.
`make docker`	`docker build -t ratchet-gateway .`	Build the gateway image (`ratchet-gateway`).
`make clean`	`rm -rf .pytest_cache .ruff_cache .coverage build dist *.egg-info sim_results.json ratchet.db` `find . -name __pycache__ -type d -prune -exec rm -rf {} +`	Remove caches, build output, `sim_results.json` and `ratchet.db`.

Run it in live mode

`python -m ratchet.live`

When to use it: Live mode on Nebius: run Night School (night), advance the release gate (tick), reload adapters after an endpoint restart (reload), seal savings receipts (seal), apply retention now (purge) or export an adapter with its licenses (export).

Help (the docstring of ratchet/live.py):

Live-mode Night School on Nebius: train on a Serverless Job, hot-load the adapter into vLLM, evaluate.

    python -m ratchet.live night        # run one night against the live ledger
    python -m ratchet.live tick         # advance the release gate (run every few minutes via cron)
    python -m ratchet.live reload       # re-load live/canary/shadow adapters after an endpoint restart (the gateway
                                        # also does this by itself when vLLM answers 404 for an adapter: AdapterReloader)
    python -m ratchet.live purge [--client NAME] [--older-than-days N] [--dry-run]
                                        # apply the retention windows now (FR-13.4); the gateway does it hourly
    python -m ratchet.live export --version N --out DIR
                                        # an adapter with its licenses: the adapter folder (when synced locally), its
                                        # BOM and NOTICE.md in DIR; prints the BOM's sha256 (NFR-9)
    python -m ratchet.live seal         # seal the savings receipts of every day that has ended (run hourly via cron:
                                        # a ledger day starts at the first request's time of day, not 00:00 UTC; the
                                        # gateway also seals when a receipt is asked for)

Environment (in addition to ratchet.config):
    RATCHET_DATA_DIR      local folder synced to the bucket mounted in jobs/endpoints (default ./data)
    RATCHET_SYNC_CMD      command that uploads RATCHET_DATA_DIR to the bucket, e.g.
                          "aws s3 sync ./data s3://my-bucket/ratchet --endpoint-url https://storage.eu-north1.nebius.cloud"
                          (run before training, and again after a night that trained, to upload the adapter's BOM
                          into its folder; never with --delete, which would remove the weights the job wrote there)
    RATCHET_TRAIN_CMD     command that runs the training job and waits for it; {version} and {data}
                          are substituted, e.g. "bash deploy/train_job.sh {version}"
    RATCHET_ADAPTER_ROOT  where the endpoint sees adapters (default /data/adapters)

After a night that trained, the adapter's BOM and NOTICE.md go to RATCHET_DATA_DIR/adapters/<name>/ (ratchet_bom.json,
NOTICE.md) and RATCHET_SYNC_CMD runs again, so the bucket's adapter folder holds the weights and their licenses together.

`python -m ratchet.live export`

When to use it: Hand an adapter to someone else with its bill of materials and NOTICE.md.

Help (–help):

usage: python -m ratchet.live export [-h] --version VERSION --out OUT

Copy an adapter with its licenses into a folder: the adapter files (when
RATCHET_DATA_DIR/adapters/<name> holds them), its bill of materials (ratchet_bom.json) and
NOTICE.md; prints the BOM's sha256.

options:
  -h, --help         show this help message and exit
  --version VERSION  the version id (as in /ratchet/versions/{id}/bom)
  --out OUT          the folder to write; it must not exist yet or be empty

`python -m ratchet.live purge`

When to use it: Apply the retention windows now instead of waiting for the gateway’s hourly run.

Help (–help):

usage: python -m ratchet.live purge [-h] [--client CLIENT] [--older-than-days OLDER_THAN_DAYS]
                                    [--dry-run]

Apply the retention windows now (FR-13.4): purge the text of examples not seen for
RATCHET_EXAMPLE_RETENTION_DAYS and fold request rows older than RATCHET_REQUEST_RETENTION_DAYS
into daily sums.

options:
  -h, --help            show this help message and exit
  --client CLIENT       only this API client's examples and request rows
  --older-than-days OLDER_THAN_DAYS
                        this window, in whole days, for both examples and requests (instead of the
                        settings)
  --dry-run             count what would go and change nothing

`python -m ratchet.config`

When to use it: Check the environment before you start or update a gateway: exit 2 means the gateway would not start.

Help (–help):

usage: python -m ratchet.config [-h] {check} ...

Check Ratchet's settings.

positional arguments:
  {check}
    check     list every configuration error and warning; exit 0, 1 (warnings) or 2 (errors)

options:
  -h, --help  show this help message and exit

python -m ratchet.config check --help

usage: python -m ratchet.config check [-h] [--json]

options:
  -h, --help  show this help message and exit
  --json      print one JSON object

`python -m ratchet.cost`

When to use it: Print the cost report (savings per token and by GPU-hours paid) and record when the student endpoint was up.

Help (–help):

usage: python -m ratchet.cost [-h] {report,endpoint,close} ...

Ratchet cost accounting

positional arguments:
  {report,endpoint,close}
    report              print the cost report and the recommendation
    endpoint            record the student endpoint going up or down
    close               close an open interval of a kind (a job whose end was lost)

options:
  -h, --help            show this help message and exit

python -m ratchet.cost report --help

usage: python -m ratchet.cost report [-h] [--db DB] [--days DAYS] [--json]

options:
  -h, --help   show this help message and exit
  --db DB      the live ledger file (default: RATCHET_DB)
  --days DAYS  only the last N ledger days
  --json       print the whole report as JSON

python -m ratchet.cost endpoint --help

usage: python -m ratchet.cost endpoint [-h] [--db DB] [--usd-per-hour USD_PER_HOUR] [--gpus GPUS]
                                       [--at AT] [--note NOTE]
                                       {up,down}

positional arguments:
  {up,down}

options:
  -h, --help            show this help message and exit
  --db DB               the live ledger file (default: RATCHET_DB)
  --usd-per-hour USD_PER_HOUR
                        USD per GPU-hour (default: RATCHET_STUDENT_GPU_USD_PER_HOUR)
  --gpus GPUS           GPUs (default: RATCHET_STUDENT_GPUS)
  --at AT               when, if not now: seconds since the epoch or ISO 8601 (UTC unless it says)
  --note NOTE           a note stored with the interval

python -m ratchet.cost close --help

usage: python -m ratchet.cost close [-h] [--db DB] [--at AT]
                                    {student_endpoint,training_job,eval_job}

positional arguments:
  {student_endpoint,training_job,eval_job}

options:
  -h, --help            show this help message and exit
  --db DB               the live ledger file (default: RATCHET_DB)
  --at AT               when it ended: seconds since the epoch or ISO 8601 (default: now)

`python -m ratchet.admin`

When to use it: Operator actions from a shell, with an audit trail: erase an end user’s data or export the ledger’s tables.

Help (–help):

usage: python -m ratchet.admin [-h] [--db DB]
                               {versions,rollback,retire,promote,flag,pin,unpin,audit-log,erase,export} ...

Manual interventions, with an audit trail; erase stored data and export tables.

positional arguments:
  {versions,rollback,retire,promote,flag,pin,unpin,audit-log,erase,export}
    versions            every version, the runtime switches and the task pins
    rollback            roll a canary back
    retire              retire a live, canary or shadow version
    promote             promote a version to a canary stage or to live
    flag                show the runtime switches, or set one
    pin                 pin a task to the teacher
    unpin               remove a task's runtime pin
    audit-log           the newest admin actions
    erase               erase stored examples for good (a dry run without --apply)
    export              one table as NDJSON

options:
  -h, --help            show this help message and exit
  --db DB               the ledger file (default: RATCHET_DB)

python -m ratchet.admin versions --help

usage: python -m ratchet.admin versions [-h] [--json]

options:
  -h, --help  show this help message and exit
  --json

python -m ratchet.admin rollback --help

usage: python -m ratchet.admin rollback [-h] --reason REASON id

positional arguments:
  id

options:
  -h, --help       show this help message and exit
  --reason REASON  why, in 3 to 500 characters

python -m ratchet.admin retire --help

usage: python -m ratchet.admin retire [-h] --reason REASON id

positional arguments:
  id

options:
  -h, --help       show this help message and exit
  --reason REASON  why, in 3 to 500 characters

python -m ratchet.admin promote --help

usage: python -m ratchet.admin promote [-h] --to {canary,live} [--traffic-pct TRAFFIC_PCT]
                                       [--force] [--load-adapter] --reason REASON
                                       id

positional arguments:
  id

options:
  -h, --help            show this help message and exit
  --to {canary,live}
  --traffic-pct TRAFFIC_PCT
                        the canary's share, 1 to 99 (default: the next stage)
  --force               allow a rejected, rolled-back or retired version or a candidate
  --load-adapter        load the adapter into the student endpoint first
  --reason REASON       why, in 3 to 500 characters

python -m ratchet.admin flag --help

usage: python -m ratchet.admin flag [-h] [--reason REASON] [--force] [--json]
                                    [{student_disabled,releases_frozen,training_paused,audit_rate}]
                                    [value]

positional arguments:
  {student_disabled,releases_frozen,training_paused,audit_rate}
  value                 on|off for a switch; 0 to 1 or 'default' for audit_rate

options:
  -h, --help            show this help message and exit
  --reason REASON
  --force               allow an audit rate of 0 while a canary is in flight
  --json

python -m ratchet.admin pin --help

usage: python -m ratchet.admin pin [-h] --reason REASON task

positional arguments:
  task

options:
  -h, --help       show this help message and exit
  --reason REASON  why, in 3 to 500 characters

python -m ratchet.admin unpin --help

usage: python -m ratchet.admin unpin [-h] --reason REASON task

positional arguments:
  task

options:
  -h, --help       show this help message and exit
  --reason REASON  why, in 3 to 500 characters

python -m ratchet.admin audit-log --help

usage: python -m ratchet.admin audit-log [-h] [--limit LIMIT] [--json]

options:
  -h, --help     show this help message and exit
  --limit LIMIT  1 to 1000 (default 50)
  --json

python -m ratchet.admin erase --help

usage: python -m ratchet.admin erase [-h] --reason REASON [--client CLIENT] [--end-user END_USER]
                                     [--hash HASHES] [--contains CONTAINS] [--apply]
                                     [--actor ACTOR]

options:
  -h, --help           show this help message and exit
  --reason REASON      why (stored redacted in the erasure log)
  --client CLIENT      an API key name: the examples it sent (and the scope of --end-user)
  --end-user END_USER  the value a client sends as user or safety_identifier; '-' reads it from
                       stdin
  --hash HASHES        an example hash (repeat for more)
  --contains CONTAINS  text in the stored (already redacted) messages, answer or conditions
  --apply              erase; without it nothing changes
  --actor ACTOR        who asks, for the erasure log: a name of letters, digits, '_', '.', '-'
                       (default 'cli')

python -m ratchet.admin export --help

usage: python -m ratchet.admin export [-h] [--task TASK] [--split {train,golden}] [--since SINCE]
                                      [--client CLIENT] [--out OUT]
                                      {examples,requests,versions,events,erasures}

positional arguments:
  {examples,requests,versions,events,erasures}

options:
  -h, --help            show this help message and exit
  --task TASK
  --split {train,golden}
  --since SINCE         seconds since the epoch, or an ISO 8601 date or date-time (UTC)
  --client CLIENT       an API key name
  --out OUT             file to write (default standard output)

`python -m ratchet.importer`

When to use it: Import labelled examples (human labels or an allowed teacher’s answers) from a JSONL file.

Help (–help):

usage: python -m ratchet.importer [-h] --task TASK --source SOURCE [--attest ATTEST]
                                  [--golden-only] [--client CLIENT] [--db DB] [--dry-run]
                                  file

Import labelled data into the ledger.

positional arguments:
  file             a JSONL file (OpenAI chat fine-tuning format, or Ratchet's train.jsonl); '-'
                   reads stdin

options:
  -h, --help       show this help message and exit
  --task TASK      the registered task the rows belong to
  --source SOURCE  'human' (with --attest) or 'teacher:<model>' (an open teacher)
  --attest ATTEST  where the labels come from and that you own them (required with --source human)
  --golden-only    hold every row out of training for good (golden split)
  --client CLIENT  the API key name to attribute the rows to (for erasure and export by client)
  --db DB          ledger file (default RATCHET_DB)
  --dry-run        import into an in-memory copy and report; write nothing

`python -m ratchet.liverun`

When to use it: Measure a live run: rounds of synthetic traffic with Night School between them and a control arm sent straight to the teacher.

Help (–help):

usage: python -m ratchet.liverun [-h] {plan,smoke,run,report,verify} ...

Live-run harness

positional arguments:
  {plan,smoke,run,report,verify}

options:
  -h, --help            show this help message and exit

python -m ratchet.liverun plan --help

usage: python -m ratchet.liverun plan [-h] [--base-url BASE_URL] [--client CLIENT]
                                      [--workload WORKLOAD] [--seed SEED] [--rounds ROUNDS]
                                      [--requests REQUESTS] [--max-usd MAX_USD]
                                      [--control-share CONTROL_SHARE]
                                      [--control-max-usd CONTROL_MAX_USD]
                                      [--concurrency CONCURRENCY] [--batch BATCH]
                                      [--until-settled | --no-until-settled]
                                      [--max-extend MAX_EXTEND] [--nights {inprocess,off}]
                                      [--on-night-failure {stop,continue}]
                                      [--drift-round DRIFT_ROUND]
                                      [--untuned-arm | --no-untuned-arm] [--scout-runs SCOUT_RUNS]
                                      [--plan-share PLAN_SHARE]
                                      [--plan-train-hours PLAN_TRAIN_HOURS]
                                      [--plan-night-minutes PLAN_NIGHT_MINUTES] [--out OUT]
                                      [--no-endpoint-check] [--gate-profile {planned,as-set}]

options:
  -h, --help            show this help message and exit
  --base-url BASE_URL
  --client CLIENT       the harness key's name in RATCHET_API_KEYS
  --workload WORKLOAD   tickets, research, mixed, or file:PATH
  --seed SEED
  --rounds ROUNDS
  --requests REQUESTS
  --max-usd MAX_USD
  --control-share CONTROL_SHARE
  --control-max-usd CONTROL_MAX_USD
  --concurrency CONCURRENCY
  --batch BATCH
  --until-settled, --no-until-settled
  --max-extend MAX_EXTEND
  --nights {inprocess,off}
  --on-night-failure {stop,continue}
  --drift-round DRIFT_ROUND
  --untuned-arm, --no-untuned-arm
  --scout-runs SCOUT_RUNS
  --plan-share PLAN_SHARE
  --plan-train-hours PLAN_TRAIN_HOURS
  --plan-night-minutes PLAN_NIGHT_MINUTES
  --out OUT
  --no-endpoint-check
  --gate-profile {planned,as-set}
                        planned (run's default): refuse unless RATCHET_SEQUENTIAL=1 and
                        RATCHET_CANARY_AUDIT_RATE=0.5 are in force

python -m ratchet.liverun smoke --help

usage: python -m ratchet.liverun smoke [-h] [--base-url BASE_URL] [--client CLIENT]
                                       [--workload WORKLOAD] [--seed SEED] [--rounds ROUNDS]
                                       [--requests REQUESTS] [--max-usd MAX_USD]
                                       [--control-share CONTROL_SHARE]
                                       [--control-max-usd CONTROL_MAX_USD]
                                       [--concurrency CONCURRENCY] [--batch BATCH]
                                       [--until-settled | --no-until-settled]
                                       [--max-extend MAX_EXTEND] [--nights {inprocess,off}]
                                       [--on-night-failure {stop,continue}]
                                       [--drift-round DRIFT_ROUND]
                                       [--untuned-arm | --no-untuned-arm]
                                       [--scout-runs SCOUT_RUNS] [--plan-share PLAN_SHARE]
                                       [--plan-train-hours PLAN_TRAIN_HOURS]
                                       [--plan-night-minutes PLAN_NIGHT_MINUTES] [--out OUT]
                                       [--no-endpoint-check] [--gate-profile {planned,as-set}]
                                       [--repeat-of RUN_DIR]

options:
  -h, --help            show this help message and exit
  --base-url BASE_URL
  --client CLIENT       the harness key's name in RATCHET_API_KEYS
  --workload WORKLOAD   tickets, research, mixed, or file:PATH
  --seed SEED
  --rounds ROUNDS
  --requests REQUESTS
  --max-usd MAX_USD
  --control-share CONTROL_SHARE
  --control-max-usd CONTROL_MAX_USD
  --concurrency CONCURRENCY
  --batch BATCH
  --until-settled, --no-until-settled
  --max-extend MAX_EXTEND
  --nights {inprocess,off}
  --on-night-failure {stop,continue}
  --drift-round DRIFT_ROUND
  --untuned-arm, --no-untuned-arm
  --scout-runs SCOUT_RUNS
  --plan-share PLAN_SHARE
  --plan-train-hours PLAN_TRAIN_HOURS
  --plan-night-minutes PLAN_NIGHT_MINUTES
  --out OUT
  --no-endpoint-check
  --gate-profile {planned,as-set}
                        planned (run's default): refuse unless RATCHET_SEQUENTIAL=1 and
                        RATCHET_CANARY_AUDIT_RATE=0.5 are in force
  --repeat-of RUN_DIR   run the same plan and seed again

python -m ratchet.liverun run --help

usage: python -m ratchet.liverun run [-h] [--base-url BASE_URL] [--client CLIENT]
                                     [--workload WORKLOAD] [--seed SEED] [--rounds ROUNDS]
                                     [--requests REQUESTS] [--max-usd MAX_USD]
                                     [--control-share CONTROL_SHARE]
                                     [--control-max-usd CONTROL_MAX_USD]
                                     [--concurrency CONCURRENCY] [--batch BATCH]
                                     [--until-settled | --no-until-settled]
                                     [--max-extend MAX_EXTEND] [--nights {inprocess,off}]
                                     [--on-night-failure {stop,continue}]
                                     [--drift-round DRIFT_ROUND]
                                     [--untuned-arm | --no-untuned-arm] [--scout-runs SCOUT_RUNS]
                                     [--plan-share PLAN_SHARE]
                                     [--plan-train-hours PLAN_TRAIN_HOURS]
                                     [--plan-night-minutes PLAN_NIGHT_MINUTES] [--out OUT]
                                     [--no-endpoint-check] [--gate-profile {planned,as-set}]
                                     [--repeat-of RUN_DIR] [--resume RUN_DIR]

options:
  -h, --help            show this help message and exit
  --base-url BASE_URL
  --client CLIENT       the harness key's name in RATCHET_API_KEYS
  --workload WORKLOAD   tickets, research, mixed, or file:PATH
  --seed SEED
  --rounds ROUNDS
  --requests REQUESTS
  --max-usd MAX_USD
  --control-share CONTROL_SHARE
  --control-max-usd CONTROL_MAX_USD
  --concurrency CONCURRENCY
  --batch BATCH
  --until-settled, --no-until-settled
  --max-extend MAX_EXTEND
  --nights {inprocess,off}
  --on-night-failure {stop,continue}
  --drift-round DRIFT_ROUND
  --untuned-arm, --no-untuned-arm
  --scout-runs SCOUT_RUNS
  --plan-share PLAN_SHARE
  --plan-train-hours PLAN_TRAIN_HOURS
  --plan-night-minutes PLAN_NIGHT_MINUTES
  --out OUT
  --no-endpoint-check
  --gate-profile {planned,as-set}
                        planned (run's default): refuse unless RATCHET_SEQUENTIAL=1 and
                        RATCHET_CANARY_AUDIT_RATE=0.5 are in force
  --repeat-of RUN_DIR   run the same plan and seed again
  --resume RUN_DIR

python -m ratchet.liverun report --help

usage: python -m ratchet.liverun report [-h] [--bill-usd BILL_USD] [--bill-note BILL_NOTE]
                                        [--repeat REPEAT_DIR] [--publish] [--allow-partial]
                                        run_dir

positional arguments:
  run_dir

options:
  -h, --help            show this help message and exit
  --bill-usd BILL_USD
  --bill-note BILL_NOTE
  --repeat REPEAT_DIR   attach a repeat run's comparison
  --publish
  --allow-partial

python -m ratchet.liverun verify --help

usage: python -m ratchet.liverun verify [-h] dir

positional arguments:
  dir

options:
  -h, --help  show this help message and exit

Check the evidence offline

`python -m ratchet.bom`

When to use it: Make a signing key, and verify an adapter’s signed bill of materials and its NOTICE.md offline.

Help (–help):

usage: python -m ratchet.bom [-h] {keygen,verify} ...

Signed bill of materials for each trained adapter.

positional arguments:
  {keygen,verify}
    keygen         print a new signing key and its public key
    verify         check a BOM document (file or - for stdin)

options:
  -h, --help       show this help message and exit

python -m ratchet.bom keygen --help

usage: python -m ratchet.bom keygen [-h]

options:
  -h, --help  show this help message and exit

python -m ratchet.bom verify --help

usage: python -m ratchet.bom verify [-h] [--public-key PUBLIC_KEY] [--notice NOTICE] path

positional arguments:
  path

options:
  -h, --help            show this help message and exit
  --public-key PUBLIC_KEY
                        the base64 public key you trust
  --notice NOTICE       a NOTICE.md to check against the notice the BOM signed

`python -m ratchet.stats`

When to use it: Plan how many golden rows a certificate needs, compute exact bounds, and recheck every number in a signed BOM.

Help (–help):

usage: python -m ratchet.stats [-h] {plan,bound,mcnemar,check} ...

Exact statistics behind Ratchet's certificates.

positional arguments:
  {plan,bound,mcnemar,check}
    plan                golden rows needed to certify a precision, for 0 to --max-errors wrong
                        answers
    bound               one-sided Clopper-Pearson lower bound on a rate
    mcnemar             exact one-sided McNemar p-value from the discordant pairs
    check               verify a BOM and recompute its certificate from the signed counts

options:
  -h, --help            show this help message and exit

python -m ratchet.stats plan --help

usage: python -m ratchet.stats plan [-h] [--target TARGET] [--delta DELTA]
                                    [--max-errors MAX_ERRORS]

options:
  -h, --help            show this help message and exit
  --target TARGET
  --delta DELTA
  --max-errors MAX_ERRORS

python -m ratchet.stats bound --help

usage: python -m ratchet.stats bound [-h] --successes SUCCESSES --n N [--delta DELTA]

options:
  -h, --help            show this help message and exit
  --successes SUCCESSES
  --n N
  --delta DELTA

python -m ratchet.stats mcnemar --help

usage: python -m ratchet.stats mcnemar [-h] --b B --c C

options:
  -h, --help  show this help message and exit
  --b B       pairs where the first side wins
  --c C       pairs where the second side wins

python -m ratchet.stats check --help

usage: python -m ratchet.stats check [-h] [--public-key PUBLIC_KEY] path

positional arguments:
  path                  a BOM document (GET /ratchet/versions/{id}/bom), or - for stdin

options:
  -h, --help            show this help message and exit
  --public-key PUBLIC_KEY
                        the base64 Ed25519 public key you trust

`python -m ratchet.proof`

When to use it: Check offline that a request was, or was not, in an adapter’s training set, and that no golden row was trained on.

Help (–help):

usage: python -m ratchet.proof [-h] {key,verify,heldout} ...

Committed training sets: prove from an adapter's signed BOM alone that an example was, or was not,
in its training or golden set.

positional arguments:
  {key,verify,heldout}
    key                 print the example key a request's teacher answer is stored under
    verify              check a membership or non-membership proof against a BOM
    heldout             check that every golden example is absent from the training set

options:
  -h, --help            show this help message and exit

python -m ratchet.proof key --help

usage: python -m ratchet.proof key [-h] --messages MESSAGES [--task TASK] [--no-redact]

options:
  -h, --help           show this help message and exit
  --messages MESSAGES  a JSON file: a messages list or a chat request body (- for stdin)
  --task TASK          the task (default: the body's metadata.ratchet_task, else 'default')
  --no-redact          the gateway runs with RATCHET_REDACT_PII=0

python -m ratchet.proof verify --help

usage: python -m ratchet.proof verify [-h] --bom BOM [--public-key PUBLIC_KEY]
                                      [--expect {included,absent}]
                                      path

positional arguments:
  path                  the proof (POST /ratchet/versions/{id}/proof), or - for stdin

options:
  -h, --help            show this help message and exit
  --bom BOM             the adapter's BOM (GET /ratchet/versions/{id}/bom)
  --public-key PUBLIC_KEY
                        the base64 Ed25519 public key you trust
  --expect {included,absent}

python -m ratchet.proof heldout --help

usage: python -m ratchet.proof heldout [-h] --bom BOM [--public-key PUBLIC_KEY] path

positional arguments:
  path                  the certificate (GET /ratchet/versions/{id}/heldout), or - for stdin

options:
  -h, --help            show this help message and exit
  --bom BOM             the adapter's BOM (GET /ratchet/versions/{id}/bom)
  --public-key PUBLIC_KEY
                        the base64 Ed25519 public key you trust

`python -m ratchet.receipt`

When to use it: Verify a savings receipt, its chain and an inclusion proof offline, or seal a day from a ledger file.

Help (–help):

usage: python -m ratchet.receipt [-h] {verify,seal} ...

Verify or seal verified savings receipts

positional arguments:
  {verify,seal}
    verify       check a receipt (file, or - for stdin)
    seal         seal every day that has ended in a ledger file, offline

options:
  -h, --help     show this help message and exit

python -m ratchet.receipt verify --help

usage: python -m ratchet.receipt verify [-h] [--public-key PUBLIC_KEY] [--prev PREV]
                                        [--proof PROOF] [--rows ROWS]
                                        receipt

positional arguments:
  receipt

options:
  -h, --help            show this help message and exit
  --public-key PUBLIC_KEY
                        the base64 public key you trust (GET /ratchet/bom/public-key)
  --prev PREV           the same client's previous receipt: check the chain
  --proof PROOF         a request's inclusion proof (GET /ratchet/receipts/{period}/proof/{id})
  --rows ROWS           the receipt's leaves (GET /ratchet/receipts/{period}/leaves): recompute
                        everything

python -m ratchet.receipt seal --help

usage: python -m ratchet.receipt seal [-h] [--db DB] [--period PERIOD] [--client CLIENT]

options:
  -h, --help       show this help message and exit
  --db DB          the ledger file (default: RATCHET_DB)
  --period PERIOD  seal through this ledger day and print its receipt
  --client CLIENT  with --period: print this client's receipt (default: the gateway's)

`python -m ratchet.replay`

When to use it: Ask what past traffic would have cost and served at another threshold or audit rate, from a ledger file.

Help (–help):

usage: python -m ratchet.replay [-h] [--db DB] [--version VERSION] [--sweep SWEEP]
                                [--audit-rate AUDIT_RATE] [--target TARGET]
                                [--since-day SINCE_DAY] [--until-day UNTIL_DAY] [--json]

Replay a version's logged traffic under another threshold or audit rate

options:
  -h, --help            show this help message and exit
  --db DB               the ledger file (default: RATCHET_DB)
  --version VERSION     one version (default: every version with replayable traffic)
  --sweep SWEEP         thresholds LO:HI:STEP (default 0.30:1.00:0.05)
  --audit-rate AUDIT_RATE
                        price audits at this rate (default: as logged)
  --target TARGET       precision the suggestion needs (default 0.97)
  --since-day SINCE_DAY
                        first ledger day
  --until-day UNTIL_DAY
                        last ledger day
  --json                print the reports as JSON

Demos and simulation

`python -m ratchet.scout`

When to use it: Run Scout from a terminal against any gateway, as an outside client would.

Help (–help):

usage: python -m ratchet.scout [-h] [--base-url BASE_URL] [--api-key API_KEY] [--max-results MAX_RESULTS]
                   question

Scout: multi-agent research through Ratchet

positional arguments:
  question

options:
  -h, --help            show this help message and exit
  --base-url BASE_URL
  --api-key API_KEY
  --max-results MAX_RESULTS

`python -m sim.simulate`

When to use it: Run the whole loop offline with mock models (simulation): days of traffic, nights of training, releases and rollbacks.

Help (–help):

usage: python -m sim.simulate [-h] [--days DAYS] [--requests REQUESTS] [--threshold THRESHOLD]
                   [--audit-rate AUDIT_RATE] [--min-shadow MIN_SHADOW] [--min-audits MIN_AUDITS]
                   [--min-new MIN_NEW] [--drift-day DRIFT_DAY]
                   [--bad-deploy-version BAD_DEPLOY_VERSION]
                   [--workload {tickets,research,mixed,schema,tools}] [--seed SEED]
                   [--release {wilson,sequential}] [--promote-evidence]
                   [--canary-audit-rate CANARY_AUDIT_RATE] [--calibration {point,ltt}]
                   [--cert-target CERT_TARGET] [--cert-delta CERT_DELTA] [--paired-gate]
                   [--min-certified MIN_CERTIFIED] [--compare-release] [--receipts]
                   [--teacher-reasoning-mean TEACHER_REASONING_MEAN]
                   [--teacher-reasoning-cv TEACHER_REASONING_CV] [--out OUT] [--db DB]

Ratchet offline simulation

options:
  -h, --help            show this help message and exit
  --days DAYS
  --requests REQUESTS   requests per day
  --threshold THRESHOLD
  --audit-rate AUDIT_RATE
  --min-shadow MIN_SHADOW
  --min-audits MIN_AUDITS
  --min-new MIN_NEW
  --drift-day DRIFT_DAY
                        day a new product launches (0 = never)
  --bad-deploy-version BAD_DEPLOY_VERSION
                        fault injection: this student version is pushed corrupted to production
  --workload {tickets,research,mixed,schema,tools}
                        tickets, Scout research calls, both (mixed), structured outputs (schema)
                        or tool requests (tools)
  --seed SEED
  --release {wilson,sequential}
                        canary rule: the Wilson bound per stage, or the anytime-valid e-process

  --promote-evidence    with --release sequential: promote a canary stage only on an e-value that
                        precision is above the floor
  --canary-audit-rate CANARY_AUDIT_RATE
                        the canary's own audit rate, more than 0 and at most 1 (default: --audit-
                        rate)
  --calibration {point,ltt}
                        threshold rule: the point rule at 97%, or Learn-then-Test over a fixed
                        grid
  --cert-target CERT_TARGET
                        the precision a certificate must show
  --cert-delta CERT_DELTA
                        a certificate's error rate
  --paired-gate         compare each candidate with the live version row by row on the same golden
                        rows (exact McNemar tests)
  --min-certified MIN_CERTIFIED
                        reject a candidate whose certified lower bound is below this (0 = off)
  --compare-release     run twice with the same seed, once per --release rule, and compare the
                        rolled-back versions
  --receipts            seal each day's savings receipt and print it next to the true all-teacher
                        cost
  --teacher-reasoning-mean TEACHER_REASONING_MEAN
                        the mock teacher bills this many reasoning tokens on average instead of
                        exactly 300 (e.g. 420)
  --teacher-reasoning-cv TEACHER_REASONING_CV
                        with --teacher-reasoning-mean: the spread (coefficient of variation,
                        default 0.3)
  --out OUT
  --db DB               write the run's ledger to this SQLite file (it must not exist yet), e.g.
                        for python -m ratchet.replay

Build, train and check the repo

`python -m ratchet.nightschool.train`

When to use it: Inside the training job’s container only: fine-tune a LoRA adapter on train.jsonl. Night School starts it for you.

Help (–help):

usage: python -m ratchet.nightschool.train [-h] --data DATA [--base BASE] --out OUT [--epochs EPOCHS] [--lr LR] [--rank RANK]
                [--max-seq MAX_SEQ] [--batch BATCH] [--targets TARGETS]

Ratchet Night School: LoRA fine-tune

options:
  -h, --help         show this help message and exit
  --data DATA
  --base BASE
  --out OUT
  --epochs EPOCHS
  --lr LR
  --rank RANK
  --max-seq MAX_SEQ
  --batch BATCH
  --targets TARGETS  comma-separated LoRA target modules, or 'all-linear'. Nemotron-H mixes
                     attention, Mamba (in_proj/out_proj) and MLP layers; if vLLM refuses the Mamba
                     modules on day 1, use q_proj,k_proj,v_proj,o_proj,up_proj,down_proj.

`python scripts/day1_check.py`

When to use it: Before live mode: check the model IDs, logprobs and JSON mode on Token Factory with a real key (costs a few cents).

Help (the docstring of scripts/day1_check.py):

Day-1 check: confirm the Token Factory pieces Ratchet depends on actually work.

    NEBIUS_API_KEY=... python scripts/day1_check.py

Checks: the model IDs exist, the teacher answers, logprobs come back, JSON mode works, the
teacher accepts the generation parameters the gateway sends it (`max_tokens` at
RATCHET_TEACHER_MAX_TOKENS, `stop`, `seed`, `temperature`, `response_format`; ADR 0017), and
whether it takes `response_format` json_schema (strict) with an answer that matches the schema
. A teacher that refuses json_schema is a warning, not a failure: the gateway then sends
json_object with the schema in the system message. Costs a few cents.

`python scripts/reference_check.py`

When to use it: Check (or rewrite) the pinned reference runs that hold the simulation’s behaviour still.

Help (–help):

usage: python scripts/reference_check.py [-h] [--only {tickets-seed7,mixed-seed1,research-seed7}] [--write]
                          [--captured-from CAPTURED_FROM]

Check (or rewrite) the three pinned reference runs of v0.4.0's behaviour.

options:
  -h, --help            show this help message and exit
  --only {tickets-seed7,mixed-seed1,research-seed7}
                        check this workload only (repeatable)
  --write               rewrite the fixture from fresh runs of every workload
  --captured-from CAPTURED_FROM
                        with --write: the release or commit the runs reproduce, stored in the
                        fixture

`python scripts/regen_sim_reference.py`

When to use it: Regenerate (or check) the seed-7 reference simulation fixture the web app’s numbers are pinned to.

Help (–help):

usage: python scripts/regen_sim_reference.py [-h] [--check]

Regenerate (or check) the pinned reference simulation fixture.

options:
  -h, --help  show this help message and exit
  --check     compare a fresh run with the fixture and write nothing

Command line

Make targets

Run it in live mode

`python -m ratchet.live`

`python -m ratchet.live export`

`python -m ratchet.live purge`

`python -m ratchet.config`

`python -m ratchet.cost`

`python -m ratchet.admin`

`python -m ratchet.importer`

`python -m ratchet.liverun`

Check the evidence offline

`python -m ratchet.bom`

`python -m ratchet.stats`

`python -m ratchet.proof`

`python -m ratchet.receipt`

`python -m ratchet.replay`

Demos and simulation

`python -m ratchet.scout`

`python -m sim.simulate`

Build, train and check the repo

`python -m ratchet.nightschool.train`

`python scripts/day1_check.py`

`python scripts/reference_check.py`

`python scripts/regen_sim_reference.py`

Start here

How it works

Scout

Run it

Security

API & reference

Help