Rescue-Brain Test Ladder

The four rules

What makes a number admissible

These are the rules that stop the ladder from producing numbers that flatter us. They are older than this bake-off and they apply to every gate in the pipeline.

No number without its N. A median with an interquartile range over a stated number of placements, or nothing. A single mission is a demo, not a measurement — the 822 s pilot run is labelled as such.
The baseline is measured, not remembered. v0 is a git tag re-run under the full protocol, not a figure someone recalls from a previous week. The detector baseline is reproducible to identical metrics.
The judge set is frozen. Never trained on, never curated against, never used to fit a calibration constant. Same for the 56 frozen frames.
One variable per comparison. Every delta we report changes imagery or task or precision — never two at once. This is the rule the ladder exists to enforce.

The design

Two axes, four cells, four honest deltas

Evaluating an edge VLM has two independent axes: whether the imagery is rendered or real, and whether the model answers one frame or drives a whole mission. Varying both at once produces a number nobody can attribute. So the four measurements sit at the corners of a factorial, and the deltas are read along its edges.

Each solid edge changes one variable, so each yields an attributable delta. The two horizontal edges are two independent estimates of the same quantity — what closing the loop costs — which makes them a consistency check on each other. The dashed diagonal, L0 straight to L3, changes both variables at once: it is the comparison that looks most impressive on a slide and means nothing.

The four deltas and what each one is allowed to claim
Delta	Variable changed	What it measures	Held constant
L0 → L1	imagery	Sim-to-real perception gap	Static task. Hardware is confounded unless the 4-bit arms are also scored on the B300 — which is why round 2 requires that.
L0 → L2	task	Cost of closing the loop, on rendered imagery	Synthetic imagery, same engine on the same Orin
L1 → L3	task	Cost of closing the loop, on real imagery	Real imagery, same engine on the same Orin
L2 → L3	imagery + physics	Sim-to-real mission gap, on matched placements	Protocol, log schema, scorer, brain

Execution order is L0, L1, L2, L3 — cheapest and most available first. That is a scheduling order, not a traversal: the claims live on the edges above.

The rungs

What runs where, and what it reports

Metrics change at the static/mission boundary and are never compared across it. A QA accuracy is not a mission result, and a mission success rate is not an accuracy.

Layers L0–L3 · two configurations from L1 onward
Layer	Where	Metrics	Baseline in hand	State
L0	B300 bf16 + 4-bit	Distance, direction and MCQ accuracy as a task mean; tok/s and peak memory per hardware	Cosmos3-Edge 47.1 % Gemma-4-E4B 33.6 % k = 1.15 → 31.8→38.8 %	Round 1 done
L1	jetson1 bench INT4 only	False-alarm rate on non-lying poses — the deciding metric — plus per-pose confusion and McNemar between arms with the discordant counts shown. Then lying recall and recall per distance bucket once the judge set lands. Decision latency; peak memory standalone and with the rover stack co-resident	Cosmos3-Edge bf16 recall 30/30 FA 23/30	Round 1 done
L2	Isaac on lw-lab1 brain on jetson1	Found within 1 m / 5 m; time to report; false reports; VLM calls; distance driven. Median + IQR, bootstrap 95 % CI, paired Wilcoxon across placements	pilot: 822 s, 56 looks, confirm 15 m (N = 1 — not a baseline)	Protocol ready
L3	jetson0 on the rover marked floor	As L2, plus sim-to-real gap per metric on matched placements. Comms cut mid-run; report arrives on reconnect	none — the first real run is the baseline	Week 3

Pruning rule

L0 and L1 are the elimination rounds. L2 and L3 carry at most two configurations: the winner, plus gemma-4-e4b-it-qat-q4_0 as the incumbent control — it is what the rover runs today, so it is the baseline a swap has to beat, not a second candidate. From L1 onward, deployment precision only: bf16 peaked at 16.0 GB and cannot fit an 8 GB Orin, so a bf16 mission arm would be measuring something that can never ship. bf16 stays at L0, as the reference the quantisation delta is measured against.

Instruments

Three sets of frames, and what each one can settle

A layer is not a dataset. L1 is the real static cell, and it draws on two sources with very different statistical weight — so the question "which model is better" and the question "how far can it see" get answered by different frames.

What each instrument decides · exact McNemar power against the 13.5-point gap round 1 measured
Instrument	N	Axis it settles	Power	Available
Spatial-QA val pairs rendered warehouse	986	Distance, direction, MCQ accuracy	~1.00	bf16 done
Blackbox mission frames real sensor, past missions	474 of 2,223	Posture false alarms on real frames — the axis the recorded prediction turns on	1.00 (20-pt gap) 0.94 (10-pt gap)	On disk now
judge-v1 taped floor, measured	40 → 120 positives	Lying recall and the distance buckets — what blackbox cannot supply	0.21 at 40 0.31 at 56	Capture + labelling

Why the judge set is not the instrument for the model comparison

It was scoped for the training-data ablation, where small N is tolerable because the question is directional. For a paired model comparison it is underpowered: at 40 to 56 items, a real 13.5-point difference goes unnoticed two times in three. The judge set can catch a collapse or a reversal, not a ranking — so its McNemar is reported as a consistency check on the blackbox result, with its power stated, never as an independent verdict.

The archive of past missions carries the weight instead: real sensor, real clutter, already deduplicated, and dense in exactly the class that decides this — non-lying poses, where bf16 Cosmos3-Edge fired on 23 of 30. A 20-point difference there is detectable at N = 150 and certain by 300; the selected slice of 474 reaches 0.94 power even on a 10-point gap. It runs today. What it cannot give is lying people: it is workplace footage, so the positive class is nearly absent and no distance was ever measured. That is the judge set's job, and the only one it is sized for.

Hardware in the loop

One brain, two worlds, one scorer

The sim-to-real mission gap is only a real number if everything except the world is identical. So the model does not run on the workstation during L2 — it runs on the bench Orin, in the loop, executing the same INT4 engine the rover will execute, contending with the detector for the same 8 GB.

Hosting the model on the B300 during L2 would be faster to set up and would produce latencies the rover can never reproduce — and would leave L2 and L3 differing in hardware, runtime and world at once. If the network round-trip inflates latency, we measure the transport and subtract it rather than relocating the brain.

Upstream

The data is gated before the GPU sees it

The ladder tests a trained model. A separate set of gates tests the training set first, because a clean evaluation of a model trained on bad data is a precise answer to the wrong question.

Dataset quality gates, run before any training run
Gate	Question	Instrument
A1	Is it well-formed?	Schema and hygiene checks; resolution rule — render 1280×720, derive by resize, never upscale
A2	Is it the same frame twice?	pHash / dHash near-duplicate pass, NeMo Curator semantic dedupe
A3	Does it cover the problem?	Coverage table across 8 distance buckets, pose classes, lighting, smoke, hard negatives
A3b	Do labels survive Cosmos Transfer?	Reasoner's reading of the repainted frame vs the renderer's own COCO box, per control strength
A4	Are the boxes right?	Teacher disagreement ranked by FiftyOne compute_mistakenness; two annotators and Cohen's κ on the judge set
A5	Will training on it help?	CCDM-MMD and SDQM as training-free proxies, then TSTR as the ground truth. FID/KID reported, never ranked on

The question the ladder exists to answer

Does the leaderboard predict the mission?

Nothing links L0 to L3 except an assumption: that the model which answers more spatial questions correctly will also find more casualties. The ladder is built to test that assumption rather than inherit it. Either answer is a result worth reporting — and "the higher QA score did not survive the closed loop" would be the more useful finding of the two.

Prediction recorded 10 Sep, before L1 ran

Cosmos3-Edge will lose at L2 and L3 on false reports, not on detection — and the deciding number will be the false-alarm rate at L1, not the 47.1 % at L0.

The L1 baseline in hand is bf16 Cosmos3-Edge at 30/30 lying recall with 23/30 false alarms on non-lying poses: near-perfect recall while firing on 77 % of everything else. False reports are exactly what the SubT scoring rule punishes. If the prediction holds, the single SFT run should target false alarms rather than distance calibration — and a prediction written down before the data arrives cannot be retrofitted afterwards.

What L1 measured, 11 Sep — INT4 on the Orin

On detection the prediction's axis held: on 24 real frames at INT4, Cosmos3-Edge found 9–10 of 12 targets to Gemma's 0–1. The false-report side — the half the prediction turned on — is under re-measurement.

The prediction read Cosmos's weakness off the bf16 arm, which fired on 23 of 30 non-lying poses. Whether that habit carries to the precision the rover actually runs is what the 474-frame blackbox slice measures at power, under the production parser; no false-report figure is claimed from the 24 frames. On detection, Cosmos found 9–10 of 12 targets to Gemma's 0–1 (paired p = 0.002–0.004). The SFT run the prediction aimed at Cosmos's false alarms waits on that measurement, and the judge set on the recall side; 24 frames is a first look, not the instrument.

Measured so far, up the ladder

The prediction is testable because the lower rungs are already closed. This is what each rung that has run has actually produced — read up from L0, not across from L0 to L3.

L0 · closed

Synthetic Spatial-QA, B300

Cosmos3-Edge scored 47.1 % task-mean accuracy to Gemma's 33.6 % — a 13.5-point gap over 986 pairs. Distance calibration is fitted per artefact (bf16 k = 1.15), lifting held-out distance accuracy 31.8 → 38.8 %. And 4-bit is statistically indistinguishable from bf16 after a per-arm re-fit (p = 0.931 on distance), so the precision the rover actually ships costs no measured accuracy.

L1 · closed

Real frames, INT4 on the Orin

On 24 real frames: 9–10 of 12 targets found to Gemma's 0–1 (McNemar p = 0.002–0.004), false-alarm rate under re-measurement, English commands 13/15 with 5/5 on stop synonyms — and the JSON schema emitted 0 of 24 times, which is why the contract moved into a parser and the geometry into code. The 474-frame blackbox slice and the judge set are what raise this to full power.

L2 · pilot only

Isaac in the loop, brain on jetson1

One mission has run end to end: 822 s to confirm a casualty at 15 m over 56 VLM looks, the INT4 engine co-resident with the detector on the 8 GB Orin. N = 1 — proof the loop closes on the real engine, not a baseline. The scored protocol across 20 placements is what turns it into one.

Outside the lab

Comparable to something other than ourselves

No single standard covers a rover looking for a person indoors, so each layer borrows the closest established method and states honestly how closely it follows it. We never claim certification.

External comparability per layer
Layer	Reference	How closely
L0 / L1	COCO mAP protocol; AI City Challenge test splits; SARD	Aligned metrics
L1 runtime	MLPerf-style edge reporting — configuration, power mode and N stated with every figure	Report structure
L2 / L3 navigation	ISO 18646-2:2024 — pose repeatability, obstacle detection, path deviation	Run as specified
L2 / L3 mission	DARPA SubT scoring; ObjectNav Success / SPL / DTS	Aligned metrics
L3 method	ASTM E2853 and NIST response-robot test methods	In the spirit of
Gates A1–A5	ISO/IEC 5259 dataset-quality dimensions	Report structure

Honest status

What's settled, and what isn't

Recall + range only

The judge set does not exist yet

Captured Friday morning, then two-annotator labelling before it is usable. It gates lying recall and the distance buckets — not the model ranking, which runs on past-mission frames instead. L3 reuses its floor markings, so the floor gets taped once for both: 7 conditions × 8 distance buckets = 56 frames per lighting condition, ground truth measured off the tape rather than estimated.

Cleared 11 Sep

The INT4 engine exists and runs on the Orin

Recorded here as a risk on 10 Sep: Cosmos3-Edge was absent from the TensorRT Edge-LLM supported-model matrix, and the one demonstrated Cosmos VLM ran on Thor at a precision Orin does not support. It closed a day later. The 4-bit engine was built, exported and executed on the 8 GB Orin Nano — 2.24 GiB of engines, ~4,830 MB full-VLM peak, 52.8 tok/s decode, 169 INT4 nodes verified — after three issues and two patches to the builder. The Gemma 4-bit arm stays as the incumbent control, not a fallback for a missing engine.

Contended

L2 competes with rendering for the same GPU

Isaac needs RT cores, so the missions and the synthetic-data renders share lw-lab1. That is a sequencing decision, not a capacity assumption.