UBR Stack · team 15 · NVIDIA Open Models Codefest 2026

Edge VLM Bake-off

Which vision-language model belongs on the rover — and in which seat. Two layers of evidence: L0, 986 synthetic pairs at bf16 on a B300, and L1, 24 real frames at 4-bit on the Orin Nano that ships. They agree on the ranking and disagree on what it means: the two models are good at disjoint halves of the job.

L0 9 Sep · bf16 · B300 · n = 986 + 456 L1 10–11 Sep · INT4 · Orin Nano 8 GB · n = 24 Revised 11 Sep

Verdict

Cosmos perceives — and, it turns out, decides

The 9 September version said "swap the rover to Cosmos3-Edge". The 10 September version split the roles — Cosmos perception, Gemma the brain. The measurement on 11 September closed that split the other way: given a conversational prompt with thinking on, Cosmos interprets operator commands at 13/15 with 5/5 on stop synonyms, so one model now does both jobs. Geometry stays deterministic in code; Gemma, as a text-only E2B, is the measured fallback if conversation or tooling need a fine-tune.

nvidia / Cosmos3-Edge · INT4-AWQ · TensorRT

perception + operator interface

9–10 of 12targets on real frames

Flat across three IoU thresholds, false-alarm rate under re-measurement, 6.0 s per look (10.1 s with thinking on) at ~4.8 GB. Never emits the output schema — the contract lives in a parser, and the geometry in code. With a conversational prompt and thinking on it also reads operator commands: 13/15, 5/5 stop synonyms.

google / Gemma 4 E4B · Q4_0 · llama.cpp

measured fallback

0–1 of 12targets on real frames

Writes the contract almost perfectly (schema validity 0.958) and cannot point: 0–1 of 12 targets, boxes in the right region and the wrong shape. 13.9 s per look at 5.6 GB. Kept as the measured text-only E2B fallback.

The earlier split would have needed both models resident — 8,406 MB of 7,485 usable. One model does not: Cosmos alone peaks ~4.8 GB, leaving ~2.6 GB on an idle board (~0.9 GB once the voice stack is up). The residency problem is resolved by the decision, not worked around. Precision is 0.529, so a second confirmation gates every approach — that problem is assigned, not solved.

L1 · real frames, on the board

The two models fail in opposite ways

Benchmark ugv-grounded-reasoning-v0: 24 items, 12 scored target instances, 14 hard negatives, deliberately skewed to the rescue target — 14 of 24 ask for a person lying down in a hi-vis vest. Same prompt, same scorer, same board for every arm. The frames are a workplace recording of people; only aggregate numbers leave the bench.

Headline · Orin Nano 8 GB · deployment precision on both sides
Model	Schema validity	Targets @ IoU 0.5	False triggers	Per look	Peak resident
Cosmos3-Edge INT4-AWQ · thinking on	0.0	9–10 of 12	under re-measurement	10.1 s	~4,830 MB
Gemma 4 E4B Q4_0 · llama.cpp	0.958	0–1 of 12	under re-measurement	13.9 s	5,642 MB

9 of 12 is recall through the tolerant parser (precision 0.529, F1 0.621); 10 of 12 is geometry-only, any box matched at IoU 0.5, identical extraction for both models. Paired McNemar on targets: 9–0 discordance p = 0.0039, 10–0 p = 0.0020.

Cosmos3-Edge, thinking on Gemma 4 at 1280×720 Gemma 4 at 640×360

The curve shapes are the diagnosis. Gemma collapses 4 → 2 → 1 → 0 as the threshold tightens: right region, wrong shape. Cosmos is flat at 10 across three thresholds — accurate boxes, not marginal ones scraping a gate. Its recognition is not the problem on either side; Gemma named the parcel, the office chair and the solar panel correctly and still could not box them.

Every arm, strict scorer

Why every Cosmos row reads 0.0 — the strict scorer gates on the schema
Arm	Runtime	Thinking	Strict F1	Targets	Schema validity
NF4, transformers (control)	transformers	off	0.40	3 / 12	0.125
NF4, transformers	transformers	on	0.0	0 / 12	0.0
AWQ-INT4, Edge-LLM	TensorRT v0.10.1	off	0.0	0 / 12	0.0
AWQ-INT4, Edge-LLM	TensorRT v0.10.1	on	0.0	0 / 12	0.0
Gemma 4 E4B, 640×360	llama.cpp b9602	off	0.0	0 / 12	0.958
Gemma 4 E4B, 1280×720	llama.cpp b9602	off	0.074	1 / 12	0.875

Scored on geometry alone, the same rows read 7, 1, 8, 10 — and with a parser that respects the model's own target call and recovers entities from JSON that json.loads rejects, the AWQ thinking-on arm reaches precision 0.529, recall 0.750, F1 0.621. The strict metric was measuring the contract, not the perception underneath it.

The drawback

It will not write our schema, and the runtime cannot make it

Zero of 24 valid outputs, under every condition tried. This is the one thing between a model that finds 9 of 12 targets and a usable perception backend, so it was worked to the end.

Everything tried, in order — every rewrite bought format and paid in detections
Attempt	Format	Targets	What it showed
Reasoning on	24 / 24 parse	10 / 12	Fixed every syntax slip — parseable outputs went 11 → 24 — and fixed the contract not at all. The two failures are separable, and only the syntax one was ours.
Syntax repair + default the omitted fields	1 / 24 valid	—	A lenient parser does not rescue a strict metric.
Flat, three fixed box slots	24 / 24	0 / 12	Mandatory empty slots induced confabulation — BOX1=1,2,3,4 · BOX2=5,6,7 — rather than making omissions visible.
Flat, one slot, yes/no gate first	24 / 24	0 / 12	FOUND=yes fell to 2 of 24 when the gate preceded the evidence.
Flat + worked example	21 / 24	0 / 12	Worse than the nested original, and a first attempt was contaminated by an example near one ground truth — re-run clean.
ROS 2 Detection2DArray	24 / 24	0 / 12	Emitted a detections: block every time, wrote center: {x, y} where the message has center.position.x, found nothing. Knowing a format is not being able to fill it.
Grammar / constrained decoding	—	—	Unavailable. Edge-LLM documents structured output as "rejected rather than ignored". logitBias reaches out-of-enum values only — not an omission, not geometry.
Nested prompt + tolerant parser	0 / 24 valid	9 / 12	Keep the prompt it perceives best under; recover each bbox_2d with the nearest following role in code. The best result on the benchmark.

Why this is the correct layering, not a workaround

The model was never the right place for the contract. A deterministic converter cannot omit a field, cannot misindent, cannot invent an out-of-enum value.

The verbose nested prompt is doing perceptual work, not formatting work — every terse format stripped it and lost the detections with it. Two mechanisms fit the runs: generation runway, where writing the entity around the box lets the model talk itself into the object before committing coordinates (thinking-on lifted detections 8 → 10); and parameterisation, since ROS had ample runway and failed, and the thing it does differently is centre-plus-size instead of corners. One experiment separates them: the nested prompt with centre-plus-size coordinates and nothing else changed. It has not been run.

It also answers the ROS question properly. The perception node emits Detection2DArray from code, from boxes the parser extracted. The output really is ROS 2; the model never has to know.

L0 · synthetic renders, bf16, one B300

The first layer, kept for what it still shows

986 validation pairs and 456 multiple-choice items on a warehouse benchmark, zero-shot, reasoning off, referenced regions drawn onto the image as numbered outlines. It ranked the models the same way L1 does, and its distance finding still stands. Two of its numbers do not, and are corrected below.

Cosmos3-Edge Gemma 4 E4B chance

Chance is drawn per task. Corrected 11 Sep: multiple-choice chance was drawn at 25 % by habit on the first version of this page and at 12.5 % on the second; the options are not always four, and the real figure for this item set is 14.22 %. That moves Gemma's 12.50 % from "at chance" to below chance. Cosmos clears chance on every task, by only 3.5 points on multiple choice.

82 % one-sided is a systematic offset. 45 % is a coin flip around the truth — noise, not bias.

Fitted on one random half, evaluated on the other, both directions; the two constants agreed (1.148 and 1.161). The same treatment makes Gemma worse, which is what noise does. k is re-fitted per precision — the bf16 constant is not carried onto the 4-bit engine. Percentages are over the 449 items where both distance values parse, not the 486 of the chart above.

Two L0 numbers withdrawn

Corrections to the 9 September version of this page
Then	Now	Why
43.9 tok/s on the B300, "2.3× the incumbent"	Withdrawn. The board's own figures stand instead: 52.8 tok/s decode at ~4.8 GB full-VLM peak (3.4 GB the LLM engine alone).	The same model at the same precision on the same B300 measured 76.17 tok/s in a different container — 1.7× apart from Python-side overhead alone. There was no reference to be unlike.
"Gemma 4 keeps the brain; Cosmos takes only perception"	Superseded 11 Sep: Cosmos takes both jobs. Gemma becomes a text-only E2B fallback.	The "Gemma 4 stays" call rested on tool-calling (21/21 vs 0/21) and the Para!→Go! inversion. PT-PT is now out of scope, and a fair re-run (conversational prompt, thinking on) reads English commands 13/15 with 5/5 stop synonyms. Tool-calling is unmeasured under that prompt — the one leg left to re-run.

Before a live mission

Four conditions, one of which is not yet met

One model, so the board fits — with the voice stack, only just. The earlier two-model split needed 8,406 MB of 7,485 usable and did not fit. One model does: Cosmos alone peaks ~4.8 GB, ~2.6 GB clear on an idle board. Once Whisper, Piper, the detector and the recorder are up, that headroom falls to ~0.9 GB (derived, not yet measured co-resident), so it is comfortable, not roomy. The E2B fallback, if ever needed, does not fit alongside the full voice stack — it would have to load and unload between phases.
Open-vocabulary hits are navigation hints, never reports. confirm_only is safe because the detector proposes and the VLM adjudicates. Where Cosmos adds value — the open-vocabulary case — that inverts: Cosmos proposes and nothing adjudicates, because the detector has no class for the thing. Precision 0.529 then lands straight on the robot. A false approach costs time; a false report costs the score and sends people to the wrong place. Until something independent confirms, the box says go and look, not found.
Thinking stays on, per request. It is what takes detections from 8 to 10, and it is a per-request flag. A look costs ~10.1 s with it, 6.0 s without — at decision points, never in a servo loop, and still cheaper than the 13.9 s Gemma look it would sit beside. Do not overwrite generation_prompt to force it; that destroys the per-request choice.
n = 24, one benchmark, never run in a live loop. The detection result clears p = 0.004 and is not noise. Restraint on hard negatives — the property a confirmer would actually be deployed for — is under re-measurement, which makes the 40-hard-negative capture the cheapest thing left to run. Everything here is bench evidence.