UBR Stack · team 15 · NVIDIA Open Models Codefest 2026
What the first three working days produced — 9 to 11 September. Nine public artefacts, three issues and two patches into NVIDIA's own repository, and a 4-bit reasoner running on the edge board that finds targets on real frames the incumbent misses. Several of the results overturned a claim of our own — including one in our own published dataset on day 3, which is the least comfortable and most useful kind.
Every row was produced under a written protocol with its N recorded. Nothing here is an estimate carried over from a previous week.
| What | Result | N / conditions | State |
|---|---|---|---|
| L0 · VLM bake-off, synthetic renders |
Cosmos3-Edge 47.1 % Gemma-4-E4B 33.6 % |
986 pairs + 456 MCQ bf16, zero-shot, one B300 |
Done |
| Distance calibration | k fitted per artefact bf16 1.155 · AWQ 1.21 |
held out, +6.9 pts (bf16) halves 1.148 / 1.161 AWQ far bucket needs 1.34 |
Done |
| Detector, real frames | false alarms on 3.2 % of frames |
2,223 real frames sim-trained model |
Done |
| Domain gap | all render sets at ~400× the noise floor |
real-vs-real MMD floor 0.000089 over 5 splits |
Done |
| Transfer 2.5 throughput | 173.3 s/clip · 37.6 GB 5.78 GPU-h, 0 failures |
120 clip-runs 40 clips × 3 strengths |
Done |
| Label survival through Transfer | ~13 % adjudicated 0 % keep the vest |
33 candidate boxes, human-checked |
Done |
| 4-bit edge artefact | 2.2 GB from 4.9 GB INT4 language, FP16 vision |
calibrated on 198 of our own rover frames |
Done |
| Same artefact, run on the edge board |
2.24 GiB of engines 3,377 MB LLM engine alone ~4,830 MB full VLM |
Orin Nano 8 GB (7,485 MB usable) 169 INT4 nodes · KV reserve 448 MiB quote the full-VLM figure |
Done |
| Edge decode throughput | 52.8 tok/s decode 2,306 tok/s prefill |
TensorRT engine, batch 1 build time 2 m 21 s |
Done |
| L1 · grounding on real frames |
Cosmos3-Edge 9–10 of 12 Gemma-4-E4B 0–1 of 12 |
24 items · 12 targets · IoU 0.5 same extraction both arms paired p = 0.0039 (9–0) · 0.0020 (10–0) |
Done |
| L1 · precision / recall | P 0.529 · R 0.750 · F1 0.621 | AWQ, thinking on, tolerant parser half the proposed boxes are wrong |
Done |
| Hard-negative false triggers | under re-measurement | re-scored under the production parser; powered run on the 474-frame slice pending | Pending |
| Schema adherence | 24 of 24 valid JSON (thinking on) 0 of 24 obey the contract |
contract recovered in code instead runtime has no grammar path |
Resolved |
| Head-to-head on the board | peak ~4,830 vs 5,642 MB 6.0–10.1 s vs 13.9 s per look |
same board, same method TensorRT vs llama.cpp — part of the speed is the runtime |
Done |
Both Cosmos paths can make a rendered scene photoreal. The question that decides whether either is usable for training data is narrower: after the repaint, does the annotation still describe the picture? We measured it frame by frame across a 93-frame clip.
So the conclusion is not "Transfer is better". It is that per-frame conditioning beats bigger models for holding a subject in place, and that Transfer's ceiling is a supply problem — the control signal covers too little of the frame. That is fixable by changing the control (depth and edge constrain the whole frame), and not by turning the strength dial, which is already at its ceiling.
| Artefact | Contents | Licence | |
|---|---|---|---|
| 4-bit edge reasoner INT4-AWQ |
INT4 AWQ + ONNX, calibrated on 198 of our own rover frames. Confirmed bring-up on an 8 GB Orin Nano — 52.8 tok/s, 2.24 GiB of engines, ~4,830 MB full-VLM peak (3,377 MB the LLM engine alone) on a 7,485 MB board. On real frames it finds 10 of 12 targets the incumbent misses. | OpenMDW 1.1 | Executed |
| The uncalibrated control NF4-bnb |
Naive post-training quantisation, published as the control so the AWQ claim has something to be measured against rather than asserted alone. | OpenMDW 1.1 | Published |
| Detector rescue-target-yolo26n |
Weights, two ONNX exports, configs, the full recipe — and its own ceiling-effect warning on the card, because the hold-out it was validated on cannot separate anything. | AGPL-3.0 | Published |
| Synthetic detection set isaac-sdg-rescue-target |
133,760 images across 37 shards, 109 clips with instance masks, merged COCO with 559,085 train annotations over 15 classes, per-image distance and bearing, provenance for 139 runs. Broken and repaired on day 3, then promoted — pinnable at revision="v3.0". | CC BY 4.0 (annotations) |
Repaired |
| Sim-to-real survival cosmos3-i2v-survival-sdg |
A negative result, published as one. The card explicitly refuses to read its n ≤ 13 cells as a curve — the point is the mechanism, not a trend line. | CC BY 4.0 | Published |
| Harness nvidia-codefest-2026 |
13 commits, ~3,450 lines. Frame selector, evaluation scripts, batch runners, the blackbox indexer. | Apache-2.0 | Published |
| Ada recipe edgellm-on-ada |
The full path to running Edge-LLM and a Cosmos3-Edge INT4 checkpoint on a consumer Ada card — a configuration the official matrix omits. Carries no NVIDIA source: the patch, the scripts, and the versions it was proven on. Deliberately disposable — see below. | Apache-2.0 | Published |
| Blackwell Ultra recipe edgellm-on-blackwell-ultra |
The same path for the B300 / Blackwell Ultra (SM103) — a datacentre part the official matrix omits. Carries no NVIDIA source: the patches (#205 + #207), the scripts, and the versions the INT4 and FP16 engines were proven on. Deliberately disposable — see below. | Apache-2.0 | Published |
| Frozen evaluation slices | 474-frame evaluation list plus a 256-frame calibration list, disjoint, seeded, with the selector that produced them. | Apache-2.0 | Published |
The most reusable thing on that list is probably none of the models. It is JETSON_ORIN_NANO_NOTES.md — seven staging traps, each found the hard way, three of which produce artefacts that pass onnx.checker while being silently broken. A passing validator is not a working export, and nothing we read beforehand said so.
Five filings across day 3, against NVIDIA/TensorRT-Edge-LLM. The patches are the smaller contributions. The three issues document things no patch of ours can fix, and the largest of them is about NVIDIA's own newest datacentre part.
| Item | What it contains | State |
|---|---|---|
| Issue #204 | The XQA NVRTC include-path bug, three further traps found on the same build, and the question of whether SM89 belongs in the support matrix at all. The three are a CUDA 12.8 floor that is nowhere stated, CUTLASS_DSL=OFF failing far downstream instead of at configure time, and ENABLE_CUTE_DSL silently rejecting a comma-separated list. | Open |
| PR #205 fixes #204 |
Passes the CUDA include path to the XQA NVRTC JIT. Two commits, DCO signed, clean under git-clang-format and cmake-format. Reviewed and approved the same day: the revised commit resolves vector_types.h through the roots FindNVRTC.cmake honours, with a warned fallback for a standalone NVRTC that ships no runtime headers. One of the two approving reviews the repository requires to merge; the approver is a community member, not a maintainer. | Approved 1 of 2 |
| Issue #206 | SM103 — the B300, Blackwell Ultra — is missing from five separate SM allowlists across the build and the attention path, so the INT4-AWQ path cannot be built for it. Each of the five lists neighbouring Blackwell parts and omits 103 alone, which reads as an oversight rather than a deliberate exclusion. Reproduced on main and on v0.10.1, on a B300 SXM6 at compute capability 10.3, CUDA 13.0, TensorRT 10.13.2 — against the same toolchain path verified working on SM89. | Open |
| PR #207 fixes #206 |
Enables SM103 across the attention SM allowlists. INT4 and FP16 engines verified on the B300 after the change. Cross-references #205, since both patches touch the same toolchain path. | Open |
| Issue #208 | The ONNX-less direct builder cannot resolve decoder tensors for the multimodal cosmos3_edge checkpoint — and renaming them to make it try yields a numerically wrong engine. Two findings in one: a resolution gap, and the numeric defect behind the obvious workaround. This is the experiment that would have separated "the export damages something" from "this is the model", and it is why that question is still open. | Open |
Why the issues outrank the patches. #205 fixes one thing. #204 documents four, and every one of them presents as something other than what it is — which is why each cost about an hour to find rather than a minute, and three of the four need no code change at all, only a sentence in the documentation. #206 is not ours to fix — a supported part the build refuses in five places is a matrix problem, not a configuration one — and #207 shows the fix is a handful of allowlist entries once someone decides to make it. #208 is the one we could not work around at all. Cheap for a maintainer to accept, expensive for the next team to rediscover.
Designed to become worthless
The value of edgellm-on-ada drops to near zero the day PR #205 merges, and edgellm-on-blackwell-ultra the day #207 does. That is the intended outcome, not a risk — and with one of #205's two required approvals in, it is closer than it was this morning.
A repository that exists to route around an upstream gap should die when the gap closes. Neither carries any NVIDIA source — only the patches, the scripts and the exact versions they were proven on — so there is nothing in them to maintain once the fixes are upstream. Publishing each workaround and its fix on the same day is what keeps it honest.
No adoption figures here. At three days old, a download count is noise in both directions — a high one would flatter us and a low one would mean nothing — so this is a judgement about usefulness rather than a measurement of uptake. Ranked, including the item we would tell someone to skip.
And one place the rule cost us something real. The 24 items that would settle the 4-bit schema question are frames of identifiable people at work, so they never leave the bench. What left was the metrics, the structural diagnostics, and the frozen prompt template with its stdlib-only scorer — enough to run the same comparison on anyone's images. Slower and less convincing than just sending the failing cases, and still the right trade: a rule that has never cost anything is not a rule.
None of these is a result. They are the rules under which the results were taken, and they are the part another team could adopt on Monday without downloading anything of ours.
| Practice | What it changes |
|---|---|
| Paired McNemar on every quantisation claim |
No quantisation delta is reported as a bare difference of two accuracies. Same items, same order, discordant counts b and c shown. It is what turned "AWQ 0.0 versus NF4 0.40" from a conclusion into a 3–0 discordance at p = 0.25 — which is to say, into nothing. |
| A published environment floor |
Environment-sliced scores are stated against a measured floor (87.9 % / 99.2 %) rather than against zero, so a high per-environment number is read as the increment it actually is. |
| Re-fitting k per precision |
The distance-calibration constant is re-fitted on each precision's own outputs, never carried over from bf16. A calibration constant fitted on one precision and applied to another quietly attributes the quantiser's error to the model. |
| A pre-upload gate on referential integrity |
Nothing reaches Hugging Face until five checks pass, the load-bearing one being that every metadata row's image actually exists inside a shard. Its absence is exactly what let a public release ship with annotations pointing at 15,528 files that were not there. This is the one habit on the list we learned by breaking something in public. |
| Correcting the chance baseline |
We had been scoring multiple-choice against 25 % chance by habit. The real figure for this question set is 14.22 % — the options are not always four. That single correction reclassifies Gemma's 12.50 % from "weak" to below chance, which is a different kind of statement and changes what the SFT run is even for. |
| Item | Where it stands | |
|---|---|---|
| Judge set | Capture protocol written and scheduled; supplies lying positives with measured distances. Blocks recall and range only — the model ranking moved to real mission frames. | Scheduled |
| Board headroom | Resolved by the one-model decision. The earlier split kept Cosmos and Gemma resident and did not fit — 8,406 MB against 7,485 usable. One model does: Cosmos alone peaks ~4,830 MB, ~2.6 GB clear on an idle board, ~0.9 GB once Whisper, Piper, the detector and the recorder are up (derived, not yet measured co-resident). The E2B fallback does not fit alongside the full voice stack — it would load and unload between phases if ever needed. | Resolved |
| Who confirms an open-vocabulary box | confirm_only is safe because the detector proposes and the VLM adjudicates. The open-vocabulary case — where Cosmos is meant to add value — inverts that: Cosmos proposes and nothing adjudicates, because the detector has no class for the thing. Precision 0.529 then lands straight on the robot's behaviour. Until something independent confirms, open-vocabulary hits are navigation hints, never reports: a false approach costs time, a false report costs the score and sends people to the wrong place. | Rule |
| 4-bit instruction following | Settled in the only way the runtime allows. The engine never obeys the contract (0 of 24 under every prompt tried, including a ROS 2 Detection2DArray it knew the name of and not the shape of), and Edge-LLM has no grammar or constrained-decoding path — structured output is "rejected rather than ignored". So the contract moved out of the model into a deterministic parser that recovers boxes and roles from the nested prompt the model perceives best under. Still open: the three-way full-precision / NF4 / AWQ run under one harness, to say formally whether quantisation touched this at all. | Resolved |
| Where that run happens | The 24 evaluation items are workplace frames of identifiable people. Our standing rule is derived frames only, nothing published, aggregate numbers alone leaving the bench — so the frames do not move. The prompt template and the stdlib-only scorer are both shareable, which means the same comparison can run on anyone's images, or on our own 16 GB card. Full precision needs ~9 GiB and will not fit the board either way. | Decision |
| Multi-control Transfer | Depth and edge weights already downloaded. The re-run is what turns the label-loss dead end into a fixed lever. | Queued |
| Rendered-output licence | Whether rendering with NVIDIA-supplied assets constrains the licence of published images. Unanswered in public; our cards now split it — annotations CC BY 4.0, renders under NVIDIA's terms with the assets not redistributed. | Asked |
| Closed-loop missions | Protocol and sizing settled; the renderer and the brain share one 16 GB card, so what the layer may claim was narrowed accordingly. | Sized |
The call to make before the next scrum
Two of the four ablation columns do not exist yet, and one of them depends on a lever we have just shown to be under-constrained.
The off-the-shelf reference is an afternoon's work and our sim-trained model is measured. The two remaining columns need Transfer output that preserves labels, and real training data. Better to narrow the claim now to off-the-shelf versus our sim-trained model on real frames — which is a complete result on its own — than to promise a four-column table and arrive with two.
And the memory call behind it is made. The edge reasoner finds targets on real frames the incumbent cannot, and running Cosmos and Gemma together to use that needs 8,406 MB of a 7,485 MB board — it does not fit. So Cosmos takes both jobs, perception and the operator interface, with Gemma kept as a text-only fallback that loads only between phases if ever needed: one model does what two could not. What is still owed before the scrum is the measurement under load — Cosmos alone peaks ~4,830 MB, which leaves roughly 0.9 GB clear once Whisper, Piper, the detector and the recorder are up, and that co-resident figure is derived, not yet measured.