UBR Stack · NVIDIA Open Models Codefest 2026 · working plan v3 · 12 Sep

Rescue-Brain Pipeline

The deliverable is an autonomous search-and-rescue robot: it searches, decides what to approach, and reports on its own — and keeps working when the link is cut. We build and prove it through a reusable data → train → edge → sim → field pipeline with a measured number at every gate, demonstrated on rescue and re-run on a second target to show the method transfers. Every week ends with a number that did not exist the week before.

How the organisers pick the GTC Berlin five — and where each criterion is answered
Progress
One gate metric per week, stage by stage
Relevance
Civil protection & SAR — a global public-sector problem, made sharply concrete by the 2026 disaster year
Uniqueness
Real rover, real judge set, an off-the-shelf baseline our own models must beat, a transfer test, and fixes sent upstream to NVIDIA's edge toolchain
Realistic use-case
Synthetic ≠ field accuracy; the physical rover is the final gate
Technology
Isaac Sim · Cosmos Transfer · Cosmos3-Edge · TAO · TensorRT Edge-LLM · RAPIDS · NeMo Curator
Why now — the 2026 field 2026 has been an exceptionally destructive year for exactly these hazards. Earthquakes killed about 7,100 people worldwide — Venezuela (M7.5, June, 6,500+), Colombia's Chocó (M7.4, August, 331), Indonesia's Flores (M7.7) and the Philippines' Mindanao (M7.8). In late August a glacier collapse above Langtang Lirung sent a flood across the Nepal–Tibet border: 1,385+ dead, 5,400+ missing, ~1.6 million affected. And Europe logged its fourth-worst wildfire season since 2012 — 489,826 ha burned by early August, Spain and France hardest hit, Portugal 47,776 ha — the same hazard behind our transfer test. Each is a ground search-and-rescue problem in comms- and GPS-degraded terrain, which is what this pipeline is built for.

MECHANISMHow data becomes a measured brain

SOURCES QC GATE B300 · TRAIN JETSON · EDGE EVALUATION Isaac Replicator renders 133,760 @ 1280×720 (v3) lw-lab1 (RTX) Rover blackbox ~108 h · 791 person segments real sensor domain, indoor NVIDIA PhysicalAI sets Spatial-QA · SmartSpaces negatives NuRec real rooms → sim scenes Judge set 474 real frames + mannequin capture never enters training Five layers hygiene · standby removal dedupe: pHash + NeMo Curator coverage table, empty cells label QA: FiftyOne + teacher CCDM-MMD / SDQM vs real RAPIDS for the joins every source, same gate subsets only, attribution logged Cosmos Transfer 2.5 orbit clips + masks → photoreal seg-only loses labels → multi-control Detector sweep B0/B1 → D1 → D2 → D3 off-the-shelf first, then ours Cosmos3-Edge — one brain perception + operator language SFT on the language side only, if needed Orin Nano 8 GB distilled nano detector Cosmos3-Edge, Edge-LLM INT4 ≤ 5 GB · ≥ 10 tok/s Sim harness — Isaac + Nav2 N = 20 placements across 8 buckets 20 paired placements per arm Real harness — marked floor same placements, mannequin cable pulled mid-run One scorer — SubT rule found ≤ 1 m / ≤ 5 m · time · false reports one scorer · sim and real read side by side labels frames subsets curated variants, labels re-checked distil engine brain same protocol same JSON log same JSON log ground truth only — to the scorer, never to training NuRec reconstructed rooms → benchmark scenes 2–3 (Isaac, lw-lab1) empty coverage cells and failures → new SDG runs (2–4) · transfer test T re-enters here with target = fire/smoke

Three sources plus subsets of NVIDIA PhysicalAI datasets pass a single QC gate before the B300 touches them; the judge set bypasses training entirely and feeds only the scorer. Sim and real harnesses run the same placements and write the same log to one scorer. For perception that makes the sim-to-real gap a subtraction — same model, different imagery. For missions it does not: the sim brain runs on a workstation GPU and the rover brain on an Orin, so we compare the two directly rather than treating one as the other's baseline. What the scorer finds missing goes back to lw-lab1 as the next render run.

EVIDENCEThe work behind each gate

Companion pages · what has been measured so far
Field notesThree Days In →What 9–11 September actually produced: measured results with their N, the findings that changed the plan, and what is not settled. BlockersBlocker Board →What is open, what it blocks, and what unblocks it — the honest state of the risks. GuidancePointers Followed →The five resources suggested at kickoff, and what came of each — what we committed to, what we're testing, and the one gap we own. Bake-offCosmos vs Gemma →The two edge VLMs on real frames: who perceives, who writes the contract, who decides. Grounded F1, restraint, runtime. MethodTest Ladder →How a model is judged: L0 synthetic, L1 real frames, L2 Isaac missions, L3 rover — read along the edges, never the diagonal. BaselineYOLO vs PeopleNet →The off-the-shelf detector reference our own model must beat — B0 (COCO person) and B1 (PeopleNet, TensorRT-ready) with a posture rule. DataDatasets →What we built and published, the sources we used, and the data challenges — closed and still open. CommunityContributions →Public models, datasets and code, and the issues and patches sent upstream to NVIDIA's edge toolchain.

STAGESFive gates, then one transfer test

1
Week 1

Dataset quality

Score the hybrid set before training anything: coverage (distance bins incl. past the 20 m wall — a body of a few dozen pixels at the detector's input resolution — pose classes with hard negatives, lighting, smoke), label consistency (do the masks and boxes still agree after Transfer?), and the sim-to-real gap: the same detector scored on sim and on real frames. Build the judge set: a mannequin capture on a taped floor supplies the lying positives with measured distances; the 474-frame past-mission slice supplies the negatives.
Gate metric
Coverage table · label-consistency % · off-the-shelf vs ours on real frames · recall per distance bucket · QC report structured by ISO/IEC 5259 dimensions
NVIDIA tech
Isaac ReplicatorCosmos Transfer 2.5RAPIDS cuDF
Replicator on lw-lab1 (RTX); Transfer on the B300. Measured: segmentation-only control is under-constrained — a single-instance mask covers ~1 % of the frame, labels survive ~13 % and 0 % keep the hi-vis vest. A static-subject control reaches ~0.90, so the mechanism works when adequately supplied: moving to multi-control (edge + seg + depth), not tuning strength.
2
Weeks 1–2

Train

Detector sweep: YOLO26n vs PeopleNet × 640 / 1280 × ±Transfer variants × ±hard-negative pseudo-labels (teacher = Cosmos3-Super or an open-vocabulary grounder — and treated as a noisy label, not ground truth: a prompt change alone swung its box count 3× on identical frames). Any bounded SFT of Cosmos3-Edge currently targets the operator-language side (tool-calls, an UNKNOWN reply); the perception false-alarm rate is under re-measurement on the 474-frame slice. Ablation against an off-the-shelf reference: B0 pretrained COCO person and B1 PeopleNet as shipped, each plus a fixed posture rule, then D1 sim-trained → D2 +Transfer → D3 +real. The headline claim is the step from B to D, not the steps among the Ds.
Gate metric
Confidence vs distance, median + IQR over N = 20 placements across 8 buckets, both models · false-alarm rate per threshold on the 474 real mission frames · COCO mAP · the posture rule's false-alarm rate on non-lying poses, with lying recall held. External: AI City test splits (SmartSpaces, Spatial-QA) + SARD lying-person recall
NVIDIA tech
TAONeMo / LoRAB300
Sweeps as background processes in one srun
3
Week 2

Edge

Distil the winning detector into the nano the rover actually runs. Build the TensorRT Edge-LLM INT4 engine for Cosmos3-Edge on the Orin Nano. Orin is FP16 / INT8 / INT4 only.
Gate metric
Met, 11 Sep. INT4-AWQ on the Orin: 2.24 GiB engines, ~4.83 GB full-VLM peak, 52.8 tok/s decode, 6.0 / 10.1 s per look — under ≤ 5 GB and 5× the ≥ 10 tok/s floor. Still open: watts, p95, co-resident memory with the voice stack, and the SM103↔Orin engine-equivalence gate.
NVIDIA tech
TensorRTTensorRT Edge-LLMJetPack 7.2
4
Week 3

Sim rover

Closed-loop mission in Isaac Sim on lw-lab1: briefing → search → detect → verify → mark → re-plan → report, with scripted comms loss. N seeded scenarios. This is where the agent is measured, not only the models — and three of its decisions get their own numbers: decide to approach (a casualty in view that the rover chose not to close on), decide to report and the harder decide not to, and behaviour under comms loss. The brain is decided (11 Sep): Cosmos3-Edge does both perception and the operator interface; geometry stays deterministic in code, with a stop matcher ahead of the model. The mission harness runs, with one scored shakedown so far.
Gate metric
SubT rule: found within 1 m (ours) and 5 m of ground truth — the report's position, not the rover's · time to report · false reports · VLM calls · distance driven · Success / SPL / distance-to-goal (ObjectNav). N = 20 paired placements per arm; McNemar on the binary metrics, Wilcoxon on the continuous ones, one primary metric declared before the runs; median + IQR, bootstrap CI. Baseline today: one mission, 822 s, 56 VLM looks
NVIDIA tech
Isaac Sim 6.0Nav2
Depends on the planner track's closed loop
5
Weeks 3–4

Physical rover

Same mission on the bench with a mannequin. The network cable is pulled mid-run: the rover keeps searching; the field report arrives on reconnect. Real rover-camera frames are the final generalisation gate for every model.
Gate metric
Same metrics as stage 4 → the mission sim-to-real gap · report on reconnect · video · ISO 18646-2 navigation tests on the same floor (pose repeatability, obstacle detection/avoidance, path deviation) · E2853-style scoring sheet
NVIDIA tech
Jetson Orin Nano 8 GB
T
Week 4 · 3 days

Transfer test

Re-run stages 1–4 on a second target with the same rover, sensors, scenes and tools: early fire / smoke ignition — UBR's founding problem. Only the target class and the assess_detection prompt change. No physical-rover stage. Capped at one column and three days, with no scope growth — it is the reusability proof, not a second project. What the cap frees goes into stage 4/5 autonomy.
Gate metric
Second column of the same table: same pipeline, new target, X days, judge-set P/R Y
NVIDIA tech
Identical stack — that is the point

COMPARABILITYWhat each number is compared against outside the lab

LayerExternal referenceStatusWe report
NavigationISO 18646-2:2024 — service-robot navigation test methodsrun as specifiedpose repeatability, obstacle detection/avoidance, path deviation — rover and sim, same geometry
MissionDARPA SubT scoring rule + ObjectNav metrics (Anderson et al. 2018, Habitat)aligned metricsfound ≤ 1 m / ≤ 5 m, false reports, time to report; Success, SPL, distance-to-goal
Physical apparatusASTM E2853 search tasks / NIST response-robot methods (RoboCup Rescue, ERL Emergency)in the spirit ofscaled search lane, mannequin as simulated victim, E2853-style scoring sheet — no certification claim
PerceptionCOCO mAP; NVIDIA AI City Challenge test splits (SmartSpaces 3D HOTA, Spatial-QA accuracy); SARDaligned splitsmAP@0.5 / 0.5:0.95 on hold-out; scores on public splits; SARD lying-person recall per size bucket
Edge runtimeMLPerf-style reportingin the spirit oftok/s, TTFT, p50/p95 latency, peak memory, watts at fixed precision
Dataset qualityISO/IEC 5259 series — data quality for analytics and MLreport structurecompleteness, accuracy, uniqueness, representativeness (CCDM-MMD / SDQM), provenance

No single standard covers "ground rover finds a prone casualty indoors". We run what can be run, report in the format of the rest, and never say "certified".

RULESWhat keeps the numbers honest

  • Never train on the 60 frozen frames or on the judge set. New training and validation sets for every experiment.
  • Real rover-camera frames are the final generalisation gate for every model. Synthetic results are diagnostics, not field accuracy.
  • Ablation by source is mandatory — sim-only / sim+Transfer / sim+Transfer+real. It is the proof that the hybrid dataset earned its place — the kind of result judges remember.
  • The transfer test must not touch the rover code path. New tools or new sensors means it is not a transfer test, and it is out of scope.
  • The baseline is measured, not remembered — and it is not ours. B0 (pretrained COCO person) and B1 (PeopleNet as shipped) are the reference; our sim-trained model is D1, a result rather than a baseline. If B1 + a ten-line posture rule matches D1 on real frames, the synthetic pipeline earned nothing — and we would rather find that out ourselves.
  • The gates are a reporting order, not a build order. The whole path — frame → detector → VLM → decision → report → scorer — is stood up end to end on ten frames, one sim mission and one rover run before any gate is filled. A gate's number may be a placeholder; the code path it runs through may not.
  • No claim below the noise floor. Two runs of the same model in different containers agree on ~95% of items (88% on distance), so a smaller gap is not evidence. And a worked example shares no content word with any test item — three contaminations were caught exactly this way.
  • Keep the agent in front. Detector numbers feed assess_detection; the demo is still the search → verify → comms-cut → report loop.
  • The decisions are the product. Approach, report, and above all not report, each measured per mission. Any SFT budget now goes to the operator-language side — tool-calls, an UNKNOWN reply, and multilingual capability (demonstrated in European Portuguese) — since 4-bit perception costs nothing once calibrated and the model is no longer asked to emit the contract.

DEPENDENCIESTo confirm this week

Planner track: closed loop in Isaac — running
The L2 mission harness runs on lw-lab1 and has scored one shakedown mission; the paired comparison has not been run. One open call: run arm C's paired placements on jetson1, or accept the workstation engine behind an SM89 equivalence gate.
Answered — decision on the engine
Mannequin capture for the judge set — scheduled for Day 4
It supplies the lying positives with measured distances — lying recall per distance bucket. With the capture on Day 4, those numbers arrive late: until then the 474-frame slice carries the false-alarm side, and the earlier ablation is detection and false-alarm rate, not lying recall by distance.
Day 4 · 30 Sep
lw-lab1 is double-booked — measured render-bound
One RTX 4060 Ti 16 GB serves both the render backlog and the missions, and Isaac needs RT cores so neither can move to the B300. Measured: missions run at rtf 0.34 and staging costs ~4× a mission, so placement ordering, not the model, drives the schedule.
Sequence, do not assume
Licence of Isaac-rendered output
Whether rendering with NVIDIA SimReady assets constrains the licence of the published images. The one public forum thread asking this was closed with "contact Enterprise Sales", so it needs an answer from inside NVIDIA.
Asked 11 Sep

CALENDARSessions and exit criteria (Lisbon time)

WhenSessionStageExit criterion
Wed 09 SepDay 1 kickoff — done—Live GCS + rover demo to mentors; cluster access, 1× B300
09 – 16 Sepasync1 → 2Judge set captured; B0/B1 off-the-shelf reference measured on real frames; coverage table; Transfer strength swept and its limits published; edge INT4 artefact built; 10 edge tools stubbed
Wed 16 Sep 09:00–16:00Day 2 scrum2Ablation table B0/B1 → D1 on real frames; the edge brain decided with its INT4 numbers in hand; briefing → edge agent → field report on a recorded scene
16 – 23 Sepasync3 → 4Edge INT4 gate already met (see stage 3); this week: SM103↔Orin engine-equivalence gate, k per range bucket on the Orin, co-resident memory, and the 20-placement L2 comparison
Wed 23 Sep 09:00–16:00Day 3 scrum4 + 5Closed-loop mission in Isaac (N scenarios, median/IQR); bench mission with cable pull
23 – 30 SepasyncTSame pipeline on fire/smoke ignition, stages 1–4, three days; second column of the table
Wed 30 Sep 09:00–16:00Day 4 scrumfreezeDry run < 5 min; backup video; both columns of the gate table filled
Wed 07 Oct 10:00–16:00Day 5 final—Present
Tue 20 OctGTC Berlin—If top 5
UB Robotics · team UBR Stack · NVIDIA Open Models Codefest 2026
working plan v3 · 12 Sep