# VECTOR × HyperCity: final adversarial system audit
| | |
|---|---|
| date | 2026-10-25 |
| branch | `claude/vector-extremis-forensic-audit-f5vtxg` (`main` untouched) |
| protocol frozen | commit `63bc12a`, before any held-out seed ran (`AUDIT_PROTOCOL.md`) |
| held-out evaluation | commit `8c72db6`: 800 of 800 trials completed, 0 failed |
| post-hoc diagnosis | commit `8558862` and later, fresh seeds only (§5.7, `DIAGNOSIS.md`) |
| simulator | SUMO 1.28.0 through TraCI, 1 s step |
| **audit completeness** | **COMPLETE.** All six acceptance gates A–F pass (§6). |
| **system performance verdict** | **FAIL against the strongest baseline.** VECTOR is not better than SUMO's native actuated control in any of the 16 scenarios, and it is significantly worse in B after Holm correction. It is better than fixed-time control in 15 of 16 scenarios. Against a simple vehicle-actuated rule it is comparable to better, with no Holm rejection. It gives emergency vehicles large, consistent gains. No signal-safety constraint was violated in any trial, under the harness's external interlock. |
These two verdicts are separate. A complete audit is not a passing system: the
runner finishing tells you nothing about VECTOR.
> This is simulation and only simulation. Nothing here establishes real-world
> safety, regulatory compliance, deployment readiness or operational
> effectiveness. Nothing here supports connecting VECTOR to live public
> traffic signals (§9).
## 1. Executive summary
**The question.** Can VECTOR, as implemented, control a connected
36-intersection simulated city? How does it compare with credible non-VECTOR
control under adversarial conditions? The comparison uses paired seeds, with
every failure kept.
**What was run.**
* The network: `city36`, a 6 × 6 SUMO grid. It has **36 connected,
signal-controlled intersections**, validated from the network file and
pinned by canonical hash.
* The arms, five in total:
* VECTOR with its advisory shadow stack;
* VECTOR without the shadow;
* SUMO's native actuated programme, gap-out logic with matched timing (the
strongest baseline available);
* a simple vehicle-actuated rule controller;
* 30 s fixed-time control.
* The scenarios: 16 (R0 and A–O).
* The seeds: 10 held-out seeds (95000–95009), run once after the protocol was
frozen.
* The trials: 800 held-out trials, all completed, plus 105 smoke trials and
200 post-hoc diagnostic trials on fresh seeds.
**What VECTOR is, in the execution path.** A per-junction rule-based control
law, `control_logic.decide_signals`. It has four rules:
1. emergency override;
2. no-demand rest;
3. starvation guard at 90 s red;
4. vehicle-count priority.
A rate-bounded queue observer filters its input, and a separate safety layer
enforces its timing.
The forecasting and tiered-intelligence stack runs only **in shadow**, with no
path to the decision. This is verified on every paired trial.
Many named modules are **not executed in the SUMO loop at all**:
* routing, coordination and perception;
* the Sentinel, Serpentine, Aegis, Atlas, Punisher, Phoenix, Hyperion,
Terminus, Anarchy, ORACLE and PULSAR stack;
* the learned models;
* the `SignalSafetyGovernor`.
A "Decision Memory Engine" does not exist in the repository (§2).
**Headline results, held-out, paired by seed.** "Better" means lower P1, the
mean time loss per requested ordinary vehicle.
| comparison | result |
|---|---|
| VECTOR vs **native actuated** (primary, Holm over 16) | Never resolved better. **Worse in B: +73.6 s/veh [26.6, 120.5], Holm p = 0.031.** Unadjusted, also worse in C, D, H, J, K and O (+29 to +111 s/veh). Not resolved in the other 9. |
| VECTOR vs **vehicle-actuated** (secondary, Holm over 16) | No Holm rejection; the smallest is p_holm = 0.0625. Unadjusted, better in 7 scenarios (−20 to −127 s/veh) and worse in K (+35.6 [6.2, 64.9]). |
| VECTOR vs **fixed-time 30 s** (exploratory) | Better in 15 of 16 (−84 to −218 s/veh). J is not resolved. |
| **emergency vehicles** vs native | Ambulance −42 to −61 s, fire engine −65 to −138 s, police −39 s per trip, all with CIs excluding 0. The exceptions are in saturated K: the ambulance is not resolved, and the police car arrives in only 5 of 10 runs under VECTOR (7 of 10 under native). |
| **safety** | 0 timing violations, 0 illegal signal states and 0 invalid requests in all 640 controlled trials. 0 collisions and 0 teleports under VECTOR. One baseline trial had a SUMO rear-end collision (§4.6). |
| **runtime** | VECTOR decide p95 2.0 ms, p99 2.8 ms, max 225 ms. The 36-junction cycle p95 is 24 ms, against a 1000 ms step. The shadow causes almost all of it: without the shadow, decide p95 is 0.06 ms. |
| **prediction (shadow)** | MAE is worse than persistence in 15 of 16 scenarios, and on every seed of each. Error doubles after the concept shift in H (1.25 → 2.60 vehicles). No model influences control. |
**Why VECTOR loses where it loses** (§5). Across the 160 paired held-out
cells, VECTOR's excess time loss over native tracks its excess *ineffective
green* (r = 0.91) and *wasted green* (r = 0.92). Ineffective green is green on
a queue that is not discharging.
The law counts halting vehicles as demand. A queue behind a blocked
downstream link therefore looks like demand, and VECTOR keeps serving it until
max green. Native gap-out logic sees no vehicles crossing its detectors and
moves on. The law has no downstream or discharge term, and no junction knows
anything about its neighbours.
A post-hoc diagnosis on fresh seeds (§5.7) tests whether the result repeats
and which VECTOR mechanism contributes: on 10 fresh seeds, VECTOR is again worse than
native in all five re-tested scenarios.
* R0: +21 s/veh [4, 38].
* J: +67 [17, 117].
* K: +116 [93, 140].
* C: +18 [−2, 38], borderline.
* B: +24 [−23, 70], not resolved. The held-out B effect (+74) did not
replicate in size, so its Holm rejection stands as the pre-registered
result, but its magnitude is probably overstated.
VECTOR's two mechanisms trade off:
* Green commitment costs about 15 s/veh at moderate demand (R0, C), but it is
worth 65–103 s/veh under saturation (J, K).
* The queue observer is worth 14–37 s/veh.
No ablation closes the gap to native under congestion. That places the
deficit in the control law itself, not in either mechanism.
**Pilot readiness.** The evidence supports **neither connection to live
signals nor an uncontrolled field trial.** The defensible next step is an
offline, shadow-only research collaboration on a partner's historical data
(§9).
## 2. System architecture: what the code actually runs
### 2.1 Verified execution path
Every statement below was checked against the code and against a real trial.
The checks were:
* `docs/architecture/DATAFLOW.md` (verified map, 331 architecture tests);
* `artifacts/final-adversarial-audit/integrity/module_inventory.json`. This
file comes from one real 36-junction VECTOR trial, run with call counters on
the control path and a record of every module loaded.
```mermaid
flowchart TB
SUMO[("SUMO 1.28.0 · city36 · 36 signalised junctions · 1 s step")]
SUMO -- "halting vehicles per approach lane
(end of the previous step)" --> SENS["audit sensor model
(only scenarios G, L)"]
SENS --> OBS["Observation, one per junction:
queues NS/EW, phase, time in phase,
time since green, emergency view, stale flag"]
SUMO -- "emergency vehicle on an approach lane
(simulator ground truth, confidence 0.95)" --> OBS
OBS --> EC["ExtremisCanonical adapter
(drops the stale flag)"]
EC --> QO["QueueObserver
queue belief, ±0.5 veh/s per step"]
QO --> LAW["control_logic.decide_signals
1 emergency · 2 no demand · 3 starvation (red ≥ 90 s) · 4 vehicle count"]
TT["SignalTimingTracker"] --> LAW
LAW -- "NS or EW" --> SA["SignalAdapter (harness safety layer)
min 8 s · max 45 s · yellow 3 s · all-red 2 s"]
SA -- "setRedYellowGreenState" --> SUMO
SUMO -- "reported state must equal commanded,
else the trial fails" --> IL["IntervalLog
(violations rebuilt from SUMO's reports)"]
LAW -. "after the phase is settled" .-> ORCH["IntelligenceOrchestrator (shadow)
StateEstimator → SENTRY/PREDATOR/APEX,
5 forecasters, ShadowLedger, Counterfactual"]
```
There is **no edge from the shadow back to the decision**:
* in the code, `canonical_controller.decide` runs the shadow after `phase` is
final;
* in the data, the signal sequence with and without the shadow is identical
in every paired held-out trial (§4.7).
**Not on the path at all.** These modules are not even *imported* during a
36-junction VECTOR trial:
* Sentinel;
* Serpentine (protocol and routing);
* Aegis, Atlas, Punisher, Phoenix, Hyperion, Terminus;
* Anarchy (ORACLE, PULSAR, its Fusion Engine);
* the perception Fusion Engine and the camera/detector path;
* the `SignalSafetyGovernor`;
* the MORL/GNN/causal learners;
* temporal (sequence) inference.
A **"Decision Memory Engine" does not exist** in the repository. The protocol
stack runs only under `run_extremis.py`, with synthetic perception and a
simulated controller. It never runs against SUMO.
### 2.2 Module table
| module | runs in the live loop? | output reaches a decision? | logged? | calls in the inventory trial (600 s, 36 junctions) | on invalid, late or missing output |
|---|---|---|---|---|---|
| `QueueObserver` (state estimate) | yes | yes: the law reads its belief, not the raw count | yes, `belief` in the trace | 19 274 updates (2 per decision) | an impossible change is clamped to ±0.5 veh/s per step; NaN and negatives are mapped to 0 upstream (`_lane_count`) |
| `control_logic.decide_signals` (decision) | yes | **is** the decision | yes, `reason` | 9 637 | `INVALID_STATE` → all-red proposal, mapped to the current approach by the adapter (defect D6) |
| `SignalTimingTracker` | yes | yes (red times for rule 3) | partly (snapshot) | every decision | — |
| `SignalAdapter` (harness) | yes | enforces timing on the decision | yes, `effect` | 9 637 applies | an invalid request is replaced by the current approach and counted |
| `IntelligenceOrchestrator` + `StateEstimator` | yes (shadow, arm `vector`) | **no** | yes, `shadow` | 9 637 | an exception is caught, counted as `shadow_failures`, and control continues (scenario M); forecasters are skipped once a 50 ms wall-clock budget is spent, so its output depends on machine load |
| forecasters: persistence, physics, Kalman, spectral | yes (shadow) | **no** | the mean only | 9 637 / 3 103 / 3 103 / 3 103 | an exception is recorded in the registry and the model is skipped; the stack still reports itself healthy (§5.9) |
| autoregressive forecaster | **never called** (registered, not invoked) | no | — | 0 | — |
| `ShadowLedger` (prediction verification) | yes (shadow) | no | `ledger_verdicts_by_junction` | 9 637 settle / 9 637 record | — |
| `TrainingGate` | constructed | holds every model at UNTRAINED: nothing is authorised | — | — | — |
| `IncidentView` (INC1) | loaded, inert (production passes no incident source) | no | — | 0 | — |
| emergency detection | the harness reads SUMO ground truth | yes (rule 1) | `ev` | — | one emergency per junction: the input contract has a single slot |
| routing / rerouting | **absent** | — | route changes counted from SUMO's `vehroute` output | 0 | — |
| coordination between junctions | **absent**: 36 independent instances, no shared state, no offsets | — | — | — | — |
A module that only writes a log entry is not counted as integrated. Only the
`QueueObserver`, `decide_signals`, the timing tracker and the emergency input
influence what SUMO executes. Everything in the shadow is integrated as a
measured observer, not as a controller.
### 2.3 What a "control cycle" is here
A **traffic pass** is one SUMO step of 1 s. In each step, for each of the 36
junctions in fixed order:
1. If the junction's `SignalAdapter.decision_due()` is true, the harness:
* builds that junction's `Observation`;
* calls its controller;
* applies the result through the adapter.
A decision is due when the minimum green has elapsed (then every second
while green) or at the end of clearance.
2. The commanded state is set.
3. SUMO advances.
4. The reported states are checked and the measurements are taken.
**Modules cooperate only within one junction's decision:** observer → law →
adapter (and, in shadow, the orchestrator). No module combines information
across junctions. The *cycle latency* in §4.5 is the wall time of step 1 for
all 36 junctions together. One recorded cycle, stage by stage, is evidence
E18.
## 3. Experimental methodology
| item | value |
|---|---|
| simulator | SUMO 1.28.0 (`eclipse-sumo` wheel), TraCI, step 1 s, teleporting disabled |
| network | `city36`: 6 × 6 grid, 36 signalised junctions, 4 two-lane arterials, 8 one-lane side streets, 180–260 m links, 24 fringe nodes; static and actuated variants pinned by canonical hash |
| demand | Poisson per second and OD route (552 routes); rates by OD class, calibrated with the baselines only; evaluation at 1.15 × base; demand 0–2100 s, drain to 2700 s |
| scenarios | 16 (R0, A–O), `scenario_matrix.json` |
| arms | VECTOR (+ shadow), VECTOR without shadow, native actuated, vehicle-actuated, fixed-time 30 s |
| seeds | calibration 90000–90001 (baselines only); smoke 91000–91001; **held-out 95000–95009**, run once after the freeze (`63bc12a`); diagnostic 96000–96009 (post-hoc) |
| trials | 16 × 5 × 10 = 800 held-out trials (plus 105 smoke, 200 diagnostic) |
| primary endpoint | mean time loss over all requested ordinary vehicles; unfinished trips are charged their loss so far |
| statistics | paired by seed; 95% t and bootstrap intervals; Wilcoxon signed-rank; Holm over 16 scenarios in two named families |
| integrity checks | harness ≡ validated runner (VECTOR and native, same seed); shadow non-influence (every paired trial); health gate per trial (accounting, teleports, SUMO state = commanded state, timing violations) |
| reproduction | §10 and `AUDIT_PROTOCOL.md` |
### 3.1 Network
The network comes from `topology_report.json`, which is validated from the
SUMO file, not assumed:
| check | value |
|---|---|
| junctions in the file | 60: 36 traffic-light junctions and 24 dead-end fringe nodes |
| signal-controlled intersections | **36** (C00–C55) |
| connected | strongly connected between the controlled junctions; each one is reachable from every entry |
| roads, lanes and links | 168 directed edges, 224 lanes, 480 controlled links |
| routes | 552 OD routes, all legal (528 with turns) |
| measurement | 192 stop-line detectors |
| validation problems | 0 |
The network is a synthetic regular grid, chosen because no real 30+ junction
network with real demand was available. It is not a model of any city.
Evidence E01 (the whole grid under VECTOR) and E02 (the topology validation)
show it.
### 3.2 Baselines
No baseline was written to make VECTOR look better.
* **Native actuated** is SUMO's own actuated programme: gap-out on its own
detectors, with min and max green matched to VECTOR's adapter. It is the
strongest controller here.
* **Vehicle-actuated** and **fixed-time 30 s** are the repository's existing
rule controllers. They run through the same `SignalAdapter`, the same
harness and the same safety checks as VECTOR.
The harness was checked against the validated runner (`stus/sumo/multi`). On
seed 91001, VECTOR and native give identical results on 13 compared metrics
(`integrity/equivalence_91001.json`).
### 3.3 Scenarios
All windows are in simulated seconds. Demand runs from 0 to 2100 s at 1.15 ×
base, and every trial ends at 2700 s. Emergency vehicles are named `amb1`,
`fire1` and `pol1`.
| id | what is injected |
|---|---|
| R0 | reference: base demand, no event |
| A | demand surge × 4 from W1 and E1 (row-1 corridor), 600–1500 |
| B | three simultaneous congestion events: lane blocks on C12→C22 (600–1500) and C43→C44 (650–1450), plus a × 3 surge from S0 and W0 |
| C | ambulance W1→E1 at 900 s, through 6 signalised junctions |
| D | ambulance W1→E1 at 900 s and fire engine S2→N2 at 909 s, converging on C21 from conflicting approaches |
| E | D plus police S4→N4 at 949 s, which conflicts at C41 |
| F | capacity cut on C12→C13: one lane blocked and speed at 30%, 700–1500 |
| G | sensor noise (SD 0.35), 15% dropout and 2% phantom spikes of 25 vehicles, at every junction, all run |
| H | concept shift from 900 s: side-street demand × 4, arterial × 0.4 |
| I | downstream spillback: both lanes of C31→C41 blocked, plus a × 2 surge from W1, 600–1500 |
| J | demand moves: the west half × 2 from 300–1200 s, then the east half × 2 from 1200–2100 s |
| K | conflicting objectives: side-street × 3 and arterial × 3 surges, plus ambulance W4→E4, fire engine N2→S3 and police E4→W1 at 950–970 s |
| L | stale (10 s delay), contradictory (approaches swapped at C11, C14, C41 and C44), dropped (10%) and invalid (1%) observations, 600–1500 |
| M | VECTOR's forecasters fail at 900 s and the whole advisory orchestrator at 1500 s |
| N | recovery: both lanes of C21→C31 blocked with a × 4 surge, 600–1200; measured as the network recovers to 2100 s |
| O | combined stress: corridor surge, downstream blockage, capacity cut, ambulance, fire engine and 2 random emergency vehicles |
Scenarios H, J and K saturate the network for **every** controller. The
native completion rate there is 66–82%. Those scenarios are kept as defined.
## 4. Results (held-out seeds 95000–95009)
The full tables are in `RESULTS_TABLES_heldout.md` (T1–T9), generated from
`artifacts/final-adversarial-audit/results/heldout/`. This section gives the
results that decide the verdict.
### 4.1 Execution and integrity
* **Trials.** 800 attempted and 800 completed (`trials/heldout/attempts.jsonl`).
0 failed, 0 retried, 0 discarded.
* **Health gate.** 799 of 800 pass. The one failure is the vehicle-actuated
arm in G, seed 95009: one SUMO rear-end collision and teleport (§4.6). It is
kept in every analysis. Excluding it would move VECTOR − vehicle-actuated in
G from −20.2 to −24.5 s/veh and would not change any conclusion.
* **Harness equivalence.** The harness gives the same results as the validated
runner for both VECTOR and native (§3.2).
* **Shadow non-influence.** In 160 of 160 paired held-out trials, the signal
sequence is identical with and without the shadow (§4.7).
### 4.2 Primary: VECTOR vs native actuated (P1, Holm over 16 scenarios)
| scenario | VECTOR | native | VECTOR − native [95% t-CI] | seeds better/worse | Holm p |
|---|---|---|---|---|---|
| R0 | 170.0 | 182.0 | −12.0 [−69.9, 46.0] | 4/6 | 1.000 |
| A | 330.1 | 302.6 | +27.5 [−22.8, 77.7] | 4/6 | 1.000 |
| **B** | 314.7 | 241.2 | **+73.6 [26.6, 120.5]** | 0/10 | **0.031** |
| C | 173.8 | 140.7 | +33.1 [16.3, 49.8] | 1/9 | 0.059 |
| D | 175.6 | 146.2 | +29.4 [12.7, 46.2] | 1/9 | 0.076 |
| E | 173.4 | 173.1 | +0.3 [−57.7, 58.3] | 2/8 | 1.000 |
| F | 174.1 | 162.1 | +11.9 [−26.8, 50.7] | 3/7 | 1.000 |
| G | 163.6 | 182.0 | −18.3 [−73.4, 36.7] | 5/5 | 1.000 |
| H | 458.1 | 406.9 | +51.2 [19.9, 82.5] | 2/8 | 0.117 |
| I | 242.7 | 258.9 | −16.2 [−65.6, 33.3] | 4/6 | 1.000 |
| J | 591.8 | 522.7 | +69.1 [7.0, 131.2] | 2/8 | 0.371 |
| K | 453.1 | 341.8 | +111.2 [67.5, 154.9] | 1/9 | 0.059 |
| L | 187.6 | 182.0 | +5.7 [−55.0, 66.3] | 3/7 | 1.000 |
| M | 170.0 | 182.0 | −12.0 [−69.9, 46.0] | 4/6 | 1.000 |
| N | 292.3 | 290.8 | +1.5 [−56.3, 59.4] | 4/6 | 1.000 |
| O | 326.5 | 261.5 | +65.1 [11.5, 118.6] | 1/9 | 0.301 |
P1 is in seconds per requested vehicle. Bootstrap intervals agree with the
t-intervals (T2).
Reading the table:
* M ≡ R0 for both arms, because the shadow changes nothing and native is not
exposed to the faults.
* G and L share native's R0 runs, because native uses SUMO's detectors, not
the audit's sensor model.
* VECTOR's point estimate is lower than native's in 4 scenarios (R0, G, I,
M), and none of those is resolved.
* With 10 pairs, only effects of about one SD of the per-seed difference are
resolvable.
* Secondary endpoints (T5) point the same way. Under the congestion scenarios
B, H, J, K and O, VECTOR has more ineffective and wasted green, more
spillback and lower completion. Under K, completion is −10.4 pp [−15.7,
−5.1].
### 4.3 Secondary and exploratory baselines
**Against vehicle-actuated** (secondary family, T3). There is no Holm
rejection; the smallest is 0.0625, in A, J, L and N.
* Unadjusted, VECTOR is better in R0 (−24.9), A (−61.6), G (−20.2),
J (−127.3), L (−42.9), M (−24.9) and N (−95.9 s/veh).
* It is worse in K (+35.6 [6.2, 64.9]).
* The vehicle-actuated rule starves approaches badly: its max red wait is
500–1500 s. VECTOR's starvation guard prevents that.
**Against fixed-time 30 s** (exploratory, T4). VECTOR is better in 15 of 16
scenarios, by 84–218 s/veh, and on 8–10 of 10 seeds in each. J is not
resolved (−33.4 [−81.3, 14.5]).
Against the simplest controllers, VECTOR is a working adaptive controller.
Against a well-configured actuated controller, it is not better.
### 4.4 Emergency vehicles
Detection is the simulator's ground truth. Ordinary vehicles do not yield.
Only the signal timing can help.
| scenario | vehicle | VECTOR | native | veh-act | fixed | VECTOR − native [95% CI] |
|---|---|---|---|---|---|---|
| C | ambulance | 121 (10/10) | 168 | 191 | 227 | **−46.5 [−62.1, −30.9]** |
| D | ambulance | 126 (10/10) | 168 | 197 | 226 | **−42.0 [−54.1, −29.9]** |
| D | fire engine | 138 (10/10) | 204 | 209 | 278 | **−65.7 [−87.5, −43.9]** |
| E | ambulance | 127 (10/10) | 171 | 194 | 227 | **−43.8 [−59.4, −28.2]** |
| E | fire engine | 139 (10/10) | 205 | 208 | 299 | **−66.7 [−87.8, −45.6]** |
| E | police | 111 (10/10) | 150 | 152 | 213 | **−39.3 [−56.3, −22.3]** |
| K | ambulance | 169 (10/10) | 171 | 225 | 422 | −1.8 [−28.9, 25.3] |
| K | fire engine | 253 (10/10) | 391 | 474 | 319 (9/10) | **−138.0 [−193.0, −83.0]** |
| K | police | 1572 (**5/10**) | 1553 (7/10) | — (0/10) | 1576 (2/10) | +107 [−362, 576] (arrived pairs only) |
| O | ambulance | 116 (10/10) | 177 | 172 | 197 | **−61.1 [−76.7, −45.5]** |
| O | fire engine | 150 (10/10) | 215 | 198 | 299 (8/10) | **−64.8 [−85.5, −44.1]** |
Values are mean travel time in seconds, with arrivals out of 10 where not
every trip finished.
* **This is VECTOR's clearest strength.** In C, D, E and O every emergency
vehicle is faster under VECTOR than under every baseline. That includes the
conflicting arrivals in D and E (evidence E10–E12, E22).
* **The general-traffic cost** of those overrides is not separately resolved.
C differs from R0 only by the ambulance, but per-seed outcomes are too
sensitive to compare across scenarios (§5.5).
* **Under saturation the override does not guarantee arrival.** In K the
police car (E4→W1) is still in the network at 2700 s in 5 of 10 VECTOR runs.
VECTOR made on average 1075 priority decisions for it, without effect: a
green signal cannot move a vehicle through a gridlocked downstream link.
Native got it through in 7 of 10 runs (E26).
* The input contract carries **one** emergency vehicle per junction. When two
are on a junction's approaches, the harness shows VECTOR only the one
dispatched most recently (`max` over requested departure). The other is
invisible to VECTOR until then. This is a harness choice forced by the
single slot.
### 4.5 Runtime
The figures are from T8, measured on one 4-vCPU cloud machine with Python
timing. There is no real-time guarantee.
| arm | decide p50 | p95 | p99 | max | 36-junction cycle p95 | wall per trial |
|---|---|---|---|---|---|---|
| VECTOR (+ shadow) | 0.175 ms | 1.983 ms | 2.803 ms | 224.7 ms | 24.39 ms | 55.9 s |
| VECTOR (no shadow) | 0.021 ms | 0.056 ms | 0.082 ms | 20.4 ms | 1.73 ms | 22.3 s |
Control itself is cheap. The advisory shadow multiplies decide latency
roughly 35-fold at p95, and it produces the 225 ms worst case. A cycle at p95
uses 2.4% of the 1 s step.
### 4.6 Safety
These checks run in every trial and cover all 800.
* **Signal timing.** 0 timing violations (min green, yellow, all-red, max
green), 0 illegal states and 0 invalid requests. Every
interval was rebuilt from SUMO's reported state, not from the command.
* **Every step,** SUMO's reported state equalled the commanded state.
* **Teleporting** is disabled, and no route was changed.
* **Collisions.** VECTOR had 0 collisions and 0 teleports in 320 trials.
**The one collision.** The vehicle-actuated arm in G, seed 95009, had two
colliding vehicles and one collision teleport:
`Teleporting vehicle 'v1341'; collision with vehicle 'v1330', lane='C41_C31_1', gap=-0.10, time=876.00, stage=move`
* It reproduces byte for byte (same signal-sequence hash `1b903a85…`, same
P1).
* It is a rear-end gap violation within a lane, not a junction conflict:
illegal states are 0.
* The SUMO log is kept in
`integrity/sumo_log_G_vehicle_actuated_95009.log`.
It is a car-following event in a baseline arm, and it is reported, not
removed.
**What this does not establish.** The signal-safety constraints are enforced
by the harness's `SignalAdapter`, and VECTOR's own `SignalSafetyGovernor` is
not in the path. The result therefore shows that VECTOR's requests are safe
under an external interlock, not that VECTOR is safe without one.
### 4.7 Shadow non-influence and module degradation
* **Non-influence.** The signal-sequence sha256 of `vector` equals that of
`vector_noshadow` in 160 of 160 held-out pairs, and in 21 of 21 smoke
pairs.
* **Scenario M.** The forecasters raise from 900 s, and the whole orchestrator
raises from 1500 s.
* 255 217 orchestrator failures were caught and counted, and the control
sequence was unchanged (E19).
* This shows **isolation, not graceful degradation**. The failed modules
never influenced control, so there was nothing to degrade.
* **The stack does not see its own failure.** From 900 s to 1500 s, all
10 058 shadow decisions on the trace seed report `degraded: FULL,
fallback: false` with **0 forecasts** (§5.9).
### 4.8 Prediction (shadow only)
The forecast is the total queue at the junction, 8 s ahead, scored against
what SUMO then measured (T7). Persistence is "the reading now".
* The shadow ensemble's MAE is **worse than persistence in 15 of 16
scenarios, on every seed** (0/10 each). Examples:
* R0: 1.60 against 1.18 vehicles;
* J: 2.34 against 1.50.
* The exception is G, on 10 of 10 seeds: 1.78 against 2.62. There the phantom
spikes corrupt the reading that persistence repeats, and the ensemble
smooths them.
* **Concept shift, H.** Shadow MAE is 1.25 before 900 s and 2.60 after.
Persistence goes from 1.11 to 1.64. Nothing adapts online (E05, E06).
No model is authorised to influence control (the training gate holds them
all at UNTRAINED). The forecasts cost most of the decide latency and add no
predictive skill over the trivial forecast.
**The shadow is not reproducible.** The orchestrator runs forecasters only
while a 50 ms wall-clock compute budget lasts (`ComputeBudget` in
`extremis/intelligence/orchestrator.py`). Which forecasts exist therefore
depends on machine load.
* Re-running held-out `vector` trials reproduces every control, traffic,
safety and emergency field exactly.
* The shadow's forecast counts and scores differ slightly between runs
(`integrity/reproduction_check.json`).
* The T7 prediction numbers are thus valid for this machine and load, not
bit-for-bit.
* Control is unaffected, because nothing reads the shadow.
### 4.9 Observation faults
* **Sensor noise, G.** VECTOR is not resolved against native (−18.3
[−73.4, 36.7]). It is better than vehicle-actuated (−20.2 [−39.7, −0.8]),
which sees the same faults. The observer's ±0.5 veh/s bound absorbs the
phantom spikes. In E04, at C00, a shown jump to 33 vehicles (true value 6)
moves the belief by 0.5. The same bound also delays real changes: in the same
window the EW belief lags the true queue by 10–15 vehicles in both
directions (§5.2).
* **Stale, swapped and invalid observations, L.** VECTOR is not resolved
against native, and better than vehicle-actuated (−42.9 [−60.3, −25.6]).
But **nothing in VECTOR detects the corruption.**
* The adapter drops the stale flag.
* At swapped junctions VECTOR serves the wrong approach. In E23, at C44,
32–42 vehicles queue on EW while VECTOR, shown them on NS, extends NS
green to max green and then serves NS again.
* Only the max-green and starvation limits bound the damage.
## 5. Failure analysis
### 5.1 Where VECTOR loses
Against native, VECTOR is worse (unadjusted) in seven scenarios:
* the congestion scenarios B, H, J, K and O, where queues stop discharging
because a downstream link is full or blocked;
* the emergency scenarios C and D, which add one or two emergency vehicles to
R0's demand. Part of that gap is per-seed sensitivity, not mechanism (§5.5).
The fresh-seed replication tests it (§5.7).
It is not resolved in R0, A, E, F, G, I, L, M and N.
### 5.2 Mechanism: green held on a queue that cannot move
* **How the law reads demand.** `decide_signals` ranks approaches by halting
vehicle count, the `QueueObserver` belief. It extends green while that
approach still "has demand". A queue that cannot discharge stays a large
halting count, so it keeps winning.
* **What the law lacks.** It has no term for discharge, downstream occupancy
or spillback. No junction knows its neighbour's state.
* **How native differs.** Native gap-out logic ends green when its detectors
see no vehicle crossing, so it yields a blocked approach early.
* **The held-out data are consistent with this.** Over the 160 paired cells
(`DIAGNOSIS.md` D4):
* VECTOR − native in P1 correlates with VECTOR − native in ineffective green
at r = 0.91, in wasted green at r = 0.92, and in gridlock events at
r = 0.79;
* the fitted slope is about 15 s/veh more time loss per 1000 extra
junction-seconds of ineffective green;
* in R0, where nothing blocks, VECTOR's ineffective green is *lower* than
native's.
This is correlation across cells, labelled exploratory. §5.7 tests the
mechanism directly with VECTOR's own ablations.
* **One junction, decision by decision** (scenario I, C41, evidence E15 and
E27). Vehicles stalled at the stop line block both lanes of the C31→C41
approach from 641/662 s.
* VECTOR serves that approach EW green for the full 45 s.
* After clearance it serves EW again, because 29–34 vehicles are still
halting there.
* The queue grows during its own green, and the reason string labels it
"discharging 29".
* NS, with 2–7 vehicles that could move, waits.
* **Observer lag adds to it** (`DIAGNOSIS.md` D5). The `QueueObserver` bounds
every change to ±0.5 veh/s.
* On the trace seed, its belief is off the true queue by ≥ 5 vehicles in
3.2% of approach-decisions in R0. That rises to 6.0% in B, 8.2% in J and
10.1% in L.
* The lag works both ways. It can hold a green after a queue has
discharged, and it is slow to see a queue that has formed.
### 5.3 Starvation cap
VECTOR's longest red wait is 105 s in every trial; native's is 55 s (T5: +50
s in every scenario). That bound is a design choice: starvation at red ≥ 90 s,
plus clearance. It prevents the vehicle-actuated rule's 500–1500 s
starvation. But it lets VECTOR hold a side approach red twice as long as
native.
### 5.4 Emergency priority under saturation
In K the override fires (on average 1075 priority decisions for the police
car), but a green cannot clear a gridlocked downstream link. VECTOR has no
routing, so it cannot send the vehicle around the jam. The other two
emergency vehicles in K still arrive: the fire engine is 138 s faster than
under native; the ambulance is not resolved.
### 5.5 Sensitivity of per-seed outcomes
C is R0 plus one ambulance. Under native, adding that single vehicle changes
P1 by −214 to +29 s/veh across seeds; under VECTOR, by −40 to +49. The
mechanism is that one extra vehicle shifts SUMO's random stream and the
network's gridlock-or-not trajectory.
The consequences:
* Paired differences *within* a scenario are valid.
* Comparisons *across* scenarios that share demand are not: for example,
"native improves when an ambulance is added" is not a real effect.
* Ten seeds sample this sensitivity coarsely, which is why the post-hoc
replication (§5.7) matters.
### 5.6 Controller-dependent incident exposure (a validity threat, checked)
The blockages in B, F, I, N and O stall *the first vehicle found halted at the
lane's stop line* inside the window. Their onset therefore depends on the
controller (`DIAGNOSIS.md` D6).
* In B, VECTOR's trials had more blocked lane-steps than native's: 1498
against 1222 on average, more in 8 of 10 seeds. If that explained VECTOR's
loss, B would partly measure exposure rather than control.
* It does not. Within B, VECTOR's excess time loss is *negatively* correlated
with its excess exposure (r = −0.44), and the fitted loss at equal exposure
is +89 s/veh. On seed 95009, VECTOR's C12→C22 blockage never engaged, and
VECTOR still lost by 177 s/veh.
* In F, I, N and O, exposure differs by less than 6% on average.
This design choice is recorded as a limitation (§8). A fixed-onset blockage
would remove it.
### 5.7 Post-hoc diagnosis on fresh seeds
**Post-hoc and exploratory** (`AUDIT_PROTOCOL.md` §13, `DIAGNOSIS.md`). The
diagnosis ran after the held-out results were seen:
* on 10 fresh seeds, 96000–96009;
* in five scenarios: R0, B, C, J and K;
* on four arms: VECTOR without the shadow (identical control to `vector`),
native, and VECTOR's two registered ablations, used unchanged.
200 of 200 trials completed, and all passed the health gate. VECTOR was not
modified. No multiplicity correction is applied.
**Replication: VECTOR − native, P1, fresh seeds.**
| scenario | held-out (§4.2) | fresh seeds | seeds better/worse (fresh) |
|---|---|---|---|
| R0 | −12.0 [−69.9, 46.0] | **+21.1 [4.3, 37.9]** | 1/9 |
| B | +73.6 [26.6, 120.5] | +23.7 [−22.6, 70.0] | 4/6 |
| C | +33.1 [16.3, 49.8] | +17.9 [−1.8, 37.6] | 2/8 |
| J | +69.1 [7.0, 131.2] | **+67.0 [16.9, 117.2]** | 2/8 |
| K | +111.2 [67.5, 154.9] | **+116.3 [92.5, 140.2]** | 0/10 |
* **The direction replicates everywhere.** On fresh seeds VECTOR is never
better than native, and it is worse in R0, a result the held-out seeds had
left unresolved.
* **The size does not always replicate.** K and J are stable. B is about a
third of its held-out size, with an interval that includes 0. Ten seeds
sample the network's sensitivity (§5.5) coarsely, so single-scenario
magnitudes should be read with that in mind.
**Ablations: what each VECTOR mechanism contributes** (ablation − VECTOR, P1,
s/veh; positive means the ablation is worse):
| scenario | remove the queue observer | remove green commitment | without commitment, vs native |
|---|---|---|---|
| R0 | +16.7 [8.8, 24.5] | **−15.1 [−26.6, −3.7]** | +6.0 [−6.1, 18.1] |
| B | +2.8 [−34.3, 40.0] | +8.9 [−43.0, 60.8] | +32.6 [−25.8, 91.0] |
| C | +14.2 [7.0, 21.5] | **−17.1 [−27.8, −6.3]** | +0.8 [−22.3, 23.9] |
| J | +13.0 [−27.0, 52.9] | +102.9 [54.1, 151.7] | +170.0 [125.5, 214.4] |
| K | +37.1 [13.0, 61.3] | +65.6 [44.3, 86.8] | +181.9 [157.1, 206.7] |
* **The queue observer helps**, by 14–37 s/veh where resolved. It is not the
cause of the losses.
* **Green commitment is a trade-off.**
* At moderate demand (R0, C) it costs about 15–17 s/veh, and it is the
main source of VECTOR's excess ineffective green there (−858 and −924
junction-seconds without it, D3). Without commitment, VECTOR cannot be
distinguished from native in R0 and C.
* Under saturation (J, K) commitment is worth 65–103 s/veh. Presumably it
stops the law from switching on every queue fluctuation. That mechanism
was not separately tested.
* **Under congestion no variant comes close to native.** In K every VECTOR
variant is 116–182 s/veh worse, on 10 of 10 seeds. Neither mechanism is the
problem. The law has no notion of discharge or downstream capacity (§5.2),
and no ablation adds one.
These findings locate the deficit; they do not fix it. Any change to VECTOR
motivated by them would have to be evaluated on new, pre-registered seeds.
### 5.8 Corrupted input is not detected
In L, VECTOR serves swapped approaches because it has no consistency check
between approaches, over time or against neighbours (§4.9, E23). The
`stale` flag the harness provides is dropped by the `ExtremisCanonical`
adapter.
### 5.9 The advisory stack misreports its health
* **The symptom.** When every forecaster raises (M, 900–1500 s), the
orchestrator still reports `degraded: FULL, fallback: false` and emits 0
forecasts.
* **What gets counted.** Only orchestrator-level exceptions (after 1500 s)
are counted as `shadow_failures`. Forecaster failures are recorded per
model in the registry, but they are not surfaced as a degraded state.
* **Consequence today:** none, because nothing reads the stack.
* **Consequence if the shadow were ever promoted:** its health signal could
not be trusted.
## 6. Acceptance gates
| gate | requirement | result | evidence |
|---|---|---|---|
| A environment | ≥ 30 validated signalised junctions, connected, legal routes, reproducible scenarios, baselines verified | **PASS:** 36 junctions, strongly connected, 552 of 552 legal routes, 0 validation problems, harness ≡ validated runner | `topology/topology_report.json`, `integrity/equivalence_91001.json`, E01, E02 |
| B integration | modules mapped to code; control path traced end to end; influence verified where claimed; gaps listed | **PASS.** The audit gate passes; the integration finding is largely negative: 4 components influence control, the shadow has none (verified), and most named modules are absent (§2) | `integrity/module_inventory.json`, traces, E18, §2.2 |
| C adversarial coverage | all 16 scenarios on all arms; ≥ 2 emergency kinds; combined stress; failures kept | **PASS:** 16 × 5 × 10, 3 emergency kinds plus random emergency vehicles, O combined, 0 failures, every attempt logged | `trials/heldout/attempts.jsonl` |
| D measurable outcomes | traffic, emergency, prediction, runtime and safety metrics, paired, with uncertainty | **PASS** | `results/heldout/*.json`, `RESULTS_TABLES_heldout.md` |
| E evidence | ≥ 15 genuine screenshots with IDs, run links and hashes; raw logs kept | **PASS:** 27 genuine screenshots, E01–E27 (14 `sumo-gui` images from 8 held-out trial runs, 13 dashboard renderings of recorded results), each with an ID, run link, sim time, claim, limitation and a sha256 re-verified on disk; all 8 GUI runs reproduce their headless trials exactly; raw trials, traces and logs kept | `evidence/EVIDENCE_MANIFEST.json`, `EVIDENCE_GALLERY.md` |
| F reproducibility | commands, environment, commit, configuration and seeds recorded | **PASS:** commands, environment, commits (freeze `63bc12a`), network hashes, scenario config hashes and seeds recorded. 8 held-out trials (5 arms) re-run at `8558862`: every control, traffic, safety and emergency field identical. Only VECTOR's shadow forecast counts vary, by 1–2 in ~50 000, because its compute budget is wall-clock (§4.8). One command runs the whole audit (`run_audit.py`) | §10, `trials/heldout/preflight.json`, `integrity/reproduction_check.json` |
### 6.1 Evidence notes
* **Two kinds of image.**
* `sumo-gui` screenshots, taken over TraCI during a held-out trial on seed
95000. Each GUI run was checked against the headless trial of the same
scenario, arm and seed: the signal-sequence hash and P1 are equal, so each
image shows exactly the reported experiment.
* Screenshots of the audit dashboard rendering recorded result files. The
sha256 of every source file is recorded.
* **No image is drawn, edited or composited.** Each is re-hashed on disk
when the gallery is generated.
* **Captions were checked against the data and corrected where they were
wrong.** The GUI captions were written in the capture plan *before* the
runs. On inspection, six did not match the recorded state, and each was
rewritten from the recorded state:
* E07: the queue was at C01, not on every approach.
* E08: the change was to green, not to yellow.
* E13 and E14: the queue shown was on a different street.
* E15: no spillback was visible at C31.
* E20: the network had not drained.
E13, E14 and E15 were also re-framed and re-captured, to show the blockage
they describe. Re-captured images whose view did not change (E01, E07–E09,
E16 and E20) are byte-identical to the first capture, because rendering is
deterministic. The superseded first-capture files are not part of the
evidence set.
* **The full list** with captions, claims, limitations, run links and
hashes is in `EVIDENCE_GALLERY.md` and `evidence/EVIDENCE_MANIFEST.json`.
* Simulator images: E01, E03, E07–E16, E20, E21.
* Dashboard images: E02, E04–E06, E17–E19, E22–E27.
## 7. What VECTOR can and cannot do (simulation evidence only)
**It can:**
* control all 36 junctions of a connected grid for 45 simulated minutes,
under 16 adversarial conditions, with no signal-safety violation behind an
external interlock;
* beat fixed-time control almost everywhere;
* match or beat a simple vehicle-actuated rule in 15 of 16 scenarios. It is
worse in K;
* give emergency vehicles trips 39–138 s faster than native control, and
faster still than the other baselines. Under saturation (K) this does not
hold for every vehicle;
* survive the failure of its whole advisory stack with no change to control;
* make decisions in about 0.06 ms without the shadow, or about 2 ms with it,
at p95.
**It cannot, or has not been shown to:**
* outperform a well-configured actuated controller, in any scenario. It is
worse under congestion, significantly so in B;
* handle downstream blockage: it has no discharge or spillback term and no
coordination;
* detect stale, swapped or invalid observations;
* route or reroute any vehicle;
* forecast better than persistence (its shadow models);
* report its own degradation correctly, or produce reproducible advisory
output (the shadow is wall-clock dependent);
* use perception: no camera or detector path was in the loop;
* run with any of the named higher-level modules (Sentinel, Serpentine,
Aegis, Atlas, Punisher, Phoenix, Hyperion, Terminus, Anarchy, ORACLE,
PULSAR, the fusion engines, the learned models) in the loop;
* run on real data or a real network. Neither was used.
## 8. Limitations
**This is simulation, and only simulation.** Nothing here establishes
real-world safety, regulatory compliance, deployment readiness or operational
effectiveness.
* **No perception in the loop.** "Observation" is SUMO's halting count per
approach. In G and L it is corrupted by the audit's sensor model. The camera,
detector and `perception_adapter` path exists only in `run_extremis.py` and
was not exercised.
* **Emergency detection is ideal.** It is the simulator's ground truth, with
confidence 0.95, no false positives and no missed vehicles. The input admits
one emergency vehicle per junction. SUMO's blue-light device is off, so
ordinary vehicles never yield; only the signals can help an emergency
vehicle.
* **No rerouting exists**, in any arm. Every vehicle keeps its assigned route,
which SUMO's `vehroute` output confirms. Scenario F's "rerouting" is
therefore a test of its absence.
* **No coordination exists.** There are 36 independent controllers with no
offsets or shared state. A corridor result is the sum of local decisions.
* **One synthetic network.** It is a regular grid with uniform turning rules,
two lane counts and Poisson demand from fixed OD classes. There are no
pedestrians, buses, protected-left phases, detectors with real failure
physics, or real counts.
* **The demand level was set with the baselines on two calibration seeds.**
Several scenarios, by design, push the network past saturation (H, J, K for
every controller).
* **Ten paired seeds per cell** resolve paired differences of about 1 SD.
Smaller effects are "not resolved". Per-seed outcomes are sensitive to small
perturbations (§5.5).
* **The native baseline is not exposed to injected sensor faults.** It uses
SUMO's own detectors, so in G and L it is a reference, not a like-for-like
comparison.
* **Latency is measured on one 4-vCPU cloud machine** with Python-level
timing. There is no hard real-time guarantee, and there is no
hardware-in-the-loop.
* **Prediction exists only in shadow**, with an 8 s horizon on total junction
queue. No model is authorised to influence control. Nothing updates its
parameters online. The forecasters keep a short history; that is state, not
learning.
* **Shadow forecasts depend on wall-clock time.** The advisory orchestrator
skips forecasters when its 50 ms budget runs out, so the prediction scores
vary slightly with machine load (§4.8). Control is deterministic.
* **The safety interlock is the harness's.** VECTOR's own
`SignalSafetyGovernor` was not in the path.
* **Incident onset depends on the controller.** A blockage stalls the first
vehicle halted at its stop line, so exposure differs between arms. It was
checked and does not explain the B result (§5.6), but a fixed-onset design
would be cleaner.
* **The diagnosis is post-hoc** (§5.7). It uses fresh seeds and VECTOR's
existing ablations, but it was designed after seeing the held-out results,
so it is exploratory.
## 9. Pilot-readiness assessment
The evidence supports **neither connection to live public signals nor an
uncontrolled field trial.** VECTOR is outperformed by standard actuated
control under congestion. It cannot detect corrupted input. Its safety
interlock in these runs was external. And none of this was tested on a real
network or with real data.
The defensible next step is a **supervised research collaboration**, offline
and shadow-only, in this order (`docs/pilot/PILOT_PLAN.md`):
1. **A data-sharing agreement** with a road authority or university
transport group. Historical detector counts and signal timing plans for a
corridor or small district would be enough.
2. **A calibrated simulation of that site.** Build the network, calibrate
demand to counts, and validate the native or field controller against the
field data before VECTOR is run at all.
3. **The comparison:** VECTOR against the site's own controller, on
pre-registered seeds, with this audit's harness, safety layer and
statistics, reviewed by the partner's engineers.
4. **Fix the known defects first**, each re-evaluated on fresh seeds:
* a discharge or downstream term in the law (§5.2);
* observation consistency checks (§5.8);
* truthful degradation reporting (§5.9);
* VECTOR's own safety governor in the path.
5. **Only after that, and with authorisation:** a closed test track or a
hardware-in-the-loop bench with a real signal controller. VECTOR would
still run in shadow, beside the certified controller.
Contacting partners, sending outreach and signing agreements are decisions
for the project owner. None was done by this audit.
## 10. Reproduction
**Environment.**
* Ubuntu 24.04, Python 3.11.
* SUMO 1.28.0 from `requirements-sumo.txt`, installed with
`pip install --target ~/.vector-deps/sumo-1.28.0 -r requirements-sumo.txt`
(`stus/sumo/env.py` finds it).
* numpy 2.4 and scipy 1.17.
* For the GUI evidence: `xvfb-run`.
* For the dashboard captures and the PDF: Chromium with Playwright on
Node 22, and python-markdown 3.7.
* The reference machine had 4 vCPUs and 15 GB RAM. The held-out run took
1.6 h of wall time with 4 workers (6.4 trial-hours).
**Commands,** from the repository root. One command runs every stage, in
order, and logs each stage's exit code and duration to
`artifacts/final-adversarial-audit/audit_run_log.jsonl`:
```bash
python audit/final_adversarial/run_audit.py # all stages; --list shows them; --stages picks some
```
The stages are these individual commands:
```bash
# 0. preflight: network hashes, >= 30 junctions, equivalence with the validated runner
python audit/final_adversarial/topology_report.py
python audit/final_adversarial/scenario_matrix.py
python audit/final_adversarial/equivalence.py --seed 91001
python audit/final_adversarial/module_inventory.py
# 1. held-out evaluation (refuses a dirty tree; records the commit in preflight.json)
python audit/final_adversarial/run_suite.py --phase heldout --seeds 95000-95009 --trace-seed 95000 --workers 4
# 2. analysis and tables
python audit/final_adversarial/analyze.py --phase heldout
python audit/final_adversarial/make_tables.py --phase heldout
# 3. post-hoc diagnosis (fresh seeds)
python audit/final_adversarial/run_suite.py --phase diagnostic --seeds 96000-96009 \
--scenarios R0 B C K J --arms vector_noshadow diag_no_observer diag_no_commitment native_actuated
python audit/final_adversarial/diagnose.py
python audit/final_adversarial/reproduce_check.py # re-runs 8 held-out trials, compares every field
# 4. evidence
xvfb-run -a -s "-screen 0 1920x1300x24" python audit/final_adversarial/capture_gui.py \
--seed 95000 --out artifacts/final-adversarial-audit/evidence
python3 -m http.server 8899 --bind 127.0.0.1 & # serves the repository for the dashboard
node audit/final_adversarial/capture_dashboard.js audit/final_adversarial/dashboard_plan.json \
artifacts/final-adversarial-audit/evidence
python audit/final_adversarial/make_gallery.py
# 5. dossier PDF
python audit/final_adversarial/build_pdf.py
```
**Determinism.** A trial's control and traffic outcome is fully determined by
its scenario, arm and seed. Trials run in separate processes and do not share
state. SUMO runs with a fixed seed
and teleporting disabled.
`reproduce_check.py` re-ran eight held-out trials, covering all five arms, at
commit `8558862`. Every control, traffic, safety and emergency field was
identical to the stored result (`integrity/reproduction_check.json`).
The only differences were in VECTOR's shadow forecast scoring: 1–2 forecasts
in about 50 000, changing scores in the third decimal. The cause is the
advisory orchestrator's wall-clock compute budget (§4.8).
Each `sumo-gui` evidence run also reproduced its headless trial exactly: the
same signal-sequence hash and the same P1.
## 11. Artifact index
| what | where |
|---|---|
| protocol (frozen), trace schema, scenario matrix | `docs/audit/final-adversarial/AUDIT_PROTOCOL.md`, `TRACE_SCHEMA.md`, `scenario_matrix.json` |
| results tables (held-out, smoke), post-hoc diagnosis | `RESULTS_TABLES_heldout.md`, `RESULTS_TABLES_smoke.md`, `DIAGNOSIS.md` |
| evidence gallery and manifest | `EVIDENCE_GALLERY.md`; `artifacts/final-adversarial-audit/evidence/EVIDENCE_MANIFEST.json` |
| this dossier as PDF (committed; the HTML beside it is regenerated by `build_pdf.py`) | `artifacts/final-adversarial-audit/dossier/FINAL_DOSSIER.pdf` |
| trial summaries (every attempt) | `artifacts/final-adversarial-audit/trials/{smoke,heldout,diagnostic}/` |
| control-cycle traces (trace seed 95000, every arm) | `artifacts/final-adversarial-audit/traces/heldout/` |
| analysis outputs | `artifacts/final-adversarial-audit/results/{heldout,smoke,diagnostic}/` |
| network, integrity and calibration | `artifacts/final-adversarial-audit/{topology,integrity,calibration}/` |
| audit code | `audit/final_adversarial/` (harness, scenarios, runner, analysis, evidence, dashboard) |