# Final adversarial audit: protocol (frozen before the held-out evaluation)

This file is committed **before** any held-out seed is run. The commit that
freezes it is the commit every held-out trial records (`preflight.json` →
`commit`). `run_suite.py` refuses held-out runs on a dirty tree.

## 1. Question

Can VECTOR, as it is implemented, control a 36-junction simulated city through
observational response? How does it compare with credible non-VECTOR control
under adversarial conditions? This is measured on paired seeds, with every
failure kept.

## 2. What is under test (verified from code, not names)

The SUMO control path is `extremis_canonical`, one fresh instance per junction.
Per junction, it runs:

`Observation` → `ExtremisCanonical` adapter → `QueueObserver` (rate-bounded
queue belief) → `control_logic.decide_signals`.

`decide_signals` applies four rules in order:

1. emergency override (confidence ≥ 0.75);
2. no-demand fallback;
3. starvation prevention (red ≥ 90 s);
4. vehicle-count priority.

The proposal passes through `SignalAdapter`:
* min green 8 s, max green 45 s, yellow 3 s, all-red 2 s;
* an invalid request is replaced and recorded.

SUMO's reported state must equal the commanded state every step.
`IntervalLog` re-derives every interval from SUMO's own reports.

**Attached in shadow** (arm `vector`): VECTOR's advisory stack. This is an
`IntelligenceOrchestrator` per junction (StateEstimator, SENTRY/PREDATOR/APEX
tiers), with the 5 registered forecasters (persistence, physics, Kalman,
spectral, autoregressive), the counterfactual service, and the ShadowLedger.
By construction it runs after the phase is settled. The audit verifies that
it changes nothing: arm `vector_noshadow`, same seeds, must give identical
signal sequences.

**Not in the SUMO path, so not testable as integrated components here**
(`docs/architecture/DATAFLOW.md`):
* perception (camera, detector, `perception_adapter`);
* routing (SERPENTINE, the A* router);
* the protocol stack: Sentinel, Serpentine, Aegis, Atlas, Punisher, Phoenix,
  Hyperion, Terminus, Anarchy/ORACLE/PULSAR/Fusion/MORL. It runs only in
  `run_extremis.py` with synthetic perception;
* the training gate;
* learned control models;
* any coordination between junctions.

The audit reports each as **not in the execution path**, never as passed.

## 3. Network

`city36` (`stus/sumo/multi/topology.py`) is a 6 × 6 grid with 36 signalised
junctions:
* 4 two-lane arterials (rows 1 and 4, columns 1 and 4) and 8 one-lane side
  streets;
* links of 180–260 m;
* 24 fringe entry/exit nodes.

Built by `netconvert` (SUMO 1.28.0), with canonical hashes pinned:
* `city36_static`: `8a9ade80…a305`;
* `city36_actuated_matched`: `6f2c50a9…cdca`.

Gate A evidence is `artifacts/final-adversarial-audit/topology/topology_report.json`.

## 4. Demand and calibration

There is one route per ordered fringe pair: 552 OD routes, all legal.
Arrivals are Poisson per second and route; the rates are by OD class.

**Calibration** (`audit/final_adversarial/calibrate.py`) used the baselines
only: `native_actuated` and `fixed_time_30s`. It ran on calibration seeds
90000–90001 at scales 0.2–0.5 of an initial guess
(`artifacts/final-adversarial-audit/calibration/sweep1.json`).

* **VECTOR was not run during calibration.**
* The base was fixed at the 0.45 level: arterial–arterial 0.009,
  arterial–side and side–arterial 0.0027, side–side 0.0009 veh/s per route.
* Every evaluation scenario runs at 1.15 × base.
* Demand enters for 0–2100 s; the run lasts 2700 s, so the network drains.
  A trip that never completes is a measured failure, charged its time loss so
  far.

## 5. Scenarios

Definitions are in `audit/final_adversarial/scenarios.py`; the config hashes
are in `scenario_matrix.json`.

| id | condition |
|---|---|
| R0 | reference: base demand, no event |
| A | row-1 corridor surge ×4, 600–1500 s |
| B | two blockages in different districts, plus a south-west surge ×3 |
| C | ambulance W1→E1 along the row-1 arterial, dispatched at 900 s |
| D | C, plus a fire engine S2→N2 converging on C21 from the conflicting approach (free-flow arrivals 940 and 941 s) |
| E | D, plus police S4→N4 converging on C41 (971 and 975 s) |
| F | capacity reduction on C12→C13: lane 0 blocked and speed 30%, 700–1500 s |
| G | sensor faults on every junction: noise σ 0.35, 15% dropout (stale), 2% phantom spikes of +25 |
| H | concept shift from 900 s: side–side ×4, side–arterial ×2, arterial–arterial ×0.4 |
| I | downstream exit C31→C41 fully blocked 600–1500 s, plus a W1 surge ×2 |
| J | demand ×2 in the west half for 300–1200 s, then in the east half for 1200–2100 s |
| K | side-street surge ×3, row-4 surge ×3, and ambulance/fire/police with a conflict at C24 |
| L | 600–1500 s: readings 10 s late, 10% dropout, NS/EW swapped at C11/C14/C41/C44, 1% invalid (NaN or −5) |
| M | VECTOR's forecasters made to raise at 900 s, and the advisory orchestrator at 1500 s |
| N | recovery: surge ×4 and a full blockage of C21→C31, cleared at 1200 s |
| O | combined: row-1 surge ×3, C31→C41 blocked 800–1400 s, ambulance + fire engine converging on C24, speed cut on C14→C15 1100–1700 s, 2 random emergency vehicles |

Emergency vehicles are vClass `emergency`:
* ambulance: 6.5 m;
* fire engine: 10 m, slower acceleration;
* police: 4.8 m.

SUMO's blue-light device is off, as in every earlier study: ordinary vehicles
do not yield.

**Sensor faults** act on the queue counts the rule controllers are shown.
`native_actuated` uses SUMO's own detectors and is **not exposed to them**. In
G and L, the like-for-like comparison is therefore VECTOR against
`vehicle_actuated` and `fixed_time_30s`.

## 6. Arms

| arm | what |
|---|---|
| `vector` | `extremis_canonical` + the advisory stack in shadow (§2) |
| `vector_noshadow` | `extremis_canonical` exactly as in production (no shadow) |
| `native_actuated` | SUMO's actuated programme with the adapter's timing |
| `vehicle_actuated` | the repository's actuated rule baseline, same adapter |
| `fixed_time_30s` | fixed-time rule baseline, same adapter |

* **No uncontrolled reference.** Junctions without signals at these volumes
  would not be a meaningful or safe comparison in SUMO.
* **Strongest baseline.** On grid2x2 the earlier held-out study found
  `native_actuated` the strongest (it beat VECTOR in 4 of 5 scenarios). It is
  therefore the primary comparator.

## 7. Seeds

| range | use | VECTOR seen? |
|---|---|---|
| 90000–90099 | calibration | no |
| 91000–91099 | smoke / development | yes (no VECTOR parameter is tuned; nothing in VECTOR is changed) |
| **95000–95009** | **held-out evaluation, run once after this freeze** | — |

* **Sample size.** 10 paired seeds × 16 scenarios × 5 arms = 800 trials,
  about 4 h on this machine. Ten pairs resolve a paired difference of about
  1.0 SD of the per-seed differences (two-sided t-test, α = 0.05,
  power 0.8). Smaller effects will be reported **not resolved**, which is a
  result.
* **Traces.** Seed 95000 is the trace seed: every arm writes the per-decision
  trace.

## 8. Endpoints

* **Primary.** P1: mean time loss over all requested ordinary vehicles, s/veh.
  * Arrived vehicles are charged SUMO's `timeLoss`.
  * Vehicles still running are charged their time loss so far.
  * Vehicles never inserted are charged the wait since their requested
    departure.
* **Secondary**, each defined in `analyze.py`:
  * p95 time loss;
  * completion rate;
  * throughput;
  * total delay (veh·h);
  * mean and median waiting time;
  * travel time of arrived trips (mean and p95);
  * network halting (mean, max, in the window);
  * spillback lane-steps: a halting vehicle within 10 m of the upstream end of
    an internal lane;
  * gridlock events: ≥ 120 s with ≥ 5 halting and no stop-line pass;
  * longest red wait (starvation);
  * stops per junction on through arterial trips;
  * congestion Gini over junctions;
  * vehicles still running at the end.
* **Emergency**, per vehicle:
  * dispatch and arrival;
  * travel time and time loss;
  * waiting and stops, and where the waiting happened;
  * priority decisions (VECTOR calls with EMERGENCY_OVERRIDE for that vehicle);
  * route changes;
  * state (arrived, running or never inserted).
* **Prediction** (VECTOR's shadow): forecast total queue at t + 8 s against
  SUMO's halting at t + 8 s, scored by MAE, RMSE and bias, overall and by
  tier. It is compared with persistence. In H it is also scored before and
  after the shift, and by 300 s bins.
* **Safety:**
  * timing violations, illegal states and invalid requests;
  * teleports (disabled) and collisions;
  * route changes;
  * shadow failures;
  * the shadow non-influence check.
* **Runtime:** controller decide latency (p50, p95, p99, max), the latency of
  the whole 36-junction decision cycle, and wall time per trial.

## 9. Statistics

For each scenario, endpoint and baseline, the paired differences
(VECTOR − baseline, by seed) are reported with:
* mean and SD;
* a 95% t-interval;
* a 95% bootstrap interval (10 000 resamples, `Random(0)`);
* the Wilcoxon signed-rank p (exact);
* the seeds where VECTOR is better or worse;
* the relative difference.

Two families are named in advance:
* **Primary:** P1, VECTOR vs `native_actuated`, over the 16 scenarios,
  Holm-adjusted at α = 0.05.
* **Secondary:** the same against `vehicle_actuated`.

Everything else is exploratory and labelled so. A difference whose interval
contains 0 is **not resolved**.

## 10. Gates

| gate | passes when |
|---|---|
| A environment | ≥ 30 validated signalised junctions, connected, legal routes, reproducible scenarios, baselines verified (equivalence check) |
| B integration | modules mapped to executable code; the control path traced end to end; influence verified where claimed; gaps listed |
| C adversarial coverage | all 16 scenarios attempted on all arms; ≥ 2 emergency kinds; combined stress attempted; failures kept |
| D measurable outcomes | traffic, emergency, prediction, runtime and safety metrics collected; paired; with uncertainty |
| E evidence | ≥ 15 genuine screenshots with IDs, run links, hashes; raw logs and tables kept |
| F reproducibility | commands, environment, commit, configuration, seeds recorded |

Audit completeness and the system verdict are reported separately.

## 11. Changes before the freeze

Each change was made before any held-out seed and was not driven by VECTOR's
results.

1. **The emergency routes of C, D, E and O were redesigned.**
   * SUMO's fastest paths for the first design (W1→N4, S1→E4) ran along side
     streets and shared **no** junction, so D and E would have tested no
     conflict.
   * Found by route analysis, not by results.
   * The new routes converge from conflicting approaches within 1–4 s at free
     flow.
2. **Harness instrumentation was added after the smoke run:** each junction's
   last decision, for the screenshot hook, and p99 latency. Neither changes
   control.
3. **Two measurement-only metrics were added after smoke B showed VECTOR
   gridlocking.** Both are computed from SUMO for every arm alike, and the
   equivalence check is re-run after them.
   * `ineffective_green_s`: green on an approach with ≥ 2 halting vehicles
     and no stop-line crossing for ≥ 10 s. The approach cannot discharge,
     because it is blocked or its downstream link is full.
   * `wasted_green_s`: green on an approach with no crossing for ≥ 5 s, while
     ≥ 3 vehicles halt on the conflicting approach.

   They are diagnostic endpoints S18 and S19, explanatory only and not part of
   either Holm family.
4. **Smoke results kept, not discarded.** The smoke runs of C/D/E/O on seed
   91000 used the first emergency-vehicle design. Their files record that
   design's config hashes. C/D/E/O and B were re-smoked on 91001 with the
   frozen definitions.

No intensity, rate or window of any scenario was changed after VECTOR's smoke
results were seen. Scenarios H, J and K saturate the network for **every** arm
(native completion 59–75% on the smoke seed). They are kept as defined:
"every controller gridlocks" is an adversarial result, not a defect of the
scenario.

## 12. Retention

| what | where | committed |
|---|---|---|
| every trial summary (smoke and held-out), including failed attempts (`*.error.json`) and `attempts.jsonl` | `artifacts/final-adversarial-audit/trials/` | yes |
| held-out decision traces (trace seed 95000, every arm) | `artifacts/final-adversarial-audit/traces/heldout/` | yes |
| smoke traces (105 files, 44 MB, development only) | outside the repository | no; sha256 of each in `traces/smoke_manifest.json` |
| screenshots and their manifest | `artifacts/final-adversarial-audit/evidence/` | yes |

The freeze commit is the one recorded in `trials/heldout/preflight.json`.

## 13. Post-hoc diagnosis (added after the held-out evaluation)

This section was written **after** the held-out results were seen. It changes
nothing above and re-tests no held-out result.

* **Purpose.** To locate why VECTOR loses to the native actuated controller in
  the congestion scenarios, and to check whether those losses repeat on seeds
  the held-out evaluation did not use.
* **Seeds.** A new range, `diagnostic`, 96000–96099, never used before. The
  diagnosis uses 96000–96009.
* **Scenarios.** R0, B (the one Holm rejection), C and K (Holm p 0.059), and J
  (saturated).
* **Arms.**
  * `vector_noshadow`: the same control as `vector` (§4.7 of the dossier shows
    the two are identical);
  * `native_actuated`;
  * `diag_no_observer` = `extremis_canonical_no_observer`;
  * `diag_no_commitment` = `extremis_canonical_no_commitment`.

  The last two are VECTOR's own registered ablations in
  `benchmarks/controllers.py`, used unchanged. `run_suite.py` refuses them
  outside the diagnostic range.
* **VECTOR is not modified.** No parameter, rule or threshold of VECTOR was
  changed after the held-out results.
* **Analysis.** `diagnose.py` writes `results/diagnostic/diagnosis.json` and
  `DIAGNOSIS.md`. It reports paired differences with t-intervals, uncorrected
  and labelled exploratory. Its mechanism check uses only the held-out trial
  table.