The latency crisis: one component, ninety-seven percent

After the intelligence layer was built, the system produced brutal latency numbers. p95 around 100 ms. Tail spikes. Instability under stress. Not remotely real-time capable.

This is the most useful thing that has happened to the project, so it is written down in more detail than it strictly needs.

Looking in the wrong place

The obvious suspects were the new components. ORACLE runs a world model. MORL does multi-objective optimisation. Anarchy spawns simulations. Those are the expensive- sounding things, and I spent weeks assuming the cost was where the sophistication was.

It was not. I profiled everything properly, and Sentinel — the anomaly detector, the simplest component, the one written first and trusted longest — was consuming roughly 97% of runtime.

It was the component I had never suspected precisely because it was the one I understood best.

Why it was so expensive

Sentinel ran heavy statistical checks on every cycle, with unbounded work patterns. Unbounded is the operative word: the cost varied with conditions, so it was not a constant overhead that shows up cleanly in an average. It amplified into tail latency spikes, and because Sentinel sits at the head of the pipeline, every spike poisoned everything downstream.

Every stage after it inherited the delay and reported its own latency honestly, which made the whole pipeline look uniformly slow rather than pointing anywhere.

The fix

• Rolling window metrics with O(1) incremental statistics
• A 5 ms hard cap on Sentinel's time budget
• Heavy analysis moved to bounded background tasks
• Strict per-stage budget enforcement across the entire pipeline
• Early-exit logic when risk or latency indicators exceed thresholds
• Spike detection with temporary mitigation windows afterwards

One bottleneck. One fix. The system went from unstable to real-time.

The numbers

Full snapshot: 0.607 ms average, 1.124 ms p95, 1.634 ms p99, 1.908 ms maximum. Zero dropped events on the bus, zero ordering violations, 32 of 32 regression tests passing.

What I actually learned

Find the bottleneck before optimising anything. Everything else was noise. Six weeks of careful work on components that were not the problem produced nothing, and would have produced nothing however well I had done it.

Suspicion should follow measurement, not architecture. I searched where the complexity was, because complexity feels expensive. Cost does not care how sophisticated a component looks.

Unbounded is the defect, not slow. Sentinel was not slow on average. It was unbounded, which meant its worst case was unrelated to its typical case — and in a real-time system the worst case is the specification. This is why every stage now carries a hard budget.

Tail risk is a first-class problem. Average latency means nothing if p99 is catastrophic. The early system had a tolerable average and an unusable tail.