Sentinel

VECTOR archive › Governance stack › Sentinel

A system cannot govern what it cannot see. Sentinel is the component that sees, and everything downstream of it is acting on a description of the system rather than the system itself — which makes the fidelity of that description the upper bound on the quality of every decision that follows.

Historical context

Sentinel was the first protocol built, and for a period it was the only one. A system that merely reported anomalies to a human was already more useful than one that did not, and everything else in the escalation chain was added because reporting turned out to be insufficient at machine speed.

It is also the component that nearly ended the project. Its original implementation performed unbounded statistical work on every cycle and consumed roughly 97% of runtime, holding p95 latency near 100 ms — while being the last component anyone suspected, because it was the simplest and the most trusted. The full account is in the latency crisis, and the constraint it produced — that every stage carries a hard time budget — now applies system-wide.

Responsibilities

Sentinel owns the collection of runtime metrics, the detection of statistical anomalies within them, and the categorisation and severity scoring of what it finds.

Sentinel never owns interpretation. It does not decide whether the system is in trouble — that is Serpentine's judgement, formed from Sentinel's output. It does not act, contain, throttle or repair. The separation is deliberate and follows the boundary between seeing and deciding: a detector that also decides inherits every one of its own weaknesses into the decision, silently.

Inputs

Runtime telemetry, sampled every second: latency, CPU, memory, GPU, queue depth and throughput. Sentinel consumes no output from any other protocol, which makes it the only component in the chain with no upstream dependency — and the reason it can still function when everything above it has failed.

Outputs

Two streams, deliberately separated by priority.

Raw metrics travel at medium priority: they are the ordinary substrate other components reason from. Detected anomalies escalate at high priority immediately.

The priority split matters more than it appears. A bus that treats "something is wrong" as ordinary traffic delivers that message behind whatever queue had already formed — which is longest precisely when something is wrong.

Internal behaviour

Detection is statistical rather than threshold-based: z-score analysis against a rolling history of twenty samples. A fixed threshold answers "is this value large", which requires knowing in advance what large means for every metric under every condition. A rolling comparison answers "is this value unusual for this system right now", which is the question that generalises.

The window is short by design. Twenty samples adapts quickly enough to follow genuine regime changes rather than alarming through all of them, at the cost of adapting to a degradation that arrives slowly enough — a tradeoff discussed below.

Since the latency work, statistics are maintained incrementally at constant cost per sample, heavier analysis is displaced into bounded background tasks, and the whole stage runs under a 5 ms hard cap with early exit when risk or latency indicators exceed thresholds.

Interactions

telemetry → Sentinel → Serpentine → (rest of the chain)

Sentinel executes first, every cycle, unconditionally. Its only consumer in the governance chain is Serpentine, which synthesises its output into a single stress state. Its anomaly stream is also what the intelligence layer's trigger conditions are evaluated against — see the intelligence layer.

Nothing calls Sentinel. It is not a service that answers questions; it is a stage that runs.

Failure modes

Sentinel stops producing. The chain has no input. Every downstream protocol is reasoning from a stale description of a system that is still changing, and — worse — nothing downstream can distinguish "no anomalies" from "no observations". This is the most dangerous failure in the stack, because it is silent and it looks like health. The mitigation is that absence of output is itself an observable condition rather than an inferred one.

Sentinel exceeds its budget. Bounded by the 5 ms cap: the stage exits with what it has rather than running long. Partial detection is a degradation. An unbounded head of the pipeline is a system-wide failure, which is exactly what the latency crisis was.

Sentinel detects spuriously. Excess anomalies propagate as elevated stress, which narrows autonomy unnecessarily. This is the correct direction to fail in — it costs capability rather than safety — but sustained false positives erode the credibility of the whole chain, which is a slower and more serious harm.

Design tradeoffs

The rolling window is short. Twenty samples means a fault that degrades slowly enough can become the new baseline without ever registering as anomalous. Detecting that class of drift needs a longer horizon, and a longer horizon adapts too slowly to legitimate regime change. There is no window length that solves both; this one favours responsiveness, and the gap is real.

Statistical detection has no semantics. Sentinel knows a value is unusual. It does not know whether unusual is bad. That judgement is deferred to Serpentine, which keeps this component simple and means detection carries no domain knowledge at all.

The time cap can hide work. A stage that exits early under load is doing less detection at precisely the moment there is more to detect. Accepted deliberately: late detection in a real-time system is indistinguishable from no detection, and the unbounded alternative was measured and was worse.

Future evolution

The clearest gap is adversarial input. Sentinel currently assumes its telemetry is honest but noisy. A sensor that is wrong deliberately — reporting plausible values that are false — defeats statistical detection entirely, because the readings are designed to be unremarkable. An adversarial robustness layer is identified as future work in the v1 changelog and is not implemented.

The second is multi-horizon detection: maintaining several windows simultaneously so that slow drift and fast spikes are both visible without one window having to serve both. This is inference about a natural direction rather than a planned change.