Aegis

VECTOR archive › Governance stack › Aegis

Most catastrophic failures in distributed systems are not caused by the component that failed first. They are caused by the redistribution of its work onto components that were already near capacity. Aegis exists to make a local failure stay local.

Historical context

Detection and classification produce an accurate description of a system getting worse, and nothing else. Aegis is the point at which the chain stops observing and starts acting, and it is deliberately the first protocol with that authority — because the earliest possible intervention is also the cheapest and the most reversible.

Responsibilities

Aegis owns isolation, throttling, and the global circuit breakers. It decides which nodes stop receiving work and which have their intake limited.

Aegis never owns repair or recovery. It removes a failing component from the path; it does not attempt to fix it — that is Punisher — and it does not bring it back — that is Phoenix. Nor does it redistribute the work it has displaced, which belongs to Atlas.

The narrowness is the point. Containment that also repairs is containment that can be delayed by a repair attempt.

Inputs

The stress state from Serpentine, and per-node error rates and load figures originating with Sentinel.

Outputs

Isolation and throttling decisions, circuit breaker state, and — for every action — a log entry carrying explicit reasoning.

The reasoning is not incidental. An isolation decision without a recorded rationale is indistinguishable after the fact from an arbitrary one, and the operator reviewing an incident needs to know why a node was removed, not merely that it was. This is explainability before automation applied at the level of a single protocol.

Internal behaviour

Three mechanisms of increasing severity.

Isolation. Nodes exceeding a 20% error rate are removed from the working set. The threshold is low deliberately: a node failing one request in five is not marginal, it is broken, and the cost of removing a functioning node is capacity while the cost of retaining a broken one is propagated errors.

Throttling. Nodes above 95% load have their intake limited rather than being removed. A heavily loaded node is not faulty; it is oversubscribed, and removing it transfers the oversubscription elsewhere. Throttling is the gentler instrument for the case where the node is still doing useful work.

Global circuit breakers. Under catastrophic state, breakers activate system-wide. This is no longer per-node judgement — it is the recognition that individual decisions have stopped being sufficient.

Interactions

Serpentine → Aegis → Atlas → Punisher → Phoenix

Aegis runs before Atlas, and the order is load-bearing: containment precedes rebalancing, so that work is never redistributed onto a node that should already have been isolated. Reversing them would move traffic onto failing capacity.

Failure modes

Over-isolation. Aggressive removal reduces capacity, which raises load on what remains, which raises error rates, which triggers further isolation. This is the characteristic cascade of a containment layer, and it is caused by the protection rather than the fault. Throttling exists partly to provide a response that does not reduce capacity.

Under-isolation. A node failing in a way that does not raise its error rate above threshold — returning wrong answers rather than errors — is not caught at all. Aegis observes failure, not correctness, and a component that fails silently is invisible to it. This is a genuine gap and not one the current design addresses.

Isolating the wrong node. Error rates attribute failure to where it surfaced, which is not always where it originated. A healthy node depending on a failing one exhibits errors of its own.

Design tradeoffs

Fixed thresholds across heterogeneous components. 20% and 95% are applied uniformly. A component whose normal error rate is legitimately higher is isolated unnecessarily; one whose failure appears well below the threshold is not caught. Per-component thresholds would fit better and would multiply the configuration surface that has to be reasoned about during an incident.

Availability is spent to buy containment. Every isolation reduces capacity, and the protocol accepts degraded throughput to prevent propagated failure. Under sufficiently broad degradation this is the wrong trade, and Aegis has no mechanism for recognising that it has isolated too much.

Future evolution

The identified gap is silent failure — detecting components that are wrong rather than erroring. That requires correctness signals Aegis does not currently receive, and plausibly belongs upstream in detection rather than here.

Dependency-aware isolation, so that attribution follows the call graph rather than the surface where errors appeared, is the second. Both are inference about direction, not planned work.