Phoenix

VECTOR archive › Governance stack › Phoenix

Restoring a distributed system is not the reverse of failing. Components come back in an order that matters, carrying state that may or may not be valid, into a system whose shape has changed while they were absent. Phoenix owns that process.

Historical context

Repair without recovery leaves a system that is technically fixed and not actually serving. Something has to decide when a repaired component rejoins, in what order, and with what state — and the naive approach, restarting everything simultaneously, tends to produce a thundering-herd effect against dependencies that have not themselves recovered.

Responsibilities

Phoenix owns restart sequencing, container rebuilds, and the restoration of global runtime state.

Phoenix never owns the decision that a component is repairable — that is Punisher — and it does not decide whether the system is safe to operate, which remains with Serpentine and VIHAAN. Phoenix restores; it does not judge.

Inputs

The set of components requiring restoration, the dependency graph between them, per-node load, and the runtime state to be restored.

Outputs

Sequenced restarts, container rebuilds, and a restored global runtime state.

Internal behaviour

Three mechanisms.

Dependency-ordered restarts. Components come back in an order that respects what they depend on. A service restarted before its dependency fails immediately and requires another restart, which is both slower and noisier than waiting.

Container rebuilds above 85% load. A heavily loaded node is rebuilt rather than restarted. The distinction assumes that a node in that condition may carry accumulated state contributing to its problem — a rebuild discards it, at the cost of a slower return.

Global runtime state restoration. Recovered components rejoin a system with coherent shared state rather than reconstructing it independently and disagreeing.

The governing disposition is conservatism, stated explicitly: Phoenix performs more restarts than strictly necessary, never fewer. Restarting something that did not need it costs milliseconds. Failing to restart something that did costs the recovery, and the failure is discovered later, in a system everyone believes has recovered.

Interactions

Punisher → Phoenix → (system returns to normal operation)

Phoenix is the last protocol in the ordinary path. Beyond it the chain contains only Hyperion and Terminus, which are not part of normal operation.

Its output is observed by Sentinel on subsequent cycles, which is how recovery is confirmed: the system does not assert that it has recovered, it observes that it has.

Failure modes

Restart loop. A component that fails on restart is restarted again. Without a bound this consumes the recovery indefinitely; whether Phoenix bounds its own attempts is not stated in the available sources and should not be assumed.

Restored state that is invalid. State captured during degradation may encode the degradation. Restoring it returns the system to the condition it was recovering from — the failure mode a rebuild is designed to escape, which is why the load threshold exists.

Recovery under continuing stress. Bringing components back into a system still under pressure adds load to something already struggling. Sequencing limits the rate; it does not remove the risk.

Design tradeoffs

Deliberate over-restarting. Recovery is slower than the minimum, always. The alternative — restarting exactly what is needed — requires knowing exactly what is needed, and being wrong is discovered late and cheaply hidden. This is a preference for a small known cost over a rare large one.

Rebuild versus restart is decided by load. A single proxy for a judgement about accumulated state. It is a reasonable heuristic and it is a heuristic.

Future evolution

Verified recovery — confirming a component is actually healthy before proceeding to the next in the sequence, rather than inferring it from a successful restart — is the clearest direction. Partial recovery, restoring the most critical path first and the remainder afterwards, is the second. Both are inference.