The data pipeline

VECTOR archive › Architecture › The data pipeline

Most of the difference between a model that works in evaluation and one that works in production is decided before any model is trained, in how the data was split. The data pipeline is where that decision lives.

Problem

Traffic is a time series, and time series break the assumption that most machine learning tooling is built on. Randomly partitioning observations into training and test sets lets the model learn from the future and be evaluated on the past. The resulting scores are excellent and meaningless.

The failure is silent, and it is flattering, which is a bad combination.

Context

The environment is non-stationary — see the philosophy on why variable environments are named in the system's definition. Conditions have regimes, and phenomena occur at more than one timescale: a signal cycle, a rush hour and a seasonal pattern are all real, and a single sampling resolution captures at most one of them well.

Constraints

Evaluation has to reflect deployment, which means training only on the past. Multiple timescales have to be available simultaneously. And the output has to be consumable by the training code without a second transformation step where assumptions can diverge.

Alternatives considered

Random splits. Standard, convenient, and invalid for time series for the reason above.

A single held-out tail. Train on everything before a date, test after it. Honest, and it evaluates the model against exactly one period — so a model that happens to suit that period looks better than it is.

Chosen architecture

DataPipeline in core/data_pipeline.py, built around three stages.

Multi-resolution resampling (resample_multi_resolution) produces several resolutions of the same series rather than committing to one, so phenomena at different timescales remain visible.

Feature engineering (engineer_features) operates per resolution and returns an EngineeredDatasetBundle — the features and the metadata describing them travel together, which is what stops a feature set and its description drifting apart.

Walk-forward splitting (walk_forward_splits, producing WalkForwardSplit) trains on a window and evaluates on the period immediately following it, repeatedly, advancing through the series. Every evaluation is a genuine forecast, and the model is scored across many regimes rather than one.

The result is exposed as a TimeSeriesTorchDataset via build_torch_dataset, so the boundary between preparation and training is a typed dataset rather than a convention.

Subsystem relationships

Feeds the learned components in learning/ and models/. It does not feed the governance stack: the protocols operate on live telemetry through the event bus, not on engineered datasets. Training and operation are deliberately separate paths.

Data flow

raw series → resample_multi_resolution → {resolution: frame}
           → engineer_features → EngineeredDatasetBundle
           → walk_forward_splits → [WalkForwardSplit, …]
           → build_torch_dataset → TimeSeriesTorchDataset

Tradeoffs

Walk-forward is expensive. Many train/evaluate cycles instead of one. The cost is compute, and it buys an estimate that means something.

Multi-resolution multiplies the feature space, which is in tension with the reduction argued for in seven variables. Resolution breadth and variable count are separate axes, and the resolution of that tension is not visible from the pipeline alone.

Recency versus regime coverage. A model validated across many regimes is not necessarily the best model for the current one.

Failure modes

Leakage through a feature. Walk-forward prevents leakage through splitting; a feature computed over the whole series before splitting reintroduces it, and the split strategy cannot detect that.

Insufficient history. Walk-forward needs enough series to produce meaningful windows; short histories yield few splits and a noisy estimate.

Regime change after training. The pipeline measures performance across past regimes. It cannot say anything about a regime that has not occurred — which is why the runtime system has VIHAAN rather than relying on validation.

Future evolution

The clearest gap is a feature store with explicit point-in-time semantics, which would make leakage through feature computation structurally impossible rather than a discipline. Inference, not a documented plan.