A trained model is an artefact, not an operating service. Its behaviour depends on data pipelines, feature definitions, configuration, serving infrastructure and the systems that consume its predictions. Those dependencies continue to change after deployment. A model can therefore pass every offline test and still fail silently in use.
MLOps addresses this gap when teams treat it as an operating model rather than a deployment toolchain. The model joins a controlled lifecycle: observe the whole decision path, preserve the evidence needed to reproduce behaviour, assign responsibility across hand-offs and validate every intervention. Automation supports these disciplines; it does not replace them.
Operate the decision system, not the model file
The useful operational boundary is the service that turns inputs into actions and outcomes. That boundary includes upstream producers, transformations, the model, decision thresholds and downstream consumers. A prediction logged to a table may feed another pipeline just as surely as a prediction returned by an API. Undeclared consumers make even a beneficial model change hazardous because its effects escape the tests written for the original use case.
This wider boundary also prevents a category error. A serving endpoint can return results within its latency target while a broken feature makes those results useless. Conversely, an accurate model can miss the service’s latency or availability requirements. Production readiness needs evidence about both computational health and decision quality.
Start with a contract for the system. Name the input producers and their obligations, the feature semantics, the prediction consumers, the action taken on each output and the metric that shows whether that action remains useful. The contract exposes where an alert should go and which team can change the relevant behaviour.
Instrument four distinct views
Useful telemetry separates four signals that are often collapsed into a single “model monitoring” dashboard.
Inputs need structural and statistical checks. Validate schemas, types, units, ranges, missingness and category frequencies before inference where practical. Compare distributions with a relevant reference window, not merely with an arbitrary training extract. Also monitor freshness and volume: an unchanged distribution of stale records is not a healthy feed.
Predictions reveal how the deployed artefact behaves. Record score distributions, predicted classes, confidence where meaningful, decision-threshold crossings and the actions taken. Slice these measures by important cohorts and by model version. A stable aggregate can conceal a sharp change in one region or customer group.
Service health covers request rates, errors, latency, resource use, dependency failures and queue depth. These measurements belong to ordinary service operations, but ML adds details such as feature-lookup latency and model-loading failures. Instrumentation must not turn prediction into an unbounded logging workload; asynchronous export, sampling and retention limits manage that cost, at the price of reduced visibility.
Downstream quality joins predictions to labels or outcomes when they become available. Measure the task’s actual loss and the operational result, such as false approvals, manual overrides or wasted reviews. Label delay, selective labelling and noisy annotators complicate this view. If only cases selected by the model receive labels, the resulting sample cannot describe performance on cases the model rejected.
Join these views with a request or event identifier and attach the model, pipeline and configuration versions to each observation. Otherwise an outcome may arrive weeks later with no reliable way to identify the decision that produced it. High-cardinality identifiers and raw payloads can create cost and privacy problems, so define what to retain, who may read it and when to aggregate or delete it.
Version the evidence behind behaviour
Source control alone cannot reconstruct a model. The same code can produce different artefacts when the training extract, feature transformation, dependency version or random seed changes. Configuration deserves equal scrutiny: data selection, hyperparameters, thresholds and post-processing can alter behaviour without a code change.
Give each training run an immutable record linking code, input data, configuration, environment, evaluation results and output artefacts. A deployment record should identify the promoted model and the serving configuration. Data snapshots or content-addressed references make the lineage concrete; an unversioned pointer to “latest” does not.
Reproducibility has limits. Parallel training and external services can introduce nondeterminism, while retaining every intermediate dataset costs storage. Record seeds and dependencies, then test whether the pipeline can reproduce results within an accepted tolerance. Retain the artefacts and lineage needed for rollback and investigation rather than promising bit-for-bit reproduction everywhere.
Versioning also changes incident response. A team can compare a suspect release with its predecessor, replay a fixed evaluation set and restore a known configuration. Without that chain, rollback becomes guesswork even if an old model binary remains in storage.
Give every interface an owner
Shared responsibility fails when an alert has no authorised recipient. Assign ownership at the interfaces where causes and remedies differ.
Upstream data owners handle producer outages, schema changes and semantic drift. ML owners maintain feature logic, training evaluation and model-quality investigations. Service owners handle capacity, latency, deployment and rollback. Product or domain owners define acceptable decision errors and judge downstream outcomes. One person may hold several roles in a small team, but each role still needs a named recipient and an escalation path.
The runbook should connect alerts to those roles. A missing-feature alert belongs first with the producer or feature pipeline, not with an automatic training job. A latency breach may require scaling or a smaller model. A fall in downstream quality demands joint investigation because the cause could sit in the model, the policy using its scores or the outcome measurement itself. Incident review should then update contracts, tests or ownership rather than stopping at the repaired component.
Drift is evidence, not a diagnosis
Terminology varies, so an operating team should define its terms mathematically. Data drift here means a change in the input distribution, p(X). Concept drift means a change in the relationship between inputs and targets, p(y|X). Performance degradation is an observed increase in task loss or a deterioration in the chosen downstream measure.
These conditions are related but not interchangeable. Seasonal inputs can move p(X) while the learned relationship remains valid. A change in user behaviour can alter p(y|X) with little visible movement in the inputs. A broken unit conversion can cause both input drift and poor predictions, but the remedy is to repair the conversion. Representative labelled outcomes, or a defensible downstream proxy, are needed to establish actual degradation.
Drift detectors therefore triage attention; they do not measure accuracy by proxy without assumptions. Choose reference and comparison windows that match the process, inspect important slices and account for normal seasonal movement. Sensitive tests on large samples can flag trivial differences, while repeated tests across many features create false alarms. Alert thresholds should reflect the cost of investigation and the cost of missing a consequential change.
Retraining is a candidate change
Retraining addresses a correctable deficiency in the model, not every operational fault. New, representative labelled data may supply the missing evidence; they do not guarantee an improvement. Retraining cannot repair a failed producer, reconcile different training and serving feature code, or correct a downstream policy that applies the wrong threshold. Training on corrupted recent data can instead preserve the fault and amplify a feedback loop.
Treat a retrained model as a release candidate. Validate it against the current model and a simple baseline on data that respects time order. Check aggregate quality, critical slices, calibration and decision thresholds, then test serving compatibility and resource requirements. A canary or controlled online comparison provides evidence under production conditions. Keep the previous version available until the candidate meets the promotion criteria.
The same gates apply to scheduled retraining. Schedules suit predictable label arrival and changing processes, but they do not prove that fresh data is representative or that the candidate is better. Automatic training with manual promotion can be a sound division of labour; automatic promotion needs especially strong tests and a safe rollback path.
An effective MLOps practice can start small: one system contract, four telemetry views, a versioned release record, named owners and a runbook that separates diagnosis from action. More orchestration is justified only when it strengthens that control loop. The goal is not continuous model replacement, but continuous evidence about whether the decision system still deserves to run.
Sources and further reading
- D. Sculley et al., Hidden Technical Debt in Machine Learning Systems (NeurIPS 2015): system boundaries, undeclared consumers, configuration debt, feedback loops and live monitoring.
- Eric Breck et al., Data Validation for Machine Learning (SysML 2019): schemas, training–serving skew, actionable alerts and the limits of sensitive distribution tests.
- Janis Klaise et al., Monitoring and Explainability of Models in Production (ICML workshop, July 2020): label-dependent quality, input and output proxies, drift detection and monitoring architecture.
- Cedric Renggli et al., A Data Quality-Driven View of MLOps (February 2021): distribution-shift definitions, data quality and statistically careful model validation.
- Georgios Symeonidis et al., MLOps—Definitions, Tools and Challenges (January 2022): lifecycle components, data and experiment versioning, monitoring and the practical barriers to retraining with delayed labels.