MLOps: from a notebook to a reliable production lifecycle

A model file is only one part of an ML system. The prediction also depends on data collection, feature definitions, preprocessing, dependencies, deployment configuration, and the environment in which users act on its output.

Consider a model that predicts whether a customer will cancel a subscription within thirty days. A good notebook score does not answer whether the features will exist at prediction time, whether labels are mature enough to evaluate, whether the deployed model receives the same transformations, or what the team should do when performance changes. MLOps is the discipline of making that entire lifecycle repeatable, observable, and recoverable.

Define the decision and its information boundary

Start with a prediction contract. Specify the entity, prediction time, target, label horizon, feature availability, output, and intended action. For example: at the beginning of each day, score active customers using events available before midnight, predicting cancellation within the next thirty days.

The action matters. If a retention team can contact only five hundred people a day, precision or expected benefit among the top five hundred may be more useful than overall accuracy. If the intervention changes outcomes, historical labels no longer describe an untouched process. Evaluation must account for how predictions affect the world.

Write down an ordinary baseline: a simple rule, a regularized linear model, or the current business process. An elaborate pipeline should improve a meaningful decision enough to justify its maintenance cost. “The model is deployed” is a delivery milestone, not evidence of value.

Build datasets that respect time

A training example needs the features that would have been available at its prediction time. Joining every historical customer row to today’s customer profile leaks later information into earlier examples. A cancellation flag added after the outcome is especially damaging, but subtler leaks arise through revised account status, late-arriving events, or aggregates whose windows extend past the prediction time.

A point-in-time join selects feature values that were valid and available at the historical cutoff. Distinguish event time from ingestion time when delayed records matter. The Feast documentation on point-in-time joins describes one implementation approach; the temporal requirement exists even if you do not use a feature store.

Check the implementation’s actual time filters. An event-time join can still admit a value corrected or backfilled after the historical prediction. Reproducing what was available then may require an additional ingestion-time condition; support and defaults differ between stores.

Labels also need time to mature. If the outcome window is thirty days, a customer scored yesterday cannot yet be labeled as “did not cancel within thirty days.” Exclude immature outcomes or model the censoring problem explicitly. Otherwise the most recent data may look artificially negative.

Split by time for the future-prediction task, and by customer where the evaluation question requires generalization to unseen customers. Fit imputers, encoders, scalers, and feature selection on the training portion only. A pipeline that stores a model but loses its fitted preprocessing has not preserved the prediction function.

Version the complete experiment

An experiment should identify the source-code revision, data snapshot or reproducible query, feature definitions, split boundaries, configuration, dependency environment, random seeds, and produced artifacts. Record evaluation results with the same provenance.

A seed is helpful but not a guarantee of bit-for-bit reproducibility across different devices, parallel execution paths, and library versions. Decide which level of reproducibility is needed: exact artifacts, numerically similar results, or a reproducible scientific conclusion. Save the trained artifact when reproducing a deployment exactly matters.

A useful artifact manifest is a small machine-readable document:

{
  "model_id": "churn-candidate-017",
  "code_revision": "example-source-revision",
  "dataset_snapshot": "customers-2026-09-30-v3",
  "feature_schema": "churn-features-v4",
  "target": "cancel_within_30_days",
  "training_cutoff": "2026-08-31",
  "evaluation_window": ["2026-09-01", "2026-09-30"],
  "preprocessing_artifact": "preprocessor.pkl",
  "model_artifact": "model.bin",
  "metrics_artifact": "evaluation.json"
}

These are illustrative identifiers, not a record of a real training run. In a real system, add checksums, the resolved environment, ownership, and access controls. Treat model artifacts and serialization formats according to their trust requirements; loading an arbitrary serialized Python object can execute code.

Make the pipeline a sequence of explicit gates

A useful pipeline separates ingestion, validation, feature construction, training, evaluation, packaging, and promotion. Each step produces named artifacts and has a clear reason to fail.

flowchart LR
  Snap["Data snapshot"] --> Checks{"Schema and quality checks"}
  Checks -- pass --> Feat["Temporal features"]
  Checks -- fail --> Stop["Stop: keep current model"]
  Feat --> Train["Training"]
  Train --> Eval{"Offline evaluation"}
  Eval -- fail --> Stop
  Eval -- pass --> Pack["Packaging"]
  Pack --> Gate{"Approval or policy gate"}
  Gate -- approved --> Deploy["Controlled deployment"]
  Gate -- rejected --> Stop

Data validation checks types, missingness, allowable ranges, key uniqueness, freshness, and relationships between fields. A schema-compatible dataset can still be semantically wrong: a unit changing from rupees to thousands of rupees may preserve the numeric type. Add checks tied to domain meaning and compare distributions with an appropriate reference window.

Training success means the optimization process completed, not that the model is acceptable. A failed evaluation must leave the currently deployed version intact. Pipeline retries should reuse or regenerate artifacts deliberately rather than silently mixing outputs from different data snapshots.

Google Cloud’s MLOps architecture guidance distinguishes code delivery, data validation, model validation, and automated training workflows. The specific tooling is optional; the separation of concerns is useful across platforms.

Evaluate decisions and important slices

For a binary classifier, a probability of $0.8$ has a stronger meaning than a high ranking score. If the output is used as a probability, check calibration: among comparable predictions near $0.8$, is the observed event rate near 80% once labels mature?

Ranking quality, probability calibration, and thresholded decisions are different properties. A model can rank customers well while producing overconfident probabilities. It can also improve overall discrimination while becoming worse for a small but important customer segment.

For an illustrative threshold policy, expected benefit per selected customer might be represented as:

$V(x)$ is the benefit of preventing cancellation and $C(x)$ is the intervention cost. The probability of cancellation is not the same as the probability that an intervention prevents it. A churn classifier alone does not estimate treatment effect. To measure intervention value, design an experiment or another defensible causal evaluation.

Compare candidates with the current model on a fixed evaluation contract. Include important slices, uncertainty in the estimates, latency, memory, and operational constraints. Repeatedly tuning against one benchmark can overfit the evaluation process itself, even if the model never directly trains on those rows.

Choose serving around the decision’s timing

Batch inference suits decisions made periodically, such as a daily retention worklist. It can simplify infrastructure and produce reproducible outputs for a known snapshot. Online inference is appropriate when a fresh response is required during an interaction, but introduces latency budgets, feature-fetch failures, and concurrency concerns.

Package preprocessing with the model or establish a rigorously shared feature contract. A difference in category mapping, timezone, missing-value handling, or aggregation window can create training-serving skew even when the model weights are identical.

Validate request schemas and model compatibility at the serving boundary. Define behavior for missing features and unavailable dependencies. A fallback may be a simpler model, a last known score with a freshness limit, or a decision to abstain. Pick the fallback according to the product’s risk, not just its convenience.

A model registry can organize versions, metadata, and deployment references. The MLflow registry documentation describes such capabilities. A registry entry is not a substitute for an evaluation gate or a rollback plan.

Deploy a model as a controlled experiment

A shadow deployment runs a candidate on real requests without allowing its predictions to control the user-facing decision. It helps check input compatibility, performance, and output differences, but it does not reveal the business effect of acting on those predictions.

A canary routes a limited share of eligible traffic to the candidate. Set technical and model-behavior guardrails, and define rollback conditions in advance. An A/B experiment can estimate the effect of a policy under suitable assignment and analysis, but it requires attention to interference, sample size, delayed outcomes, and consistent treatment allocation.

Keep the previous compatible model and its preprocessing available. Rolling back only the weights may fail if a feature schema has changed. Plan for code, feature, and data compatibility together. Record which model served each prediction so delayed labels can be joined to the correct version.

Monitor three different layers

Service health includes latency, error rate, resource usage, throughput, and feature-fetch failures. These signals can identify an unavailable system, but not a subtly inaccurate one.

Data health includes missingness, freshness, category changes, range violations, and distribution shifts. A changed distribution is evidence that something changed, not proof that prediction quality deteriorated. A successful marketing campaign can change the customer mix while the model remains useful.

Outcome quality includes metrics computed after labels become available, segmented by model version, prediction time, and important groups. In the thirty-day churn task, genuine outcome monitoring has a delay. Short-term proxy signals may help detect problems sooner, but should not be relabeled as measured ground-truth performance.

Distinguish a change in $p(x)$, the input distribution, from a change in $p(y\mid x)$, the relationship between inputs and outcomes. Observing the first does not establish the second. Model outputs and business interventions can also create feedback loops that influence future training data. The paper Hidden Technical Debt in Machine Learning Systems is an important starting point for understanding these system-level dependencies.

Retrain for a reason, then validate again

A scheduled retraining job can keep a model current, but automatic retraining is not automatic improvement. New data may be incomplete, mislabeled, or affected by a broken pipeline. Distribution changes may require a feature correction or a revised target rather than another optimization run.

Use explicit triggers: sufficient new mature labels, a known regime change, a sustained outcome-quality issue, or a planned experiment. Every candidate still passes data checks and the same promotion contract. Keep the ability to stop training or promotion independently of the serving system.

Record a model’s owner, supported use, limitations, data lineage, evaluation windows, and retirement criteria. Operational ownership includes responding when labels stop arriving, a data provider changes meaning, or a downstream team starts using the score for a new decision.

Practice: recover from a deliberately broken pipeline

Build a small batch-scoring pipeline with versioned inputs, fitted preprocessing, a model artifact, and a prediction manifest. Add a temporal validation split and a simple baseline. Make the pipeline reject a missing required feature and refuse to promote a candidate that fails its evaluation gate.

Then introduce three failures: a delayed source table, a category mapping that differs between training and inference, and an evaluation window containing immature labels. Each should produce a specific diagnostic rather than a mysteriously worse dashboard score.

Finally, deploy two versions to a local serving environment and roll back the candidate while retaining compatible preprocessing. A reliable ML system is not merely able to train a model. It can explain which data produced an answer, recognize when its assumptions no longer hold, and recover without guessing.