MLOps

The first model is quickly built. The hard part comes after. The training ran once on a laptop, nobody is sure what data it used, and nobody checks whether the model still gives good results today. MLOps applies proven software engineering principles to machine learning: pipelines prepare data, train, test and deploy, every version is traceable, and in operation input data, prediction quality and costs are monitored. The same applies to applications built on language models, for which the term LLMOps has caught on. Model development itself is covered under Machine Learning Development.

How you notice it is missing

MLOps usually becomes an issue when several of these apply.

• A model runs in production, but nobody can say what data it was trained on.
• Retraining takes a while because only one person can do it by hand.
• Predictions get noticeably worse, and it only comes to light when business teams complain.
• There are several copies of the same model on different machines.
• A prompt or model in an LLM application was changed, and nobody knows whether the answers got better or worse.
• The bill for compute or language models keeps rising without anyone being able to say what for.

What belongs in from the start

Experiment tracking records the parameters, data, metrics and results of every training run, so experiments can be compared and repeated. A model registry keeps the model versions with their status, such as “in review” or “production”, and points to the data and code the model was built from. Together with versioning of data and code, you can always say which combination produced a given result. A CI/CD pipeline sends changes through automated tests and releases them in controlled steps.

A feature store, a shared place for computed features so that training and production use the same values, only pays off once several models need the same features. We make the cost of compute, storage and, for language models, tokens visible per model and application.

Releasing a new version without guessing

We keep development, testing and operation separate. Before a model enters the registry, it passes automated tests: for data quality, for accuracy on a fixed test set and for technical requirements such as response time. Deployment goes through the pipeline, not by hand.

For the rollout, shadow mode has proven its worth. The new model gets the same inputs as the old one, and its predictions are compared but decide nothing. After that a small share of requests goes to the new version, and if the metrics look right, the share is increased. The last working version stays ready so that switching back is quick if something goes wrong. The logs from these steps also count toward documentation duties. Which documentation and logging the EU AI Act requires for which systems is covered under EU AI Act.

When the model gets worse

A model loses quality in operation when reality changes, even if nothing happens to the model. So we monitor several levels. Data quality checks catch missing values, new categories or failed source systems before they affect predictions. With drift, the distribution of the input data or the relationship between inputs and outcome shifts. Prediction quality can be checked once the actual outcomes are known. If they arrive late, proxy metrics help. Response time, error rates and compute costs are watched like for any other application.

When a threshold is crossed, that triggers an alert or a new training run. Whether the result then goes live directly or is reviewed first depends on the risk. A model that retrains itself on faulty data gets worse quickly, so a check belongs before release, and for decisions with bigger consequences a sign-off by a person.

Running language model applications

With language model applications you often do not train a model of your own. Still, things keep changing: prompts, the providers’ model versions, knowledge sources, tool connections. We version prompts and configurations like code, so changes can be traced and rolled back. A fixed set of typical questions with expected answers checks every change automatically, complemented by ratings from language models or people. A “small” prompt change often has big effects, and without this test nobody notices.

With tracing, every request is logged with its intermediate steps, from document retrieval and tool calls to the answer, so errors can be followed back. Token usage and response times are evaluated per application. And when a provider replaces or retires a model, the evaluation shows whether the application still works well with the new one. How assistants and custom applications are chosen and designed is covered under Generative AI for Business. This page is about running them afterwards.

Starting small

If you begin with one model, you do not need infrastructure for a hundred. Tracking, registry, tests and monitoring can be built with open tools such as MLflow, and the pipelines usually run in the CI/CD tools the team already has. The platforms of the cloud providers or of Databricks pay off when many teams and models come together. Clean interfaces matter more than any single product, because vendors rename products or drop features, and then one part should be replaceable without rebuilding everything.

Two things are almost always missing in practice. One is a fixed test set with known results, without which you cannot tell whether a new version is better. The other is somebody who reacts to an alert. So we decide who acts on which alarm while setting things up.