Putting a machine learning model into production used to be the most difficult aspect of AI engineering. Teams struggled with developing training pipelines, writing deployment scripts and building monitoring dashboards. MLOps emerged as an answer to this problem; it provided a framework for creating repeatable processes around machine learning models.
However, large language models (LLMs) have changed the game.
While it has become relatively simple to get a model to produce some kind of response, producing a quality response one that is safe and cost-effective has emerged as the next challenge. The difference between producing a response and producing a high-quality response is what gives rise to the current popularity of LLMOps in the engineering community.
However, LLMOps is not intended to replace MLOps. Rather, LLMOps is built upon MLOps, and recognizing the relationship between these two areas of focus can help teams avoid unnecessary spending due to misunderstanding.
A Quick Refresher on MLOps
An Overview of MLOps Machine Learning Operations (MLOps) refers to the collection of best practices used to transition a machine learning model developed in an interactive programming environment (such as a Jupyter Notebook) into a reliable production system.
The Google MLOps Framework outlines various stages of maturity for organizations using MLOps; these range from manual, one-off processes to completely automated systems for training and deploying models. The main components of MLOps include:
- Collecting, validating, and managing versions of your data
- Creating features and training models
- Tracking experiments and registering your models
- Developing Continuous Integration/Continuous Delivery (CI/CD) pipelines for your models and pipelines
- Deploying, monitoring, and detecting drift in your models
- Training new models if your current ones start performing poorly
- Consider a bank’s credit risk model, or a retail store’s demand forecasting.
- Input data is structured.
- Output data is numeric or categorical.
Accuracy can be evaluated by comparing predicted outputs with actual outputs.
Where LLMs Break the Old Assumptions
The “old” assumptions about how to develop applications using LLMs don’t hold true anymore. The outputs of LLMs are open-ended textual responses. The “training” process for many LLMs was done by someone else. Judging the quality of an LLM’s output is much more difficult.
For example, consider a customer support assistant. There is no one correct answer to use as a baseline for comparing responses. Responses may be factually correct but off-brand; responses may provide assistance but pose risks; responses may appear well-written but were invented. Teams will need to take these changes into account when managing their projects.
When developing LLM applications, there are several components that need to be managed:
- Prompts: These are treated like code and versioned, tested, and reviewed.
- Retrieval Systems: These systems involve vector databases, chunking strategies, and keeping documents up to date.
- Model Choice: Teams can switch between hosted models and open-source models based on cost and quality.
- Guardrails: These are used to filter out sensitive information, dangerous content, and irrelevant requests.
- Evaluation: This involves using rubrics for scoring, human reviews, and automatic checks on free-form outputs.
- Cost and Latency: Token usage becomes a legitimate expense. Most people refer to all of the above practices combined as LLMOps development.
MLOps vs LLMOps at a Glance
| Area | MLOps | LLMOps |
| Main asset | Trained model and features | Prompts, retrieval, model and orchestration |
| Output | Predictions, scores, labels | Open-ended text, actions, structured responses |
| Evaluation | Accuracy, precision, recall | Groundedness, relevance, safety, human ratings |
| Common failure | Data or concept drift | Hallucination, prompt injection, inconsistent tone |
| Cost driver | Training and serving compute | Token usage and API calls |
| Feedback loop | Retrain on new data | Refine prompts, retrieval and evaluation sets |
The table simplifies things, of course. In real projects the lines blur quickly.
It’s an Evolution, Not a Replacement
A very common misconception about LLMOps is that it has made MLOps redundant. The truth is many large-scale applications utilize both.
For example, a fraud detection system could employ a classic machine learning model to assign scores to each transaction; while at the same time utilizing an LLM to generate summaries of the cases where a transaction has been flagged.
The first part requires traditional Machine Learning Operations (MLOps); however, the second part will require LLMOps. If either of these components are missing from the final product, the product may appear to be incomplete.
The fundamentals of MLOps also apply directly to LLM-based applications. These include versioning, reproducibility, automated testing, monitoring and access controls. All of these factors are equally important to developing a reliable LLM-based application as they are to developing a reliable Gradient Boosted Tree Model-based application.
Therefore, organizations who have developed strong MLOps practices typically find adapting to LLM applications much easier since they are accustomed to working with pipelines and feedback loops.
Where Teams Fall Short: Treating Prototypes Like Products
A chatbot using an API looks great for a demonstration, but without evaluation and monitoring no one knows if its performance will hold up when actual users begin to use it.
- Failing to Evaluate: When success is based on “it feels okay”, any changes to the prompt or model result in regression.
- Allowing Costs to Grow Unchecked: Long prompts, repeated calls and large models may take what was once a promising feature and turn it into a costly problem.
- Becoming Locked-In to a Single Vendor: Models develop over time and vendors’ prices change. Architectures which make switching models difficult limit your options.
- Forgetting Security: Security issues such as prompt injection and data exposure are engineering issues from the very beginning of the project and do not need to be reviewed at a later date.
What a Healthy LLMOps Practice Looks Like
Looping teams through defining tasks & success criteria, building an honest evaluation set, testing prompts & retrieval against the evaluation set, deploying behind guardrails, monitoring real usage, collecting failures & feeding them back into the evaluation set.
Every cycle makes the system slightly more reliable; everything is based upon data and not intuition. Observability is key to this process. By logging prompts, retrieved context, responses, latency, and cost, we can make problem-solving systematic rather than intuitive.
The Skills Behind Modern AI Engineering
The skills for an engineer to be successful in today’s modern world of Artificial Intelligence (AI) are the reason why we see AI Engineering Services today as a blend of many different types of skills such as Data Engineering, Model Evaluation, Cloud Infrastructure, Security, Product Thinking.
Although it would be rare to find a single individual with knowledge of each area; the team will collectively cover them. When you are looking to hire AI ML Developers, you should focus on their experience with building working systems that have gone through the process of shipping beyond the “demo” phase. You should also ask if they understand how to evaluate outputs from non-deterministic models.
Finally, you should inquire about their experience with monitoring, retraining and prompt versioning in a production environment.
Starting with One Project
Not all organizations need an entire LLMOps infrastructure on Day 1. The best way to begin is by picking out one high-impact use case that would benefit greatly from LLMOps.
Once you’ve identified that use case, determine your definition of success for that particular use case. Then develop enough monitoring and evaluation tools to ensure that you are confident in the performance of the LLM in this specific use case.
From there, you can standardize these patterns and apply them to subsequent projects. Existing MLOps teams will be able to leverage their current MLOps capabilities and develop them further.
New MLOps teams will be able to utilize simple tools to support their development efforts and eventually mature those tools into something larger.
Final Thoughts
MLOps has shown the industry that models are living systems, not just a one-off delivery. LLMOps takes this concept further by showing how in the world of LLMs the definition of “model” now includes prompts, retrieved knowledge and open-ended behavior.
Organizations who view both as part of an ongoing engineering discipline will release AI products that remain effective long after their initial release. Businesses seeking experienced assistance in MLOps development services, LLM applications and production AI can utilize an integrated approach to AI engineering that combines product and operations from initial prototype development through continued support.
