Research · AI Infrastructure & Operations

MLOps & LLMOps

MLOps applies software engineering and operations discipline to machine learning. LLMOps extends it to generative AI, adding prompts, retrieval pipelines, guardrails and model providers to what must be versioned, tested and monitored.

The LLMOps Control Stack

MLOps gave teams versioned models and automated release pipelines. Generative AI and agents add parts that change system behavior without any code change: prompts, retrieval indexes, tool connections and the model provider itself. The CorpExcellence LLMOps Control Stack groups these into six layers. Each layer needs an owner and a change process, and one trace runs through all of them so any output can be explained after the fact.

Agents and applications

The top layer is what users touch: assistants, workflows and agents. Agents call tools, more and more often through the Model Context Protocol (MCP). The list of tools an agent may use is configuration, and it deserves the same review as code. Log every tool call with the identity the agent acted under.

Release control

Ship changes in stages. Run a new prompt or model on a slice of traffic and compare results with the current version. Keep a one-step rollback to the last known-good configuration. Treat a provider’s model upgrade as a release too, even though you did not write it.

Evaluation gates

This is the layer most teams skip and later regret. Every change to a model, prompt, retrieval setting or tool definition runs against a versioned test set before release. Gates block regressions automatically, and people review the cases an automated check cannot judge. Keep the test sets under version control as well, since a changed test set changes what passing means.

Model gateway

Route all model calls through one gateway. It holds credentials, applies quotas and fallbacks, records which model version answered, and lets you move between providers without rewriting applications. Gateways are now a standard product category, offered both by cloud providers and by open-source projects.

Asset registry

Keep models, prompts, retrieval index versions and tool definitions in one registry, with a record of which version is live in each environment. Pin specific provider model versions instead of floating aliases that can change underneath you.

Data pipelines

Training data, documents, embeddings and retrieval indexes need lineage and access controls. A stale or over-shared retrieval index is an LLMOps failure even when the model itself is fine. Re-index on a schedule and remove documents that users should no longer see.

Is your gateway recording what each request costs?

AI Inference Cost Control includes an inference cost data dictionary, a model routing policy template and an architecture review checklist for production AI.

Get the full report →

Where the industry is now

Tracing for agents is moving toward open standards. OpenTelemetry’s GenAI semantic conventions now define model spans, agent spans and MCP spans, but as of September 2026 they remain in Development status, so attribute names may still change. MCP itself moved to neutral governance when Anthropic donated it to the Agentic AI Foundation under the Linux Foundation in December 2025. A practical requirement for any LLMOps tool you buy: it must export traces in OpenTelemetry format, so you are not locked into one vendor’s view of your system.

Common pitfalls

• Editing production prompts directly, outside version control.
• Logging outputs without the model version that produced them.
• Letting each team pick its own gateway and tracing tool.
• Granting agents broad tool access during a pilot and never narrowing it.