From Predictable Programs to Probabilistic Systems: What Changes When Software Uses AI?

A conventional expense application can reject a claim because its amount exceeds a defined limit. A language model can read the employee's explanation and suggest how to classify it. Both can be useful, but they provide different kinds of assurance. The first implements an explicit rule. The second produces an interpretation that needs to be evaluated against the organization's actual policy. The first behaves like a predictable program; the second makes the application part of a probabilistic system.

This difference changes how an application should be designed, tested, and supported. It does not mean that conventional software is always predictable or that every model response varies. It means that the basis for trusting an output changes when learned behavior becomes part of the decision path.

What a language model contributes

Google's machine-learning material explains language models in terms of predicting tokens from context. Tokens are units used to represent text, and transformer architectures use attention to process relationships within that context. ^[1] Training creates learned parameters that influence which outputs the model generates.

At runtime, the application supplies instructions and information, and a decoding process selects an output. Sampling settings can affect variation. Even a repeatable answer can be wrong, however, because repeatability is not the same property as factual accuracy or suitability for a business process.

A useful distinction is between a calculated result and an interpreted result. If a fee must equal a stated percentage of an amount, conventional code can enforce that calculation. If a complaint must be summarized, a model may provide flexibility that an exact rule cannot easily express. A single application can use both.

The application still needs firm rules

Take an invoice service as an example. A model extracts a supplier name and proposes an expense category from a scanned document. Conventional code checks the supplier identifier, validates the amount, and determines whether the user has approval authority. A separate workflow routes ambiguous cases for review.

This design gives each component a specific responsibility. The model does not gain payment authority simply because it can describe a payment. The output must pass checks appropriate to the action, and uncertain extraction should not silently become a confirmed accounting record.

The same principle applies to generated database queries, configuration changes, and support responses. The application should distinguish proposing a result from authorizing its use. That boundary is an architectural decision that deserves an explicit place in the design.

Context becomes part of the behavior

An answer can change when the instructions, retrieved documents, conversation history, or available tools change. Anthropic's context-engineering guidance describes context as a finite resource and applies the term to everything the model receives, not only the prompt. ^[2]

For an enterprise application, this creates practical questions. Which policy version was supplied? Did the user have access to the retrieved passage? Was a previous conversation still relevant? Did the system receive the complete record? Troubleshooting requires visibility into the information used for the particular interaction. Our model evaluation and monitoring research goes further on this topic.

Record identifiers for important inputs and configuration versions while limiting sensitive content in logs. The purpose is to make failures explainable enough to investigate. Recording everything indefinitely creates a separate information-management problem and is not a substitute for a deliberate logging design.

Testing must allow variation without accepting errors

A summary can use different words and remain correct. A payment amount cannot be approximately correct merely because the wording is fluent. Test criteria should separate acceptable variation from requirements that admit no discretion.

For example, a team could evaluate whether a summary includes the relevant deadline, avoids unsupported claims, identifies conflicting evidence, and links to the correct source. A different test should confirm that an unauthorized user cannot access the underlying document. Combining these into a single satisfaction score would conceal distinct failure types.

Microsoft's RAG evaluation guidance separates assessment of generated answers into dimensions that include groundedness, relevance, and completeness. ^[3] These dimensions are useful reminders that a readable answer can still omit a critical qualification or rely on evidence that does not support its conclusion.

Uncertainty needs an operational response

NIST's Generative AI Profile identifies confabulation, including confidently stated false content, as a risk requiring attention. ^[4] A disclaimer beneath the answer cannot do all the work of managing that risk. The application needs behavior for cases where evidence is missing or the request exceeds its intended scope.

A practical design might return the relevant documents without a synthesized conclusion, request a missing identifier, or transfer the case to a specialist. The appropriate response depends on the service. An inability to answer can be the correct result when the alternative is an unsupported decision.

Avoid treating a model's verbal expression of confidence as a calibrated probability. If the application needs a threshold for automatic processing, establish it through evaluation of the complete system and review its performance on the relevant cases.

Build a service whose limits are visible

The conceptual diagram shows two paths joining at a validation boundary. Explicit business rules handle defined obligations, while the model handles interpretation. The application decides what can proceed, what needs review, and what must stop.

The value of this arrangement is the question it forces a team to answer: which component establishes that an action is permitted and correct? Once that responsibility is clear, probabilistic behavior becomes an engineering concern that can be evaluated and bounded, rather than an excuse for unexplained outcomes.

References

1. Google for Developers. LLMs: What's a Large Language Model?. Living documentation.
2. Anthropic. Effective Context Engineering for AI Agents. September 29, 2025.
3. Microsoft Learn. RAG: Large Language Model End-to-End Evaluation Phase. Living documentation.
4. NIST. Generative Artificial Intelligence Profile, AI 600-1. July 2024.

Sources checked October 2026. CorpExcellence.com articles are best-effort research and analysis, not professional advice.