A system that passed its test has been tested once
The prevailing approach treats assurance as a gate. A system is built, tested against cases assembled before launch, approved, and then released into production, where it is expected to continue behaving. In a business process this is a false assumption, because the inputs change every week, the rules change every quarter, and the dependencies change without notice.
The consequence is familiar: a system that behaved acceptably in testing degrades in use, the degradation is noticed by a customer rather than a monitor, and the response is a piece of work to re-establish the old behaviour, followed by another period of quiet drift.
Correctness tests have to be drawn from the business
A test set assembled from the cases that seemed interesting during build tells you about those cases and nothing else. Test sets that survive contact with production are drawn from the actual distribution of the work: the routine transactions, the awkward ones that arrive once a quarter, the near-misses that were corrected, and the cases escalated because the outcome mattered.
Two properties make such a set useful rather than decorative. It is representative, in the sense that its composition reflects the real population rather than the build team's interests, and it is versioned, so that a later change can be measured against the same cases.
Regression and drift arrive on different clocks
A production workflow depends on more than the model. It depends on the schema of a source table, on a downstream service that reshapes a field, on a payment service that starts rejecting a value it used to accept, and on a colleague who changes a threshold in a configuration screen to unblock a busy afternoon. Each is small, and each is invisible to the component that depends on it.
This is why a live system needs its dependencies watched as well as its own output. A model can hold its accuracy while the pipeline around it changes the meaning of the data it reads. The relationship between the two layers of context is in why an AI agent needs more than database access.
Silent workflow failures are worse than loud ones
The failures that reach a customer are the least important category. More consequential are the ones that are absorbed: a case that takes the exception path every day, a review queue that grows, a record written with a field nobody notices is wrong. They are absorbed because the human process still functions, and the cost appears later as rework and delay.
Monitoring has to be designed around absorption rather than around crashes. The useful signals are the rate at which cases take the exception path, the size of the approval queues, the interval between a case arriving and a person acting on it, and the volume of work handled without human review. The last of these grows quietly as trust builds, and it deserves a threshold of its own. The measures to watch once a workflow is live are in continuous operational intelligence.
Three kinds of change need different responses, and treating them as one produces a testing regime that is either wasteful or blind.
| What changed | Typical signal | Sensible response |
|---|---|---|
| Source data and schema | Field missing, type changed, volume shifted | Contract test on the source, then re-run the case set |
| Model behaviour | Accuracy on the fixed set moves, output format shifts | Scheduled evaluation against the versioned set |
| Business rules and thresholds | Exception rate rises, approval queue grows | Review the configuration change with the business owner |
| Process and dependencies | Step order changes, a downstream rule alters | End-to-end journey test on the critical paths |
The model itself is often the least frequent source of change. The data and the rules around it move far more often, and each of them changes the behaviour of the system without any release.
The business process changes and the system inherits the old rule
When a process is redesigned, the system usually keeps the previous behaviour, because nobody connected the two pieces of work. The old routing rule still sends a case to a queue that no longer exists, or the old approval threshold remains in force after the policy changed. Nothing errors. The system is doing exactly what it was built to do, in a process that no longer exists.
This is the most under-managed change of all, because it is usually not planned as a change to the system at all. The practical control is a rule: when a process is redesigned, the systems that participate in it are named in the design, and a named person checks that the automated behaviour still matches the new process.
Change management has to name who decides
The question that stalls most change reviews is not what to do but who decides. A model update, a prompt change, a new source and a new rule have different risk profiles, and combining them into a single release makes the highest-risk one cheap to ship. Separating them means a change to a retrieval source can be reviewed by a data owner, while a change to how decisions are made goes to the business owner.
Each change needs a stated trigger for re-testing, a named person who authorises it, a record of what changed and what was re-run, and a post-change observation period before the change is treated as settled. Without those four, a change log becomes a record of activity rather than of assurance. Keeping such a capability alive after the pilot team has moved on is in maintaining enterprise AI after the pilot.
Making this a standing capability
- Assemble a versioned case set drawn from real production traffic, including the cases that were escalated and corrected.
- Write down the changes that require re-testing, and who is authorised to make each of them.
- Instrument the live workflow for the signals of absorption, not only for failures, and set thresholds on each.
- Add an end-to-end check on the critical paths, so that a change in a dependency is detected as a broken journey rather than as a degraded accuracy figure.
- Name the process owner who reviews the results on a fixed cycle.
The measures that show the capability is working are the interval between a change and the re-test that covers it, the number of live changes not accompanied by a re-run, the exception-path rate and its trend, and the number of silent failures found by monitoring rather than by a customer.
Where a standing regime is not worth the cost
For a process with low consequence, low volume and no external party, a full continuous regime is disproportionate. A small fixed case set re-run monthly, a handful of monitored signals and a named owner for the exceptions is usually enough. What is not defensible is having no answer at all for a system that moves value or touches a person outside the business.