AI agents need a new rhythm
Teams can make an AI agent appear capable in days. That is not the same as making one dependable across the range of situations real customers create.
Traditional software delivery gives teams a familiar rhythm: define a unit of work, build it, test it, accept it, and release it. That rhythm does not disappear for AI agents, but the steps change, and so does the tempo at which a team can perform them. During build, the loop from change to evaluated result turns many times a day, and releases can arrive weekly rather than monthly or quarterly.
Agent behavior is probabilistic. A change meant to improve one interaction can degrade another, and a platform update can alter behavior when the team has changed nothing.
The requirements are also unusually easy to lose. Agent instructions and business requirements are written in the same language, so teams often move directly from an idea to a prompt. The agent appears to work, but the reasoning behind it has not been preserved. The organization may no longer know why a behavior matters, how it should be evaluated, whether it blocks release, or who owns the decision.
The gap becomes visible during acceptance. Subject matter experts and business stakeholders encounter behaviors they did not expect. Without a shared specification, every observation can look like a defect and a new release requirement. Acceptance becomes a second requirements process. After release, the same pattern becomes a stream of reactive fixes.
AI agents need to change faster than scheduled test cycles can support. Automated evaluations create that tempo. A durable specification preserves the decisions, the priorities, and the behaviors that already work, and that is what keeps the tempo sustainable.
One method connects two cycles
The methodology has five recurring steps: Specify, Configure, Evaluate, Release, and Observe.
Specify, Configure, and Evaluate turn expertise into an agent the organization can release with confidence. Observe turns customer interactions and outcomes into what the organization should change next. Every accepted change returns to Specify and moves through the same cycle before reaching customers.
The point is not to choose between expert judgment and production learning. Running both at a higher tempo is what improves the agent quickly without risking the behaviors that already work.
1Specify
Turn expertise into explicit behaviors and evaluations before turning it into agent instructions.
The specification describes each behavior in three parts:
- Behavior and why it matters: what the agent should or should not do in a given situation, and the business reason behind it
- Evaluation: how the team will judge the behavior, including the threshold it has to meet
- Dependencies: the tools and knowledge the behavior needs. Tools are usually the enterprise services the agent calls, such as APIs. Knowledge is the policies and product information used by the agent
A behavior and its evaluation are two sides of the same decision. A behavior without an evaluation is an opinion. An evaluation without a named behavior is a test that nobody can adjudicate.
Each behavior is classified as a Must or a Should. Musts define the release boundary and include what the agent must do or must not do. Shoulds improve quality without automatically preventing release and include what the agent should do or should not do. The distinction keeps every desirable improvement from becoming an urgent release requirement.
Defining what good looks like requires complementary expertise. Subject matter experts own the specification and decide whether a behavior is a Must or a Should. Conversation designers, who bring backgrounds in linguistics and human factors, translate policies and procedures into effective conversations and expose interaction requirements others miss.
The specification lives outside the AI agent platform, where the business and delivery team can review it together. It does not need to describe the entire future state before work begins. Start at the level of detail available. Describing the behavior and its evaluation will reveal where the team needs to go deeper.
Late fee dispute
MustOwner: Dana Whitfield, Billing Operations
- Behavior
A customer disputes a late fee. Confirm it from the account, explain it with the actual due and payment dates, and waive it if the customer is eligible. Other fees, disputes older than 90 days, and accounts in collections transfer to a live agent with context.
- Why
High volume. The fee is small and a live agent's time is not. Customers in good standing get the waiver; most others accept a specific explanation. Eligibility comes from a billing service, not the agent's judgment.
- Evaluation
Pass only if the agent cites the correct dates and amount, follows the eligibility result, and transfers every out-of-scope case.
Threshold: 0 failures in 60 varied runs.- Dependencies
-
- Look Up Account (fee, due date, payment date)
- Check Fee Waiver Eligibility
- Waive Fee
- Late Fee Policy
- Transfer with Attached Data
2Configure
Configure the AI agent and its platform from the specification.
Translate the specification into prompts or instructions, connections to tools, ingested knowledge, and automated evaluations. If a tool is not ready, use a stub or mock, a placeholder that returns realistic data, so work can continue without waiting for the complete integration.
The direction matters. Derive the agent from the specification, never the reverse. Update the specification first, then change the configuration. If the team changes the configuration first and documents it later, the specification will quickly stop being the thing anyone trusts.
This separation also protects what the organization knows from the platform's shorter life cycle. Models, orchestration techniques, and interfaces are changing rapidly, usually monthly and sometimes weekly. The policies, procedures, behaviors, evaluations, knowledge, and business decisions in the specification should survive them.
3Evaluate
Run automated evaluations to quickly assess performance and establish production readiness.
Run evaluations against varied simulated interactions and compare the results with agreed thresholds. Review failures and a sample of apparent successes, with the greatest attention on Musts.
If a Must does not meet its threshold, return to Specify or Configure. Once the focused problem is resolved, rerun the broader suite. A targeted improvement can disturb behavior nobody intended to change, so the evaluation set must protect the whole agent rather than only the latest fix.
Evaluations must run repeatedly because a generative system can produce different results from the same input. During build, focused evaluations run many times a day and the full set runs several times a week. They should also run after configuration changes, before every release, and on a schedule after release. Models and platforms can change underneath the agent, and tools or knowledge can fail silently even when the delivery team has released nothing.
Automation changes the delivery tempo. A team that can determine within an hour whether anything broke can improve the agent frequently. A team that depends on manual testing at scheduled points must either move slowly or accept more risk.
Acceptance then becomes an extension of evaluation rather than a new requirements process at the end. Stakeholders review the same behaviors, thresholds, and results throughout implementation. By the time the agent is ready for release, the release boundary has already been applied.
4Release
Decide whether to deploy the latest candidate, and whether to deploy the current release more widely.
Release is an event, not another evaluation stage. It may introduce a new configuration, increase traffic to one already in production, or compare a new version with the current one. During early releases, a new version may reach customers weekly, and the traffic it receives may change once or twice a week.
The sponsor decides when to release because readiness and timing are different questions. Evaluation determines whether the agent is ready. The sponsor decides when the business should act on that result.
Traffic provides another control. The organization can increase or reduce the share of customers reaching the agent as production performance becomes clear, instead of treating release as an all-or-nothing commitment.
5Observe
Learn from customer interactions and outcomes, then update the specification.
Simulation can confirm only cases someone imagined. It cannot reproduce every way customers will express themselves, every condition surrounding an interaction, or every effect the agent will have on the business.
Sample apparent failures, apparent successes, and outliers to understand what the agent is doing and what customers are asking it to do. Measure the business outcome. An agent can behave as specified without producing the intended result, and it can improve a headline metric while creating problems that aggregate reporting hides.
What production reveals begins as an observation, not a defect. It becomes a defect only when it shows that the agent violated a specified behavior. Its priority comes from the specification, based on whether that behavior is a Must or a Should.
Other observations may reveal a missing behavior, an incomplete evaluation, a deliberate business change, or an expectation that differs from what was agreed. A defect may originate in the specification, a tool, or the knowledge rather than in the agent instructions.
Subject matter and conversation design experts analyze observations before making changes. Any changes then move through Specify, Configure, and Evaluate before reaching customers.
Expertise reduces what the organization has to learn the expensive way in production. Customer interactions and outcomes improve the specification beyond what experts could anticipate. Each cycle strengthens the other.
What changes for leaders
The methodology does more than organize work. It changes how the organization makes decisions about AI agents.
- Release criteria are agreed early. The specification, not the agent's prompts and instructions, defines what has to work before release and what can wait. Stakeholders review those behaviors and their evaluation results as the work develops, instead of reconstructing requirements at the end.
- Changes become more frequent and safer. Comprehensive and automated evaluations demonstrate when versions are ready for release and whether any changes caused other disruptions.
- The organization keeps what it learns. Models and platforms change often, and the team may not control the timing. The specification resides outside the platform and holds the behaviors, evaluations, dependencies, and reasoning behind each decision, so the organization has something durable while the models and platforms move.
Speed and rigor stop looking like opposites. The same foundation that helps the team reach production also helps it improve the agent after release.
Where to start
Whether the catalyst is ambition, imperative, or mandate, it has to become a business case before it can become an agent. The business case names the outcomes the business expects, and it is the input to the methodology.