Deployment reveals the challenges

Your agent is live. Some run as expected and produce little that a customer or the business would notice. Others produce results, but maintaining them takes constant attention, because every change is manual and every release is a risk. Meanwhile, leadership expects a broader rollout, the next use case, and faster progress. Going live revealed the hard parts.

What did your deployment prove?

Most first deployments are capability proofs: of a concept, an operation, a platform, or an integration. They answer one question: can this be done here? If yours succeeded, it established one or both of two capabilities, and that is a real result.

  • Solution capability: how well does the agent work in production? Going live settled whether it works. The question is how consistently it performs its intended behaviors, and its integrations with adjacent systems, under real conditions.
  • Delivery capability: can your team change it reliably? It can specify what the agent should do, observe whether it does, evaluate changes, and release them.

Establishing one is progress. Establishing both is a foundation most organizations do not have, but neither is the same as establishing value.

A capability is value you could produce. Business value is value you have produced: a need resolved without a transfer, less customer effort, a service level met, capacity freed, cost to serve reduced. It is what keeps an initiative funded.

The reverse also holds. Value without delivery capability is a ceiling, not a foundation: a real result that depends on manual effort and a few people to keep producing it.

Without value, the capability itself was tested gently. A use case that does not matter much brings business rules nobody argues about, integrations nobody insists on, volume that strains nothing, and stakeholders who never show up. Failure is cheap, so failure handling is never exercised.

Two boxes labeled Capability and Value joined by a plus sign.
An initiative must provide both enterprise capability and business value to continue.

So the question is not whether your agent works but where it stands. On the matrix below, proven means value produced, not capability proved. A successful platform proof that has not yet moved an outcome sits in the unproven row, and that is the normal place to start.

Where does your deployment stand?

Two dimensions place any deployed agent, and they are independent. An agent can be strong on one and weak on the other.

Outcome performance. Is it producing value?

Whether the agent produces the intended customer and business outcomes, consistently, in production. Sounding fluent, passing a demonstration, or handling simple requests does not count. Nor does meeting the deployment’s objectives, since a capability proof can meet every objective without producing an outcome. The hard question: has the outcome moved on a number someone outside the AI team already tracks, such as cost to serve, containment, or CSAT? If you cannot tell, treat the answer as no, and make naming that number the first move.

Delivery repeatability. Can you reliably do more?

Whether the organization can specify, observe, evaluate, change, release, and reuse the agent reliably, and whether the solution has the capabilities this requires. It does not accrue with time; an agent can be live for a year and remain fragile. The hard question: can you reproduce a production failure and show that a change fixed it without breaking anything else? If it takes a specific person, answer no.

A two by two matrix of outcome performance against delivery repeatability with four positions: Unproven and Fragile, Proven but Fragile, Unproven but Repeatable, and Proven and Repeatable, each with its condition.
  • Proven but Fragile: a tracked number has moved, but every change is slow and unverified.
  • Proven and Repeatable: the value is proven, and changes are easy and reliable, without side effects.
  • Unproven and Fragile: no tracked number has moved, and every change is slow and unverified.
  • Unproven but Repeatable: the value is unproven, but changes are easy and reliable, without side effects.
Place your agent by the harder test on each axis. Answer no wherever you cannot prove yes; the honest position is usually lower and further left than the comfortable one.

This is a decision aid, not a maturity score. Different initiatives sit in different quadrants, and one initiative moves as its scope, performance, and method change. Find yours, then choose the move.

Choose your next move

The two dimensions decide different things. Performance decides whether to improve the solution or expand it. Repeatability decides whether to strengthen the delivery method or reuse it. Most positions call for both, in different proportions.

The same two by two matrix with the four cells relabeled as moves: Improve and Strengthen, Strengthen, Then Expand, Improve or Repurpose, and Expand Deliberately.
  • Proven but Fragile: strengthen, then expand. Make changes repeatable, then extend the reach.
  • Proven and Repeatable: expand deliberately. Add one expansion at a time and verify the key indicators after each.
  • Unproven and Fragile: improve and strengthen. Fix what it does and how it changes, together.
  • Unproven but Repeatable: improve or repurpose. Improve a key indicator or change the use case.
The same matrix, with the next move for each position. Find your position, then make its move.

Whatever your position, do not expand or repeat a deployment whose outcome is still unproven. Every added channel or journey inherits the same unmeasured outcome and the same fragility, so the underperformance compounds. The next section takes each move in turn, including when to repurpose the capability for a different use case.

Improve the unproven, expand the proven, strengthen the fragile

Improve

When the outcome is unproven, find out why before expanding anything. Then fix, deepen, or repurpose.

  • Fix the behavior. Turn each production failure, such as a wrong answer or a dropped step, into a specified behavior with an automated evaluation.
  • Deepen the scope. Add the intents, exceptions, and transactions that carry the value when the agent covers too little of the journey to move the number.
  • Repurpose the solution. Carry the solution and delivery capability to a use case that can produce consequential value.

Expand

When the outcome is proven, apply what works, one expansion at a time.

  • Grow reach. Take a proven capability to more customers, channels, languages, regions, or business units.
  • Deepen resolution. Handle more intents, exceptions, personalization, or transactions within the current journey.
  • Extend adjacently. Reuse the foundation for a related customer journey or operational problem.

Strengthen

When delivery is fragile, make the next change cheaper than the last, whichever move you choose.

  • Capture what the agent should do. Turn what production reveals into a specification: each behavior with an evaluation and a threshold. Treat a behavior that failed and a behavior nobody had specified the same way: write it down, with its evaluation.
  • Test after every change. Run automated evaluations on every change and on a schedule, so a fix or a platform update cannot quietly undo what worked.
  • Build the method and the team. Run the specification, the automated evaluations, and the release decision as one repeatable cycle, owned by the business and delivery teams together.

None of this requires a new platform. The evaluation, observation, and release capabilities usually exist already, in the contact center platform, the cloud platform, or tooling another team runs. What is missing is the method that connects them: a specification the business owns, evaluations that run on every change, and release criteria someone is accountable for. That method, and the patterns from building it on the platforms enterprises already run, is what PTP brings.

How one enterprise chose its next move

An insurer piloted an AI agent to answer coverage and benefits questions for one small customer group. It reached production the traditional way: requirements, build, then acceptance, where the business joined and found issue after issue. Every fix meant changing the instructions and retesting by hand, and the cycle continued after go-live. An eight-week pilot ran sixteen. Calls were being automated, but the only measure was whether a call avoided a transfer, which counted a customer who gave up as a success. Unproven and fragile.

The first move was to name the number: services completed for the customer, live agent handle time with and without the AI agent, and escalations sorted into good and bad. The move itself was improve and strengthen, together. Each observed issue became a specified behavior, a must or a should, with an automated evaluation written before the fix and run many times after it. Live agents began receiving a summary of what the agent had done. Acceptance stopped being a manual test cycle and became a review of interactions the evaluations had already exercised. In about six weeks the rate of good automation doubled, from about 15 percent to about 30 percent.

Then the same integration, method, and measurements carried the agent to a second line of business. That deployment shipped on its eight-week plan, with roughly a third of the effort of the first.

The diagnosis matrix with a marker in Unproven and Fragile labeled Pilot, a curved arrow to a marker in Proven and Repeatable labeled Six weeks later, and an arrow to the upper right corner of the grid.
One move, drawn on the grid: name the number, improve and strengthen together, then expand.

Make your next move specific

Use this for the initiative in front of you.

  1. Outcome

    Which customer or business outcome must this move improve, and what evidence from the current deployment shows it can? Agree on the number with the people who track it.

  2. Scope

    Which single behavior, transaction, journey, segment, or channel can you specify, evaluate, and release now?

  3. Delivery

    Do you have a method to specify each behavior with its evaluation, evaluate every change automatically, and release against agreed criteria, and a team to run it?

  4. Dependencies

    Which integrations, knowledge, and evaluations will this move reuse or need, and what reusable capability will it leave behind?

  5. Observation

    Once it is live, how will you observe the outcome and the behaviors, and how quickly can you release the next change?

If you can answer all five, you have a next move. If you cannot answer the first, the move is improve or repurpose, not expand. If you cannot answer the second, narrow the move until you can. If you cannot answer the last three, you have found the repeatability gap.

Know what your deployment proved and where it stands. Then make the move that creates value now and makes the next one easier.