Agents vs Chatbots: A Production Checklist

A familiar pattern: a company deploys a chatbot, deflection goes up for a few weeks, then the hard conversations arrive and the bot fails with confidence. The model was rarely the problem. A chatbot and an agent are different kinds of system, and production judges them by different standards. This is the checklist we use before we call something an agent.

Deflection vs resolution

A chatbot is measured on deflection: how many conversations avoided a human. An agent is measured on resolution: how many tasks it actually completed.

The two diverge. A bot can deflect most conversations and resolve almost nothing, because users give up and phone instead. Resolution means the system does the work: checks the order, changes the booking, issues the invoice.

Tools and memory

An agent reads and writes in your systems: CRM, calendar, order management, payments. Without tools it is an assistant.

Two requirements follow:

• Real integrations. Write access with clear permissions, not a read-only FAQ index.
• Memory with a policy. The agent remembers what matters inside a task, and across conversations where appropriate, and it is explicit about what it stores.

Limits

An agent that can act can also act wrong. Every action it can take belongs to one of three lists:

• Does alone. Low-risk, reversible work: answering, following up, reconciling.
• Asks you. Anything above a threshold: large payments, discounts, exceptions.
• Never does. Actions no agent should take: signing contracts, changing bank details.

The agent that reschedules an appointment should not be able to refund a year of invoices without a person. Lists start conservative and widen as the evidence earns it.

Evaluation

A language model cannot be unit-tested like a function. The same input can produce different valid outputs, and a small prompt change can break a flow that worked yesterday.

Agents need an eval harness: real scenarios with expected outcomes, run on every change. At minimum:

• Task evals. Did the agent complete the booking, with the right data, in the right system?
• Regression evals. Past failures, replayed on every release.
• Adversarial evals. Prompt injection, out-of-scope requests, contradictory instructions.

Observability

For any action the agent took, you need to answer: what did it see, what did it decide, what did it call, and what did it cost. That means tracing every run end to end, logging every tool call with inputs and outputs, and tracking cost per completed task.

The checklist

1. Does it complete tasks end to end in real systems, or only talk about them?
2. Is every action on one of the three lists, with thresholds that trigger human approval?
3. Is there an eval harness that runs on every change, with real failure cases in it?
4. Can you trace any single run and explain why the agent did what it did?
5. Do you know the cost per resolved task?
6. Do you know your resolution rate, not just your deflection rate?
7. When the agent is wrong, does it escalate, or fail silently?

Seven yeses: you have an agent. Anything less: you have a chatbot, which is fine as long as you staff and measure it like one.

If you want a read on where your system sits, talk to us or see how we build custom AI agents.