Five questions to ask before putting an AI agent to work in your business
October’s research on AI agents boils down to five questions worth answering before you connect one to your CRM, your inbox or your invoicing.
An AI agent is not a chatbot that answers questions. It’s a system that does things: it looks something up in your CRM, drafts an email, logs an order or sorts invoices, connected to the tools your business already uses. So when it gets something wrong, it doesn’t leave a poor answer on a screen. It leaves an order logged incorrectly, or an email sent to the wrong person.
Every week I read through what’s published on arXiv about applied AI. The papers from the last week of October fit neatly into five questions worth answering before you put an agent to work.
1. What happens when a connection fails?
An agent depends on everything it’s connected to, and connections fail: a service stops responding, a key expires, data arrives in a different format.
One research team built a way to inject realistic faults into those connections, drawn from real incidents logged across 1,774 open-source projects, and measured how different agents coped. When faults were persistent, the quality of the work dropped sharply. Moving from a 20-billion-parameter model to a 120-billion-parameter one didn’t fix it. What made the difference was how the software around the model handled errors.
In practice, the question has very concrete answers: are there retries, does someone get an alert when something fails, and what does the system do in the meantime? If nobody can answer, the first person to notice the failure will be a customer.
2. Where does a person check, and why there?
Agents that run for months build up what you might call memory: customer details, preferences, past decisions. And that information goes stale.
One October paper makes a point I think is among the most useful of the month: whether a stored fact needs re-checking depends on what’s about to be done with it. Suggesting a meeting slot from a fortnight-old detail is harmless. Issuing an invoice from that same detail is not. Using that principle, the authors report a 33.9% lower average verification cost in one of their test environments.
For a business, that means placing human review according to the cost of getting it wrong, not evenly. Routine work goes through on its own. Anything that touches money or reaches a customer gets checked by a person first. Reviewing everything eats the time savings; reviewing nothing is a gamble.
3. What does each task cost, and how many succeed?
The one industrial case this week comes from DoorDash, which is hardly a small business: a multi-agent system that reads tickets, queries live data and prepares code changes. Out of 50 cases, 45 ended in changes approved by engineers, at an average of $4.79 and 13.8 minutes per task.
What’s useful here isn’t the size of the company but how they measured it: success rate, cost per task and time per task. Those are the three numbers you need before deciding whether an automation pays off, set against what the job costs when done by hand today. If a query takes someone twenty minutes and an agent gets it right nine times out of ten for a few pounds, the decision is easy. Without the numbers there’s no decision, only a hunch.
4. How often do your documents change?
Many assistants answer from a company’s own documents: price lists, product sheets, procedures. To do that, documents are split into chunks and indexed. What rarely gets mentioned is that every time a document changes, part of that index has to be reprocessed, and that costs money every month.
A study using real revisions to Wikipedia articles compared three ways of splitting text. With fixed-size chunks, 57–68% of the work had to be redone. Splitting by paragraphs and sections, 38–56%. And answer quality barely moved between methods: less than two percentage points apart.
If your documents rarely change, it makes no difference. If they change every week, it’s a small technical choice that shows up on the bill.
5. Have you tested it on your own cases?
To check whether an assistant gives good answers, it’s increasingly common to use another AI model as a judge. A study of more than a million scores across 27 languages found that which model acts as judge matters more than how you prompt it, and that evaluations are less stable in languages with fewer resources than English.
A second paper from the same week adds that agent rankings shift with the model, the software around it and the benchmark, and that similar scores can hide very different behaviour.
The takeaway is simple: published leaderboards are a starting point, not a substitute for twenty or thirty real cases from your own business run through the system, reviewed by someone who knows the work. And if you serve customers in more than one language, run them in each one.
None of these five questions is about which model to choose. They’re about what happens around it: failures, checks, costs, upkeep and testing. That’s what separates an impressive demo from something your team uses every day without thinking about it. And it’s why it pays to get the process right before automating it.
If you have a process in mind and want to run these five questions over it, that’s what the discovery call is for.