AI voice agents for customer contact: what can they do, and what not?

How a voice agent works

A voice agent consists of three parts working in sequence. Speech recognition turns what the caller says into text. A language model works out what the question is and what the answer should be, and can look up data in your systems along the way. Speech synthesis turns the answer back into a voice.

The difference from an old-fashioned phone menu is that the caller can simply say what they want, in their own words, instead of pressing a key. The difference from a person is that the system only knows what you have given it and may only do what you allow.

Where a voice agent is strong

The best uses are calls that happen often, have a clear structure and require little judgement:

• Answering questions that keep coming back: opening hours, the status of an order, what is needed for an appointment.
• Requesting and recording details: name, customer number, the problem and its urgency, so a colleague picks up the call prepared.
• Routing to the right person or department, based on what the caller says rather than a menu.
• Making, moving or confirming appointments in a calendar the system can access.
• Being reachable outside office hours, or when all lines are busy, so a caller is not left hanging in a queue.

Where the limits are

A language model sounds convincing, even when it is not sure. That is the main reason to scope a voice agent tightly. Less suitable are:

• Calls where empathy or negotiation is central, such as a complaint from an angry customer or a sensitive personal matter.
• Decisions with large consequences, such as an exception to the terms or a promise about money, without a person checking.
• Questions whose answer is not in your systems or documentation. What the system does not know, it should not guess.
• Poor connections, lots of background noise or strong dialects, where speech recognition still makes mistakes.

What setting it up well means

The difference between a voice agent that helps and one that irritates lies almost entirely in how it is set up. A few things that belong to that:

• A clear task. Start with one kind of call, not with the whole of customer service at once.
• A fixed way out. The caller can always ask for a person, and the system transfers as soon as it is not sure.
• Answers from your own sources. The system looks things up in your systems and documentation, instead of answering from general knowledge.
• Honesty. The system says at the start that the caller is speaking with an AI assistant.
• Reviewing and adjusting. Calls are kept and read back, so mistakes are found and the setup improves.

Privacy and the GDPR

A phone call contains personal data, sometimes sensitive. Agree up front where the speech and the text are processed, how long calls are kept and who can access them. Tell callers that the call is handled by an AI assistant and, if so, that it is recorded. Choose speech and language model vendors that fit your requirements for processing and storage, and record that in a data processing agreement.

Connecting it to your systems

A voice agent that only talks helps little. The value lies in what it can look up and record: the status of an order in your ERP, an appointment in the calendar, a ticket in your customer service system. Without those integrations it stays with general answers; with them the system can really help the caller further, and everything discussed ends up where your colleagues see it.

The handover to a colleague belongs to that integration too. When the system transfers a caller, ideally it is already clear who is calling, what the question is and what has been tried. That way the caller does not have to tell their story again, and the colleague starts the call where the system stopped.

How I approach it

I start with the calls you handle today: which questions keep coming back, what does a colleague have to look up for them, and which calls do you want to keep with people? That shows which task a voice agent can take over, and which not.

Then I build the system around that task, connected to the systems you already use, with a fixed way out to a colleague. It is tested on real call scenarios before it goes live, and after delivery I stay available for adjustments and maintenance.

Frequently asked questions

Will a caller notice they are talking to a computer?

Usually, yes, and that is fine. The system should say at the start that the caller is speaking with an AI assistant. What counts is not whether it sounds human, but whether the caller is helped quickly or transferred when needed.

Does a voice agent replace my staff on the phone?

No, it takes over the repetitive work. The calls that are always the same go to the system, so colleagues have time for the calls where a person makes the difference.

Does it work in Dutch?

Yes. Speech recognition and language models work well in Dutch, although strong dialects, poor connections and lots of background noise remain harder. That is one reason to build in a fixed way out to a colleague.