How to build an AI voice agent (step by step)

How I built two AI voice agents, one on ElevenLabs and one open source in my own voice, step by step, with every problem I hit.

Navid Moazzezvon Navid Moazzez·Aktualisiert 3. Okt. 2026·38 Min. Lesezeit·

An AI voice agent is software that holds a spoken conversation: it listens, works out what you want, and answers out loud, on a website, in an app or on a phone line.

I've built two of them.

The first runs on ElevenLabs, in my cloned voice. The second is open source: a voice model I trained on 142 minutes of my own recordings, with Claude Code writing the code.

In a 16-question test call, my open-source agent started answering 1.35 seconds after each question, on average. My ElevenLabs voice, saying the same answers, started after 1.19 seconds.

In this guide, I'll show you:

  • What an AI voice agent is, and how it works
  • How to build one on ElevenLabs, step by step, with screenshots from my account
  • How to build and change ElevenLabs agents from Claude Code, with their CLI and MCP server
  • The other platforms worth knowing, each with its price
  • The open-source agent I built, and the free Voicebox app
  • How to build your own interface, like my NavidBot
  • How to put your own voice in your agent, with recordings you can compare
  • Every problem I hit, and how I fixed it
  • What it costs, how to test it, and the mistakes to avoid

You don't need to code for your first agent. And every number here comes from my own tests, or from the company's own pages, read on October 2, 2026.

key_takeaways.mdTL;DR

Key takeaways

An AI voice agent runs a loop: speech to text hears you, an AI model writes the answer, and text to speech says it, in about 1 second.
ElevenLabs Agents is the quickest way I know to build one: write a prompt, pick a voice and a model, test it, then put it on your site or a phone number. The free plan includes 15 minutes of calls a month.
From Claude Code, you can build and change ElevenLabs agents with their CLI, their MCP server or their skills.
Open source (LiveKit, Pipecat) costs less per minute at high volume, but you run it yourself, and a GPU for your own voice model costs about $2.50 an hour.
An instant clone of your voice needs a few seconds to 2 minutes of audio. A professional clone needs an hour or more.

What is an AI voice agent?

An AI voice agent is an AI you talk to out loud, the way you'd talk to a person on the phone.

You speak, it understands, and it answers in a natural voice. It can also do things during the call, like book a meeting, look up an order or pass you to a person.

For example, a voice agent can:

  • Answer your phone – pick up every call, answer common questions and book appointments, day and night
  • Talk on your website – a call button on your site that answers from your own pages
  • Make calls – reminders and follow-ups, to people who agreed to get them
  • Speak as you – answer questions in your own voice

That last one is what I built. My agent answers questions about AI and the creator economy, in my voice.

A voice agent is close to a chatbot, except you talk instead of type. ElevenLabs agents can take typed messages too, and mine does.

And a phone menu makes you press 1 or 2. A voice agent lets you say what you need in your own words.

Why build an AI voice agent?

Build one when people already want to talk to you, and you can't answer every time.

For example, that could be a business that misses calls after hours, a coach who gets the same questions every week, or a site where visitors would rather ask than search.

An AI receptionist pays off where calls already go unanswered. So check that first.

And it's cheap to try. ElevenLabs' free plan includes 15 minutes of calls a month, and LiveKit's free plan includes 1,000 agent minutes.