🎙️ Tutorial 11: Voice Agents

Master voice-enabled AI agents with the OpenAI Agents SDK! This tutorial demonstrates how to build conversational voice agents using speech-to-text, text-to-speech, and intelligent agent workflows for natural voice interactions.

🎯 What You'll Learn

• Voice Pipeline Architecture: Complete speech ↔ text ↔ speech workflow
• Static Voice Processing: Turn-based voice interaction with recorded audio
• Streaming Voice Processing: Real-time voice conversation with live audio
• Multi-Language Support: Automatic language detection and agent handoffs
• Voice-Optimized Tools: Design tools specifically for voice interactions
• Audio Management: Recording, playback, and streaming audio utilities

🧠 Core Concept: Voice Agents

Voice agents combine the power of AI language models with speech processing to create natural conversational interfaces. Think of voice agents as AI assistants you can talk to naturally that:

• Listen to your speech and convert it to text
• Process your request with intelligent AI agents
• Use tools and make decisions like text-based agents
• Convert responses back to natural speech
• Handle multi-turn conversations seamlessly

┌─────────────────────────────────────────────────────────────┐
│                     VOICE AGENT SYSTEM                      │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  🎤 USER SPEECH                                             │
│       │                                                     │
│       ▼                                                     │
│  ┌─────────────┐    1. SPEECH-TO-TEXT                       │
│  │   AUDIO     │    ◦ Convert speech to text                │
│  │  PIPELINE   │    ◦ Handle multiple languages             │
│  └─────────────┘                                            │
│       │                                                     │
│       ▼                                                     │
│  ┌─────────────┐    2. AGENT PROCESSING                     │
│  │    AGENT    │    ◦ Multi-agent workflows                 │
│  │ ECOSYSTEM   │    ◦ Tool calling & handoffs               │
│  └─────────────┘    ◦ Context management                    │
│       │                                                     │
│       ▼                                                     │
│  ┌─────────────┐    3. TEXT-TO-SPEECH                       │
│  │   SPEECH    │    ◦ Convert response to speech            │
│  │ SYNTHESIS   │    ◦ Natural voice output                  │
│  └─────────────┘                                            │
│       │                                                     │
│       ▼                                                     │
│  🔊 AI RESPONSE                                             │
└─────────────────────────────────────────────────────────────┘

🚀 Tutorial Overview

This tutorial demonstrates three core voice interaction patterns:

1. Static Voice Processing (static/)

• Turn-based interaction: Record → Process → Respond
• Complete audio processing: Full utterance before processing
• Simpler implementation: Easier to understand and debug
• Best for: Voice commands, structured interactions

2. Streaming Voice Processing (streamed/)

• Real-time interaction: Continuous listening and responding
• Live audio streaming: Process audio as it arrives
• Activity detection: Automatic speech start/stop detection
• Best for: Natural conversations, voice assistants

3. Realtime Voice Processing (realtime/)

• Ultra-low latency: WebSocket-based persistent connections
• Interruption handling: Natural conversation interruptions
• Realtime API: OpenAI's newest voice technology
• Best for: Live conversations, minimal latency requirements

📁 Project Structure

11_voice/
├── README.md                          # This file - voice agents overview
├── static/                            # Static voice processing example
│   ├── agent.py                      # Complete static voice agent
│   ├── util.py                       # Audio recording and playback utilities
│   ├── requirements.txt              # Dependencies for static example
│   ├── env.example                   # Environment variables
│   └── README.md                     # Static voice documentation
├── streamed/                         # Streaming voice processing example
│   ├── agent.py                      # Real-time streaming voice agent
│   ├── util.py                       # Streaming audio utilities
│   ├── requirements.txt              # Dependencies for streaming example
│   ├── env.example                   # Environment variables
│   └── README.md                     # Streaming voice documentation
├── realtime/                         # Realtime voice processing example
│   ├── agent.py                      # Basic realtime voice agent
│   ├── requirements.txt              # Dependencies for realtime example
│   ├── env.example                   # Environment variables
│   └── README.md                     # Realtime voice documentation
└── __init__.py                       # Module initialization

🎯 Learning Objectives

By the end of this tutorial, you'll understand:

• ✅ How to build complete voice interaction pipelines
• ✅ The difference between static and streaming voice processing
• ✅ How to implement multi-language voice agents with handoffs
• ✅ Best practices for voice-optimized agent design
• ✅ Real-time audio processing and streaming techniques

🚀 Getting Started

Prerequisites

1. Install OpenAI Agents SDK with voice support:

   pip install 'openai-agents[voice]'
   

2. Install audio dependencies:

   pip install sounddevice numpy soundfile librosa
   

3. Set up environment variables:

   cp static/env.example static/.env
   cp streamed/env.example streamed/.env
   # Edit .env files and add your OpenAI API key
   

Quick Start Options

Option 1: Static Voice (Recommended for beginners)

cd static/
python agent.py

Option 2: Streaming Voice (Advanced)

cd streamed/
python agent.py

Option 3: Realtime Voice (Ultra-low latency)

cd realtime/
python agent.py

🧪 Voice Agent Capabilities

Multi-Language Support

Both examples include:

• English Agent: Primary assistant with full tool access
• Spanish Agent: Specialized Spanish-speaking assistant
• French Agent: Specialized French-speaking assistant
• Automatic Language Detection: Seamless handoffs based on detected language

Voice-Optimized Tools

• get_weather(city): Weather information with voice-friendly responses
• get_time(): Current time with natural speech output
• calculate_tip(bill, percentage): Tip calculations for voice queries
• set_reminder(message, minutes): Voice-activated reminders (streaming only)
• get_news_summary(): Voice-friendly news updates (streaming only)

Audio Processing Features

• High-Quality Recording: 24kHz audio capture
• Real-Time Playback: Low-latency audio output
• Activity Detection: Automatic speech boundary detection (streaming)
• Error Recovery: Robust audio pipeline error handling

🔧 Key Voice Agent Patterns

1. Basic Voice Pipeline

from agents.voice import VoicePipeline, SingleAgentVoiceWorkflow

pipeline = VoicePipeline(
    workflow=SingleAgentVoiceWorkflow(agent)
)

2. Static Audio Processing

from agents.voice import AudioInput

audio_buffer = record_audio(duration=5.0)
audio_input = AudioInput(buffer=audio_buffer)
result = await pipeline.run(audio_input)

3. Streaming Audio Processing

from agents.voice import StreamedAudioInput

streamed_input = StreamedAudioInput()
result = await pipeline.run(streamed_input)

# Push audio chunks in real-time
streamed_input.push_audio(audio_chunk)

4. Multi-Language Agent Setup

spanish_agent = Agent(
    name="Spanish",
    handoff_description="A spanish speaking agent.",
    instructions="Speak in Spanish only..."
)

main_agent = Agent(
    name="Assistant", 
    handoffs=[spanish_agent, french_agent],
    instructions="If user speaks Spanish, handoff to Spanish agent..."
)

💡 Voice Agent Best Practices

Agent Design for Voice

1. Concise Instructions: Voice interactions work best with brief instructions
2. Conversational Responses: Design for natural speech patterns
3. Clear Tool Descriptions: Voice-friendly tool naming and descriptions
4. Language Handling: Implement clear language detection logic

Audio Quality

1. Good Hardware: Use quality microphones and speakers
2. Noise Reduction: Minimize background noise during recording
3. Audio Levels: Ensure appropriate input/output volume levels
4. Latency Optimization: Configure audio buffers for minimal delay

Error Handling

1. Graceful Failures: Handle audio device failures gracefully
2. Network Issues: Implement retry logic for API calls
3. User Interruptions: Allow clean exit from voice sessions
4. Resource Cleanup: Properly close audio streams and resources

🧪 Example Voice Interactions

English Conversations

• "Tell me a joke" → Humorous response
• "What's the weather in London?" → Weather tool call
• "What time is it?" → Current time
• "Calculate a 18% tip on a $75 bill" → Tip calculation

Language Switching

• "Hola, ¿qué tiempo hace en Madrid?" → Spanish agent response
• "Bonjour, quelle heure est-il?" → French agent response
• Seamless language detection and agent handoffs

Multi-Turn Conversations (Streaming)

• Natural back-and-forth dialogue
• Context preservation across turns
• Tool usage within conversations

📊 Static vs Streaming Comparison

Feature	Static Voice	Streaming Voice
Processing	Turn-based	Real-time
Complexity	Simpler	More complex
Latency	Higher	Lower
Use Cases	Commands, queries	Conversations
Activity Detection	Manual	Automatic
Resource Usage	Lower	Higher
User Experience	Structured	Natural

🚨 Requirements & Dependencies

Core Dependencies

• openai-agents[voice]: Voice-enabled Agents SDK
• sounddevice: Real-time audio I/O
• numpy: Audio data processing
• soundfile: Audio file operations (optional)
• librosa: Audio resampling (optional)

System Requirements

• Python 3.8+: Required for async support
• Audio Hardware: Microphone and speakers/headphones
• Processing Power: Sufficient CPU for real-time audio processing
• Network: Stable internet for OpenAI API calls

🔗 Related Documentation

• Voice Quickstart: Official voice agent setup guide
• Voice Pipelines: Advanced pipeline configuration
• Agent Fundamentals: Basic agent concepts
• Multi-Agent Systems: Agent handoffs and orchestration

🚨 Troubleshooting

Audio Issues

• No microphone input: Check audio device permissions and settings
• Poor audio quality: Verify microphone levels and background noise
• Playback problems: Test speaker/headphone configuration
• Latency issues: Optimize audio buffer sizes

Voice Pipeline Issues

• Transcription errors: Ensure clear speech and good audio quality
• Agent responses: Verify API keys and network connectivity
• Language detection: Test with clear language examples
• Handoff failures: Check agent instructions and handoff logic

Performance Issues

• High CPU usage: Monitor real-time processing load
• Memory leaks: Ensure proper cleanup of audio streams
• Network timeouts: Implement retry logic for API calls
• Resource conflicts: Check for audio device conflicts

💡 Pro Tips

• Start with Static: Master static voice processing before attempting streaming
• Test Audio Setup: Verify hardware configuration before development
• Monitor Debug Output: Use callbacks to understand pipeline behavior
• Optimize for Voice: Design agents specifically for conversational interaction
• Handle Edge Cases: Plan for network issues, audio failures, and user interruptions

🔗 Next Steps

After mastering voice agents:

• Production Deployment: Scale voice agents for real-world applications
• Custom Voice Models: Integrate specialized speech recognition/synthesis
• Multi-Modal Agents: Combine voice with vision and text capabilities
• Enterprise Voice Solutions: Build robust voice applications for business use