Integrating conversational AI voice models such as Gemini Live or OpenAI Realtime API into live multi-party WebRTC calls typically forces engineering teams into a fragile architecture: spinning up headless browsers or heavy C++ GStreamer workers on the AI backend just to speak Opus over SRTP. When designing the AI gateway inside Slyber SFU Server, I eliminated server-side WebRTC duplication by introducing a lightweight, full-duplex binary WebSocket pipe (/ai/ws).
Combined with the zero-allocation Go routing engine, this gateway allows any Python, Node.js, or Go orchestrator to stream raw Opus frames into a WebRTC room while preserving hardware-accelerated browser playback and end-to-end encryption.
1. Decoupling WebRTC Transport from AI Inference (/ai/ws)
Instead of managing ICE candidates, DTLS handshakes, and RTCP feedback inside your AI microservice, the SFU exposes an authenticated WebSocket endpoint at /ai/ws?room=room-id&mode=mux (gateway documentation):
- Human-to-AI audio extraction. As participants speak in the room, the SFU strips SRTP/RTP headers and forwards raw Opus frames over the WebSocket prefixed with
[len(speaker_id), rfc6464_audio_level, speaker_id...]so your AI backend knows exactly which participant is talking. - Multi-bot multiplexing (
mode=mux). A single WebSocket connection can dynamically spawn up to 6 independent AI agents in the same room via JSON control commands (spawn_agentandremove_agent), each assigned its own SSRC and track ID so clients display individual active-speaker indicators per bot. - Single-frame HTTP injection. For one-off voice alerts or announcements, backend services can push a raw Opus payload via
POST /ai?room=room-idwithout maintaining a persistent socket.
2. Smoothing Bursty Neural Audio with a 20ms RTP Pacer
Multimodal LLMs generate speech chunks unevenly—frequently delivering 400–800 ms of audio in a single burst followed by network jitter. Injecting bursty packets directly into a WebRTC session causes browser jitter buffer underruns and audible clicks.
Inside Slyber SFU Server, every spawned AI slot is backed by an isolated ring buffer (up to 600 frames / 12 seconds of speech) driven by a strict 20ms ticker that stamps outbound Opus packets with monotonic sequence numbers and exact +960 RTP timestamp increments at 48 kHz.
3. Sub-60ms VAD Barge-In Without Decrypting Audio
Natural human-AI dialogue requires immediate interruption: the moment a user starts speaking, the bot must stop talking. Decrypting Opus payloads on the SFU to run neural voice activity detection would burn CPU cores and break RFC 9605 SFrame end-to-end encryption.
Instead, the SFU inspects the unencrypted RFC 6464 (urn:ietf:params:rtp-hdrext:ssrc-audio-level) RTP header extension on incoming human audio packets:
// When human voice energy crosses threshold for 3 consecutive frames (~60ms):
1. Auto-flush immediately clears all queued unspoken frames in the bot's 20ms pacer
2. SFU emits {"type": "ai_vad_barge_in", "participant_id": "user-1"} over /ai/ws
3. After 360ms of trailing silence, SFU emits {"type": "ai_vad_speech_end"}| Pipeline Stage | Mechanism | Latency & Scalability Benefit |
| AI Ingress / Egress | Multiplexed binary WebSocket (/ai/ws?mode=mux) | Up to 6 distinct AI voices over 1 TCP socket |
| Jitter Elimination | 20ms server-side RTP pacer (+960 timestamp step) | Smooth browser playback from bursty LLM output |
| Interruption Detection | RFC 6464 header inspection (3-frame onset) | <60ms barge-in flush with zero audio decryption |
| Zero-Trust Compatibility | Works alongside SFrame E2EE + MLS key rotation | Compatible with REQUIRE_E2EE=true rooms |
Because voice energy is read from the RTP header extension, barge-in detection operates at line rate with zero cryptographic overhead—even when REQUIRE_E2EE=true is enforced across the room. Explore the full deployment benchmarks on the Slyber SFU Server case study or get started on slyber.com.