How an SFU Server Works

Visualization 

Step 1. Why a frame cannot be sent through the tube whole (RTP and SRTP)

• Each cube is placed into a standard cylindrical RTP capsule with a sequence number and exact departure timestamp stamped on its casing (so the cubes can be reassembled in the right order on the viewer's side).
• On the outside, for the duration of its flight through the street tube, the capsule is locked with an SRTP travel combination lock so no one on the street can swap out the cube.

Step 2. Why there is a switch in the city center instead of a reassembly plant (MCU vs. SFU)

1. The old assembly plant (MCU): Capsules from all 20 people flew into a large hall. Workers opened every capsule, took out all the cubes, glued the 20 faces into a single composite panel, sawed it back into cubes, and repacked them into capsules for each viewer. This took precious fractions of a second and required a massive facility.
2. The high speed automated switch (SFU): A modern hub - is a compact transparent junction where capsules containing casts are never opened at all. A capsule from you flies into the SFU junction at full speed, a sensor reads only the address tag on the outside in a millisecond, and an automated switch instantly fires copies of that capsule into the viewers' tubes.

Step 3. The switch cannot shrink the cargo, and the elevator tube is narrow (Simulcast and SVC)

• Method 1 - Three separate queues (Simulcast): The sender's machine at home creates three independent copies of the same frame at once - large (50 capsules), medium (15 capsules), and small (5 capsules) - and sends all three streams to the SFU junction. The switch at the station simply routes the large stream into the wide tube, and the small one - into the narrow elevator tube.
• Method 2 - Three color modular kit (SVC): To avoid sending three separate copies from home to the station and wasting tube space, the sender transmits a single set of capsules split by color:
• 5 green capsules carry the base outline of the frame (enough to assemble a low-resolution picture);
• 10 yellow capsules add medium sharpness on top of the base outline;
• 35 blue capsules add fine high-definition details. Now, when the viewer's tube in the elevator narrows, the SFU switch cuts off the blue and yellow capsules with a single flap movement, letting only the 5 green ones into the narrow tube!

Step 4. How the switch in the city center knows that a tube in an elevator 50 km away is starting to clog (TWCC, GCC, BWE)

1. Finish line interval counter (TWCC): Upon leaving the SFU junction, every capsule is stamped with a continuous sequence number (1, 2, 3...). Right before the entrance to the viewer's phone sits an optical ring with a stopwatch. It records the exact arrival time of each capsule and every half second sends a brief TWCC report back to the station: "Capsules #101, 102, and 103 left the station 5 milliseconds apart, but arrived at my finish line stretched out to 25 milliseconds apart!"
2. Queue build up sensor (GCC): The GCC computing unit at the SFU station reads this report and sees simple mechanics: if the time gap between capsules at the exit has grown, they have started slowing down in a queue before the narrow bottleneck of a router or cell tower. The capsules are still intact, but congestion in the tube is already growing.
3. Bandwidth valve (BWE): Receiving the signal from the GCC sensor, the BWE regulator calculates the actual diameter of the tube at that exact second and immediately turns the flap on the SFU switch: "Throughput dropped - cut off the blue and yellow SVC layers, feed only the green base layer into the tube!"

Step 5. How to hide the cubes from the SFU station itself without blinding the switch (Insertable Streams and MLS)

• Internal opaque case (Insertable Streams): Right in the sender's room, before the frame cube drops into the transparent RTP capsule, a side mechanism seals the cube itself inside an opaque internal case locked with the private key of the room's participants. Meanwhile, an open colored label remains on the outside wall of the RTP capsule itself ("Green SVC layer, frame #10"). Now the SFU switch clearly sees the colored label and cuts off extra layers for the elevator, while the cube inside the case remains a black box to the station.
• Key distribution pyramid (MLS): So that 1,000 people in a call share the same key to this internal case and have it change instantly whenever anyone leaves (otherwise the person who left could read others' capsules), participants are linked into a pyramid of pairs (by 2, by 4, by 8 people). When one person leaves the room, the new key is passed up through just 10 levels of the pyramid from bottom to top instead of cycling through all 1,000 participants one by one.

Step 6. Fourth consequence: How to route a tube out of your gated courtyard and pass street checkpoints (STUN, TURN, DPI, ICE, Happy Eyeballs)

• External courtyard gate sign (STUN): Sitting inside the courtyard behind the router's fence, your machine only knows its internal apartment number, and cannot see which external hatch in the courtyard wall its tube exits through onto the street. The machine fires a test ball to a street STUN beacon, which sends back a reply: "Your tube exits onto the street through external hatch #48200".
• Inspection checkpoints (DPI) and the bypass sleeve (TURN): On some streets stand ISP DPI scanners that X-ray the tubes and slam the valve shut as soon as they spot RTP capsules inside. And in strict office buildings, direct hatches to the street are welded shut altogether. In those cases, capsules are routed through a TURN bypass transit hub inside the standard opaque shell of a regular website (HTTPS on port 443). The DPI scanner sees standard banking traffic and lets it through nonstop.
• Automatic working tube selection (ICE and Happy Eyeballs): Between home and the SFU station there are always several routing options: a direct overhead tube (UDP), a ground rail tube (ICE-TCP), an old IPv4 bridge, a new IPv6 bridge, and a TURN bypass sleeve. The ICE dispatcher gathers a list of all available hatches (in Trickle ICE mode it tests the very first hatch immediately without waiting for the rest, while the SFU station itself in ICE-Lite mode simply waits for incoming balls at its fixed address). So the user does not have to wait for each bridge to be tested one after another, the Happy Eyeballs starter valve fires test balls into parallel tubes simultaneously: whichever tube returns an echo first within hundredths of a second - opens immediately for the main stream.

Step 7. Finale: Why for 1,000,000 viewers a single tube is replaced with a multi barrel cylinder and district warehouses (QUIC and MoQ)

• If you route a million people through an old single track tube (TCP), where all capsules are linked into one rigid queue, the very first stuck video capsule will block the entire tube behind it, audio included.
• And if you try to distribute a million streams via classic WebRTC (RTP over UDP), the SFU junction will have to turn a million individual speed valves and re encrypt every capsule personally for each viewer - because standard city warehouses (CDNs) do not know how to copy an unnamed RTP stream.

• Hose with dozens of independent inner barrels (QUIC): Instead of a single tube between the server and viewer, a wide protected QUIC sleeve is laid. At its end sits a floating coupling (Connection ID): when a person leaves home and switches from Wi-Fi to 5G, the coupling snaps onto the new tube in a millisecond without stopping the stream. And inside the sleeve itself, like in a revolver cylinder, run dozens of separate smooth barrels. If a capsule gets stuck in one barrel, cargo keeps flying through all the neighboring barrels inside the same sleeve without the slightest hitch.
• A dedicated barrel for every second of video (MoQ): The MoQ dispatcher distributes the media stream across these internal QUIC barrels following three simple rules:
• Top barrel audio only: voice capsules always fly out first, bypassing any video queues.
• Every new second of video is loaded into its own adjacent barrel! At the front of the barrel goes a heavy starter capsule (the full keyframe for that second), followed down the same barrel by 30 lightweight motion capsules for that second.
• Overtaking a jam without waiting: if a viewer steps into an elevator and a capsule gets stuck in Barrel #10 (for the 10th second), the system does not wait for it to be pushed through! In that exact fraction of a second, a fresh starter capsule for the new second is fired down the neighboring Barrel #11, bypassing the jam, while the stuck 10th barrel is simply shut off and discarded.
• Duplication through standard city warehouses (MoQ Relays / CDN): Because each one second magazine in MoQ has clear serial markings on the outside ("Channel 1, Video, Second #11"), the central SFU station no longer needs to run a million tubes to viewers. It sends just one magazine to ordinary district warehouses along the route (CDN servers), and those, without ever opening the encrypted capsules, simply clone the ready made one second magazines from their shelves into hundreds of thousands of local viewers' QUIC sleeves.

Glossary of Terms and Standards

WebRTC (Web Real Time Communication)

MCU (Multipoint Control Unit)

SFU (Selective Forwarding Unit)

RTP and SRTP (Real time Transport Protocol / Secure RTP)

SVC (Scalable Video Coding)

BWE (Bandwidth Estimation)

TWCC (Transport Wide Congestion Control)

GCC (Google Congestion Control)

STUN (Session Traversal Utilities for NAT)

TURN (Traversal Using Relays around NAT)

ICE, ICE Lite, Trickle ICE, and ICE TCP

DPI (Deep Packet Inspection)

Insertable Streams (WebRTC Encoded Transform)

MLS (Messaging Layer Security)

WHIP and WHEP (WebRTC HTTP Ingestion / Egress Protocol)

QUIC (Quick UDP Internet Connections)

MoQ (Media over QUIC)

Happy Eyeballs (Fast Fallback)

Why You Need an SFU

How the Server Processes Hundreds of Thousands of Packets per Second

1. Asynchronous Batching I/O

2. Lock Free Architecture

• Worker process isolation A completely independent worker process or thread (Worker) runs on each CPU core with its own event loop and its own routing table. In stateless relays (for example, in high load TURN servers), threads can share a single port via the Linux kernel option SO_REUSEPORT, where the network card distributes packets across cores by hashing addresses and ports (5 tuple). Meanwhile, in servers like Mediasoup, where a specific room (Router) strictly lives inside one specific Worker process, each worker process listens on its own dedicated port (WebRtcServer), or an eBPF filter (SO_ATTACH_REUSEPORT_EBPF) is placed in front of them to route packets straight to the right core based on the session identifier (ice ufrag). Threads share no memory with each other, so they need no mutexes.
• Lock free ring buffers If one thread still needs to pass a packet to another (for instance, when the sender and viewer are served by different cores), an in memory ring queue is used. For a "single producer - single consumer" pair, no locks are needed at all one thread modifies only the write index, and the other - only the read index using fast atomic operations and memory barriers (Acquire / Release) in just a few CPU cycles. And if multiple threads write to the queue at once (MPSC / MPMC), pointers are advanced via hardware atomic compare and swap instructions (CAS - Compare And Swap, taking 15–40 cycles) without system calls or stopping threads.

3. Strict CPU Core Pinning (CPU Affinity)

• Loss of hot L1 and L2 cache Every physical CPU core has its own ultra-fast memory - L1 and L2 cache (access time around 1–4 nanoseconds versus 60–100 nanoseconds for main RAM). While a thread runs on a single core, the RTP packet processing code and SRTP encryption contexts sit right inside the L1/L2 cache. As soon as the thread migrates to a neighboring core, its cache there is empty, and the processor stalls for hundreds of cycles fetching data from slow main memory (Cache Miss).
• Cold address translation buffer On the new physical core, the cache that translates virtual memory addresses to physical ones (TLB) lacks entries for that thread's memory pages, triggering a series of expensive TLB Miss penalties when accessing network buffers.
• NUMA architecture penalties On multi socket servers, migrating a thread to a core on a different CPU socket forces it to pull network buffers from remote memory across the inter processor interconnect at higher latency.

What Happens to Video in an Elevator and How BWE, TWCC, and GCC Work

1. Splitting a frame into dozens of packets A single 1080p video frame (especially a full keyframe - I frame) weighs between 30 and 100 KB. Since the maximum size of a single network packet on the internet (MTU) is around 1,200–1,500 bytes, one video frame is sliced into 30–80 separate RTP packets.
2. Cell tower buffer overflow and packet drops When a burst of dozens of packets hits the narrow bottleneck of the radio link in the elevator, the memory queue at the base station overflows in a fraction of a second (Bufferbloat), and the carrier's router starts forcibly dropping packets that do not fit (Packet Loss).
3. Decoder stall and retransmission storm (PLI Storm) The video decoder in the phone cannot assemble a frame if even a few of its 50 packets are lost. Moreover, the delta frames (P frames) following a keyframe store only mathematical changes relative to the previous frame. Having lost its reference point, the decoder leaves the last intact image on screen and sends an emergency PLI (Picture Loss Indication) signal to the server - "send a new full keyframe". If the server responds by sending a new heavy keyframe into the elevator, it will clog the narrow channel for good.

How the TWCC Data Collection Protocol Works

1. Continuous outbound numbering The sender (for example, the SFU when distributing a stream to a viewer) adds a special header extension to every outgoing RTP packet - a single monotonically increasing packet number (Transport Wide Sequence Number 1, 2, 3, 4...), and records the exact departure time of each number in its own memory with microsecond precision.
2. Arrival log on the phone The recipient's smartphone receives these packets and logs the exact arrival time of each number according to its internal clock.
3. Feedback receipt Every 50–100 milliseconds, the receiver packs these measurements into a compact control packet called RTCP (TransportLayerFeedback) and sends it back "Packets #101–120 received, arrival intervals between them were so many quarter-milliseconds (250 µs steps), and packet #115 did not arrive".

How the GCC Algorithm Calculates Channel Speed from TWCC Data

• Delay based controller (primary) It compares the time intervals at which packets left the sender (tsend) with the intervals at which they arrived at the receiver (trecv). The difference between them is called inter-packet delay variation (One Way Delay Gradient). If the server sent two packets 5 ms apart, and they arrived at the phone 8 ms apart, then 12 ms, then 18 ms - a mathematical trend filter (Trendline filter or Kalman filter) inside GCC immediately raises an alarm the queue in the router buffer in front of the phone has started to grow. No packets are lost yet and the picture is still smooth, but the channel is already congested. GCC immediately issues a command to lower the bitrate, preventing video freezes ahead of time.
• Loss based controller (safety net) It inspects the fraction of lost packets in the TWCC receipt. If loss is under 2% - the algorithm allows a gradual bitrate increase to probe the channel ceiling. If loss is between 2% and 10% - the speed is held at its current level. If more than 10% of packets are lost - the bitrate is urgently slashed in proportion to the loss rate.

Simulcast and SVC How to Shrink the Picture Without Transcoding

Simulcast (Three Separate Pipelines)

SVC (Layered Nesting Doll Principle)

The ICE Protocol Family and Working with a TURN Server

All ICE Modes Explained in Plain English

• Full ICE ( RFC 8445) The standard mode used by browsers and mobile apps when calling each other directly. Both devices first gather all their possible addresses (candidates) a local address on the home network (Host candidate), the router's external address discovered via a STUN server (Server Reflexive candidate), and a fallback address on a TURN relay (Relayed candidate). Then both devices mutually probe all address pairs with check packets until they find the shortest working route.
• ICE Lite (Lightweight Server ICE) A simplified mode designed specifically for SFU media servers. Because an SFU always sits in a data center on a fixed public IP address and does not hide behind a home router, it does not need to query STUN servers or initiate connectivity checks toward clients. In ICE Lite mode, the server simply publishes its IP and port and replies to incoming checks from the browser. This saves server CPU time and speeds up connection setup.
• Trickle ICE ( RFC 8838) In legacy ICE mode, the browser first spent several seconds gathering every single one of its addresses (polling network cards, STUN, and TURN) and only then sent them to the server in one large batch. In Trickle ICE mode, the browser sends candidates incrementally in parallel as they are discovered sending the local address in the very first millisecond, and the STUN address 20 ms later. As a result, the connection to the SFU is established almost instantaneously.
• ICE TCP (Direct ICE over TCP, RFC 6544) On some public Wi Fi networks and in offices, system administrators block the UDP protocol. The ICE TCP standard allows an SFU to open a regular TCP port and accept media packets directly from the browser without involving a TURN server (each RTP packet is prefixed with a two-byte length header according to RFC 4571). It comes in three subtypes passive (the server waits for an incoming connection - the primary mode for an SFU), active (the client opens the connection), and so (simultaneous open).
• ICE Restart A mechanism for fast connection recovery when switching networks. When a person walks out of their house during a call and their phone switches from Wi Fi to 4G/5G, its IP address changes. To avoid dropping the entire call and recreating the room session, the browser generates new ICE credentials (ice ufrag and ice pwd), establishes a new route to the SFU in a fraction of a second, and continues sending video into the exact same stream.

Why Do You Need a TURN Server in Front of an SFU If You Have ICE Lite and ICE TCP?

• Disguising as HTTPS (TURN over TLS) On a direct ICE TCP connection to an SFU, the handshake begins with plaintext unencrypted STUN packets. A DPI inspection system on port 443 sees that the payload is not genuine TLS web traffic and drops the connection. Only when connecting to a TURN server via turns443 does the browser first establish a full cryptographic TLS session (the exact same kind used when logging into online banking), hiding all WebRTC packets inside it.
• Traversing office HTTP proxies In locked-down corporate networks where computers have no direct internet access at all, the browser can punch a tunnel to the TURN server through the standard corporate proxy server using the HTTP CONNECT method.
• Single-port multiplexing In modern architecture, a smart load balancer (or a TURN module built into the SFU) sits on external port 443, accepts the disguised TLS stream, strips the outer TLS wrapper in server memory, and immediately passes the packets into the SFU core without network forwarding delays.

Server Geo Cascading

End to End Encryption in Group Calls Insertable Streams and MLS

How Insertable Streams Work (WebRTC Encoded Transform)

1. Once the video codec (for example, VP9 or AV1) compresses a video frame, the browser passes that frame's byte array to a dedicated background script (Web Worker).
2. The script encrypts only the picture itself (the payload) according to the SFrame (Secure Frame, RFC 9605) standard using a secret group key that the server does not possess.
3. The encrypted frame returns to the browser, which packs it into standard RTP packets and attaches an open outer label - the unencrypted RTP header and the Dependency Descriptor.
4. As a result, the SFU reads the open label ("this is frame #540, medium 540p layer, not a keyframe"), freely switches quality layers to match viewers' internet speeds, yet sees only encrypted noise inside the box itself.

How the MLS Protocol Works (Messaging Layer Security, RFC 9420)

• All room participants are placed in the bottom nodes (leaves) of a virtual binary tree, while the shared group media encryption key sits at the very top (the root of the tree).
• Each participant knows the secret keys only along their direct branch from their leaf up to the root of the tree.
• When one person leaves a 1,024 participant room, the system does not need to send 1,024 individual messages - it only needs to update 10 nodes along a single branch of the tree 2(1024)=10. Thanks to logarithmic complexity O(N), key rotation in massive conferences takes just a few milliseconds.

Universal WHIP and WHEP Gateways and Connecting AI

• WHIP (WebRTC HTTP Ingestion Protocol, RFC 9725 - server ingress) Allows any streaming software (including OBS Studio and FFmpeg) to start broadcasting into any SFU in a fraction of a second by sending session parameters (SDP Offer) in a single standard HTTP POST request to the server's web URL and receiving a 201 Created response (SDP Answer).
• WHEP (WebRTC HTTP Egress Protocol - viewer egress) Allows any lightweight player in a browser, mobile app, or Smart TV to pull a live stream from an SFU via an ordinary web link in a single HTTP request without bundling heavy proprietary SDKs for a specific server.
• Internal AI gateways Modern SFUs feature direct server side interfaces for plugging in neural networks (speech recognition and synthesis models or multimodal voice assistants). The neural network joins the room at the server memory level as a virtual participant, receives a clean audio stream without internet latency, and instantly returns a generated voice response.

Deep Dive into QUIC and MoQ The Future of Video Delivery

1. Complex and slow connection setup To start a call, a device must complete a chain of separate handshakes (ICE probing, followed by a cryptographic DTLS handshake, followed by SRTP setup).
2. Incompatibility with inexpensive Content Delivery Networks (CDNs) In WebRTC, every viewer maintains an individual encrypted session with the SFU server. If 100,000 people watch a broadcast, you have to rent a hundred powerful SFU servers, because standard caching CDN servers (Cloudflare, Akamai, Fastly) only know how to serve static files (as in HLS/DASH with 5–15 seconds of latency) and do not understand streaming WebRTC.

How the QUIC Protocol Works (RFC 9000)

• Instant start (1 RTT and 0 RTT) In QUIC, TLS 1.3 encryption is built directly into the transport protocol itself. Connection setup and key exchange happen simultaneously in a single round trip (1 RTT), and when reconnecting to a known server, media data flies from the very first millisecond (0 RTT).
• Independent streams without queue blocking (No Head of Line Blocking) If you send audio and video over standard TCP, losing a single packet from a heavy video frame forces the operating system to halt reading the entire TCP connection until the lost packet is retransmitted. Because of this, audio stutters right alongside the video, even though the voice packets arrived at the phone long ago. In QUIC, dozens of lightweight independent logical streams (QUIC Streams) and fast datagrams (QUIC Datagrams, RFC 9221) are created inside a single connection. If a packet is lost in a video stream, the neighboring voice stream keeps playing without a single hiccup.
• Seamless Connection Migration In older protocols, a connection is strictly tied to an "IP address + port" pair. When you leave home and your smartphone switches from Wi Fi to 5G, the IP address changes and the connection drops (requiring an ICE Restart). In QUIC, every connection is identified by a cryptographic Connection ID (CID) in the packet header. When the phone switches networks to 5G, it simply keeps sending packets with the old Connection ID from the new IP address, and the server recognizes it on the fly without dropping a single frame.

How the MoQ Standard Works (Media over QUIC)

• Object data model (Tracks → Groups → Objects) Instead of an endless continuous stream of packets (as in RTP), MoQ organizes a broadcast into a clear hierarchy of named objects. A broadcast channel is divided into Tracks separate audio, separate 1080p video, separate 360p video, separate subtitles. Each video track is sliced into Groups - short independent video segments (typically a GOP - Group of Pictures lasting 1–2 seconds and starting with a full keyframe). Inside a group sit individual Objects - specific video frames streamed frame by frame as they leave the camera with tens of milliseconds of latency.
• A dedicated QUIC stream per group of pictures and smart priorities Each new group of frames (GOP) is sent in its own independent QUIC stream with an explicitly assigned Delivery Priority. If a viewer walks into an elevator and an older group of video frames gets stuck in transit, the server does not try to endlessly retransmit stale video - it gives maximum priority to 1) audio, and 2) the freshest group of frames, while simply closing the old stuck stream with a STOP_SENDING command. Thanks to this, the video never falls behind the live conversation.
• Inexpensive scaling via caching relays (MoQ Relays / CDN) Because every frame in MoQ has a unique name (Broadcast ID + Track ID + Group Number + Object Number), intermediary servers no longer need to understand video codecs or decrypt media payloads. A standard edge CDN server (MoQ Relay) receives a single stream from the publisher with 100 ms latency, stores the named chunks in fast RAM, and distributes them to hundreds of thousands of viewers using a publish/subscribe (Pub/Sub) model, while simultaneously allowing viewers to rewind the live broadcast back several seconds.

Why UDP Blocking Is Not a Threat

• Door UDP443 Accepts around 90% of all media traffic at maximum speed and minimal latency. A smart network demultiplexer at the entrance inspects the first bytes of each packet and routes traffic in nanoseconds sending STUN/DTLS/SRTP packets to the classic WebRTC engine, and QUIC packets - to the MoQ engine.
• Door TCP443 Accepts signaling control commands, HTTP WHIP and WHEP requests, direct ICE TCP connections, fallback HTTP/2 streams, and TURN over TLS traffic disguised as an ordinary HTTPS website, guaranteeing 100% connectivity for calls and broadcasts across any network in the world.

Ready Made Platforms for Building Video Servers

• Mediasoup (C++ and Rust) An ultra fast low level toolkit library with a Shared Nothing architecture for those who want to squeeze the maximum number of streams out of every CPU core and write their own custom room management business logic in Node.js or Rust.
• Janus WebRTC Server © A lightweight modular industry veteran with a minimal memory footprint, well suited for embedded systems and bridging browsers with legacy office SIP telephony and RTSP surveillance cameras.
• Pion / Ion SFU (Go) A flexible suite of WebRTC protocol libraries written in Go that lets you quickly build your own specialized media server, custom gateway, or server side recorder from scratch.
• Slyber SFU Server (Go) A turnkey distributed media server out of the box featuring built in geo cascading, an integrated TURN/TLS gateway on port 443, WHIP/WHEP support, and a ready made framework for connecting voice and video AI agents.