How Meta Deployed AV1 for Real-Time Video Calls Challenges, ML Eligibility, and Error Resilience

Why Meta Bet on AV1 for Real-Time Communication

In 2023, Meta began rolling out AV1 support for high-end devices in Messenger and WhatsApp. By 2026, AV1 is enabled on the majority of mobile devices across Meta's Real-Time Communication (RTC) apps. The motivation is simple: AV1 delivers the same visual quality as H.264/AVC while using at least 20% less bandwidth — and often much more on capable devices.

For users in emerging markets where video bitrates can drop to 10–400 kbps, this difference is dramatic. In side-by-side tests at 100 kbps, H.264 video appears noticeably blurry while AV1 remains much clearer. AV1 also excels at screen content (text, UI) thanks to palette mode and intra-block copy, which are part of its main profile.

But bringing AV1 to real-time calls is far harder than using it for video-on-demand. RTC requires end-to-end latency under 300ms, real-time encoding/decoding on battery-powered devices, and graceful handling of network fluctuations. Meta's engineering blog (source linked below) details exactly how they tackled each obstacle.

Source: Meta Engineering Blog

The Core Challenges & Solutions

1. Encoder/Decoder Selection

Problem: Open-source AV1 encoders (like libAOM) increased power consumption by 14% on a Pixel 8 compared to H.264, and added 1.7 MB to the app binary (600 kB compressed). For a company serving billions of users, binary size affects update success rates, startup time, and crash rates.

Solution: Meta adopted an internal low-complexity encoder with power consumption similar to H.264 baseline. They developed an ultra-low-complexity preset that matches H.264 encoding complexity while maintaining AV1's compression gains. For decoding, they chose dav1d for its superior power efficiency.

Binary size mitigation:

• Dynamic-download approach failed due to network/device issues.
• Instead, they optimized the quantization matrix (QM) tool (halved its size), removed unused tools (saving 60 kB), and shared codec libraries across features (e.g., video message transcoding).

2. Device Eligibility: ML-Based Approach

Problem: Android has thousands of device models. Relying on specs like RAM, release year, or OS version proved unreliable. Some 2023 octa-core phones couldn't even encode 320×180@15fps in real time due to CPU throttling.

Solution: Meta built an ML-based device eligibility framework that uses real-world performance metrics (not lab data) to compute an rtc_score. The model was iteratively refined:

• Model V1.1 (Aug 2025): Expanded AV1 traffic.
• Model V2: Two-tier approach separating high-end and low-end devices.

3. Codec Complexity Adaptation

Even eligible devices struggle during calls. Meta implemented three adaptive mechanisms:

# Pseudocode: Adaptive encoder preset adjustment
if encoding_latency > THRESHOLD_HIGH:
    decrease_encoder_complexity()
elif encoding_latency < THRESHOLD_LOW:
    increase_encoder_complexity()

# If still too high, switch to H.264
if encoding_latency > MAX_ALLOWED:
    switch_codec_to_h264()

# Peer decoding check
if peer_decoding_latency > PEER_THRESHOLD:
    sender_switches_to_h264()

This enables an asymmetric codec design: mid-range phones send H.264 but receive AV1 from high-end peers, significantly boosting coverage.

4. Rate Control: Preventing Overshoot & Undershoot

Problem: Bitrate spikes cause congestion and video freezes. VBV (Video Buffering Verifier) delay must stay under 200ms. Overshoot is obvious, but undershoot also hurts — it misleads bandwidth estimation and slows bitrate ramp-up.

Solution:

• Encoder tracks VBV buffer status and reduces rate of subsequent frames after overshoot.
• Key-frame bitrate is strictly controlled to avoid spikes.
• Reference Picture Resampling (RPR) allows resolution changes without generating a key frame.
• Revised algorithm to prevent both overshoot and undershoot.

5. Error Resilience: Temporal Layers & LTR

Temporal Layers (TL):

• Frames organized into base layer (T0) and enhancement layers (T1, T2…).
• If enhancement packets are lost, decoding continues with base layer — no freeze.
• FEC protects base layer; retransmissions are prioritized by layer.
• TL is turned on adaptively only when packet loss rises, to avoid compression efficiency loss.

Long-Term Reference (LTR):

• Periodically, the encoder emits an LTR frame stored in a 4-frame buffer.
• Upon packet loss, the receiver requests an LTRP (LTR-predicted) frame — instantly resynchronizing without a costly key frame.
• LTR is implemented via a proprietary RTP header extension that carries an explicit LTR indicator and frame_id for ACK tracking.

# LTR request handling (simplified)
def handle_ltr_request(request):
    if has_acked_ltr_in_buffer():
        encode_ltrp_frame()
    else:
        encode_key_frame()  # fallback

LTR reuses existing periodic high-quality frames, so there's minimal overhead.

Limitations and Caveats

While Meta's approach is impressive, it's not a silver bullet:

• Power consumption still higher on low-end devices, even with optimized encoder.
• Group calls remain challenging — decoding multiple AV1 streams simultaneously is much harder than 1:1.
• Hardware AV1 support is needed for truly widespread, high-quality adoption across all device tiers.
• Temporal layers reduce compression efficiency when always enabled; adaptive switching adds complexity.
• LTR requires tight network-encoder coordination — not all codec implementations support this.

Next Steps for Your Learning

If you're implementing or evaluating AV1 for your own RTC product:

1. Start with dav1d decoder — it's the most power-efficient open-source option.
2. Build a device capability model using real performance metrics, not just specs.
3. Implement adaptive codec switching — don't assume a device that can decode AV1 can also encode it.
4. Test rate control under frequent bitrate/resolution changes — this is where most encoders fail.
5. Consider temporal layers and LTR for lossy networks, but measure the compression trade-off.

For deeper background on codec optimization techniques, check out Beyond Matrix Math: How NVIDIA Blackwell Ultra Tackles the Softmax Bottleneck — it covers similar trade-offs in AI inference. And if you're curious about other 2026 tech trends, CSS in 2026: Alpha Function, Grid Lanes, and What Happened at CSS Day explores new frontend capabilities.

Conclusion: AV1 Is the Future of RTC — But It Takes Engineering

Meta's journey shows that adopting AV1 for real-time communication is not just a codec swap. It requires:

• Low-complexity encoder design that matches H.264 power consumption.
• ML-based device eligibility that learns from real-world data.
• Adaptive codec and preset switching that responds to device health.
• Sophisticated rate control to avoid both overshoot and undershoot.
• Temporal layers and LTR for graceful degradation under packet loss.

The result? AV1 now powers the majority of mobile devices in Messenger and WhatsApp calls, delivering better quality to users on constrained networks. As hardware support expands and ML models improve, coverage and quality will only increase.

Key takeaway for developers: AV1's benefits are real, but deploying it in an RTC product requires a holistic system-level approach — not just integrating a library. Start small, measure everything, and iterate.