Why Netflix Built Its Own LLM Serving Stack

Most teams consume LLMs through hosted APIs. Netflix went further—running the full stack from model deployment to inference inside its existing production environment. This wasn't just about avoiding vendor lock-in; it was about achieving low latency, deep customization, and seamless integration with the company's ML infrastructure.

The platform's architecture reflects a key principle: LLMs should not be special snowflakes. Every model, from XGBoost ensembles to large-scale LLMs, is scored through a unified JVM-based serving system that handles routing, A/B testing, feature fetching, inference, and logging. This unified approach reuses existing client libraries, health checking, and deployment pipelines, reducing operational complexity.

But building this platform came with hard trade-offs. Four decisions shaped it—engine selection, model packaging, API surface design, and deployment strategy. Each one constrained the next, and production revealed lessons that design phases often miss.

Netflix GPU inference cluster with Triton and vLLM servers Algorithm Concept Visual

Key Design Decisions and Their Production Lessons

1. Choosing vLLM as the Paved-Path Engine

Netflix initially built on TensorRT-LLM, a performant engine integrated with Triton. By summer 2025, open-source engines had largely closed the performance gap, and the workload mix had broadened to include embedding generation, prefill-only inference, autoregressive decoding, and custom models. After re-benchmarking, they selected vLLM for its:

  • Flexibility: Loads custom architectures without multi-step compilation.
  • Extensibility: Hooks for custom decoding logic, essential for constrained decoding.
  • Debuggability: Easier to inspect failures than compiled engines.
  • Familiarity: Many ML practitioners already used vLLM in research.

2. Packaging Models: The vLLM Backend vs. Python Backend

Triton offers two ways to package models, and the choice impacts maintainability significantly:

  • Python backend: Authors define explicit I/O tensor specs at packaging time. These specs freeze in the artifact and must match the frontend's expectations. Every frontend upgrade touching I/O specs requires coordinated changes, or requests fail at runtime.
  • vLLM backend: The artifact is just a JSON config pointing to model weights and tokenizer. Triton generates I/O specs dynamically, allowing models and frontend to evolve independently.

The vLLM backend is the architecturally correct default, but production exposed two issues:

  • Version mismatch: Triton's vLLM backend is compiled against a specific vLLM API. When they drift, the backend fails to load entirely. The platform must pin compatible versions and prevent model authors from overriding them.
  • Custom model logic: Models requiring custom preprocessing/postprocessing or non-standard execution must use the Python backend for full control. This escape hatch remains necessary for a subset of models.

3. OpenAI-Compatible HTTP Frontend

To avoid making LLMs special, Netflix exposes both gRPC and an OpenAI-compatible API. The OpenAI-compatible interface has become the de facto standard for the LLM ecosystem, so adopting it enables seamless graduation from hosted models to fine-tuned self-hosted ones.

Behind the API, Netflix reuses NVIDIA's Triton OpenAI-compatible frontend, but patched a critical gap: response_format was silently dropped before reaching vLLM, so JSON output requests proceeded without guided decoding and could return malformed JSON with no error. They now translate response_format into vLLM's guided decoding parameters at request time.

4. Deployment Strategies: Red-Black vs. Versioned

GPU deployments take longer to bring up than CPU services, and I/O schemas may change between versions. Netflix offers two strategies:

  • Red-Black: Deploys a new version alongside the current one, shifts traffic in phases, and supports atomic rollback. Ideal when the model interface is stable, but fails when I/O schema changes require coordinated consumer updates.
  • Versioned: Maintains independent deployments for each (modelId, modelVersion) pair. Consumers can wait for the new version to be ready before switching, while old versions keep serving legacy traffic. The trade-off is temporary GPU cost during transition.

Recommendation: Embed variable configurations (like tensor shapes) directly into the inference model to make it version-agnostic, enabling the cheaper Red-Black path.

Operational Notes: Boot Sequence and Metrics

Two operational details hit production gaps:

  • Cold-start latency: Downloading large LLMs at startup is slow. Netflix materializes models on Amazon FSx at announcement time, so warm starts hit a high-performance file system.
  • Unified metrics: vLLM writes metrics to PROMETHEUS_MULTIPROC_DIR as .db files; Triton reports its own. The built-in bridge surfaces only 9 of 40+ vLLM metrics. Netflix added a lightweight HTTP proxy merging both into a single /metrics endpoint, so existing dashboards and alerts work without modification.

Developer monitoring LLM serving metrics and constrained decoding IT Technology Image

Deep Dive: Constrained Decoding at Scale

Some production workloads require fine-grained control over token generation. Netflix pushes constraints inside the decode loop using vLLM's custom logits processor interface, modeling each constraint as a state machine. This ensures outputs are compliant by construction, avoiding costly post-inference validation.

Why the First Implementation Didn't Scale

In vLLM V0, custom logits processors run per-request. The GPU produces logits for the whole batch, but the CPU processes them sequentially due to the GIL. CPU time grows linearly with batch size, causing tail latencies. This bottleneck is invisible in single-request benchmarks but surfaces under realistic concurrency.

vLLM V1: Batch-Level Design

vLLM V1 moved logits processing to batch level. Netflix rewrote the processor to operate on batch-level data structures, and reimplemented the hot path in C++ with multi-threading to bypass the GIL. The V1 API requires explicit tracking of batch membership changes via update_state(batch_update), which is more complex but necessary for correctness.

Operational Hardening

Stateful constraint logic introduced two issues:

  • Partial prefills: V1 performs chunked prefilling, so a request can be prefilled over multiple engine steps. BatchUpdate lacks granularity to tell if a request was fully or partially prefilled, so they added internal tracking.
  • Preemption: Under memory pressure, vLLM may evict a partially completed request and reschedule it later with a different prompt. This breaks the state machine's assumption that the output token list grows monotonically. They detect when token history shrinks and reinitialize from the new prompt.

Architecture diagram of Netflix LLM serving platform with gRPC and HTTP paths Developer Related Image

Limitations and Next Steps

While the platform is robust, several areas remain under active development:

  • System prompt compression to reduce prompt length without sacrificing quality.
  • Asynchronous scheduling of vLLM V1 for better resource utilization.
  • Vectorized logits processors running as fused GPU kernels instead of CPU code.
  • Lower-precision model variants to decrease memory footprint and increase throughput.

Netflix will continue collaborating with the open-source community to evolve this space.

Conclusion

Netflix's in-house LLM serving platform demonstrates that running LLMs at scale requires careful trade-offs and constant iteration. The lessons—version pinning, silent API gaps, packaging trade-offs—highlight the importance of production feedback in shaping architecture. For teams considering a similar path, start with a unified serving layer and prioritize operational observability from day one.

For more on optimizing performance in production, check out our guide on query performance insights from Vercel CLI. And if you're exploring new commerce standards, see our analysis of the Universal Commerce Protocol.

Original source: Netflix Tech Blog

This content was drafted using AI tools based on reliable sources, and has been reviewed by our editorial team before publication. It is not intended to replace professional advice.