Why ZippyDB Needed a Proxy Layer
ZippyDB is Meta's most widely used key-value store, backing product metadata, counters, and configuration at billions of operations per second. But its direct-access model created a dense many-to-many mesh of TLS connections: a typical client held tens of thousands of outbound connections, and a typical database host accepted tens of thousands of inbound ones. Every new client cohort made every database host worse.
Proxies appear wherever a large, diverse client population talks to a shared backend—connection poolers, service mesh sidecars, CDN edges, API gateways. Interposing between many callers and a shared resource buys three things: it bounds the problem, creates a home for shared work, and creates a control point. The tradeoff is a hop and one more tier to operate, but the trade pays off when the client population is large, diverse, and not yours to change.
The real win isn't the one-time connection reduction—it's the change in scaling behavior. Fan-in becomes independent of the client population.
This is a classic platform engineering pattern: when you can't change a million clients, insert a tier you can change. For a related case study on platform engineering at scale, see how Santander slashed infrastructure provisioning from 90 days to hours.
Source: Meta Engineering Blog
Collapsing the Request Stream: Batching and Coalescing
The proxy's shared vantage point enables something no client library can: combining work across unrelated callers. A shared batcher on each host groups requests headed for the same destination, keyed by use case and physical shard, and merges them into one backend RPC.
# Conceptual illustration of request coalescing at the proxy tier
# (Meta's implementation is C++, but the pattern is language-agnostic)
import asyncio
from collections import defaultdict
class BatchCoalescer:
def __init__(self, linger_ms=2, max_size=100):
self.linger_ms = linger_ms
self.max_size = max_size
# Map: (use_case, shard) -> list of pending futures
self.batches = defaultdict(list)
async def request(self, use_case: str, shard: int, key: bytes):
loop = asyncio.get_event_loop()
fut = loop.create_future()
bucket = self.batches[(use_case, shard)]
bucket.append((key, fut))
if len(bucket) >= self.max_size:
await self._flush(use_case, shard)
else:
# Schedule a flush after the linger window
loop.call_later(self.linger_ms / 1000, self._schedule_flush, use_case, shard)
return await fut
async def _flush(self, use_case, shard):
batch = self.batches.pop((use_case, shard), [])
if not batch:
return
# Coalesce duplicate keys: fetch once, fan out
unique_keys = {k for k, _ in batch}
results = await self._backend_fetch(use_case, shard, unique_keys)
for key, fut in batch:
if not fut.done():
fut.set_result(results[key])
Two benefits aren't about efficiency at all. First, coalescing under a hot key: thousands of simultaneous callers collapse into a single backend read, so a hot key can never become a stampede against one replica. Second, it retires fragile client-side batching libraries that were CPU-hungry and a steady source of incidents.
Batching ships with two safety mechanisms: idle eviction (batch-map entries idle beyond a TTL are erased on the next flush) and an in-flight cap (rejects new executions when coroutines pile up faster than they drain).
The Arithmetic of Fan-In Reduction
Model the fleet as balls into bins. Throw B balls (shards a host touches) into H bins (hosts) and count distinct bins hit. The expected number is:
E[distinct bins] = H * (1 - (1 - 1/H)^B)
And a specific bin is hit with probability:
p = 1 - (1 - 1/H)^B
Fan-out is distinct bins hit; fan-in is that probability times the caller population. With round mock figures—20 regions, 500,000 database hosts, 30,000 proxy hosts, 1,000,000 clients, 50,000 shards per client—the model yields a ~19x reduction in total persistent connections, because each backend connection multiplexes many clients.
But the one-time reduction isn't the real point. In the direct model, database-host fan-in is linear in the client population: every new cohort makes every database host worse. With ZGateway, fan-in reduces to approximately R · S_host—regions times shard density per host—independent of both fleets. An unbounded number driven by everyone else becomes a bounded number you control.
What to Watch Out For
- Extra hop and operational tier: The proxy adds latency and a fleet to operate. It pays off only when the client population is large, diverse, and not yours to change.
- Batching OOM risk: Holding requests in memory is dangerous. The idle eviction TTL and in-flight cap are not optional—they're what make batching safe by default.
- Cache tier reshuffling cost: Moving a weight on a cache tier means moving keys. The load balancer must throttle changes to avoid costly reshuffling.
- Heterogeneous hosts: Equal treatment of unequal hosts produces hot outliers. A single fixed balancing policy cannot serve both a calm tier and a tier in shock.
Next Steps
If you're designing a similar tier, start by measuring your fan-in distribution. Then study ServiceRouter-style weighted consistent hashing and AIMD concurrency control. For the client-side counterpart, see Python typing adoption trends and the challenges that remain.
Operating ZGateway in Production
ZGateway runs across dozens of regions in a handful of tiers: one large general-purpose tier for the long tail, dedicated tiers for the largest customers, and a separate high-throughput proxy tier. They differ in footprint by more than an order of magnitude, and individual tiers aren't uniform due to stacking. Rich per-use-case observability is what makes admission control and load balancing safe on shared infrastructure.
Tenant Isolation and Admission Control
Discriminant Load Shedding (DLS) maps every request to a per-tenant bucket, keyed by use case and split by priority. When a tenant floods the tier, its own bucket fills and its excess is shed while every other bucket keeps draining. In a controlled overload at >90% CPU across ~1,350 active tenant buckets, only 6—the actual noisy neighbors—were shedding; the other ~1,344 executed 99.9% of requests with zero rejections.
Cross-Region Resilience
Three mechanisms extend ZGateway beyond strict regionality: global routing (a routing table spanning regions), mega-regions (grouping geographically close regions into one locality), and rings (declaring exactly which regions back each other up, in what proportion). The failover signal matters as much as the routing—a simple regional CPU average smooths over exactly the hot conditions you need to catch.
The Road Ahead
Three directions stand out: agent-operated heuristics (exposing control loops as a structured surface for AI agents), co-location (pushing part of the gateway beside the ZServer host for latency-critical workloads), and a multi-process gateway (splitting responsibilities for hard fault isolation). Together they turn ZGateway from a smart tier into a programmable one.
The takeaway: When you can't change a million clients, build the tier you can change. Batching, coalescing, admission control, and cross-region failover all become possible at once—solved once by the team that knows the backend best.