Cloudflares Code Orange How Fail Small Engineering Prevents Global Outages

Why This Matters

In November and December 2025, Cloudflare experienced two global outages that disrupted traffic for millions of users. The root causes? A machine learning classifier rollout in Bot Management and a control flag in the global configuration system. Both incidents shared a deadly common pattern: code that assumed inputs would always be valid, with no graceful degradation.

Cloudflare’s response was an intensive, company-wide engineering initiative called “Code Orange: Fail Small.” Over two quarters, the team rebuilt how configuration changes are deployed, how failures are absorbed, and how institutional knowledge is captured and enforced. This article breaks down the three pillars of that initiative and what they mean for the reliability of your infrastructure.

Reference: The full official post is available at the Cloudflare Blog.

Pillar 1: Health-Mediated Configuration Deployments

The Problem

Before Code Orange, configuration changes could reach Cloudflare’s network instantly. There was no staged rollout, no real-time health monitoring, and no automatic rollback. A single bad config file could bring down the entire network.

The Solution: Snapstone

Cloudflare built a new internal component called Snapstone. It bundles configuration changes into packages and releases them gradually, applying the same health-mediation principles used for software deployments.

# Conceptual Snapstone deployment flow (simplified)
stages:
  - name: canary
    percentage: 2
    health_checks:
      - latency_p99 < 200ms
      - error_rate < 0.1%
    rollback_on_failure: true
  - name: regional
    percentage: 25
    health_checks:
      - latency_p99 < 250ms
      - error_rate < 0.5%
    rollback_on_failure: true
  - name: global
    percentage: 100
    health_checks:
      - latency_p99 < 300ms
      - error_rate < 1.0%
    rollback_on_failure: true

Key characteristics of Snapstone:

• Flexible units: Teams define any configuration unit (data file, control flag, etc.) that needs health mediation.
• Automatic rollback: If health checks fail at any stage, the deployment is reverted before it affects production traffic.
• Unified approach: Previously, teams had to build their own canary deployment tooling. Snapstone provides a single, reusable system.

Related: For a broader view on how progressive rollout patterns apply to AI agent orchestration, check out Vercel Chat SDK Adapter Directory — it uses similar wave-based deployment principles.

Pillar 2: Reducing the Impact of Failure

The Principle: Fail Gracefully

Cloudflare’s teams reviewed every critical service for failure modes and implemented three strategies:

1. Fail stale: Use the last known good configuration when new data can’t be read.
2. Fail open: If no old config is available, serve traffic with reduced functionality rather than dropping it.
3. Fail close: Only block traffic if serving degraded data would cause worse outcomes.

Example: Bot Management ML Classifier

If the November 2025 outage scenario repeated today:

• The system would detect unreadable data at the canary stage (2% traffic).
• It would refuse the new config and fall back to the old one.
• If the old config was also unavailable, it would fail open — serving traffic without the latest ML model — rather than dropping requests.

Process Segmentation for Blast Radius

Cloudflare has begun segmenting services into independent copies for different customer cohorts. For example, the Workers runtime now runs separate instances for free vs. paid customers. Changes are deployed to the free segment first. In a seven-day period, the Workers deployment process was triggered more than 50 times, each in waves that propagate to the edge.

Also see: Spotify’s approach to release process dashboards and automation offers complementary insights on managing deployment frequency and blast radius. Read more in Inside Spotify’s Release Engine Dashboard Design & Automation Insights.

Pillar 3: The Codex — AI-Enforced Engineering Standards

From Incidents to Institutional Memory

The November and December outages were caused by two simple coding mistakes:

• A Rust service called .unwrap() instead of handling an error.
• Lua code indexed an object that didn’t exist.

To prevent these patterns from recurring, Cloudflare built the Codex — a living repository of engineering standards written through an RFC process and enforced by AI code review agents.

How the Codex Works

1. Domain experts write RFCs to codify best practices (e.g., “Do not use .unwrap() outside of tests and build.rs.”)
2. Approved RFCs generate Codex rules with a simple format: “If you need X, use Y” plus a link to the RFC.
3. AI agents enforce rules at every stage of the SDLC: design review, code review, deployment, and incident analysis.
4. Incidents surface gaps that become new RFCs, creating a continuous improvement flywheel.

Practical Impact

Before Code Orange, a dangerous merge request could escape review and cause a global outage. Now, the same violation is caught at the PR stage — the blast radius shrinks from millions of affected requests to a single developer getting immediate, actionable feedback.

Limitations & Caveats

While Code Orange is a massive step forward, it’s not a silver bullet:

• Complexity: Snapstone and process segmentation add operational overhead. Smaller teams may find the tooling heavy.
• False positives: AI code review agents can flag safe patterns as violations, requiring manual review overhead.
• Human factors: Drills and muscle memory are only as good as their frequency. Cloudflare’s April 2026 drill involved 200+ engineers, but sustaining that practice across all teams is a challenge.

Next Steps

If you’re responsible for reliability at your organization, consider:

• Audit your configuration deployment pipeline: Are you deploying configs instantly or progressively?
• Implement health-mediated rollouts: Even a simple canary with manual health checks is better than no staging.
• Codify your incident learnings: Start a lightweight “Codex” for your team — a living document of rules enforced by linters or CI gates.

Cloudflare’s work is a powerful case study in how engineering culture, tooling, and process can converge to prevent catastrophic failures. The job is never done, but the direction is clear: fail small, learn fast, and encode your knowledge so it never has to be rediscovered through an outage.