Context: Why Another Workflow Engine?

Multi-step business processes — especially those spanning services, waiting on human input, or mixing sync/async steps — need persistent state. When a server restarts, the process must know what happened, what to retry, and what's still pending. Building this per-process means re-solving partial failure, timers, and recovery over and over.

Most existing engines solve this with a central orchestration service (self-hosted or managed). That gives a unified platform, but adds a shared dependency. Airbnb wanted a different model: each service owns its workflows inside its existing deployment and database. That requirement shaped Skipper, now open source under Apache 2.0.

근거자료: Airbnb Engineering — Skipper is now open source

Embedded workflow engine running inside Java service replicas with shared database

Architecture: Runtime Inside Your Replicas

With Skipper, your service replicas also run the workflow runtime and scheduler. They share a database that stores workflow state and scheduled work. The scheduler polls for runnable tasks and uses leases to coordinate ownership — if a replica dies, another recovers the work.

This removes a separately deployed orchestration tier. The tradeoff: your team now owns workflow execution, database capacity, and recovery.

A Workflow Is Just a Class

Skipper has two core abstractions:

  • Workflow — orchestration (decisions, ordering, waits)
  • Action — side-effecting work (API calls, DB writes); results are checkpointed for replay
class PublishListing : Workflow() {
    private val listing = actions()
    @StateField var reviewApproved: Boolean? = null

    @WorkflowMethod
    suspend fun publish(request: PublicationRequest): Boolean {
        if (request.requiresReview) {
            // Waits without holding a thread; survives redeploys
            val reviewed = waitUntil(
                { reviewApproved != null },
                Duration.ofDays(1)
            )
            if (!reviewed || reviewApproved != true) return false
        }
        listing.publish(request.listingId, request.requestId)
        listing.notifyHost(request.listingId, request.requestId)
        return true
    }

    @SignalMethod
    fun recordReview(approved: Boolean) {
        reviewApproved = approved
    }
}

waitUntil hibernates the workflow; recordReview is a signal from an API handler or queue consumer. Each action must be idempotent (e.g., dedupe by stable request ID) — Skipper does not make downstream calls transactional.

Recovery via Checkpoints and Replay

On resume, Skipper re-executes the workflow method from the top. Actions with persisted checkpoints return saved results. This requires deterministic workflow code: clock reads, randomness, and I/O belong in actions, not the workflow body. By default, checkpoints are batched until completion or a wait/retry — reducing DB writes but widening the replay window on crash. Actions can request immediate checkpointing to narrow it.

Skipper architecture diagram showing workflow runtime and scheduler in service process Dev Environment Setup

Where Skipper Fits — and Where It Doesn't

DimensionCentral Orchestrator (Temporal, Cadence, Step Functions)Skipper (Embedded)
DeploymentSeparate cluster/serviceInside your service replicas
Ops ownershipPlatform teamService team
Latency per stepRound-trip to orchestratorIn-process (DB + downstream only)
Multi-team platformStrongWeaker (per-service)
Language supportPolyglotJVM (Java/Kotlin)
StorageManagedMySQL, SQLite (Postgres/DynamoDB planned)

Good fit: Java/Kotlin services with robust ops that own a multi-step process and want durable execution without a new cluster. Especially when the alternative is hand-rolled retry jobs, state tables, and polling loops.

Poor fit: Polyglot orgs wanting a single shared platform, or teams without DB capacity planning discipline.

Caveats

  • At-least-once, not exactly-once. Side effects and checkpoints aren't atomic — design idempotency carefully.
  • Determinism is a hard requirement. Any nondeterminism in workflow code breaks replay.
  • You own the DB. Capacity, migrations, and recovery timing (lease settings) are on you.
  • JVM-only today. No Python/Go/Node support.
  • Ecosystem maturity. Admin UI and tooling exist, but replay debugging and evolution tooling are still evolving.

What's in the Release

Core engine, MySQL & SQLite adapters, test utilities (WorkflowTest with controllable time), admin UI, signals/queries/retries/compensation, and a Kotlin DSL called Skipper State Machine for state/event/transition-style processes.

Comparison of embedded orchestration vs central workflow cluster for durable execution Development Concept Image

Outlook & Next Steps

Airbnb plans to add storage plugins (Postgres, DynamoDB) and improve tooling for workflow evolution and replay debugging. Community contributions around compatibility checks and replay visualization are explicitly welcomed.

If you're evaluating Skipper:

  1. Audit your existing retry/state-table code — that's your migration surface.
  2. Prototype with WorkflowTest before touching production DBs.
  3. Enforce determinism with lint rules or code review checklists.
  4. Plan DB capacity and lease tuning alongside rollout.

Further reading:

This content was drafted using AI tools based on reliable sources, and has been reviewed by our editorial team before publication. It is not intended to replace professional advice.