Stop Copy-Pasting ETL Scripts Specification-Driven Composition on AWS

The Hidden Cost of Script-Based Pipelines

Every data team has been there. You start with a clean transform_orders.py. Six months later, you have transform_orders_v2.py, transform_orders_fixed.py, and transform_orders_final_USE_THIS_ONE.py. The transformation logic is duplicated across a dozen workflows, and a single business rule change cascades into a week of refactoring.

This is the scalability bottleneck that specification-driven composition addresses. Instead of embedding workflow intent inside imperative code, you declare what the pipeline should produce in a structured specification (JSON or YAML), and a composer assembles the how from reusable, versioned capabilities.

The pattern is particularly relevant in regulated environments — healthcare, finance, life sciences — where governance requires that business users author intent without touching execution code, and where audit trails must be explicit and versioned.

The core insight: workflow intent and processing logic are two different concerns, and conflating them is what kills you at scale.

This guide walks through the pattern and its serverless AWS implementation. For a deeper foundation on scaling Python workloads across clusters before you architect your pipeline layer, see our practical Ray on AWS tutorial.

The Three-Layer Architecture

Specification-driven composition organizes a workflow into three distinct layers:

1. Intent layer — Specifications define workflow behavior declaratively.
2. Composition layer — The composer validates specs and assembles pipelines.
3. Processing layer — Capability processors execute transformation steps.

The Specification

A specification is a declarative JSON/YAML document describing datasets, mappings, and transformations. It contains no processing logic — only intent.

{
  "source": ["raw_orders"],
  "target": ["clean_orders"],
  "mappings": [
    {
      "source_field": "order_date",
      "target_field": "order_date_iso",
      "capability": "format_date@1.2.0"
    },
    {
      "source_field": "amount",
      "target_field": "amount_usd",
      "capability": "normalize_currency@2.0.1"
    }
  ]
}

Note the pinned capability versions (@1.2.0). This is what makes runs reproducible — you never silently pick up a breaking change in a transformation.

The Composer

The composer is the brain. It does not perform transformations. It:

• Parses and schema-validates the specification
• Queries the capability registry (backed by Amazon OpenSearch Service) to resolve capability ARNs and metadata
• Compiles the spec into an Amazon States Language (ASL) definition
• Starts an AWS Step Functions state machine

# Composer Lambda — simplified skeleton
import json, boto3

sfn = boto3.client("stepfunctions")
opensearch = boto3.client("opensearchserverless")

def lambda_handler(event, context):
    spec = json.loads(event["spec_body"])
    validate_schema(spec)  # raise on invalid

    states = {}
    for i, mapping in enumerate(spec["mappings"]):
        cap = resolve_capability(mapping["capability"])  # OpenSearch lookup
        states[f"step_{i}"] = {
            "Type": "Task",
            "Resource": cap["arn"],
            "Next": f"step_{i+1}" if i + 1 < len(spec["mappings"]) else "Done"
        }
    states["Done"] = {"Type": "Succeed"}

    asl = {"StartAt": "step_0", "States": states}
    sfn.create_state_machine(name=spec["id"], definition=json.dumps(asl),
                             roleArn="arn:aws:iam::123456789012:role/sfn-exec")
    return {"status": "composed", "steps": len(spec["mappings"])}

The Capability Registry

Treat the registry as a governed artifact, not a lookup table. Capability definitions live in version control. Your CI/CD pipeline validates metadata and runs tests before publishing. Specification authors reference explicit versions.

The Capability Pipeline

Once assembled, Step Functions invokes each capability Lambda in sequence. Each processor receives only the fields its mapping references — a natural least-privilege boundary. Traces flow to CloudWatch Logs.

Where This Pattern Breaks (And Where It Shines)

⚠️ Limitations You Must Acknowledge

It is overkill for small pipelines. If you have fewer than three to five workflows, the composer, registry, and spec schema are pure overhead. A single well-tested script beats a distributed architecture every time.

The registry becomes a bottleneck if ungoverned. Without CI validation and version pinning, you trade duplicated scripts for a duplicated registry — same disease, different symptom.

Composer complexity is real. Writing a schema validator, a capability resolver, and an ASL compiler is non-trivial. Budget for it.

🔐 Security Notes

• Use SSE-KMS with a customer-managed key on spec and data buckets.
• Enforce HTTPS via bucket policy (aws:SecureTransport).
• Enable node-to-node encryption on OpenSearch.
• Tag sensitive fields in the spec ("sensitivity": "PHI") and derive target sensitivity from source tag + capability behavior. The composer should generate masking artifacts (e.g., Lake Formation column grants) automatically.

📊 How to Measure Success

Track three metrics:

If those numbers don't move, the pattern isn't paying for itself.

Getting Started This Week

Don't rewrite everything. Pick one existing pipeline that has three or more variants (monthly finance reports from different source systems are ideal). Then:

1. Describe it as a specification in JSON.
2. Implement 3–5 reusable capabilities as Lambda functions.
3. Wire S3 uploads to a composer Lambda.
4. Measure onboarding time for the next variant.

If the second variant takes days instead of weeks, you've validated the pattern.

Next Steps

• Event-driven wiring: The AWS Lambda event-driven architectures guide shows how to connect S3 uploads to your composer.
• Orchestration: The Step Functions + Lambda integration guide covers capability processor invocation.
• Observability: Publish custom CloudWatch metrics for composition success rate and step latency.

Related Reading

• React Server Components Security Alert — CVE-2025-55184 — if your data pipelines serve a web frontend, this is worth a look.

The real win isn't the architecture. It's that domain users can now author workflows in a language they understand, and engineers stop being a human compiler between business intent and execution.

Source: AWS Architecture Blog — Specification-driven composition for flexible data workflows