How Spotify Automated 1,800 Dataset Migrations with Background Coding Agents (Honk Part 4)

The Pain of Dataset Migrations at Scale

If you’ve ever been responsible for deprecating a widely-used dataset, you know the feeling: a long tail of downstream consumers, manual pull requests, and endless coordination meetings. At Spotify, two heavily-used user datasets needed to be deprecated to make way for new versions with richer dimensions. The catch? ~1,800 direct downstream data pipelines, spanning three different pipeline frameworks: SQL-based BigQuery Runner, dbt, and Scala-based Scio. The manual effort was estimated at 10 engineering weeks.

Facing that prospect, the team turned to their internal background coding agent, Honk, combined with Backstage and Fleet Management tools. This is the story of how they pulled it off — and what they learned for the future.

Source: Spotify Engineering Blog

Step 1: Finding the Target Repositories with Backstage

Before any code change, you need to know what to change. Spotify used Backstage’s endpoint lineage and Codesearch plugins to map every downstream consumer of the deprecated datasets. Each endpoint’s Backstage page provided a clear list of repositories that needed migration. With Codesearch, they wrote queries to find target repositories across their GitHub Enterprise landscape, then orchestrated everything using the Fleetshift plugin.

Step 2: Context Engineering — The Hard Part

As discussed in Part 2 of the series, context engineering is the most critical — and hardest — part of working with background coding agents. Honk had to handle three different pipeline frameworks:

What Didn’t Work

• Using a human-written migration guide as context: Honk made incorrect assumptions about field mappings.
• Trying to cover Scio comprehensively: Without access to Claude skills or MCPs, the prompt became unwieldy. The team decided to pause Scio migrations and focus on the other two frameworks.

What Worked

• Writing explicit mapping tables in the context file — no assumptions.
• Specifying where NOT to migrate: For fields requiring human judgment, Honk left them unchanged but added comments with links to migration guides.

Pro tip: If your codebase lacks unit tests, Honk’s ability to verify its work is limited. The BigQuery Runner and dbt repos at Spotify rarely had build-time tests, so the team had to rely on downstream teams for manual verification.

Step 3: Rollout and Results

Using Fleetshift, Spotify rolled out 240 automated migration PRs. The Fleetshift UI provided a dashboard to monitor progress, click through PRs, and communicate with owning teams — invaluable for troubleshooting.

Result: 10 engineering weeks saved. Thousands of downstream pipelines migrated without manual toil.

Limitations and Lessons Learned

1. Standardization is Key

The success of large-scale agentic migrations depends on consolidating and standardizing your data landscape. Inconsistent frameworks (like Scio) are much harder to automate.

2. Testing Infrastructure Matters

Without unit tests, agents can’t verify their own work. Enforcing testing requirements across repositories is a prerequisite for autonomous coding.

3. Context Engineering is Still an Art

Even with good context, the agent will make mistakes if the mapping is not explicit. Use tables, not prose.

4. Don’t Forget the Human-in-the-Loop

Some decisions require human judgment. Leave clear comments and links for the reviewers.

What’s Next for Honk?

The Honk team is working on a feature that will allow the agent to gather its own context — reading JIRA tickets, documentation, and schemas before making changes. This will reduce the need for upfront context engineering and improve code change quality.

Combined with broader strategic pushes toward standardization and testing, Spotify expects Honk to tackle even more complex migrations autonomously.

Next Steps for Your Team

• Audit your data landscape: Identify frameworks that are inconsistently used.
• Standardize where possible: The more uniform your codebase, the easier automation becomes.
• Invest in testing: Agents that can run tests and iterate are far more reliable.
• Start small: Try automating a single migration with a well-defined context file before scaling.

If you’re interested in the underlying Python ecosystem that powers such workflows, check out our analysis of Python 3.14.3 new features. And for building visual, code-first AI pipelines, see our introduction to Daggr.

Conclusion

Spotify’s experience with Honk, Backstage, and Fleet Management proves that background coding agents can dramatically reduce manual toil — but only when the infrastructure supports them. Standardization, testing, and careful context engineering are the foundations of success.

The 10 engineering weeks saved on this single migration is just the beginning. As Honk gains the ability to gather its own context and as the underlying Claude Code agents improve, the potential for autonomous software maintenance grows exponentially.

Key takeaway: Don’t expect magic from agents alone. Invest in your codebase’s consistency and testability first, and the agents will follow.