The Problem: Data Insights at Scale
At Spotify, with over 70,000 datasets and petabytes of data, no single data expert can know everything. The traditional approach—asking a colleague on Slack—doesn't scale. Simply feeding all schemas to an LLM fails: context windows are too small, and schemas don't capture semantics. A column with type INT64 might contain legacy test data or real user counts, and the model can't tell the difference.
The Solution: A Context Layer
Spotify built a data assistant (called Vedder) that sits between raw data and the LLM. It uses a ReAct loop to reason, generate SQL, and run queries, returning answers with sources. The key innovation is the cluster model: a domain-specific context pack curated by human experts.
Each cluster includes:
- Datasets: relevant tables with full schema, column profiling (e.g., common values like 'US', 'GB'), and partition info.
- Pairs: vetted question-SQL examples that teach the model the correct query patterns.
- Docs: business context, terminology, and gotchas.
The curation is owned by data scientists and analytics engineers who deeply understand the data. They decide what to include, and they approve each pair.
Why Human Curation Matters
Spotify tested an automated approach: using the full query history to generate question-SQL pairs. But when curators reviewed them, they accepted only 12.5%. The rest were ad-hoc exploration, debugging sessions, or queries that taught wrong patterns. Query history is noisy; the signal doesn't label itself.
So every example runs through an expert. This isn't about replacing humans—it's about amplifying their expertise in a scalable way.
The Cluster Model in Practice
Here's a simplified example of how a curated pair might look:
# Example of a curated question-SQL pair in a cluster
curated_pairs = [
{
"question": "How many active users in the US last week?",
"sql": """
SELECT COUNT(DISTINCT user_id) AS active_users
FROM user_activity
WHERE country = 'US'
AND activity_date >= DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY)
AND is_test_user = FALSE
"""
}
]
This pair teaches the LLM to filter out test users and use the correct date range—something a raw schema wouldn't convey.
Keeping Clusters Healthy
Data changes constantly. Clusters have a health score based on signals like:
- Schema drift: pairs referencing renamed columns degrade immediately.
- Coverage: how well the context answers real user questions.
- Reproducibility: can the generated SQL be rerun successfully?
Cluster owners see these scores on a dashboard and decide where to spend curation time.
Limitations and Caveats
- Curation is labor-intensive: experts must review each pair, and this doesn't scale without dedicated tools.
- Context staleness: even with health scores, there's a lag between data changes and curation updates.
- Not a universal solution: the architecture works well at Spotify because they have a strong data culture. For companies with messy data, the curation overhead might be prohibitive.
Next Steps for Learning
- Explore the ReAct paper for more on reasoning loops.
- Look into MCP servers for integrating AI assistants with IDEs.
- Study how Spotify's approach could be adapted to your own data stack.
Conclusion
Spotify's data assistant shows that the key to trustworthy AI data insights is not just a better model—it's a curated context layer built by human experts. By encoding domain knowledge into clusters, they've made it possible for non-SQL users to get reliable answers at scale.
If you're building a similar system, focus on: ownership, curation, and feedback loops. Let experts curate the context, monitor health, and continuously improve.
For more on scaling data infrastructure, check out our article on how Meta scaled FFmpeg for billions of videos. And for a look at how major tech companies handle AI governance in disconnected environments, see our piece on Microsoft's sovereign cloud.