Random Access Parquet (RAP) Turning Your Data Lake into a Low-Latency Serving Layer

From Batch to Interactive: The Data Lake Revolution

The data lake has traditionally been the home of batch analytics—running massive SQL queries over petabytes of data for reports and ML pipelines. But with the rise of AI agents and real-time personalization, there's a growing need to access individual records (e.g., a user's listening history) with latency measured in milliseconds, not minutes.

Spotify's engineering team faced this exact challenge. Their exabytes of data in GCS (Google Cloud Storage) were perfect for batch processing, but too slow for online point queries. The bottleneck wasn't the storage layer—GCS can deliver a single request in 30-100ms, and S3 Express One Zone even faster. The real problem was the query engines: distributed SQL engines like Trino or BigQuery add seconds of overhead for job scheduling and planning, even for a simple lookup.

In this deep dive, we'll explore Random Access Parquet (RAP), a technique developed by Spotify to bridge this gap. RAP enables interactive point queries directly on your existing Parquet files, without copying data to a separate key-value store. We'll cover the core concept, the external index, and a set of file-layout optimizations that can dramatically reduce latency.

The Core Idea: Replacing Dependent Reads with a Single Lookup

The fundamental issue with point queries in a data lake is the chain of dependent reads. To find a single row in a Parquet file, the engine must:

1. Fetch the file footer.
2. Parse row group metadata.
3. Scan the key column to locate matching rows.
4. Use column and page indexes to find the corresponding pages in each value column.

Each step requires a round-trip to storage, and each round-trip adds latency. The RAP approach eliminates this chain by using an external index that maps each key directly to the file and row number where its data resides. Given a key, the reader can issue precise ranged reads in parallel, fetching only the necessary bytes.

The External Index Structure

The index is a multimap, where a single key can have entries across multiple files and partitions. Each entry contains:

• Key: The lookup key (e.g., user ID).
• File: Which Parquet file (dictionary-encoded ordinal).
• Row numbers: The rows within that file.
• Value count (optional): For pagination.

This is fundamentally different from Parquet's built-in page indexes or Bloom filters, which are probabilistic and narrow a scan. The external index is definitive—it returns the exact files and rows, eliminating the scan entirely.

Code Example: A Simple RAP Index Lookup

While the full implementation is complex, here's a simplified Python example to illustrate the concept:

# Simple RAP index lookup (pseudo-code)

def get_user_data(user_id: str, rap_index: dict) -> list:
    """
    Retrieve user data using a RAP index.
    """
    # 1. Lookup the index (O(1) operation)
    if user_id not in rap_index:
        return []

    # 2. Get the file and row numbers
    entries = rap_index[user_id]
    results = []

    for entry in entries:
        file_path = entry['file']
        row_numbers = entry['row_numbers']

        # 3. Read only the specific rows from the Parquet file
        #    (using pyarrow or similar)
        data = read_parquet_rows(file_path, row_numbers)
        results.extend(data)

    return results

# Example index (in reality, this would be distributed)
rap_index = {
    'user_123': [
        {'file': 's3://data-lake/2026/07/01/user_events.parquet', 'row_numbers': [42, 43]},
        {'file': 's3://data-lake/2026/07/02/user_events.parquet', 'row_numbers': [10]}
    ]
}

# Query
user_data = get_user_data('user_123', rap_index)
print(user_data)

Note: This is a simplified illustration. Production implementations handle distributed indexes, caching, and parallel reads.

Optimizing File Layout for Point Queries

The external index tells you where to read, but the file layout determines how much you read. To minimize latency and I/O, RAP applies several optimizations that concentrate a key's data and reduce the number of read operations.

Key Optimizations Summary

The Biggest Win: Reducing Read Operations

In a standard Parquet file, fetching a key's values across N columns requires N parallel reads. The most effective optimization is to reduce the number of reads to one. This can be achieved by:

• Storing data as a single blob or Variant column (e.g., JSON, Protobuf). This is natural for applications that consume the data as a document.
• Interleaving columns: Physically placing data from different columns adjacent for each key. This allows a single contiguous ranged read to fetch all columns at once, while remaining valid Parquet for standard readers.

The Future of Data Lakes: A Single Serving Layer

RAP has significant implications for data architecture. It allows the data lake to serve both analytical and interactive workloads, eliminating the need to maintain separate copies in specialized serving systems. This changes the economics of which data can be served online—historical data, long-tail entities, and low-traffic features all become viable for interactive access.

However, RAP is not a silver bullet. It requires building and maintaining an external index, and the file-layout optimizations may not be suitable for all workloads. For example, interleaving columns can hurt performance for single-column scans in batch analytics.

Limitations and Caveats:

• Index maintenance: The index must be built and updated as new data arrives, adding pipeline complexity.
• File layout tradeoffs: Optimizations like interleaving columns can negatively impact batch query performance.
• Data types: ZSTD frame resets require PLAIN encoding, which may be less compact for certain data types.

Next Steps for Learning:

1. Experiment with Parquet internals: Use tools like pyarrow.parquet to understand row groups, pages, and column indexes.
2. Explore secondary indexes: Learn how to implement hash tables and sorted indexes for multi-dimensional lookups.
3. Consider space-filling curves: Techniques like Z-ordering can complement secondary indexes for better data locality.

Together with related trends, such as the evolution of on-device AI, the ability to quickly access and reason over large datasets becomes even more critical. For more on this, see our analysis of Google's AI Edge on-device function calling.

Recommended Reading:

• The Engineering Challenges of Building Meta Ray-Ban Display

• Google AI Edge: On-Device Function Calling