DuckLake Redefines Data Lakehouse Architecture by Combining Relational Metadata with Plain Parquet Storage

The modern data lakehouse landscape has long been defined by complexity. Traditional open table formats such as Apache Iceberg and Delta Lake rely on intricate webs of JSON, Avro, and metadata files just for an analytics query to determine which Parquet files are relevant to execute. This multi-layered metadata approach has historically introduced friction, high storage overhead, and severe latency penalties for iterative and transactional workloads. Enter DuckLake, an emerging open table format that proposes a radical architectural simplification: managing metadata inside a genuine relational database while keeping the underlying data entirely in plain Parquet files.

DuckLake is the product of continuous database research spearheaded by the team behind DuckDB, an embedded analytical database system that originated as a research prototype at CWI (Centrum Wiskunde & Informatica) in the Netherlands. As organizations increasingly seek alternatives to expensive, fully managed cloud data warehouses—such as Snowflake, Google BigQuery, and Databricks—DuckLake and its companion client-server protocol, Quack, aim to offer a lightweight, high-performance, and cost-effective alternative for modern data architecture.

The Architectural Blueprint: Simplifying the Lakehouse

To understand the appeal of DuckLake, one must examine the operational hurdles of conventional open table formats. In ecosystems like Apache Iceberg, data is stored in object storage alongside multi-tier metadata trees. Every time a query is initiated, the engine must perform multiple round trips to object storage to parse metadata manifests before reading a single row of actual data.

DuckLake drastically reduces this overhead. The entire table structure and file mappings are registered within a standard SQL database serving as the catalog. When an analytical query is executed against DuckLake, the engine asks a single, direct SQL question to the catalog to retrieve the precise list of Parquet files required. The metadata lives in a real database—such as PostgreSQL, SQLite, or DuckDB itself—while the heavy analytical data remains untouched in standard, unproprietary Parquet files.

This separation of concerns eliminates the need for specialized file-based metadata readers. Furthermore, because the catalog sits inside a standard relational database, DuckLake inherits standard ACID (Atomicity, Consistency, Isolation, Durability) transaction guarantees out of the box, avoiding the need to custom-build transactional machinery on top of raw object storage.

The Evolution of DuckDB and the Genesis of DuckLake

The roots of DuckLake trace back to the foundational development of DuckDB. Pedro Holanda, the lead developer of DuckLake, joined the DuckDB project in 2018 while completing his PhD at CWI in the Database Architectures Group. Alongside co-creators Hannes Mühleisen and Mark Raasveldt, Holanda helped engineer an analytical database tailored specifically for data scientists and software developers seeking the seamless in-process experience of SQLite combined with the blazing-fast vectorised execution of a columnar database.

Over the years, the DuckDB team observed a fundamental shift in user behavior. Data scientists and data engineers were actively trying to bypass traditional server-heavy databases due to deployment friction, high provisioning costs, and slow data transfer protocols between applications and database instances. DuckDB successfully solved this by embedding directly into the application memory space, allowing zero-copy data sharing with frameworks like Pandas, NumPy, and Apache Arrow.

However, as users began scaling DuckDB workflows to massive datasets housed in cloud object storage, the limitations of traditional file-based analytical formats became apparent. This realization led to the conceptualization of DuckLake: a format designed to bring the speed, simplicity, and transactional rigor of embedded databases directly to cloud storage layers. Guillermo Sanchez Dionis, who joined the DuckDB and MotherDuck engineering ecosystem after years of hands-on experience running enterprise-grade data platforms like BigQuery and Snowflake, brought a user-centric operational perspective to the project, contributing heavily to both DuckLake and the Quack network protocol.

Overcoming High Contention with the Quack Protocol

One of the most persistent bottlenecks in open table formats has been their poor performance under heavy transactional contention. Traditional formats are optimized for massive, infrequent batch writes rather than frequent, concurrent, small-scale updates. In multi-writer scenarios, high-contention environments typically cause frequent transaction conflicts, leading to excessive round trips, rollback cycles, and degraded throughput.

To address this, the engineering team developed Quack, a modern HTTP-based client-server protocol tailored specifically for DuckDB and DuckLake workflows. By shifting transaction handling and retry logic directly to the server side, Quack eliminates unnecessary network round trips between the client and the catalog.

Recent benchmarks under heavy concurrency highlight the efficacy of this approach. When utilizing PostgreSQL as a catalog backend under high transaction contention, DuckLake handles roughly five transactions per second due to client-side retry overhead. However, when paired with Quack and an in-server DuckDB catalog, DuckLake sustains up to 200 transactions per second under equivalent load. This level of concurrency under heavy contention represents a performance benchmark that few, if any, other open table formats can currently match.

Mitigating the Small File Problem via Data Inlining

A notorious challenge in cloud-based data lakes is the "small file problem." When streaming pipelines or frequent transactional updates insert small batches of records continuously, systems generate thousands of tiny data and metadata files. This proliferation of files severely degrades query planning and execution speeds due to high object storage request latency.

DuckLake addresses this architectural flaw through a feature known as data inlining. Instead of immediately writing a new Parquet file to object storage for every minor insertion, DuckLake can temporarily inline small batches of data directly into a structured table within the relational catalog.

These inlined records are tracked with snapshot identifiers and deletion flags. Once the accumulated data volume crosses a predetermined size threshold, DuckLake flushes the data out to a standardized, highly optimized Parquet file on object storage. This mechanism protects the data lake from performance degradation caused by streaming micro-writes, yielding performance improvements of up to three orders of magnitude in high-ingestion scenarios compared to traditional raw metadata architectures.

Production Readiness and Ecosystem Implications

With the release of DuckLake 1.0 and subsequent stability hardening, the project has transitioned firmly into production-grade territory. Core maintenance has focused heavily on robust checkpointing, automated file compaction, orphan file cleanup, and efficient deletion handling, ensuring long-term operational stability for growing datasets.

Early adopters and enterprise platforms—including developer-focused data services and analytical tooling providers—have increasingly adopted DuckLake as a foundational building block for their software stacks. By providing native support for alternative compute engines, such as experimental Apache Spark writers and DataFusion integrations, DuckLake ensures that organizations are not locked into a single proprietary execution engine.

Furthermore, the introduction of the "frozen DuckLake" pattern offers novel cost-saving opportunities. By maintaining a read-only DuckDB catalog file alongside Parquet data on low-cost object storage, organizations can execute batch updates periodically while allowing an unlimited number of concurrent client applications to query the data lake locally without running a dedicated, always-on database server.

Industry Outlook

As data engineering teams increasingly push back against the ballooning costs and complex operational overhead of legacy lakehouse platforms, simpler, standards-based alternatives are gaining momentum. By stripping away redundant metadata layers and anchoring table architecture in proven relational database principles, DuckLake demonstrates that high performance, ACID compliance, and extreme scalability do not require architectural bloat.

As DuckDB 2.0 approaches release—bringing advanced asynchronous I/O capabilities and further performance enhancements—DuckLake and the Quack protocol are well-positioned to influence the next generation of cloud-native data architectures, bridging the gap between localized analytical processing and distributed cloud storage.