DuckLake Redefines Data Lake Architecture by Replacing Complex Metadata Formats with Standard SQL and Plain Parquet

Data engineering has long been plagued by operational bloat. For years, organizations attempting to build scalable data lakes have had to navigate an intricate labyrinth of nested JSON files, Avro manifests, and fragmented metadata layers just to determine which core storage files are relevant to a query. This multi-tiered complexity has driven up compute costs, slowed down analytical execution times, and introduced significant friction into modern data stacks. Enter DuckLake, a novel open-source table format that fundamentally challenges the status quo by collapsing traditional lakehouse overhead into a single, elegant paradigm: metadata lives inside a standard relational database, while the data itself remains in plain Apache Parquet files.

The project, which has rapidly gained traction within the developer and data science communities, represents a massive step forward in the quest for operational simplicity. By leveraging real databases for transactional cataloging and leaving raw data files untouched in object storage like AWS S3 or Azure Blob Storage, DuckLake bridges the gap between the flexibility of data lakes and the ACID-compliant reliability of traditional data warehouses.

Origins and Development: From CWI Research to Production-Ready Architecture

The conceptual roots of DuckLake trace back to the evolving philosophy of DuckDB, the ultra-fast, in-process analytical database management system. Pedro Holanda, the lead DuckLake developer, joined the DuckDB ecosystem in 2018 when the project was still an emerging research prototype at CWI (Centrum Wiskunde & Informatica), the national research institute for mathematics and computer science in the Netherlands. Working alongside co-creators Hannes Mühleisen and Mark Raasveldt, Holanda initially focused on foundational performance challenges, including optimizing CSV and analytical file processing.

As DuckDB expanded from a niche academic project into a widely adopted analytical engine utilized by data scientists and software engineers worldwide, the engineering team identified a gaping architectural void in how companies managed large-scale data lakes. Traditional open table formats like Apache Iceberg and Delta Lake were designed primarily for batch processing environments, growing outward from complex, file-based metadata structures. While effective for massive, single-pass batch operations, these formats struggled under heavy transactional concurrency and streaming write workloads.

Guillermo Sanchez Dionis joined the DuckLake initiative to help refine the project’s product strategy and core protocol development, contributing significantly to both DuckLake and the newly introduced Quack client-server protocol. Together, Holanda, Dionis, and the broader DuckDB development team spent months stabilizing DuckLake, leading up to its 1.0 production release and laying the groundwork for upcoming iterations.

The Architectural Shift: One SQL Query Replaces Nested Metadata

To understand the innovation behind DuckLake, one must examine the inefficiencies of legacy open table formats. In ecosystems like Apache Iceberg, queries require an engine to traverse multiple layers of JSON metadata, manifest lists, and individual manifest files before a single byte of actual Parquet data can be scanned. This multi-step traversal introduces latency, particularly when dealing with frequent, small-scale insertions—a common bottleneck known in the industry as the "small files problem."

DuckLake eliminates this overhead entirely. Instead of maintaining a sprawling file tree of metadata, DuckLake stores catalog information in a robust relational database. Depending on the user’s requirements, this catalog can be hosted on PostgreSQL, SQLite, an in-process DuckDB instance, or the high-performance Quack server protocol.

When a query is executed against a DuckLake table, the engine issues a single, highly optimized SQL question to the catalog database. The catalog instantly returns a precise list of relevant Parquet files alongside necessary column statistics. The compute engine then reads only the exact data files required, bypassing unnecessary round-trips to object storage.

Furthermore, DuckLake incorporates an innovative feature known as data inlining. For small, high-frequency write operations—such as streaming insertions or minor transactional updates—data is temporarily buffered directly within the relational catalog’s internal tables. This prevents the generation of thousands of fragmented, sub-optimal Parquet files in object storage. Once accumulated volumes cross a specific threshold, the system transparently flushes the inlined data into properly sized Parquet files (typically targeting 512 megabytes), maintaining peak query performance without manual administrative intervention.

Unprecedented Transactional Scale with the Quack Protocol

One of the most remarkable breakthroughs associated with the DuckLake ecosystem is its performance under heavy transactional contention. Historically, open table formats have suffered from poor concurrency when handling multiple simultaneous write operations. High-contention workloads on platforms utilizing Iceberg or Delta Lake often degrade to a handful of transactions per second due to snapshot conflict resolution overheads and frequent client-server round-trips.

DuckLake, when paired with the new Quack protocol, shatters these traditional limitations. Quack serves as a modern, lightweight client-server communication layer built on top of standard HTTP, designed specifically to integrate seamlessly with DuckDB and DuckLake. By handling conflict resolution and transaction retries natively on the server side rather than forcing round-trip failures back to the client application, DuckLake achieves an astonishing 200 transactions per second under heavy concurrency.

Benchmark evaluations indicate that no other open-source table format currently approaches this level of transactional throughput. This capability transforms data lakes from rigid, read-heavy analytical repositories into nimble environments capable of supporting real-time operational applications and streaming data pipelines.

Adoption, Production Readiness, and the Path Forward

With the stabilization of DuckLake 1.0 and ongoing feature refinements leading toward subsequent releases, the format has officially moved past its experimental phase. Enterprise adopters and software providers—including platforms like Firebolt and MotherDuck—have increasingly integrated DuckLake into their core architectures, utilizing its streamlined design to lower cloud infrastructure costs and eliminate vendor lock-in.

Unlike proprietary data warehousing solutions that trap user data inside closed ecosystems, DuckLake remains fundamentally open. The underlying data is stored as standard Parquet, ensuring that organizations retain total ownership and flexibility. If an engineering team wishes to migrate from DuckDB to Apache Spark, DataFusion, or another compatible compute engine, the transition requires minimal friction.

As data volumes continue to explode across global enterprises, the demand for architectural simplicity has never been higher. By trimming away the excess metadata bloat of previous-generation lakehouses and anchoring table management in standard relational databases, DuckLake offers a compelling blueprint for the future of scalable, high-performance data engineering.