Accelerating Apache Spark Query Execution with Apache Iceberg Materialized Views in AWS Glue Data Catalog

Data-driven organizations often grapple with the paradox of the "analytical bottleneck," where critical business intelligence dashboards and legacy reporting systems rely on complex, multi-table SQL queries that are both resource-intensive and prohibitively expensive to modify. As data volumes in modern data lakes expand into the petabyte scale, the latency associated with joins, heavy aggregations, and window functions has become a primary driver of rising infrastructure costs and delayed decision-making. Historically, the solution required manually rewriting these queries or constructing custom ETL (Extract, Transform, Load) pipelines—both of which introduce significant regression risks and engineering overhead.

However, a technical shift is underway with the integration of automatic query rewrite capabilities in Amazon EMR 7.12.0 and AWS Glue 5.1. This development allows for the seamless acceleration of existing SQL workloads by leveraging Apache Iceberg materialized views (MVs) without requiring any changes to the original source code.

The Evolution of Query Optimization

The challenge of optimizing large-scale analytical queries has long been a focal point for database engineers. Traditional methods, such as standard views or custom-built caches, often fell short. Standard views in AWS Glue typically require re-computation upon every execution, offering no performance gain for heavy processing. Custom ETL pipelines, while effective at storing results, require manual intervention and constant maintenance to ensure that the data remains synchronized with the base tables.

The introduction of automatic query rewrite in the AWS ecosystem represents a departure from these manual workflows. By utilizing a metadata cache, the Apache Spark optimizer can now intercept incoming queries, analyze their logical plans, and cross-reference them against available materialized views. If a match is detected—whether it is a structural match involving complex aggregations or an exact match for specific window functions—the optimizer transparently redirects the execution to the pre-computed MV. If no suitable MV exists, the query proceeds to the base tables as usual, ensuring that business logic remains consistent and correct.

Chronology and Technical Implementation

The rollout of this functionality began in mid-2024, signaling a commitment to deep integration between the AWS Glue Data Catalog and the Apache Iceberg open-table format. The process functions through an opt-in mechanism, requiring users to set the configuration spark.sql.optimizer.answerQueriesWithMVs.enabled=true during the creation of a Spark session.

Once enabled, the system operates on a sophisticated matching algorithm. When a query is submitted, the Spark optimizer decomposes the logical plan into subtrees. The system then evaluates these subtrees against the metadata of existing materialized views. This granular approach is critical; it means an MV does not need to perfectly replicate the entirety of a query to be useful. By substituting only the relevant portions of the plan, the engine can significantly reduce the computational load on the cluster, even for complex queries that might otherwise require scanning terabytes of raw data.

Comparative Analysis of Performance Tiers

To illustrate the efficacy of this approach, engineering teams have modeled performance across three distinct tiers using the TPC-DS 3 TB benchmark. The findings reveal a clear correlation between the specificity of the materialized view and the resulting performance gains.

At the "Tier 1" level, which involves a single-table pre-aggregate of sales data without joining or filtering, organizations can expect roughly a 5x increase in query speed. This tier offers the broadest utility, as it can be reused across any per-customer aggregation. Moving to "Tier 2," where the materialized view incorporates pre-joined tables and a baked-in filter, the performance gain improves to approximately 10x.

The most dramatic improvements appear at "Tier 3," which utilizes exact-match materialized views. By storing the precise output of a ranked analytical query, the system achieves a 20x or greater reduction in execution time. While the storage footprint for these MVs is negligible—often accounting for only a fraction of a percent of the original dataset—the trade-off lies in the specificity of the view, as Tier 3 MVs are generally limited to the exact query shape they were designed to serve.

Economic and Operational Implications

The introduction of automatic query rewrite has notable implications for cloud expenditure and operational efficiency. While materialized views do incur costs—specifically the compute time required to refresh the view and the storage costs for the resulting files—the ability to offload heavy processing from real-time query execution often leads to a net reduction in overall cloud spend.

By shifting the computational burden to a scheduled, background process in the AWS Glue Data Catalog, organizations can ensure that their BI dashboards remain responsive without triggering expensive, on-demand re-computations of massive fact tables. This shift is particularly beneficial for ISV applications and legacy reports where the original SQL is "frozen" and cannot be modified to account for new data structures or schema optimizations.

Industry Reception and Best Practices

Industry analysts note that this capability addresses a long-standing pain point for data engineers who previously faced a binary choice: either tolerate slow, expensive queries or embark on high-risk code refactoring projects. The consensus among database practitioners is that the key to success lies in identifying high-cost, recurring query patterns.

For organizations looking to adopt this technology, the recommended path forward is iterative. Data teams are encouraged to begin by monitoring their query logs to identify the most expensive, frequently executed tasks. Once a candidate is identified, creating an exact-match materialized view serves as an ideal proof-of-concept. Following validation—which can be confirmed by inspecting the Spark execution plan to ensure that the "BatchScan" is pointing to the MV rather than the base table—teams can then move to broader, multi-purpose materialized views that support a wider variety of query shapes.

Addressing Potential Limitations

Despite the clear advantages, the system is not without considerations. The overhead of the optimizer checking for MV matches can, in very specific low-latency scenarios, introduce minor delays. Furthermore, because the optimizer checks for data staleness, there is a risk that if an MV is not refreshed according to a regular schedule, it may be bypassed entirely in favor of the base tables to ensure data accuracy. This underscores the importance of a robust, automated refresh strategy integrated into the AWS Glue Data Catalog.

Conclusion

The move toward automatic, metadata-driven query rewriting signifies a broader trend in the data engineering space: the abstraction of performance optimization away from the developer and into the query engine itself. By leveraging Apache Iceberg’s capabilities, AWS has provided a mechanism that allows organizations to maximize the value of their existing data lakes while minimizing the complexity of maintaining legacy analytical systems. As data volumes continue to grow, the ability to transparently accelerate workloads without disrupting the underlying SQL will likely become a standard expectation for any enterprise-grade data platform. Through this integration, businesses gain the agility to scale their analytical capabilities, ensuring that performance remains consistent even as the complexity and scale of their data environments evolve.