United Airlines Overcomes Data Silos with AWS Glue Data Catalog Federation to Unify Analytics Platforms
United Airlines has successfully executed a major architectural pivot in its data engineering strategy, leveraging AWS Glue Data Catalog federation to eliminate the need for costly data duplication between its Databricks-managed lakehouse and Amazon Redshift analytics environments. By enabling seamless, real-time access to petabytes of customer interaction data stored in Amazon S3, the airline has streamlined its reporting infrastructure while achieving significant operational cost reductions.
The initiative, co-developed by United Airlines engineers Ankit Aggarwal and Raja Kalluri alongside AWS architects, addresses a common challenge in enterprise data management: the fragmentation of data across disparate platforms. For an organization as large as United, which processes billions of events daily, the ability to maintain a single source of truth without building and maintaining complex, fragile synchronization pipelines represents a substantial shift in operational efficiency.
The Problem of Data Proliferation
For years, United Airlines relied on a medallion architecture—progressing data from bronze to silver to gold layers—hosted on Amazon Simple Storage Service (Amazon S3). While this architecture provided a robust foundation for data storage, it created a visibility gap. The airline’s extensive user interaction data, which tracks customer engagement, feature adoption, and conversion behaviors across web and mobile platforms, was cataloged exclusively within Databricks Unity Catalog.
Because Amazon Redshift lacked native visibility into the Unity Catalog, the data team was forced to adopt a strategy of data movement. This meant extracting, transforming, and loading (ETL) data into Amazon Redshift Managed Storage (RMS). This legacy approach introduced several friction points: it required the development of custom sync pipelines that were prone to schema drift, necessitated redundant storage, and imposed a latency lag that hampered the ability of analysts to make decisions based on near-real-time information. Furthermore, the cost of maintaining this redundant infrastructure—specifically the dedicated ETL clusters and associated storage—was mounting, reaching approximately $30,000 per month in unnecessary expenses.
Chronology of the Federation Deployment
The transition to a federated model followed a phased rollout, prioritizing the most critical business domains to ensure stability before a wider enterprise implementation.
- Strategic Planning Phase: The team identified the "User Interaction" data layer as the primary candidate for federation. This domain, comprising several double-digit terabytes of data, provided the highest value for immediate analytics.
- Configuration and Testing: Engineers utilized the AWS Glue Data Catalog federation to bridge the gap between Databricks and Redshift. By configuring a resource link database within the default Glue catalog, they created a pointer-based architecture that allows Redshift to query data without moving it.
- Initial Deployment: The first 30 tables were successfully moved to the production federated environment. This milestone served as the proof-of-concept, confirming that query performance remained high and that the security governance models remained intact.
- Current Scaling Phase: With the initial success, the team is currently rolling out an additional 70 tables. The project roadmap includes the integration of several hundred additional tables across various business domains over the coming months.
Technical Mechanics: How Federation Works
The technical success of this deployment hinges on the interaction between three primary layers: the Databricks Unity Catalog, the AWS Glue Data Catalog, and Amazon Redshift Serverless.
At the core of the solution is the "resource link" pattern. When a user executes a SQL query in Amazon Redshift, the query engine does not search for the data within Redshift’s own storage. Instead, it contacts the AWS Glue Data Catalog. Because of the resource link, Glue resolves the metadata by reaching out to the Databricks Unity Catalog via the Iceberg REST API.
This process is governed by AWS Lake Formation, which handles credential vending. When the query is initiated, Lake Formation ensures that the IAM role associated with the Redshift namespace has the appropriate permissions to access the specific S3 objects. Once verified, the data is retrieved directly from the S3 bucket where it resides, effectively turning Amazon Redshift into a high-performance query engine that operates on external data as if it were local.
Strategic Implications and Cost Analysis
The shift from a replication-heavy model to a federated query model has provided United Airlines with measurable improvements across four key operational areas:
Operational Simplification: By consolidating data processing within the Databricks ecosystem and using Redshift solely as a query engine, the airline has eliminated the need for manual catalog synchronization. This removes the "drift" that often occurs when two separate platforms try to maintain identical data versions.
Disaster Recovery (DR) Resilience: Previously, DR protocols for the data platform were dependent on Redshift snapshots, which occurred on a three-hour cycle. This left a significant window for potential data loss. With the current architecture, the airline leverages Amazon S3 cross-region replication. Because the data now resides on S3 rather than being locked inside a specific Redshift cluster, a new Redshift Serverless workgroup can be spun up in a secondary region to query the replicated S3 data, resulting in a near-continuous recovery point objective.
Financial Efficiency: The most direct impact is the reduction of redundant infrastructure costs. By eliminating the need for ETL-specific clusters and the associated storage fees, United Airlines has projected a cost saving of roughly $360,000 annually. These funds can now be reallocated toward more innovative data initiatives rather than maintenance of data pipelines.
Democratized Access: Approximately 100 data analysts were granted immediate access to the user interaction datasets during the first phase of the rollout. This represents a significant increase in productivity, as these analysts no longer need to submit requests for data to be moved or replicated to the Redshift environment.
Security and Governance Frameworks
A primary concern for a major airline is the strict governance of customer data. United Airlines managed this by unifying identity governance through Azure Active Directory (AD).
The security architecture relies on a single-identity model. AD group memberships determine a user’s access levels in both the Databricks Unity Catalog and the AWS environment. When a user runs a query through Redshift, the system validates their credentials via the SAML-authenticated JDBC driver. Lake Formation then performs a secondary check to ensure the user has the necessary permissions granted at the table level. This dual-layered approach ensures that even as data becomes more accessible, it remains highly controlled and compliant with internal data privacy mandates.
Future Outlook
The success of United Airlines’ federation strategy highlights a growing trend among large-scale enterprises: the move away from monolithic data warehouses toward a "data lakehouse" philosophy. By treating data as a product that can be queried across platforms rather than a static asset that must be moved, companies can respond to market fluctuations with greater agility.
While this solution is currently optimized for Apache Iceberg tables and S3-based storage, the underlying principles of the AWS Glue Data Catalog federation suggest a future where cross-platform interoperability becomes the standard rather than the exception. For United Airlines, the immediate focus remains on completing the migration of the remaining hundreds of tables, ensuring that the entire enterprise analytics stack benefits from the agility and security provided by this unified, federated approach. The project stands as a benchmark for how legacy-heavy organizations can modernize their data stacks without engaging in the high-risk, time-consuming process of total platform migration.
