Iceberg Catalog Wars: Polaris vs Unity vs Glue 2026

Q: What does an Iceberg catalog actually do?

An Iceberg catalog is the service that makes tables usable rather than just present in storage. It does four jobs. First, discovery: it maps namespaces and table names to the current metadata file location in object storage, so an engine can find a table without knowing its physical path. Second, commit arbitration: when an engine writes, the catalog performs the atomic compare-and-swap of the table’s metadata pointer, which is where Iceberg’s ACID guarantee and its optimistic concurrency actually live — two writers racing are serialised by the catalog, not by the object store. Third, access control: the catalog is the natural enforcement point for who can read, write or administer which namespaces and tables, since every engine passes through it. Fourth, credential vending: modern catalogs hand engines short-lived, scoped storage credentials for exactly the table locations a request is authorised to touch, replacing long-lived bucket-wide keys. The files and Iceberg metadata in object storage are open and portable; these four functions are services, and every one of them is operated by whichever catalog you chose. That asymmetry — open files, proprietary control plane — is why catalog choice, not table format, is now the consequential lakehouse decision.

Q: What is the Iceberg REST catalog specification, and what does it not cover?

The Iceberg REST catalog specification is an open HTTP protocol, maintained in the Apache Iceberg project, that standardises how engines talk to catalogs: listing namespaces, resolving tables to their current metadata, creating and committing changes to tables, and obtaining vended storage credentials. Before it, every catalog needed a bespoke connector inside every engine, which throttled interoperability; with it, any engine implementing the REST client can work against any compliant catalog — Spark, Trino, Flink, DuckDB and Snowflake all speak it, and catalogs from Apache Polaris to AWS Glue’s Iceberg REST endpoint implement the server side. What it deliberately does not cover is everything above the wire protocol: there is no standard model for role-based access control, column masking, row filtering, lineage, audit, multi-catalog management, federation, or policy. Each vendor implements those in its own way, which means every grant, masking rule and governance workflow you build is written in one vendor’s dialect and cannot be exported to a competitor. The spec is the best thing that happened to lakehouse interoperability, and it is routinely oversold: REST support guarantees your engines can connect, not that your governance can move. Protocol interoperability and platform portability are different claims — the spec makes the first and is silent on the second.

Q: Apache Polaris vs Unity Catalog vs AWS Glue vs Gravitino — how do they differ?

They embody three different strategies. Apache Polaris — co-created by Snowflake and Dremio, donated to the ASF in 2024, and graduated to a top-level Apache project in February 2026 — bets on neutrality: a fully open catalog implementing the Iceberg REST spec plus the production layer the spec omits (RBAC, credential vending, multi-catalog management, federation), with no engine to sell. You self-host it or use Snowflake’s managed Open Catalog. Unity Catalog is Databricks’ bet that governance depth wins: the open-sourced core (June 2024) provides the interface, but the compelling features — fine-grained ACLs, lineage, masking, Lakehouse Federation — are deepest in the managed Databricks product, which is where it pulls you. AWS’s Glue Iceberg REST endpoint and S3 Tables represent the dissolve-into-the-cloud strategy: the catalog becomes another AWS service, governance is expressed in IAM and Lake Formation grants, integration is unmatched inside AWS and non-existent outside it. Apache Gravitino, also a top-level Apache project with its 1.2.0 release in March 2026, answers a different question: rather than being the one Iceberg catalog, it federates many metadata systems — Hive metastores, Iceberg catalogs, Kafka topics, model registries — behind one metadata and authorization layer, targeting organisations that already have catalog sprawl rather than those choosing their first.

Q: Why is the catalog the new lock-in if Iceberg is an open format?

Because openness at one layer relocates the competition to the layer above. Iceberg standardised the table: any engine can read the Parquet data files and Iceberg metadata sitting in your object store, so the classic warehouse lock-in — proprietary storage format — is gone. But a table is inert without its catalog: you cannot find it, write to it safely, or govern access to it except through the catalog service, and the catalog’s value-added layer — the RBAC model, masking rules, lineage graph, audit trail, federation configuration — is standardised by no specification. Every grant you write accrues to one vendor’s model. The switching cost is therefore not moving data (the files never move); it is re-expressing your entire governance surface in a different dialect and re-pointing every producer and consumer. For a modest estate of a few hundred tables and a few dozen principals that is realistically measured in engineer-months, and it scales with grant count, which grows monthly. This is textbook control-point economics: vendors converged enthusiastically on the open format and the open protocol precisely because the catalog above them is where differentiation — and pricing power — now lives. Recognising that early is what keeps the decision deliberate instead of accidental.

Q: Which Iceberg catalog should I choose in 2026?

Choose by gravity, not by feature matrix. If 90% of your compute is Databricks, take Unity Catalog’s managed governance — fighting your platform’s native catalog buys friction and little else. If you are all-in on AWS and your security team already thinks in IAM and Lake Formation, the Glue Iceberg REST endpoint with S3 Tables means one less system to operate, monitor and audit, at the price of governance expressed in AWS primitives that will not travel. The neutral options earn their operational cost when the premise is genuinely multi-engine — say Snowflake writing, Trino and DuckDB reading, Flink streaming in: one Apache Polaris instance (self-managed, or Snowflake’s managed Open Catalog if you do not want to run it) with credential vending beats per-engine catalogs the moment you have two engines. Apache Gravitino is the answer to a different question — not “which catalog” but “how do I stop having six” — federating existing metastores and catalogs behind one layer. Whatever you pick, two disciplines preserve the exit: engines connect through the REST protocol only, never a proprietary SDK path; and authorization intent lives as code in your own repository, compiled into the catalog’s grant model, so a future migration re-targets a compiler rather than reverse-engineering years of accumulated click-ops.

Q: Can I run two catalogs over the same Iceberg tables?

For writes, no — and this is one of the fastest ways to corrupt a lakehouse’s consistency story. Iceberg’s ACID guarantee depends on a single arbiter performing the atomic swap of each table’s metadata pointer. If the same table is registered in two catalogs and both accept commits, the catalogs race: each can swap its own notion of the current metadata without knowledge of the other, and you get split-brain — lost updates, readers seeing different table states depending on which catalog they resolved through, and no error anywhere while it happens. Every table needs exactly one catalog that owns its commit path. What is legitimate is read-side federation: one catalog designated as the commit arbiter, with other systems federating or mirroring its metadata for discovery — Polaris supports external catalog federation, and Gravitino’s entire design is presenting many underlying catalogs through one layer, without seizing their commit paths. The operational rule is simple: one lake, one commit arbiter per table; federate reads if organisational reality demands it; and treat any pipeline that registers an existing table into a second catalog with write access as an incident, because that is what it is about to become. Migrations should re-point atomically — register, verify, cut over, deregister — never run dual-write.

Q: What is credential vending and why does it matter for security?

Credential vending is the catalog issuing engines short-lived, narrowly scoped storage credentials at query time — a token good for reading exactly the S3, ADLS or GCS locations backing the tables the request is authorised to touch, and nothing else. It replaces the older pattern in which every engine held long-lived, bucket-wide storage keys, and the catalog’s access control was therefore decorative: any engine or notebook with the bucket key could bypass table-level grants entirely by reading the files directly. With vending, object-storage access is derived from catalog authorisation on every request, so the table-level RBAC actually binds. The security consequence cuts both ways. Positively, you get real least privilege on the data plane, credentials that expire in minutes rather than persisting in configuration files, and a single audit point through which every data access flows. But it also promotes the catalog into a critical security boundary: it holds the master capability to mint storage access, so its own authentication, RBAC rigor, availability and audit logging now gate the entire lake. A catalog with weak or afterthought RBAC plus credential vending is effectively an open proxy to your object store. Evaluate a catalog’s vending and RBAC implementation with the same severity you would apply to an identity provider, because that is the role it now plays.

Q: How expensive is migrating from one Iceberg catalog to another?

The data is the cheap part — with both catalogs speaking Iceberg, table content in object storage does not move at all, and re-registering tables’ metadata pointers in a new catalog is mechanical. The real cost is everything the REST specification does not standardise. Every namespace layout, every role, every grant, every masking and row-filter rule, every service-account integration and every audit or lineage dependency was expressed in the old catalog’s governance dialect and must be re-expressed in the new one’s — which rarely maps one-to-one, so a human decides case by case what each grant should become. Then every producer and consumer — Spark jobs, Trino clusters, streaming writers, BI connections, notebooks — must be re-pointed and re-authenticated, ideally with a verification window proving reads and writes agree before cutover. For a modest estate of 200 tables and 40 principals, two to three engineer-months — roughly $60,000–$90,000 of engineering time — is a realistic floor, and cost scales with grant count rather than data volume, which is why it only grows. Two disciplines shrink it dramatically: grants-as-code from day one, so migration means re-targeting a compiler at a new backend; and REST-only engine connections, so re-pointing is a configuration change rather than a connector rewrite. Without those, "we will migrate later" is a decision to never migrate.

Satyam Kumar

Zurück zum Blog

ai-architecture

Iceberg Won. Now the Fight Is Over the Catalog

By Satyam KumarAugust 11, 202610 min read

apache iceberg iceberg catalog apache polaris unity catalog aws glue apache gravitino rest catalog lakehouse data catalog credential vending table format s3 tables data governance open table formats snowflake open catalog 2026

Iceberg Won. Now the Fight Is Over the Catalog

Frequently Asked Questions

Artikel teilen

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

KI & Cloud Architekt. Ich helfe Teams, Systeme zu bauen, die auf Millionen skalieren.

LinkedIn GitHub

Iceberg Won. Now the Fight Is Over the Catalog

Frequently Asked Questions

Artikel teilen

Comments

Leave a comment