TL;DR: Data is the ledger of how your business runs. When the format that ledger lives in belongs to one vendor, and there’s no clear path off it, you haven’t bought a data platform you’ve quietly handed over part ownership of your business. Open table formats exist to prevent exactly that, by separating your data from any one vendor’s compute engine. This piece traces how that movement got here, why the industry’s current “convergence” story deserves skepticism, and why choice — not any one engine, SQL or AI — must be the design principle enterprises insist on.
In 2005, the Commonwealth of Massachusetts did something that made Microsoft very unhappy: it decided that public records (e.g., birth certificates, court filings, the state’s own institutional memory) could no longer live exclusively inside a single vendor’s proprietary file format.¹ The state’s argument wasn’t about price, or features, or which word processor had a prettier user experience. It was an argument based on longevity and control. A public record has to be readable in twenty years, by whoever needs to read it, regardless of whether the vendor that created the format still exists, still supports it, or still likes you.
Microsoft’s response tells you everything about how format fights actually get resolved. It didn’t win the argument on the merits; it got its own format standardized too, submitting what became Office Open XML (OOXML) to Ecma International that same November, a year later approved as ECMA-376 and ratified as ISO/IEC 29500 in 2008.² The OpenDocument Format it was racing against was already approved as ISO/IEC 26300 in 2006.³ Two formats, two standards bodies, one underlying admission from both sides: whoever controls the format controls whether you can leave. The European Union already told Microsoft as much in 2004, warning that the “winner” of the format question would be whichever format became a genuine international standard, not one that shipped with the most market share.⁴ Ultimately, the battle centered on own an institution’s, organization’s or business’ own records.
We’re having a historical rhyming moment today about enterprise data, but this time, the thing reading the record isn’t a person opening a document. It’s an AI agent or SQL Compute engine, and the record might contain your customer list, your pricing model, or your regulated patient data. That changes what’s at stake, and it’s why the Open Table Format movement deserves to be read as more than a storage-engineering footnote. It’s the data layer’s answer to the same sovereignty question Massachusetts asked twenty years ago.
I’m going to use Open Table Format (OTF) throughout this piece with a specific, operational meaning, so let’s define it once. An OTF is a metadata specification — not a database, not a file format on its own — that sits on top of ordinary files in object storage and tells any compute engine, unambiguously, which files constitute a table’s current correct state, how that state changed over time, and how to change it safely while someone else is reading it. Apache Iceberg, Delta Lake, and Apache Hudi are the three that matter. What they all share is the property that makes this piece worth writing: the data and the engine that queries it are no longer the same purchase decision.
I’ll also use data sovereignty with a specific meaning: an enterprise’s retained ability to keep its data in a jurisdiction, format, and access pattern of its own choosing, and to point any compute engine — including one it didn’t have last year — at that data without first re-platforming it into someone else’s proprietary custody. That’s the thread connecting the 2005 document fight to the 2026 data fight.
For most of the last two decades, a table wasn’t a portable thing it was a commitment. Data stored in a proprietary warehouse’s internal format, or tracked by a vendor-specific catalog could only be read by that vendor’s own engine. Moving to a different engine because it was faster, cheaper, or a better fit for a new workload meant a full export and reload from the old to the new, which is ultimately risky migration. In essence proprietary formats were lock-in, manifested.
In the Hadoop ecosystem the problem got acute. Hive defined a table as all the files sitting in a directory, with partitions as subdirectories and a central metastore tracking paths. Reasonable for the mid-2000s. It degraded badly at scale: listing files across thousands of partitions could take minutes before a query even started; there was no way to atomically update more than one partition at once; concurrent readers and writers had no isolation and could return inconsistent results; and a query filtering on a derived value — say, a month extracted from a timestamp — triggered a full table scan because the partitioning scheme had no way to express that relationship.
Circa 2017 Netflix hit every one of those limits in production. Engineers there (i.e., Ryan Blue and Daniel Weeks, who’d later found Tabular) set out to fix three specific problems: guarantee real atomic transactions, allow file-level rather than whole-partition changes, and reduce operational burdens of running large tables.⁵,⁶ The result was Apache Iceberg, and its existence is the direct, documented answer to a proprietary-adjacent problem: a table format so tightly coupled to one engine’s assumptions that fixing it required leaving the format behind entirely. Tight coupling of engine and format is pattern worth teasing apart: each generation of data infrastructure has offered convenience today in exchange for dependency pain that surfaces later usually at a trigger like a contract renewal. The open table format movement is the industry’s most serious attempt yet to break that cycle, by making the format itself, not any one vendor’s implementation of it, the thing the customer actually owns.
Concretely, an open table format changes four things:
Before there could be a table format battle, there had to be a shared file format worth fighting over. That’s Apache Parquet, and it wasn’t built to solve the table problem it solved a narrower, earlier one. Analytical queries against Hadoop were slow because the data underneath them was stored row by row. Twitter and Cloudera built Parquet jointly in 2013, drawing on the record-shredding techniques described in Google’s Dremel paper, and it became a top-level Apache project in April 2015.⁶ Parquet stores all the values for one column together, compresses them as a homogeneous block, and embeds schema and per-column statistics directly in the file — so an engine can skip whole columns and row groups it doesn’t need, without any external schema registry. That combination is why Parquet became the default storage format for data lakes for NAS systems, S3-compatible object storage AWS S3, Azure Data Lake Storage, and Google Cloud Storage. It is also why every subsequent lakehouse technology chose to build on it rather than replace it. It’s now natively readable by SQL Server, DuckDB, Spark, Flink, Trino, Presto, every major cloud analytical service, and Postgres — the last of these via pg_lake, which Snowflake open-sourced in November 2025 under Apache 2.0.⁹ What Parquet doesn’t do is answer the question a table format exists to answer: which set of files, right now, constitutes the current correct state of this table, and how do I change that state safely while someone else is reading it? Parquet files are immutable and on their own, uncoordinated. The metadata layer above them is what turns a directory of files into something that behaves like a table.
Each of the three table formats closed that gap independently, but were designed to solve different production problems. Let’s take a look at each one.
The clearest evidence the format battle had a winner isn’t a benchmark. It’s what the losing side’s strongest competitor did next. In June 2024, Databricks acquired Tabular, the commercial company founded by the creators of Apache Iceberg, in a deal reported at roughly two billion dollars — due to a bidding war that reportedly involved Snowflake and Confluent. The acquisition was announced during Snowflake’s own annual conference — yeah that’s not a coincidence.¹⁴,¹⁵ Why the large purchase price you might ask for a three-year-old company with ~40 employees and roughly a million dollars in ARR? Simply put, the acquirer bought at such a steep price to control the Iceberg roadmap.
The year before, Databricks had already responded to Iceberg’s momentum with UniForm, a compatibility layer generating Iceberg-readable metadata alongside Delta. UniForm is useful engineering and an architectural admission: if your proprietary format needs to impersonate the open one, the open one has already won the neutrality argument. Acquiring Tabular put the people who wrote the Iceberg spec, and who still shape its direction, onto Databricks’ payroll. Basically, this enables them significant influence over an open standard inside the Apache Software Foundation without controlling it outright. Ryan Blue framed the move publicly as working toward interoperability “so that you can take advantage of the amazing work of both communities and don’t need to worry about the underlying format.”¹⁶ Read plainly, a company spent two billion dollars to stop competing on format and start competing on the platform or everything built on top of it.
But wait there’s more… By June 2025, Databricks announced native Iceberg support through its own catalog’s REST Catalog API. By its 2026 Data and AI Summit, it went further with GA support for Iceberg v3 tables directly in its runtime. There’s also a joint roadmap targeting Delta 5.0 and Iceberg v4 converging on a single shared metadata structure by Q4 2026. Databricks CEO Ali Ghodsi put the destination in nine words: ”It doesn’t matter if it’s in Delta or Iceberg.”¹⁷ Which is of course quite an appealing sentence, and it’s also worth reading twice. The convergence is real at the metadata layer — Iceberg v3 and Delta 5.0 are, in fact, moving toward a shared on-disk structure. But the performance and platform advantage Databricks sells alongside that convergence (e.g., Photon’s query acceleration, Unity Catalog’s governance model, LTAP, etc.) still lives inside Databricks’ managed runtime, not inside the open Iceberg spec itself. So their pitch resolves to something more like: your table format is free to move; the platform that makes it fast and is coupled to our portfolio is not. That’s a softer, later-stage version of the same lock-in Iceberg was invented to escape in the first place. In other words it is a soft on-ramp back into one vendor’s platform, dressed as an interoperability win.
It’s worth naming what this rhymes with. In 2005, Microsoft didn’t win the document-format argument by being more open than OpenDocument, but it got its own format standardized in parallel, submitted through a body it had more influence over, and closed enough of the gap that the market never had to fully choose. Buying the standard-setters is a more direct version of the same move, and it has the benefit of not requiring the acquirer to be dishonest, or even wrong about the engineering. It requires customers to stop asking who benefits when “convergence” is announced. Let’s contrast that with what Snowflake actually did with pg_lake: open-sourced outright, Apache 2.0, no runtime string attached. With pg_lake, a customer can take that code and run it without ever touching a Snowflake account.⁷ That’s open in meaningfully different way, where the technology provider or vendor gave away the keys. While the other is positioning itself to still hold a spare set after the doors are declared “open.”
None of this makes Databricks a villain. Cornering a standard you don’t fully control is a rational, well-worn competitive play. Microsoft ran a version of it in 2005 and ultimately lost the procurement argument to a coalition that insisted on genuine neutrality and future of choice. The behavior is the thing worth being wary of, not the company running it, and the same wariness applies to any vendor — including us — the day incentives point the same way. Our own position, for what it’s worth, is that a customer should be able to point any SQL engine, and increasingly any AI engine, at their own governed tables, full stop, with no asterisk about which runtime makes it feature rich or fast. That’s not a slogan; it’s a design constraint. If a vendor’s roadmap only works at full capability inside their box, the box is the product, that’s lock-in manifest.
Here’s the part the format battle history doesn’t cover on its own, and the reason I wanted to revisit this section: everything above was written as an argument about (SQL) compute flexibility. Choose whichever SQL engine fits the workload, without re-platforming the data. That’s still true and valuable. But we’re now pointing something new at these tables like retrieval pipelines, agentic SQL generation, fine-tuning jobs. As a result the sovereignty stakes are different in kind, not just degree.
Consider what’s actually being asked of enterprise data teams right now: connect an AI agent to the customer table. Let a copilot query the claims database. Fine-tune a model on internal support tickets. Every one of those requests is a request to give some compute engine — increasingly, an AI-native one — direct access to data that may be subject to GDPR residency requirements, the EU AI Act’s provisions on training data governance, HIPAA, sector-specific financial regulation, or simple competitive-IP protection. If that data sits inside a proprietary warehouse format or platform, the honest answer to “can I use a different AI vendor’s model against this data next year” is the same answer enterprises gave in 2019 when asked “can I switch SQL engines”: not without a migration project. This is where the OTF architecture stops being a compute-flexibility story and becomes a sovereignty story. Because storage is decoupled from the engine, an enterprise can keep its data in one place (one jurisdiction, one governed catalog, one Iceberg table) and point whichever AI engine it trusts at that moment at the data, without duplicating it, exporting it, or handing custody to a vendor. The data stays put; the catalog stays the single record of truth, and whichever engine gets pointed at it is a decision the enterprise keeps making, not one it made once by accident. That’s not a hypothetical extension of the OTF argument; it’s the same “one copy, many engines” property from earlier in this piece, now applied to a category of engine that didn’t exist when Iceberg was designed.
Let’s review this article’s discourse. Hive’s directory-based tables broke down under real production load, and Netflix’s fix, Iceberg, succeeded because it refused engine lock-in. Delta Lake solved a genuine correctness problem but was born, by its founders’ admission, out of a user lock-in concern. Databricks’ two-billion-dollar acquisition of Tabular is best read not as a bet on Tabular’s revenue, but as the cost of controlling the winning standard. And for what its worth the document-format battle of the 2000s resolved the same way: not because one side proved technical superiority, but because institutions decided owning the format of their own records mattered more than any single vendor’s feature set. In every version of this story, the format the customer doesn’t own is the format that eventually gets used against them. AI just adds a new, higher-stakes reason to care: it’s no longer only about which engine reads your data. It’s about which vendor’s AI roadmap your data is quietly locked into.
If you’re evaluating a data platform, or an AI platform that touches your in 2026, three key points follow from this history:
None of this requires abandoning any given vendor’s engine or model. It requires refusing to let the data be the thing that gets locked to a vendor’s roadmap, regardless of which engine, SQL or AI, happens to be querying it this year. So the next time a vendor announces “convergence” between two formats, four questions are worth asking before the announcement gets filed under good news:
None of these questions require distrust of any specific vendor’s intentions. They require treating “open” as a claim to be verified against your own exit cost, not a claim to be taken at the press release’s word — which is exactly what Massachusetts’ procurement office should have asked in 2004, and largely did.
Massachusetts didn’t win its argument by building a better word processor. It won by insisting the record itself had to outlast any single vendor’s product decisions. That’s the position worth taking on data heading into the next phase of AI adoption: the table format is not a technical implementation detail to be decided by whichever platform team ships first. It’s the thing that determines, years from now, whether you’re choosing your AI engine or being chosen by it.
Let’s call it the architecture of choice, because that’s what it actually is: not a single choice made once, under pressure, at a renewal date. It is retaining the ability to keep choosing, continuously, because the format never took the choice away from you to begin with. Open table formats gave enterprises that back for SQL compute. The same architecture is what lets them keep it as compute increasingly means an AI agent instead of a query planner.
Data is the ledger of how your business actually runs. It contains who your customers are, what they paid, what broke, what worked, what your model learned and from what. Hand over unrestricted custody of that ledger to a vendor with no clear, tested path back out, and you haven’t bought a data platform. You’ve handed over part ownership of the business itself, on terms you didn’t get to negotiate later. Own your business. Own your data.