Knowledge Base
Apache Iceberg Reference
A definitive, authoritative reference for every major Apache Iceberg concept — from the core table format and metadata layer to catalogs, query engines, operational patterns, and agentic data architectures. Each entry is written to be a standalone resource, deeply interlinked across the knowledge base.
222 terms across 14 categories
Core Concepts
- ACID Transactions in Apache Iceberg Apache Iceberg delivers full ACID transaction guarantees on object storage through optimistic concurrency control and at …
- Apache Iceberg Spec v1 vs v2 Apache Iceberg Spec v2 introduced row-level deletes (delete files), sequence numbers, required field tracking, and impro …
- Apache Iceberg Spec v3 Apache Iceberg Spec v3 introduces deletion vectors for more efficient row-level deletes, the Variant data type for semi- …
- Apache Iceberg Spec v4 (Current State) Apache Iceberg Spec v4 is in early community discussion and proposal stages as of 2025, with potential features includin …
- Apache Iceberg Table Format The Apache Iceberg table format is a specification defining how data files, metadata files, manifests, and snapshots are …
- Apache Iceberg vs Apache Hudi Apache Iceberg and Apache Hudi are both open table formats for cloud lakehouses — Iceberg prioritizes multi-engine inter …
- Apache Iceberg vs Delta Lake Apache Iceberg and Delta Lake are the two dominant open table formats for cloud lakehouses — Iceberg offers superior mul …
- Data Lakehouse A data lakehouse is a modern data architecture that combines the low-cost, scalable storage of a data lake with the reli …
- Hidden Partitioning in Apache Iceberg Hidden partitioning in Apache Iceberg separates the physical partition layout from the logical table schema, allowing th …
- Iceberg Column Mapping Iceberg column mapping decouples the logical column names in the schema from the physical field names in data files usin …
- Iceberg Deletion Vectors Deletion vectors are a Spec v3 enhancement to Apache Iceberg's row-level delete mechanism, replacing positional delete f …
- Iceberg Open Table Format vs. Delta Lake vs. Apache Hudi Apache Iceberg, Delta Lake, and Apache Hudi are the three dominant open table formats competing to be the storage founda …
- Iceberg Sequence Number The Iceberg sequence number is a monotonically increasing integer assigned to each snapshot and each data/delete file, i …
- Iceberg Snapshot References Iceberg snapshot references are named pointers (branches and tags) stored in the table metadata that reference specific …
- Iceberg Sort Order An Iceberg sort order is a table-level specification stored in metadata that defines how data should be physically order …
- Iceberg Table Properties Iceberg table properties are key-value configuration settings stored in the table metadata that control write behavior, …
- Iceberg Table Statistics (Puffin) Iceberg table statistics are advanced column-level metrics — including NDV (number of distinct values) estimates using A …
- Iceberg Views Apache Iceberg Views are named, stored SQL queries managed by the Iceberg catalog that appear as virtual tables to downs …
- Open Table Format Comparison (Iceberg, Delta Lake, Hudi, Paimon) A comprehensive comparison of the four major open table formats — Apache Iceberg, Delta Lake, Apache Hudi, and Apache Pa …
- Partition Evolution in Apache Iceberg Partition evolution in Apache Iceberg lets you change a table's partitioning scheme at any time without rewriting existi …
- Schema Evolution in Apache Iceberg Schema evolution in Apache Iceberg allows you to safely add, drop, rename, reorder, and widen columns in a table without …
- Time Travel in Apache Iceberg Time travel in Apache Iceberg lets you query a table as it existed at any past snapshot or timestamp, enabling reproduci …
- What is Apache Iceberg? Apache Iceberg is an open, high-performance table format for huge analytic datasets stored in data lakes, enabling ACID …
File & Metadata Layer
- Apache Iceberg ORC Format Apache ORC (Optimized Row Columnar) is an alternative columnar storage format supported by Apache Iceberg alongside Parq …
- Apache Iceberg Snapshot An Iceberg snapshot is an immutable, point-in-time view of a table's complete data state, recorded as a manifest list th …
- Apache Parquet and Iceberg Apache Parquet is the default and recommended data file format for Apache Iceberg tables, providing columnar storage, ri …
- Iceberg Avro Metadata Format Apache Avro is the metadata file format used for all Apache Iceberg manifest files and manifest lists, providing schema- …
- Iceberg Data Files Iceberg data files are the immutable columnar files (Parquet, ORC, or Avro) that store the actual table data in object s …
- Iceberg Delete Files Iceberg delete files record row-level deletions without rewriting data files, enabling efficient UPDATE, DELETE, and MER …
- Iceberg Encryption Apache Iceberg supports column-level and file-level encryption through its encryption specification, enabling sensitive …
- Iceberg Equality Deletes Equality delete files in Apache Iceberg record column values identifying rows to be deleted, enabling row-level deletes …
- Iceberg FileIO API The Iceberg FileIO API is an abstraction layer that decouples the Iceberg table format from specific storage system impl …
- Iceberg Manifest File An Iceberg manifest file is an Avro metadata file that tracks a subset of an Iceberg table's data files, recording each …
- Iceberg Manifest List An Iceberg manifest list is a file associated with each snapshot that lists all the manifest files making up that snapsh …
- Iceberg Metadata File The Iceberg metadata file (metadata.json) is the top-level entry point for an Iceberg table, recording the full history …
- Iceberg Positional Deletes Positional delete files in Apache Iceberg record the exact file path and row position of deleted rows, enabling efficien …
- Iceberg Puffin Files Puffin is the Apache Iceberg file format for storing advanced table statistics and indexes beyond the basic min/max boun …
Catalogs
- Apache Gravitino Apache Gravitino is an open-source multi-source metadata hub that provides unified metadata management across heterogene …
- Apache Polaris Catalog Apache Polaris is an open-source implementation of the Apache Iceberg REST Catalog specification, co-created by Dremio a …
- AWS Glue Catalog for Apache Iceberg AWS Glue Data Catalog is Amazon's managed metadata catalog service with native support for Apache Iceberg tables via the …
- Hive Metastore Catalog for Iceberg The Hive Metastore (HMS) is the original Iceberg catalog implementation, using a relational database to store Iceberg ta …
- Iceberg Catalog Migration Iceberg catalog migration moves tables between catalog implementations (HMS to Polaris, Glue to Nessie, JDBC to REST Cat …
- Iceberg JDBC Catalog The Iceberg JDBC Catalog uses any JDBC-compatible relational database (PostgreSQL, MySQL, SQLite) as a persistent metada …
- Iceberg Multi-Catalog Architecture Multi-catalog architectures in Apache Iceberg use multiple catalog instances to achieve environment isolation, domain se …
- Iceberg REST Catalog The Iceberg REST Catalog is a standardized HTTP API specification for Apache Iceberg catalog operations, enabling any en …
- Iceberg REST Catalog API Reference The Apache Iceberg REST Catalog specification defines a standardized HTTP API for catalog operations — namespace managem …
- Project Nessie Project Nessie is an open-source transactional metadata catalog for Apache Iceberg with Git-like branching and merging s …
- Snowflake Open Catalog Snowflake Open Catalog is a managed Apache Polaris service offered by Snowflake that provides a vendor-neutral Iceberg R …
- What is an Iceberg Catalog? An Apache Iceberg catalog is the service responsible for tracking the current metadata file location for each Iceberg ta …
Operations & Optimization
- Copy-on-Write (CoW) in Iceberg Copy-on-Write (CoW) is an Iceberg write mode where UPDATE and DELETE operations rewrite entire affected data files to pr …
- Expire Snapshots in Apache Iceberg Expiring snapshots in Apache Iceberg is the maintenance operation that removes old snapshot metadata (and optionally the …
- Iceberg Bloom Filters Bloom filter indexes in Apache Iceberg enable probabilistic row-level skipping by allowing query engines to determine wi …
- Iceberg Branching and Tagging Iceberg table branches and tags are named references to specific snapshots or independent snapshot chains, enabling Git- …
- Iceberg Concurrent Write Handling Apache Iceberg uses optimistic concurrency control with atomic catalog commits to safely handle multiple simultaneous wr …
- Iceberg Cost Optimization Cost optimization for Apache Iceberg lakehouses targets storage costs (snapshot expiration, compression, tiering), compu …
- Iceberg Data Skipping Data skipping in Apache Iceberg is the multi-level mechanism by which query engines eliminate irrelevant files and row g …
- Iceberg Incremental Reads Iceberg incremental reads enable processing only the new or changed data between two snapshots by using the snapshot dif …
- Iceberg Maintenance Scheduling Production Apache Iceberg maintenance requires scheduling compaction, snapshot expiration, orphan file cleanup, and mani …
- Iceberg Orphan Files Orphan files in Apache Iceberg are data files written to object storage during failed transactions that were never commi …
- Iceberg Performance Tuning Guide A comprehensive guide to optimizing Apache Iceberg query and write performance, covering partition pruning effectiveness …
- Iceberg Predicate Pushdown Predicate pushdown in Apache Iceberg propagates WHERE clause filter conditions from the query layer down through the man …
- Iceberg Rewrite Manifests Rewriting Iceberg manifests is a maintenance operation that consolidates many small manifest files into fewer, larger on …
- Iceberg Table Clustering Table clustering in Apache Iceberg co-locates related rows within the same data files to maximize column statistics sele …
- Iceberg Table Compaction Iceberg compaction is the maintenance process of merging small data files into optimally sized files, applying pending d …
- Iceberg Table Design Best Practices Iceberg table design best practices cover partition strategy, sort order selection, file format and compression choices, …
- Iceberg Table Rollback Rolling back an Apache Iceberg table reverts its current state to a prior snapshot, effectively undoing all writes since …
- Iceberg Upsert (MERGE INTO) Iceberg upsert operations using MERGE INTO enable atomic insert-or-update workflows against Iceberg tables, implementing …
- Iceberg Write Distribution Modes Iceberg write distribution modes control how data is distributed across parallel write tasks before being written to out …
- Merge-on-Read (MoR) in Iceberg Merge-on-Read (MoR) is an Iceberg write strategy where UPDATE and DELETE operations write small delete files instead of …
- Row-Level Deletes in Apache Iceberg Row-level deletes in Apache Iceberg enable precise removal or modification of individual rows within existing data files …
- Small File Problem in Apache Iceberg The small file problem in Apache Iceberg occurs when frequent write transactions generate many small Parquet files, degr …
- Z-Order Clustering in Apache Iceberg Z-Order (or Z-curve) clustering in Apache Iceberg is a multi-dimensional data layout optimization that co-locates rows w …
Engines & Integrations
- Apache Airflow and Apache Iceberg Apache Airflow is the most widely used workflow orchestration platform for Iceberg data pipelines, providing scheduling, …
- Apache Doris and Apache Iceberg Apache Doris is a high-performance real-time analytical database with native Iceberg external catalog support, enabling …
- Apache Flink and Apache Iceberg Apache Flink is the leading stream processing engine for Apache Iceberg, enabling real-time data ingestion with exactly- …
- Apache Kafka and Apache Iceberg Apache Kafka and Apache Iceberg form the backbone of real-time lakehouse pipelines — Kafka provides the event streaming …
- Apache Spark and Apache Iceberg Apache Spark is the most feature-complete query engine for Apache Iceberg, providing full DDL, DML, time travel, stored …
- Apache Superset and Apache Iceberg Apache Superset is the leading open-source business intelligence tool that queries Apache Iceberg tables through SQL con …
- Databricks and Apache Iceberg Databricks supports Apache Iceberg through UniForm (Delta-to-Iceberg automatic metadata generation) and native Iceberg c …
- dbt and Apache Iceberg dbt (data build tool) transforms raw Iceberg table data into clean, tested, documented analytical models using SQL, with …
- Dremio and Apache Iceberg Dremio is an Agentic Lakehouse platform that provides a fully integrated Iceberg experience through its Intelligent Quer …
- DuckDB and Apache Iceberg DuckDB is an embedded analytical database with a native Apache Iceberg extension that enables direct, high-performance S …
- Hive and Apache Iceberg Apache Hive 4.x has native Iceberg support, enabling Hive SQL to read and write Iceberg tables as first-class objects wh …
- Presto and Apache Iceberg PrestoDB is the Meta-maintained fork of the original Presto query engine with an Iceberg connector that supports Iceberg …
- PyIceberg: Python Library for Apache Iceberg PyIceberg is the official Python library for Apache Iceberg, providing a pure-Python client for reading, writing, and ma …
- Snowflake Iceberg Tables Snowflake Iceberg Tables let organizations store Iceberg data in their own object storage (external volumes) while using …
- StarRocks and Apache Iceberg StarRocks is a high-performance OLAP query engine with native Apache Iceberg external table support via its Multi-Catalo …
- Trino and Apache Iceberg Trino (formerly PrestoSQL) is a distributed SQL query engine with native Apache Iceberg support, optimized for interacti …
Agentic & AI
- Agentic Lakehouse An Agentic Lakehouse is a data lakehouse architecture purpose-built for AI agents and autonomous analytics, combining op …
- Iceberg AI Readiness Iceberg AI readiness describes the architectural properties that make Apache Iceberg tables ideal for AI and machine lea …
- Iceberg AI Semantic Layer The AI Semantic Layer on Apache Iceberg translates raw Iceberg table data into AI-understandable business context throug …
- Iceberg Apache Arrow Flight Apache Arrow Flight provides a high-throughput, low-latency RPC protocol for transferring Apache Arrow columnar data fro …
- Iceberg Feature Store Apache Iceberg is used as the offline storage layer in ML feature stores, providing point-in-time correct feature retrie …
- Iceberg LLM Grounding and RAG for Structured Data LLM grounding with Apache Iceberg uses governed, versioned Iceberg tables as the authoritative data source for LLM respo …
- Iceberg Natural Language Analytics Natural language analytics on Apache Iceberg enables business users and AI agents to ask questions in plain English and …
- LangChain and Apache Iceberg LangChain agents can query Apache Iceberg lakehouses using SQL tools and Arrow Flight connections, enabling natural lang …
- MCP and Apache Iceberg Model Context Protocol (MCP) servers for Apache Iceberg enable AI agents and LLMs to discover, query, and reason over Ic …
Cloud-Specific Integrations
- Amazon EMR and Apache Iceberg Amazon EMR (Elastic MapReduce) is AWS's managed Spark and Flink cluster service that supports Apache Iceberg as a first- …
- Amazon S3 Tables for Apache Iceberg Amazon S3 Tables is an AWS managed service that provides Apache Iceberg table storage and catalog directly within Amazon …
- AWS Athena and Apache Iceberg Amazon Athena is a serverless SQL query engine with native Apache Iceberg support via the AWS Glue Data Catalog, enablin …
- BigQuery and Apache Iceberg Google BigQuery supports Apache Iceberg tables through BigLake managed tables and Biglake Metastore, enabling BigQuery S …
- Google Cloud and Apache Iceberg Google Cloud's Apache Iceberg stack integrates BigQuery, Cloud Storage, Biglake Metastore, and Cloud Dataplex to provide …
- Microsoft Fabric and Apache Iceberg Microsoft Fabric supports Apache Iceberg tables through OneLake's open format integration and mirrored Fabric tables, en …
Modern Lakehouse Concepts & Interoperability
- Apache DataFusion A highly extensible Rust-native SQL query engine that supports native querying and management of Apache Iceberg tables.
- Apache XTable Translations An open-source translation layer that converts table metadata bidirectionally among Apache Iceberg, Delta Lake, and Apac …
- Bauplan A Python-native serverless lakehouse engine that uses Apache Iceberg as its core storage format to run pipelines with Gi …
- CDC Log Ingestion Pipelines Data pipelines that capture database transaction logs and apply those insert, update, and delete events to lakehouse tab …
- ClickHouse A fast, open-source columnar database management system that supports native reading, querying, and writing of Apache Ic …
- Columnar Memory Layouts A memory architecture that groups data values sequentially by columns rather than rows, enabling efficient vectorized qu …
- Debezium CDC Engines The standard open-source framework built on Apache Kafka Connect that monitors database transaction logs and translates …
- Decoupled Compute and Storage An architecture separating the compute resources used to execute queries from the storage systems where data files resid …
- Delta Lake UniForm Metadata A features in Delta Lake (Universal Format) that automatically generates Apache Iceberg metadata alongside Delta logs du …
- Dynamic Filter Pushdown A query optimization technique that uses runtime join results to dynamically filter scan operations on the opposing tabl …
- Google Cloud BigLake A storage virtualization and governance engine in Google Cloud that unifies data lakes and warehouses, supporting manage …
- Microsoft Fabric OneLake A unified SaaS data lake for Microsoft Fabric that supports Apache Iceberg through metadata shortcuts and table format v …
- Modern Data Stack (MDS) A cloud-native suite of analytical tools built around a central cloud data warehouse or lakehouse, emphasizing modularit …
- Object Storage Prefix Hashing A layout strategy that prepends random hash prefixes to object paths to distribute storage request workloads across diff …
- PuppyGraph A real-time, zero-ETL graph query engine that allows users to query relational data lakes, including Apache Iceberg, as …
- Read Amplification A storage performance metric representing the ratio of physical data read from storage compared to the logical data requ …
- Space Amplification A storage metric representing the ratio of physical disk space occupied by a table compared to the size of its active lo …
- Spice.ai A data acceleration platform that enables application and AI developers to query, federate, and write to Apache Iceberg …
- Split Planning Loops The execution phase where query engine coordinators analyze table partition statistics to group data files into parallel …
- Time To First Byte (TTFB) A latency metric measuring the duration from when a client engine initiates an object storage read request to when the s …
- VeloDB A commercial data warehouse built on Apache Doris that offers native integration with Apache Iceberg tables for fast que …
- Vortex File Format A high-performance columnar storage format succeeding Parquet, optimized for GPU-native decompression, in-memory Arrow c …
- Write Amplification A performance metric measuring the ratio of physical data bytes written to storage compared to the logical data bytes up …
- Zero-Copy Cloning A metadata-only operation that duplicates a table state instantly by pointing the new table clone to the existing data f …
Lakehouse Catalogs & Governance
- Catalog Federation The capability of a query engine to connect to multiple disjoint metadata catalogs simultaneously, presenting them as a …
- Catalog Namespaces Hierarchical logical containers (similar to databases, schemas, or folders) used within an Iceberg catalog to organize t …
- Data Lineage Tracking The practice of documenting and visualizing the lifecycle, transformations, and flow of data from its source to its fina …
- Glue Catalog IAM Policies AWS Identity and Access Management policies that regulate user and compute engine access to AWS Glue Data Catalog databa …
- Glue Catalog Lake Formation A security service layer built on the AWS Glue Data Catalog that enables fine-grained, column-level, row-level, and cell …
- Iceberg DynamoDB Catalog An AWS serverless catalog implementation that stores Iceberg table pointers in Amazon DynamoDB, using conditional writes …
- Iceberg Hive Catalog Lock Manager A coordination system that uses the Hive Metastore lock APIs to serialize concurrent table updates, ensuring commit safe …
- Iceberg JDBC Catalog Locks A database row-level locking mechanism used by SQL-backed Iceberg catalogs to coordinate concurrent commits and ensure t …
- Nessie Git-like Branching A version control model in Project Nessie that allows teams to manage database catalog tables using branches, tags, and …
- Nessie Merging The operation in Project Nessie that merges commits from an isolated staging branch back into the main production branch …
- Nessie Tagging A version control feature in Project Nessie that creates named, immutable reference pointers to freeze the state of the …
- Polaris Catalog Sharing A secure metadata sharing feature in Apache Polaris that allows external accounts and partner engines to access catalog …
- Polaris RBAC Model The Role-Based Access Control framework in Apache Polaris that secures catalog metadata resources by mapping service pri …
- Polaris Service Principals Programmatic identities created within Apache Polaris to authenticate query engines, ETL pipelines, and applications usi …
- REST Catalog Credential Vending A security architecture where the Iceberg REST catalog dynamically issues short-lived, scoped storage credentials to que …
- REST Catalog OAuth2 Token Flow The standardized authentication protocol used by Iceberg REST clients to exchange credentials for secure Bearer tokens t …
- Single Source of Truth (SSOT) A data design principle where a central repository serves as the definitive reference for the state and metadata definit …
- Snowflake External Table Catalog Sync The automated or manual synchronization process that refreshes Snowflake metadata pointers to match updates made to Iceb …
- Snowflake Managed Iceberg Tables Iceberg tables where Snowflake serves as the catalog manager, executing all DML writes and writing physical Parquet file …
- Unity Catalog Delta-Iceberg Compatibility The capabilities in Unity Catalog that allow Delta Lake tables to be read as Apache Iceberg tables by generating complia …
Dremio-Specific Engine & Optimizations
- Dremio Acceleration Engine The Dremio Acceleration Engine is the dedicated resource allocation mechanism within Dremio executors responsible for bu …
- Dremio Aggregation Reflections Dremio Aggregation Reflections are pre-computed data structures that store pre-aggregated metrics and measures grouped b …
- Dremio Arrow Flight SQL Dremio Arrow Flight SQL is a high-performance database connectivity protocol based on Apache Arrow and gRPC, transferrin …
- Dremio Column-Level Masking Dremio Column-Level Masking is a dynamic data security feature that obfuscates or replaces sensitive column values (such …
- Dremio Columnar Cloud Cache (C3) Dremio Columnar Cloud Cache (C3) is a block-level storage caching mechanism on executor nodes that stores copies of data …
- Dremio Coordinator Node The control plane node in a Dremio cluster responsible for query parsing, optimization, metadata catalog management, and …
- Dremio Data Reflections Matching Dremio Data Reflections Matching is the query planner compiler logic that automatically identifies, evaluates, and subst …
- Dremio Engine Auto-scaling The automated process of dynamically scaling Dremio execution engine pods in a Kubernetes cluster to optimize performanc …
- Dremio External Queries Dremio External Queries are SQL pass-through commands that allow users to execute native database queries directly on un …
- Dremio Iceberg Metadata Sync Dremio Iceberg Metadata Sync is the background coordination process that updates Dremio's catalog pointer to reference t …
- Dremio Join Co-segmentation Dremio Join Co-segmentation is a query planning optimization that aligns join keys with matching table partition structu …
- Dremio LDAP Integration The configuration of Dremio authentication to validate user credentials and map groups against LDAP or Active Directory …
- Dremio Metadata Caching Dremio Metadata Caching is the process of storing table metadata (such as schemas, partition statistics, and file lists) …
- Dremio Parquet Vectorized Reader Dremio Parquet Vectorized Reader is an engine component that reads columnar Parquet data directly into in-memory Apache …
- Dremio Physical Datasets (PDS) Dremio Physical Datasets (PDS) are physical tables, views, or raw file directories registered from external data sources …
- Dremio Raw Reflections Dremio Raw Reflections are pre-computed data layouts that preserve raw, row-level columns from a dataset, optimized with …
- Dremio Reflections Dremio Reflections are pre-computed query acceleration structures that automatically optimize and speed up SQL queries u …
- Dremio Row-Level Security (RLS) Dremio Row-Level Security (RLS) is an access control mechanism that dynamically filters table rows returned by a query b …
- Dremio Sabot Engine The Sabot Engine is Dremio's highly optimized, vectorized query execution engine built on Apache Arrow, designed to proc …
- Dremio Spaces Dremio Spaces are logical namespace containers in the Dremio catalog where data teams organize, collaborate on, and secu …
- Dremio SQL Runner Dremio SQL Runner is the integrated web-based query editor and SQL interface inside the Dremio console used for writing …
- Dremio User Defined Functions (UDFs) Dremio User Defined Functions (UDFs) are custom calculations created in SQL that allow data teams to encapsulate reusabl …
- Dremio Virtual Datasets (VDS) Dremio Virtual Datasets (VDS) are logical views defined in the semantic layer that allow data teams to clean, join, and …
Governance & Security
- Iceberg Access Control Patterns Iceberg access control is implemented at the catalog layer through the Iceberg REST Catalog RBAC model, providing namesp …
- Iceberg Audit Logging Iceberg audit logging captures a complete record of all catalog interactions, table reads, write commits, schema changes …
- Iceberg Data Lineage Iceberg data lineage is the ability to trace the origin, transformation history, and downstream consumption of data in I …
- Iceberg Data Masking Data masking in Apache Iceberg protects sensitive column values from unauthorized consumers by applying masking function …
- Iceberg Multi-Tenancy Patterns Multi-tenancy in Apache Iceberg isolates multiple tenants, teams, or environments in a shared lakehouse using namespace …
Table Format Maintenance & Operations
- Iceberg Bin-Packing Compaction A fast, non-sorting compaction strategy in Apache Iceberg that combines small data files into larger files to reduce rea …
- Iceberg Manifest Merging An automatic, write-path optimization in Apache Iceberg that merges small manifest files during commits to prevent metad …
- Iceberg Metadata Pruning An automated housekeeping process in Apache Iceberg that limits the accumulation of historical table metadata JSON files …
- Iceberg Orphan Files Penalty The performance and cost overhead incurred when unreferenced, abandoned files accumulate in object storage due to failed …
- Iceberg Partition-Level Compaction A targeted maintenance strategy that runs file compaction only on specific partitions, optimizing high-churn data subset …
- Iceberg Rollback Snapshot Procedures SQL procedures used to revert the active state of an Apache Iceberg table to a specific snapshot ID or historical timest …
- Iceberg Sort-Based Compaction A compaction strategy in Apache Iceberg that combines small data files and sorts the records globally by specified colum …
- Iceberg Spark Procedure add_files A Spark SQL procedure in Apache Iceberg used to register existing Parquet or ORC data files directly into an Iceberg tab …
- Iceberg Spark Procedure expire_snapshots A Spark SQL procedure in Apache Iceberg used to remove expired snapshots and physically delete their unreferenced data a …
- Iceberg Spark Procedure register_table A Spark SQL procedure in Apache Iceberg used to register an existing metadata JSON file as a table within the active cat …
- Iceberg Spark Procedure remove_orphan_files A Spark SQL procedure in Apache Iceberg used to identify and delete files in table storage that are not referenced in an …
- Iceberg Spark Procedure rewrite_data_files A Spark SQL metadata procedure in Apache Iceberg used to consolidate small files and apply sorting or clustering strateg …
- Iceberg Spark Procedure rewrite_manifests A Spark SQL maintenance procedure in Apache Iceberg used to reorganize and merge small manifest files to optimize query …
- Iceberg Spark Procedure rewrite_position_deletes A Spark SQL maintenance procedure in Apache Iceberg used to consolidate small position delete files to improve query rea …
- Iceberg Z-Order Compaction A multi-dimensional clustering compaction strategy in Apache Iceberg that sorts data along a Z-order space-filling curve …
Iceberg Specification, Schema & Internals
- Iceberg Bucket Partition Transform An Iceberg partition transform that hashes values of a source column into a user-specified number of buckets to distribu …
- Iceberg Column Projection The process of reading only the specific fields requested by a query from physical storage files using unique field IDs …
- Iceberg Date/Time Partition Transforms A set of native partition transforms in Apache Iceberg that partition data by year, month, day, or hour using source dat …
- Iceberg Decimal Type Widening A metadata-only schema evolution rule in Apache Iceberg that permits promoting the precision of a decimal field while ke …
- Iceberg File Content Type A field in the manifest entry schema that indicates the type of data stored in a file, distinguishing between data files …
- Iceberg File Path Spec The set of structural rules in Apache Iceberg that defines how metadata files, manifest files, and physical data files a …
- Iceberg Identity Partition Transform The default partition transform in Apache Iceberg that partitions data directly by the values of a source column without …
- Iceberg Lock Manager A catalog helper interface in Apache Iceberg used to manage transactional locks and secure pointer updates on catalogs t …
- Iceberg Manifest Entry Schema The Avro schema definition that specifies how Iceberg tracks data or delete files within manifest files, detailing colum …
- Iceberg Manifest Entry Status An integer status code in the Iceberg manifest entry schema that defines the state of a file in a snapshot: 0 (EXISTING) …
- Iceberg Manifest List Schema The Avro schema definition that tracks the set of manifest files comprising an Iceberg snapshot, including partition sum …
- Iceberg Metadata Table Files A virtual system table in Apache Iceberg that lists all active data and delete files, exposing their file paths, sizes, …
- Iceberg Metadata Table History A virtual system table in Apache Iceberg that exposes the historical timeline of committed snapshots, timestamps, and li …
- Iceberg Metadata Table Snapshots A virtual system table in Apache Iceberg that lists all active snapshots, exposing their creation times, parent IDs, ope …
- Iceberg Metrics Mode A configuration property in Apache Iceberg that controls the depth of column-level statistics stored in manifest files t …
- Iceberg Nested Type System The set of structural data types—structs, lists, and maps—defined by the Iceberg specification that use unique field IDs …
- Iceberg Optimistic Concurrency Control (OCC) A concurrency model in Apache Iceberg that assumes low conflict rates, allowing concurrent writers to prepare changes in …
- Iceberg Parent Snapshot ID A reference property stored in snapshot metadata that links a snapshot to its immediate ancestor, establishing the table …
- Iceberg Sequence Number Inheritance The mechanism in Apache Iceberg where data and delete files inherit their commit sequence numbers dynamically from the m …
- Iceberg Snapshot Expiration Age A table configuration property that defines the maximum age of snapshots before they become eligible for deletion during …
- Iceberg Snapshot Summary A map of key-value metadata properties embedded inside a snapshot that describes the write operation and counts of affec …
- Iceberg Spec V3 File Encryption The specification for securing Iceberg tables using envelope encryption and AES-GCM to encrypt data and metadata files.
- Iceberg Spec V3 Object-Store Storage Layout A storage layout configuration in Apache Iceberg that uses random prefix hashes to distribute data files evenly across c …
- Iceberg Table Metadata Schema The JSON schema specification for the Iceberg table metadata file, tracking schemas, partition specs, snapshots, and tab …
- Iceberg Truncate Partition Transform An Iceberg partition transform that truncates string values to a specific width or groups numeric values into fixed-size …
Patterns & Architecture
- Iceberg CDC (Change Data Capture) CDC with Apache Iceberg enables real-time synchronization of operational database changes (inserts, updates, deletes) in …
- Iceberg Data Mesh Architecture A data mesh on Apache Iceberg uses Iceberg tables as the storage standard for domain-owned data products, with the Icebe …
- Iceberg Lakehouse Federation Iceberg lakehouse federation enables querying Iceberg tables across multiple catalogs, cloud environments, and storage p …
- Iceberg Streaming Ingestion Iceberg streaming ingestion is the pattern of continuously writing data from event streams, Kafka topics, and CDC feeds …
- Iceberg Table Migration from Hive Migrating from Apache Hive tables to Apache Iceberg converts existing Parquet files into Iceberg-managed tables with ful …
- Medallion Architecture with Apache Iceberg The Medallion Architecture (Bronze/Silver/Gold) is a multi-layer data organization pattern where raw data flows through …
- Write-Audit-Publish (WAP) Pattern The Write-Audit-Publish (WAP) pattern is a data pipeline quality assurance workflow using Apache Iceberg branches to wri …
📚 Go Deeper on Apache Iceberg
Alex Merced has authored three hands-on books covering Apache Iceberg, the Agentic Lakehouse, and modern data architecture. Pick up a copy to master the full ecosystem.