Mooncake: Real-Time Columnstore & Apache Iceberg Lakehouse for PostgreSQL on Mooncake

Architecture

Mon, 01 Jan 0001 00:00:00 +0000

pg_mooncake (v0.3.0) combines three specialized components inside a single PostgreSQL deployment:

pg_duckdb (v1.2.0): Embeds the DuckDB v2.0.0 vectorized analytical execution engine inside PostgreSQL and routes queries against non-heap table access methods.
moonlink (v0.0.1): A Rust library and background worker embedded inside pg_mooncake that ingests PostgreSQL logical replication streams, maintains an in-memory Apache Arrow write buffer and Roaring Bitmap deletion vectors, flushes Parquet files, and emits Apache Iceberg v2 table metadata.
duckdb_mooncake (v0.0.2): A DuckDB 2.0 C++/Rust storage extension loaded automatically by pg_mooncake from a local extension repository (mooncake.extension_repository). It connects to Moonlink over a local Unix domain socket RPC (pg_mooncake/moonlink.sock) to read consistent snapshots across both flushed Parquet files and buffered Arrow batches.

Together, they expose a native PostgreSQL Table Access Method (USING mooncake) managed through CALL mooncake.create_table(dst, src).

Create Columnstore Table

Mon, 01 Jan 0001 00:00:00 +0000

In pg_mooncake, columnstore tables are created as real-time mirrors of existing PostgreSQL rowstore (heap or partitioned) tables using the mooncake.create_table procedure.

Procedure Signature

CALL mooncake.create_table(
    dst text,
    src text,
    src_uri text DEFAULT NULL,
    table_config json DEFAULT NULL
);

Parameter	Type	Default	Description
`dst`	`text`	(Required)	Name of the destination columnstore mirror table to create (supports `schema.table`, defaults to `public` if unqualified).
`src`	`text`	(Required)	Name of the source PostgreSQL rowstore or partitioned table to replicate from (supports `schema.table`).
`src_uri`	`text`	`NULL`	Connection URI (`postgresql://user:pass@host:port/dbname`) of a remote PostgreSQL instance when replicating from an external database. When `NULL`, replicates from the current local database.
`table_config`	`json`	`NULL`	Optional JSON object configuring Moonlink storage and ingestion settings (such as `write_buffer_size` or initial data copy behavior).

Parameter Name: src_uri (Not source_uri)

Earlier upstream documentation contained a typo showing source_uri => 'postgresql://...'. The actual parameter name in pg_mooncake’s SQL definition (src/functions.rs) is src_uri. Always use src_uri => 'postgresql://...' when calling mooncake.create_table with named arguments.

Hello World Tutorial

Mon, 01 Jan 0001 00:00:00 +0000

This tutorial walks through the core operations of pg_mooncake: creating a regular PostgreSQL rowstore table, creating a real-time columnstore mirror (USING mooncake), running transactional INSERT, UPDATE, and DELETE statements, testing rollback semantics, and joining columnstore and rowstore tables.

1. Enable the Extension

Connect to your PostgreSQL database and ensure pg_mooncake (along with pg_duckdb) is enabled:

CREATE EXTENSION IF NOT EXISTS pg_mooncake CASCADE;

2. Create a Rowstore Table & Columnstore Mirror

First, create a standard PostgreSQL heap table (user_activity) for your transactional writes, then call mooncake.create_table to create its real-time columnstore mirror (user_activity_columnstore):

Real-Time Operational Analytics (HTAP)

Mon, 01 Jan 0001 00:00:00 +0000

If you are running slow analytical queries (dashboards, customer-facing metrics, multi-table aggregations) on PostgreSQL tables, pg_mooncake gives you a straightforward route to accelerate them:

Keep your application write path 100% unchanged.
Create a real-time columnstore mirror with CALL mooncake.create_table(dst, src).
Route analytical SELECT queries to the columnstore mirror table.

Key Advantages

Schema changes supported: ALTER TABLE ... ADD COLUMN and DROP COLUMN on the source table automatically sync to the columnstore mirror.
Fast UPDATE and DELETE handling: moonlink uses in-memory Apache Arrow buffers and Roaring Bitmap deletion vectors, handling high-frequency OLTP updates and deletes far more efficiently than traditional Parquet/Iceberg writers.
Zero-ETL: No Debezium, Kafka, or Flink clusters to operate—pg_mooncake uses PostgreSQL’s built-in logical replication.
Transactional consistency: Analytical queries wait for the current PostgreSQL WAL LSN barrier before scanning, guaranteeing a consistent view of committed data.

Implementation Pattern: Write to Rowstore, Read from Columnstore

-- 1. Your existing OLTP table (for transactional writes and point lookups)
CREATE TABLE orders (
    order_id SERIAL PRIMARY KEY,
    customer_id INTEGER NOT NULL,
    total_amount DECIMAL(10, 2) NOT NULL,
    order_status TEXT NOT NULL DEFAULT 'pending',
    order_date TIMESTAMP NOT NULL DEFAULT NOW()
);

-- 2. Create the real-time columnstore mirror (for analytics)
CALL mooncake.create_table('orders_analytics', 'orders');

-- 3. Write operations: always use the original rowstore table
INSERT INTO orders (customer_id, total_amount) VALUES (123, 99.99);
UPDATE orders SET total_amount = 149.99, order_status = 'shipped' WHERE order_id = 1;

-- 4. Analytical read operations: query the columnstore mirror
SELECT
    customer_id,
    COUNT(*) AS order_count,
    SUM(total_amount) AS lifetime_value
FROM orders_analytics
WHERE order_date >= CURRENT_DATE
GROUP BY customer_id;

Manual Query Routing

Explicit Table Routing

pg_mooncake does not automatically rewrite queries targeting orders to target orders_analytics. You choose which table to query based on your workload:

Roadmap

Mon, 01 Jan 0001 00:00:00 +0000

Note

The roadmap items below reflect the historical upstream pg_mooncake v0.2 design goals published at docs.mooncake.dev/pg/roadmap, together with the current maintenance scope of the pgsty/pg_mooncake fork (v0.3.0).

PGSTY Maintenance Focus (`v0.3.0+`)

The PGSTY fork of the Mooncake suite (pg_mooncake v0.3.0, duckdb_mooncake v0.0.2, moonlink v0.0.1) prioritizes long-term production stability, PostgreSQL version coverage, and security hardening:

PostgreSQL 14–18 Compatibility: Keeping pg_mooncake synchronized with current pgrx (0.19.3) across PostgreSQL 14, 15, 16, 17, and 18 on Linux (x86_64 and aarch64, RHEL/Rocky EL8–EL10, Debian 12/13, Ubuntu 22.04/24.04).
DuckDB 2.0.0 & pg_duckdb 1.2.0 Alignment: Maintaining ABI compatibility between pg_mooncake, pg_duckdb, and duckdb_mooncake as DuckDB evolves its TableCatalogEntry, TableFilterSet, and StorageExtension APIs.
Correctness & Fail-Fast Guards: Enforcing runtime pg_duckdb version checks, LSN-consistent snapshot reads on columnstore tables, automatic REPLICA IDENTITY FULL propagation across partitioned table hierarchies, and explicit 0A000 errors on deprecated helpers.
Dependency Security Hygiene: Regularly auditing and upgrading Rust dependencies across all three lockfiles (pg_mooncake, moonlink, duckdb_mooncake/moonlink_ffi) against the RustSec Advisory Database.

Historical Upstream Roadmap (`v0.2`)

The following four capabilities were outlined by Mooncake Labs for the v0.2 architecture:

v0.1 Architecture (Delta Lake & USING columnstore)

Mon, 01 Jan 0001 00:00:00 +0000

Legacy pg_mooncake v0.1 Archive (Delta Lake Engine)

This page documents pg_mooncake v0.1, which used direct CREATE TABLE ... USING columnstore DDL and Delta Lake storage. Starting in v0.2 and PGSTY’s v0.3.0 release, pg_mooncake uses moonlink real-time columnstore mirrors (CALL mooncake.create_table + USING mooncake) and Apache Iceberg v2. For current documentation, see Introduction and v0.1 vs v0.2 & v0.3.

In pg_mooncake v0.1, table metadata is stored inside PostgreSQL catalogs for transactional consistency, while columnstore data is written as Parquet files with Delta Lake metadata on local disk or object storage.

Archiving Large Postgres Tables

Mon, 01 Jan 0001 00:00:00 +0000

pg_mooncake’s columnstore mirror writes data as compressed Parquet files (on local disk or S3 / Cloudflare R2 object storage), providing cost-effective, durable history while PostgreSQL heap storage only maintains hot recent partitions.

Architecture Pattern

The partition archiving pattern follows three rules:

Create a columnstore mirror (CALL mooncake.create_table) of a time- or range-partitioned PostgreSQL table.
Periodically DROP or TRUNCATE old partitions from the PostgreSQL rowstore once they age out of your hot operational window.
Retain complete history in the columnstore mirror—pg_mooncake intentionally ignores DROP TABLE and TRUNCATE events on child partitions of a partitioned source table, so all historical rows remain queryable in the columnstore!

Step-by-Step Walkthrough

Below is a complete example using a range-partitioned transactions table.

Known Limitations

Mon, 01 Jan 0001 00:00:00 +0000

Before deploying pg_mooncake (v0.3.0) in production, review the following architectural constraints and known limitations.

1. Read-Only Columnstore Mirror Tables

Tables created with CALL mooncake.create_table(dst, src) use the USING mooncake table access method and are read-only replicas.

Direct INSERT, UPDATE, DELETE, or COPY statements targeting the USING mooncake table are rejected by PostgreSQL (ERROR:cannot modify table "..." because it uses the mooncake access method).
Always execute DML operations against the source rowstore table (src); changes replicate automatically via moonlink.

2. Primary Keys & `REPLICA IDENTITY FULL`

To replicate UPDATE and DELETE operations accurately over PostgreSQL logical replication, the source table must provide complete old-row identification in the WAL stream:

License

Mon, 01 Jan 0001 00:00:00 +0000

The PGSTY Mooncake ecosystem preserves the exact upstream license terms of each repository created by Mooncake Labs, Inc.

Summary by Repository

Component	Repository	License	Notes
`pg_mooncake`	`pgsty/pg_mooncake`	`MIT`	Permissive open-source license (`Copyright (c) 2024 Mooncake Labs`).
`duckdb_mooncake`	`pgsty/duckdb_mooncake`	`MIT`	Permissive open-source license (`Copyright (c) 2025 Mooncake Labs`).
`moonlink` (Engine)	`pgsty/moonlink`	`BSL-1.1` + Exception	Business Source License 1.1 with the `PG_MOONCAKE LINKING & DISTRIBUTION EXCEPTION`, converting to `Apache-2.0` on `2029-05-05`.
`moonlink` (Client Crates)	`moonlink_rpc`, `moonlink_proto`, `moonlink_error`, `moonlink_table_metadata`	`MIT`	Explicitly licensed under MIT so `pg_mooncake` and `duckdb_mooncake` remain pure MIT works when linking against them.
Documentation Site	`pgsty/mooncake.pgsty.com`	`CC-BY-4.0` / `Apache-2.0`	Documentation prose under Creative Commons Attribution 4.0 International (`CC-BY-4.0`); site build scripts under `Apache-2.0`.

`pg_mooncake` & `duckdb_mooncake` — MIT License

Both pg_mooncake and duckdb_mooncake are distributed under the MIT License:

Moonlink Ingestion Engine

Mon, 01 Jan 0001 00:00:00 +0000

Moonlink (pgsty/moonlink, current release v0.0.1) is a Rust library and standalone service designed to write and maintain real-time Apache Iceberg v2 and Delta Lake tables for operational systems and event streams without Apache Spark or Kafka.

Inside pg_mooncake, Moonlink runs as an embedded PostgreSQL background worker (mooncake_bgworker) communicating over a local Unix domain socket ($PGDATA/pg_mooncake/moonlink.sock). It can also be compiled and deployed as a standalone HTTP REST and gRPC ingestion server (moonlink_service).

Query Columnstore Table & Supported Query Shapes

Mon, 01 Jan 0001 00:00:00 +0000

Columnstore mirror tables created with CALL mooncake.create_table(dst, src) are queried using standard PostgreSQL SELECT syntax. Behind the scenes, pg_mooncake and pg_duckdb 1.2.0 translate and route analytical scans to the embedded DuckDB v2.0.0 vectorized execution engine.

How Queries Work

When you query a USING mooncake table:

LSN Synchronization: duckdb_mooncake checks PostgreSQL’s current WAL write LSN and waits for moonlink to ingest all committed transactions up to that LSN.
Segment Elimination: DuckDB 2.0 inspects multi-file Parquet column min/max statistics to skip files and row groups that cannot match your WHERE predicates.
Zero-Copy Hybrid Scan: Flushed Parquet files, Roaring Bitmap deletion vectors, and the in-memory Apache Arrow write buffer are scanned in parallel across DuckDB worker threads (duckdb.threads).

You can verify that DuckDB is executing your query by running EXPLAIN:

v0.1 Get Started

Mon, 01 Jan 0001 00:00:00 +0000

Legacy pg_mooncake v0.1 Archive (Delta Lake Engine)

Historical Docker Image (Obsolete)

docker pull mooncakelabs/pg_mooncake
docker run --name mooncake-demo -e POSTGRES_HOST_AUTH_METHOD=trust -p 5432:5432 -d mooncakelabs/pg_mooncake
docker run -it --rm --link mooncake-demo:postgres mooncakelabs/pg_mooncake psql -h postgres -U postgres

Building `v0.1` from Source

git submodule update --init --recursive
make release -j$(nproc)
make install

Create the Extension

CREATE EXTENSION pg_mooncake;

v0.1 Hello World (USING columnstore)

Mon, 01 Jan 0001 00:00:00 +0000

Legacy pg_mooncake v0.1 Archive (Delta Lake Engine)

Create the Extension

CREATE EXTENSION pg_mooncake;

Create a `USING columnstore` Table (`v0.1`)

CREATE TABLE user_activity (
    user_id BIGINT,
    activity_type TEXT,
    activity_timestamp TIMESTAMP,
    duration INT
) USING columnstore;

Inserts, Updates, and Deletes

INSERT INTO user_activity VALUES
    (1, 'login', '2024-01-01 08:00:00', 120),
    (2, 'page_view', '2024-01-01 08:05:00', 30),
    (3, 'logout', '2024-01-01 08:30:00', 60),
    (4, 'error', '2024-01-01 08:13:00', 60);

UPDATE user_activity SET duration = 180 WHERE user_id = 1;
DELETE FROM user_activity WHERE user_id = 2;

Transactions

BEGIN;
INSERT INTO user_activity VALUES
    (5, 'login', '2024-01-02 10:00:00', 200),
    (6, 'login', '2024-01-02 10:30:00', 90);
ROLLBACK;
SELECT * FROM user_activity;

Joins

-- Between two columnstore tables
SELECT * FROM user_activity ua
JOIN user_activity ua2 ON ua.user_id = ua2.user_id;

-- Between a regular PostgreSQL table and a columnstore table
SELECT * FROM users JOIN user_activity ON users.user_id = user_activity.user_id;

Attribution

Mon, 01 Jan 0001 00:00:00 +0000

Upstream Creators

pg_mooncake, duckdb_mooncake, and moonlink were created by Mooncake Labs. All core architectural credit — including the hybrid rowstore + columnstore design, the Moonlink real-time Iceberg buffer engine, logical replication WAL ingestion, and the DuckDB execution bridge — belongs to the original Mooncake Labs authors and contributors.

Repository Lineage

PGSTY Repository	Upstream Source	Upstream Baseline Commit	PGSTY Release Tag
`pgsty/pg_mooncake`	`Mooncake-Labs/pg_mooncake`	`v0.2.0` (`be17cd0`) + post-0.2 upstream commits	`v0.3.0` (`1a9cbfd65be9a6486d020120acb49ef0c6800d68`)
`pgsty/duckdb_mooncake`	`Mooncake-Labs/duckdb_mooncake`	`ba79184`	`v0.0.2` (`49765a369ad88986916ce53c969ed2ce4ba66fac`)
`pgsty/moonlink`	`Mooncake-Labs/moonlink`	`f2e1715`	`v0.0.1` (`b298ef7444890e3c8af5544e350232518bcfd8e5`)
`pgsty/pg_duckdb`	`duckdb/pg_duckdb`	`v1.2.0`	`v1.2.0` (`8756099e267c47f0b86c24de4cf995327f1f4279`)

Documentation Provenance & Recovery

The original upstream documentation website (https://docs.mooncake.dev/) was taken offline after the upstream project transitioned direction, and no public documentation source repository remained available on GitHub.

Download & Install

Mon, 01 Jan 0001 00:00:00 +0000

Pre-built native RPM and DEB packages for pg_mooncake (including the embedded moonlink ingestion engine and duckdb_mooncake DuckDB extension) are published across 16 Linux OS / CPU architecture targets (el8, el9, el10, d12, d13, u22, u24, u26 × x86_64 / aarch64) for PostgreSQL 14, 15, 16, 17, and 18 (80 binary packages across 5 PostgreSQL major versions).

pg_mooncakev0.3.0

PostgreSQL Table Access Method (USING mooncake) and columnstore mirror procedures (mooncake.create_table, create_snapshot, optimize_table, list_tables), built with pgrx 0.19.3 for PostgreSQL 14–18.

moonlinkv0.0.1

Rust real-time ingestion engine embedded inside pg_mooncake’s background worker (and available as a standalone HTTP/RPC server), managing logical replication, Arrow write buffers, Roaring deletion vectors, and Apache Iceberg v2 snapshots.

duckdb_mooncakev0.0.2

DuckDB 2.0 storage extension bundled with pg_mooncake into a local extension repository (mooncake.extension_repository), enabling pg_duckdb 1.2.0 to scan Moonlink tables with multi-file Parquet statistics pruning.

duckdb_mooncake & DuckDB 2.0 Integration

Mon, 01 Jan 0001 00:00:00 +0000

duckdb_mooncake (pgsty/duckdb_mooncake, current release v0.0.2, MIT license) is a C++ and Rust FFI storage extension for DuckDB v2.0.0. It allows DuckDB (both standalone and embedded inside PostgreSQL via pg_duckdb 1.2.0) to attach directly to a moonlink metadata directory and execute vectorized scans over Moonlink tables.

Role Inside `pg_mooncake 0.3.0`

When pg_mooncake executes a query against a USING mooncake table inside PostgreSQL, pg_duckdb delegates the table scan to DuckDB. pg_mooncake automatically initializes the DuckDB session by running:

Apache Iceberg v2 & Object Storage Configuration

Mon, 01 Jan 0001 00:00:00 +0000

Every columnstore mirror table in pg_mooncake (v0.3.0) is backed by moonlink, which writes data as standard Parquet files and maintains Apache Iceberg v2 table metadata.

By default, Moonlink writes files to the local filesystem under $PGDATA/pg_mooncake/. You can configure pg_mooncake to write directly to Amazon S3, Cloudflare R2, MinIO, or Google Cloud Storage (GCS) so external engines (DuckDB, Apache Spark, Trino, Snowflake, PyIceberg) can query your PostgreSQL tables directly from object storage.

The Postgres Lakehouse

Mon, 01 Jan 0001 00:00:00 +0000

The Lakehouse architecture stores data once in open table formats (Apache Iceberg or Delta Lake) on object storage so multiple compute engines—PostgreSQL, DuckDB, Apache Spark, Snowflake, Trino, and Python (Polars / PyIceberg)—can share the same dataset without copying it.

pg_mooncake makes PostgreSQL a first-class citizen of your lakehouse in both directions:

Postgres $\rightarrow$ Iceberg Lakehouse (Zero-ETL Export): Replicate operational PostgreSQL tables directly into Apache Iceberg v2 tables on S3, Cloudflare R2, or local storage via moonlink.
Lakehouse $\rightarrow$ Postgres (Direct In-DB Scan): Query external Parquet, Apache Iceberg, and Delta Lake tables directly from PostgreSQL and join them with live operational tables via pg_duckdb.

1. Export Operational Postgres Tables as Apache Iceberg v2

Traditionally, landing PostgreSQL data in Apache Iceberg required deploying Debezium, Kafka, Schema Registry, and Spark/Flink consumers. With pg_mooncake, it takes three SQL commands:

v0.1 Cloud Storage (mooncake.create_secret)

Mon, 01 Jan 0001 00:00:00 +0000

Legacy pg_mooncake v0.1 Archive (Delta Lake Engine)

Current v0.3.0 Cloud Storage Setup

In v0.2 and v0.3.0, credentials are inserted into duckdb.secrets rather than mooncake.create_secret. See Iceberg & Object Storage.

v0.1 on Neon Postgres

Mon, 01 Jan 0001 00:00:00 +0000

Legacy pg_mooncake v0.1 Archive (Delta Lake Engine)

pg_mooncake v0.1 was offered as an experimental extension on Neon.

Enable the Extension on Neon (`v0.1`)

SET neon.allow_unstable_extensions = 'true';
CREATE EXTENSION pg_mooncake;

Cloud Storage Requirement

Because Neon does not allow extensions to persist columnstore files on local compute disk, using pg_mooncake v0.1 on Neon required configuring an external S3 or Cloudflare R2 bucket as described in v0.1 Cloud Storage, and disabling Neon’s scale-to-zero feature to preserve local metadata caches.

Log Analytics with OpenTelemetry

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL is rarely used for large-scale log analytics because uncompressed heap tables and indexes grow expensive quickly, and full-table scans over millions of log entries are slow.

With pg_mooncake, you can combine partitioned PostgreSQL log ingestion with an object-storage-backed columnstore mirror:

Low storage cost: Historical logs are compressed into columnar Parquet files on S3, Cloudflare R2, or local disk.
Simple self-hosting: Just PostgreSQL + object storage—no Elasticsearch or ClickHouse cluster required.
Fast vectorized scans: DuckDB 2.0 scans millions of log records per second with segment elimination on timestamps.

Implementation Pattern

Combine a time-partitioned rowstore table for log ingestion with a USING mooncake columnstore mirror for analytical queries:

Query Your Data Lake in Postgres

Mon, 01 Jan 0001 00:00:00 +0000

Because pg_mooncake (v0.3.0) embeds DuckDB v2.0.0 via pg_duckdb 1.2.0, you can query external files in your data lake (local disk, S3, Cloudflare R2, GCS, HTTP, or HuggingFace hf:// datasets) directly from PostgreSQL without loading them into heap tables first.

Supported External Formats

Apache Parquet (duckdb.read_parquet)
CSV / TSV (duckdb.read_csv)
JSON / NDJSON (duckdb.read_json)
Apache Iceberg (duckdb.iceberg_scan)
Delta Lake (duckdb.delta_scan)

v0.2 / v0.3 (duckdb.*) vs Legacy v0.1 (mooncake.*) Functions

In pg_mooncake v0.2 and v0.3.0, external file readers are provided directly by pg_duckdb (duckdb.read_parquet, duckdb.read_csv, duckdb.iceberg_scan, duckdb.delta_scan). For the legacy v0.1 mooncake.read_parquet syntax, see v0.1 Query External Files.

Trademark Policy

Mon, 01 Jan 0001 00:00:00 +0000

Independence Statement

mooncake.pgsty.com and the repositories under github.com/pgsty (pgsty/pg_mooncake, pgsty/duckdb_mooncake, pgsty/moonlink) are maintained independently by PGSTY (Pigsty) for the PostgreSQL open-source community.

This site and the packages distributed by Pigsty / PGEXT are not affiliated with, endorsed by, or sponsored by Mooncake Labs, Inc.

Third-Party Trademarks

Mooncake, pg_mooncake, duckdb_mooncake, and Moonlink are names originated by Mooncake Labs, Inc. They are used on this website solely for nominative, descriptive identification of the open-source PostgreSQL and DuckDB extensions and their compatibility lineage.
PostgreSQL and Postgres are registered trademarks of the PostgreSQL Community Association of Canada.
DuckDB is a trademark of the DuckDB Foundation / DuckDB Labs.
Apache Iceberg, Apache Parquet, Apache Arrow, and Apache are trademarks of the Apache Software Foundation.
Delta Lake is a trademark of LF Projects, LLC.
Pigsty and PGSTY are trademarks of Ruohang Feng (@Vonng).

v0.1 Reference Overview

Mon, 01 Jan 0001 00:00:00 +0000

Legacy pg_mooncake v0.1 Archive (Delta Lake Engine)

v0.1 vs v0.2 & v0.3 Evolution

Mon, 01 Jan 0001 00:00:00 +0000

pg_mooncake has evolved through three major architectural milestones:

v0.1 (Legacy Delta Lake Engine): Introduced direct CREATE TABLE ... USING columnstore tables backed by Delta Lake and DuckDB (pg_duckdb).
v0.2 (Moonlink & Apache Iceberg Redesign): Replaced direct columnstore writes with a columnstore mirror architecture (CALL mooncake.create_table(dst, src) + USING mooncake) powered by moonlink logical replication, in-memory Arrow write buffers, Roaring deletion vectors, and Apache Iceberg v2 metadata.
v0.3.0 (PGSTY Maintained Release): Upgraded pg_mooncake to pgrx 0.19.3 (PostgreSQL 14–18), ported duckdb_mooncake v0.0.2 to pg_duckdb 1.2.0 and DuckDB v2.0.0, fixed Moonlink (v0.0.1) snapshot LSN hangs and partitioned table replica identity handling, and patched 20+ RustSec dependency advisories.

Looking for v0.1 Documentation?

Complete documentation for the legacy v0.1 release (USING columnstore, mooncake.create_secret, mooncake.columnstore_tables, mooncake.read_parquet) is preserved in the v0.1 Archive (Delta Lake).

v0.1 Columnstore DDL (USING columnstore)

Mon, 01 Jan 0001 00:00:00 +0000

Legacy pg_mooncake v0.1 Archive (Delta Lake Engine)

Create Columnstore Table (`v0.1`)

In v0.1, columnstore tables were created directly with the USING columnstore clause:

SBOM & Security Audit

Mon, 01 Jan 0001 00:00:00 +0000

This page documents the exact submodule pin matrix (versions.json) and the dependency security audit performed for pg_mooncake v0.3.0, duckdb_mooncake v0.0.2, and moonlink v0.0.1.

Component & Submodule Pin Matrix (`versions.json`)

pgsty/pg_mooncake records its exact submodule and upstream dependency pins in versions.json:

Component	Path	Repository	Tag / Branch	Commit SHA	License
`pg_mooncake`	`/`	`pgsty/pg_mooncake`	`v0.3.0`	`1a9cbfd65be9a6486d020120acb49ef0c6800d68`	`MIT`
`duckdb_mooncake`	`duckdb_mooncake`	`pgsty/duckdb_mooncake`	`v0.0.2`	`49765a369ad88986916ce53c969ed2ce4ba66fac`	`MIT`
`duckdb` (nested)	`duckdb_mooncake/duckdb`	`duckdb/duckdb`	`v2.0.0`	`d54cbbe3df17e8e5ce68312f483b4155836112aa`	`MIT`
`moonlink`	`moonlink` (`duckdb_mooncake/moonlink`)	`pgsty/moonlink`	`v0.0.1`	`b298ef7444890e3c8af5544e350232518bcfd8e5`	`BSL-1.1` + Exception (`MIT` client crates)
`pg_duckdb`	`pg_duckdb`	`pgsty/pg_duckdb`	`v1.2.0`	`8756099e267c47f0b86c24de4cf995327f1f4279`	`MIT`
`rust-postgres`	`rust-postgres`	`sfackler/rust-postgres`	`master`	`14c8e599f5551a8caa96ff5023685a6f537a1455`	`Apache-2.0 OR MIT`

Core Rust Toolchain & Framework Pins

Dependency	Version	Role
`pgrx` / `cargo-pgrx`	`0.19.3`	PostgreSQL extension framework for Rust (PostgreSQL 14–18).
`iceberg`	`0.6.0`	Apache Iceberg table metadata and catalog client in `moonlink`.
`arrow` / `parquet`	`55.2.0`	Columnar memory format and Parquet writer in `moonlink`.
`datafusion`	`48.0.1`	Expression evaluation and pruning in `moonlink`.
`opendal`	`0.53.1`	Unified object storage abstraction (`fs`, `s3`, `gcs`, `azblob`).
`tonic` / `prost`	`0.14.5` / `0.14.3`	gRPC / IPC protocol serialization between `pg_mooncake` and `moonlink`.
`tokio`	`1.53.0`	Async runtime for background ingestion and object store I/O.

2026-11-05 Cargo Dependency Security Audit

All three Cargo workspaces were audited using cargo-audit 0.22.2 against the RustSec Advisory Database (commit 965bf9ad083b58e6340d65829d85217b18be9290, 927 advisories loaded).

v0.1 Querying Columnstore Outside Postgres

Mon, 01 Jan 0001 00:00:00 +0000

Legacy pg_mooncake v0.1 Archive (Delta Lake Engine)

Current v0.3.0 Metadata Function

In v0.2 and v0.3.0, use SELECT * FROM mooncake.list_tables(); to inspect table LSNs and Apache Iceberg v2 warehouse locations.

v0.1 GUC Configurations

Mon, 01 Jan 0001 00:00:00 +0000

Legacy pg_mooncake v0.1 Archive (Delta Lake Engine)

`v0.1` GUC Parameters

-- Default cloud storage bucket
ALTER DATABASE <database_name> SET mooncake.default_bucket = 's3://my-bucket';

-- Memory allocated to DuckDB (v0.1.1+, default 4GB)
ALTER DATABASE <database_name> SET mooncake.maximum_memory = '6GB';
CALL mooncake.reset_duckdb();

-- Number of threads for DuckDB (v0.1.1+, default -1 = all cores)
ALTER DATABASE <database_name> SET mooncake.maximum_threads = -1;

-- Local write cache (default true)
ALTER DATABASE <database_name> SET mooncake.enable_local_cache = false;

-- Memory cache for Parquet metadata (default true)
ALTER DATABASE <database_name> SET mooncake.enable_memory_metadata_cache = false;

v0.1 Load Data

Mon, 01 Jan 0001 00:00:00 +0000

Legacy pg_mooncake v0.1 Archive (Delta Lake Engine)

From PostgreSQL Heap Tables (`v0.1`)

INSERT INTO users_columnstore
SELECT * FROM users_rowstore;

From Parquet, CSV, or JSON Files (`v0.1`)

-- Using COPY
COPY users_columnstore FROM 's3://bucket/file.parquet';
COPY users_columnstore FROM 's3://bucket/folder/*.parquet';
COPY users_columnstore FROM 's3://bucket/data.csv' WITH (FORMAT 'csv', HEADER true);

-- Using INSERT with mooncake.read_parquet
INSERT INTO users_columnstore
SELECT * FROM mooncake.read_parquet('s3://bucket/file.parquet') AS (
    id INT,
    name TEXT,
    created_at TIMESTAMP
);

From Iceberg, Delta Lake, or HuggingFace (`v0.1`)

-- Iceberg
INSERT INTO users_columnstore
SELECT * FROM iceberg_scan('s3://bucket/table') AS (id INT, name TEXT, created_at TIMESTAMP);

-- Delta Lake
INSERT INTO users_columnstore
SELECT * FROM delta_scan('s3://bucket/table') AS (id INT, name TEXT, created_at TIMESTAMP);

-- HuggingFace
COPY users_columnstore FROM 'hf://datasets/OpenLeecher/lmsys_chat_1m_clean/data/train-00000-of-00004.parquet';

v0.1 Query External Files

Mon, 01 Jan 0001 00:00:00 +0000

Legacy pg_mooncake v0.1 Archive (Delta Lake Engine)

Current v0.3.0 External File Readers

In v0.2 and v0.3.0, use duckdb.read_parquet, duckdb.read_csv, duckdb.iceberg_scan, and duckdb.delta_scan as documented in Query Your Data Lake in Postgres.

v0.1 Supported Queries & Query Shapes

Mon, 01 Jan 0001 00:00:00 +0000

Legacy pg_mooncake v0.1 Archive (Delta Lake Engine)

For the updated v0.3.0 guide on query shapes and DuckDB 2.0 execution, see Query Columnstore & Query Shapes.

v0.1 Data Archiving

Mon, 01 Jan 0001 00:00:00 +0000

Legacy pg_mooncake v0.1 Archive (Delta Lake Engine)

Current v0.3.0 Partition Archiving Pattern

In v0.2 and v0.3.0, you no longer need manual INSERT INTO ... DELETE FROM batch scripts—simply mirror a partitioned table with CALL mooncake.create_table and DROP old partitions. See Archiving Large Postgres Tables.

v0.1 Operational Analytics (Trigger + pg_cron Staging)

Mon, 01 Jan 0001 00:00:00 +0000

Legacy pg_mooncake v0.1 Archive (Delta Lake Engine)

Replaced by mooncake.create_table in v0.2 / v0.3.0

In v0.1, syncing a rowstore table to a USING columnstore table required triggers, a staging table, and a pg_cron job. In v0.2 and v0.3.0, all of this is replaced by a single CALL mooncake.create_table('orders_analytics', 'orders');. See Real-Time Operational Analytics.

v0.1 Data Warehousing & dbt

Mon, 01 Jan 0001 00:00:00 +0000

Legacy pg_mooncake v0.1 Archive (Delta Lake Engine)

Daily Aggregation Pipeline (`v0.1`)

CREATE TABLE daily_sales_summary (
    date DATE,
    product_id INT,
    category_id INT,
    region_id INT,
    total_quantity INT,
    total_revenue DECIMAL(18, 2)
) USING columnstore;

Integration with dbt (`v0.1`)

In v0.1, dbt models could target USING columnstore by setting default_table_access_method in an on-run-start hook:

Page Not Found

Mon, 01 Jan 0001 00:00:00 +0000

The page you are looking for has moved, been deleted, or does not exist. Use the search bar (⌘K) or return to the Mooncake Documentation.

pg_mooncake 0.3.0: First PGSTY Release with DuckDB 2.0, pgrx 0.19.3, and PostgreSQL 14–18 Support

Thu, 05 Nov 2026 00:00:00 +0000

pg_mooncake 0.3.0 — released alongside duckdb_mooncake v0.0.2 and moonlink v0.0.1 — is the first release of the Mooncake real-time analytics suite maintained by PGSTY:

pgsty/pg_mooncake v0.3.0 (MIT)
pgsty/duckdb_mooncake v0.0.2 (MIT)
pgsty/moonlink v0.0.1 (BSL-1.1 with PG_MOONCAKE LINKING & DISTRIBUTION EXCEPTION, client crates MIT)

Binary RPM and DEB packages for PostgreSQL 14, 15, 16, 17, and 18 across RHEL / Rocky / AlmaLinux 8, 9, 10 and Debian 12/13, Ubuntu 22.04/24.04 (x86_64 and aarch64) are available through the PGEXT repository and Pigsty.

Porting pg_mooncake and duckdb_mooncake to DuckDB 2.0, pg_duckdb 1.2.0, and pgrx 0.19.3

Thu, 05 Nov 2026 00:00:00 +0000

When Mooncake Labs paused public updates to pg_mooncake after v0.2.0, the extension was pinned to an earlier DuckDB 1.3.x / pg_duckdb snapshot and pgrx 0.12. Meanwhile, the PostgreSQL ecosystem moved forward to PostgreSQL 18, pgrx 0.19.3, and pg_duckdb 1.2.0 (which embeds DuckDB v2.0.0).

This article walks through the engineering changes required to bring the entire Mooncake stack — pg_mooncake v0.3.0, duckdb_mooncake v0.0.2, and moonlink v0.0.1 — onto DuckDB 2.0.0, pg_duckdb 1.2.0, and pgrx 0.19.3.

Mooncake is joining Databricks!

Wed, 01 Oct 2025 00:00:00 +0000

Thirteen months ago, we started Mooncake to bring the data intelligence stack that powers giants like OpenAI to every vibe-coded app and developer. From first prompt to IPO.

Today, we’re joining forces with Databricks. The mission continues: make Lakebase and the Lakehouse feel like one.

For a decade, we thought the final form of a database was one that could do it all. Fast updates, transactions, analytics. For database people, this was HTAP. It never quite landed. Zhou and Cheng built it. Pranav tried to sell it. It was hard.

Mooncake: a real-time storage engine for Apache Iceberg

Mon, 11 Aug 2025 00:00:00 +0000

This technical overview assumes familiarity with Apache Iceberg, an open table format for analytical datasets. Iceberg is part of the general “Lakehouse” movement, where users:

Store large datasets using open file formats (Parquet + Metadata) in Object Storage
Query them directly by engines like Spark, Trino, Flink, and DuckDB using SQL
Get the benefits of ‘database features’ like schema evolution & time-travel
Scale practically infinitely, and aren’t locked into a vendor

Iceberg was built for slow-changing data + batch analytics & processing. That world is pretty clear and only improving with engines getting better at reading open formats.

No, Iceberg won’t fix your data stack.

Mon, 21 Jul 2025 00:00:00 +0000

Life is expectations vs reality. Here is one for Apache Iceberg – a favorite amongst the data community.

Expectations: Iceberg is the solution to all my data pains

See, the core of Iceberg is pretty neat:

Keep all data in object storage in open file formats like Parquet.
Fool any execution engine (DuckDB, ClickHouse, Spark, Snowflake, etc.) into thinking these files are a table and query them with SQL directly.
[Ongoing progress] Get exceptional performance on these tables, without having to load data into a system-specific format.

And with clever design you get:

but, my Postgres (+ lakehouse) is free

Fri, 13 Jun 2025 00:00:00 +0000

Disclaimer: Your Postgres and Lakehouse are free and open too. You don’t need to be (or spend like) a giant to use them.

Why Postgres + Lakehouse?

Reason #1: If you believe in AI, this is the agent stack

Agents need memory. Both working memory and longer-term knowledge.

Working memory has to be instant, reliable, and accurate, just like app state. What’s the best place to store it? Postgres. That’s true for both apps and agents. This state is typically tied to an individual user, tenant, or agent.

HTAP is Dead

Mon, 05 May 2025 00:00:00 +0000

This blog is inspired by Jordan Tigani’s blog titled “Big Data is Dead”. Jordan and I actually spent some time building an HTAP database at SingleStore.

The good old days (’80s)

Back in the ’80s, one relational database did everything. Transactions (OLTP) during the day and reports after hours (OLAP). Databases like Oracle V2 and IBM DB2 ran OLTP and OLAP on the same system; largely because data sets still fit on a few disks and compute was costly.

What does Analytics in Postgres really mean?

Sun, 23 Mar 2025 00:00:00 +0000

Few areas in databases are hotter right now than extending Postgres to support analytics queries. This isn’t a particularly novel area either –– there have been plenty of attempts in the last decade:

Here is what it means to our customers –– and why the problem remains largely unsolved, despite our first attempt with pg_mooncake.

Level 0: I just run analytics queries on my Postgres tables

You’ll be surprised how far this can go with proper indexes, statistics, query tuning, and materialized views—people really overthink this.

pg_mooncake v0.2: what we're cooking

Tue, 11 Feb 2025 00:00:00 +0000

With v0.1, we proved that Postgres can deliver great analytics performance. Our approach combined two key pieces—DuckDB vectorized engine and a columnstore table—making pg_mooncake the only extension in Postgres that integrates both.

The results speak for themselves: pg_mooncake ranks in ClickBench’s top 10, and we built the fastest Parquet-based engine along the way.

Now, with v0.2, we’re tackling the next big challenge; we call it out-of-the-box analytics.

First, why in-Postgres analytics?

Applications often have their first analytics needs far before they have their first data warehouse or analytics system.

How to improve DuckDB performance on Parquet

Wed, 22 Jan 2025 00:00:00 +0000

Last week, we ran ClickBench on Postgres and ranked in the top 10. We showed that Postgres can be extended with pg_mooncake to support serious analytics workloads.

In the announcement, we got a lot of questions on how pg_mooncake outperforms DuckDB on Parquet files even though internally, it embeds DuckDB in Postgres and stores data in Parquet format.

See also the discussion on X / Twitter.

The best part: pg_mooncake does not modify DuckDB at all…

Clickbench says Postgres is a great analytics database

Tue, 14 Jan 2025 00:00:00 +0000

TL;DR: We spent a few months optimizing PostgreSQL and made it to the Top 10 on ClickBench, a benchmark typically dominated by specialized analytics databases.

What’s more, all compute is within Postgres, and all tables are managed directly by PostgreSQL—it’s not a simple wrapper. This is the story of pg_mooncake.

What Is ClickBench?

ClickBench is the definitive benchmark for real-time analytics databases, originally designed to showcase the performance of ClickHouse. It evaluates databases on their ability to handle real-world analytics workloads, including high-volume table scans and complex aggregations.

wth are S3 tables

Sun, 12 Jan 2025 00:00:00 +0000

Amazon announced table buckets during re:Invent last year. An S3 bucket that’s also a table? What does that mean?

TL;DR: S3 tables allow Iceberg tables to exist without a catalog. Let me explain.

Iceberg vs Delta Lake

What? I thought Iceberg and Delta are just table formats? Why are we comparing the way they’re implemented?

A key difference between Iceberg and Delta Lake is “Delta tables can exist without a catalog, but Iceberg cannot”, due to their metadata design:

The easiest way to Analytics in Postgres: pg_mooncake v0.1

Sat, 11 Jan 2025 00:00:00 +0000

We announced pg_mooncake to the world on Day 60.

Today, Day 131, pg_mooncake comes out of beta.

pg_mooncake wins on ease and performance. So, we doubled down for v0.1. That’s it.

The Easiest Way to Fast Analytics in Postgres

With a native columnstore table in Postgres, pg_mooncake brings familiar table semantics to the way you run analytics. Columnstore tables support transactional operations like INSERT, SELECT, UPDATE, DELETE, and COPY, alongside joins with regular Postgres heap tables.

pg_mooncake 🤝 drizzle: typed SDKs for your analytic workloads

Wed, 08 Jan 2025 00:00:00 +0000

A thing that surprised us in the pg_mooncake launch was how much developers cared about ORMs for their analytical workloads. We saw two typical requests:

“I have a bunch of Parquet files. I want full ORM support over them.”
“I use Drizzle/Prisma for my transactional database. I also want to use it for my analytics and dashboards.”

We’ve always believed that columnar storage in Postgres must look and feel like a regular heap table. It’s where we innovated with pg_mooncake columnstore tables; you can run transactions, updates, deletes, and joins with heap tables.

pg_mooncake + Neon: serverless analytics in Postgres

Tue, 07 Jan 2025 00:00:00 +0000

pg_mooncake launched in beta two months ago. It adds columnar storage and a vectorized engine to Postgres.

Over the last two months, with our customers, we’ve realized we accidentally built the best serverless database for analytics. Here’s the story:

1. pg_mooncake 🤝 Neon storage engine

Neon innovated significantly on its storage engine.

There’s a great thread by Nikita on this. This storage engine is how Neon can achieve true separation of storage & compute. This leads to an exceptional DX around branching, scale-to-zero, read replicas, instant provisioning… All of these are goodies of a true serverless Postgres.

pg_mooncake: Fast Analytics in Postgres with Columnstore Tables and DuckDB

Wed, 06 Nov 2024 00:00:00 +0000

pg_mooncake brings columnstore tables with DuckDB execution to Postgres for fast analytics.

It supports transactional inserts, updates, and deletes, just like regular Postgres tables. Data is written with Delta Lake (and Apache Iceberg in v0.2+) table metadata, allowing it to be queried outside of Postgres.

Why another Postgres analytics extension?

Postgres is the most popular database these days, but it’s bad at analytics. Numerous attempts have been made to address this, but with limited success so far. Fast analytics require a distinct set of techniques, such as columnar storage, vectorized execution, and late materialization, etc. Early projects like Citus & Timescale added columnar storage, but there was no good vectorized execution to leverage the format.

Mooncake: next-gen search and analytic systems

Wed, 30 Oct 2024 00:00:00 +0000

We are Zhou, Cheng, and Pranav. And we’ve spent the last decade building databases at MemSQL (now SingleStore).

We love complex systems, deep-tech engineering, and naturally – physics 🙂 Data systems, much like physics, are built on certain principles. The principles need to be unanimous and durable.

Building systems that are simply ’technically’ better is futile. The success of a data system depends on the principles it embodies, and changing them along the way is extremely difficult. Closed-source companies can’t go to open source and expect adoption. Databases can’t just abandon their storage format and be effective stateless engines.

Reflections from the Iceberg Meetup

Tue, 08 Oct 2024 00:00:00 +0000

Last month, I spoke at the Iceberg meetup at Snowflake HQ. Public speaking has never been my favorite, and knowing Snowflake CEO Sridhar and Tabular CEO Ryan were in the audience did not help.

It was day 18 of starting Mooncake Labs, and I spoke about a general indexing framework for Iceberg. You can find the details here. Here were some of my learnings:

1. Great products are a reflection of its community.

I spent the last decade building a proprietary database. Every time a feature request came in, engineering and product leaders would spend hours prioritizing its impact. Almost every time, it boiled down to one question: How much ARR? I watched my best engineers spend months building awkward hooks and extensions that did not fit the bigger picture. I often thought, Just ask the customer to make a PR :) The end product of this type of prioritization process tends to look like:

Things we learnt

Sun, 08 Sep 2024 00:00:00 +0000

The title should really be “Things we learnt, the hard way”.

As we start this new journey, we spent some time thinking about the axioms of data today. These are things we (or no one for that matter) can dispute.

1. If Postgres can do it, Postgres will do it

We’ve seen this one plenty of times now – it started with JSON, then moved to geo-spatial (PostGIS), and now we’re seeing it with vector search (pgvector).

Mooncake: Real-Time Columnstore & Apache Iceberg Lakehouse for PostgreSQL on Mooncake

Architecture

Create Columnstore Table

Procedure Signature

Hello World Tutorial

1. Enable the Extension

2. Create a Rowstore Table & Columnstore Mirror

Real-Time Operational Analytics (HTAP)

Key Advantages

Implementation Pattern: Write to Rowstore, Read from Columnstore

Manual Query Routing

Roadmap

PGSTY Maintenance Focus (v0.3.0+)

Historical Upstream Roadmap (v0.2)

v0.1 Architecture (Delta Lake & USING columnstore)

Archiving Large Postgres Tables

Architecture Pattern

Step-by-Step Walkthrough

Known Limitations

1. Read-Only Columnstore Mirror Tables

2. Primary Keys & REPLICA IDENTITY FULL

License

Summary by Repository

pg_mooncake & duckdb_mooncake — MIT License

Moonlink Ingestion Engine

Query Columnstore Table & Supported Query Shapes

How Queries Work

v0.1 Get Started

Historical Docker Image (Obsolete)

Building v0.1 from Source

Create the Extension

v0.1 Hello World (USING columnstore)

Create the Extension

Create a USING columnstore Table (v0.1)

Inserts, Updates, and Deletes

Transactions

Joins

Attribution

Upstream Creators

Repository Lineage

Documentation Provenance & Recovery

Download & Install

duckdb_mooncake & DuckDB 2.0 Integration

Role Inside pg_mooncake 0.3.0

Apache Iceberg v2 & Object Storage Configuration

The Postgres Lakehouse

1. Export Operational Postgres Tables as Apache Iceberg v2

v0.1 Cloud Storage (mooncake.create_secret)

v0.1 on Neon Postgres

Enable the Extension on Neon (v0.1)

Cloud Storage Requirement

Log Analytics with OpenTelemetry

Implementation Pattern

Query Your Data Lake in Postgres

Supported External Formats

Trademark Policy

Independence Statement

Third-Party Trademarks

v0.1 Reference Overview

v0.1 vs v0.2 & v0.3 Evolution

v0.1 Columnstore DDL (USING columnstore)

Create Columnstore Table (v0.1)

SBOM & Security Audit

Component & Submodule Pin Matrix (versions.json)

Core Rust Toolchain & Framework Pins

2026-11-05 Cargo Dependency Security Audit

v0.1 Querying Columnstore Outside Postgres

v0.1 GUC Configurations

v0.1 GUC Parameters

v0.1 Load Data

From PostgreSQL Heap Tables (v0.1)

From Parquet, CSV, or JSON Files (v0.1)

From Iceberg, Delta Lake, or HuggingFace (v0.1)

v0.1 Query External Files

v0.1 Supported Queries & Query Shapes

v0.1 Data Archiving

v0.1 Operational Analytics (Trigger + pg_cron Staging)

v0.1 Data Warehousing & dbt

Daily Aggregation Pipeline (v0.1)

Integration with dbt (v0.1)

PGSTY Maintenance Focus (`v0.3.0+`)

Historical Upstream Roadmap (`v0.2`)

2. Primary Keys & `REPLICA IDENTITY FULL`

`pg_mooncake` & `duckdb_mooncake` — MIT License

Building `v0.1` from Source

Create a `USING columnstore` Table (`v0.1`)

Role Inside `pg_mooncake 0.3.0`

Enable the Extension on Neon (`v0.1`)

Create Columnstore Table (`v0.1`)

Component & Submodule Pin Matrix (`versions.json`)

`v0.1` GUC Parameters

From PostgreSQL Heap Tables (`v0.1`)

From Parquet, CSV, or JSON Files (`v0.1`)

From Iceberg, Delta Lake, or HuggingFace (`v0.1`)

Daily Aggregation Pipeline (`v0.1`)

Integration with dbt (`v0.1`)