v0.1 Archive: Welcome to pg_mooncake (Delta Lake) on Mooncake

v0.1 Architecture (Delta Lake & USING columnstore)

Mon, 01 Jan 0001 00:00:00 +0000

Legacy pg_mooncake v0.1 Archive (Delta Lake Engine)

This page documents pg_mooncake v0.1, which used direct CREATE TABLE ... USING columnstore DDL and Delta Lake storage. Starting in v0.2 and PGSTY’s v0.3.0 release, pg_mooncake uses moonlink real-time columnstore mirrors (CALL mooncake.create_table + USING mooncake) and Apache Iceberg v2. For current documentation, see Introduction and v0.1 vs v0.2 & v0.3.

In pg_mooncake v0.1, table metadata is stored inside PostgreSQL catalogs for transactional consistency, while columnstore data is written as Parquet files with Delta Lake metadata on local disk or object storage.

v0.1 Get Started

Mon, 01 Jan 0001 00:00:00 +0000

Legacy pg_mooncake v0.1 Archive (Delta Lake Engine)

Historical Docker Image (Obsolete)

docker pull mooncakelabs/pg_mooncake
docker run --name mooncake-demo -e POSTGRES_HOST_AUTH_METHOD=trust -p 5432:5432 -d mooncakelabs/pg_mooncake
docker run -it --rm --link mooncake-demo:postgres mooncakelabs/pg_mooncake psql -h postgres -U postgres

Building `v0.1` from Source

git submodule update --init --recursive
make release -j$(nproc)
make install

Create the Extension

CREATE EXTENSION pg_mooncake;

v0.1 Hello World (USING columnstore)

Mon, 01 Jan 0001 00:00:00 +0000

Legacy pg_mooncake v0.1 Archive (Delta Lake Engine)

Create the Extension

CREATE EXTENSION pg_mooncake;

Create a `USING columnstore` Table (`v0.1`)

CREATE TABLE user_activity (
    user_id BIGINT,
    activity_type TEXT,
    activity_timestamp TIMESTAMP,
    duration INT
) USING columnstore;

Inserts, Updates, and Deletes

INSERT INTO user_activity VALUES
    (1, 'login', '2024-01-01 08:00:00', 120),
    (2, 'page_view', '2024-01-01 08:05:00', 30),
    (3, 'logout', '2024-01-01 08:30:00', 60),
    (4, 'error', '2024-01-01 08:13:00', 60);

UPDATE user_activity SET duration = 180 WHERE user_id = 1;
DELETE FROM user_activity WHERE user_id = 2;

Transactions

BEGIN;
INSERT INTO user_activity VALUES
    (5, 'login', '2024-01-02 10:00:00', 200),
    (6, 'login', '2024-01-02 10:30:00', 90);
ROLLBACK;
SELECT * FROM user_activity;

Joins

-- Between two columnstore tables
SELECT * FROM user_activity ua
JOIN user_activity ua2 ON ua.user_id = ua2.user_id;

-- Between a regular PostgreSQL table and a columnstore table
SELECT * FROM users JOIN user_activity ON users.user_id = user_activity.user_id;

v0.1 Cloud Storage (mooncake.create_secret)

Mon, 01 Jan 0001 00:00:00 +0000

Legacy pg_mooncake v0.1 Archive (Delta Lake Engine)

Current v0.3.0 Cloud Storage Setup

In v0.2 and v0.3.0, credentials are inserted into duckdb.secrets rather than mooncake.create_secret. See Iceberg & Object Storage.

v0.1 on Neon Postgres

Mon, 01 Jan 0001 00:00:00 +0000

Legacy pg_mooncake v0.1 Archive (Delta Lake Engine)

pg_mooncake v0.1 was offered as an experimental extension on Neon.

Enable the Extension on Neon (`v0.1`)

SET neon.allow_unstable_extensions = 'true';
CREATE EXTENSION pg_mooncake;

Cloud Storage Requirement

Because Neon does not allow extensions to persist columnstore files on local compute disk, using pg_mooncake v0.1 on Neon required configuring an external S3 or Cloudflare R2 bucket as described in v0.1 Cloud Storage, and disabling Neon’s scale-to-zero feature to preserve local metadata caches.

v0.1 Reference Overview

Mon, 01 Jan 0001 00:00:00 +0000

Legacy pg_mooncake v0.1 Archive (Delta Lake Engine)

v0.1 Columnstore DDL (USING columnstore)

Mon, 01 Jan 0001 00:00:00 +0000

Legacy pg_mooncake v0.1 Archive (Delta Lake Engine)

Create Columnstore Table (`v0.1`)

In v0.1, columnstore tables were created directly with the USING columnstore clause:

v0.1 Querying Columnstore Outside Postgres

Mon, 01 Jan 0001 00:00:00 +0000

Legacy pg_mooncake v0.1 Archive (Delta Lake Engine)

Current v0.3.0 Metadata Function

In v0.2 and v0.3.0, use SELECT * FROM mooncake.list_tables(); to inspect table LSNs and Apache Iceberg v2 warehouse locations.

v0.1 GUC Configurations

Mon, 01 Jan 0001 00:00:00 +0000

Legacy pg_mooncake v0.1 Archive (Delta Lake Engine)

`v0.1` GUC Parameters

-- Default cloud storage bucket
ALTER DATABASE <database_name> SET mooncake.default_bucket = 's3://my-bucket';

-- Memory allocated to DuckDB (v0.1.1+, default 4GB)
ALTER DATABASE <database_name> SET mooncake.maximum_memory = '6GB';
CALL mooncake.reset_duckdb();

-- Number of threads for DuckDB (v0.1.1+, default -1 = all cores)
ALTER DATABASE <database_name> SET mooncake.maximum_threads = -1;

-- Local write cache (default true)
ALTER DATABASE <database_name> SET mooncake.enable_local_cache = false;

-- Memory cache for Parquet metadata (default true)
ALTER DATABASE <database_name> SET mooncake.enable_memory_metadata_cache = false;

v0.1 Load Data

Mon, 01 Jan 0001 00:00:00 +0000

Legacy pg_mooncake v0.1 Archive (Delta Lake Engine)

From PostgreSQL Heap Tables (`v0.1`)

INSERT INTO users_columnstore
SELECT * FROM users_rowstore;

From Parquet, CSV, or JSON Files (`v0.1`)

-- Using COPY
COPY users_columnstore FROM 's3://bucket/file.parquet';
COPY users_columnstore FROM 's3://bucket/folder/*.parquet';
COPY users_columnstore FROM 's3://bucket/data.csv' WITH (FORMAT 'csv', HEADER true);

-- Using INSERT with mooncake.read_parquet
INSERT INTO users_columnstore
SELECT * FROM mooncake.read_parquet('s3://bucket/file.parquet') AS (
    id INT,
    name TEXT,
    created_at TIMESTAMP
);

From Iceberg, Delta Lake, or HuggingFace (`v0.1`)

-- Iceberg
INSERT INTO users_columnstore
SELECT * FROM iceberg_scan('s3://bucket/table') AS (id INT, name TEXT, created_at TIMESTAMP);

-- Delta Lake
INSERT INTO users_columnstore
SELECT * FROM delta_scan('s3://bucket/table') AS (id INT, name TEXT, created_at TIMESTAMP);

-- HuggingFace
COPY users_columnstore FROM 'hf://datasets/OpenLeecher/lmsys_chat_1m_clean/data/train-00000-of-00004.parquet';

v0.1 Query External Files

Mon, 01 Jan 0001 00:00:00 +0000

Legacy pg_mooncake v0.1 Archive (Delta Lake Engine)

Current v0.3.0 External File Readers

In v0.2 and v0.3.0, use duckdb.read_parquet, duckdb.read_csv, duckdb.iceberg_scan, and duckdb.delta_scan as documented in Query Your Data Lake in Postgres.

v0.1 Supported Queries & Query Shapes

Mon, 01 Jan 0001 00:00:00 +0000

Legacy pg_mooncake v0.1 Archive (Delta Lake Engine)

For the updated v0.3.0 guide on query shapes and DuckDB 2.0 execution, see Query Columnstore & Query Shapes.

v0.1 Data Archiving

Mon, 01 Jan 0001 00:00:00 +0000

Legacy pg_mooncake v0.1 Archive (Delta Lake Engine)

Current v0.3.0 Partition Archiving Pattern

In v0.2 and v0.3.0, you no longer need manual INSERT INTO ... DELETE FROM batch scripts—simply mirror a partitioned table with CALL mooncake.create_table and DROP old partitions. See Archiving Large Postgres Tables.

v0.1 Operational Analytics (Trigger + pg_cron Staging)

Mon, 01 Jan 0001 00:00:00 +0000

Legacy pg_mooncake v0.1 Archive (Delta Lake Engine)

Replaced by mooncake.create_table in v0.2 / v0.3.0

In v0.1, syncing a rowstore table to a USING columnstore table required triggers, a staging table, and a pg_cron job. In v0.2 and v0.3.0, all of this is replaced by a single CALL mooncake.create_table('orders_analytics', 'orders');. See Real-Time Operational Analytics.

v0.1 Data Warehousing & dbt

Mon, 01 Jan 0001 00:00:00 +0000

Legacy pg_mooncake v0.1 Archive (Delta Lake Engine)

Daily Aggregation Pipeline (`v0.1`)

CREATE TABLE daily_sales_summary (
    date DATE,
    product_id INT,
    category_id INT,
    region_id INT,
    total_quantity INT,
    total_revenue DECIMAL(18, 2)
) USING columnstore;

Integration with dbt (`v0.1`)

In v0.1, dbt models could target USING columnstore by setting default_table_access_method in an on-run-start hook:

v0.1 Archive: Welcome to pg_mooncake (Delta Lake) on Mooncake

v0.1 Architecture (Delta Lake & USING columnstore)

v0.1 Get Started

Historical Docker Image (Obsolete)

Building v0.1 from Source

Create the Extension

v0.1 Hello World (USING columnstore)

Create the Extension

Create a USING columnstore Table (v0.1)

Inserts, Updates, and Deletes

Transactions

Joins

v0.1 Cloud Storage (mooncake.create_secret)

v0.1 on Neon Postgres

Enable the Extension on Neon (v0.1)

Cloud Storage Requirement

v0.1 Reference Overview

v0.1 Columnstore DDL (USING columnstore)

Create Columnstore Table (v0.1)

v0.1 Querying Columnstore Outside Postgres

v0.1 GUC Configurations

v0.1 GUC Parameters

v0.1 Load Data

From PostgreSQL Heap Tables (v0.1)

From Parquet, CSV, or JSON Files (v0.1)

From Iceberg, Delta Lake, or HuggingFace (v0.1)

v0.1 Query External Files

v0.1 Supported Queries & Query Shapes

v0.1 Data Archiving

v0.1 Operational Analytics (Trigger + pg_cron Staging)

v0.1 Data Warehousing & dbt

Daily Aggregation Pipeline (v0.1)

Integration with dbt (v0.1)

Building `v0.1` from Source

Create a `USING columnstore` Table (`v0.1`)

Enable the Extension on Neon (`v0.1`)

Create Columnstore Table (`v0.1`)

`v0.1` GUC Parameters

From PostgreSQL Heap Tables (`v0.1`)

From Parquet, CSV, or JSON Files (`v0.1`)

From Iceberg, Delta Lake, or HuggingFace (`v0.1`)

Daily Aggregation Pipeline (`v0.1`)

Integration with dbt (`v0.1`)