Snowflake Just Fixed Postgres CDC by Making It Push, Not Pull

There's a specific kind of 3 AM that every data engineer knows.

Your Postgres CDC pipeline is down. WAL is piling up, the disk is filling fast, and Debezium is lagging 30 minutes behind. You're manually truncating replication slots while praying nothing gets lost.

If that sounds familiar, Snowflake's latest engineering blog is worth your time.

They didn't patch the existing CDC stack. They threw it out and started over: a Postgres extension that pushes changes straight into object storage, turning replication into clockwork.

Why Pull-Based CDC Keeps Breaking

Postgres CDC, in its standard form, works like this: logical decoding turns WAL records into row-level operations, a replication slot exposes them over the network, and an external consumer—usually Debezium—pulls them into Kafka, then onward to the destination.

That last hop is where it falls apart. The consumer knows nothing about what's happening inside Postgres. Schema change? The consumer hits it mid-stream and explodes. Table dropped? The consumer keeps waiting for changes that will never come. Network partition or Postgres crash? The consumer can't tell them apart, so it either replays or skips.

And then there's snapshot alignment, exactly-once recovery, upsert performance degradation on large tables… any one of these can trigger the PagerDuty call.

Snowflake's insight: stop pulling. Make Postgres push.

A CDC Engine Inside Postgres Itself

They wrote snowflake_cdc, a Postgres extension that runs inside the database process. It continuously batches changes and writes them to Iceberg tables (compressed Parquet) on S3. Snowflake picks them up and applies them—no Kafka, no connectors, no external infrastructure.

Four stages, each running a beat behind the last:

1. Write: Data hits the table, WAL grows
2. Decode: Background workers translate past WAL into row-level changes
3. Capture: The extension packs those changes into Iceberg changelogs
4. Apply: Snowflake merges batches at exact transaction boundaries

None of them block each other. The decoder uses Postgres's "historic snapshot" mechanism, so it reads the catalog as it existed at write time—even if the table has since been altered or dropped.

And because the extension lives inside Postgres, it sees schema changes and DDL transactions first-hand. Nobody downstream has to guess.

Transactions Across Systems

Here's the part I keep coming back to.

Anyone who's built ETL or CDC pipelines knows the pain: transactions feel like cheating inside a single database, and the moment you cross a system boundary they're gone. Now you're hand-rolling idempotency, deduplication, ordering, and partial failure recovery—and every one of those picks the worst possible night to fail.

Snowflake wraps both ends in transactions:

• Postgres side: One transaction commits multiple change batches to the Iceberg changelogs. All or nothing.
• Snowflake side: One transaction merges those batches into the target tables, stopping precisely on a Postgres transaction boundary.

Foreign keys stay intact. Joins come out right. Nobody needs an upsert.

Speaking of upserts—plenty of CDC tools turn every operation into one because it's "simpler." You pay for that later:

• Intermediate states on the destination don't line up
• Every insert turns into an expensive columnar lookup against the target table
• Merging historical data with incremental changes gets close to impossible

Snowflake avoids upserts entirely because their replication is transaction-driven. An insert is an append. A delete is a delete. No matching, no ambiguity. Insert-heavy tables—usually your largest ones—replicate at near-zero overhead.

Live Views: Decoupling Apply Frequency from Query Latency

This one took me a second read.

Traditional CDC makes you pick: apply often and keep latency low, or apply rarely and keep queries fast. Merging isn't free, so you lose on one end or the other.

Snowflake's live views sidestep the choice by merging unapplied changelog data with target table data at query time. Filters and projections push down directly to the Parquet files. Sub-minute latency, even when applies are hours apart.

So apply frequency and query freshness stop being the same dial. Batch your merges for efficiency; queries stay current anyway.

What This Means for You

Most teams don't have Snowflake's engineering resources, and you probably shouldn't build your own CDC from scratch. Three of these ideas travel anyway:

1. Push beats pull. A source database that pushes its own changes is far steadier than an outsider reading its WAL. If your sync pipeline is flaky, move logic toward the source before throwing more hardware at it.
2. Transactions are the foundation. Wrapping cross-system operations in transactions eliminates an entire class of failures. Even a simple checkpoint-file-plus-merge is more reliable than stateless upserts.
3. Apply frequency is independent of query freshness. You can approximate live views with materialized views plus UNION on delta data. Batch your merges, query the union.

One thing you can do today:

Check your CDC pipeline's lag. If it's over 5 minutes, look at your WAL retention policy first—don't jump straight to adding machines. If lag is volatile, consider a minimal push layer on the source side. Even a simple pgoutput-to-S3 script removes a failure point compared to running Kafka in the middle.

---

✨ Originally drafted by DeepSeek, reviewed and polished by Claude.

Sources:

• How we pushed CDC into Postgres, and turned replication into clockwork - Snowflake Blog