The 16-Year-Old SQLite Bug That Took Tailscale Half a Year to Catch

It's 3 AM and your phone lights up: the production database is corrupt, and PRAGMA integrity_check is dumping red everywhere. You check git log. You check recent changes. Nothing. It's not your fault—but your users don't care about whose fault it is.

Tailscale's engineers lived through that 3 AM call, over and over, for six months. The culprit they eventually pinned down: a data race that had been hiding in SQLite for at least 16 years. The SQLite team gave it a name—the WAL-Reset bug.

19 corruptions, six months, no common thread

It started last August. Tailscale's control plane is split into shards; each shard runs one SQLite database, opened by exactly one Go process. That's the textbook single-writer setup SQLite was designed for, and it had run without incident since 2022.

Then a pipeline reading their S3 backups flagged an error. They ran integrity_check against the backup and found real corruption. SQLite corruption is possible, but it's the kind of thing that shouldn't happen in normal operation. They repaired it. They found no cause.

Then it happened again. And again. Nineteen separate corruption incidents in six months. The maddening part: there was no common factor. Not the shard, not the customer, not a feature, not the time of day, not load. There was nothing to reproduce.

The write that vanished into thin air

With no way to reproduce the bug, they had to catch it red-handed—deploying passive forensic telemetry in production and waiting, the last thing any engineer wants to do. Meanwhile they built a transaction-logging pipeline: every SQL statement that modified the database got streamed to a separate log, so they could replay transactions against a known-good backup instead of rolling back.

That pipeline never got used for its intended rescue. Instead, it handed them the clue. On two occasions, the replay didn't apply cleanly. A write that had been committed was invisible to later reads. Data had disappeared without a single error.

In a single-writer SQLite database, that should be impossible—unless something went wrong at the exact point where the write happened.

WAL, checkpoints, and a 16-year-old race

SQLite stores data as pages. When you update the database, new pages don't go straight into the main file. They go into a sidecar file called the WAL—the Write-Ahead Log, the "log before data" rule made concrete. The WAL can't grow forever, so periodically its pages get copied back into the main database file. That's a checkpoint.

Here's the analogy: the WAL is the pass-through window in a kitchen, and a checkpoint is the waiter carrying plates from the window to the tables. Normally, pick up ten plates, the window has ten fewer. But if the window holds ten plates and the waiter reports moving twenty, something else just cleared the window behind your back.

That's exactly what Tailscale saw: during checkpoints, SQLite reported copying more pages than the WAL actually contained. What the SQLite team found was a rare data race between a checkpoint and a write transaction. If a write lands at a specific instant during a checkpoint, the checkpoint mistakenly believes some pages have already been copied back—when they haven't. Those pages never make it to the database file, and that data is gone forever. Meanwhile, indexes that reference those missing pages do get written, so the file is structurally consistent and silently wrong.

SQLite estimates the bug had been there for at least 16 years. It survived that long because the trigger conditions are so narrow the maintainers had to write code to deliberately force it in testing. The fix adds a check that detects when the WAL has been reset by another thread during a checkpoint.

Why Tailscale, of all people

This is the part worth remembering: SQLite didn't fail because it's unreliable. Tailscale got bitten because they walked off the well-trodden path.

Most people let SQLite decide when to checkpoint, and never think about it. Tailscale took manual control of checkpointing and ran it aggressively, so they could take fast, consistent backups. The SQLite maintainers were blunt: that's precisely why they hit a race other users don't. A bug with a rare trigger will eventually find the people who poke the trigger most often.

Tailscale's own framing is the lesson: running boring technology is fine. Running boring technology in a non-standard way is the risk. The default paths are bomb-proof because millions of users have already hammered them. The moment you deviate from the main road for some special requirement, you become the only person testing that road.

Don't mistake "should be fine" for a backup strategy

The tempting takeaway here is "SQLite has bugs, avoid it." Wrong. Every database has bugs—PostgreSQL unearthed a 15-year-old WAL-related defect back in 2018. The only difference is whether you hit that particular code path. A database's reliability promise is always conditional on "you haven't triggered that condition yet."

What should actually make you nervous is the sentence we all mutter to ourselves: "This database has run fine for a decade, it should be fine." Even Tailscale's fix wasn't clean. They upgraded to the patched release, and promptly triggered a false alarm—the 3.52.0 release changed text-to-float rounding, which broke their virtual-column timestamp indexes and produced "fake" corruption. The SQLite team had to withdraw 3.52.0 and ship 3.51.3 with only the WAL-Reset fix. Even the "upgrade to fix it" step found a new way to bite.

So don't just stare at the bug. Three concrete things you can do today:

1. Check your version. Look at the SQLite version in your project. If it's in the affected range, upgrade (the stable line is now 3.53.x, the one with the self-healing index). If you don't upgrade, at least know what you're betting on.
2. Run a recovery drill. Not after the data is gone. Pull your latest backup right now, run integrity_check and a replay, and prove your restore path actually works instead of "should work in theory."
3. Audit your non-standard usage. Are you manually overriding a default somewhere—checkpoint, vacuum, connection pool, isolation level? Every time you step off the default path, you sign yourself up for the testing nobody else is doing.

A bug that hid for 16 years isn't nonexistent. It just hadn't met anyone yet. Before it meets you, drill the things you can drill.

✨ Drafted by DeepSeek, reviewed and polished by Claude.

Sources:

• Tailscale: How we tracked down a 16-year-old SQLite bug
• SQLite: The WAL-Reset Bug
• SQLite: How To Corrupt An SQLite Database File