MagicTools
Pitfall NotesBy CooconAugust 17, 202615 views7 min read

Zsh History Disappearing? A 10-Year-Old Bug, Fixed in 5.9.2

First, the conclusion, because incorrect answers to this problem are circulating too widely online:

  • Symptom: ~/.zsh_history is occasionally truncated, leaving only very old entries; years of new history disappear entirely; the file itself has no garbled text or partial lines.
  • Root Cause: The code path in Zsh that rewrites the history file upon exit, when interrupted by SIGINT (Ctrl+C), writes back only a half-read history file—a single-process bug, unrelated to multi-terminal concurrency, and unrelated to your setopt configuration.
  • Solution: Upgrade to Zsh 5.9.2 (released 2026-07-12). Note that 5.9.1 does not have this fix; configurations like INC_APPEND_HISTORY, APPEND_HISTORY, etc., cannot prevent it.

This bug has been lurking since a commit in 2015, living for 10 years in one of the world's most popular shells. It was tracked down by Michael Stapelberg (author of the i3 window manager), whose investigation record published in August 2026 is a textbook example of a debug article. Below, we break down the root cause and the investigation process, and finally, outline what you should do now.

Symptom: The History File "Goes Back in Time"

For years, Stapelberg encountered this occasionally: certain commands executed the day before couldn't be found with Ctrl+R; checking ~/.zsh_history revealed the file only contained very old entries, and the number of residual lines varied each time. His configuration was without suspicion:

HISTSIZE=4000
HISTFILE=~/.zsh_history
SAVEHIST=10000000
setopt HIST_IGNORE_DUPS
setopt INC_APPEND_HISTORY   # incremental append was already on — the bug still occurred
unsetopt SHARE_HISTORY

This is crucial: Numerous posts online (including some AI-generated "solutions") will tell you "just enabling INC_APPEND_HISTORY will fix it". He already had it enabled. This option solves a different problem (the classic "later-closed terminal overwrites earlier-closed" from the bash era), which is entirely separate from the bug in this article.

Investigation: From inotify to Patching Zsh to Crash

The investigation chain is worth reviewing for anyone involved in system troubleshooting, with tools escalating progressively:

  1. inotify: After monitoring the entire home directory, it was found that Zsh's write pattern on exit was "read old history → write .zsh_history.newrename() to overwrite the original file". However, inotify couldn't see which process was doing it.
  2. fatrace: Could see process name and PID—ruling out the multi-process overwrite conjecture—but couldn't see how many bytes each process read or wrote.
  3. bpftrace: A custom script traced file syscalls and aggregated read/write bytes per fd, logging continuously in the background. When the bug recurred, comparing logs caught the key difference: normal exits read the history file to EOF (read = 0); during the problematic incident, it did not read to EOF—Zsh only read about 11.5MB before stopping, then wrote this incomplete content as a new file, overwriting the original.
  4. Patching Zsh to cause a crash: Checks were added in the source code—if the written history was less than 50,000 lines, it would intentionally trigger a segmentation fault before the rename, using systemd-coredump to capture the scene. A few days later, a crash did occur, and the core dump's call stack and variable values (errflag = 2, lasthist.interrupted = 1) confirmed signal intervention.

Root Cause: The Exit Compaction Path Doesn't Check the Interrupt Flag

Upon exit, Zsh performs a history "compaction": entries are incrementally appended during the session, but on exit, it reads the entire history file into memory, applies size limits, and rewrites it all. The problem lies in two segments of code on this path:

  • The read loop in readhistfile has a signal check: upon receiving SIGINT, it breaks, having only read partway (this check was introduced in commit f1c702f in March 2015, released with Zsh 5.4).
  • However, savehistfile does not check this interrupt flag and proceeds to write the incomplete history to .zsh_history.new and rename it to overwrite the original file.

The trigger is a mundane habit: when closing a terminal after work, repeatedly pressing Ctrl+D, Ctrl+C, Ctrl+D, Ctrl+C... One Ctrl+D starts a zsh process exiting and rewriting history, and the immediately following Ctrl+C interrupts its reading. The larger the history file, the slower the rewrite, and the longer the window for this to happen.

Ironically, Zsh's default HIST_SAVE_BY_COPY (write to .new first, then atomic rename) is designed as a safeguard for "if the save process is interrupted"—but it cannot protect against "the data written into .new is already incomplete".

Fix: Available in 5.9.2, Not in 5.9.1

In March 2025, Stapelberg submitted a report with a minimal reproduction to the zsh-workers mailing list (bug #53412), and Bart Schaefer subsequently provided a fix (#53454): if the read is interrupted, it marks the save as failed, no longer writing out or overwriting the incomplete history.

There was a pitfall in the release timeline: the fix was merged into the main branch in April 2025 (commit bacc78ec, cherry-picked to the release branch as a6760226), but Zsh 5.9.1 (released 2026-05-31) missed it—due to an oversight by the release engineer, and it was only included after being pointed out in Zsh 5.9.2 (released 2026-07-12). Another detail: this fix is not listed in the NEWS file for 5.9.2 (which only lists new features); the written record is in the ChangeLog: 53454: Src/hist.c: fix interrupt handling in savehistfile().

Distribution Status (as of 2026-08-17)

Distribution / Package Manager Current zsh Version Contains Fix?
Arch Linux 5.9.2-1 ✅ (Followed on release day)
Homebrew 5.9.2
Fedora 45 / rawhide 5.9.2-3
Debian sid 5.9.2-1
Debian 13 (trixie) / 12 (bookworm) 5.9-8 / 5.9-4
Ubuntu (including 24.04 LTS and development releases) ≤ 5.9-8ubuntu3
Fedora 43 / 44 5.9-20 / 5.9-21 ❌ (Backport status unconfirmed)
macOS system built-in /bin/zsh Old version Recommended to check with zsh --version and install a new version via Homebrew

If zsh --version is below 5.9.2, either upgrade, wait for distribution backports, or refer to the mitigations below.

Before Upgrading, What Can You Do?

Configuration options cannot prevent this bug, but these steps are effective:

  1. Back up the history file. The author relied on daily backups for years for self-rescue. A single cron line can do it: cp ~/.zsh_history ~/.zsh_history.bak.$(date +%u) (rotating weekly, keeping 7 copies).
  2. Add a sentinel check. A method used by users on HN: in .zshrc, check the line count of ~/.zsh_history; if it falls below a threshold, sound a loud alarm—turning "discovering loss months later" into "discovering it the same day".
  3. Break the habit of repeatedly pressing Ctrl+C to close terminals, especially on machines where the history file is large (several MB or more).
  4. To completely bypass file-based history, you can switch to external history solutions that use SQLite storage, like atuin.

Another Pitfall: An Exported HISTFILE

The original article's appendix also describes an independent "history truncation" path, which many users on HN discovered applied to them only after reading:

Tools like Emacs TRAMP export HISTFILE (the environment variable is passed to child processes), while most people's .zshrc only assigns it, without unsetting the export. Thus, when you temporarily launch bash within zsh, bash inherits HISTFILE=~/.zsh_history and then truncates your zsh history according to bash's own HISTFILESIZE (often set to tens of thousands of lines by default on many systems). As one HN user put it, "running bash in zsh ruined three years of my history."

The countermeasure: actively unset the export in .zshrctypeset +x HISTFILE.

Frequently Asked Questions (FAQ)

Zsh history lost, can enabling INC_APPEND_HISTORY fix it?

It cannot fix the bug described in this article. INC_APPEND_HISTORY solves a different classic problem of multi-session append writing; this bug occurs in the history rewrite path upon exit, and the bug discoverer's configuration already had INC_APPEND_HISTORY enabled. The only root fix is upgrading Zsh to 5.9.2.

Did Zsh 5.9.1 fix the history loss bug?

No. The fix was merged into the main branch in April 2025, but was omitted when 5.9.1 (2026-05-31) was released, and only included in 5.9.2 (2026-07-12). Use zsh --version to confirm the version; 5.9.1 also requires an upgrade.

How do I determine if my history loss is from this bug or another cause?

Three characteristics point to this bug: the file is truncated leaving only very old entries, there is no garbled text or partial lines, and it occurs after closing a terminal (especially with a habit of repeatedly pressing Ctrl+C/Ctrl+D). If you often start bash within zsh, first check if HISTFILE is exported (export -p | grep HISTFILE)—that's a separate, easier pitfall to fall into.

The history file has already been truncated, can it be recovered?

Zsh itself has no recovery mechanism; the old file overwritten by rename is gone. Look for backups: Time Machine, daily backups, or other long-running sessions still open (they still have the complete history in memory, which can be exported with fc -W /tmp/rescue_history). This is why backing up the history file and adding a sentinel check are recommended.

References

Related Articles

Protobuf LSP Setup Guide: VS Code & Neovim with Buf

Buf ships a production-grade Protobuf LSP inside the buf CLI: go-to-definition, completion, find references, rename, and diagnostics that match buf lint. Setup paths for VS Code, Neovim (0.11 native + lspconfig), and JetBrains, a minimal two-file test project, and the known pitfalls—every command verified with buf v1.72.0.

developerAug 17, 20265 min
15

Stripe's $7B OpenRouter Deal: What It Means for Developers

Bloomberg reports Stripe has agreed to buy OpenRouter for over $7 billion—5x its valuation from three months ago. We cross-checked the facts: the deal's real status, why Stripe is buying its own billing customer, the Bridge→Tempo→Metronome→OpenRouter acquisition arc, BYOK rules, five alternatives, and the one thing to do now: abstract your routing layer.

ai-tutorialsAug 17, 20267 min
14

Stripe's $7B OpenRouter Deal: Buying the Right to Route

Stripe has finalized a deal to buy OpenRouter for more than $7 billion. The money isn't for the code that forwards requests — it's for the power to decide which provider serves them. But Stripe bought Amazon's position without Amazon's lock-in.

developerAug 17, 202610 min
64

Stripe's $7B OpenRouter Deal: Buying the Right to Route

Stripe has finalized a deal to buy OpenRouter for more than $7 billion. The money isn't for the code that forwards requests — it's for the power to decide which provider serves them. But Stripe bought Amazon's position without Amazon's lock-in.

llmai-infrastructure+3
ai-tutorialsAug 17, 202611 min
65

Published by MagicTools