Dev Breakfast · 2026-09-18

AWS admits that some data in the Middle East facilities struck by Iran cannot be recovered—it's not a delay, it's unrecoverable, and the report doesn't specify which availability zones, how many customers, or if there is compensation. Cloud vendors' default backups are never contractual promises; you must first calculate RPO and retention policies yourself.

AWS says some data in Middle East facilities can't be recovered: backup is harder than you think

First, let's look at the facts: reports indicate that AWS has stated that some data cannot be recovered from facilities in the Middle East that were attacked by Iran. The original report only had a headline and a brief overview, without specifying which availability zones, how many customers are involved, whether there is a compensation plan, or if there was any rollback or remedy. The idea that 'cloud vendors back up for you' is never the default promise in the contract.

Why this is worth a coder taking a look—because it's not about geopolitics, but about the most underestimated part of every system: backups. A detailed article breaks it down: backups are not just 'storing another copy', but a whole set of trade-offs. You need to answer at least two questions first—RPO (how much data loss you can tolerate) and retention policy. The author's reference: critical financial institutions might have an RPO of less than 30 seconds, while many small businesses have 24 hours or more, and some have no disaster recovery plan at all. If RPO is set to 24 hours, that's 7 snapshots a week, 30 a month, 365 a year—without rotation, storage won't hold.

So we have rotation: keep recent ones dense, sparse ones far out, daily backups for 14 days, weekly for 7 weeks, monthly for 12 months—this is GFS rotation plus snapshot-based backups. There's another layer: file changes follow a fat-tailed distribution—most files don't change for long, very few change daily. So you shouldn't store duplicate copies, but deduplicate, using hard links to reference existing files. A file is stored only once on disk, and each snapshot points to it. Tools like rsync-based ones (e.g., rsnapshot) use this, with the benefit that rotation only deletes directory entries, not the files themselves, and saves bandwidth—when you need to transfer to a second machine in the cloud, every megabyte saved is a bill.

The real difficulty of backups is not in 'copying', but in 'recovering'—an unverified backup is equivalent to no backup. In the article, the string of adjectives gets longer and longer: incremental, deduplicated, GFS-rotated, snapshot-based... each adjective adds complexity. The author ran into issues when practicing with their 10 Docker containers: many containers create files owned by root, but cronjobs run with default users, leading to backup failures in logs; databases often load data into memory and then flush to disk in batches, so directly copying files doesn't yield a consistent database. These pitfalls have nothing to do with which cloud you use.

Comparing makes it clearer: RAID 1 does mirroring, not backups—ransomware encryption, accidental file deletion, or scripts filling the disk with zeros, the mirror will sync these exactly to the second disk. Mirroring handles disk failures, snapshots handle human errors. These two are often confused as one, until the day something happens and you realize you only have the former. As for cloud vendors, how much can be recovered and how long it takes depends on how they internally partition and replicate—this part is invisible to you and not covered by the SLA, which usually compensates for availability, not your data.

For coders, there are three specific areas to address: confirm what the RPO for critical data actually is (don't write 'the higher the better', write a number); hook backup script failure alerts into channels you actually check, not a log file no one opens; for databases, don't use file-level copying, use logical exports or their own snapshot mechanisms. As for a second copy across regions or cloud vendors—the cost is there, and whether it's worth it depends on your own calculation based on data importance.

💡 Chef's take: The most heart-wrenching sentence in that article is 'There are two types of people in the world: those who have experienced catastrophic data loss, and those who are about to'—and in backup rotation strategies, the easiest step to skip is 'recovery drills', with the reason always being no time, and the cost when something happens is all the time.

Sources:

• AWS Says It Can't Restore Some Data From Mideast Facilities Struck by Iran (WSJ)
• Backups Aren't Simple (filipovski.net)

Nvidia allows Rust to directly write GPU kernels, with two paths in parallel

Nvidia announced support for writing GPU kernels natively in Rust, with two official tracks. The original report only had a headline and this sentence, without elaborating on how the two tracks are divided or which scenarios they cover. For Rust programmers, this means an additional official entry point into the long C++-dominated GPU space; but whether it's worth migrating now depends on them clarifying the boundaries of the two tracks, available toolchains, and limitations. With new things, first understand what has actually changed before deciding to jump on board.

Sources:

• Introducing CUDA Rust: Two Tracks for Writing GPU Kernels

4B model-generated query plans are 81% faster than Postgres

Someone trained a small 4B model specifically for generating query plans, with the author claiming it's 81% faster than Postgres's built-in planner. This directly relates to you: query planning has always been the most experience-dependent and hardest-to-tune part of databases. If a small model can consistently outperform an optimizer that's been running for over twenty years, then the entry point for 'SQL tuning' will change. But my first reaction to such results is to ask about the data: 81% on which benchmark and which query patterns? Queries like TPC-H's standardized ones and online ones with lots of ORM-generated SQL are entirely different in difficulty. The original report only had a conclusion, without detailing test conditions, so note this number but don't rush to believe it.

Sources:

• Training a 4B model to produce 81% faster query plans than Postgres

Spend $9 to have Gemini train its own replacement

Someone posted a record: spending $9 to have Gemini train its own replacement. The original report only had a headline, without elaborating on where the training data came from, the scale of the replacement, or which step the $9 was spent on. So the only thing certain now is that someone has reduced the cost of 'using a large model to build a cheaper version of itself' to single-digit dollars. For coders, this is worth clicking to see how it was done, but don't rush to copy it. Model capability is determined by data, not by how much money was spent.

Sources:

• p-s-v (74)

HarnessTax test: How much does the harness affect coding agent performance?

With the same model, how much can performance differ when run with a different harness? Someone created the HarnessTax project to quantify the impact of the harness (the scaffolding around the model: tool calls, context management, retry logic, etc.) on coding agent performance. This directly relates to you—you might frequently adjust prompts and change models, but may not realize that what truly creates the gap could be the outer shell. The original report only had a title and project page, without specific numbers; whether it's worth a detailed look depends on seeing what tasks it was tested on.

Sources:

• HarnessTax: How Much Does the Harness Matter for Coding Agents?

DeepSeek v4.1 Flash compresses KV cache, ternary models are a different story

We previously discussed the capabilities of DeepSeek v4.1 Flash, and today's piece turns to another aspect: for the same model, KV cache compression has been pushed to new limits. KV cache is the most memory-intensive part of long-context inference; being able to compress it means more context can fit on the same card, with lower single inference cost—for those deploying themselves, this is more practical than benchmarks.

Additionally, there's a paper on ternary LLMs breaking the 1.58-bit barrier, which is a different matter from the above; don't confuse them. Ternary quantization saves weights, KV cache compression saves intermediate inference states, and both paths are moving towards 'running more cheaply'. Whether ability drops after compression, and by how much, depends on the data tested; just the word 'limits' in the title doesn't explain the issue.

Sources:

• Breaking the 1.58-bit Barrier for Ternary LLMs
• DeepSeek-v4.1 Flash: Pushing the Limits of KV Cache Compression

480-point hot post: One page of copyable programming tricks

A Hacker News post about 'small tricks' got 480 points, with a title so plain it's only three words, linking to a personal blog, and no summary in the body. Reaching 480 points on this platform usually means it's not about new frameworks, but about the small pitfalls you encounter daily and keep repeating—naming, boundaries, early returns, and such. It's worth clicking to see if there's one you can directly copy into your current code.

Sources:

• Small programming tricks

GitLab.com to change rate limits, only this sentence was said

GitLab.com's rate limits are changing, with the official blog only stating 'Rate limits on GitLab.com are changing', without elaborating on how they will change, when they will take effect, or which APIs will be affected. For those who use GitLab daily as a code hosting and pipeline base, this is worth a look, but currently, the only certainty is 'changing'. My attitude is simple: first wait for them to clarify the numbers and scope, then decide whether to adjust calling methods in advance.

Sources:

• Rate limits on GitLab.com are changing

---

What is the RPO set for your system—within 30 seconds, or a gamble at 24 hours? See you tomorrow morning at 8 o'clock.

This issue selected 8 items from 65 pieces of information in the past 24 hours on X / Hacker News / GitHub Trending (written hourly throughout the day, fact-checked, and compiled in the morning). Content is LLM-assisted, with original source links for each item; please cross-verify for important decisions.