The Self-Reinforcing Flywheel Is Harder Than It Looks: 40 Days of an LLM Auto-Tuning Loop
The idea
"Dev Breakfast" (码农早餐) is our daily tech newsletter on WeChat. One issue goes out at 8 a.m. every day, written by an LLM pipeline. On September 11 I added a self-optimizing loop on top of it. The plan:
- Every day at 08:50, pull the official stats (reads, recommendation-feed reads, shares, follows, completion rate)
- Score the "writing version" that produced yesterday's issue and compare it to its parent: below 0.8× rolls back, 1× or above is kept
- An LLM reads the data and proposes changes to at most 3 of 25 typed knobs, which become the new version
- The knobs cover topic-selection gates, title and opening hints, length, humor density, cover style, theme color, and follow-prompt copy
The verdict is code, not the LLM. The hard floors (fact-checking, real source links, no invented first-person experience) live outside the knobs, so no version can touch them. Better data → better writing → better data. The left foot steps on the right foot and you climb.
Looking back after 40 days, the flywheel never completed a real turn.
The 40-day ledger
| Number | |
|---|---|
| Versions | 18 (12 proposed by the optimizer, 6 set manually) |
| Rolled back | 8, including 3 manual versions |
| Issues since the loop went live | 47, totaling 2,818 reads, 44 shares, 7 follows from the article |
| Issues with ≤ 50 reads | 28 (60%) |
| Share concentration | 21 of the 44 shares came from two issues (an eBPF agent cutting overhead by 90%, LiteLLM moving to a Rust core) |
Three details say more than the totals:
- v10 had exactly the same knobs as its parent v6. That day the optimizer produced no valid proposal and just reused the parent. It was rolled back anyway. That is an accidental A/A test: same configuration, two different days of articles, and the score gap crossed the rollback line.
- Cover style went in circles: dark → light (v3) → dark (v5) → flat (v9, rolled back) → light (v14, rolled back). The masthead tagline was proposed in v8 and v11 and rolled back both times. A human finally set it in v17.
- "Drop the long deep-dive sections" was set three times by a human. On 09-16, after reading 10 days of data, I cut them (v4). On 09-18 the optimizer rolled that back based on one issue's score. That issue ended at 387 reads, the highest since the loop launched. I set it again on 09-19 (v7), and it was rolled back again. The third attempt was v17 on 10-12.
Why it never turned
1. Not enough signal: noise is bigger than the knob effect
Since the loop went live, the average issue got 60 reads, with a 1.56% share rate and a 0.25% follow rate. A version that runs a full week (7 issues, about 420 reads) is expected to produce one follow. A "version score" built by comparing one follow with zero is a coin flip.
Run the sample size on share rate instead. To detect a 20% relative difference at the usual 80% power and 5% significance, each version needs roughly 25,000 reads. At 60 reads a day, that is more than 400 days per version.
At first we judged after 1–3 issues. On 10-12 we raised it to a minimum of 7. That cut the noise somewhat, but it also meant one verdict per week, or five or six independent experiments in 40 days. Nowhere near 400 days. Every threshold is a bad trade: judge fast and you chase noise, judge accurately and you are too slow to matter.
2. The knobs don't control the biggest driver
93% of this account's reads come from the platform's recommendation feed. On 09-07, a single issue ("write .gitignore in reverse") got 1,358 reads, 78% of all reads over those 10 days. What decides whether an issue gets recommended is mostly what topic happened to be available that day, and second, the title. Whether the cover is dark or light, or humor goes up a notch, are second-order effects.
The day's topic is not a knob. It comes from the outside news stream. If you treat the first-order variable as noise and optimize second-order variables, the optimizer just learns which days were lucky.
3. The loop can game its own metric
The original score was recommendations 0.4 + follow rate 0.25 + like rate 0.2 + comment rate 0.15. Starting 08-28, the newsletter already closed with a "pick a side in the comments" question. Once the loop was live, comment rate was part of the score, and one knob, the closing-question hint, was tagged as affecting comments. In v16 the optimizer proposed making that question more precise. Across roughly 40 issues with a pick-a-side question, we got one reader comment, and it didn't pick a side.
On 10-12 we read every comment. Reader comments came from three situations only: the title overclaimed, a detail was unclear, or an expert added or pushed back. Separately there were corrections we posted ourselves. One author correction, on 08-31, would count as a positive under the old formula: the article was wrong, and the score went up. At this scale you cannot prompt comments into existence. Putting them in the score only teaches the loop to optimize the prompt.
4. The environment shifts, and the loop can't see it
From 10-16 the recommendation feed fell off a cliff. 10-15 still had 29 recommendation reads; after that, 0–1 per issue. The version was v17 the whole time, the article structure didn't change, and there were no code commits to the publishing pipeline. Our read is that this is an account-level change (platform-side throttling, or the end of a new-account boost). The cause is still unconfirmed.
To the loop, this just looks like "the score crashed after some version." It blames whatever knobs were active, rolls back, proposes again, and diligently optimizes against a cliff that has nothing to do with knobs.
5. Humans and the loop fight
A human looks at structural conclusions across many issues ("extra articles in the same push don't add traffic," "long deep-dives drag completion down to 6–16%"). The loop looks at per-version scores. When they conflict, the side that runs automatically every morning at 08:50 always wins. We first added a lock: manual versions with fewer than 3 observed issues are not judged. Later this became a minimum of 7 for every version. In effect, that admits strategic decisions should not be overturned by a loop judging on a week of noise.
What actually worked
Over two months, every change that helped came from a human reading the data, not from knob A/B tests:
- 09-16: Read 10 days of daily data, stopped multi-article pushes and the long deep-dives (extra articles got 1–22 reads each, zero recommendations)
- 09-19: Found average reading time and a five-segment drop-off breakdown in the official stats. Readers' budget was fixed at 40–60 seconds regardless of length. Side items became headline-only and the lead story was cut to 400–500 characters. Among issues with at least 10 reads, average completion went from about 23% before to about 29% afterwards (through 10-15). Topics changed over the same period, so treat this as correlation only
- 10-12: Read every comment, dropped the closing question, took comments out of the score, and made correction comments a penalty
What these three have in common: each one lays dozens of issues side by side and looks at structure, instead of comparing two version means. The conclusions are qualitative ("readers have a 40-second budget"), and knobs were just how they got implemented.
Takeaways
- Compute the sample size before you automate an experiment loop. Use your real baseline rates to get "samples needed to detect an X% difference," then divide by daily traffic. If the answer is a year, don't automate the verdict.
- Check first that your knobs control the biggest driver. If most of the variance comes from inputs you don't control (topics, platform traffic), account for them first (stratify, or use them as covariates). Otherwise the optimizer only learns which days were lucky.
- Score only on metrics the loop can't game. Metrics that a call to action can pull directly (comments, likes) become shortcuts for the optimizer. Our newsletter has since dropped every follow prompt and relies on the content alone.
- Give the loop an "environment changed" brake. When a global metric falls off a cliff (say, recommendation traffic goes to zero across the board), pause verdicts and alert a human instead of rolling back more knobs.
- Keep strategic decisions out of the loop. A loop is good at small adjustments within a direction that is already set, provided there is enough signal. The direction itself should be a human call based on structure, and the loop should not be able to overturn it.
The loop is still running. Every day it pulls data, keeps the books, and makes proposals. Our conclusion from this review: until there is enough signal, it works best as a diligent data clerk and proposer. The "left foot on right foot" step, judging and upgrading itself, belongs back with a human. The first push on any flywheel is whether the content is actually worth reading.
