Service Up, Ports Open, Certs Valid, VPN Dead for 4 Hours: Tailscale Took Over DNS and Left the Proxy Box With No Upstream
Symptoms
Two DMIT VPSes in the same Los Angeles datacenter, same setup, both running sing-box as proxy nodes (VLESS-REALITY on tcp/443, Hysteria2 on udp/443). On the evening of September 16 both nodes on the primary box (call it dmit-usa) started timing out in every client. The second box (dmit-usa-eb) kept working.
The first round of checks came back all green:
$ ssh dmit-usa 'systemctl is-active sing-box nginx; ss -lntup | grep -E ":443 |:8444|:8445"'
active
active
udp UNCONN 0 0 *:443 *:* users:(("sing-box",...))
tcp LISTEN 0 4096 127.0.0.1:8444 ... users:(("sing-box",...))
tcp LISTEN 0 511 127.0.0.1:8445 ... users:(("nginx",...))
tcp LISTEN 0 511 0.0.0.0:443 ... users:(("nginx",...))
nginx -t passed. The Let's Encrypt certificate used by Hysteria2 had 81 days left. The last sing-box restart in systemd was two days earlier, part of a certificate cleanup, with no abnormal exits since.
The only thing out of place was the log directory:
-rw-r--r-- 1 sing-box sing-box 13710190 Sep 16 23:15 sing-box.log # today, 13 MB
-rw-r--r-- 1 sing-box sing-box 7329135 Sep 16 00:30 sing-box.log.1 # yesterday, 7 MB
-rw-r--r-- 1 sing-box sing-box 50237 Sep 7 00:15 sing-box.log.10.gz # a normal day, tens of KB compressed
Tens of kilobytes a day normally, more than ten megabytes a day for the last two. The service was not dead. It was spamming something.
Diagnosis: three commands from "the log got big" to the root cause
Step 1: what is it spamming?
Normalize and count the error lines in the last 2 MB of the log:
tail -c 2000000 /worker/logs/sing-box/sing-box.log \
| grep -oE "(ERROR|failed|handshake)[^\"]{0,80}" \
| sed -E 's/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+(:[0-9]+)?//g' \
| sort | uniq -c | sort -rn | head
5307 ERROR inbound/vless[vless-in]: process connection from
2826 handshake: REALITY: failed to dial dest: lookup www.apple.com: (exchange6: SERVFAIL
2499 handshake: REALITY: failed to dial dest: lookup www.apple.com: (exchange4: SERVFAIL
40 ERROR connection: open connection to oauthaccountmanager.googleapis.com:443 ...
Two things here:
- The REALITY handshake fails because
lookup www.apple.comreturns SERVFAIL. REALITY works by having the server actually dial the borrowed target site (www.apple.com here) during the handshake and use its TLS handshake as camouflage. If the target cannot be resolved, the handshake cannot proceed, and every VLESS connection is rejected. - The
open connection to xxx:443failures further down are Hysteria2. The tunnel is already up, but outbound connections to the destination hostnames fail to resolve in exactly the same way. That is why both protocols broke at once and in the same way: clients could reach the port, but not a single page would load.
Step 2: who is answering DNS?
$ cat /etc/resolv.conf
# resolv.conf(5) file generated by tailscale
# For more info, see https://tailscale.com/s/resolvconf-overwrite
# DO NOT EDIT THIS FILE BY HAND -- CHANGES WILL BE OVERWRITTEN
nameserver 100.100.100.100
nameserver fd7a:115c:a1e0::53
search tailaca6af.ts.net
$ dig +short www.apple.com # system DNS → 100.100.100.100
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL
$ dig +short @1.1.1.1 www.apple.com # ask the public resolver directly
www-apple-com.v.aaplimg.com.
The system resolver is Tailscale's MagicDNS stub at 100.100.100.100, and it answers SERVFAIL. Bypass it and ask 1.1.1.1 directly and everything works. The network was fine. The thing that was broken was tailscaled, the process forwarding DNS on this box.
Step 3: why did tailscaled stop forwarding?
$ journalctl -u tailscaled --since today | grep -i dns | grep -v RATELIMIT | tail
Sep 16 09:52:30 tailscaled: trample: resolv.conf changed from what we expected. did some other program interfere? current contents: "nameserver 1.1.1.1\nnameserver 1.0.0.1\n"
Sep 16 09:52:30 tailscaled: dns: Resolvercfg: {Routes:{.:[1.1.1.1 1.0.0.1] ts.net.:[199.247.155.53 ...]} ...}
Sep 16 18:58:20 tailscaled: trample: resolv.conf changed from what we expected. did some other program interfere? current contents: ""
Sep 16 18:58:20 tailscaled: dns: Resolvercfg: {Routes:{.:[] ts.net.:[199.247.155.53 ...]} ...}
Sep 16 23:16:21 tailscaled: dns: resolver: forward: no upstream resolvers set, returning SERVFAIL
Three log lines tell the whole story:
| Time | What resolv.conf was rewritten to by someone else | Public upstream after tailscaled re-took control (Routes:{.:[...]}) |
Result |
|---|---|---|---|
| 09:52:30 | 1.1.1.1 / 1.0.0.1 |
[1.1.1.1 1.0.0.1] |
Fine |
| 18:58:20 | an empty file | [] |
Every public query from then on: no upstream resolvers set, returning SERVFAIL |
And at both of those timestamps the system journal shows the same event:
Sep 16 18:58:20 dhclient[533]: DHCPREQUEST for 179.253.232.64 on eth0 to 193.41.250.250 port 67
Sep 16 18:58:20 dhclient[533]: DHCPACK of 179.253.232.64 from 193.41.250.250
Sep 16 18:58:20 dhclient[533]: bound to 179.253.232.64 -- renewal in 36300 seconds.
eth0 is configured by DHCP, and dhclient rewrites /etc/resolv.conf on every lease renewal, roughly every 10 hours. The first time, it wrote a complete file, tailscaled re-took control and recorded those two addresses as the "original system upstream", and nothing happened. The second time, tailscaled read an empty file. The most likely explanation is that it hit the window between dhclient truncating the file and writing the new content. The upstream list became empty, and from then on every query not under ts.net got an immediate SERVFAIL. All 34,000 SERVFAIL lines in the sing-box log are after 18:58, not one before.
Why the other box was fine
dmit-usa-eb had Tailscale installed earlier (September 5) with an identical configuration: accept-dns on, resolv.conf managed by tailscaled, no global nameservers in the tailnet. It was simply lucky. tailscaled had not yet read an empty file during a dhclient renewal. Same landmine, not yet stepped on. This is also where the "the control box is fine" clue is most misleading. Two boxes with the same config, one working and one not, pushes your intuition toward "something was done to the broken one specifically". In reality both were in a dangerous state.
Root cause in one sentence
Tailscale on Linux defaults to --accept-dns=true, which points /etc/resolv.conf at its own stub resolver 100.100.100.100. Its ability to forward public domains depends entirely on the upstream it read from resolv.conf at the moment it took over. If the tailnet has no global nameservers configured, that is the only source. Any program that rewrites resolv.conf (dhclient, systemd-resolved, NetworkManager, cloud-init) can cause it to read an incomplete file, after which DNS on the whole machine fails silently. Tailscale's own docs call this a "DNS fight" (the link in the log is tailscale.com/s/dns-fight).
And sing-box's REALITY inbound has no DNS configuration of its own. It relies entirely on the system resolver. So a change that looks completely unrelated to the proxy (installing a mesh VPN tool) took the proxy down.
Fixes: pick one, the first is recommended
Option A (fixes the cause, both boxes in one go): In the Tailscale admin console, DNS → Global nameservers, add 1.1.1.1 and 1.0.0.1. Once the config is pushed, tailscaled's public upstream comes from the control plane and no longer depends on whatever it happens to read from resolv.conf. This is how we fixed it. No service restart was needed; after the console change, dig www.apple.com on both boxes returned NOERROR and the sing-box log stopped erroring. To verify:
tailscale dns status | sed -n '/Resolvers/,/Split DNS/p'
# 1.1.1.1 / 1.0.0.1 should be listed under Resolvers
Option B (single-box stopgap, no admin console needed): Tell Tailscale to leave DNS alone, then write resolv.conf yourself.
tailscale set --accept-dns=false
printf 'nameserver 1.1.1.1\nnameserver 1.0.0.1\n' > /etc/resolv.conf
The cost is that this box can no longer resolve *.ts.net MagicDNS names. Reaching peers by their 100.x address still works. A proxy box almost never needs MagicDNS, so the cost is negligible.
Option C (decouple sing-box from system DNS): Add a standalone dns section to the sing-box config pointing at something like 1.1.1.1. Then even if system DNS breaks again, the REALITY handshake and outbound resolution keep working. It does not fix the Tailscale problem, but it shrinks the blast radius of the proxy service. Good as a supplement to A or B.
Postmortem: why this took much longer than it should have
The actual order of investigation was: service status → ports → certificates → whether the deploy hook from the certificate cleanup two days earlier had restarted sing-box into a broken state → logs. The first four all came back clean, and "we touched certificates two days ago" looked so much like a root cause that it nearly consumed all the time.
The real lessons are three:
- "Service is running" is not "service works".
systemctl is-activeonly says the process is alive. For proxy services, what you should verify is the core dependencies: can it resolve names, can it connect outbound. - A sudden jump in log size is the earliest and cheapest signal. Tens of KB a day turning into tens of MB tells you "it is spamming errors" before any monitoring does. When you see that, go straight to the log. Do not detour.
- The most recent "unrelated" change is the most suspicious. Installing Tailscale looks like it has nothing to do with a VPN, but it rewrote resolv.conf. Anything on a server that touches the network stack should be treated as a change to every service that depends on the network.
Keeping Claude Code out of this hole: rules worth putting in CLAUDE.md
Claude Code installed the Tailscale, and Claude Code found the bug. It diagnosed correctly, but nothing stopped it during the install. Afterwards we added the following to the CLAUDE.md and tasks/lessons.md in the VPS repo. The idea is to hard-code "what to verify after installing" rather than hope the model thinks of it every time.
1. An acceptance checklist for network components
On proxy boxes (machines running sing-box), after installing anything that takes over
/etc/resolv.conf (Tailscale / systemd-resolved / NetworkManager / cloud-init), verify:
- dig www.apple.com via system DNS returns NOERROR
- count sing-box log ERROR lines before and after; the delta must be 0
- cat /etc/resolv.conf to confirm who manages it; if tailscaled, the tailnet must already
have global nameservers configured
Not passing verification means the install is not done.
2. Turn implicit dependencies into explicit facts
The sing-box config has no standalone dns section. The REALITY handshake must resolve
www.apple.com first. If system DNS on the server breaks, the VPN goes down entirely
(service/ports/certs all look fine; systemctl will not show it).
The model does not automatically know project-level facts like "the REALITY handshake depends on DNS". Write one sentence and DNS lands in its top three checks next time.
3. The order of investigation can be a rule too
Troubleshooting VPN outages: (1) systemctl / ports (2) dig www.apple.com
(3) sing-box log size and normalized error counts (4) any network-related change in the
last 48h, including ones that look unrelated. If log size is abnormally large, jump to (3).
4. Make it remember that "the control box is fine" is not evidence of exclusion
When two identically configured machines differ, one working and one not, first check
whether the working one simply has not been triggered yet. Do not assume it is safe.
None of these four are generic best practices. They are concrete facts and concrete actions for this machine and this architecture. That is exactly the most valuable content in a CLAUDE.md: project-private constraints the model cannot derive from general knowledge, where getting it wrong is expensive.
Related
- Claude Code's Auto Mode Judges a Model With a Model: this investigation ran straight into that error at the start. It took more than ten retries to get SSH execution back, and in the end a human ran the commands and pasted the output.