Service Up, Ports Open, Certs Valid, VPN Dead for 4 Hours: Tailscale Took Over DNS and Left the Proxy Box With No Upstream

Symptoms

Two DMIT VPSes in the same Los Angeles datacenter, same setup, both running sing-box as proxy nodes (VLESS-REALITY on tcp/443, Hysteria2 on udp/443). On the evening of September 16 both nodes on the primary box (call it dmit-usa) started timing out in every client. The second box (dmit-usa-eb) kept working.

The first round of checks came back all green:

$ ssh dmit-usa 'systemctl is-active sing-box nginx; ss -lntup | grep -E ":443 |:8444|:8445"'
active
active
udp   UNCONN 0 0   *:443            *:*   users:(("sing-box",...))
tcp   LISTEN 0 4096 127.0.0.1:8444  ...   users:(("sing-box",...))
tcp   LISTEN 0 511  127.0.0.1:8445  ...   users:(("nginx",...))
tcp   LISTEN 0 511  0.0.0.0:443     ...   users:(("nginx",...))

nginx -t passed. The Let's Encrypt certificate used by Hysteria2 had 81 days left. The last sing-box restart in systemd was two days earlier, part of a certificate cleanup, with no abnormal exits since.

The only thing out of place was the log directory:

-rw-r--r-- 1 sing-box sing-box 13710190 Sep 16 23:15 sing-box.log      # today, 13 MB
-rw-r--r-- 1 sing-box sing-box  7329135 Sep 16 00:30 sing-box.log.1    # yesterday, 7 MB
-rw-r--r-- 1 sing-box sing-box    50237 Sep  7 00:15 sing-box.log.10.gz # a normal day, tens of KB compressed

Tens of kilobytes a day normally, more than ten megabytes a day for the last two. The service was not dead. It was spamming something.

Diagnosis: three commands from "the log got big" to the root cause

Step 1: what is it spamming?

Normalize and count the error lines in the last 2 MB of the log:

tail -c 2000000 /worker/logs/sing-box/sing-box.log \
  | grep -oE "(ERROR|failed|handshake)[^\"]{0,80}" \
  | sed -E 's/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+(:[0-9]+)?//g' \
  | sort | uniq -c | sort -rn | head

5307 ERROR  inbound/vless[vless-in]: process connection from
2826 handshake: REALITY: failed to dial dest: lookup www.apple.com: (exchange6: SERVFAIL
2499 handshake: REALITY: failed to dial dest: lookup www.apple.com: (exchange4: SERVFAIL
  40 ERROR  connection: open connection to oauthaccountmanager.googleapis.com:443 ...

Two things here:

• The REALITY handshake fails because lookup www.apple.com returns SERVFAIL. REALITY works by having the server actually dial the borrowed target site (www.apple.com here) during the handshake and use its TLS handshake as camouflage. If the target cannot be resolved, the handshake cannot proceed, and every VLESS connection is rejected.
• The open connection to xxx:443 failures further down are Hysteria2. The tunnel is already up, but outbound connections to the destination hostnames fail to resolve in exactly the same way. That is why both protocols broke at once and in the same way: clients could reach the port, but not a single page would load.

Step 2: who is answering DNS?

$ cat /etc/resolv.conf
# resolv.conf(5) file generated by tailscale
# For more info, see https://tailscale.com/s/resolvconf-overwrite
# DO NOT EDIT THIS FILE BY HAND -- CHANGES WILL BE OVERWRITTEN

nameserver 100.100.100.100
nameserver fd7a:115c:a1e0::53
search tailaca6af.ts.net

$ dig +short www.apple.com          # system DNS → 100.100.100.100
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL
$ dig +short @1.1.1.1 www.apple.com # ask the public resolver directly
www-apple-com.v.aaplimg.com.

The system resolver is Tailscale's MagicDNS stub at 100.100.100.100, and it answers SERVFAIL. Bypass it and ask 1.1.1.1 directly and everything works. The network was fine. The thing that was broken was tailscaled, the process forwarding DNS on this box.

Step 3: why did tailscaled stop forwarding?

$ journalctl -u tailscaled --since today | grep -i dns | grep -v RATELIMIT | tail
Sep 16 09:52:30 tailscaled: trample: resolv.conf changed from what we expected. did some other program interfere? current contents: "nameserver 1.1.1.1\nnameserver 1.0.0.1\n"
Sep 16 09:52:30 tailscaled: dns: Resolvercfg: {Routes:{.:[1.1.1.1 1.0.0.1] ts.net.:[199.247.155.53 ...]} ...}
Sep 16 18:58:20 tailscaled: trample: resolv.conf changed from what we expected. did some other program interfere? current contents: ""
Sep 16 18:58:20 tailscaled: dns: Resolvercfg: {Routes:{.:[] ts.net.:[199.247.155.53 ...]} ...}
Sep 16 23:16:21 tailscaled: dns: resolver: forward: no upstream resolvers set, returning SERVFAIL

Three log lines tell the whole story:

Time	What resolv.conf was rewritten to by someone else	Public upstream after tailscaled re-took control (Routes:{.:[...]})	Result
09:52:30	1.1.1.1 / 1.0.0.1	[1.1.1.1 1.0.0.1]	Fine
18:58:20	an empty file	[]	Every public query from then on: no upstream resolvers set, returning SERVFAIL

And at both of those timestamps the system journal shows the same event:

Sep 16 18:58:20 dhclient[533]: DHCPREQUEST for 179.253.232.64 on eth0 to 193.41.250.250 port 67
Sep 16 18:58:20 dhclient[533]: DHCPACK of 179.253.232.64 from 193.41.250.250
Sep 16 18:58:20 dhclient[533]: bound to 179.253.232.64 -- renewal in 36300 seconds.

eth0 is configured by DHCP, and dhclient rewrites /etc/resolv.conf on every lease renewal, roughly every 10 hours. The first time, it wrote a complete file, tailscaled re-took control and recorded those two addresses as the "original system upstream", and nothing happened. The second time, tailscaled read an empty file. The most likely explanation is that it hit the window between dhclient truncating the file and writing the new content. The upstream list became empty, and from then on every query not under ts.net got an immediate SERVFAIL. All 34,000 SERVFAIL lines in the sing-box log are after 18:58, not one before.

Why the other box was fine

dmit-usa-eb had Tailscale installed earlier (September 5) with an identical configuration: accept-dns on, resolv.conf managed by tailscaled, no global nameservers in the tailnet. It was simply lucky. tailscaled had not yet read an empty file during a dhclient renewal. Same landmine, not yet stepped on. This is also where the "the control box is fine" clue is most misleading. Two boxes with the same config, one working and one not, pushes your intuition toward "something was done to the broken one specifically". In reality both were in a dangerous state.

Root cause in one sentence

Tailscale on Linux defaults to --accept-dns=true, which points /etc/resolv.conf at its own stub resolver 100.100.100.100. Its ability to forward public domains depends entirely on the upstream it read from resolv.conf at the moment it took over. If the tailnet has no global nameservers configured, that is the only source. Any program that rewrites resolv.conf (dhclient, systemd-resolved, NetworkManager, cloud-init) can cause it to read an incomplete file, after which DNS on the whole machine fails silently. Tailscale's own docs call this a "DNS fight" (the link in the log is tailscale.com/s/dns-fight).

And sing-box's REALITY inbound has no DNS configuration of its own. It relies entirely on the system resolver. So a change that looks completely unrelated to the proxy (installing a mesh VPN tool) took the proxy down.

Fixes: pick one, the first is recommended

Option A (fixes the cause, both boxes in one go): In the Tailscale admin console, DNS → Global nameservers, add 1.1.1.1 and 1.0.0.1. Once the config is pushed, tailscaled's public upstream comes from the control plane and no longer depends on whatever it happens to read from resolv.conf. This is how we fixed it. No service restart was needed; after the console change, dig www.apple.com on both boxes returned NOERROR and the sing-box log stopped erroring. To verify:

tailscale dns status | sed -n '/Resolvers/,/Split DNS/p'
# 1.1.1.1 / 1.0.0.1 should be listed under Resolvers

Option B (single-box stopgap, no admin console needed): Tell Tailscale to leave DNS alone, then write resolv.conf yourself.

tailscale set --accept-dns=false
printf 'nameserver 1.1.1.1\nnameserver 1.0.0.1\n' > /etc/resolv.conf

The cost is that this box can no longer resolve *.ts.net MagicDNS names. Reaching peers by their 100.x address still works. A proxy box almost never needs MagicDNS, so the cost is negligible.

Option C (decouple sing-box from system DNS): Add a standalone dns section to the sing-box config pointing at something like 1.1.1.1. Then even if system DNS breaks again, the REALITY handshake and outbound resolution keep working. It does not fix the Tailscale problem, but it shrinks the blast radius of the proxy service. Good as a supplement to A or B.

Postmortem: why this took much longer than it should have

The actual order of investigation was: service status → ports → certificates → whether the deploy hook from the certificate cleanup two days earlier had restarted sing-box into a broken state → logs. The first four all came back clean, and "we touched certificates two days ago" looked so much like a root cause that it nearly consumed all the time.

The real lessons are three:

1. "Service is running" is not "service works". systemctl is-active only says the process is alive. For proxy services, what you should verify is the core dependencies: can it resolve names, can it connect outbound.
2. A sudden jump in log size is the earliest and cheapest signal. Tens of KB a day turning into tens of MB tells you "it is spamming errors" before any monitoring does. When you see that, go straight to the log. Do not detour.
3. The most recent "unrelated" change is the most suspicious. Installing Tailscale looks like it has nothing to do with a VPN, but it rewrote resolv.conf. Anything on a server that touches the network stack should be treated as a change to every service that depends on the network.

Keeping Claude Code out of this hole: rules worth putting in CLAUDE.md

Claude Code installed the Tailscale, and Claude Code found the bug. It diagnosed correctly, but nothing stopped it during the install. Afterwards we added the following to the CLAUDE.md and tasks/lessons.md in the VPS repo. The idea is to hard-code "what to verify after installing" rather than hope the model thinks of it every time.

1. An acceptance checklist for network components

On proxy boxes (machines running sing-box), after installing anything that takes over
/etc/resolv.conf (Tailscale / systemd-resolved / NetworkManager / cloud-init), verify:
- dig www.apple.com via system DNS returns NOERROR
- count sing-box log ERROR lines before and after; the delta must be 0
- cat /etc/resolv.conf to confirm who manages it; if tailscaled, the tailnet must already
  have global nameservers configured
Not passing verification means the install is not done.

2. Turn implicit dependencies into explicit facts

The sing-box config has no standalone dns section. The REALITY handshake must resolve
www.apple.com first. If system DNS on the server breaks, the VPN goes down entirely
(service/ports/certs all look fine; systemctl will not show it).

The model does not automatically know project-level facts like "the REALITY handshake depends on DNS". Write one sentence and DNS lands in its top three checks next time.

3. The order of investigation can be a rule too

Troubleshooting VPN outages: (1) systemctl / ports (2) dig www.apple.com
(3) sing-box log size and normalized error counts (4) any network-related change in the
last 48h, including ones that look unrelated. If log size is abnormally large, jump to (3).

4. Make it remember that "the control box is fine" is not evidence of exclusion

When two identically configured machines differ, one working and one not, first check
whether the working one simply has not been triggered yet. Do not assume it is safe.

None of these four are generic best practices. They are concrete facts and concrete actions for this machine and this architecture. That is exactly the most valuable content in a CLAUDE.md: project-private constraints the model cannot derive from general knowledge, where getting it wrong is expensive.

Related

• Claude Code's Auto Mode Judges a Model With a Model: this investigation ran straight into that error at the start. It took more than ten retries to get SSH execution back, and in the end a human ran the commands and pasted the output.