Flow
Who writes resolv.conf
1Lease renews
DHCP says to use the home router for DNS.
2Router written in
The site resolvers get pushed out of the file.
3Backup lookup fails
The router doesn't know the backup server's name.
4Hook skips the write
Now DHCP leaves the file alone and the site resolvers stay.
Lease renewals on one of my hypervisors kept breaking its DNS, and the backups were what broke. Some backup jobs on that host couldn't look up the backup server's name, so they failed.
That host gets its internet address from DHCP. Along with the address, the lease says which DNS server to use: the home router. The router has never heard of my internal names. My own setup tells the host to use the site resolvers instead, and those do know the names. Both of them write the same file, resolv.conf. Each time the lease renewed, DHCP wrote the router back in, and each time my automation ran, it put the site resolvers back. A backup that started in between asked the router, got nothing, and gave up.
The funny bit is that the router was already on my list, last, as a backup. DHCP just pushed everything else off.
Last night the first attempt was a setting that tells the DHCP client to swap in my resolvers instead of the router. It didn't stick. This morning I added a small hook that runs every time the lease changes. The hook tells the DHCP client to skip writing resolv.conf, so the address still comes from DHCP while DNS stays under my control.
Why didn't the first setting work? I'd bet the DHCP client only reads that setting when it starts, and it never restarted. The hook gets read fresh every time, so it works right away. I left the old setting in, since it should kick in after a restart.
If you have a host with a DHCP uplink and your own DNS, check who writes resolv.conf. If the answer is more than one thing, pick one.
I still want to watch a backup run right after a real renewal before I call this done.