Back to Feed
article 2m read

The backup handshake that hung on one oversized packet

A hypervisor on a borrowed cellular uplink could not finish TLS to the backup server because big packets vanished without a word, and the first backup that did run flooded the uplink and dropped the WireGuard mesh.

Article DepthPractice

Flow

Steps for a small, slow uplink

What was done here, in order.

  1. 1Spot the hang

    A TLS hello that never finishes points at dropped big packets.

  2. 2Measure the real size

    Find the largest packet that passes instead of trusting defaults.

  3. 3Clamp and shrink the tunnel

    Clamp TCP at the router, then lower the tunnel MTU for everything.

  4. 4Shape the upload

    Set a shaper just under the real upload speed so keepalives get a turn.

  5. 5Guard the backup

    Check which gateway is upstream before a big job runs.

One hypervisor could not finish a TLS handshake with my backup server. It connected, sent its hello, and then nothing happened.

That host sits behind its own router on a borrowed cellular uplink until it moves to a data centre. It talks to my other sites over a WireGuard mesh.

When a TLS hello hangs like that, suspect a path-MTU black hole. Some hop drops big packets and never tells the sender to make them smaller. Small packets get through, so the connection opens. The hello is often the first big one, so that's where it sticks.

The agent first set an MSS clamp on the router's LAN side at the normal WireGuard value. That tells TCP to use smaller pieces. It still hung. A test showed packets well short of normal full size passed and bigger ones failed, so the clamp went down to fit.

Then a backup ran and filled the slow uplink. The routers send small keepalive packets to check the mesh links. Those got stuck behind the backup, the routers decided the links were dead, and the mesh dropped. Adding a shaper on the router's WAN side, set just under the real upload speed, keeps that queue on the router where every flow gets a turn. I also turned the nightly backup off for that link, even though it's the only backup the host has.

Next, the tunnel size itself became a per-site setting. A clamp only helps TCP. A smaller tunnel MTU covers everything.

Later the backup was back on with a check. Before each run it pings the internet with a TTL of two, which makes the second hop answer. If that hop is the cellular box, the run stops. At the data centre it just runs.

If you try the same thing, know the gaps. No answer counts as safe, and it only checks once. I still don't know why the cellular path is so small.

Keep reading

Pick up a connected idea or branch into a nearby one.

3 paths forward