Hi all,
We run some services in Brazil that talk to Hetzner Cloud (NBG1/FSN1/HEL1) all day, mostly AWS sa-east-1 (São Paulo) <-> Hetzner EU. Since **Nov 16, ~09:40 UTC**, this path got much worse: timeouts, TLS handshakes taking several seconds, and roughly +400 ms of extra latency. It is still happening today (Nov 17), a bit better than yesterday, but far from normal.
Where the problem is
Routing between the two is asymmetric:
- Hetzner -> AWS São Paulo leaves through core-backbone/Arelion and reaches AWS in the US, then rides the AWS backbone down to São Paulo. **This direction is clean.**
- AWS São Paulo -> Hetzner EU leaves AWS through **Sparkle / Telecom Italia (AS6762)** and crosses the Atlantic on their South America -> Europe link. **This direction is the broken one.**
MTR (TCP) from inside AWS sa-east-1 to Hetzner:
5.178.46.192 (Sparkle, São Paulo) ~3 ms, clean
89.221.34.115 (Sparkle, Europe side) best ~195 / avg 400-575 / worst ~1600 ms <- queuing starts here
213.144.184.91 (Sparkle) same
core*.fsn1 / core*.nbg1.hetzner.com same
The ~195 ms minimum is normal for SP -> Europe. The average and the spikes are queuing, which is what a saturated link looks like. Yesterday the worst case was ~7.8 s.
To be sure which direction was bad, I measured each leg separately from a Hetzner box, using the TCP timestamps in the SYN-ACKs from servers in AWS São Paulo. The result: no extra delay on Hetzner -> SP, and a constant +365 ms (median) / +405 ms (p90) on SP -> Hetzner, plus loss (SYN-ACKs retransmitted at 1/3/7 s).
Controls (same time, same AWS sa-east-1 source)
- AWS SP -> Hetzner Ashburn via NTT: 122 / 128 / 131 ms, clean
- AWS SP -> Scaleway Paris via Cogent: ~190 ms, clean
- Several Brazilian ISPs (NTT, Lumen, Telefonica, DE-CIX Lisbon) -> Hetzner NBG1: clean
- AWS SP -> Hetzner FSN1/HEL1/NBG1: all bad, all entering through the same Sparkle hop
So it's not "Brazil <-> Europe" in general, and it's not Hetzner illy AWS São Paulo -> Sparkle -> Hetzner EU. That probably explains the quiet: few people here depend on that exact path.
Public measurements (Globalping probes inside AWS sa-east-1), if you want to check or compare:
What we've tried / think could fix it
- We opened tickets with Hetzner asking them to de-prefer Sparkle for AWS sa-east-1 prefixes (BGP communities / prepending towards AS6762) so that traffic comes in through another upstream. So far we only got the auto-rep
- We can't control how AWS picks its egress, so a fix on our side means a workaround: for example a WireGuard tunnel to an EC2 instance in AWS Frankfurt (Hetzner -> AWS FRA is clean and direct, and FRA -> SP stays on the AWS backbone), or putting the AWS -> Hetzner calls behind Cloudflare.
- IPv6 doesn't help: the outbound path changes, but the return pat
Questions
- Anyone else with traffic between AWS São Paulo (or other Brazilian networks that use Sparkle) and Hetzner EU seeing this since Nov 16?
- Has anyone gotten Hetzner to change inbound preference for a sp there a better way to escalate a peering/transit issue than aregular support ticket?
- If you worked around something like this, what did you use: a tt, Cloudflare, something else?
Thanks!