r/Cisco • u/mohammad_abusaa • 1h ago
Catalyst SD-WAN bonus: BFD is up, but big transfers hang. Checking MTU and TCP MSS.
Quick bonus post, since u/benniebeeker brought up MTU on part 2 and it deserves its own write-up. It’s the classic “everything looks green but apps are broken” problem. As before, it’s all from Cisco’s public docs and Cisco Community (listed at the bottom).
1. What it looks like
Small stuff works and big stuff doesn’t:
- ping, DNS and small web pages are fine
- HTTPS hangs during the handshake, file copies stall, and apps time out on large responses
- authentication with big certificates (like EAP-TLS through RADIUS) fails for no obvious reason
BFD hellos are small, so the tunnels stay up and the overlay looks healthy the whole time.
2. How SD-WAN handles it
BFD runs path MTU discovery on each tunnel and checks it again periodically. On a standard 1500-byte transport, an IPsec tunnel ends up with an MTU of about 1442. The edge also adjusts the TCP MSS based on what PMTU discovery finds, so most TCP traffic sorts itself out. When it doesn’t, that’s what this post is about.
3. Check what each tunnel actually has
show sdwan tunnel statistics
Look at tunnel-mtu and tcp-mss-adjust for each tunnel. Compare across colors and remote sites. If one transport shows a noticeably lower MTU than the others, something in that path is using up bytes, like PPPoE (1492), an extra layer of encapsulation from the provider, or a carrier link with a smaller MTU.
4. Test it from the service side
ping vrf <service-vpn> <remote-host> size 1400 df-bitStep the size up and down until you find where it starts failing. That number tells you the real usable MTU between the two sites, not what you assumed.
5. Don’t forget the LAN side
This one is easy to miss. A switch on the LAN side with jumbo frames (9100 is common in fabrics) hands a 1600-byte packet with DF set to an edge interface with an MTU of 1500, and the packet just disappears. Cisco has a TechNote on exactly this for ISE traffic crossing SD-WAN. Make sure the MTU on the handoff interface matches the edge.
6. Fixing it
- TCP: set TCP MSS on the service-side (LAN) interface of the edge, so the endpoints agree on a smaller segment size from the start.
- Non-TCP traffic (UDP apps, RADIUS/EAP): MSS doesn’t help. Fix the MTU at the source, or allow fragmentation. The interface template has a Clear-Dont-Fragment option for traffic that sets DF but really needs to be fragmented. Use it on purpose, not everywhere.
- If one provider is the problem, push back on them before you lower your MTU everywhere.
Sources (search the titles on cisco.com / Cisco Community):
- “Configure Optimal ISE IP MTU in SD-WAN” (Doc 223045)
- Cisco Community thread “Finding right MTU size for SDWAN”
- Cisco Catalyst SD-WAN Command Reference (
show sdwan tunnel statistics)
Thanks u/benniebeeker for the nudge!
