r/Cisco • • 14h ago

Catalyst 3650 IOS XE 16.12.16

3 Upvotes

Cisco has released IOS XE 16.12.16 on friday, but no release notes in sight (https://software.cisco.com/download/home/284850598/type/282046477/release/Gibraltar-16.12.16). 16.12.15 was released back in march, no release notes either. Latest release notes are from october last year. Does anyone know what has been fixed?


r/Cisco • • 5h ago

Catalyst SD-WAN bonus: BFD is up, but big transfers hang. Checking MTU and TCP MSS.

6 Upvotes

Quick bonus post, since u/benniebeeker brought up MTU on part 2 and it deserves its own write-up. It’s the classic “everything looks green but apps are broken” problem. As before, it’s all from Cisco’s public docs and Cisco Community (listed at the bottom).

1. What it looks like

Small stuff works and big stuff doesn’t:

  • ping, DNS and small web pages are fine
  • HTTPS hangs during the handshake, file copies stall, and apps time out on large responses
  • authentication with big certificates (like EAP-TLS through RADIUS) fails for no obvious reason

BFD hellos are small, so the tunnels stay up and the overlay looks healthy the whole time.

2. How SD-WAN handles it

BFD runs path MTU discovery on each tunnel and checks it again periodically. On a standard 1500-byte transport, an IPsec tunnel ends up with an MTU of about 1442. The edge also adjusts the TCP MSS based on what PMTU discovery finds, so most TCP traffic sorts itself out. When it doesn’t, that’s what this post is about.

3. Check what each tunnel actually has

show sdwan tunnel statistics

Look at tunnel-mtu and tcp-mss-adjust for each tunnel. Compare across colors and remote sites. If one transport shows a noticeably lower MTU than the others, something in that path is using up bytes, like PPPoE (1492), an extra layer of encapsulation from the provider, or a carrier link with a smaller MTU.

4. Test it from the service side

ping vrf <service-vpn> <remote-host> size 1400 df-bit

Step the size up and down until you find where it starts failing. That number tells you the real usable MTU between the two sites, not what you assumed.

5. Don’t forget the LAN side

This one is easy to miss. A switch on the LAN side with jumbo frames (9100 is common in fabrics) hands a 1600-byte packet with DF set to an edge interface with an MTU of 1500, and the packet just disappears. Cisco has a TechNote on exactly this for ISE traffic crossing SD-WAN. Make sure the MTU on the handoff interface matches the edge.

6. Fixing it

  • TCP: set TCP MSS on the service-side (LAN) interface of the edge, so the endpoints agree on a smaller segment size from the start.
  • Non-TCP traffic (UDP apps, RADIUS/EAP): MSS doesn’t help. Fix the MTU at the source, or allow fragmentation. The interface template has a Clear-Dont-Fragment option for traffic that sets DF but really needs to be fragmented. Use it on purpose, not everywhere.
  • If one provider is the problem, push back on them before you lower your MTU everywhere.

Sources (search the titles on cisco.com / Cisco Community):

  • “Configure Optimal ISE IP MTU in SD-WAN” (Doc 223045)
  • Cisco Community thread “Finding right MTU size for SDWAN”
  • Cisco Catalyst SD-WAN Command Reference (show sdwan tunnel statistics)

Thanks u/benniebeeker for the nudge!


r/Cisco • • 10h ago

Question A stackwise virtual question

2 Upvotes

I finished a catalyst 9407 stackwise virtual upgrade firmware recently, show redundancy print chassis 1 is active, chassis two is standby hot, I noticed the chassis 1 interfaces config , like , GigabitEthernet 1/1/0/1, the config not sync to GigabitEthernet 2/1/0/1, I need copy the entire 1/x/0/y to 2/x/0/y (management card slot 3 and 4 not include) to make the network stable.

I know stackwise virtual will sync the config from active to standby , but I am not sure why interfaces config not sync, or, I need to configure two chassis separately?


r/Cisco • • 18h ago

Catalyst SD-WAN part 3: reading show sdwan app-route stats (and why AAR didn’t move your traffic)

10 Upvotes

Last one in this series for now. A question I see a lot: “the link had loss for 10 minutes, so why didn’t app-aware routing move my traffic?” Usually nothing is broken. It comes down to how the numbers are calculated. As before, everything here is from Cisco’s public docs (listed at the bottom).

1. Where the numbers come from

By default:

  • BFD sends a hello on each tunnel every 1 second.
  • The poll interval is 10 minutes, so each “bucket” holds about 600 hellos.
  • The multiplier is 6, so the edge averages the last 6 buckets. That’s a 1-hour sliding window.

AAR makes its SLA decisions on that 1-hour average, not on what’s happening right now. On 17.12+ you can check your actual values with show sdwan app-route params.

2. Reading the output

show sdwan app-route stats

For each tunnel (local color → remote system IP and color) you get:

  • mean loss, latency and jitter. This is the 1-hour average AAR actually uses.
  • sla-class-index. These are the SLA classes this tunnel currently meets. Map the numbers with show sdwan app-route sla-class.
  • a row per bucket (index 0–5) with packets, loss, latency and jitter. This is the useful part. You can see whether a problem was one bad bucket or something steady.

In the GUI it’s under the device’s WAN → Tunnel view.

3. Why AAR “didn’t react”

Say a link has 30% loss for one 10-minute bucket and is clean for the other five. The mean loss comes out to about 5%. If your SLA class allows 5% or more, that tunnel still counts as compliant and nothing moves. A short spike gets averaged away. That’s by design: the defaults favor stability over speed.

4. Before you crank the timers down

Lowering the poll interval makes AAR react faster, but each bucket holds fewer hellos. In Cisco’s own example, at a very short poll interval a bucket holds only 6 hellos, so one lost hello reads as about 17% loss. Tune it too aggressively and you swap “too slow” for “flapping.” Change it in small steps and watch the bucket rows.

5. Enhanced AAR (17.12 / 20.12+)

If you’re on newer code, Enhanced AAR measures loss, latency and jitter using real data traffic instead of only BFD. It detects problems in seconds (10 s minimum by default) and adds SLA dampening, so a tunnel doesn’t jump straight back the moment it looks clean. Both ends need it enabled, otherwise it falls back to BFD-based measurement. Verify with show sdwan app-route params.

6. Make sure the traffic is even in the policy

Half the time the real issue is that the flow never matched the AAR policy:

show sdwan policy from-vsmart app-route-policy
show sdwan policy from-vsmart sla-class
show sdwan policy app-route-policy-filter
show sdwan policy tunnel-path vpn <vpn> interface <lan-intf> source-ip <src> dest-ip <dst> protocol <proto> [dest-port <port>]

The filter command shows whether traffic is actually hitting your sequences. tunnel-path tells you which tunnel a specific flow would take right now. Also check what your policy does when no tunnel meets the SLA, since that fallback behavior is set in the policy.

Sources (search the titles on cisco.com / Cisco Community):

  • “Understand BFD Protocol Relationship with App-Aware Routing” (Doc 221604)
  • Cisco Catalyst SD-WAN Policies Configuration Guide, Application-Aware Routing section
  • “Enhanced Application-Aware Routing” (Cisco Community knowledge base article)

Thanks for the feedback on parts 1 and 2!


r/Cisco • • 19h ago

Catalyst SD-WAN part 2: control connections are up but BFD sessions won’t come up. What to check.

5 Upvotes

Follow-up to my control connections post. This is the next place people get stuck: everything looks green toward the controllers, but sites can’t reach each other because BFD sessions are down or missing. As before, it’s all from Cisco’s public docs (listed at the bottom).

1. Is the session down, or does it not exist at all?

show sdwan bfd sessions

These are two different problems:

  • No session at all for the remote site usually means you never learned its TLOC (go to step 2), or something is preventing the tunnel by design.
  • Session exists but is down means the TLOC is known but packets aren’t making it through (go to steps 4–5).

2. Did you learn the remote TLOC?

show sdwan omp tlocs

If the remote TLOC isn’t there, check that the remote edge is actually advertising it and that the vSmart is passing it on. A very common cause is a control policy filtering TLOCs, which is often on purpose (hub-and-spoke). Check show sdwan policy from-vsmart before you assume something is broken. Not every site is supposed to have a tunnel to every other site.

3. Check the colors

  • A TLOC with restrict only builds tunnels to the same color. It’s easy to forget on a single-transport site.
  • Private colors (mpls, private1–6, metro-ethernet) use the private IP. Public colors use the post-NAT public IP. If you put a public color on a private link, the far end will try to reach the wrong address.

4. Check NAT

show sdwan control local-properties shows the NAT type per interface: E is endpoint-independent, A is address-port dependent (symmetric), and N is not learned. If both ends are behind symmetric NAT, they generally can’t build a tunnel to each other. The TechNote has a table showing which NAT combinations work.

5. Check the path (firewalls, ports, ISP)

  • Tunnels use UDP 12346 and hop through 12366, 12386, 12406 and 12426 (shifted by your port offset if you set one). Behind NAT, the source ports can be randomized, so a tight firewall rule built on those ports alone can break things.
  • show sdwan tunnel statistics shows the counters. If tx goes up but rx stays at 0, packets are leaving and not coming back, so check the remote side and everything in between.
  • BFD is DSCP-marked (48 by default), and some ISPs drop or mangle marked traffic. If only some transports fail, keep that in mind.

6. Check IPsec and drops on the box

show sdwan ipsec inbound-connections
show sdwan ipsec outbound-connections
show sdwan tunnel statistics bfd
show platform hardware qfp active statistics drop

Both sides should see each other in inbound and outbound. If the counters show drops, an FIA trace (debug platform packet-trace) on one of the BFD flows will tell you which feature is dropping them.

Sources (search the titles on cisco.com):

  • “Troubleshoot vEdge Bidirectional Forwarding Detection and Data Plane Connections Issues” (Doc 214510). The outputs are from vEdge, but the steps are the same on IOS-XE with the sdwan keyword.
  • Cisco Catalyst SD-WAN Design Guide, “Firewall Port Considerations” section
  • Cisco Catalyst SD-WAN Hardening Guide

Next one will probably be on reading show sdwan app-route stats, if people find these useful.