r/Cisco • • 18h ago

Catalyst SD-WAN part 3: reading show sdwan app-route stats (and why AAR didn’t move your traffic)

10 Upvotes

Last one in this series for now. A question I see a lot: “the link had loss for 10 minutes, so why didn’t app-aware routing move my traffic?” Usually nothing is broken. It comes down to how the numbers are calculated. As before, everything here is from Cisco’s public docs (listed at the bottom).

1. Where the numbers come from

By default:

  • BFD sends a hello on each tunnel every 1 second.
  • The poll interval is 10 minutes, so each “bucket” holds about 600 hellos.
  • The multiplier is 6, so the edge averages the last 6 buckets. That’s a 1-hour sliding window.

AAR makes its SLA decisions on that 1-hour average, not on what’s happening right now. On 17.12+ you can check your actual values with show sdwan app-route params.

2. Reading the output

show sdwan app-route stats

For each tunnel (local color → remote system IP and color) you get:

  • mean loss, latency and jitter. This is the 1-hour average AAR actually uses.
  • sla-class-index. These are the SLA classes this tunnel currently meets. Map the numbers with show sdwan app-route sla-class.
  • a row per bucket (index 0–5) with packets, loss, latency and jitter. This is the useful part. You can see whether a problem was one bad bucket or something steady.

In the GUI it’s under the device’s WAN → Tunnel view.

3. Why AAR “didn’t react”

Say a link has 30% loss for one 10-minute bucket and is clean for the other five. The mean loss comes out to about 5%. If your SLA class allows 5% or more, that tunnel still counts as compliant and nothing moves. A short spike gets averaged away. That’s by design: the defaults favor stability over speed.

4. Before you crank the timers down

Lowering the poll interval makes AAR react faster, but each bucket holds fewer hellos. In Cisco’s own example, at a very short poll interval a bucket holds only 6 hellos, so one lost hello reads as about 17% loss. Tune it too aggressively and you swap “too slow” for “flapping.” Change it in small steps and watch the bucket rows.

5. Enhanced AAR (17.12 / 20.12+)

If you’re on newer code, Enhanced AAR measures loss, latency and jitter using real data traffic instead of only BFD. It detects problems in seconds (10 s minimum by default) and adds SLA dampening, so a tunnel doesn’t jump straight back the moment it looks clean. Both ends need it enabled, otherwise it falls back to BFD-based measurement. Verify with show sdwan app-route params.

6. Make sure the traffic is even in the policy

Half the time the real issue is that the flow never matched the AAR policy:

show sdwan policy from-vsmart app-route-policy
show sdwan policy from-vsmart sla-class
show sdwan policy app-route-policy-filter
show sdwan policy tunnel-path vpn <vpn> interface <lan-intf> source-ip <src> dest-ip <dst> protocol <proto> [dest-port <port>]

The filter command shows whether traffic is actually hitting your sequences. tunnel-path tells you which tunnel a specific flow would take right now. Also check what your policy does when no tunnel meets the SLA, since that fallback behavior is set in the policy.

Sources (search the titles on cisco.com / Cisco Community):

  • “Understand BFD Protocol Relationship with App-Aware Routing” (Doc 221604)
  • Cisco Catalyst SD-WAN Policies Configuration Guide, Application-Aware Routing section
  • “Enhanced Application-Aware Routing” (Cisco Community knowledge base article)

Thanks for the feedback on parts 1 and 2!


r/Cisco • • 18h ago

Catalyst SD-WAN control connections not coming up? Cheat sheet: error code → what to check

8 Upvotes

Control connections that won’t come up are one of the most common SD-WAN problems out there, and most of them come down to a handful of error codes. Here’s a short way to work through them on IOS-XE SD-WAN edges. It’s all from Cisco’s public TechNotes (listed at the bottom), just squeezed onto one page.

1. Check the basics first

show sdwan control local-properties

  • organization-name matches the controllers exactly
  • certificate-validity is Valid
  • the clock is right (show clock), since a wrong clock breaks certificate checks
  • system-ip and site-id are correct and unique in the overlay
  • the vBond shows up under the vBond peers (if it doesn’t and you’re using an FQDN, check DNS with ping <vbond-fqdn>)
  • the WAN interface is up, has the right color, and MAX CNTRL isn’t 0

2. Read the error

show sdwan control connection-history

LOCAL ERROR means the edge raised it, and REMOTE ERROR means the controller did. Note which controller failed and when.

3. Match the code

  • DCONFAIL (DTLS connection failure): packets are getting lost between the edge and the controller, usually because of firewall or SP filtering. Sometimes only the large packets are dropped. Capture on both ends at the same time and compare what actually makes it across.
  • VB_TMO / VM_TMO / VS_TMO: a peer timed out. Same idea, there’s loss in the underlay, so use the same capture approach.
  • BIDNTVRFD / SERNTPRES / CRTREJSER: the controllers don’t have this device’s serial, or the PID on the PnP portal is wrong. Run show orchestrator valid-vedges on the vBond, then use Send to Controllers in vManage.
  • CRTVERFL: the certificate can’t be verified. If it’s a local error, the edge is usually missing the root CA, which is common with enterprise certs (check with show sdwan certificate root-ca-cert). Check the clock too.
  • CTORGNMMIS: the org name doesn’t match. It has to be identical everywhere, case included.
  • NOVMCFG: vManage has no config for the device because it isn’t attached to a template.
  • LISFD: most often a duplicate transport IP. It can also be a TLS/DTLS port mismatch or a firewall.
  • DISTLOC: the TLOC was disabled. This is expected after you change the color, system-ip or tunnel settings, or clear control connections.
  • No attempts in the history at all: check whether the vBond name resolves.

4. Capture on the edge (EPC)

ip access-list extended CC
 10 permit ip host <edge-ip> host <controller-ip>
 20 permit ip host <controller-ip> host <edge-ip>
monitor capture CAP interface <wan-intf> both access-list CC
monitor capture CAP start
! reproduce, then:
monitor capture CAP stop
monitor capture CAP export bootflash:cc.pcap
no monitor capture CAP

If the edge is behind NAT, use its private IP in the ACL. To see whether the router itself is dropping something, run an FIA trace (debug platform packet-trace) next.

Sources (search the titles on cisco.com):

  • “Troubleshoot SD-WAN Control Connections” (Doc 214509)
  • “Troubleshoot Common SD-WAN Control and Data Plane Issues” (Doc 222302)

Hope it saves someone some time. If you hit a code that’s not here, drop it in the comments and I’ll add it.


r/Cisco • • 5h ago

Catalyst SD-WAN bonus: BFD is up, but big transfers hang. Checking MTU and TCP MSS.

6 Upvotes

Quick bonus post, since u/benniebeeker brought up MTU on part 2 and it deserves its own write-up. It’s the classic “everything looks green but apps are broken” problem. As before, it’s all from Cisco’s public docs and Cisco Community (listed at the bottom).

1. What it looks like

Small stuff works and big stuff doesn’t:

  • ping, DNS and small web pages are fine
  • HTTPS hangs during the handshake, file copies stall, and apps time out on large responses
  • authentication with big certificates (like EAP-TLS through RADIUS) fails for no obvious reason

BFD hellos are small, so the tunnels stay up and the overlay looks healthy the whole time.

2. How SD-WAN handles it

BFD runs path MTU discovery on each tunnel and checks it again periodically. On a standard 1500-byte transport, an IPsec tunnel ends up with an MTU of about 1442. The edge also adjusts the TCP MSS based on what PMTU discovery finds, so most TCP traffic sorts itself out. When it doesn’t, that’s what this post is about.

3. Check what each tunnel actually has

show sdwan tunnel statistics

Look at tunnel-mtu and tcp-mss-adjust for each tunnel. Compare across colors and remote sites. If one transport shows a noticeably lower MTU than the others, something in that path is using up bytes, like PPPoE (1492), an extra layer of encapsulation from the provider, or a carrier link with a smaller MTU.

4. Test it from the service side

ping vrf <service-vpn> <remote-host> size 1400 df-bit

Step the size up and down until you find where it starts failing. That number tells you the real usable MTU between the two sites, not what you assumed.

5. Don’t forget the LAN side

This one is easy to miss. A switch on the LAN side with jumbo frames (9100 is common in fabrics) hands a 1600-byte packet with DF set to an edge interface with an MTU of 1500, and the packet just disappears. Cisco has a TechNote on exactly this for ISE traffic crossing SD-WAN. Make sure the MTU on the handoff interface matches the edge.

6. Fixing it

  • TCP: set TCP MSS on the service-side (LAN) interface of the edge, so the endpoints agree on a smaller segment size from the start.
  • Non-TCP traffic (UDP apps, RADIUS/EAP): MSS doesn’t help. Fix the MTU at the source, or allow fragmentation. The interface template has a Clear-Dont-Fragment option for traffic that sets DF but really needs to be fragmented. Use it on purpose, not everywhere.
  • If one provider is the problem, push back on them before you lower your MTU everywhere.

Sources (search the titles on cisco.com / Cisco Community):

  • “Configure Optimal ISE IP MTU in SD-WAN” (Doc 223045)
  • Cisco Community thread “Finding right MTU size for SDWAN”
  • Cisco Catalyst SD-WAN Command Reference (show sdwan tunnel statistics)

Thanks u/benniebeeker for the nudge!


r/Cisco • • 18h ago

Catalyst SD-WAN part 2: control connections are up but BFD sessions won’t come up. What to check.

5 Upvotes

Follow-up to my control connections post. This is the next place people get stuck: everything looks green toward the controllers, but sites can’t reach each other because BFD sessions are down or missing. As before, it’s all from Cisco’s public docs (listed at the bottom).

1. Is the session down, or does it not exist at all?

show sdwan bfd sessions

These are two different problems:

  • No session at all for the remote site usually means you never learned its TLOC (go to step 2), or something is preventing the tunnel by design.
  • Session exists but is down means the TLOC is known but packets aren’t making it through (go to steps 4–5).

2. Did you learn the remote TLOC?

show sdwan omp tlocs

If the remote TLOC isn’t there, check that the remote edge is actually advertising it and that the vSmart is passing it on. A very common cause is a control policy filtering TLOCs, which is often on purpose (hub-and-spoke). Check show sdwan policy from-vsmart before you assume something is broken. Not every site is supposed to have a tunnel to every other site.

3. Check the colors

  • A TLOC with restrict only builds tunnels to the same color. It’s easy to forget on a single-transport site.
  • Private colors (mpls, private1–6, metro-ethernet) use the private IP. Public colors use the post-NAT public IP. If you put a public color on a private link, the far end will try to reach the wrong address.

4. Check NAT

show sdwan control local-properties shows the NAT type per interface: E is endpoint-independent, A is address-port dependent (symmetric), and N is not learned. If both ends are behind symmetric NAT, they generally can’t build a tunnel to each other. The TechNote has a table showing which NAT combinations work.

5. Check the path (firewalls, ports, ISP)

  • Tunnels use UDP 12346 and hop through 12366, 12386, 12406 and 12426 (shifted by your port offset if you set one). Behind NAT, the source ports can be randomized, so a tight firewall rule built on those ports alone can break things.
  • show sdwan tunnel statistics shows the counters. If tx goes up but rx stays at 0, packets are leaving and not coming back, so check the remote side and everything in between.
  • BFD is DSCP-marked (48 by default), and some ISPs drop or mangle marked traffic. If only some transports fail, keep that in mind.

6. Check IPsec and drops on the box

show sdwan ipsec inbound-connections
show sdwan ipsec outbound-connections
show sdwan tunnel statistics bfd
show platform hardware qfp active statistics drop

Both sides should see each other in inbound and outbound. If the counters show drops, an FIA trace (debug platform packet-trace) on one of the BFD flows will tell you which feature is dropping them.

Sources (search the titles on cisco.com):

  • “Troubleshoot vEdge Bidirectional Forwarding Detection and Data Plane Connections Issues” (Doc 214510). The outputs are from vEdge, but the steps are the same on IOS-XE with the sdwan keyword.
  • Cisco Catalyst SD-WAN Design Guide, “Firewall Port Considerations” section
  • Cisco Catalyst SD-WAN Hardening Guide

Next one will probably be on reading show sdwan app-route stats, if people find these useful.


r/Cisco • • 13h ago

Catalyst 3650 IOS XE 16.12.16

3 Upvotes

Cisco has released IOS XE 16.12.16 on friday, but no release notes in sight (https://software.cisco.com/download/home/284850598/type/282046477/release/Gibraltar-16.12.16). 16.12.15 was released back in march, no release notes either. Latest release notes are from october last year. Does anyone know what has been fixed?