Capture SNMP traps during a network cutover

Why traps during a cutover

A cutover changes many things at once: cables move, uplinks are re-patched, devices reload with new software. Your permanent monitoring system usually sees these changes late, polls on a timer of minutes, and may be on the far side of the very link you are moving. A trap is sent by the device the moment the change happens, so a receiver on the same network as you shows the sequence in real time.

The useful notifications are standard and supported almost everywhere:

Vendors add their own notifications for power supplies, fans, spanning tree and routing neighbors. MobiObs shows those with their numeric OID where it does not know the name.

Before the window

1. Pick the network for the receiver. The MobiObs device must stay reachable from the equipment throughout the change, so put it on the management VLAN, ideally on a path that the cutover does not touch. A USB-C Ethernet adapter on an out-of-band switch is the most robust choice. Read its address from the Network screen; 10.20.4.9 below.
2. Choose v2c or v3. SNMPv2c needs only a community string and works in the Free tier. SNMPv3 with authentication and privacy is the better choice where the policy requires it, and needs a matching user in MobiObs (a Pro feature). For a one-off cutover on a management network, v2c with a dedicated community is usually enough.
3. Add the trap destination on every device in scope with port 1162, because Android apps cannot listen on 162 and MobiObs uses the same port on every platform. Enable at least the link and restart notifications. The vendor guides have the exact lines, for example Cisco IOS XE (snmp-server host … udp-port 1162), Junos (destination-port 1162), Aruba and VMware ESXi. Devices that only send to 162, such as MikroTik RouterOS and PAN-OS, need a relay or NAT rule.
4. Add syslog as well. Traps say what changed; the syslog messages around them often say why. MobiObs puts both on one timeline.
5. Write down the expected sequence. For each step of the plan, note which interfaces should go down and come back. During the window you then compare the trap list with the plan rather than reading it cold.

Prove the path with a test trap

Do this before the window, from a host on the management network, so a missing trap later means a real problem and not a firewall rule.

net-snmp: test linkDown trap and inform to udp/1162

# v2c trap (linkDown)
snmptrap -v 2c -c MOBIOBS-RO 10.20.4.9:1162 '' 1.3.6.1.6.3.1.1.5.3
# v2c inform: snmptrap waits for the acknowledgement
snmptrap -v 2c -Ci -c MOBIOBS-RO 10.20.4.9:1162 '' 1.3.6.1.6.3.1.1.5.3

The empty '' argument uses the host’s uptime. MobiObs acknowledges informs, so the second command returns without an error when the path works in both directions. Then trigger one real trap from a device in scope, for example by shutting and re-enabling an unused port, and check that linkDown and linkUp both arrive.

During the cutover

Keep the SNMP Traps screen open, on the mobile device or in a browser on a laptop (the device serves the same screens on port 8080). Watch for:

• linkDown without a matching linkUp. The interface that did not come back after a re-patch. The varbinds include the interface index and usually its name.
• linkDown and linkUp repeating on one port. A flapping link: a bad cable or optic, a speed or duplex mismatch, or a port channel member that keeps joining and leaving.
• coldStart from a device you did not reload. A crash or a power problem. Line it up with the syslog messages just before it.
• authenticationFailure after the change. A monitoring system still polling with an old community or credentials, or one that now reaches the device from a new address.
• Silence. A device that should have sent traps and did not may have lost its path to the receiver. The Sources screen shows when each sender was last seen.

The top agents and top traps panels make the noisiest device obvious; filter the list to that agent to read its sequence in order.

After the cutover

• Compare the received traps with the expected sequence you wrote down. Every planned linkDown should have its linkUp.
• Keep the evidence. The Free tier keeps one hour of data; for a longer window, Pro makes retention configurable and exports traps and logs to JSON, NDJSON or CSV for the change record. See pricing.
• Remove the trap destination from each device, or leave it pointed at the permanent receiver only, and save the configuration.

Verify it works

The setup is ready when all of these are true: the snmptrap test appears on the SNMP Traps screen; the inform test returns without a timeout; a real linkDown and linkUp from one device in scope appear with the right agent address; and Sources lists every device in scope. If one device is missing, fix it before the window rather than during it.

Troubleshooting

• The test trap does not arrive. Check the port (1162), then client isolation or an ACL between the management network and the receiver.
• Traps arrive flagged. The community, or the SNMPv3 user, protocols or passwords, do not match MobiObs. Flagged traps are still shown, so the path itself is fine.
• SNMPv3 traps fail but v2c works. Check the user name, the security level and the authentication and privacy protocols; SHA-1 and SHA-256 are different protocols. The SNMP trap receiver guide explains engine IDs.
• Traps from one device stop mid-window. Its route to the receiver probably went through the link you just moved. That is why the receiver belongs on a path the cutover does not touch.
• On an iPhone or iPad, nothing arrives after a while. iOS suspends background apps; keep MobiObs in the foreground with Auto-Lock off.