Linux: forward syslog with rsyslog or syslog-ng, and send flows and traps to MobiObs

Before you start

The MobiObs device is 10.20.4.9 in the examples. It listens on udp/5514 and tcp/5514 for syslog, udp/2055 for NetFlow and IPFIX, udp/6343 for sFlow and udp/1162 for SNMP traps. Examples were written for rsyslog 8.x, syslog-ng 3.x/4.x and net-snmp 5.9 on current Debian, Ubuntu and RHEL-family distributions. Outbound traffic is rarely blocked on the host itself, but check any egress rules in nftables or a cloud security group.

rsyslog

/etc/rsyslog.d/90-mobiobs.conf (rsyslog 8.x): TCP with a disk-assisted queue

*.* action(type="omfwd" target="10.20.4.9" port="5514" protocol="tcp"
           TCP_Framing="octet-counted"
           template="RSYSLOG_SyslogProtocol23Format"
           queue.type="LinkedList" queue.size="10000"
           queue.filename="mobiobs" queue.saveOnShutdown="on"
           action.resumeRetryCount="-1" action.resumeInterval="10")

RSYSLOG_SyslogProtocol23Format produces RFC 5424 messages, and octet-counted framing (RFC 6587) keeps multi-line messages intact; MobiObs detects both automatically. The queue settings matter on a mobile collector: if the device leaves the network or is locked, rsyslog buffers messages in memory, spills to disk under its work directory, and retries indefinitely instead of blocking local logging or discarding the backlog.

rsyslog: UDP instead of TCP

*.* action(type="omfwd" target="10.20.4.9" port="5514" protocol="udp"
           template="RSYSLOG_SyslogProtocol23Format")

UDP needs no connection and cannot stall, but messages are lost if the device is unreachable. To send only part of the log, wrap the action in a filter:

rsyslog: forward only authentication and warnings

if $syslogfacility-text == "authpriv" or $syslogseverity <= 4 then {
    action(type="omfwd" target="10.20.4.9" port="5514" protocol="udp"
           template="RSYSLOG_SyslogProtocol23Format")
}

Check the syntax with rsyslogd -N1, then apply it with sudo systemctl restart rsyslog. On distributions where the systemd journal is the primary log, rsyslog reads it through imjournal or imuxsock (the default packages do this), so journal messages are forwarded too. A host with only journald needs rsyslog or syslog-ng installed to forward in syslog format.

syslog-ng

/etc/syslog-ng/conf.d/mobiobs.conf (syslog-ng 3.x / 4.x)

destination d_mobiobs {
    syslog("10.20.4.9" transport("tcp") port(5514));
};
log { source(s_src); destination(d_mobiobs); };

The syslog() driver sends RFC 5424 with octet-counted framing. For RFC 3164 over UDP, use network("10.20.4.9" transport("udp") port(5514)) instead. The source name depends on the distribution: s_src on Debian and Ubuntu, s_sys on RHEL-family systems. Check it in /etc/syslog-ng/syslog-ng.conf, then run syslog-ng --syntax-only and restart the service.

Relay equipment that only sends to 514

Some equipment cannot change the syslog port. A laptop or server running rsyslog can accept it on 514 and pass it on:

/etc/rsyslog.d/10-relay.conf: listen on udp/514, forward to MobiObs

module(load="imudp")
input(type="imudp" port="514" ruleset="to_mobiobs")
ruleset(name="to_mobiobs") {
    action(type="omfwd" target="10.20.4.9" port="5514" protocol="udp")
}

Messages keep their original hostname, but MobiObs sees the relay as the source address. Allow udp/514 in the relay's firewall.

Send a test message

util-linux logger

logger -n 10.20.4.9 -P 5514 -d "MobiObs test from $(hostname)"      # UDP
logger -n 10.20.4.9 -P 5514 -T "MobiObs test from $(hostname)"      # TCP
logger -p auth.warning "local test, forwarded by rsyslog"

The first two lines send directly, bypassing the local daemon, so they test only the network path. The last line goes through rsyslog, which tests your forwarding rule. If nothing appears, see the troubleshooting checklist.

NetFlow and sFlow from a Linux host

softflowd (NetFlow v9) and hsflowd (sFlow)

# NetFlow v9 from eth0 (use -v 10 for IPFIX)
sudo softflowd -i eth0 -v 9 -n 10.20.4.9:2055 -t maxlife=60

# /etc/hsflowd.conf (host sFlow agent)
sflow {
  sampling = 400
  polling = 20
  collector { ip = 10.20.4.9 udpport = 6343 }
  pcap { dev = eth0 }
}

softflowd turns packets seen on an interface into flow records, which is useful on a mirror port or on a Linux router. hsflowd exports host counters and, with the pcap module, sampled packets.

SNMP traps with net-snmp

Test traps and snmpd trap destinations (net-snmp 5.9)

# one-off v2c linkDown trap
snmptrap -v 2c -c MOBIOBS-RO 10.20.4.9:1162 '' 1.3.6.1.6.3.1.1.5.3

# one-off v3 authPriv trap
snmptrap -v 3 -e 0x8000000001020304 -u mobiobs -l authPriv \
    -a SHA -A AuthPass-change-me -x AES -X PrivPass-change-me \
    10.20.4.9:1162 '' 1.3.6.1.6.3.1.1.5.3

# /etc/snmp/snmpd.conf: send snmpd's own traps to MobiObs
trap2sink 10.20.4.9:1162 MOBIOBS-RO
trapsess -v 3 -u mobiobs -l authPriv -a SHA -A AuthPass-change-me -x AES -X PrivPass-change-me 10.20.4.9:1162

Enter the same community, or the same SNMPv3 user and protocols, in MobiObs. The empty '' argument tells snmptrap to use the current uptime. Restart snmpd after editing its configuration; it sends a coldStart trap on start-up.

Remove it afterwards

Delete /etc/rsyslog.d/90-mobiobs.conf (or the syslog-ng file) and restart the daemon, and remove any on-disk queue files named mobiobs* in rsyslog's work directory (usually /var/lib/rsyslog or /var/spool/rsyslog). Stop softflowd and hsflowd if you started them, and remove the trap2sink or trapsess lines from snmpd.conf.