[Oisf-users] autofp mode performance - high level

Wed Sep 4 00:33:21 UTC 2013

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

My experience with trying to run autofp mode against ~5Gbs traffic is
that the packet acquisition->decode/stream function will not be able to
process all incoming packets on a single core, even with no rules
enabled.  I believe when I tried using more than cores 0,1 (single core
w/Hyperthreading) I ran into problems.

I don't have any experience with pfring mode; but from looking at this
description it appears to function like worker mode w/AF_PACKET.

> https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Runmodes

I suspect most users are going to be using a 1Gbs (or even 100mb)
sensor.  I use libpcap/autofp on the inside interface of a 1Gb web-proxy
for example.

Modern multi-queue 10Gb NICs are really designed to integrate with the
'worker' style threading model.  I've copied the 'top' output from a
live sensor with a few Gbs of traffic below.  The flows are nicely
hashed across all active cores.

As I mentioned before, the only real issue I've found with worker mode
is that an extremely heavy single flow can crush a single core and
ultimately cause some dropped packets.

> top - 00:17:47 up 8 days,  7:48,  5 users,  load average: 14.06, 14.10, 14.33
> Tasks: 205 total,   1 running, 204 sleeping,   0 stopped,   0 zombie
> %Cpu0  : 87.4 us,  1.0 sy,  0.3 ni,  0.3 id,  0.0 wa,  0.0 hi, 10.9 si,  0.0 st
> %Cpu1  : 50.7 us, 18.9 sy,  0.0 ni, 22.2 id,  0.0 wa,  0.0 hi,  8.3 si,  0.0 st
> %Cpu2  : 78.5 us,  1.0 sy,  0.3 ni, 14.2 id,  0.0 wa,  0.0 hi,  6.0 si,  0.0 st
> %Cpu3  : 75.5 us,  1.0 sy,  0.3 ni, 14.2 id,  0.0 wa,  0.0 hi,  8.9 si,  0.0 st
> %Cpu4  : 67.9 us,  1.3 sy,  0.0 ni, 22.8 id,  0.0 wa,  0.0 hi,  7.9 si,  0.0 st
> %Cpu5  : 75.2 us,  1.0 sy,  0.7 ni, 17.2 id,  0.0 wa,  0.0 hi,  6.0 si,  0.0 st
> %Cpu6  : 73.5 us,  0.7 sy,  0.0 ni, 17.2 id,  0.0 wa,  0.0 hi,  8.6 si,  0.0 st
> %Cpu7  : 62.3 us,  1.3 sy,  1.0 ni, 29.8 id,  0.3 wa,  0.0 hi,  5.3 si,  0.0 st
> %Cpu8  : 89.1 us,  0.3 sy,  0.0 ni,  1.7 id,  0.0 wa,  0.0 hi,  8.9 si,  0.0 st
> %Cpu9  : 72.5 us,  1.0 sy,  0.0 ni, 17.9 id,  0.0 wa,  0.0 hi,  8.6 si,  0.0 st
> %Cpu10 : 86.4 us,  0.3 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi, 13.2 si,  0.0 st
> %Cpu11 : 77.5 us,  0.0 sy,  0.0 ni, 11.9 id,  0.0 wa,  0.0 hi, 10.6 si,  0.0 st
> %Cpu12 : 64.9 us,  1.0 sy,  0.0 ni, 26.5 id,  0.0 wa,  0.0 hi,  7.6 si,  0.0 st
> %Cpu13 : 88.4 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi, 11.6 si,  0.0 st
> %Cpu14 : 70.2 us,  0.7 sy,  0.7 ni, 17.2 id,  0.0 wa,  0.0 hi, 11.3 si,  0.0 st
> %Cpu15 : 59.6 us,  1.0 sy,  0.3 ni, 30.5 id,  0.0 wa,  0.0 hi,  8.6 si,  0.0 st
> KiB Mem:  49454148 total, 49283648 used,   170500 free,   410216 buffers
> KiB Swap:        0 total,        0 used,        0 free, 22697120 cached
> 
>   PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
> 23017 root      20   0 23.588g 0.022t 7.368g S  1262 47.58   5237:22 /usr/bin/suricata

- -Coop

On 9/3/2013 5:06 PM, Darren Spruell wrote:
> Had a question about a statement from listserv [1] last month re: autofp
> performance:
> 
> ...it seems autofp only works for <1Gbs traffic.
> 
> However autofp was set as the default runmode some time ago which
> suggests to me that it must be reasonable/good for most cases compared
> to other modes.
> 
> Assuming "most cases" aren't generally <1Gbps monitoring, is there any
> reason autofp wouldn't be suitable for monitoring higher traffic rates
> without problematic loss? Seems to me that it would also depend on
> acquisition method (pcap, pfring, afpacket etc.) (and of course many
> other variables).
> 
> Looking at about a ~5 Gbps traffic flow on a couple of interfaces, 8
> cores/32GB RAM/Intel 82599. Wondering if autofp mode is to be avoided to
> focus on something like workers mode, etc.
> 
> [1]
> https://lists.openinfosecfoundation.org/pipermail/oisf-users/2013-August/002954.html
> 

- -- 
Cooper Nelson
Network Security Analyst
UCSD ACT Security Team
cnelson at ucsd.edu x41042
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJSJn/RAAoJEKIFRYQsa8FWBcoIAI+6UcmKqD4kCTvHqlvKi1Mu
xfYd+P2K7VyDgBtkQqG7Eud49wyhPKC1d9TD3OVrW/lvkHNUzNetd8nmwKFNmV4U
KCPOEhxnISr2GzuiAaEay74YfiuL2aU9WMOdP9fyTVbynUAapuFrCvTz7STRTiGs
eZ4omsBTVJJ3OCEbLRVSdeS8nfAu06EjLM2vr+y+sJ1Aget31+ZZlhcKz9siF4eh
6igpZI+rMRaUlniImvz+VCLq4b8Cg4wWEzigkjTIwmCPFpOjtOjcAGRRjDZOBs0+
RMiDUyoBer68x7w5LJ+CN14Gv+/9Snx8ikNg36xcpBBrANkKX/yPxaqboSp+6jM=
=q6iw
-----END PGP SIGNATURE-----