[Oisf-users] Suricata on 8 cores, ~70K packets/sec
rmkml
rmkml at yahoo.fr
Tue Feb 15 17:12:35 UTC 2011
Thx for feedback Chris,
Do you have tested with last suricata v1.1 beta version please? or better with last git version?
Only for testing, maybe compile suricata with --enable-profiling option please?
What is your signatures set? maybe test without signatures set please?
can you test with more max-pending-packets like 5000 or more?
Regards
Rmkml
On Tue, 15 Feb 2011, Chris Wakelin wrote:
> Apologies for the long post!
>
> Here's some logs from the Suricata instance monitoring our student
> residences (no prizes for guessing which rules they trigger most often
> ...). We have an identical machine monitoring the campus network.
>
>> [7012] 15/2/2011 -- 01:02:57 - (flow-hash.c:324) <Warning> (FlowGetNew) -- [ERRCODE: SC_WARN_FLOW_EMERGENCY(158)] - Warning, engine running with FLOW_EMERGENCY bit set (ts.tv_sec: 1297731777, ts.tv_usec:777481)
>> [7028] 15/2/2011 -- 01:02:57 - (flow.c:1099) <Info> (FlowManagerThread) -- Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1297731777, ts.tv_usec:799576) flow_spare_q status(): 31% flows at the queue
>> [7012] 15/2/2011 -- 01:03:34 - (flow-hash.c:324) <Warning> (FlowGetNew) -- [ERRCODE: SC_WARN_FLOW_EMERGENCY(158)] - Warning, engine running with FLOW_EMERGENCY bit set (ts.tv_sec: 1297731814, ts.tv_usec:850460)
>> [7028] 15/2/2011 -- 01:03:34 - (flow.c:1099) <Info> (FlowManagerThread) -- Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1297731814, ts.tv_usec:870498) flow_spare_q status(): 32% flows at the queue
>
> We get lots of that eventually (and on the campus instance). We seem to
> be leaking flows somewhere. Any idea how?
>
>> [7008] 15/2/2011 -- 08:29:36 - (suricata.c:1258) <Info> (main) -- signal received
>> [7011] 15/2/2011 -- 08:29:36 - (source-pfring.c:311) <Info> (ReceivePfringThreadExitStats) -- (ReceivePfring) Packets 1256720045, bytes 999535465040
>> [7008] 15/2/2011 -- 08:29:36 - (suricata.c:1288) <Info> (main) -- time elapsed 37487s
>> [7011] 15/2/2011 -- 08:29:36 - (source-pfring.c:315) <Info> (ReceivePfringThreadExitStats) -- (ReceivePfring) Pfring Total:1256720045 Recv:1256720045 Drop:0 (0.0%).
>> [7013] 15/2/2011 -- 08:29:36 - (stream-tcp.c:3465) <Info> (StreamTcpExitPrintStats) -- (Stream1) Packets 975933372
>
> I guess that means Suricata doesn't think it missed anything.
>
>> [7027] 15/2/2011 -- 08:29:36 - (alert-fastlog.c:324) <Info> (AlertFastLogExitPrintStats) -- (Outputs) Alerts 1656
>> [7027] 15/2/2011 -- 08:29:36 - (alert-unified2-alert.c:603) <Info> (Unified2AlertThreadDeinit) -- Alert unified2 module wrote 1656 alerts
>> [7027] 15/2/2011 -- 08:29:36 - (log-httplog.c:404) <Info> (LogHttpLogExitPrintStats) -- (Outputs) HTTP requests 51027
>> [7027] 15/2/2011 -- 08:29:36 - (log-droplog.c:389) <Info> (LogDropLogExitPrintStats) -- (Outputs) Dropped Packets 0
>> [7028] 15/2/2011 -- 08:29:36 - (flow.c:1141) <Info> (FlowManagerThread) -- 5916442 new flows, 2665165 established flows were timed out, 4425494 flows in closed state
>
> Are these supposed to add up?
>
>> [7008] 15/2/2011 -- 08:29:36 - (stream-tcp-reassemble.c:352) <Info> (StreamTcpReassembleFree) -- Max memuse of the stream reassembly engine 67108863 (in use 0)
>> [7008] 15/2/2011 -- 08:29:36 - (stream-tcp.c:466) <Info> (StreamTcpFreeConfig) -- Max memuse of stream engine 33554304 (in use 0)
>> [7008] 15/2/2011 -- 08:29:36 - (detect.c:3335) <Info> (SigAddressCleanupStage1) -- cleaning up signature grouping structure... complete
>
> I have (tweaked)
>
>> max-pending-packets: 2000
>
>> flow:
>> # memcap: 33554432
>> # hash_size: 65536
>> # prealloc: 10000
>> memcap: 268435456
>> hash_size: 262144
>> prealloc: 40000
>> emergency_recovery: 30
>> prune_flows: 5
>
> (left at the defaults)
>
>> stream:
>> memcap: 33554432 # 32mb
>> checksum_validation: yes # reject wrong csums
>> inline: no # no inline mode
>> reassembly:
>> memcap: 67108864 # 64mb for reassembly
>> depth: 1048576 # reassemble 1mb into a stream
>
> Should these be increased? The machine has loads of memory (16GB).
>
> I have 8 cores, and have "set_cpu_affinity: no" and
> "detect_thread_ratio: 1.5". I'm getting roughly 60% idle on each core
> (more at the moment, and the 99.7% use by Decode1 is unusual):
>
>> top - 16:34:47 up 4 days, 23:12, 5 users, load average: 1.91, 2.00, 1.98
>> Tasks: 226 total, 3 running, 223 sleeping, 0 stopped, 0 zombie
>> Cpu0 : 1.7%us, 3.3%sy, 0.0%ni, 95.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
>> Cpu1 : 0.3%us, 2.0%sy, 0.0%ni, 97.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
>> Cpu2 : 0.6%us, 5.5%sy, 0.0%ni, 93.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
>> Cpu3 : 99.7%us, 0.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
>> Cpu4 : 2.0%us, 2.6%sy, 0.0%ni, 95.1%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st
>> Cpu5 : 11.9%us, 3.5%sy, 0.0%ni, 83.1%id, 0.0%wa, 0.0%hi, 1.5%si, 0.0%st
>> Cpu6 : 1.6%us, 3.2%sy, 0.0%ni, 95.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
>> Cpu7 : 2.6%us, 3.6%sy, 0.0%ni, 93.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
>> Mem: 16465480k total, 14030348k used, 2435132k free, 247064k buffers
>> Swap: 3905528k total, 9016k used, 3896512k free, 12264960k cached
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ P COMMAND
>> 8280 snort 20 0 1063m 537m 1864 R 99.7 3.3 30:24.07 3 Decode1
>> 8175 snort 20 0 238m 174m 1008 R 21.2 1.1 65:18.84 5 argus
>> 8281 snort 20 0 1063m 537m 1864 S 4.3 3.3 3:07.09 7 Stream1
>> 8279 snort 20 0 1063m 537m 1864 S 3.0 3.3 1:18.07 6 ReceivePfring
>> 8296 snort 20 0 1063m 537m 1864 S 2.6 3.3 0:45.02 6 FlowManagerThre
>> 8294 snort 20 0 1063m 537m 1864 S 2.3 3.3 1:19.98 6 RespondReject
>> 8295 snort 20 0 1063m 537m 1864 S 2.0 3.3 1:14.83 2 Outputs
>> 8174 snort 20 0 238m 174m 1008 S 1.3 1.1 4:33.75 1 argus
>> 8282 snort 20 0 1063m 537m 1864 S 1.3 3.3 0:54.00 4 Detect1
>> 8283 snort 20 0 1063m 537m 1864 S 1.3 3.3 0:54.16 4 Detect2
>> 8288 snort 20 0 1063m 537m 1864 S 1.3 3.3 0:54.07 7 Detect7
>> 8290 snort 20 0 1063m 537m 1864 S 1.3 3.3 0:54.14 7 Detect9
>> 8292 snort 20 0 1063m 537m 1864 S 1.3 3.3 0:54.18 4 Detect11
>> 8284 snort 20 0 1063m 537m 1864 S 1.0 3.3 0:53.96 0 Detect3
>> 8285 snort 20 0 1063m 537m 1864 S 1.0 3.3 0:54.10 2 Detect4
>> 8286 snort 20 0 1063m 537m 1864 S 1.0 3.3 0:53.94 7 Detect5
>> 8287 snort 20 0 1063m 537m 1864 S 1.0 3.3 0:54.18 0 Detect6
>> 8289 snort 20 0 1063m 537m 1864 S 1.0 3.3 0:54.01 0 Detect8
>> 8291 snort 20 0 1063m 537m 1864 S 1.0 3.3 0:53.83 0 Detect10
>> 8293 snort 20 0 1063m 537m 1864 S 1.0 3.3 0:54.09 2 Detect12
>
> (The, ahem, user ID is a historical relic ...)
>
> I'm not sure whether setting CPU affinity would help; the comment "On
> Intel Core2 and Nehalem CPU's enabling this will degrade performance"
> put me off, though in fact our CPUs are slightly older:
>
>> model name : Intel(R) Xeon(R) CPU X5355 @ 2.66GHz
>> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca lahf_lm tpr_shadow
>
> Any hints and pointers would be most welcome!
>
> Best Wishes,
> Chris
>
> --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
> Christopher Wakelin, c.d.wakelin at reading.ac.uk
> IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439
> Whiteknights, Reading, RG6 6AF, UK Fax: +44 (0)118 975 3094
> _______________________________________________
> Oisf-users mailing list
> Oisf-users at openinfosecfoundation.org
> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>
More information about the Oisf-users
mailing list