[Oisf-users] Suricata on 8 cores, ~70K packets/sec
Chris Wakelin
c.d.wakelin at reading.ac.uk
Mon Feb 21 15:06:08 UTC 2011
On 18/02/11 02:01, Victor Julien wrote:
> On 02/17/2011 09:16 AM, Chris Wakelin wrote:
>> I've tried increasing the stream reassembly memcap to 512mb, and it
>> still seems to use it all. I've not had any flow emergency mode recently
>> though.
>
> If you have the memory, try increasing it more :)
I've increased it to 1Gb which was enough for one of the servers, but
the other one still runs out. I'll try 2Gb!
>>>
>>> Again, the decode1 thread ended up using all its CPU and the packet
>>> count dropped to 5-6K per second. Strangely, I've not seen that before
>>> today.
>>
>> I've rebooted this server since (needed to turn on I/OAT in the BIOS)
>> and it seems to have behaved since in this respect.
>
> Thats odd. Please let me know if you see it happening again.
It's happening again! I've tried running "strace" against the PID of the
Decode1 thread, and got lots of
> futex(0x77e514, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x77e510, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
> futex(0x77e4e8, FUTEX_WAKE_PRIVATE, 1) = 1
but that seems to happen even when its running normally. Is there
anything else I can try to get debug info? The stats in stats.log
suggest the pkts/sec drops right down:
> -------------------------------------------------------------------
> Date: 2/21/2011 -- 14:51:37 (uptime: 0d, 00h 39m 53s)
> -------------------------------------------------------------------
> Counter | TM Name | Value
> -------------------------------------------------------------------
> tcp.sessions | Stream1 | 521613
> tcp.ssn_memcap_drop | Stream1 | 0
> tcp.pseudo | Stream1 | 91337
> tcp.segment_memcap_drop | Stream1 | 1978699
> tcp.stream_depth_reached | Stream1 | 2373
> decoder.pkts | Decode1 | 123127760
> decoder.bytes | Decode1 | 97525215743
> decoder.ipv4 | Decode1 | 127480421
> decoder.ipv6 | Decode1 | 565
> decoder.ethernet | Decode1 | 123127760
> decoder.raw | Decode1 | 0
> decoder.sll | Decode1 | 0
> decoder.tcp | Decode1 | 92834478
> decoder.udp | Decode1 | 20940545
> decoder.icmpv4 | Decode1 | 116021
> decoder.icmpv6 | Decode1 | 343
> decoder.ppp | Decode1 | 9339
> decoder.pppoe | Decode1 | 0
> decoder.gre | Decode1 | 9339
> decoder.vlan | Decode1 | 57293169
> decoder.avg_pkt_size | Decode1 | 792.065215
> decoder.max_pkt_size | Decode1 | 1518
> defrag.ipv4.fragments | Decode1 | 8766543
> defrag.ipv4.reassembled | Decode1 | 4353025
> defrag.ipv4.timeouts | Decode1 | 0
> defrag.ipv6.fragments | Decode1 | 0
> defrag.ipv6.reassembled | Decode1 | 0
> defrag.ipv6.timeouts | Decode1 | 0
> detect.alert | Detect | 436
> -------------------------------------------------------------------
> Date: 2/21/2011 -- 14:51:41 (uptime: 0d, 00h 39m 57s)
> -------------------------------------------------------------------
> Counter | TM Name | Value
> -------------------------------------------------------------------
> tcp.sessions | Stream1 | 521688
> tcp.ssn_memcap_drop | Stream1 | 0
> tcp.pseudo | Stream1 | 91337
> tcp.segment_memcap_drop | Stream1 | 1978699
> tcp.stream_depth_reached | Stream1 | 2373
> decoder.pkts | Decode1 | 123142823
> decoder.bytes | Decode1 | 97536905815
> decoder.ipv4 | Decode1 | 127495865
> decoder.ipv6 | Decode1 | 565
> decoder.ethernet | Decode1 | 123142823
> decoder.raw | Decode1 | 0
> decoder.sll | Decode1 | 0
> decoder.tcp | Decode1 | 92845506
> decoder.udp | Decode1 | 20943543
> decoder.icmpv4 | Decode1 | 116037
> decoder.icmpv6 | Decode1 | 343
> decoder.ppp | Decode1 | 9339
> decoder.pppoe | Decode1 | 0
> decoder.gre | Decode1 | 9339
> decoder.vlan | Decode1 | 57300411
> decoder.avg_pkt_size | Decode1 | 792.063260
> decoder.max_pkt_size | Decode1 | 1518
> defrag.ipv4.fragments | Decode1 | 8767564
> defrag.ipv4.reassembled | Decode1 | 4353406
> defrag.ipv4.timeouts | Decode1 | 0
> defrag.ipv6.fragments | Decode1 | 0
> defrag.ipv6.reassembled | Decode1 | 0
> defrag.ipv6.timeouts | Decode1 | 0
> detect.alert | Detect | 436
> -------------------------------------------------------------------
compare to 10 minutes after a restart:
> -------------------------------------------------------------------
> Date: 2/21/2011 -- 15:03:34 (uptime: 0d, 00h 10m 33s)
> -------------------------------------------------------------------
> Counter | TM Name | Value
> -------------------------------------------------------------------
> detect.alert | Detect | 158
> decoder.pkts | Decode1 | 43052129
> decoder.bytes | Decode1 | 33982388376
> decoder.ipv4 | Decode1 | 44627032
> decoder.ipv6 | Decode1 | 109
> decoder.ethernet | Decode1 | 43052129
> decoder.raw | Decode1 | 0
> decoder.sll | Decode1 | 0
> decoder.tcp | Decode1 | 31771056
> decoder.udp | Decode1 | 8034364
> decoder.icmpv4 | Decode1 | 38318
> decoder.icmpv6 | Decode1 | 75
> decoder.ppp | Decode1 | 0
> decoder.pppoe | Decode1 | 0
> decoder.gre | Decode1 | 0
> decoder.vlan | Decode1 | 20169649
> decoder.avg_pkt_size | Decode1 | 789.331194
> decoder.max_pkt_size | Decode1 | 1518
> defrag.ipv4.fragments | Decode1 | 3153235
> defrag.ipv4.reassembled | Decode1 | 1575023
> defrag.ipv4.timeouts | Decode1 | 0
> defrag.ipv6.fragments | Decode1 | 0
> defrag.ipv6.reassembled | Decode1 | 0
> defrag.ipv6.timeouts | Decode1 | 0
> tcp.sessions | Stream1 | 184185
> tcp.ssn_memcap_drop | Stream1 | 0
> tcp.pseudo | Stream1 | 29128
> tcp.segment_memcap_drop | Stream1 | 0
> tcp.stream_depth_reached | Stream1 | 937
Best Wishes,
Chris
--
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
Christopher Wakelin, c.d.wakelin at reading.ac.uk
IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439
Whiteknights, Reading, RG6 6AF, UK Fax: +44 (0)118 975 3094
More information about the Oisf-users
mailing list