[Oisf-devel] 77b708=WIN?

Tue Oct 4 17:24:08 UTC 2011

I recommend switching to autofp from single for PF_RING.

On Tue, Oct 4, 2011 at 12:04 PM, Chris Wakelin
<c.d.wakelin at reading.ac.uk> wrote:
> Just updated to latest git master. I've not seen this before:
>
>> Cpu0  : 87.4%us,  7.6%sy,  0.0%ni,  0.3%id,  0.0%wa,  0.0%hi,  4.6%si,  0.0%st
>> Cpu1  :  8.8%us,  1.6%sy,  0.0%ni, 87.0%id,  0.0%wa,  0.0%hi,  2.6%si,  0.0%st
>> Cpu2  : 10.5%us,  2.5%sy,  0.0%ni, 85.7%id,  0.0%wa,  0.0%hi,  1.3%si,  0.0%st
>> Cpu3  : 12.2%us,  2.0%sy,  0.0%ni, 84.8%id,  0.0%wa,  0.0%hi,  1.0%si,  0.0%st
>> Cpu4  : 96.7%us,  0.0%sy,  0.0%ni,  0.3%id,  0.0%wa,  0.0%hi,  3.0%si,  0.0%st
>> Cpu5  :  6.5%us,  2.6%sy,  0.0%ni, 90.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
>> Cpu6  : 17.7%us,  0.9%sy,  0.0%ni, 79.8%id,  0.0%wa,  0.0%hi,  1.6%si,  0.0%st
>> Cpu7  :  1.9%us,  1.9%sy,  0.0%ni, 94.8%id,  1.0%wa,  0.0%hi,  0.3%si,  0.0%st
>> Mem:  16465276k total, 10679036k used,  5786240k free,    25700k buffers
>> Swap:  3906552k total,    25876k used,  3880676k free,  3440596k cached
>>
>>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  P COMMAND
>> 10407 mysql     20   0  343m  59m 3116 R 99.8  0.4   5:39.85 0 mysqld
>> 18037 root      20   0 7400m 6.0g  49m R 99.8 38.3  46:35.11 4 Pfring4
>> 18091 snort     20   0  263m 187m 3028 S 17.9  1.2  35:09.99 1 argus
>> 18039 root      20   0 7400m 6.0g  49m S 15.2 38.3  14:17.66 6 Pfring6
>> 18036 root      20   0 7400m 6.0g  49m R 13.3 38.3  10:03.05 3 Pfring3
>> 18035 root      20   0 7400m 6.0g  49m S 11.3 38.3  12:38.72 2 Pfring2
>> 18034 root      20   0 7400m 6.0g  49m S  7.3 38.3   8:31.14 1 Pfring1
>> 18038 root      20   0 7400m 6.0g  49m S  6.0 38.3   6:58.98 5 Pfring5
>> 18040 root      20   0 7400m 6.0g  49m S  4.0 38.3   2:04.13 2 FlowManagerThre
>> 18090 snort     20   0  263m 187m 3028 S  0.7  1.2   0:13.65 5 argus
>>   131 root      20   0     0    0    0 S  0.3  0.0   0:04.81 4 kondemand/4
>> 10370 mysql     20   0  343m  59m 3116 S  0.3  0.4   2:37.97 3 mysqld
>>  1585 operator  20   0 19460 1612 1040 R  0.3  0.0   1:20.70 7 top
>
> One of the Pfring's is maxing out its CPU. I'm using
>
> * Will's "single" pf_ring runmode, with CPU affinity set
> * max-pending-packets : 2000
> * detect-engine/profile: medium
> * sgh-mpm-context: full
> * mpm-algo: ac
> * ~3500 rules
> * ~50,000 packets per second
>
> Killing suricata dumped core (7GB of it) :
>
>> #0  SigMatchSignaturesBuildMatchArraySIMD (th_v=<value optimised out>, de_ctx=<value optimised out>, det_ctx=0xaf67cd20, p=0xbabbaa70) at detect.c:918
>> 918         for (u = 0; u < det_ctx->sgh->sig_cnt; u += 64) {
>> #0  SigMatchSignaturesBuildMatchArraySIMD (th_v=<value optimised out>, de_ctx=<value optimised out>, det_ctx=0xaf67cd20, p=0xbabbaa70) at detect.c:918
>> #1  SigMatchSignaturesBuildMatchArray (th_v=<value optimised out>, de_ctx=<value optimised out>, det_ctx=0xaf67cd20, p=0xbabbaa70) at detect.c:1021
>> #2  SigMatchSignatures (th_v=<value optimised out>, de_ctx=<value optimised out>, det_ctx=0xaf67cd20, p=0xbabbaa70) at detect.c:1462
>> #3  0x0000000000428f2c in Detect (tv=0x0, p=<value optimised out>, data=0x1e4, pq=<value optimised out>, postpq=0x14) at detect.c:1857
>> #4  0x000000000041f1fc in FlowForceReassemblyForQ (q=<value optimised out>) at flow-timeout.c:452
>> #5  0x000000000041f313 in FlowForceReassembly () at flow-timeout.c:515
>> #6  0x0000000000407b14 in main (argc=6, argv=<value optimised out>) at suricata.c:1565
>
> but I got something similar at lunchtime (git master as of late Sunday I
> think) when I tried increasing max-pending-packets to 5000.
>
> It had been running fine yesterday though.
>
> Best Wishes,
> Chris
>
> On 04/10/11 17:01, Martin Holste wrote:
>> It grows at about 2-3 GB/day, and we are running some of the tagged
>> signatures, though not that many.  Recall again that I'm running about
>> 7k rules with ac full turned on, so my ram baseline is 33 GB.
>>
>> On Tue, Oct 4, 2011 at 10:20 AM, Victor Julien <victor at inliniac.net> wrote:
>>> On 10/04/2011 05:03 PM, Martin Holste wrote:
>>>> Hasn't crashed yet, which is I think as long as it's ever gone.  Ram
>>>> is up to 40 GB, so I'm glad to hear you've found and fixed a memory
>>>> leak.  I've got 100 GB more ram before it dies, so I'll let it run for
>>>> a few more days to see if the initial crashing issue is fixed.
>>>
>>> Thanks Martin. Whats your memory use growth rate?
>>>
>>> The memleak should only occur if sigs that use tags fire. There are
>>> quite a few of those it seems from quickly grepping through
>>> emerging-all.rules. We'll see :)
>>>
>>> Cheers,
>>> Victor
>>>
>>> --
>>> ---------------------------------------------
>>> Victor Julien
>>> http://www.inliniac.net/
>>> PGP: http://www.inliniac.net/victorjulien.asc
>>> ---------------------------------------------
>>>
>>>
>> _______________________________________________
>> Oisf-devel mailing list
>> Oisf-devel at openinfosecfoundation.org
>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>
>
> --
> --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
> Christopher Wakelin,                           c.d.wakelin at reading.ac.uk
> IT Services Centre, The University of Reading,  Tel: +44 (0)118 378 2908
> Whiteknights, Reading, RG6 6AF, UK              Fax: +44 (0)118 975 3094
> _______________________________________________
> Oisf-devel mailing list
> Oisf-devel at openinfosecfoundation.org
> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>