[Oisf-users] File Extraction Woes

[Peter Manev]

On Tue, Jun 3, 2014 at 4:10 PM, Jason Batchelor <jxbatchelor at gmail.com> wrote:
> I adjusted the values accordingly and for a while, things seemed to go well
> for a while. Unfortunately however, memcap inevitably increased after a
> while to a point where I began dropping packets once again. I think the
> fundamental issue here is that I just cannot keep up with memcap using
> PF_RING. I'm considering giving Ubuntu a whirl and trying with AF_PACKET,
> hopefully I will have better luck?
>

Please note that --enable-profile affects heavily performance.



>
> On Mon, Jun 2, 2014 at 1:49 PM, Peter Manev <petermanev at gmail.com> wrote:
>>
>> On Mon, Jun 2, 2014 at 8:41 PM, Jason Batchelor <jxbatchelor at gmail.com>
>> wrote:
>> > Peter,
>> >
>> > Per your suggestion, I tuned the conf file to the specs you posted
>> > earlier,
>> > changing profile to 'high' and sgh-mpm-context to 'full'. After about
>> > two
>> > hours I killed the process with the -15 flag, here are the last bits of
>> > the
>> > suricata.log file after the termination...
>> >
>> > 2/6/2014 -- 18:23:04 - <Info> - TCP segment pool of size 4 had a peak
>> > use of
>> > 6562 segments, more than the prealloc setting of 256
>> > 2/6/2014 -- 18:23:04 - <Info> - TCP segment pool of size 16 had a peak
>> > use
>> > of 3046 segments, more than the prealloc setting of 512
>> > 2/6/2014 -- 18:23:04 - <Info> - TCP segment pool of size 112 had a peak
>> > use
>> > of 41878 segments, more than the prealloc setting of 512
>> > 2/6/2014 -- 18:23:04 - <Info> - TCP segment pool of size 248 had a peak
>> > use
>> > of 34405 segments, more than the prealloc setting of 512
>> > 2/6/2014 -- 18:23:04 - <Info> - TCP segment pool of size 512 had a peak
>> > use
>> > of 26920 segments, more than the prealloc setting of 512
>> > 2/6/2014 -- 18:23:04 - <Info> - TCP segment pool of size 768 had a peak
>> > use
>> > of 22130 segments, more than the prealloc setting of 1024
>> > 2/6/2014 -- 18:23:04 - <Info> - TCP segment pool of size 1448 had a peak
>> > use
>> > of 89057 segments, more than the prealloc setting of 1024
>> > 2/6/2014 -- 18:23:04 - <Info> - TCP segment pool of size 65535 had a
>> > peak
>> > use of 2184 segments, more than the prealloc setting of 128
>> > 2/6/2014 -- 18:23:04 - <Info> - TCP segment chunk pool had a peak use of
>> > 44047 chunks, more than the prealloc setting of 250
>> > 2/6/2014 -- 18:23:04 - <Info> - host memory usage: 390144 bytes,
>> > maximum:
>> > 16777216
>> > 2/6/2014 -- 18:23:04 - <Info> - Dumping profiling data for 1 rules.
>> > 2/6/2014 -- 18:23:04 - <Info> - Done dumping profiling data.
>> > 2/6/2014 -- 18:23:04 - <Info> - file /data/suricata/keyword_perf.log
>> > mode a
>> > 2/6/2014 -- 18:23:04 - <Info> - Done dumping keyword profiling data.
>> > 2/6/2014 -- 18:23:04 - <Info> - cleaning up signature grouping
>> > structure...
>> > complete
>> > 2/6/2014 -- 18:23:04 - <Notice> - Stats for 'p4p2':  pkts: 3515741384,
>> > drop:
>> > 956825003 (27.22%), invalid chksum: 0
>> >
>> > The peak use in all cases far exceeds the prealloc settings. While I am
>> > not
>> > very well versed in understanding how *exactly this ties things up, I
>> > would
>> > venture to guess these should line up far more closely than they are?
>>
>> yes, please adjust accordingly and test again if you could.
>> btw - i see the drops are 27% now, if i remember correctly they were 50%
>> before?
>>
>> >
>> > Hopefully, this helps, I am not quite sure where to go from here
>> > however.
>> >
>> > Thanks,
>> > Jason
>> >
>>
>>
>> --
>> Regards,
>> Peter Manev
>
>



-- 
Regards,
Peter Manev