[Oisf-users] File Extraction Woes

[Jason Batchelor]

I adjusted the values accordingly and for a while, things seemed to go well
for a while. Unfortunately however, memcap inevitably increased after a
while to a point where I began dropping packets once again. I think the
fundamental issue here is that I just cannot keep up with memcap using
PF_RING. I'm considering giving Ubuntu a whirl and trying with AF_PACKET,
hopefully I will have better luck?


On Mon, Jun 2, 2014 at 1:49 PM, Peter Manev <petermanev at gmail.com> wrote:

> On Mon, Jun 2, 2014 at 8:41 PM, Jason Batchelor <jxbatchelor at gmail.com>
> wrote:
> > Peter,
> >
> > Per your suggestion, I tuned the conf file to the specs you posted
> earlier,
> > changing profile to 'high' and sgh-mpm-context to 'full'. After about two
> > hours I killed the process with the -15 flag, here are the last bits of
> the
> > suricata.log file after the termination...
> >
> > 2/6/2014 -- 18:23:04 - <Info> - TCP segment pool of size 4 had a peak
> use of
> > 6562 segments, more than the prealloc setting of 256
> > 2/6/2014 -- 18:23:04 - <Info> - TCP segment pool of size 16 had a peak
> use
> > of 3046 segments, more than the prealloc setting of 512
> > 2/6/2014 -- 18:23:04 - <Info> - TCP segment pool of size 112 had a peak
> use
> > of 41878 segments, more than the prealloc setting of 512
> > 2/6/2014 -- 18:23:04 - <Info> - TCP segment pool of size 248 had a peak
> use
> > of 34405 segments, more than the prealloc setting of 512
> > 2/6/2014 -- 18:23:04 - <Info> - TCP segment pool of size 512 had a peak
> use
> > of 26920 segments, more than the prealloc setting of 512
> > 2/6/2014 -- 18:23:04 - <Info> - TCP segment pool of size 768 had a peak
> use
> > of 22130 segments, more than the prealloc setting of 1024
> > 2/6/2014 -- 18:23:04 - <Info> - TCP segment pool of size 1448 had a peak
> use
> > of 89057 segments, more than the prealloc setting of 1024
> > 2/6/2014 -- 18:23:04 - <Info> - TCP segment pool of size 65535 had a peak
> > use of 2184 segments, more than the prealloc setting of 128
> > 2/6/2014 -- 18:23:04 - <Info> - TCP segment chunk pool had a peak use of
> > 44047 chunks, more than the prealloc setting of 250
> > 2/6/2014 -- 18:23:04 - <Info> - host memory usage: 390144 bytes, maximum:
> > 16777216
> > 2/6/2014 -- 18:23:04 - <Info> - Dumping profiling data for 1 rules.
> > 2/6/2014 -- 18:23:04 - <Info> - Done dumping profiling data.
> > 2/6/2014 -- 18:23:04 - <Info> - file /data/suricata/keyword_perf.log
> mode a
> > 2/6/2014 -- 18:23:04 - <Info> - Done dumping keyword profiling data.
> > 2/6/2014 -- 18:23:04 - <Info> - cleaning up signature grouping
> structure...
> > complete
> > 2/6/2014 -- 18:23:04 - <Notice> - Stats for 'p4p2':  pkts: 3515741384,
> drop:
> > 956825003 (27.22%), invalid chksum: 0
> >
> > The peak use in all cases far exceeds the prealloc settings. While I am
> not
> > very well versed in understanding how *exactly this ties things up, I
> would
> > venture to guess these should line up far more closely than they are?
>
> yes, please adjust accordingly and test again if you could.
> btw - i see the drops are 27% now, if i remember correctly they were 50%
> before?
>
> >
> > Hopefully, this helps, I am not quite sure where to go from here however.
> >
> > Thanks,
> > Jason
> >
>
>
> --
> Regards,
> Peter Manev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20140603/722f74e7/attachment-0002.html>