[Oisf-users] File Extraction Woes

[Jason Batchelor]

Peter,

Per your suggestion, I tuned the conf file to the specs you posted earlier,
changing profile to 'high' and sgh-mpm-context to 'full'. After about two
hours I killed the process with the -15 flag, here are the last bits of the
suricata.log file after the termination...

2/6/2014 -- 18:23:04 - <Info> - TCP segment pool of size 4 had a peak use
of 6562 segments, more than the prealloc setting of 256
2/6/2014 -- 18:23:04 - <Info> - TCP segment pool of size 16 had a peak use
of 3046 segments, more than the prealloc setting of 512
2/6/2014 -- 18:23:04 - <Info> - TCP segment pool of size 112 had a peak use
of 41878 segments, more than the prealloc setting of 512
2/6/2014 -- 18:23:04 - <Info> - TCP segment pool of size 248 had a peak use
of 34405 segments, more than the prealloc setting of 512
2/6/2014 -- 18:23:04 - <Info> - TCP segment pool of size 512 had a peak use
of 26920 segments, more than the prealloc setting of 512
2/6/2014 -- 18:23:04 - <Info> - TCP segment pool of size 768 had a peak use
of 22130 segments, more than the prealloc setting of 1024
2/6/2014 -- 18:23:04 - <Info> - TCP segment pool of size 1448 had a peak
use of 89057 segments, more than the prealloc setting of 1024
2/6/2014 -- 18:23:04 - <Info> - TCP segment pool of size 65535 had a peak
use of 2184 segments, more than the prealloc setting of 128
2/6/2014 -- 18:23:04 - <Info> - TCP segment chunk pool had a peak use of
44047 chunks, more than the prealloc setting of 250
2/6/2014 -- 18:23:04 - <Info> - host memory usage: 390144 bytes, maximum:
16777216
2/6/2014 -- 18:23:04 - <Info> - Dumping profiling data for 1 rules.
2/6/2014 -- 18:23:04 - <Info> - Done dumping profiling data.
2/6/2014 -- 18:23:04 - <Info> - file /data/suricata/keyword_perf.log mode a
2/6/2014 -- 18:23:04 - <Info> - Done dumping keyword profiling data.
2/6/2014 -- 18:23:04 - <Info> - cleaning up signature grouping structure...
complete
2/6/2014 -- 18:23:04 - <Notice> - Stats for 'p4p2':  pkts: 3515741384,
drop: 956825003 (27.22%), invalid chksum: 0

The peak use in all cases far exceeds the prealloc settings. While I am not
very well versed in understanding how *exactly this ties things up, I would
venture to guess these should line up far more closely than they are?

Hopefully, this helps, I am not quite sure where to go from here however.

Thanks,
Jason











On Mon, Jun 2, 2014 at 9:06 AM, Victor Julien <lists at inliniac.net> wrote:

> On 05/31/2014 04:05 PM, Jason Batchelor wrote:
> > I will take a peek at these parameters and report back the outcome. I
> > only have that one rule I posted in my initial email running however,
> > that just looks for me downloading a PDF.
>
> Another thing to try would be to enable profiling: recompile with
> --enable-profiling passed to configure and then enable it in your yaml.
>
> This could give some insights into what parts of the system are (over)
> loaded.
>
> --
> ---------------------------------------------
> Victor Julien
> http://www.inliniac.net/
> PGP: http://www.inliniac.net/victorjulien.asc
> ---------------------------------------------
>
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> OISF: http://www.openinfosecfoundation.org/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20140602/94524ed1/attachment-0002.html>