[Oisf-devel] Extremely long startup times on latest git

[Martin Holste]

Ah, fixed it!  Victor, you were on the right track--the problem was
that I was using a Snort rules file and not the Suricata-optimized
rule file.  Everything loads very quickly with the correct file (using
the ip/http keywords, etc.).  Actually, it's such poor performance, I
wonder if you want to put in a banner or warning or something when
loading non-IP rules.  Also, the performance is about 10x better, so
I'm wondering if flat out erroring somehow wouldn't be appropriate.
I'd hate for people to use the wrong rules file and think it was
Suricata being slow.

On Sun, Sep 18, 2011 at 6:08 PM, Victor Julien <victor at inliniac.net> wrote:
> On 09/18/2011 08:06 PM, Martin Holste wrote:
>> I'm seeing load times of greater than a half hour with a standard
>> setup, using default config values:
>>
>> [25718] 18/9/2011 -- 11:25:53 - (detect.c:2440) <Info>
>> (SigAddressPrepareStage1) -- 9301 signatures processed. 2013 are
>> IP-only rules, 2796 are inspecting packet payload, 2739 inspect
>> application layer, 0 are decoder/engine/stream event only
>> [25718] 18/9/2011 -- 11:25:53 - (detect.c:2443) <Info>
>> (SigAddressPrepareStage1) -- building signature grouping structure,
>> stage 1: adding signatures to signature source addresses... complete
>> [25718] 18/9/2011 -- 11:31:53 - (detect.c:3085) <Info>
>> (SigAddressPrepareStage2) -- building signature grouping structure,
>> stage 2: building source address list... complete
>> [25718] 18/9/2011 -- 11:59:07 - (detect.c:3642) <Info>
>> (SigAddressPrepareStage3) -- MPM memory 330428951 (dynamic 330428951,
>> ctxs 0, avg per ctx 0)
>> [25718] 18/9/2011 -- 11:59:07 - (detect.c:3644) <Info>
>> (SigAddressPrepareStage3) -- max sig id 9301, array size 1163
>> [25718] 18/9/2011 -- 11:59:07 - (detect.c:3655) <Info>
>> (SigAddressPrepareStage3) -- building signature grouping structure,
>> stage 3: building destination address lists... complete
>>
>> I think 6 minutes is a pretty long time to compile signatures (stage
>> 1), but I've seen that before.  Why does it take 28 minutes to build a
>> source address list?  I'm using the standard ET ruleset.
>
> This may be related to a change I did to allow ports in "ip only"
> signatures. Bugs in that code have caused issues like this before.
>
> My code is at commit e13181496c435f5a6b401faf7d40298608d3314c
>
> Can you test with and without that?
>
> --
> ---------------------------------------------
> Victor Julien
> http://www.inliniac.net/
> PGP: http://www.inliniac.net/victorjulien.asc
> ---------------------------------------------
>
> _______________________________________________
> Oisf-devel mailing list
> Oisf-devel at openinfosecfoundation.org
> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>