[Oisf-users] Loading Large Number of Rules produce different results

[Peter Manev]

On Mon, Apr 8, 2019 at 6:22 PM GORHAM JOHNSON, OZELINA <og1939 at att.com> wrote:
>
> When loading certain types of signatures received the follow error
>
> Critical> - [ERRCODE: SC_ERR_AHO_CORASICK(174)] - Just ran out of space in the queue.  Fatal Error.  Exiting.  Please file a bug report on this
>
>
>
> After increasing the STATE_QUEUE_CONTAINER_SIZE to 524288 the rule file loaded but found an anomaly.
>
>  Why is it that ‘STATE_QUEUE_CONTAINER_SIZE’ does not always need to be increased to load a large number of rules (see below).
> What is the highest number of rules the suricata can handle, assuming no limitation to memory?  Tried increasing STATE_QUEUE_CONTAINER_SIZE to 1048576 and get a segmentation fault at startup.
>
>
>
> Attached are two files used to load 70k rules.
>
> Using - Suricata version 4.1.2 RELEASE
>
>
>
> Test results with:  STATE_QUEUE_CONTAINER_SIZE = 524288
>
>

At first glance  -  it would be preferable if you could post  a bug
report for better investigation for the segfault and the "out of
space" err. I tried to reproduce it it with the latest git as well
(see below).

I would recommend  using Hyperscan for bigger rulesets (and better
performance ). Note if Hyperscan is installed on the system it will
automatically be picked up, you don't need to specify it (
https://github.com/OISF/suricata/blob/master/suricata.yaml.in#L1459 )
-

~/Work/Suricata/suricomp$ sudo /opt/suritest/bin/suricata --set
"mpm-algo = ac" --set "spm-algo = bm" -S  rule70k-1.rules -T
[17278] 8/4/2019 -- 23:28:46 - (suricata.c:1853) <Info>
(ParseCommandLine) -- Running suricata under test mode
[17278] 8/4/2019 -- 23:28:46 - (suricata.c:1066) <Notice> (LogVersion)
-- This is Suricata version 5.0.0-dev (rev 7f63ec185) running in
SYSTEM mode
[17278] 8/4/2019 -- 23:28:49 - (suricata.c:2991) <Notice> (main) --
Configuration provided was successfully loaded. Exiting.
~/Work/Suricata/suricomp$ sudo /opt/suritest/bin/suricata --set
"mpm-algo = ac" --set "spm-algo = bm" -S  rule70k-3.rules -T
[17294] 8/4/2019 -- 23:28:59 - (suricata.c:1853) <Info>
(ParseCommandLine) -- Running suricata under test mode
[17294] 8/4/2019 -- 23:28:59 - (suricata.c:1066) <Notice> (LogVersion)
-- This is Suricata version 5.0.0-dev (rev 7f63ec185) running in
SYSTEM mode
[17294] 8/4/2019 -- 23:29:06 - (util-mpm-ac.c:401) <Critical>
(SCACEnqueue) -- [ERRCODE: SC_ERR_AHO_CORASICK(174)] - Just ran out of
space in the queue.  Fatal Error.  Exiting.  Please file a bug report
on this
~/Work/Suricata/suricomp$ sudo /opt/suritest/bin/suricata --set
"mpm-algo = hs" --set "spm-algo = hs" -S  rule70k-3.rules -T
[17307] 8/4/2019 -- 23:29:23 - (suricata.c:1853) <Info>
(ParseCommandLine) -- Running suricata under test mode
[17307] 8/4/2019 -- 23:29:23 - (suricata.c:1066) <Notice> (LogVersion)
-- This is Suricata version 5.0.0-dev (rev 7f63ec185) running in
SYSTEM mode
[17307] 8/4/2019 -- 23:29:37 - (suricata.c:2991) <Notice> (main) --
Configuration provided was successfully loaded. Exiting.
~/Work/Suricata/suricomp$

Thank you

-- 
Regards,
Peter Manev