[Oisf-users] Loading Large Number of Rules produce different results

[GORHAM JOHNSON, OZELINA]

When loading certain types of signatures received the follow error
Critical> - [ERRCODE: SC_ERR_AHO_CORASICK(174)] - Just ran out of space in the queue.  Fatal Error.  Exiting.  Please file a bug report on this

After increasing the STATE_QUEUE_CONTAINER_SIZE to 524288 the rule file loaded but found an anomaly.

  1.   Why is it that 'STATE_QUEUE_CONTAINER_SIZE' does not always need to be increased to load a large number of rules (see below).
  2.  What is the highest number of rules the suricata can handle, assuming no limitation to memory?  Tried increasing STATE_QUEUE_CONTAINER_SIZE to 1048576 and get a segmentation fault at startup.

Attached are two files used to load 70k rules.

Using - Suricata version 4.1.2 RELEASE



Test results with:  STATE_QUEUE_CONTAINER_SIZE = 524288



rule70k-1.rules

8/4/2019 -- 15:57:10 - <Info> - 1 rule files processed. 70000 rules successfully loaded, 0 rules failed

8/4/2019 -- 15:57:10 - <Info> - Threshold config parsed: 0 rule(s) found

8/4/2019 -- 15:57:11 - <Info> - 70000 signatures processed. 0 are IP-only rules, 70000 are inspecting packet payload, 0 inspect application layer, 0 are decoder event only

8/4/2019 -- 15:57:13 - <Info> - cleaning up signature grouping structure... complete

8/4/2019 -- 15:57:13 - <Notice> - rule reload complete





rule70k-3.rules

8/4/2019 -- 15:47:58 - <Info> - 1 rule files processed. 70000 rules successfully loaded, 0 rules fai

led

8/4/2019 -- 15:47:58 - <Info> - Threshold config parsed: 0 rule(s) found

8/4/2019 -- 15:48:02 - <Info> - 70000 signatures processed. 0 are IP-only rules, 70000 are inspectin

g packet payload, 0 inspect application layer, 0 are decoder event only

8/4/2019 -- 15:48:06 - <Critical> - [ERRCODE: SC_ERR_AHO_CORASICK(174)] - Just ran out of space in t

he queue.  Fatal Error.  Exiting.  Please file a bug report on this



Thanks,

Ena Gorham Johnson
AT&T Labs
(470) 378-7867

This communication may contain information that is privileged, or confidential. If you are not the intended recipient, please note  FYI that any dissemination, distribution or copying of this communication is strictly prohibited.  Anyone who receives this message in error should notify the sender immediately by telephone or by return e-mail and delete it from his or her computer

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20190408/4dacc7d0/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rules.zip
Type: application/x-zip-compressed
Size: 3021189 bytes
Desc: rules.zip
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20190408/4dacc7d0/attachment-0001.bin>