[Oisf-users] What limit to remove?

[Mark Ashley]

Solaris 11 x86
X4600 128GB RAM, 16 x AMD8220s
Suricata master-2013-12-02

I'm seeing memory "exhaustion" issues with starting suricata with pretty
much all of the rules switched on, including the emerging ones. The process
gets up to about 3.85GB RSS and then crashes with the errors below.

I'm looking at the suricata.yaml file and trying to find out where I can
increase limits to allow the rules to be there, and suricata to cache them.
There's no issue with available RAM or a ulimit setting. It can have it all.

ta,
Mark.


[1] 18/12/2013 -- 15:01:53 - (suricata.c:1877) <Info> (PostConfLoadedSetup)
-- No 'host-mode': suricata is in IDS mode, using default setting
'sniffer-only'
[1] 18/12/2013 -- 15:01:53 - (defrag-hash.c:209) <Info> (DefragInitConfig)
-- allocated 1280000 bytes of memory for the defrag hash... 40000 buckets
of size 32
[1] 18/12/2013 -- 15:01:53 - (defrag-hash.c:236) <Info> (DefragInitConfig)
-- preallocated 50000 defrag trackers of size 136
[1] 18/12/2013 -- 15:01:53 - (defrag-hash.c:243) <Info> (DefragInitConfig)
-- defrag memory usage: 8080000 bytes, maximum: 4294967296
[1] 18/12/2013 -- 15:01:53 - (tmqh-flow.c:61) <Info> (TmqhFlowRegister) --
AutoFP mode using "Round Robin" flow load balancer
[1] 18/12/2013 -- 15:01:53 - (suricata.c:1910) <Info> (PostConfLoadedSetup)
-- Will use direct allocation instead of packet pool
[1] 18/12/2013 -- 15:01:53 - (host.c:202) <Info> (HostInitConfig) --
allocated 131072 bytes of memory for the host hash... 4096 buckets of size
32
[1] 18/12/2013 -- 15:01:53 - (host.c:227) <Info> (HostInitConfig) --
preallocated 5000 hosts of size 96
[1] 18/12/2013 -- 15:01:53 - (host.c:229) <Info> (HostInitConfig) -- host
memory usage: 651072 bytes, maximum: 17179869184
[1] 18/12/2013 -- 15:01:53 - (flow.c:383) <Info> (FlowInitConfig) --
allocated 2097152 bytes of memory for the flow hash... 65536 buckets of
size 32
[1] 18/12/2013 -- 15:01:55 - (flow.c:409) <Info> (FlowInitConfig) --
preallocated 1000000 flows of size 236
[1] 18/12/2013 -- 15:01:55 - (flow.c:411) <Info> (FlowInitConfig) -- flow
memory usage: 242097152 bytes, maximum: 4294967296
[1] 18/12/2013 -- 15:01:55 - (reputation.c:459) <Info> (SRepInit) -- IP
reputation disabled
[1] 18/12/2013 -- 15:01:55 - (util-magic.c:62) <Info> (MagicInit) -- using
magic-file /usr/local/share/misc/magic.mgc
[1] 18/12/2013 -- 15:01:55 - (suricata.c:1753) <Info> (SetupDelayedDetect)
-- Delayed detect disabled
[1] 18/12/2013 -- 15:02:11 - (detect.c:453) <Info> (SigLoadSignatures) --
49 rule files processed. 15023 rules successfully loaded, 0 rules failed
[1] 18/12/2013 -- 15:02:11 - (detect.c:2564) <Info>
(SigAddressPrepareStage1) -- 15040 signatures processed. 1039 are IP-only
rules, 5182 are inspecting packet payload, 10662 inspect application layer,
83 are decoder event only
[1] 18/12/2013 -- 15:02:11 - (detect.c:2570) <Info>
(SigAddressPrepareStage1) -- building signature grouping structure, stage
1: preprocessing rules... complete
[1] 18/12/2013 -- 15:02:13 - (detect.c:3194) <Info>
(SigAddressPrepareStage2) -- building signature grouping structure, stage
2: building source address list... complete
[1] 18/12/2013 -- 15:03:39 - (util-mpm-ac.c:430) <Error> (SCACInitNewState)
-- [ERRCODE: SC_ERR_MEM_ALLOC(1)] - SCRealloc failed: Not enough space,
while trying to allocate 15066112 bytes
[1] 18/12/2013 -- 15:03:39 - (util-mpm-ac.c:430) <Error> (SCACInitNewState)
-- [ERRCODE: SC_ERR_FATAL(171)] - Out of memory. The engine cannot be
initialized. Exiting...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20131218/61dc64e1/attachment.html>