[Oisf-users] What limit to remove?

[Victor Julien]

On 12/18/2013 07:03 AM, Mark Ashley wrote:
> To answer my own question, since it's a 32bit compiled binary, the limit
> for the process address map is 4GB.
> 
> I'll have to prune the resource usage, or re-compile the binary and
> libraries 64bit.

I would go the 64bit route. 4GB is not nearly going to be enough on a
large pipe.

Cheers,
Victor

> 
> On Wed, Dec 18, 2013 at 4:33 PM, Mark Ashley <mark at ibiblio.org
> <mailto:mark at ibiblio.org>> wrote:
> 
>     Solaris 11 x86
>     X4600 128GB RAM, 16 x AMD8220s
>     Suricata master-2013-12-02
> 
>     I'm seeing memory "exhaustion" issues with starting suricata with
>     pretty much all of the rules switched on, including the emerging
>     ones. The process gets up to about 3.85GB RSS and then crashes with
>     the errors below.
> 
>     I'm looking at the suricata.yaml file and trying to find out where I
>     can increase limits to allow the rules to be there, and suricata to
>     cache them. There's no issue with available RAM or a ulimit setting.
>     It can have it all.
> 
>     ta,
>     Mark.
> 
> 
>     [1] 18/12/2013 -- 15:01:53 - (suricata.c:1877) <Info>
>     (PostConfLoadedSetup) -- No 'host-mode': suricata is in IDS mode,
>     using default setting 'sniffer-only'
>     [1] 18/12/2013 -- 15:01:53 - (defrag-hash.c:209) <Info>
>     (DefragInitConfig) -- allocated 1280000 bytes of memory for the
>     defrag hash... 40000 buckets of size 32
>     [1] 18/12/2013 -- 15:01:53 - (defrag-hash.c:236) <Info>
>     (DefragInitConfig) -- preallocated 50000 defrag trackers of size 136
>     [1] 18/12/2013 -- 15:01:53 - (defrag-hash.c:243) <Info>
>     (DefragInitConfig) -- defrag memory usage: 8080000 bytes, maximum:
>     4294967296
>     [1] 18/12/2013 -- 15:01:53 - (tmqh-flow.c:61) <Info>
>     (TmqhFlowRegister) -- AutoFP mode using "Round Robin" flow load balancer
>     [1] 18/12/2013 -- 15:01:53 - (suricata.c:1910) <Info>
>     (PostConfLoadedSetup) -- Will use direct allocation instead of
>     packet pool
>     [1] 18/12/2013 -- 15:01:53 - (host.c:202) <Info> (HostInitConfig) --
>     allocated 131072 bytes of memory for the host hash... 4096 buckets
>     of size 32
>     [1] 18/12/2013 -- 15:01:53 - (host.c:227) <Info> (HostInitConfig) --
>     preallocated 5000 hosts of size 96
>     [1] 18/12/2013 -- 15:01:53 - (host.c:229) <Info> (HostInitConfig) --
>     host memory usage: 651072 bytes, maximum: 17179869184 <tel:17179869184>
>     [1] 18/12/2013 -- 15:01:53 - (flow.c:383) <Info> (FlowInitConfig) --
>     allocated 2097152 bytes of memory for the flow hash... 65536 buckets
>     of size 32
>     [1] 18/12/2013 -- 15:01:55 - (flow.c:409) <Info> (FlowInitConfig) --
>     preallocated 1000000 flows of size 236
>     [1] 18/12/2013 -- 15:01:55 - (flow.c:411) <Info> (FlowInitConfig) --
>     flow memory usage: 242097152 bytes, maximum: 4294967296
>     [1] 18/12/2013 -- 15:01:55 - (reputation.c:459) <Info> (SRepInit) --
>     IP reputation disabled
>     [1] 18/12/2013 -- 15:01:55 - (util-magic.c:62) <Info> (MagicInit) --
>     using magic-file /usr/local/share/misc/magic.mgc
>     [1] 18/12/2013 -- 15:01:55 - (suricata.c:1753) <Info>
>     (SetupDelayedDetect) -- Delayed detect disabled
>     [1] 18/12/2013 -- 15:02:11 - (detect.c:453) <Info>
>     (SigLoadSignatures) -- 49 rule files processed. 15023 rules
>     successfully loaded, 0 rules failed
>     [1] 18/12/2013 -- 15:02:11 - (detect.c:2564) <Info>
>     (SigAddressPrepareStage1) -- 15040 signatures processed. 1039 are
>     IP-only rules, 5182 are inspecting packet payload, 10662 inspect
>     application layer, 83 are decoder event only
>     [1] 18/12/2013 -- 15:02:11 - (detect.c:2570) <Info>
>     (SigAddressPrepareStage1) -- building signature grouping structure,
>     stage 1: preprocessing rules... complete
>     [1] 18/12/2013 -- 15:02:13 - (detect.c:3194) <Info>
>     (SigAddressPrepareStage2) -- building signature grouping structure,
>     stage 2: building source address list... complete
>     [1] 18/12/2013 -- 15:03:39 - (util-mpm-ac.c:430) <Error>
>     (SCACInitNewState) -- [ERRCODE: SC_ERR_MEM_ALLOC(1)] - SCRealloc
>     failed: Not enough space, while trying to allocate 15066112 bytes
>     [1] 18/12/2013 -- 15:03:39 - (util-mpm-ac.c:430) <Error>
>     (SCACInitNewState) -- [ERRCODE: SC_ERR_FATAL(171)] - Out of memory.
>     The engine cannot be initialized. Exiting...
> 
> 
> 
> 
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> OISF: http://www.openinfosecfoundation.org/
> 


-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------