[Oisf-users] Suricata 4.1.x possible memory leak (tcp.reassembly_memuse)
Eric Urban
eurban at umn.edu
Thu Mar 5 14:12:28 UTC 2020
Since you mentioned the SMB parser, I believe it wouldn't apply in your
case since you don't have Rust enabled.
I looked at our config in the stream section and noticed two options that
differ quite a bit, which is the prealloc-sessions being set much higher
(500000) and reassembly.depth is at the default value. From
https://suricata.readthedocs.io/en/suricata-4.1.6/configuration/suricata-yaml.html?highlight=prealloc%20sessions#stream-engine,
it is written that "To mitigate Suricata from being overloaded by fast
session creation, the option prealloc_sessions instructs Suricata to keep a
number of sessions ready in memory" (Note that there is maybe a typo there
in "prealloc_sessions" as in our config and in the suricata.yaml example
from their GitHub repo it is actually prealloc-sessions with a hyphen and
not an underscore). Though your issue isn't quite what I would expect
based on the documentation, it is worth a shot to see if this has any
effect.
--
Eric Urban
Security Analyst | University Information Security (UIS)
University of Minnesota | umn.edu
Information Security is a shared responsibility. Learn more at:
https://z.umn.edu/uis
On Tue, Mar 3, 2020 at 2:59 AM d0g d3v <d0gd3v at gmail.com> wrote:
> Hi Guys,
>
>
> After suricata upgrade from 4.0.5 to 4.1.x(0-7) it seems in my setup I experiencing some memory leak. tcp.reassembly_memuse continuously increasing up to any limit set (40gb in few minutes) and then tcp.segment_memcap_drop are triggered and suricata events drops to almost zero.
>
> On the other side, in 4.0.5 there is 15gb stream.reassembly.memcap set and is never reached. tcp.reassembly_memuse is stable around 11gb.
>
>
> My setup:
>
> CPU: 56 cores
>
> MEM: 64 GB
>
> debian 9
>
> pfring-zc
>
>
> stream:
> memcap: 20gb
> checksum-validation: no # reject wrong csums
> inline: no # auto will use inline mode in IPS mode, yes or no set it statically
> prealloc-sessions: 8096
> bypass: yes
> midstream: true
> async-oneside: true
> reassembly:
> memcap: 15gb
> depth: 20mb # reassemble 1mb into a stream
> toserver-chunk-size: 2560
> toclient-chunk-size: 2560
> randomize-chunk-size: yes
>
>
> What I tried:
>
> -disable zero copy, change pfring to afpacket - same result
>
> -filter-out smb communication (bpf) if the smb parsing introduced in 4.1. is not root cause - same result
>
> -significantly decrease stream.reassembly.depth and flow-timeouts - same results
>
>
>
> I currently can't easy try v5.0.x because of dependencies on the current system. Could you please point me to direction what else I could check?
>
> buildinfo for 4.0.5 and then 4.1.7
>
> This is Suricata version 4.0.5 RELEASE
> Features: PCAP_SET_BUFF PF_RING AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LIBJANSSON TLS MAGIC
> SIMD support: SSE_4_2 SSE_4_1 SSE_3
> Atomic intrisics: 1 2 4 8 16 byte(s)
> 64-bits, Little-endian architecture
> GCC version 6.3.0 20170516, C version 199901
> compiled with _FORTIFY_SOURCE=0
> L1 cache line size (CLS)=64
> thread local storage method: __thread
> compiled with LibHTP v0.5.27, linked against LibHTP v0.5.27
>
> Suricata Configuration:
> AF_PACKET support: yes
> PF_RING support: yes
> NFQueue support: no
> NFLOG support: yes
> IPFW support: no
> Netmap support: no
> DAG enabled: no
> Napatech enabled: no
>
> Unix socket enabled: yes
> Detection enabled: yes
>
> Libmagic support: yes
> libnss support: yes
> libnspr support: yes
> libjansson support: yes
> hiredis support: yes
> hiredis async with libevent: no
> Prelude support: yes
> PCRE jit: yes
> LUA support: no
> libluajit: no
> libgeoip: yes
> Non-bundled htp: no
> Old barnyard2 support: no
> CUDA enabled: no
> Hyperscan support: yes
> Libnet support: yes
>
> Rust support (experimental): no
> Experimental Rust parsers: no
> Rust strict mode: no
>
> Suricatasc install: yes
>
> Profiling enabled: no
> Profiling locks enabled: no
>
> Development settings:
> Coccinelle / spatch: no
> Unit tests enabled: no
> Debug output enabled: no
> Debug validation enabled: no
>
> Generic build parameters:
> Installation prefix: /usr/local
> Configuration directory: /usr/local/etc/suricata/
> Log directory: /usr/local/var/log/suricata/
>
> --prefix /usr/local
> --sysconfdir /usr/local/etc
> --localstatedir /usr/local/var
>
> Host: x86_64-pc-linux-gnu
> Compiler: gcc (exec name) / gcc (real)
> GCC Protect enabled: no
> GCC march native enabled: yes
> GCC Profile enabled: no
> Position Independent Executable enabled: no
> CFLAGS -g -O2 -march=native
> PCAP_CFLAGS -I/usr/include
> SECCFLAGS
>
>
> This is Suricata version 4.1.7 RELEASE
>
> Features: DEBUG PCAP_SET_BUFF PF_RING AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LIBJANSSON TLS MAGIC RUST
> SIMD support: SSE_4_2 SSE_4_1 SSE_3
> Atomic intrisics: 1 2 4 8 16 byte(s)
> 64-bits, Little-endian architecture
> GCC version 6.3.0 20170516, C version 199901
> compiled with _FORTIFY_SOURCE=0
> L1 cache line size (CLS)=64
> thread local storage method: __thread
> compiled with LibHTP v0.5.32, linked against LibHTP v0.5.32
>
> Suricata Configuration:
> AF_PACKET support: yes
> eBPF support: no
> XDP support: no
> PF_RING support: yes
> NFQueue support: no
> NFLOG support: yes
> IPFW support: no
> Netmap support: no
> DAG enabled: no
> Napatech enabled: no
> WinDivert enabled: no
>
> Unix socket enabled: yes
> Detection enabled: yes
>
> Libmagic support: yes
> libnss support: yes
> libnspr support: yes
> libjansson support: yes
> liblzma support: no
> hiredis support: yes
> hiredis async with libevent: no
> Prelude support: yes
> PCRE jit: yes
> LUA support: no
> libluajit: no
> GeoIP support: yes, legacy libgeoip
> Non-bundled htp: no
> Old barnyard2 support: no
> Hyperscan support: yes
> Libnet support: yes
> liblz4 support: no
>
> Rust support: yes
> Rust strict mode: no
> Rust debug mode: no
> Rust compiler: rustc 1.24.1
> Rust cargo: cargo 1.34.0
> Cargo vendor: no
>
> Install suricatasc: yes
> Install suricata-update: yes
>
> Profiling enabled: no
> Profiling locks enabled: no
>
> Development settings:
> Coccinelle / spatch: no
> Unit tests enabled: no
> Debug output enabled: yes
> Debug validation enabled: no
>
> Generic build parameters:
> Installation prefix: /usr/local
> Configuration directory: /usr/local/etc/suricata/
> Log directory: /usr/local/var/log/suricata/
>
> --prefix /usr/local
> --sysconfdir /usr/local/etc
> --localstatedir /usr/local/var
> --datarootdir /usr/local/share
>
> Host: x86_64-pc-linux-gnu
> Compiler: gcc (exec name) / gcc (real)
> GCC Protect enabled: no
> GCC march native enabled: yes
> GCC Profile enabled: no
> Position Independent Executable enabled: no
> CFLAGS -g -O2 -march=native -I${srcdir}/../rust/gen/c-headers
> PCAP_CFLAGS -I/usr/include
> SECCFLAGS
>
>
> Thanks,
>
> Peter
>
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>
> Conference: https://suricon.net
> Trainings: https://suricata-ids.org/training/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20200305/7f417da1/attachment-0001.html>
More information about the Oisf-users
mailing list