[Oisf-users] Suricata 4.1.x possible memory leak (tcp.reassembly_memuse)
d0g d3v
d0gd3v at gmail.com
Tue Mar 10 07:29:05 UTC 2020
*>Since you mentioned the SMB parser, I believe it wouldn't apply in your
case since you don't have Rust enabled. *
Actually I do have Rust enabled for 4.1.7. Old working 4.0.5 does not have
Rust support enabled but 4.1.7 does. SMB parsing is one of the main reasons
for upgrade. Now I tried to compile 4.1.7 without Rust and it seems the
issue is gone. Except there is no required SMB parser. So it seems it is
related to Rust(current version: 1.34.2) somehow.
*> ...prealloc-sessions...*
Unfortunately increasing prealloc-sessions does not help. I went up to
500000 and the behavior is the same.
Thanks,
Peter
On Thu, Mar 5, 2020 at 3:12 PM Eric Urban <eurban at umn.edu> wrote:
> Since you mentioned the SMB parser, I believe it wouldn't apply in your
> case since you don't have Rust enabled.
>
> I looked at our config in the stream section and noticed two options that
> differ quite a bit, which is the prealloc-sessions being set much higher
> (500000) and reassembly.depth is at the default value. From
> https://suricata.readthedocs.io/en/suricata-4.1.6/configuration/suricata-yaml.html?highlight=prealloc%20sessions#stream-engine,
> it is written that "To mitigate Suricata from being overloaded by fast
> session creation, the option prealloc_sessions instructs Suricata to keep a
> number of sessions ready in memory" (Note that there is maybe a typo there
> in "prealloc_sessions" as in our config and in the suricata.yaml example
> from their GitHub repo it is actually prealloc-sessions with a hyphen and
> not an underscore). Though your issue isn't quite what I would expect
> based on the documentation, it is worth a shot to see if this has any
> effect.
>
>
> --
> Eric Urban
> Security Analyst | University Information Security (UIS)
> University of Minnesota | umn.edu
> Information Security is a shared responsibility. Learn more at:
> https://z.umn.edu/uis
>
>
> On Tue, Mar 3, 2020 at 2:59 AM d0g d3v <d0gd3v at gmail.com> wrote:
>
>> Hi Guys,
>>
>>
>> After suricata upgrade from 4.0.5 to 4.1.x(0-7) it seems in my setup I experiencing some memory leak. tcp.reassembly_memuse continuously increasing up to any limit set (40gb in few minutes) and then tcp.segment_memcap_drop are triggered and suricata events drops to almost zero.
>>
>> On the other side, in 4.0.5 there is 15gb stream.reassembly.memcap set and is never reached. tcp.reassembly_memuse is stable around 11gb.
>>
>>
>> My setup:
>>
>> CPU: 56 cores
>>
>> MEM: 64 GB
>>
>> debian 9
>>
>> pfring-zc
>>
>>
>> stream:
>> memcap: 20gb
>> checksum-validation: no # reject wrong csums
>> inline: no # auto will use inline mode in IPS mode, yes or no set it statically
>> prealloc-sessions: 8096
>> bypass: yes
>> midstream: true
>> async-oneside: true
>> reassembly:
>> memcap: 15gb
>> depth: 20mb # reassemble 1mb into a stream
>> toserver-chunk-size: 2560
>> toclient-chunk-size: 2560
>> randomize-chunk-size: yes
>>
>>
>> What I tried:
>>
>> -disable zero copy, change pfring to afpacket - same result
>>
>> -filter-out smb communication (bpf) if the smb parsing introduced in 4.1. is not root cause - same result
>>
>> -significantly decrease stream.reassembly.depth and flow-timeouts - same results
>>
>>
>>
>> I currently can't easy try v5.0.x because of dependencies on the current system. Could you please point me to direction what else I could check?
>>
>> buildinfo for 4.0.5 and then 4.1.7
>>
>> This is Suricata version 4.0.5 RELEASE
>> Features: PCAP_SET_BUFF PF_RING AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LIBJANSSON TLS MAGIC
>> SIMD support: SSE_4_2 SSE_4_1 SSE_3
>> Atomic intrisics: 1 2 4 8 16 byte(s)
>> 64-bits, Little-endian architecture
>> GCC version 6.3.0 20170516, C version 199901
>> compiled with _FORTIFY_SOURCE=0
>> L1 cache line size (CLS)=64
>> thread local storage method: __thread
>> compiled with LibHTP v0.5.27, linked against LibHTP v0.5.27
>>
>> Suricata Configuration:
>> AF_PACKET support: yes
>> PF_RING support: yes
>> NFQueue support: no
>> NFLOG support: yes
>> IPFW support: no
>> Netmap support: no
>> DAG enabled: no
>> Napatech enabled: no
>>
>> Unix socket enabled: yes
>> Detection enabled: yes
>>
>> Libmagic support: yes
>> libnss support: yes
>> libnspr support: yes
>> libjansson support: yes
>> hiredis support: yes
>> hiredis async with libevent: no
>> Prelude support: yes
>> PCRE jit: yes
>> LUA support: no
>> libluajit: no
>> libgeoip: yes
>> Non-bundled htp: no
>> Old barnyard2 support: no
>> CUDA enabled: no
>> Hyperscan support: yes
>> Libnet support: yes
>>
>> Rust support (experimental): no
>> Experimental Rust parsers: no
>> Rust strict mode: no
>>
>> Suricatasc install: yes
>>
>> Profiling enabled: no
>> Profiling locks enabled: no
>>
>> Development settings:
>> Coccinelle / spatch: no
>> Unit tests enabled: no
>> Debug output enabled: no
>> Debug validation enabled: no
>>
>> Generic build parameters:
>> Installation prefix: /usr/local
>> Configuration directory: /usr/local/etc/suricata/
>> Log directory: /usr/local/var/log/suricata/
>>
>> --prefix /usr/local
>> --sysconfdir /usr/local/etc
>> --localstatedir /usr/local/var
>>
>> Host: x86_64-pc-linux-gnu
>> Compiler: gcc (exec name) / gcc (real)
>> GCC Protect enabled: no
>> GCC march native enabled: yes
>> GCC Profile enabled: no
>> Position Independent Executable enabled: no
>> CFLAGS -g -O2 -march=native
>> PCAP_CFLAGS -I/usr/include
>> SECCFLAGS
>>
>>
>> This is Suricata version 4.1.7 RELEASE
>>
>> Features: DEBUG PCAP_SET_BUFF PF_RING AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LIBJANSSON TLS MAGIC RUST
>> SIMD support: SSE_4_2 SSE_4_1 SSE_3
>> Atomic intrisics: 1 2 4 8 16 byte(s)
>> 64-bits, Little-endian architecture
>> GCC version 6.3.0 20170516, C version 199901
>> compiled with _FORTIFY_SOURCE=0
>> L1 cache line size (CLS)=64
>> thread local storage method: __thread
>> compiled with LibHTP v0.5.32, linked against LibHTP v0.5.32
>>
>> Suricata Configuration:
>> AF_PACKET support: yes
>> eBPF support: no
>> XDP support: no
>> PF_RING support: yes
>> NFQueue support: no
>> NFLOG support: yes
>> IPFW support: no
>> Netmap support: no
>> DAG enabled: no
>> Napatech enabled: no
>> WinDivert enabled: no
>>
>> Unix socket enabled: yes
>> Detection enabled: yes
>>
>> Libmagic support: yes
>> libnss support: yes
>> libnspr support: yes
>> libjansson support: yes
>> liblzma support: no
>> hiredis support: yes
>> hiredis async with libevent: no
>> Prelude support: yes
>> PCRE jit: yes
>> LUA support: no
>> libluajit: no
>> GeoIP support: yes, legacy libgeoip
>> Non-bundled htp: no
>> Old barnyard2 support: no
>> Hyperscan support: yes
>> Libnet support: yes
>> liblz4 support: no
>>
>> Rust support: yes
>> Rust strict mode: no
>> Rust debug mode: no
>> Rust compiler: rustc 1.24.1
>> Rust cargo: cargo 1.34.0
>> Cargo vendor: no
>>
>> Install suricatasc: yes
>> Install suricata-update: yes
>>
>> Profiling enabled: no
>> Profiling locks enabled: no
>>
>> Development settings:
>> Coccinelle / spatch: no
>> Unit tests enabled: no
>> Debug output enabled: yes
>> Debug validation enabled: no
>>
>> Generic build parameters:
>> Installation prefix: /usr/local
>> Configuration directory: /usr/local/etc/suricata/
>> Log directory: /usr/local/var/log/suricata/
>>
>> --prefix /usr/local
>> --sysconfdir /usr/local/etc
>> --localstatedir /usr/local/var
>> --datarootdir /usr/local/share
>>
>> Host: x86_64-pc-linux-gnu
>> Compiler: gcc (exec name) / gcc (real)
>> GCC Protect enabled: no
>> GCC march native enabled: yes
>> GCC Profile enabled: no
>> Position Independent Executable enabled: no
>> CFLAGS -g -O2 -march=native -I${srcdir}/../rust/gen/c-headers
>> PCAP_CFLAGS -I/usr/include
>> SECCFLAGS
>>
>>
>> Thanks,
>>
>> Peter
>>
>> _______________________________________________
>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
>> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>
>> Conference: https://suricon.net
>> Trainings: https://suricata-ids.org/training/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20200310/b93e0688/attachment-0001.html>
More information about the Oisf-users
mailing list