[Oisf-users] Suricata 4.1.x possible memory leak (tcp.reassembly_memuse)
Peter Manev
petermanev at gmail.com
Tue Mar 10 07:45:07 UTC 2020
On Tue, Mar 10, 2020 at 8:29 AM d0g d3v <d0gd3v at gmail.com> wrote:
>
> >Since you mentioned the SMB parser, I believe it wouldn't apply in your case since you don't have Rust enabled.
> Actually I do have Rust enabled for 4.1.7. Old working 4.0.5 does not have Rust support enabled but 4.1.7 does. SMB parsing is one of the main reasons for upgrade. Now I tried to compile 4.1.7 without Rust and it seems the issue is gone. Except there is no required SMB parser. So it seems it is related to Rust(current version: 1.34.2) somehow.
>
Can you try compiling it with the latest Rust ("rustup update" if you
have installed rustup , then do a compile - should do it) and see if
it makes a difference?
Thanks
> > ...prealloc-sessions...
> Unfortunately increasing prealloc-sessions does not help. I went up to 500000 and the behavior is the same.
>
> Thanks,
>
> Peter
>
>
> On Thu, Mar 5, 2020 at 3:12 PM Eric Urban <eurban at umn.edu> wrote:
>>
>> Since you mentioned the SMB parser, I believe it wouldn't apply in your case since you don't have Rust enabled.
>>
>> I looked at our config in the stream section and noticed two options that differ quite a bit, which is the prealloc-sessions being set much higher (500000) and reassembly.depth is at the default value. From https://suricata.readthedocs.io/en/suricata-4.1.6/configuration/suricata-yaml.html?highlight=prealloc%20sessions#stream-engine, it is written that "To mitigate Suricata from being overloaded by fast session creation, the option prealloc_sessions instructs Suricata to keep a number of sessions ready in memory" (Note that there is maybe a typo there in "prealloc_sessions" as in our config and in the suricata.yaml example from their GitHub repo it is actually prealloc-sessions with a hyphen and not an underscore). Though your issue isn't quite what I would expect based on the documentation, it is worth a shot to see if this has any effect.
>>
>>
>> --
>> Eric Urban
>> Security Analyst | University Information Security (UIS)
>> University of Minnesota | umn.edu
>> Information Security is a shared responsibility. Learn more at: https://z.umn.edu/uis
>>
>>
>> On Tue, Mar 3, 2020 at 2:59 AM d0g d3v <d0gd3v at gmail.com> wrote:
>>>
>>> Hi Guys,
>>>
>>>
>>> After suricata upgrade from 4.0.5 to 4.1.x(0-7) it seems in my setup I experiencing some memory leak. tcp.reassembly_memuse continuously increasing up to any limit set (40gb in few minutes) and then tcp.segment_memcap_drop are triggered and suricata events drops to almost zero.
>>>
>>> On the other side, in 4.0.5 there is 15gb stream.reassembly.memcap set and is never reached. tcp.reassembly_memuse is stable around 11gb.
>>>
>>>
>>> My setup:
>>>
>>> CPU: 56 cores
>>>
>>> MEM: 64 GB
>>>
>>> debian 9
>>>
>>> pfring-zc
>>>
>>>
>>> stream:
>>> memcap: 20gb
>>> checksum-validation: no # reject wrong csums
>>> inline: no # auto will use inline mode in IPS mode, yes or no set it statically
>>> prealloc-sessions: 8096
>>> bypass: yes
>>> midstream: true
>>> async-oneside: true
>>> reassembly:
>>> memcap: 15gb
>>> depth: 20mb # reassemble 1mb into a stream
>>> toserver-chunk-size: 2560
>>> toclient-chunk-size: 2560
>>> randomize-chunk-size: yes
>>>
>>>
>>> What I tried:
>>>
>>> -disable zero copy, change pfring to afpacket - same result
>>>
>>> -filter-out smb communication (bpf) if the smb parsing introduced in 4.1. is not root cause - same result
>>>
>>> -significantly decrease stream.reassembly.depth and flow-timeouts - same results
>>>
>>>
>>>
>>> I currently can't easy try v5.0.x because of dependencies on the current system. Could you please point me to direction what else I could check?
>>>
>>> buildinfo for 4.0.5 and then 4.1.7
>>>
>>> This is Suricata version 4.0.5 RELEASE
>>> Features: PCAP_SET_BUFF PF_RING AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LIBJANSSON TLS MAGIC
>>> SIMD support: SSE_4_2 SSE_4_1 SSE_3
>>> Atomic intrisics: 1 2 4 8 16 byte(s)
>>> 64-bits, Little-endian architecture
>>> GCC version 6.3.0 20170516, C version 199901
>>> compiled with _FORTIFY_SOURCE=0
>>> L1 cache line size (CLS)=64
>>> thread local storage method: __thread
>>> compiled with LibHTP v0.5.27, linked against LibHTP v0.5.27
>>>
>>> Suricata Configuration:
>>> AF_PACKET support: yes
>>> PF_RING support: yes
>>> NFQueue support: no
>>> NFLOG support: yes
>>> IPFW support: no
>>> Netmap support: no
>>> DAG enabled: no
>>> Napatech enabled: no
>>>
>>> Unix socket enabled: yes
>>> Detection enabled: yes
>>>
>>> Libmagic support: yes
>>> libnss support: yes
>>> libnspr support: yes
>>> libjansson support: yes
>>> hiredis support: yes
>>> hiredis async with libevent: no
>>> Prelude support: yes
>>> PCRE jit: yes
>>> LUA support: no
>>> libluajit: no
>>> libgeoip: yes
>>> Non-bundled htp: no
>>> Old barnyard2 support: no
>>> CUDA enabled: no
>>> Hyperscan support: yes
>>> Libnet support: yes
>>>
>>> Rust support (experimental): no
>>> Experimental Rust parsers: no
>>> Rust strict mode: no
>>>
>>> Suricatasc install: yes
>>>
>>> Profiling enabled: no
>>> Profiling locks enabled: no
>>>
>>> Development settings:
>>> Coccinelle / spatch: no
>>> Unit tests enabled: no
>>> Debug output enabled: no
>>> Debug validation enabled: no
>>>
>>> Generic build parameters:
>>> Installation prefix: /usr/local
>>> Configuration directory: /usr/local/etc/suricata/
>>> Log directory: /usr/local/var/log/suricata/
>>>
>>> --prefix /usr/local
>>> --sysconfdir /usr/local/etc
>>> --localstatedir /usr/local/var
>>>
>>> Host: x86_64-pc-linux-gnu
>>> Compiler: gcc (exec name) / gcc (real)
>>> GCC Protect enabled: no
>>> GCC march native enabled: yes
>>> GCC Profile enabled: no
>>> Position Independent Executable enabled: no
>>> CFLAGS -g -O2 -march=native
>>> PCAP_CFLAGS -I/usr/include
>>> SECCFLAGS
>>>
>>>
>>> This is Suricata version 4.1.7 RELEASE
>>>
>>> Features: DEBUG PCAP_SET_BUFF PF_RING AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LIBJANSSON TLS MAGIC RUST
>>> SIMD support: SSE_4_2 SSE_4_1 SSE_3
>>> Atomic intrisics: 1 2 4 8 16 byte(s)
>>> 64-bits, Little-endian architecture
>>> GCC version 6.3.0 20170516, C version 199901
>>> compiled with _FORTIFY_SOURCE=0
>>> L1 cache line size (CLS)=64
>>> thread local storage method: __thread
>>> compiled with LibHTP v0.5.32, linked against LibHTP v0.5.32
>>>
>>> Suricata Configuration:
>>> AF_PACKET support: yes
>>> eBPF support: no
>>> XDP support: no
>>> PF_RING support: yes
>>> NFQueue support: no
>>> NFLOG support: yes
>>> IPFW support: no
>>> Netmap support: no
>>> DAG enabled: no
>>> Napatech enabled: no
>>> WinDivert enabled: no
>>>
>>> Unix socket enabled: yes
>>> Detection enabled: yes
>>>
>>> Libmagic support: yes
>>> libnss support: yes
>>> libnspr support: yes
>>> libjansson support: yes
>>> liblzma support: no
>>> hiredis support: yes
>>> hiredis async with libevent: no
>>> Prelude support: yes
>>> PCRE jit: yes
>>> LUA support: no
>>> libluajit: no
>>> GeoIP support: yes, legacy libgeoip
>>> Non-bundled htp: no
>>> Old barnyard2 support: no
>>> Hyperscan support: yes
>>> Libnet support: yes
>>> liblz4 support: no
>>>
>>> Rust support: yes
>>> Rust strict mode: no
>>> Rust debug mode: no
>>> Rust compiler: rustc 1.24.1
>>> Rust cargo: cargo 1.34.0
>>> Cargo vendor: no
>>>
>>> Install suricatasc: yes
>>> Install suricata-update: yes
>>>
>>> Profiling enabled: no
>>> Profiling locks enabled: no
>>>
>>> Development settings:
>>> Coccinelle / spatch: no
>>> Unit tests enabled: no
>>> Debug output enabled: yes
>>> Debug validation enabled: no
>>>
>>> Generic build parameters:
>>> Installation prefix: /usr/local
>>> Configuration directory: /usr/local/etc/suricata/
>>> Log directory: /usr/local/var/log/suricata/
>>>
>>> --prefix /usr/local
>>> --sysconfdir /usr/local/etc
>>> --localstatedir /usr/local/var
>>> --datarootdir /usr/local/share
>>>
>>> Host: x86_64-pc-linux-gnu
>>> Compiler: gcc (exec name) / gcc (real)
>>> GCC Protect enabled: no
>>> GCC march native enabled: yes
>>> GCC Profile enabled: no
>>> Position Independent Executable enabled: no
>>> CFLAGS -g -O2 -march=native -I${srcdir}/../rust/gen/c-headers
>>> PCAP_CFLAGS -I/usr/include
>>> SECCFLAGS
>>>
>>>
>>> Thanks,
>>>
>>> Peter
>>>
>>> _______________________________________________
>>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>>> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
>>> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>
>>> Conference: https://suricon.net
>>> Trainings: https://suricata-ids.org/training/
>
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>
> Conference: https://suricon.net
> Trainings: https://suricata-ids.org/training/
--
Regards,
Peter Manev
More information about the Oisf-users
mailing list