[Oisf-users] Suricata 4.1.x possible memory leak (tcp.reassembly_memuse)
d0g d3v
d0gd3v at gmail.com
Wed Apr 1 06:35:26 UTC 2020
I tried Rust 1.42 with the same result. So I reverted it back on default
deb10 1.34.2. Just for a test I put another 64GB RAM to the server (128GB
total). I set stream.reassembly.memcap to 90GB. tcp.reassembly_memuse in
this setup is between 75GB-90GB, sometimes even hits the reassembly memcap.
Do you think that the new feature (rust with SMB parser) could cause such
issue? Increasing tcp.reassembly_memuse from stable ~11GB on 4.0.5 (with
much higher reassembly depth and flow timeouts) to unstable ~90GB on 4.1.7?
Thanks,
Peter
On Tue, Mar 10, 2020 at 8:45 AM Peter Manev <petermanev at gmail.com> wrote:
> On Tue, Mar 10, 2020 at 8:29 AM d0g d3v <d0gd3v at gmail.com> wrote:
> >
> > >Since you mentioned the SMB parser, I believe it wouldn't apply in your
> case since you don't have Rust enabled.
> > Actually I do have Rust enabled for 4.1.7. Old working 4.0.5 does not
> have Rust support enabled but 4.1.7 does. SMB parsing is one of the main
> reasons for upgrade. Now I tried to compile 4.1.7 without Rust and it seems
> the issue is gone. Except there is no required SMB parser. So it seems it
> is related to Rust(current version: 1.34.2) somehow.
> >
>
> Can you try compiling it with the latest Rust ("rustup update" if you
> have installed rustup , then do a compile - should do it) and see if
> it makes a difference?
> Thanks
>
> > > ...prealloc-sessions...
> > Unfortunately increasing prealloc-sessions does not help. I went up to
> 500000 and the behavior is the same.
> >
> > Thanks,
> >
> > Peter
> >
> >
> > On Thu, Mar 5, 2020 at 3:12 PM Eric Urban <eurban at umn.edu> wrote:
> >>
> >> Since you mentioned the SMB parser, I believe it wouldn't apply in your
> case since you don't have Rust enabled.
> >>
> >> I looked at our config in the stream section and noticed two options
> that differ quite a bit, which is the prealloc-sessions being set much
> higher (500000) and reassembly.depth is at the default value. From
> https://suricata.readthedocs.io/en/suricata-4.1.6/configuration/suricata-yaml.html?highlight=prealloc%20sessions#stream-engine,
> it is written that "To mitigate Suricata from being overloaded by fast
> session creation, the option prealloc_sessions instructs Suricata to keep a
> number of sessions ready in memory" (Note that there is maybe a typo there
> in "prealloc_sessions" as in our config and in the suricata.yaml example
> from their GitHub repo it is actually prealloc-sessions with a hyphen and
> not an underscore). Though your issue isn't quite what I would expect
> based on the documentation, it is worth a shot to see if this has any
> effect.
> >>
> >>
> >> --
> >> Eric Urban
> >> Security Analyst | University Information Security (UIS)
> >> University of Minnesota | umn.edu
> >> Information Security is a shared responsibility. Learn more at:
> https://z.umn.edu/uis
> >>
> >>
> >> On Tue, Mar 3, 2020 at 2:59 AM d0g d3v <d0gd3v at gmail.com> wrote:
> >>>
> >>> Hi Guys,
> >>>
> >>>
> >>> After suricata upgrade from 4.0.5 to 4.1.x(0-7) it seems in my setup I
> experiencing some memory leak. tcp.reassembly_memuse continuously
> increasing up to any limit set (40gb in few minutes) and then
> tcp.segment_memcap_drop are triggered and suricata events drops to almost
> zero.
> >>>
> >>> On the other side, in 4.0.5 there is 15gb stream.reassembly.memcap set
> and is never reached. tcp.reassembly_memuse is stable around 11gb.
> >>>
> >>>
> >>> My setup:
> >>>
> >>> CPU: 56 cores
> >>>
> >>> MEM: 64 GB
> >>>
> >>> debian 9
> >>>
> >>> pfring-zc
> >>>
> >>>
> >>> stream:
> >>> memcap: 20gb
> >>> checksum-validation: no # reject wrong csums
> >>> inline: no # auto will use inline mode in IPS
> mode, yes or no set it statically
> >>> prealloc-sessions: 8096
> >>> bypass: yes
> >>> midstream: true
> >>> async-oneside: true
> >>> reassembly:
> >>> memcap: 15gb
> >>> depth: 20mb # reassemble 1mb into a stream
> >>> toserver-chunk-size: 2560
> >>> toclient-chunk-size: 2560
> >>> randomize-chunk-size: yes
> >>>
> >>>
> >>> What I tried:
> >>>
> >>> -disable zero copy, change pfring to afpacket - same result
> >>>
> >>> -filter-out smb communication (bpf) if the smb parsing introduced in
> 4.1. is not root cause - same result
> >>>
> >>> -significantly decrease stream.reassembly.depth and flow-timeouts -
> same results
> >>>
> >>>
> >>>
> >>> I currently can't easy try v5.0.x because of dependencies on the
> current system. Could you please point me to direction what else I could
> check?
> >>>
> >>> buildinfo for 4.0.5 and then 4.1.7
> >>>
> >>> This is Suricata version 4.0.5 RELEASE
> >>> Features: PCAP_SET_BUFF PF_RING AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG
> LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LIBJANSSON TLS
> MAGIC
> >>> SIMD support: SSE_4_2 SSE_4_1 SSE_3
> >>> Atomic intrisics: 1 2 4 8 16 byte(s)
> >>> 64-bits, Little-endian architecture
> >>> GCC version 6.3.0 20170516, C version 199901
> >>> compiled with _FORTIFY_SOURCE=0
> >>> L1 cache line size (CLS)=64
> >>> thread local storage method: __thread
> >>> compiled with LibHTP v0.5.27, linked against LibHTP v0.5.27
> >>>
> >>> Suricata Configuration:
> >>> AF_PACKET support: yes
> >>> PF_RING support: yes
> >>> NFQueue support: no
> >>> NFLOG support: yes
> >>> IPFW support: no
> >>> Netmap support: no
> >>> DAG enabled: no
> >>> Napatech enabled: no
> >>>
> >>> Unix socket enabled: yes
> >>> Detection enabled: yes
> >>>
> >>> Libmagic support: yes
> >>> libnss support: yes
> >>> libnspr support: yes
> >>> libjansson support: yes
> >>> hiredis support: yes
> >>> hiredis async with libevent: no
> >>> Prelude support: yes
> >>> PCRE jit: yes
> >>> LUA support: no
> >>> libluajit: no
> >>> libgeoip: yes
> >>> Non-bundled htp: no
> >>> Old barnyard2 support: no
> >>> CUDA enabled: no
> >>> Hyperscan support: yes
> >>> Libnet support: yes
> >>>
> >>> Rust support (experimental): no
> >>> Experimental Rust parsers: no
> >>> Rust strict mode: no
> >>>
> >>> Suricatasc install: yes
> >>>
> >>> Profiling enabled: no
> >>> Profiling locks enabled: no
> >>>
> >>> Development settings:
> >>> Coccinelle / spatch: no
> >>> Unit tests enabled: no
> >>> Debug output enabled: no
> >>> Debug validation enabled: no
> >>>
> >>> Generic build parameters:
> >>> Installation prefix: /usr/local
> >>> Configuration directory: /usr/local/etc/suricata/
> >>> Log directory: /usr/local/var/log/suricata/
> >>>
> >>> --prefix /usr/local
> >>> --sysconfdir /usr/local/etc
> >>> --localstatedir /usr/local/var
> >>>
> >>> Host: x86_64-pc-linux-gnu
> >>> Compiler: gcc (exec name) / gcc (real)
> >>> GCC Protect enabled: no
> >>> GCC march native enabled: yes
> >>> GCC Profile enabled: no
> >>> Position Independent Executable enabled: no
> >>> CFLAGS -g -O2 -march=native
> >>> PCAP_CFLAGS -I/usr/include
> >>> SECCFLAGS
> >>>
> >>>
> >>> This is Suricata version 4.1.7 RELEASE
> >>>
> >>> Features: DEBUG PCAP_SET_BUFF PF_RING AF_PACKET HAVE_PACKET_FANOUT
> LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS
> HAVE_LIBJANSSON TLS MAGIC RUST
> >>> SIMD support: SSE_4_2 SSE_4_1 SSE_3
> >>> Atomic intrisics: 1 2 4 8 16 byte(s)
> >>> 64-bits, Little-endian architecture
> >>> GCC version 6.3.0 20170516, C version 199901
> >>> compiled with _FORTIFY_SOURCE=0
> >>> L1 cache line size (CLS)=64
> >>> thread local storage method: __thread
> >>> compiled with LibHTP v0.5.32, linked against LibHTP v0.5.32
> >>>
> >>> Suricata Configuration:
> >>> AF_PACKET support: yes
> >>> eBPF support: no
> >>> XDP support: no
> >>> PF_RING support: yes
> >>> NFQueue support: no
> >>> NFLOG support: yes
> >>> IPFW support: no
> >>> Netmap support: no
> >>> DAG enabled: no
> >>> Napatech enabled: no
> >>> WinDivert enabled: no
> >>>
> >>> Unix socket enabled: yes
> >>> Detection enabled: yes
> >>>
> >>> Libmagic support: yes
> >>> libnss support: yes
> >>> libnspr support: yes
> >>> libjansson support: yes
> >>> liblzma support: no
> >>> hiredis support: yes
> >>> hiredis async with libevent: no
> >>> Prelude support: yes
> >>> PCRE jit: yes
> >>> LUA support: no
> >>> libluajit: no
> >>> GeoIP support: yes, legacy libgeoip
> >>> Non-bundled htp: no
> >>> Old barnyard2 support: no
> >>> Hyperscan support: yes
> >>> Libnet support: yes
> >>> liblz4 support: no
> >>>
> >>> Rust support: yes
> >>> Rust strict mode: no
> >>> Rust debug mode: no
> >>> Rust compiler: rustc 1.24.1
> >>> Rust cargo: cargo 1.34.0
> >>> Cargo vendor: no
> >>>
> >>> Install suricatasc: yes
> >>> Install suricata-update: yes
> >>>
> >>> Profiling enabled: no
> >>> Profiling locks enabled: no
> >>>
> >>> Development settings:
> >>> Coccinelle / spatch: no
> >>> Unit tests enabled: no
> >>> Debug output enabled: yes
> >>> Debug validation enabled: no
> >>>
> >>> Generic build parameters:
> >>> Installation prefix: /usr/local
> >>> Configuration directory: /usr/local/etc/suricata/
> >>> Log directory: /usr/local/var/log/suricata/
> >>>
> >>> --prefix /usr/local
> >>> --sysconfdir /usr/local/etc
> >>> --localstatedir /usr/local/var
> >>> --datarootdir /usr/local/share
> >>>
> >>> Host: x86_64-pc-linux-gnu
> >>> Compiler: gcc (exec name) / gcc (real)
> >>> GCC Protect enabled: no
> >>> GCC march native enabled: yes
> >>> GCC Profile enabled: no
> >>> Position Independent Executable enabled: no
> >>> CFLAGS -g -O2 -march=native
> -I${srcdir}/../rust/gen/c-headers
> >>> PCAP_CFLAGS -I/usr/include
> >>> SECCFLAGS
> >>>
> >>>
> >>> Thanks,
> >>>
> >>> Peter
> >>>
> >>> _______________________________________________
> >>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> >>> Site: http://suricata-ids.org | Support:
> http://suricata-ids.org/support/
> >>> List:
> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> >>>
> >>> Conference: https://suricon.net
> >>> Trainings: https://suricata-ids.org/training/
> >
> > _______________________________________________
> > Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> > Site: http://suricata-ids.org | Support:
> http://suricata-ids.org/support/
> > List:
> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> >
> > Conference: https://suricon.net
> > Trainings: https://suricata-ids.org/training/
>
>
>
> --
> Regards,
> Peter Manev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20200401/45917265/attachment-0001.html>
More information about the Oisf-users
mailing list