<div dir="ltr"><i>>Since you mentioned the SMB parser, I believe it wouldn't apply in your case since you don't have Rust enabled. </i><br><div>Actually I do have Rust enabled for 4.1.7. Old working 4.0.5 does not have Rust support enabled but 4.1.7 does. SMB parsing is one of the main reasons for upgrade. Now I tried to compile 4.1.7 without Rust and it seems the issue is gone. Except there is no required SMB parser. So it seems it is related to Rust(current version: 1.34.2) somehow.</div><div><br></div><div><i>> ...prealloc-sessions...</i></div><div>Unfortunately increasing prealloc-sessions does not help. I went up to 500000 and the behavior is the same.<br></div><div><br></div><div>Thanks,</div><div><br></div><div>Peter</div><div><i><br></i></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 5, 2020 at 3:12 PM Eric Urban <<a href="mailto:eurban@umn.edu" target="_blank">eurban@umn.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Since you mentioned the SMB parser, I believe it wouldn't apply in your case since you don't have Rust enabled.<br></div><div><br></div><div>I looked at our config in the stream section and noticed two options that differ quite a bit, which is the prealloc-sessions being set much higher (500000) and reassembly.depth is at the default value. From <a href="https://suricata.readthedocs.io/en/suricata-4.1.6/configuration/suricata-yaml.html?highlight=prealloc%20sessions#stream-engine" target="_blank">https://suricata.readthedocs.io/en/suricata-4.1.6/configuration/suricata-yaml.html?highlight=prealloc%20sessions#stream-engine</a>, it is written that "To mitigate Suricata from being overloaded by fast session creation, the option prealloc_sessions instructs Suricata to keep a number of sessions ready in memory" (Note that there is maybe a typo there in "prealloc_sessions" as in our config and in the suricata.yaml example from their GitHub repo it is actually prealloc-sessions with a hyphen and not an underscore). Though your issue isn't quite what I would expect based on the documentation, it is worth a shot to see if this has any effect.<br></div><div><br></div><div><br></div><div><div><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><span>-- <br></span></div><div dir="ltr"><span>Eric Urban</span><br><span></span><div dir="ltr"><span> Security Analyst | University Information Security (UIS)<br> University of Minnesota | <a href="http://umn.edu" target="_blank">umn.edu</a><br>Information Security is a shared responsibility. Learn more at: <a href="https://z.umn.edu/uis" target="_blank">https://z.umn.edu/uis</a><br></span></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Mar 3, 2020 at 2:59 AM d0g d3v <<a href="mailto:d0gd3v@gmail.com" target="_blank">d0gd3v@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><pre style="white-space:pre-wrap;color:rgb(0,0,0)">Hi Guys,</pre><pre style="white-space:pre-wrap;color:rgb(0,0,0)"><br></pre><pre style="white-space:pre-wrap;color:rgb(0,0,0)">After suricata upgrade from 4.0.5 to 4.1.x(0-7) it seems in my setup I experiencing some memory leak. tcp.reassembly_memuse continuously increasing up to any limit set (40gb in few minutes) and then tcp.segment_memcap_drop are triggered and suricata events drops to almost zero. </pre><pre style="white-space:pre-wrap;color:rgb(0,0,0)">On the other side, in 4.0.5 there is 15gb stream.reassembly.memcap set and is never reached. tcp.reassembly_memuse is stable around 11gb.</pre><pre style="white-space:pre-wrap;color:rgb(0,0,0)"><br></pre><pre style="white-space:pre-wrap;color:rgb(0,0,0)">My setup:</pre><pre style="white-space:pre-wrap;color:rgb(0,0,0)"><span style="font-family:Arial,Helvetica,sans-serif">CPU: 56 cores</span><br></pre><pre style="white-space:pre-wrap;color:rgb(0,0,0)">MEM: 64 GB</pre><pre style="white-space:pre-wrap;color:rgb(0,0,0)">debian 9</pre><pre style="white-space:pre-wrap;color:rgb(0,0,0)">pfring-zc</pre><pre style="white-space:pre-wrap;color:rgb(0,0,0)"><br></pre><pre style="white-space:pre-wrap;color:rgb(0,0,0)">stream:<br> memcap: 20gb<br> checksum-validation: no # reject wrong csums<br> inline: no # auto will use inline mode in IPS mode, yes or no set it statically<br> prealloc-sessions: 8096<br> bypass: yes<br> midstream: true<br> async-oneside: true<br> reassembly:<br> memcap: 15gb<br> depth: 20mb # reassemble 1mb into a stream<br> toserver-chunk-size: 2560<br> toclient-chunk-size: 2560<br> randomize-chunk-size: yes<br></pre><pre style="white-space:pre-wrap;color:rgb(0,0,0)"><br></pre><pre style="white-space:pre-wrap;color:rgb(0,0,0)">What I tried:</pre><pre style="white-space:pre-wrap;color:rgb(0,0,0)">-disable zero copy, change pfring to afpacket - same result</pre><pre style="white-space:pre-wrap;color:rgb(0,0,0)">-filter-out smb communication (bpf) if the smb parsing introduced in 4.1. is not root cause - same result</pre><pre style="white-space:pre-wrap;color:rgb(0,0,0)">-significantly decrease stream.reassembly.depth and flow-timeouts - same results</pre><pre style="white-space:pre-wrap;color:rgb(0,0,0)"> </pre><pre style="white-space:pre-wrap;color:rgb(0,0,0)">I currently can't easy try v5.0.x because of dependencies on the current system. Could you please point me to direction what else I could check?</pre><pre style="white-space:pre-wrap;color:rgb(0,0,0)">buildinfo for 4.0.5 and then 4.1.7</pre><pre style="white-space:pre-wrap;color:rgb(0,0,0)">This is Suricata version 4.0.5 RELEASE<br>Features: PCAP_SET_BUFF PF_RING AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LIBJANSSON TLS MAGIC<br>SIMD support: SSE_4_2 SSE_4_1 SSE_3<br>Atomic intrisics: 1 2 4 8 16 byte(s)<br>64-bits, Little-endian architecture<br>GCC version 6.3.0 20170516, C version 199901<br>compiled with _FORTIFY_SOURCE=0<br>L1 cache line size (CLS)=64<br>thread local storage method: __thread<br>compiled with LibHTP v0.5.27, linked against LibHTP v0.5.27<br><br>Suricata Configuration:<br> AF_PACKET support: yes<br> PF_RING support: yes<br> NFQueue support: no<br> NFLOG support: yes<br> IPFW support: no<br> Netmap support: no<br> DAG enabled: no<br> Napatech enabled: no<br><br> Unix socket enabled: yes<br> Detection enabled: yes<br><br> Libmagic support: yes<br> libnss support: yes<br> libnspr support: yes<br> libjansson support: yes<br> hiredis support: yes<br> hiredis async with libevent: no<br> Prelude support: yes<br> PCRE jit: yes<br> LUA support: no<br> libluajit: no<br> libgeoip: yes<br> Non-bundled htp: no<br> Old barnyard2 support: no<br> CUDA enabled: no<br> Hyperscan support: yes<br> Libnet support: yes<br><br> Rust support (experimental): no<br> Experimental Rust parsers: no<br> Rust strict mode: no<br><br> Suricatasc install: yes<br><br> Profiling enabled: no<br> Profiling locks enabled: no<br><br>Development settings:<br> Coccinelle / spatch: no<br> Unit tests enabled: no<br> Debug output enabled: no<br> Debug validation enabled: no<br><br>Generic build parameters:<br> Installation prefix: /usr/local<br> Configuration directory: /usr/local/etc/suricata/<br> Log directory: /usr/local/var/log/suricata/<br><br> --prefix /usr/local<br> --sysconfdir /usr/local/etc<br> --localstatedir /usr/local/var<br><br> Host: x86_64-pc-linux-gnu<br> Compiler: gcc (exec name) / gcc (real)<br> GCC Protect enabled: no<br> GCC march native enabled: yes<br> GCC Profile enabled: no<br> Position Independent Executable enabled: no<br> CFLAGS -g -O2 -march=native<br> PCAP_CFLAGS -I/usr/include<br> SECCFLAGS<br></pre><pre style="white-space:pre-wrap;color:rgb(0,0,0)"><br></pre><pre style="white-space:pre-wrap;color:rgb(0,0,0)">This is Suricata version 4.1.7 RELEASE</pre><pre style="white-space:pre-wrap;color:rgb(0,0,0)">Features: DEBUG PCAP_SET_BUFF PF_RING AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LIBJANSSON TLS MAGIC RUST<br>SIMD support: SSE_4_2 SSE_4_1 SSE_3<br>Atomic intrisics: 1 2 4 8 16 byte(s)<br>64-bits, Little-endian architecture<br>GCC version 6.3.0 20170516, C version 199901<br>compiled with _FORTIFY_SOURCE=0<br>L1 cache line size (CLS)=64<br>thread local storage method: __thread<br>compiled with LibHTP v0.5.32, linked against LibHTP v0.5.32<br><br>Suricata Configuration:<br> AF_PACKET support: yes<br> eBPF support: no<br> XDP support: no<br> PF_RING support: yes<br> NFQueue support: no<br> NFLOG support: yes<br> IPFW support: no<br> Netmap support: no<br> DAG enabled: no<br> Napatech enabled: no<br> WinDivert enabled: no<br><br> Unix socket enabled: yes<br> Detection enabled: yes<br><br> Libmagic support: yes<br> libnss support: yes<br> libnspr support: yes<br> libjansson support: yes<br> liblzma support: no<br> hiredis support: yes<br> hiredis async with libevent: no<br> Prelude support: yes<br> PCRE jit: yes<br> LUA support: no<br> libluajit: no<br> GeoIP support: yes, legacy libgeoip<br> Non-bundled htp: no<br> Old barnyard2 support: no<br> Hyperscan support: yes<br> Libnet support: yes<br> liblz4 support: no<br><br> Rust support: yes<br> Rust strict mode: no<br> Rust debug mode: no<br> Rust compiler: rustc 1.24.1<br> Rust cargo: cargo 1.34.0<br> Cargo vendor: no<br><br> Install suricatasc: yes<br> Install suricata-update: yes<br><br> Profiling enabled: no<br> Profiling locks enabled: no<br><br>Development settings:<br> Coccinelle / spatch: no<br> Unit tests enabled: no<br> Debug output enabled: yes<br> Debug validation enabled: no<br><br>Generic build parameters:<br> Installation prefix: /usr/local<br> Configuration directory: /usr/local/etc/suricata/<br> Log directory: /usr/local/var/log/suricata/<br><br> --prefix /usr/local<br> --sysconfdir /usr/local/etc<br> --localstatedir /usr/local/var<br> --datarootdir /usr/local/share<br><br> Host: x86_64-pc-linux-gnu<br> Compiler: gcc (exec name) / gcc (real)<br> GCC Protect enabled: no<br> GCC march native enabled: yes<br> GCC Profile enabled: no<br> Position Independent Executable enabled: no<br> CFLAGS -g -O2 -march=native -I${srcdir}/../rust/gen/c-headers<br> PCAP_CFLAGS -I/usr/include<br> SECCFLAGS<br></pre><pre style="white-space:pre-wrap;color:rgb(0,0,0)"><br></pre><pre style="white-space:pre-wrap;color:rgb(0,0,0)">Thanks,</pre><pre style="white-space:pre-wrap;color:rgb(0,0,0)">Peter</pre></div>
_______________________________________________<br>
Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org" target="_blank">oisf-users@openinfosecfoundation.org</a><br>
Site: <a href="http://suricata-ids.org" rel="noreferrer" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" rel="noreferrer" target="_blank">http://suricata-ids.org/support/</a><br>
List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" rel="noreferrer" target="_blank">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
<br>
Conference: <a href="https://suricon.net" rel="noreferrer" target="_blank">https://suricon.net</a><br>
Trainings: <a href="https://suricata-ids.org/training/" rel="noreferrer" target="_blank">https://suricata-ids.org/training/</a></blockquote></div>
</blockquote></div>