<div dir="ltr">I tried Rust 1.42 with the same result. So I reverted it back on default deb10 1.34.2. Just for a test I put another 64GB RAM to the server (128GB total). I set <span style="color:rgb(0,0,0);white-space:pre-wrap">stream.reassembly.memcap to 90GB. </span><span style="color:rgb(0,0,0);white-space:pre-wrap">tcp.reassembly_memuse in this setup is between 75GB-90GB, sometimes even hits the reassembly memcap. </span><div>Do you think that the new feature (rust with SMB parser) could cause such issue? Increasing tcp.reassembly_memuse from stable ~11GB on 4.0.5 (with much higher reassembly depth and flow timeouts) to unstable ~90GB on 4.1.7?</div><div><br></div><div>Thanks,</div><div><br></div><div>Peter</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Mar 10, 2020 at 8:45 AM Peter Manev <<a href="mailto:petermanev@gmail.com" target="_blank">petermanev@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Tue, Mar 10, 2020 at 8:29 AM d0g d3v <<a href="mailto:d0gd3v@gmail.com" target="_blank">d0gd3v@gmail.com</a>> wrote:<br>
><br>
> >Since you mentioned the SMB parser, I believe it wouldn't apply in your case since you don't have Rust enabled.<br>
> Actually I do have Rust enabled for 4.1.7. Old working 4.0.5 does not have Rust support enabled but 4.1.7 does. SMB parsing is one of the main reasons for upgrade. Now I tried to compile 4.1.7 without Rust and it seems the issue is gone. Except there is no required SMB parser. So it seems it is related to Rust(current version: 1.34.2) somehow.<br>
><br>
<br>
Can you try compiling it with the latest Rust ("rustup update" if you<br>
have installed rustup , then do a compile - should do it) and see if<br>
it makes a difference?<br>
Thanks<br>
<br>
> > ...prealloc-sessions...<br>
> Unfortunately increasing prealloc-sessions does not help. I went up to 500000 and the behavior is the same.<br>
><br>
> Thanks,<br>
><br>
> Peter<br>
><br>
><br>
> On Thu, Mar 5, 2020 at 3:12 PM Eric Urban <<a href="mailto:eurban@umn.edu" target="_blank">eurban@umn.edu</a>> wrote:<br>
>><br>
>> Since you mentioned the SMB parser, I believe it wouldn't apply in your case since you don't have Rust enabled.<br>
>><br>
>> I looked at our config in the stream section and noticed two options that differ quite a bit, which is the prealloc-sessions being set much higher (500000) and reassembly.depth is at the default value. From <a href="https://suricata.readthedocs.io/en/suricata-4.1.6/configuration/suricata-yaml.html?highlight=prealloc%20sessions#stream-engine" rel="noreferrer" target="_blank">https://suricata.readthedocs.io/en/suricata-4.1.6/configuration/suricata-yaml.html?highlight=prealloc%20sessions#stream-engine</a>, it is written that "To mitigate Suricata from being overloaded by fast session creation, the option prealloc_sessions instructs Suricata to keep a number of sessions ready in memory" (Note that there is maybe a typo there in "prealloc_sessions" as in our config and in the suricata.yaml example from their GitHub repo it is actually prealloc-sessions with a hyphen and not an underscore). Though your issue isn't quite what I would expect based on the documentation, it is worth a shot to see if this has any effect.<br>
>><br>
>><br>
>> --<br>
>> Eric Urban<br>
>> Security Analyst | University Information Security (UIS)<br>
>> University of Minnesota | <a href="http://umn.edu" rel="noreferrer" target="_blank">umn.edu</a><br>
>> Information Security is a shared responsibility. Learn more at: <a href="https://z.umn.edu/uis" rel="noreferrer" target="_blank">https://z.umn.edu/uis</a><br>
>><br>
>><br>
>> On Tue, Mar 3, 2020 at 2:59 AM d0g d3v <<a href="mailto:d0gd3v@gmail.com" target="_blank">d0gd3v@gmail.com</a>> wrote:<br>
>>><br>
>>> Hi Guys,<br>
>>><br>
>>><br>
>>> After suricata upgrade from 4.0.5 to 4.1.x(0-7) it seems in my setup I experiencing some memory leak. tcp.reassembly_memuse continuously increasing up to any limit set (40gb in few minutes) and then tcp.segment_memcap_drop are triggered and suricata events drops to almost zero.<br>
>>><br>
>>> On the other side, in 4.0.5 there is 15gb stream.reassembly.memcap set and is never reached. tcp.reassembly_memuse is stable around 11gb.<br>
>>><br>
>>><br>
>>> My setup:<br>
>>><br>
>>> CPU: 56 cores<br>
>>><br>
>>> MEM: 64 GB<br>
>>><br>
>>> debian 9<br>
>>><br>
>>> pfring-zc<br>
>>><br>
>>><br>
>>> stream:<br>
>>> memcap: 20gb<br>
>>> checksum-validation: no # reject wrong csums<br>
>>> inline: no # auto will use inline mode in IPS mode, yes or no set it statically<br>
>>> prealloc-sessions: 8096<br>
>>> bypass: yes<br>
>>> midstream: true<br>
>>> async-oneside: true<br>
>>> reassembly:<br>
>>> memcap: 15gb<br>
>>> depth: 20mb # reassemble 1mb into a stream<br>
>>> toserver-chunk-size: 2560<br>
>>> toclient-chunk-size: 2560<br>
>>> randomize-chunk-size: yes<br>
>>><br>
>>><br>
>>> What I tried:<br>
>>><br>
>>> -disable zero copy, change pfring to afpacket - same result<br>
>>><br>
>>> -filter-out smb communication (bpf) if the smb parsing introduced in 4.1. is not root cause - same result<br>
>>><br>
>>> -significantly decrease stream.reassembly.depth and flow-timeouts - same results<br>
>>><br>
>>><br>
>>><br>
>>> I currently can't easy try v5.0.x because of dependencies on the current system. Could you please point me to direction what else I could check?<br>
>>><br>
>>> buildinfo for 4.0.5 and then 4.1.7<br>
>>><br>
>>> This is Suricata version 4.0.5 RELEASE<br>
>>> Features: PCAP_SET_BUFF PF_RING AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LIBJANSSON TLS MAGIC<br>
>>> SIMD support: SSE_4_2 SSE_4_1 SSE_3<br>
>>> Atomic intrisics: 1 2 4 8 16 byte(s)<br>
>>> 64-bits, Little-endian architecture<br>
>>> GCC version 6.3.0 20170516, C version 199901<br>
>>> compiled with _FORTIFY_SOURCE=0<br>
>>> L1 cache line size (CLS)=64<br>
>>> thread local storage method: __thread<br>
>>> compiled with LibHTP v0.5.27, linked against LibHTP v0.5.27<br>
>>><br>
>>> Suricata Configuration:<br>
>>> AF_PACKET support: yes<br>
>>> PF_RING support: yes<br>
>>> NFQueue support: no<br>
>>> NFLOG support: yes<br>
>>> IPFW support: no<br>
>>> Netmap support: no<br>
>>> DAG enabled: no<br>
>>> Napatech enabled: no<br>
>>><br>
>>> Unix socket enabled: yes<br>
>>> Detection enabled: yes<br>
>>><br>
>>> Libmagic support: yes<br>
>>> libnss support: yes<br>
>>> libnspr support: yes<br>
>>> libjansson support: yes<br>
>>> hiredis support: yes<br>
>>> hiredis async with libevent: no<br>
>>> Prelude support: yes<br>
>>> PCRE jit: yes<br>
>>> LUA support: no<br>
>>> libluajit: no<br>
>>> libgeoip: yes<br>
>>> Non-bundled htp: no<br>
>>> Old barnyard2 support: no<br>
>>> CUDA enabled: no<br>
>>> Hyperscan support: yes<br>
>>> Libnet support: yes<br>
>>><br>
>>> Rust support (experimental): no<br>
>>> Experimental Rust parsers: no<br>
>>> Rust strict mode: no<br>
>>><br>
>>> Suricatasc install: yes<br>
>>><br>
>>> Profiling enabled: no<br>
>>> Profiling locks enabled: no<br>
>>><br>
>>> Development settings:<br>
>>> Coccinelle / spatch: no<br>
>>> Unit tests enabled: no<br>
>>> Debug output enabled: no<br>
>>> Debug validation enabled: no<br>
>>><br>
>>> Generic build parameters:<br>
>>> Installation prefix: /usr/local<br>
>>> Configuration directory: /usr/local/etc/suricata/<br>
>>> Log directory: /usr/local/var/log/suricata/<br>
>>><br>
>>> --prefix /usr/local<br>
>>> --sysconfdir /usr/local/etc<br>
>>> --localstatedir /usr/local/var<br>
>>><br>
>>> Host: x86_64-pc-linux-gnu<br>
>>> Compiler: gcc (exec name) / gcc (real)<br>
>>> GCC Protect enabled: no<br>
>>> GCC march native enabled: yes<br>
>>> GCC Profile enabled: no<br>
>>> Position Independent Executable enabled: no<br>
>>> CFLAGS -g -O2 -march=native<br>
>>> PCAP_CFLAGS -I/usr/include<br>
>>> SECCFLAGS<br>
>>><br>
>>><br>
>>> This is Suricata version 4.1.7 RELEASE<br>
>>><br>
>>> Features: DEBUG PCAP_SET_BUFF PF_RING AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LIBJANSSON TLS MAGIC RUST<br>
>>> SIMD support: SSE_4_2 SSE_4_1 SSE_3<br>
>>> Atomic intrisics: 1 2 4 8 16 byte(s)<br>
>>> 64-bits, Little-endian architecture<br>
>>> GCC version 6.3.0 20170516, C version 199901<br>
>>> compiled with _FORTIFY_SOURCE=0<br>
>>> L1 cache line size (CLS)=64<br>
>>> thread local storage method: __thread<br>
>>> compiled with LibHTP v0.5.32, linked against LibHTP v0.5.32<br>
>>><br>
>>> Suricata Configuration:<br>
>>> AF_PACKET support: yes<br>
>>> eBPF support: no<br>
>>> XDP support: no<br>
>>> PF_RING support: yes<br>
>>> NFQueue support: no<br>
>>> NFLOG support: yes<br>
>>> IPFW support: no<br>
>>> Netmap support: no<br>
>>> DAG enabled: no<br>
>>> Napatech enabled: no<br>
>>> WinDivert enabled: no<br>
>>><br>
>>> Unix socket enabled: yes<br>
>>> Detection enabled: yes<br>
>>><br>
>>> Libmagic support: yes<br>
>>> libnss support: yes<br>
>>> libnspr support: yes<br>
>>> libjansson support: yes<br>
>>> liblzma support: no<br>
>>> hiredis support: yes<br>
>>> hiredis async with libevent: no<br>
>>> Prelude support: yes<br>
>>> PCRE jit: yes<br>
>>> LUA support: no<br>
>>> libluajit: no<br>
>>> GeoIP support: yes, legacy libgeoip<br>
>>> Non-bundled htp: no<br>
>>> Old barnyard2 support: no<br>
>>> Hyperscan support: yes<br>
>>> Libnet support: yes<br>
>>> liblz4 support: no<br>
>>><br>
>>> Rust support: yes<br>
>>> Rust strict mode: no<br>
>>> Rust debug mode: no<br>
>>> Rust compiler: rustc 1.24.1<br>
>>> Rust cargo: cargo 1.34.0<br>
>>> Cargo vendor: no<br>
>>><br>
>>> Install suricatasc: yes<br>
>>> Install suricata-update: yes<br>
>>><br>
>>> Profiling enabled: no<br>
>>> Profiling locks enabled: no<br>
>>><br>
>>> Development settings:<br>
>>> Coccinelle / spatch: no<br>
>>> Unit tests enabled: no<br>
>>> Debug output enabled: yes<br>
>>> Debug validation enabled: no<br>
>>><br>
>>> Generic build parameters:<br>
>>> Installation prefix: /usr/local<br>
>>> Configuration directory: /usr/local/etc/suricata/<br>
>>> Log directory: /usr/local/var/log/suricata/<br>
>>><br>
>>> --prefix /usr/local<br>
>>> --sysconfdir /usr/local/etc<br>
>>> --localstatedir /usr/local/var<br>
>>> --datarootdir /usr/local/share<br>
>>><br>
>>> Host: x86_64-pc-linux-gnu<br>
>>> Compiler: gcc (exec name) / gcc (real)<br>
>>> GCC Protect enabled: no<br>
>>> GCC march native enabled: yes<br>
>>> GCC Profile enabled: no<br>
>>> Position Independent Executable enabled: no<br>
>>> CFLAGS -g -O2 -march=native -I${srcdir}/../rust/gen/c-headers<br>
>>> PCAP_CFLAGS -I/usr/include<br>
>>> SECCFLAGS<br>
>>><br>
>>><br>
>>> Thanks,<br>
>>><br>
>>> Peter<br>
>>><br>
>>> _______________________________________________<br>
>>> Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org" target="_blank">oisf-users@openinfosecfoundation.org</a><br>
>>> Site: <a href="http://suricata-ids.org" rel="noreferrer" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" rel="noreferrer" target="_blank">http://suricata-ids.org/support/</a><br>
>>> List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" rel="noreferrer" target="_blank">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
>>><br>
>>> Conference: <a href="https://suricon.net" rel="noreferrer" target="_blank">https://suricon.net</a><br>
>>> Trainings: <a href="https://suricata-ids.org/training/" rel="noreferrer" target="_blank">https://suricata-ids.org/training/</a><br>
><br>
> _______________________________________________<br>
> Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org" target="_blank">oisf-users@openinfosecfoundation.org</a><br>
> Site: <a href="http://suricata-ids.org" rel="noreferrer" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" rel="noreferrer" target="_blank">http://suricata-ids.org/support/</a><br>
> List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" rel="noreferrer" target="_blank">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
><br>
> Conference: <a href="https://suricon.net" rel="noreferrer" target="_blank">https://suricon.net</a><br>
> Trainings: <a href="https://suricata-ids.org/training/" rel="noreferrer" target="_blank">https://suricata-ids.org/training/</a><br>
<br>
<br>
<br>
-- <br>
Regards,<br>
Peter Manev<br>
</blockquote></div>