Hi all! Just as you suggested, I did another "tuning round" to my suricata.yaml file, specially to the timeouts section. I was confused about this settings, because I think that stream and flow wasn't relationed.<br>
Here are my new settings, I must wait to monday (more load to our datacenter network) to see the effects:<br><br>flow-timeouts:<br><br>  default:<br>    new: 10 # 30<br>    established: 20 # 300<br>    closed: 0<br>    emergency-new: 5 # 10<br>
    emergency-established: 10 # 100<br>    emergency-closed: 0<br>  tcp:<br>    new: 10 # 60<br>    established: 20 # 3600<br>    closed: 2 # 120<br>    emergency-new: 5 # 10<br>    emergency-established: 10 # 300<br>    emergency-closed: 2 # 20<br>
  udp:<br>    new: 5 # 30<br>    established: 10 # 300<br>    emergency-new: 2 # 10<br>    emergency-established: 5 # 100<br>  icmp:<br>    new: 5 # 30<br>    established: 5 # 300<br>    emergency-new: 2 # 10<br>    emergency-established: 2 # 100<br>
<br>I will keep you informed, thanks A LOT for your help!<br>Best regards!<br><div class="gmail_extra"><br><br><div class="gmail_quote">2012/12/1 Peter Manev <span dir="ltr"><<a href="mailto:petermanev@gmail.com" target="_blank">petermanev@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi ,<br><br>Martin is very right about the flow-timeouts - very important, not to forget to adjust those.<br>300 sec is 5 min...on a busy network .... -<br>
tcp:<br>established: 3600 - (default)<br>1 hr can have some serious impact :) <br>
<br>It is funny you mention about the drops.. I just had a quick chat with Victor about drops in general just a few days ago.<br>Here is some of our values/results on one of our test boxes  (9.5Gb/s traffic):<br><br>YAML:<br>

flow-timeouts:<br><br>  default:<br>    new: 5 #30<div class="im"><br>    established: 10#  300<br>    closed: 0<br></div>    emergency-new: 1 #10<br>    emergency-established: 2 #100<br>    emergency-closed: 0<br>  tcp:<br>
    new: 5 #60<br>    established: 300 # 3600<br>
    closed: 10 #30<br>    emergency-new: 1 # 10<br>    emergency-established: 5 # 300<br>    emergency-closed: 20 #20<br>  udp:<br>    new: 5 #30<br>    established: 5 # 300<br>    emergency-new: 5 #10<br>    emergency-established: 5 # 100<br>

  icmp:<br>    new: 5 #30<br>    established: 5 % 300<br>    emergency-new: 5 #10<br>    emergency-established: 5 # 100<br><br>......<br>stream:<br>  memcap: 16gb<br>  max-sessions: 20000000<br>  prealloc-sessions: 10000000<div class="im">
<br>
  checksum-validation: no      # reject wrong csums<br></div>  #checksum-validation: yes      # reject wrong csums<br>  inline: no                    # no inline mode<br>  reassembly:<br>    memcap: 12gb<br>    #memcap: 8gb<br>

    depth: 12mb                  # reassemble 1mb into a stream<br>    toserver-chunk-size: 2560<br>    toclient-chunk-size: 2560<br><br># Host table:<br>#<br># Host table is used by tagging and per host thresholding subsystems.<br>

#<br>host:<br>  hash-size: 4096<br>  prealloc: 1000<br>  memcap: 16777216<br><br>......<br># Defrag settings:<br><br>defrag:<br>  #trackers: 262144  # number of defragmented flows to follow<br>  #max-frags:  262144   #number of fragments per-flow<br>

  trackers: 65535<br>  max-frags: 65535 # number of fragments per-flow<br>  prealloc: yes<br>  timeout: 10<br> <br><br><br>Al this is using af_packet 16 threads , on a 16CPU(with Hyperthrd) box 32 GB RAM, with some special intel 10G NIC tuning, ubuntu LTS 12.04, running latest git with 7K EmThr rules.<br>

Some more info:<br><br><br><br>pevman@suricata:~$ sudo grep -n "drop" /var/data/regit/log/suricata/stats.log | tail -48<br>2504179:capture.kernel_drops      | AFPacketeth31             | 0<br>2504209:tcp.ssn_memcap_drop       | AFPacketeth31             | 0<br>

2504218:tcp.segment_memcap_drop   | AFPacketeth31             | 0<br>2504224:capture.kernel_drops      | AFPacketeth32             | 0<br>2504254:tcp.ssn_memcap_drop       | AFPacketeth32             | 0<br>2504263:tcp.segment_memcap_drop   | AFPacketeth32             | 0<br>

2504269:capture.kernel_drops      | AFPacketeth33             | 0<br>2504299:tcp.ssn_memcap_drop       | AFPacketeth33             | 0<br>2504308:tcp.segment_memcap_drop   | AFPacketeth33             | 0<br>2504314:capture.kernel_drops      | AFPacketeth34             | 0<br>

2504344:tcp.ssn_memcap_drop       | AFPacketeth34             | 0<br>2504353:tcp.segment_memcap_drop   | AFPacketeth34             | 0<br>2504359:capture.kernel_drops      | AFPacketeth35             | 0<br>2504389:tcp.ssn_memcap_drop       | AFPacketeth35             | 0<br>

2504398:tcp.segment_memcap_drop   | AFPacketeth35             | 0<br>2504404:capture.kernel_drops      | AFPacketeth36             | 0<br>2504434:tcp.ssn_memcap_drop       | AFPacketeth36             | 0<br>2504443:tcp.segment_memcap_drop   | AFPacketeth36             | 0<br>

2504449:capture.kernel_drops      | AFPacketeth37             | 0<br>2504479:tcp.ssn_memcap_drop       | AFPacketeth37             | 0<br>2504488:tcp.segment_memcap_drop   | AFPacketeth37             | 0<br>2504494:capture.kernel_drops      | AFPacketeth38             | 0<br>

2504524:tcp.ssn_memcap_drop       | AFPacketeth38             | 0<br>2504533:tcp.segment_memcap_drop   | AFPacketeth38             | 0<br>2504539:capture.kernel_drops      | AFPacketeth39             | 0<br>2504569:tcp.ssn_memcap_drop       | AFPacketeth39             | 0<br>

2504578:tcp.segment_memcap_drop   | AFPacketeth39             | 0<br>2504584:capture.kernel_drops      | AFPacketeth310            | 0<br>2504614:tcp.ssn_memcap_drop       | AFPacketeth310            | 0<br>2504623:tcp.segment_memcap_drop   | AFPacketeth310            | 0<br>

2504629:capture.kernel_drops      | AFPacketeth311            | 0<br>2504659:tcp.ssn_memcap_drop       | AFPacketeth311            | 0<br>2504668:tcp.segment_memcap_drop   | AFPacketeth311            | 0<br>2504674:capture.kernel_drops      | AFPacketeth312            | 0<br>

2504704:tcp.ssn_memcap_drop       | AFPacketeth312            | 0<br>2504713:tcp.segment_memcap_drop   | AFPacketeth312            | 0<br>2504719:capture.kernel_drops      | AFPacketeth313            | 0<br>2504749:tcp.ssn_memcap_drop       | AFPacketeth313            | 0<br>

2504758:tcp.segment_memcap_drop   | AFPacketeth313            | 0<br>2504764:capture.kernel_drops      | AFPacketeth314            | 0<br>2504794:tcp.ssn_memcap_drop       | AFPacketeth314            | 0<br>2504803:tcp.segment_memcap_drop   | AFPacketeth314            | 0<br>

2504809:capture.kernel_drops      | AFPacketeth315            | 0<br>2504839:tcp.ssn_memcap_drop       | AFPacketeth315            | 0<br>2504848:tcp.segment_memcap_drop   | AFPacketeth315            | 0<br>2504854:capture.kernel_drops      | AFPacketeth316            | 0<br>

2504884:tcp.ssn_memcap_drop       | AFPacketeth316            | 0<br>2504893:tcp.segment_memcap_drop   | AFPacketeth316            | 0<br><br><b>pevman@suricata:~$ suricata --build-info</b><br>[10384] 1/12/2012 -- 14:28:44 - (suricata.c:560) <Info> (SCPrintBuildInfo) -- This is Suricata version 1.4dev (rev 005f7a2)<br>

[10384] 1/12/2012 -- 14:28:44 - (suricata.c:633) <Info> (SCPrintBuildInfo) -- Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 PF_RING AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK HAVE_HTP_TX_GET_RESPONSE_HEADERS_RAW HAVE_NSS PROFILING<br>

[10384] 1/12/2012 -- 14:28:44 - (suricata.c:647) <Info> (SCPrintBuildInfo) -- 64-bits, Little-endian architecture<br>[10384] 1/12/2012 -- 14:28:44 - (suricata.c:649) <Info> (SCPrintBuildInfo) -- GCC version 4.6.3, C version 199901<br>

[10384] 1/12/2012 -- 14:28:44 - (suricata.c:655) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_1<br>[10384] 1/12/2012 -- 14:28:44 - (suricata.c:658) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_2<br>

[10384] 1/12/2012 -- 14:28:44 - (suricata.c:661) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4<br>[10384] 1/12/2012 -- 14:28:44 - (suricata.c:664) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8<br>

[10384] 1/12/2012 -- 14:28:44 - (suricata.c:667) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_16<br>[10384] 1/12/2012 -- 14:28:44 - (suricata.c:671) <Info> (SCPrintBuildInfo) -- compiled with -fstack-protector<br>

[10384] 1/12/2012 -- 14:28:44 - (suricata.c:677) <Info> (SCPrintBuildInfo) -- compiled with _FORTIFY_SOURCE=2<br>[10384] 1/12/2012 -- 14:28:44 - (suricata.c:680) <Info> (SCPrintBuildInfo) -- compiled with libhtp 0.2.11, linked against 0.2.11<br>

<br>pevman@suricata:~$ sudo grep -n "uptime" /var/data/regit/log/suricata/stats.log | tail -4<br>2503442:Date: 12/1/2012 -- 14:27:56 (uptime: 0d, 18h 07m 04s)<br>2504174:Date: 12/1/2012 -- 14:28:15 (uptime: 0d, 18h 07m 23s)<br>

2504906:Date: 12/1/2012 -- 14:28:34 (uptime: 0d, 18h 07m 42s)<br>2505638:Date: 12/1/2012 -- 14:28:53 (uptime: 0d, 18h 08m 01s)<br><br>pevman@suricata:~$ sudo tcpstat -i eth3<br>Time:1354365172 n=6106758       avg=984.85      stddev=663.77   bps=9622763462.40<br>

Time:1354365177 n=6126927       avg=981.51      stddev=663.29   bps=9621826076.80<br>Time:1354365182 n=6110921       avg=984.19      stddev=662.02   bps=9622922160.00<br>Time:1354365187 n=6126978       avg=981.50      stddev=662.38   bps=9621846648.00<br>

Time:1354365192 n=6109322       avg=984.46      stddev=661.25   bps=9623061092.80<br>Time:1354365197 n=6146841       avg=978.24      stddev=662.73   bps=9620970840.00<br>^CTime:1354365202       n=112243        avg=982.41      stddev=663.97   bps=176430308.80<br>

<br>pevman@suricata:~$ uname -a<br>Linux suricata 3.2.0-30-generic #48-Ubuntu SMP Fri Aug 24 16:52:48 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux<br>pevman@suricata:~$<br><br><img alt=""><br>
<br><br><br>hope it helps.<br><br>thanks<div><div class="h5"><br><br><div class="gmail_quote">On Sat, Dec 1, 2012 at 3:54 AM, Martin Holste <span dir="ltr"><<a href="mailto:mcholste@gmail.com" target="_blank">mcholste@gmail.com</a>></span> wrote:<br>

<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Adjust your default timeouts much lower so that streams are taken out of the connection pool more quickly.<br><br>This config is aggressive, but I think you'll find it does the trick.  If it doesn't work, I'd like to know:<br>



<br>flow-timeouts:<br><br>  default:<br>    new: 1 # 30<br>    established: 10 #300<br>    closed: 0<br>    emergency_new: 1 #10<br>    emergency_established: 1 #100<br>    emergency_closed: 0<br>  tcp:<br>    new: 1 #60<br>



    established: 10 #3600<br>    closed: 0 #120<br>    emergency_new: 1 #10<br>    emergency_established: 5 #1 #300<br>    emergency_closed: 20<br>  udp:<br>    new: 1 #30<br>    established: 1 #300<br>    emergency_new: 1 #10<br>



    emergency_established: 1 #100<br>  icmp:<br>    new: 1 #30<br>    established: 1 #300<br>    emergency_new: 1 #10<br>    emergency_established: 1 #100<div><div><br><br><div class="gmail_extra">
<br><br><div class="gmail_quote">
On Fri, Nov 30, 2012 at 4:15 PM, Dave Remien <span dir="ltr"><<a href="mailto:dave.remien@gmail.com" target="_blank">dave.remien@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>Fernando,</div><div><br></div>If I'm reading your config file right, you're asking for 8.3 million sessions of 512KB each? I think that works out to 4.3TB of RAM; rather more than the 64GB memcap.<div>



<br></div>
<div>Cheers,</div><div><br></div><div>Dave<div><div><br><br><div class="gmail_quote">On Fri, Nov 30, 2012 at 10:24 AM, Fernando Sclavo <span dir="ltr"><<a href="mailto:fsclavo@gmail.com" target="_blank">fsclavo@gmail.com</a>></span> wrote:<br>




<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">-----BEGIN PGP SIGNED MESSAGE-----<br>
Hash: SHA1<br>
<br>
Hello all!<br>
I'm installing an IDS on our company, monitoring two core switches with<br>
a sustained traffic of about 2gbps each. The server is a Dell R715, 32<br>
cores, 192Gb RAM with two Intel X520 nics. Suricata version is 1.4b3.<br>
The problem we are facing, is with tcp.segment_memcap_drop increasing<br>
continuosly once time tcp.reassembly_memuse reaches their max size (64gb!!)<br>
The related suricata.yaml stanza is:<br>
<br>
stream:<br>
  memcap: 24gb<br>
  checksum-validation: no      # reject wrong csums<br>
  inline: no                  # auto will use inline mode in IPS mode,<br>
yes or no set it statically<br>
  max-sessions: 8388608<br>
  prealloc-sessions: 8388608<br>
  reassembly:<br>
    memcap: 64gb<br>
    depth: 512kb                  # reassemble 1mb into a stream<br>
    toserver-chunk-size: 2560<br>
    toclient-chunk-size: 2560<br>
<br>
Thanks in advance!<br>
-----BEGIN PGP SIGNATURE-----<br>
Version: GnuPG v1.4.11 (GNU/Linux)<br>
Comment: Using GnuPG with undefined - <a href="http://www.enigmail.net/" target="_blank">http://www.enigmail.net/</a><br>
<br>
iQIcBAEBAgAGBQJQuOviAAoJEDtYYV2Ws9eJD18P/2+QZR+6BXnk/FfXQeCw43Xh<br>
qynGiI3qnrg3SSaGdiWDrm0b8UuVuq/HXaAdIo0hzeDNgRLWjBKnnz4b3UA3HyIH<br>
cKpPUsEFUyc55KPSDzDW2mCGB/V//7f/Ude5DXG7/CZ9+xJu1jhuePfuE9Nl1yIi<br>
o3xmlI1mXXXc82rs0VGKDJ0ZwoN+/zmcnp1sW5mG42CKR2Hr9PcVKzP0IHbNZlHI<br>
Q0ishhXNrKcGCpHn9/J9gg44af6+7a0EdnOZOEgRNtOILfK6C5N4p5cwZfMAkYnL<br>
AcswoaER4ftBV49WpfWjTeOhEQxYaGFM8QURB0f30ODqMDoDUKX6lwjXm6+ZfQqr<br>
Y+mGzX/WFCeFI2A4KqgNamZi1IKKd83j0AxH8nYhWa9kPtws75L5iGYAQOE5yoVw<br>
oTnEncPlSLK+Mb/fhoc0crNeMkCKDV6uCFgpE/JKUtogG25nmcbSAIoE3Esa9iYq<br>
dRww7KhOZttLRXjZeRkm/bl1CmBDXDJ2sZQ8jZtqpGeFlIMi4BYCyQAKsKWyAji4<br>
9LrDvtnew/jvWLCpNOfPrHWjRM+XbpD+k4YWO1imRWU6Or+E4Fgx9oiFNd9ni/DY<br>
l2NrSkq9RIixCVqrpNkWsEwCxN2pftJ4h0sXqTqkkhi8Ofhui60o1uNAOqMGURoN<br>
U30CUPowHUvuwnguE781<br>
=vy1s<br>
-----END PGP SIGNATURE-----<br>
_______________________________________________<br>
Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org" target="_blank">oisf-users@openinfosecfoundation.org</a><br>
Site: <a href="http://suricata-ids.org" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" target="_blank">http://suricata-ids.org/support/</a><br>
List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" target="_blank">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
OISF: <a href="http://www.openinfosecfoundation.org/" target="_blank">http://www.openinfosecfoundation.org/</a><br>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888">-- <br><div>".... We are such stuff</div><div>As dreams are made on; and our little life</div><div><div>Is rounded with a sleep."</div>



</div><div>-- Shakespeare, The Tempest - Act 4</div>
<br>
</font></span></div>
<br>_______________________________________________<br>
Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org" target="_blank">oisf-users@openinfosecfoundation.org</a><br>
Site: <a href="http://suricata-ids.org" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" target="_blank">http://suricata-ids.org/support/</a><br>
List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" target="_blank">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
OISF: <a href="http://www.openinfosecfoundation.org/" target="_blank">http://www.openinfosecfoundation.org/</a><br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org" target="_blank">oisf-users@openinfosecfoundation.org</a><br>
Site: <a href="http://suricata-ids.org" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" target="_blank">http://suricata-ids.org/support/</a><br>
List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" target="_blank">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
OISF: <a href="http://www.openinfosecfoundation.org/" target="_blank">http://www.openinfosecfoundation.org/</a><br></blockquote></div><br><br clear="all"><br>-- <br></div></div><div>Regards,</div>
<div>Peter Manev</div><br>
</blockquote></div><br></div>