[Oisf-users] tcp.segment_memcap_drop couldn't be kept at zero, no matters how much memory we assign
Victor Julien
lists at inliniac.net
Mon Dec 3 13:59:01 UTC 2012
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 12/02/2012 01:32 AM, Fernando Sclavo wrote:
> Hi all! Just as you suggested, I did another "tuning round" to my
> suricata.yaml file, specially to the timeouts section. I was
> confused about this settings, because I think that stream and flow
> wasn't relationed. Here are my new settings, I must wait to monday
> (more load to our datacenter network) to see the effects:
>
> flow-timeouts:
>
> default: new: 10 # 30 established: 20 # 300 closed: 0
> emergency-new: 5 # 10 emergency-established: 10 # 100
> emergency-closed: 0 tcp: new: 10 # 60 established: 20 # 3600
> closed: 2 # 120 emergency-new: 5 # 10 emergency-established: 10 #
> 300 emergency-closed: 2 # 20 udp: new: 5 # 30 established: 10 #
> 300 emergency-new: 2 # 10 emergency-established: 5 # 100 icmp: new:
> 5 # 30 established: 5 # 300 emergency-new: 2 # 10
> emergency-established: 2 # 100
>
> I will keep you informed, thanks A LOT for your help!
If that didn't help I recommend that you try our 1.3-git code. We just
fixed a bug that could lead to memory leaks in this area, so it may
very well be that this causes your issue.
https://github.com/inliniac/suricata/tree/master-1.3.x
Cheers,
Victor
> Best regards!
>
>
> 2012/12/1 Peter Manev <petermanev at gmail.com
> <mailto:petermanev at gmail.com>>
>
> Hi ,
>
> Martin is very right about the flow-timeouts - very important, not
> to forget to adjust those. 300 sec is 5 min...on a busy network
> .... - tcp: established: 3600 - (default) 1 hr can have some
> serious impact :)
>
> It is funny you mention about the drops.. I just had a quick chat
> with Victor about drops in general just a few days ago. Here is
> some of our values/results on one of our test boxes (9.5Gb/s
> traffic):
>
> YAML: flow-timeouts:
>
> default: new: 5 #30
>
> established: 10# 300 closed: 0 emergency-new: 1 #10
> emergency-established: 2 #100 emergency-closed: 0 tcp: new: 5 #60
> established: 300 # 3600 closed: 10 #30 emergency-new: 1 # 10
> emergency-established: 5 # 300 emergency-closed: 20 #20 udp: new: 5
> #30 established: 5 # 300 emergency-new: 5 #10
> emergency-established: 5 # 100 icmp: new: 5 #30 established: 5 %
> 300 emergency-new: 5 #10 emergency-established: 5 # 100
>
> ...... stream: memcap: 16gb max-sessions: 20000000
> prealloc-sessions: 10000000
>
> checksum-validation: no # reject wrong csums
> #checksum-validation: yes # reject wrong csums inline: no
> # no inline mode reassembly: memcap: 12gb #memcap: 8gb depth: 12mb
> # reassemble 1mb into a stream toserver-chunk-size: 2560
> toclient-chunk-size: 2560
>
> # Host table: # # Host table is used by tagging and per host
> thresholding subsystems. # host: hash-size: 4096 prealloc: 1000
> memcap: 16777216
>
> ...... # Defrag settings:
>
> defrag: #trackers: 262144 # number of defragmented flows to
> follow #max-frags: 262144 #number of fragments per-flow
> trackers: 65535 max-frags: 65535 # number of fragments per-flow
> prealloc: yes timeout: 10
>
>
>
> Al this is using af_packet 16 threads , on a 16CPU(with Hyperthrd)
> box 32 GB RAM, with some special intel 10G NIC tuning, ubuntu LTS
> 12.04, running latest git with 7K EmThr rules. Some more info:
>
>
>
> pevman at suricata:~$ sudo grep -n "drop"
> /var/data/regit/log/suricata/stats.log | tail -48
> 2504179:capture.kernel_drops | AFPacketeth31 | 0
> 2504209:tcp.ssn_memcap_drop | AFPacketeth31 | 0
> 2504218:tcp.segment_memcap_drop | AFPacketeth31 | 0
> 2504224:capture.kernel_drops | AFPacketeth32 | 0
> 2504254:tcp.ssn_memcap_drop | AFPacketeth32 | 0
> 2504263:tcp.segment_memcap_drop | AFPacketeth32 | 0
> 2504269:capture.kernel_drops | AFPacketeth33 | 0
> 2504299:tcp.ssn_memcap_drop | AFPacketeth33 | 0
> 2504308:tcp.segment_memcap_drop | AFPacketeth33 | 0
> 2504314:capture.kernel_drops | AFPacketeth34 | 0
> 2504344:tcp.ssn_memcap_drop | AFPacketeth34 | 0
> 2504353:tcp.segment_memcap_drop | AFPacketeth34 | 0
> 2504359:capture.kernel_drops | AFPacketeth35 | 0
> 2504389:tcp.ssn_memcap_drop | AFPacketeth35 | 0
> 2504398:tcp.segment_memcap_drop | AFPacketeth35 | 0
> 2504404:capture.kernel_drops | AFPacketeth36 | 0
> 2504434:tcp.ssn_memcap_drop | AFPacketeth36 | 0
> 2504443:tcp.segment_memcap_drop | AFPacketeth36 | 0
> 2504449:capture.kernel_drops | AFPacketeth37 | 0
> 2504479:tcp.ssn_memcap_drop | AFPacketeth37 | 0
> 2504488:tcp.segment_memcap_drop | AFPacketeth37 | 0
> 2504494:capture.kernel_drops | AFPacketeth38 | 0
> 2504524:tcp.ssn_memcap_drop | AFPacketeth38 | 0
> 2504533:tcp.segment_memcap_drop | AFPacketeth38 | 0
> 2504539:capture.kernel_drops | AFPacketeth39 | 0
> 2504569:tcp.ssn_memcap_drop | AFPacketeth39 | 0
> 2504578:tcp.segment_memcap_drop | AFPacketeth39 | 0
> 2504584:capture.kernel_drops | AFPacketeth310 | 0
> 2504614:tcp.ssn_memcap_drop | AFPacketeth310 | 0
> 2504623:tcp.segment_memcap_drop | AFPacketeth310 | 0
> 2504629:capture.kernel_drops | AFPacketeth311 | 0
> 2504659:tcp.ssn_memcap_drop | AFPacketeth311 | 0
> 2504668:tcp.segment_memcap_drop | AFPacketeth311 | 0
> 2504674:capture.kernel_drops | AFPacketeth312 | 0
> 2504704:tcp.ssn_memcap_drop | AFPacketeth312 | 0
> 2504713:tcp.segment_memcap_drop | AFPacketeth312 | 0
> 2504719:capture.kernel_drops | AFPacketeth313 | 0
> 2504749:tcp.ssn_memcap_drop | AFPacketeth313 | 0
> 2504758:tcp.segment_memcap_drop | AFPacketeth313 | 0
> 2504764:capture.kernel_drops | AFPacketeth314 | 0
> 2504794:tcp.ssn_memcap_drop | AFPacketeth314 | 0
> 2504803:tcp.segment_memcap_drop | AFPacketeth314 | 0
> 2504809:capture.kernel_drops | AFPacketeth315 | 0
> 2504839:tcp.ssn_memcap_drop | AFPacketeth315 | 0
> 2504848:tcp.segment_memcap_drop | AFPacketeth315 | 0
> 2504854:capture.kernel_drops | AFPacketeth316 | 0
> 2504884:tcp.ssn_memcap_drop | AFPacketeth316 | 0
> 2504893:tcp.segment_memcap_drop | AFPacketeth316 | 0
>
> *pevman at suricata:~$ suricata --build-info* [10384] 1/12/2012 --
> 14:28:44 - (suricata.c:560) <Info> (SCPrintBuildInfo) -- This is
> Suricata version 1.4dev (rev 005f7a2) [10384] 1/12/2012 -- 14:28:44
> - (suricata.c:633) <Info> (SCPrintBuildInfo) -- Features:
> PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 PF_RING AF_PACKET
> HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK
> HAVE_HTP_TX_GET_RESPONSE_HEADERS_RAW HAVE_NSS PROFILING [10384]
> 1/12/2012 -- 14:28:44 - (suricata.c:647) <Info> (SCPrintBuildInfo)
> -- 64-bits, Little-endian architecture [10384] 1/12/2012 --
> 14:28:44 - (suricata.c:649) <Info> (SCPrintBuildInfo) -- GCC
> version 4.6.3, C version 199901 [10384] 1/12/2012 -- 14:28:44 -
> (suricata.c:655) <Info> (SCPrintBuildInfo) --
> __GCC_HAVE_SYNC_COMPARE_AND_SWAP_1 [10384] 1/12/2012 -- 14:28:44 -
> (suricata.c:658) <Info> (SCPrintBuildInfo) --
> __GCC_HAVE_SYNC_COMPARE_AND_SWAP_2 [10384] 1/12/2012 -- 14:28:44 -
> (suricata.c:661) <Info> (SCPrintBuildInfo) --
> __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4 [10384] 1/12/2012 -- 14:28:44 -
> (suricata.c:664) <Info> (SCPrintBuildInfo) --
> __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8 [10384] 1/12/2012 -- 14:28:44 -
> (suricata.c:667) <Info> (SCPrintBuildInfo) --
> __GCC_HAVE_SYNC_COMPARE_AND_SWAP_16 [10384] 1/12/2012 -- 14:28:44 -
> (suricata.c:671) <Info> (SCPrintBuildInfo) -- compiled with
> -fstack-protector [10384] 1/12/2012 -- 14:28:44 - (suricata.c:677)
> <Info> (SCPrintBuildInfo) -- compiled with _FORTIFY_SOURCE=2
> [10384] 1/12/2012 -- 14:28:44 - (suricata.c:680) <Info>
> (SCPrintBuildInfo) -- compiled with libhtp 0.2.11, linked against
> 0.2.11
>
> pevman at suricata:~$ sudo grep -n "uptime"
> /var/data/regit/log/suricata/stats.log | tail -4 2503442:Date:
> 12/1/2012 -- 14:27:56 (uptime: 0d, 18h 07m 04s) 2504174:Date:
> 12/1/2012 -- 14:28:15 (uptime: 0d, 18h 07m 23s) 2504906:Date:
> 12/1/2012 -- 14:28:34 (uptime: 0d, 18h 07m 42s) 2505638:Date:
> 12/1/2012 -- 14:28:53 (uptime: 0d, 18h 08m 01s)
>
> pevman at suricata:~$ sudo tcpstat -i eth3 Time:1354365172 n=6106758
> avg=984.85 stddev=663.77 bps=9622763462.40 Time:1354365177
> n=6126927 avg=981.51 stddev=663.29 bps=9621826076.80
> Time:1354365182 n=6110921 avg=984.19 stddev=662.02
> bps=9622922160.00 Time:1354365187 n=6126978 avg=981.50
> stddev=662.38 bps=9621846648.00 Time:1354365192 n=6109322
> avg=984.46 stddev=661.25 bps=9623061092.80 Time:1354365197
> n=6146841 avg=978.24 stddev=662.73 bps=9620970840.00
> ^CTime:1354365202 n=112243 avg=982.41 stddev=663.97
> bps=176430308.80
>
> pevman at suricata:~$ uname -a Linux suricata 3.2.0-30-generic
> #48-Ubuntu SMP Fri Aug 24 16:52:48 UTC 2012 x86_64 x86_64 x86_64
> GNU/Linux pevman at suricata:~$
>
>
>
>
>
> hope it helps.
>
> thanks
>
>
> On Sat, Dec 1, 2012 at 3:54 AM, Martin Holste <mcholste at gmail.com
> <mailto:mcholste at gmail.com>> wrote:
>
> Adjust your default timeouts much lower so that streams are taken
> out of the connection pool more quickly.
>
> This config is aggressive, but I think you'll find it does the
> trick. If it doesn't work, I'd like to know:
>
> flow-timeouts:
>
> default: new: 1 # 30 established: 10 #300 closed: 0 emergency_new:
> 1 #10 emergency_established: 1 #100 emergency_closed: 0 tcp: new: 1
> #60 established: 10 #3600 closed: 0 #120 emergency_new: 1 #10
> emergency_established: 5 #1 #300 emergency_closed: 20 udp: new: 1
> #30 established: 1 #300 emergency_new: 1 #10 emergency_established:
> 1 #100 icmp: new: 1 #30 established: 1 #300 emergency_new: 1 #10
> emergency_established: 1 #100
>
>
>
>
> On Fri, Nov 30, 2012 at 4:15 PM, Dave Remien <dave.remien at gmail.com
> <mailto:dave.remien at gmail.com>> wrote:
>
> Fernando,
>
> If I'm reading your config file right, you're asking for 8.3
> million sessions of 512KB each? I think that works out to 4.3TB of
> RAM; rather more than the 64GB memcap.
>
> Cheers,
>
> Dave
>
>
> On Fri, Nov 30, 2012 at 10:24 AM, Fernando Sclavo
> <fsclavo at gmail.com <mailto:fsclavo at gmail.com>> wrote:
>
> Hello all! I'm installing an IDS on our company, monitoring two
> core switches with a sustained traffic of about 2gbps each. The
> server is a Dell R715, 32 cores, 192Gb RAM with two Intel X520
> nics. Suricata version is 1.4b3. The problem we are facing, is
> with tcp.segment_memcap_drop increasing continuosly once time
> tcp.reassembly_memuse reaches their max size (64gb!!) The related
> suricata.yaml stanza is:
>
> stream: memcap: 24gb checksum-validation: no # reject wrong
> csums inline: no # auto will use inline mode in
> IPS mode, yes or no set it statically max-sessions: 8388608
> prealloc-sessions: 8388608 reassembly: memcap: 64gb depth: 512kb
> # reassemble 1mb into a stream toserver-chunk-size: 2560
> toclient-chunk-size: 2560
>
> Thanks in advance! _______________________________________________
> Suricata IDS Users mailing list:
> oisf-users at openinfosecfoundation.org
> <mailto:oisf-users at openinfosecfoundation.org> Site:
> http://suricata-ids.org | Support:
> http://suricata-ids.org/support/ List:
> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>
>
OISF: http://www.openinfosecfoundation.org/
>
>
>
>
> -- ".... We are such stuff As dreams are made on; and our little
> life Is rounded with a sleep." -- Shakespeare, The Tempest - Act 4
>
>
> _______________________________________________ Suricata IDS Users
> mailing list: oisf-users at openinfosecfoundation.org
> <mailto:oisf-users at openinfosecfoundation.org> Site:
> http://suricata-ids.org | Support:
> http://suricata-ids.org/support/ List:
> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>
>
OISF: http://www.openinfosecfoundation.org/
>
>
>
> _______________________________________________ Suricata IDS Users
> mailing list: oisf-users at openinfosecfoundation.org
> <mailto:oisf-users at openinfosecfoundation.org> Site:
> http://suricata-ids.org | Support:
> http://suricata-ids.org/support/ List:
> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>
>
OISF: http://www.openinfosecfoundation.org/
>
>
>
>
> -- Regards, Peter Manev
>
>
>
>
> _______________________________________________ Suricata IDS Users
> mailing list: oisf-users at openinfosecfoundation.org Site:
> http://suricata-ids.org | Support:
> http://suricata-ids.org/support/ List:
> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>
>
OISF: http://www.openinfosecfoundation.org/
>
- --
- ---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
- ---------------------------------------------
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/
iEYEARECAAYFAlC8sBoACgkQiSMBBAuniMf8GACdEMV0uJRPN8fcuhuKmG5QYOjW
/YUAn2AkN6F0slQpbkbDxqLJOunSMzrB
=dwrv
-----END PGP SIGNATURE-----
More information about the Oisf-users
mailing list