[Oisf-users] tcp.segment_memcap_drop couldn't be kept at zero, no matters how much memory we assign
Peter Manev
petermanev at gmail.com
Sat Dec 1 13:03:16 UTC 2012
Hi ,
Martin is very right about the flow-timeouts - very important, not to
forget to adjust those.
300 sec is 5 min...on a busy network .... -
tcp:
established: 3600 - (default)
1 hr can have some serious impact :)
It is funny you mention about the drops.. I just had a quick chat with
Victor about drops in general just a few days ago.
Here is some of our values/results on one of our test boxes (9.5Gb/s
traffic):
YAML:
flow-timeouts:
default:
new: 5 #30
established: 10# 300
closed: 0
emergency-new: 1 #10
emergency-established: 2 #100
emergency-closed: 0
tcp:
new: 5 #60
established: 300 # 3600
closed: 10 #30
emergency-new: 1 # 10
emergency-established: 5 # 300
emergency-closed: 20 #20
udp:
new: 5 #30
established: 5 # 300
emergency-new: 5 #10
emergency-established: 5 # 100
icmp:
new: 5 #30
established: 5 % 300
emergency-new: 5 #10
emergency-established: 5 # 100
......
stream:
memcap: 16gb
max-sessions: 20000000
prealloc-sessions: 10000000
checksum-validation: no # reject wrong csums
#checksum-validation: yes # reject wrong csums
inline: no # no inline mode
reassembly:
memcap: 12gb
#memcap: 8gb
depth: 12mb # reassemble 1mb into a stream
toserver-chunk-size: 2560
toclient-chunk-size: 2560
# Host table:
#
# Host table is used by tagging and per host thresholding subsystems.
#
host:
hash-size: 4096
prealloc: 1000
memcap: 16777216
......
# Defrag settings:
defrag:
#trackers: 262144 # number of defragmented flows to follow
#max-frags: 262144 #number of fragments per-flow
trackers: 65535
max-frags: 65535 # number of fragments per-flow
prealloc: yes
timeout: 10
Al this is using af_packet 16 threads , on a 16CPU(with Hyperthrd) box 32
GB RAM, with some special intel 10G NIC tuning, ubuntu LTS 12.04, running
latest git with 7K EmThr rules.
Some more info:
pevman at suricata:~$ sudo grep -n "drop"
/var/data/regit/log/suricata/stats.log | tail -48
2504179:capture.kernel_drops | AFPacketeth31 | 0
2504209:tcp.ssn_memcap_drop | AFPacketeth31 | 0
2504218:tcp.segment_memcap_drop | AFPacketeth31 | 0
2504224:capture.kernel_drops | AFPacketeth32 | 0
2504254:tcp.ssn_memcap_drop | AFPacketeth32 | 0
2504263:tcp.segment_memcap_drop | AFPacketeth32 | 0
2504269:capture.kernel_drops | AFPacketeth33 | 0
2504299:tcp.ssn_memcap_drop | AFPacketeth33 | 0
2504308:tcp.segment_memcap_drop | AFPacketeth33 | 0
2504314:capture.kernel_drops | AFPacketeth34 | 0
2504344:tcp.ssn_memcap_drop | AFPacketeth34 | 0
2504353:tcp.segment_memcap_drop | AFPacketeth34 | 0
2504359:capture.kernel_drops | AFPacketeth35 | 0
2504389:tcp.ssn_memcap_drop | AFPacketeth35 | 0
2504398:tcp.segment_memcap_drop | AFPacketeth35 | 0
2504404:capture.kernel_drops | AFPacketeth36 | 0
2504434:tcp.ssn_memcap_drop | AFPacketeth36 | 0
2504443:tcp.segment_memcap_drop | AFPacketeth36 | 0
2504449:capture.kernel_drops | AFPacketeth37 | 0
2504479:tcp.ssn_memcap_drop | AFPacketeth37 | 0
2504488:tcp.segment_memcap_drop | AFPacketeth37 | 0
2504494:capture.kernel_drops | AFPacketeth38 | 0
2504524:tcp.ssn_memcap_drop | AFPacketeth38 | 0
2504533:tcp.segment_memcap_drop | AFPacketeth38 | 0
2504539:capture.kernel_drops | AFPacketeth39 | 0
2504569:tcp.ssn_memcap_drop | AFPacketeth39 | 0
2504578:tcp.segment_memcap_drop | AFPacketeth39 | 0
2504584:capture.kernel_drops | AFPacketeth310 | 0
2504614:tcp.ssn_memcap_drop | AFPacketeth310 | 0
2504623:tcp.segment_memcap_drop | AFPacketeth310 | 0
2504629:capture.kernel_drops | AFPacketeth311 | 0
2504659:tcp.ssn_memcap_drop | AFPacketeth311 | 0
2504668:tcp.segment_memcap_drop | AFPacketeth311 | 0
2504674:capture.kernel_drops | AFPacketeth312 | 0
2504704:tcp.ssn_memcap_drop | AFPacketeth312 | 0
2504713:tcp.segment_memcap_drop | AFPacketeth312 | 0
2504719:capture.kernel_drops | AFPacketeth313 | 0
2504749:tcp.ssn_memcap_drop | AFPacketeth313 | 0
2504758:tcp.segment_memcap_drop | AFPacketeth313 | 0
2504764:capture.kernel_drops | AFPacketeth314 | 0
2504794:tcp.ssn_memcap_drop | AFPacketeth314 | 0
2504803:tcp.segment_memcap_drop | AFPacketeth314 | 0
2504809:capture.kernel_drops | AFPacketeth315 | 0
2504839:tcp.ssn_memcap_drop | AFPacketeth315 | 0
2504848:tcp.segment_memcap_drop | AFPacketeth315 | 0
2504854:capture.kernel_drops | AFPacketeth316 | 0
2504884:tcp.ssn_memcap_drop | AFPacketeth316 | 0
2504893:tcp.segment_memcap_drop | AFPacketeth316 | 0
*pevman at suricata:~$ suricata --build-info*
[10384] 1/12/2012 -- 14:28:44 - (suricata.c:560) <Info> (SCPrintBuildInfo)
-- This is Suricata version 1.4dev (rev 005f7a2)
[10384] 1/12/2012 -- 14:28:44 - (suricata.c:633) <Info> (SCPrintBuildInfo)
-- Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 PF_RING AF_PACKET
HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK
HAVE_HTP_TX_GET_RESPONSE_HEADERS_RAW HAVE_NSS PROFILING
[10384] 1/12/2012 -- 14:28:44 - (suricata.c:647) <Info> (SCPrintBuildInfo)
-- 64-bits, Little-endian architecture
[10384] 1/12/2012 -- 14:28:44 - (suricata.c:649) <Info> (SCPrintBuildInfo)
-- GCC version 4.6.3, C version 199901
[10384] 1/12/2012 -- 14:28:44 - (suricata.c:655) <Info> (SCPrintBuildInfo)
-- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_1
[10384] 1/12/2012 -- 14:28:44 - (suricata.c:658) <Info> (SCPrintBuildInfo)
-- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_2
[10384] 1/12/2012 -- 14:28:44 - (suricata.c:661) <Info> (SCPrintBuildInfo)
-- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4
[10384] 1/12/2012 -- 14:28:44 - (suricata.c:664) <Info> (SCPrintBuildInfo)
-- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8
[10384] 1/12/2012 -- 14:28:44 - (suricata.c:667) <Info> (SCPrintBuildInfo)
-- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_16
[10384] 1/12/2012 -- 14:28:44 - (suricata.c:671) <Info> (SCPrintBuildInfo)
-- compiled with -fstack-protector
[10384] 1/12/2012 -- 14:28:44 - (suricata.c:677) <Info> (SCPrintBuildInfo)
-- compiled with _FORTIFY_SOURCE=2
[10384] 1/12/2012 -- 14:28:44 - (suricata.c:680) <Info> (SCPrintBuildInfo)
-- compiled with libhtp 0.2.11, linked against 0.2.11
pevman at suricata:~$ sudo grep -n "uptime"
/var/data/regit/log/suricata/stats.log | tail -4
2503442:Date: 12/1/2012 -- 14:27:56 (uptime: 0d, 18h 07m 04s)
2504174:Date: 12/1/2012 -- 14:28:15 (uptime: 0d, 18h 07m 23s)
2504906:Date: 12/1/2012 -- 14:28:34 (uptime: 0d, 18h 07m 42s)
2505638:Date: 12/1/2012 -- 14:28:53 (uptime: 0d, 18h 08m 01s)
pevman at suricata:~$ sudo tcpstat -i eth3
Time:1354365172 n=6106758 avg=984.85 stddev=663.77
bps=9622763462.40
Time:1354365177 n=6126927 avg=981.51 stddev=663.29
bps=9621826076.80
Time:1354365182 n=6110921 avg=984.19 stddev=662.02
bps=9622922160.00
Time:1354365187 n=6126978 avg=981.50 stddev=662.38
bps=9621846648.00
Time:1354365192 n=6109322 avg=984.46 stddev=661.25
bps=9623061092.80
Time:1354365197 n=6146841 avg=978.24 stddev=662.73
bps=9620970840.00
^CTime:1354365202 n=112243 avg=982.41 stddev=663.97
bps=176430308.80
pevman at suricata:~$ uname -a
Linux suricata 3.2.0-30-generic #48-Ubuntu SMP Fri Aug 24 16:52:48 UTC 2012
x86_64 x86_64 x86_64 GNU/Linux
pevman at suricata:~$
hope it helps.
thanks
On Sat, Dec 1, 2012 at 3:54 AM, Martin Holste <mcholste at gmail.com> wrote:
> Adjust your default timeouts much lower so that streams are taken out of
> the connection pool more quickly.
>
> This config is aggressive, but I think you'll find it does the trick. If
> it doesn't work, I'd like to know:
>
> flow-timeouts:
>
> default:
> new: 1 # 30
> established: 10 #300
> closed: 0
> emergency_new: 1 #10
> emergency_established: 1 #100
> emergency_closed: 0
> tcp:
> new: 1 #60
> established: 10 #3600
> closed: 0 #120
> emergency_new: 1 #10
> emergency_established: 5 #1 #300
> emergency_closed: 20
> udp:
> new: 1 #30
> established: 1 #300
> emergency_new: 1 #10
> emergency_established: 1 #100
> icmp:
> new: 1 #30
> established: 1 #300
> emergency_new: 1 #10
> emergency_established: 1 #100
>
>
>
>
> On Fri, Nov 30, 2012 at 4:15 PM, Dave Remien <dave.remien at gmail.com>wrote:
>
>> Fernando,
>>
>> If I'm reading your config file right, you're asking for 8.3 million
>> sessions of 512KB each? I think that works out to 4.3TB of RAM; rather more
>> than the 64GB memcap.
>>
>> Cheers,
>>
>> Dave
>>
>>
>> On Fri, Nov 30, 2012 at 10:24 AM, Fernando Sclavo <fsclavo at gmail.com>wrote:
>>
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA1
>>>
>>> Hello all!
>>> I'm installing an IDS on our company, monitoring two core switches with
>>> a sustained traffic of about 2gbps each. The server is a Dell R715, 32
>>> cores, 192Gb RAM with two Intel X520 nics. Suricata version is 1.4b3.
>>> The problem we are facing, is with tcp.segment_memcap_drop increasing
>>> continuosly once time tcp.reassembly_memuse reaches their max size
>>> (64gb!!)
>>> The related suricata.yaml stanza is:
>>>
>>> stream:
>>> memcap: 24gb
>>> checksum-validation: no # reject wrong csums
>>> inline: no # auto will use inline mode in IPS mode,
>>> yes or no set it statically
>>> max-sessions: 8388608
>>> prealloc-sessions: 8388608
>>> reassembly:
>>> memcap: 64gb
>>> depth: 512kb # reassemble 1mb into a stream
>>> toserver-chunk-size: 2560
>>> toclient-chunk-size: 2560
>>>
>>> Thanks in advance!
>>> -----BEGIN PGP SIGNATURE-----
>>> Version: GnuPG v1.4.11 (GNU/Linux)
>>> Comment: Using GnuPG with undefined - http://www.enigmail.net/
>>>
>>> iQIcBAEBAgAGBQJQuOviAAoJEDtYYV2Ws9eJD18P/2+QZR+6BXnk/FfXQeCw43Xh
>>> qynGiI3qnrg3SSaGdiWDrm0b8UuVuq/HXaAdIo0hzeDNgRLWjBKnnz4b3UA3HyIH
>>> cKpPUsEFUyc55KPSDzDW2mCGB/V//7f/Ude5DXG7/CZ9+xJu1jhuePfuE9Nl1yIi
>>> o3xmlI1mXXXc82rs0VGKDJ0ZwoN+/zmcnp1sW5mG42CKR2Hr9PcVKzP0IHbNZlHI
>>> Q0ishhXNrKcGCpHn9/J9gg44af6+7a0EdnOZOEgRNtOILfK6C5N4p5cwZfMAkYnL
>>> AcswoaER4ftBV49WpfWjTeOhEQxYaGFM8QURB0f30ODqMDoDUKX6lwjXm6+ZfQqr
>>> Y+mGzX/WFCeFI2A4KqgNamZi1IKKd83j0AxH8nYhWa9kPtws75L5iGYAQOE5yoVw
>>> oTnEncPlSLK+Mb/fhoc0crNeMkCKDV6uCFgpE/JKUtogG25nmcbSAIoE3Esa9iYq
>>> dRww7KhOZttLRXjZeRkm/bl1CmBDXDJ2sZQ8jZtqpGeFlIMi4BYCyQAKsKWyAji4
>>> 9LrDvtnew/jvWLCpNOfPrHWjRM+XbpD+k4YWO1imRWU6Or+E4Fgx9oiFNd9ni/DY
>>> l2NrSkq9RIixCVqrpNkWsEwCxN2pftJ4h0sXqTqkkhi8Ofhui60o1uNAOqMGURoN
>>> U30CUPowHUvuwnguE781
>>> =vy1s
>>> -----END PGP SIGNATURE-----
>>> _______________________________________________
>>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>>> Site: http://suricata-ids.org | Support:
>>> http://suricata-ids.org/support/
>>> List:
>>> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>> OISF: http://www.openinfosecfoundation.org/
>>>
>>
>>
>>
>> --
>> ".... We are such stuff
>> As dreams are made on; and our little life
>> Is rounded with a sleep."
>> -- Shakespeare, The Tempest - Act 4
>>
>>
>> _______________________________________________
>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
>> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>> OISF: http://www.openinfosecfoundation.org/
>>
>
>
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> OISF: http://www.openinfosecfoundation.org/
>
--
Regards,
Peter Manev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20121201/45458272/attachment-0001.html>
More information about the Oisf-users
mailing list