[Oisf-users] tcp.segment_memcap_drop couldn't be kept at zero, no matters how much memory we assign

Tue Dec 4 14:18:21 UTC 2012

   Off course!
Here it is: https://dl.dropbox.com/u/9555438/_usr_bin_suricata.0.crash.gz

On 12/04/2012 09:49 AM, Peter Manev wrote:

Hi,

I think a core dump would be very useful - for the dev folks.
Do you think this is possible?

On Tue, Dec 4, 2012 at 1:01 PM, Fernando Sclavo <fsclavo at gmail.com> wrote:

>  Hi, with the suggestions you gave me, and with some more reading, I
> almost keep the IDS without drops, but, unfortunatelly Suricata crash,
> always with the same error:
>
> [45249.968058] AFPacketeth58[2096] general protection ip:477861
> sp:7fbe28cc5bb0 error:0 in suricata[400000+1a5000]
>
> Please let me know how can I help to debug this problem.
> Thanks!
>
>
>
> On 12/01/2012 10:03 AM, Peter Manev wrote:
> > Hi ,
> >
> > Martin is very right about the flow-timeouts - very important, not to
> forget to adjust those.
> > 300 sec is 5 min...on a busy network .... -
> > tcp:
> > established: 3600 - (default)
> > 1 hr can have some serious impact :)
> >
> > It is funny you mention about the drops.. I just had a quick chat with
> Victor about drops in general just a few days ago.
> > Here is some of our values/results on one of our test boxes (9.5Gb/s
> traffic):
> >
> > YAML:
> > flow-timeouts:
> >
> > default:
> > new: 5 #30
> > established: 10# 300
> > closed: 0
> > emergency-new: 1 #10
> > emergency-established: 2 #100
> > emergency-closed: 0
> > tcp:
> > new: 5 #60
> > established: 300 # 3600
> > closed: 10 #30
> > emergency-new: 1 # 10
> > emergency-established: 5 # 300
> > emergency-closed: 20 #20
> > udp:
> > new: 5 #30
> > established: 5 # 300
> > emergency-new: 5 #10
> > emergency-established: 5 # 100
> > icmp:
> > new: 5 #30
> > established: 5 % 300
> > emergency-new: 5 #10
> > emergency-established: 5 # 100
> >
> > ......
> > stream:
> > memcap: 16gb
> > max-sessions: 20000000
> > prealloc-sessions: 10000000
> > checksum-validation: no # reject wrong csums
> > #checksum-validation: yes # reject wrong csums
> > inline: no # no inline mode
> > reassembly:
> > memcap: 12gb
> > #memcap: 8gb
> > depth: 12mb # reassemble 1mb into a stream
> > toserver-chunk-size: 2560
> > toclient-chunk-size: 2560
> >
> > # Host table:
> > #
> > # Host table is used by tagging and per host thresholding subsystems.
> > #
> > host:
> > hash-size: 4096
> > prealloc: 1000
> > memcap: 16777216
> >
> > ......
> > # Defrag settings:
> >
> > defrag:
> > #trackers: 262144 # number of defragmented flows to follow
> > #max-frags: 262144 #number of fragments per-flow
> > trackers: 65535
> > max-frags: 65535 # number of fragments per-flow
> > prealloc: yes
> > timeout: 10
> >
> >
> >
> > Al this is using af_packet 16 threads , on a 16CPU(with Hyperthrd) box
> 32 GB RAM, with some special intel 10G NIC tuning, ubuntu LTS 12.04,
> running latest git with 7K EmThr rules.
> > Some more info:
> >
> >
> >
> > pevman at suricata:~$ sudo grep -n "drop"
> /var/data/regit/log/suricata/stats.log | tail -48
> > 2504179:capture.kernel_drops | AFPacketeth31 | 0
> > 2504209:tcp.ssn_memcap_drop | AFPacketeth31 | 0
> > 2504218:tcp.segment_memcap_drop | AFPacketeth31 | 0
> > 2504224:capture.kernel_drops | AFPacketeth32 | 0
> > 2504254:tcp.ssn_memcap_drop | AFPacketeth32 | 0
> > 2504263:tcp.segment_memcap_drop | AFPacketeth32 | 0
> > 2504269:capture.kernel_drops | AFPacketeth33 | 0
> > 2504299:tcp.ssn_memcap_drop | AFPacketeth33 | 0
> > 2504308:tcp.segment_memcap_drop | AFPacketeth33 | 0
> > 2504314:capture.kernel_drops | AFPacketeth34 | 0
> > 2504344:tcp.ssn_memcap_drop | AFPacketeth34 | 0
> > 2504353:tcp.segment_memcap_drop | AFPacketeth34 | 0
> > 2504359:capture.kernel_drops | AFPacketeth35 | 0
> > 2504389:tcp.ssn_memcap_drop | AFPacketeth35 | 0
> > 2504398:tcp.segment_memcap_drop | AFPacketeth35 | 0
> > 2504404:capture.kernel_drops | AFPacketeth36 | 0
> > 2504434:tcp.ssn_memcap_drop | AFPacketeth36 | 0
> > 2504443:tcp.segment_memcap_drop | AFPacketeth36 | 0
> > 2504449:capture.kernel_drops | AFPacketeth37 | 0
> > 2504479:tcp.ssn_memcap_drop | AFPacketeth37 | 0
> > 2504488:tcp.segment_memcap_drop | AFPacketeth37 | 0
> > 2504494:capture.kernel_drops | AFPacketeth38 | 0
> > 2504524:tcp.ssn_memcap_drop | AFPacketeth38 | 0
> > 2504533:tcp.segment_memcap_drop | AFPacketeth38 | 0
> > 2504539:capture.kernel_drops | AFPacketeth39 | 0
> > 2504569:tcp.ssn_memcap_drop | AFPacketeth39 | 0
> > 2504578:tcp.segment_memcap_drop | AFPacketeth39 | 0
> > 2504584:capture.kernel_drops | AFPacketeth310 | 0
> > 2504614:tcp.ssn_memcap_drop | AFPacketeth310 | 0
> > 2504623:tcp.segment_memcap_drop | AFPacketeth310 | 0
> > 2504629:capture.kernel_drops | AFPacketeth311 | 0
> > 2504659:tcp.ssn_memcap_drop | AFPacketeth311 | 0
> > 2504668:tcp.segment_memcap_drop | AFPacketeth311 | 0
> > 2504674:capture.kernel_drops | AFPacketeth312 | 0
> > 2504704:tcp.ssn_memcap_drop | AFPacketeth312 | 0
> > 2504713:tcp.segment_memcap_drop | AFPacketeth312 | 0
> > 2504719:capture.kernel_drops | AFPacketeth313 | 0
> > 2504749:tcp.ssn_memcap_drop | AFPacketeth313 | 0
> > 2504758:tcp.segment_memcap_drop | AFPacketeth313 | 0
> > 2504764:capture.kernel_drops | AFPacketeth314 | 0
> > 2504794:tcp.ssn_memcap_drop | AFPacketeth314 | 0
> > 2504803:tcp.segment_memcap_drop | AFPacketeth314 | 0
> > 2504809:capture.kernel_drops | AFPacketeth315 | 0
> > 2504839:tcp.ssn_memcap_drop | AFPacketeth315 | 0
> > 2504848:tcp.segment_memcap_drop | AFPacketeth315 | 0
> > 2504854:capture.kernel_drops | AFPacketeth316 | 0
> > 2504884:tcp.ssn_memcap_drop | AFPacketeth316 | 0
> > 2504893:tcp.segment_memcap_drop | AFPacketeth316 | 0
> >
> > *pevman at suricata:~$ suricata --build-info*
> > [10384] 1/12/2012 -- 14:28:44 - (suricata.c:560) <Info>
> (SCPrintBuildInfo) -- This is Suricata version 1.4dev (rev 005f7a2)
> > [10384] 1/12/2012 -- 14:28:44 - (suricata.c:633) <Info>
> (SCPrintBuildInfo) -- Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1
> PF_RING AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1
> HAVE_HTP_URI_NORMALIZE_HOOK HAVE_HTP_TX_GET_RESPONSE_HEADERS_RAW HAVE_NSS
> PROFILING
> > [10384] 1/12/2012 -- 14:28:44 - (suricata.c:647) <Info>
> (SCPrintBuildInfo) -- 64-bits, Little-endian architecture
> > [10384] 1/12/2012 -- 14:28:44 - (suricata.c:649) <Info>
> (SCPrintBuildInfo) -- GCC version 4.6.3, C version 199901
> > [10384] 1/12/2012 -- 14:28:44 - (suricata.c:655) <Info>
> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_1
> > [10384] 1/12/2012 -- 14:28:44 - (suricata.c:658) <Info>
> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_2
> > [10384] 1/12/2012 -- 14:28:44 - (suricata.c:661) <Info>
> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4
> > [10384] 1/12/2012 -- 14:28:44 - (suricata.c:664) <Info>
> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8
> > [10384] 1/12/2012 -- 14:28:44 - (suricata.c:667) <Info>
> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_16
> > [10384] 1/12/2012 -- 14:28:44 - (suricata.c:671) <Info>
> (SCPrintBuildInfo) -- compiled with -fstack-protector
> > [10384] 1/12/2012 -- 14:28:44 - (suricata.c:677) <Info>
> (SCPrintBuildInfo) -- compiled with _FORTIFY_SOURCE=2
> > [10384] 1/12/2012 -- 14:28:44 - (suricata.c:680) <Info>
> (SCPrintBuildInfo) -- compiled with libhtp 0.2.11, linked against 0.2.11
> >
> > pevman at suricata:~$ sudo grep -n "uptime"
> /var/data/regit/log/suricata/stats.log | tail -4
> > 2503442:Date: 12/1/2012 -- 14:27:56 (uptime: 0d, 18h 07m 04s)
> > 2504174:Date: 12/1/2012 -- 14:28:15 (uptime: 0d, 18h 07m 23s)
> > 2504906:Date: 12/1/2012 -- 14:28:34 (uptime: 0d, 18h 07m 42s)
> > 2505638:Date: 12/1/2012 -- 14:28:53 (uptime: 0d, 18h 08m 01s)
> >
> > pevman at suricata:~$ sudo tcpstat -i eth3
> > Time:1354365172 n=6106758 avg=984.85 stddev=663.77 bps=9622763462.40
> > Time:1354365177 n=6126927 avg=981.51 stddev=663.29 bps=9621826076.80
> > Time:1354365182 n=6110921 avg=984.19 stddev=662.02 bps=9622922160.00
> > Time:1354365187 n=6126978 avg=981.50 stddev=662.38 bps=9621846648.00
> > Time:1354365192 n=6109322 avg=984.46 stddev=661.25 bps=9623061092.80
> > Time:1354365197 n=6146841 avg=978.24 stddev=662.73 bps=9620970840.00
> > ^CTime:1354365202 n=112243 avg=982.41 stddev=663.97 bps=176430308.80
> >
> > pevman at suricata:~$ uname -a
> > Linux suricata 3.2.0-30-generic #48-Ubuntu SMP Fri Aug 24 16:52:48 UTC
> 2012 x86_64 x86_64 x86_64 GNU/Linux
> > pevman at suricata:~$
> >
> >
> >
> >
> >
> > hope it helps.
> >
> > thanks
> >
> > On Sat, Dec 1, 2012 at 3:54 AM, Martin Holste <mcholste at gmail.com
> <mailto:mcholste at gmail.com> <mcholste at gmail.com>> wrote:
> >
> > Adjust your default timeouts much lower so that streams are taken out of
> the connection pool more quickly.
> >
> > This config is aggressive, but I think you'll find it does the trick. If
> it doesn't work, I'd like to know:
> >
> > flow-timeouts:
> >
> > default:
> > new: 1 # 30
> > established: 10 #300
> > closed: 0
> > emergency_new: 1 #10
> > emergency_established: 1 #100
> > emergency_closed: 0
> > tcp:
> > new: 1 #60
> > established: 10 #3600
> > closed: 0 #120
> > emergency_new: 1 #10
> > emergency_established: 5 #1 #300
> > emergency_closed: 20
> > udp:
> > new: 1 #30
> > established: 1 #300
> > emergency_new: 1 #10
> > emergency_established: 1 #100
> > icmp:
> > new: 1 #30
> > established: 1 #300
> > emergency_new: 1 #10
> > emergency_established: 1 #100
> >
> >
> >
> >
> > On Fri, Nov 30, 2012 at 4:15 PM, Dave Remien <dave.remien at gmail.com
> <mailto:dave.remien at gmail.com> <dave.remien at gmail.com>> wrote:
> >
> > Fernando,
> >
> > If I'm reading your config file right, you're asking for 8.3 million
> sessions of 512KB each? I think that works out to 4.3TB of RAM; rather more
> than the 64GB memcap.
> >
> > Cheers,
> >
> > Dave
> >
> >
> > On Fri, Nov 30, 2012 at 10:24 AM, Fernando Sclavo <fsclavo at gmail.com
> <mailto:fsclavo at gmail.com> <fsclavo at gmail.com>> wrote:
> >
>
> Hello all!
> I'm installing an IDS on our company, monitoring two core switches with
> a sustained traffic of about 2gbps each. The server is a Dell R715, 32
> cores, 192Gb RAM with two Intel X520 nics. Suricata version is 1.4b3.
> The problem we are facing, is with tcp.segment_memcap_drop increasing
> continuosly once time tcp.reassembly_memuse reaches their max size (64gb!!)
> The related suricata.yaml stanza is:
>
> stream:
>   memcap: 24gb
>   checksum-validation: no      # reject wrong csums
>   inline: no                  # auto will use inline mode in IPS mode,
> yes or no set it statically
>   max-sessions: 8388608
>   prealloc-sessions: 8388608
>   reassembly:
>     memcap: 64gb
>     depth: 512kb                  # reassemble 1mb into a stream
>     toserver-chunk-size: 2560
>     toclient-chunk-size: 2560
>
> Thanks in advance!
>
> > _______________________________________________
> > Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> <mailto:oisf-users at openinfosecfoundation.org><oisf-users at openinfosecfoundation.org>
> > Site: http://suricata-ids.org | Support:
> http://suricata-ids.org/support/
> > List:
> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> > OISF: http://www.openinfosecfoundation.org/
> >
> >
> >
> >
> > --
> > ".... We are such stuff
> > As dreams are made on; and our little life
> > Is rounded with a sleep."
> > -- Shakespeare, The Tempest - Act 4
> >
> >
> > _______________________________________________
> > Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> <mailto:oisf-users at openinfosecfoundation.org><oisf-users at openinfosecfoundation.org>
> > Site: http://suricata-ids.org | Support:
> http://suricata-ids.org/support/
> > List:
> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> > OISF: http://www.openinfosecfoundation.org/
> >
> >
> >
> > _______________________________________________
> > Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> <mailto:oisf-users at openinfosecfoundation.org><oisf-users at openinfosecfoundation.org>
> > Site: http://suricata-ids.org | Support:
> http://suricata-ids.org/support/
> > List:
> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> > OISF: http://www.openinfosecfoundation.org/
> >
> >
> >
> >
> > --
> > Regards,
> > Peter Manev
> >
>
>
>

-- 
Regards,
Peter Manev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20121204/2d8b4750/attachment-0002.html>