[Oisf-users] memcap drops etc

Thu Dec 6 12:18:53 UTC 2012

On Thu, Dec 6, 2012 at 12:58 PM, Peter Manev <petermanev at gmail.com> wrote:
>
>
> On Thu, Dec 6, 2012 at 12:26 PM, Christophe Vandeplas
> <christophe at vandeplas.com> wrote:
>>
>> trying to reply to all the questions, also from Anoop.
>>
>> On Thu, Dec 6, 2012 at 11:55 AM, Peter Manev <petermanev at gmail.com> wrote:
>> > Hi Cristophe,
>> >
>> > sorry  - i missed the info from you.
>> > Ok HW is definitely enough for that traffic.
>> >
>> > Do you use af_packet?
>>
>> no, I'll activate it on this IDS by using the  eth2 interface only.
>> Fortunately that's an IDS where the bond0 was not really necessary,
>> but we prefer to keep every IDS as identical as possible. I'll have to
>> dig into the AF_PACKET documentation to understand how I should
>> configure it to receive on two physical interfaces.
>>
>> > Is Suriata running on all 8 cores?
>>
>> yep, on every machine it uses CPU from all cores.
>>
>> > bond0 interface - is that bridged by any chance?
>>
>> nope, that is/was not bridged. As I just switched to direct interface
>> usage with AF_PACKET to eth2. This is not relevant anymore.
>>
>> /etc/network/interfaces is
>> auto eth2
>> iface eth2 inet manual
>>     pre-up ifconfig $IFACE up promisc
>>     post-down ifconfig $IFACE down
>>     bond-master bond0
>>
>> # bonding interfaces for easier sniffing
>> auto bond0
>> iface bond0 inet manual
>>     pre-up ifconfig $IFACE up promisc
>>     post-down ifconfig $IFACE down
>>     bond-mode balance-rr
>>     bond-miimon 100
>>     bond-slaves none
>>
>>
>> > Do you have checksums enabled or disabled?
>>
>> enabled (as shown below)
>>
>> > FlowTimeout values - you should try to lower them.
>>
>> ok,
>>
>> > Can you describe the ruleset you're using?
>>
>>  44538 signatures processed. 711 are IP-only rules, 43495 are
>> inspecting packet payload, 13901 inspect application layer, 0 are
>> decoder event only
>
> do i read this correctly - 44K rules? :)

yep :-)   we mainly use privately-shared lists of dns names/ips, ...
of targeted attacks.
Hostnames generate http, dns (udp/tcp) rules, so it grows quite fast)

> But more importantly - which Suriacta ver are you using?

Suricata 1.3.4 from the ubuntu ppa repo.

New config is reloaded with suricata -D --pidfile
/var/run/suricata.pid -c /etc/suricata/suricata.yaml --af-packet=eth2
--runmode=workers
raised flow.memcap to 3gb   (was 2gb)
raised stream.memcap to 3gb (was 2gb)
lowered stream.reassembly.memcap to 1mb (default)

So that 6 GB of the 8 GB, that gives 2 GB for the rest of suri and the
OS. Is that a correct interpretation?

>From the new stats (http://pastebin.com/HNWRUBEM ) I see that the
tcp.reassembly_gap is raising quickly (997)
There are no drops
capture.kernel_drops      | AFPacketeth21             | 0
tcp.ssn_memcap_drop       | AFPacketeth21             | 0
tcp.segment_memcap_drop   | AFPacketeth21             | 0

At startup suri now says:
6/12/2012 -- 13:04:11 - <Info> - 11 rule files processed. 43201 rules
succesfully loaded, 60 rules failed
6/12/2012 -- 13:05:49 - <Info> - 44538 signatures processed. 711 are
IP-only rules, 43495 are inspecting packet payload, 13901 inspect
application layer, 0 are decoder event only
6/12/2012 -- 13:05:49 - <Info> - building signature grouping
structure, stage 1: adding signatures to signature source addresses...
complete
6/12/2012 -- 13:05:50 - <Info> - building signature grouping
structure, stage 2: building source address list... complete
6/12/2012 -- 13:06:16 - <Info> - building signature grouping
structure, stage 3: building destination address lists... complete
6/12/2012 -- 13:06:29 - <Info> - Threshold config parsed: 0 rule(s) found
6/12/2012 -- 13:06:29 - <Info> - Core dump size set to unlimited.
6/12/2012 -- 13:06:29 - <Info> - fast output device (regular)
initialized: fast.log
6/12/2012 -- 13:06:29 - <Info> - Unified2-alert initialized: filename
unified2.alert, limit 32 MB
6/12/2012 -- 13:06:29 - <Info> - http-log output device (regular)
initialized: http.log
6/12/2012 -- 13:06:29 - <Info> - Using round-robin cluster mode for
AF_PACKET (iface eth2)
6/12/2012 -- 13:06:29 - <Info> - Enabling mmaped capture on iface eth2
6/12/2012 -- 13:06:29 - <Info> - Going to use 1 thread(s)
6/12/2012 -- 13:06:29 - <Info> - RunModeIdsAFPSingle initialised
6/12/2012 -- 13:06:29 - <Info> - stream "max-sessions": 262144
6/12/2012 -- 13:06:29 - <Info> - stream "prealloc-sessions": 32768
6/12/2012 -- 13:06:29 - <Info> - stream "memcap": 3221225472
6/12/2012 -- 13:06:29 - <Info> - stream "midstream" session pickups: disabled
6/12/2012 -- 13:06:29 - <Info> - stream "async-oneside": disabled
6/12/2012 -- 13:06:29 - <Info> - stream "checksum-validation": enabled
6/12/2012 -- 13:06:29 - <Info> - stream."inline": disabled
6/12/2012 -- 13:06:29 - <Info> - Enabling zero copy mode
6/12/2012 -- 13:06:29 - <Info> - stream.reassembly "memcap": 1073741824
6/12/2012 -- 13:06:29 - <Info> - stream.reassembly "depth": 1048576
6/12/2012 -- 13:06:29 - <Info> - stream.reassembly "toserver-chunk-size": 2560
6/12/2012 -- 13:06:29 - <Info> - stream.reassembly "toclient-chunk-size": 2560
6/12/2012 -- 13:06:29 - <Info> - AF_PACKET RX Ring params:
block_size=32768 block_nr=52 frame_size=1584 frame_nr=1040
6/12/2012 -- 13:06:30 - <Info> - all 1 packet processing threads, 3
management threads initialized, engine started.

>>
>>
>> the ruleset is very simple with tcp, http and udp filters. Nothing
>> really spectacular.
>> I wouldn't expect the ruleset to be a problem because CPU load is very
>> very low. (even on the 130Mbps IDS it's only at 150-180% of the 800%
>> available)
>>
>>
>> I'll re-read what Victor said and will continue hunting for the cause.
>> Thanks for all these fast replies !
>>
>> Christophe
>>
>> >
>> > thank you
>> >
>> >
>> > On Thu, Dec 6, 2012 at 11:40 AM, Christophe Vandeplas
>> > <christophe at vandeplas.com> wrote:
>> >>
>> >> On Thu, Dec 6, 2012 at 11:21 AM, Peter Manev <petermanev at gmail.com>
>> >> wrote:
>> >> > Hi,
>> >> >
>> >> > what (how much) traffic do you average?
>> >>
>> >> Hello Peter,
>> >>
>> >> That was written in my mail, one of the IDSses sees only 15Mbps during
>> >> the day on average. Spikes up to 40Mbps (but very short spikes 4 times
>> >> a day). That should certainly be feasible with such a system.
>> >>
>> >> Once I get that IDS working fine I'll finetune the settings of the
>> >> others. (150 Mbps and 80 Mbps on average during the day)
>> >>
>> >>
>> >> > On Thu, Dec 6, 2012 at 11:17 AM, Christophe Vandeplas
>> >> > <christophe at vandeplas.com> wrote:
>> >> >>
>> >> >> Hello,
>> >> >>
>> >> >>
>> >> >> Almost all my IDSses are having
>> >> >> tcp.segment_memcap_drop
>> >> >> tcp.reassembly_gap
>> >> >>
>> >> >> And some of them have
>> >> >> tcp.ssn_memcap_drop
>> >> >>
>> >> >> I have been playing around with the memory settings in suricata, but
>> >> >> I
>> >> >> must admit it still looks very unclear to me, any help would really
>> >> >> be
>> >> >> appreciated.
>> >> >>
>> >> >> To attack this problem I'm now concentrating my efforts on the IDS
>> >> >> dealing with the least traffic: during the day average of 15 Mbps.
>> >> >> The IDS has 8 virtual-cores (4-core + ht = 8 ), and 8 GB of ram. And
>> >> >> is sniffing using -i on a bond0 interface.
>> >> >>
>> >> >> The stats file is here: http://pastebin.com/kSVFDHRM
>> >> >>
>> >> >>
>> >> >> Outputs that are on: fast, unified2, http, stats, syslog.
>> >> >> I did not change anything in the threading section.
>> >> >> Defrag is also default:
>> >> >> defrag:
>> >> >>   max-frags: 65535
>> >> >>   prealloc: yes
>> >> >>   timeout: 60
>> >> >>
>> >> >> Raised flow:
>> >> >> flow:
>> >> >>   memcap: 2gb
>> >> >>   hash-size: 65536
>> >> >>   prealloc: 10000
>> >> >>   emergency-recovery: 30
>> >> >>   prune-flows: 5
>> >> >>
>> >> >> Flow-timeouts are default, and I raised stream memcaps:
>> >> >> stream:
>> >> >>   memcap: 2gb
>> >> >>   checksum-validation: yes      # reject wrong csums
>> >> >>   inline: no                    # no inline mode
>> >> >>   reassembly:
>> >> >>     memcap: 1gb
>> >> >>     depth: 8mb                  # reassemble 1mb into a stream
>> >> >>     toserver-chunk-size: 2560
>> >> >>     toclient-chunk-size: 2560
>> >> >>
>> >> >>
>> >> >> Any advice to further finetune is welcome !
>> >> >>
>> >> >> Thanks a lot
>> >> >> Christophe
>> >> >> _______________________________________________
>> >> >> Suricata IDS Users mailing list:
>> >> >> oisf-users at openinfosecfoundation.org
>> >> >> Site: http://suricata-ids.org | Support:
>> >> >> http://suricata-ids.org/support/
>> >> >> List:
>> >> >> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>> >> >> OISF: http://www.openinfosecfoundation.org/
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Regards,
>> >> > Peter Manev
>> >> >
>> >
>> >
>> >
>> >
>> > --
>> > Regards,
>> > Peter Manev
>> >
>
>
>
>
> --
> Regards,
> Peter Manev
>