[Oisf-users] memcap drops etc
Peter Manev
petermanev at gmail.com
Thu Dec 6 13:30:05 UTC 2012
On Thu, Dec 6, 2012 at 1:18 PM, Christophe Vandeplas <
christophe at vandeplas.com> wrote:
> On Thu, Dec 6, 2012 at 12:58 PM, Peter Manev <petermanev at gmail.com> wrote:
> >
> >
> > On Thu, Dec 6, 2012 at 12:26 PM, Christophe Vandeplas
> > <christophe at vandeplas.com> wrote:
> >>
> >> trying to reply to all the questions, also from Anoop.
> >>
> >> On Thu, Dec 6, 2012 at 11:55 AM, Peter Manev <petermanev at gmail.com>
> wrote:
> >> > Hi Cristophe,
> >> >
> >> > sorry - i missed the info from you.
> >> > Ok HW is definitely enough for that traffic.
> >> >
> >> > Do you use af_packet?
> >>
> >> no, I'll activate it on this IDS by using the eth2 interface only.
> >> Fortunately that's an IDS where the bond0 was not really necessary,
> >> but we prefer to keep every IDS as identical as possible. I'll have to
> >> dig into the AF_PACKET documentation to understand how I should
> >> configure it to receive on two physical interfaces.
> >>
> >> > Is Suriata running on all 8 cores?
> >>
> >> yep, on every machine it uses CPU from all cores.
> >>
> >> > bond0 interface - is that bridged by any chance?
> >>
> >> nope, that is/was not bridged. As I just switched to direct interface
> >> usage with AF_PACKET to eth2. This is not relevant anymore.
> >>
> >> /etc/network/interfaces is
> >> auto eth2
> >> iface eth2 inet manual
> >> pre-up ifconfig $IFACE up promisc
> >> post-down ifconfig $IFACE down
> >> bond-master bond0
> >>
> >> # bonding interfaces for easier sniffing
> >> auto bond0
> >> iface bond0 inet manual
> >> pre-up ifconfig $IFACE up promisc
> >> post-down ifconfig $IFACE down
> >> bond-mode balance-rr
> >> bond-miimon 100
> >> bond-slaves none
> >>
> >>
> >> > Do you have checksums enabled or disabled?
> >>
> >> enabled (as shown below)
> >>
> >> > FlowTimeout values - you should try to lower them.
> >>
> >> ok,
> >>
> >> > Can you describe the ruleset you're using?
> >>
> >> 44538 signatures processed. 711 are IP-only rules, 43495 are
> >> inspecting packet payload, 13901 inspect application layer, 0 are
> >> decoder event only
> >
> > do i read this correctly - 44K rules? :)
>
> yep :-) we mainly use privately-shared lists of dns names/ips, ...
> of targeted attacks.
> Hostnames generate http, dns (udp/tcp) rules, so it grows quite fast)
>
> > But more importantly - which Suriacta ver are you using?
>
> Suricata 1.3.4 from the ubuntu ppa repo.
>
I am glad to hear that you are using PPA :)
you should/could also give 1.4rc1 a try - located in our suricata-beta
repository
>
> New config is reloaded with suricata -D --pidfile
> /var/run/suricata.pid -c /etc/suricata/suricata.yaml --af-packet=eth2
> --runmode=workers
> raised flow.memcap to 3gb (was 2gb)
> raised stream.memcap to 3gb (was 2gb)
> lowered stream.reassembly.memcap to 1mb (default)
>
> So that 6 GB of the 8 GB, that gives 2 GB for the rest of suri and the
> OS. Is that a correct interpretation?
>
>
> From the new stats (http://pastebin.com/HNWRUBEM ) I see that the
> tcp.reassembly_gap is raising quickly (997)
> There are no drops
> capture.kernel_drops | AFPacketeth21 | 0
> tcp.ssn_memcap_drop | AFPacketeth21 | 0
> tcp.segment_memcap_drop | AFPacketeth21 | 0
>
> But this seems weird, the decoder sees less packets (272) than the
> kernel. While the kernel reports no drops. Or that's perhaps because
> it's somewhere in between?
> capture.kernel_packets | AFPacketeth21 | 948484
> capture.kernel_drops | AFPacketeth21 | 0
> decoder.pkts | AFPacketeth21 | 948756
>
>
>
>
>
> At startup suri now says:
> 6/12/2012 -- 13:04:11 - <Info> - 11 rule files processed. 43201 rules
> succesfully loaded, 60 rules failed
> 6/12/2012 -- 13:05:49 - <Info> - 44538 signatures processed. 711 are
> IP-only rules, 43495 are inspecting packet payload, 13901 inspect
> application layer, 0 are decoder event only
> 6/12/2012 -- 13:05:49 - <Info> - building signature grouping
> structure, stage 1: adding signatures to signature source addresses...
> complete
> 6/12/2012 -- 13:05:50 - <Info> - building signature grouping
> structure, stage 2: building source address list... complete
> 6/12/2012 -- 13:06:16 - <Info> - building signature grouping
> structure, stage 3: building destination address lists... complete
> 6/12/2012 -- 13:06:29 - <Info> - Threshold config parsed: 0 rule(s) found
> 6/12/2012 -- 13:06:29 - <Info> - Core dump size set to unlimited.
> 6/12/2012 -- 13:06:29 - <Info> - fast output device (regular)
> initialized: fast.log
> 6/12/2012 -- 13:06:29 - <Info> - Unified2-alert initialized: filename
> unified2.alert, limit 32 MB
> 6/12/2012 -- 13:06:29 - <Info> - http-log output device (regular)
> initialized: http.log
> 6/12/2012 -- 13:06:29 - <Info> - Using round-robin cluster mode for
> AF_PACKET (iface eth2)
> 6/12/2012 -- 13:06:29 - <Info> - Enabling mmaped capture on iface eth2
> 6/12/2012 -- 13:06:29 - <Info> - Going to use 1 thread(s)
> 6/12/2012 -- 13:06:29 - <Info> - RunModeIdsAFPSingle initialised
> 6/12/2012 -- 13:06:29 - <Info> - stream "max-sessions": 262144
> 6/12/2012 -- 13:06:29 - <Info> - stream "prealloc-sessions": 32768
> 6/12/2012 -- 13:06:29 - <Info> - stream "memcap": 3221225472
> 6/12/2012 -- 13:06:29 - <Info> - stream "midstream" session pickups:
> disabled
> 6/12/2012 -- 13:06:29 - <Info> - stream "async-oneside": disabled
> 6/12/2012 -- 13:06:29 - <Info> - stream "checksum-validation": enabled
> 6/12/2012 -- 13:06:29 - <Info> - stream."inline": disabled
> 6/12/2012 -- 13:06:29 - <Info> - Enabling zero copy mode
> 6/12/2012 -- 13:06:29 - <Info> - stream.reassembly "memcap": 1073741824
> 6/12/2012 -- 13:06:29 - <Info> - stream.reassembly "depth": 1048576
> 6/12/2012 -- 13:06:29 - <Info> - stream.reassembly "toserver-chunk-size":
> 2560
> 6/12/2012 -- 13:06:29 - <Info> - stream.reassembly "toclient-chunk-size":
> 2560
> 6/12/2012 -- 13:06:29 - <Info> - AF_PACKET RX Ring params:
> block_size=32768 block_nr=52 frame_size=1584 frame_nr=1040
> 6/12/2012 -- 13:06:30 - <Info> - all 1 packet processing threads, 3
> management threads initialized, engine started.
>
you mention you have 8 cores. Did you configure 8 threads for AF_PACKET in
the yaml section?
af-packet:
- interface: eth0
# Number of receive threads (>1 will enable experimental flow pinned
# runmode)
threads: 1
you could try changing threads:1 to threads:8
that however should not be an issue in this case ...since your traffic is
only 15Mbs
>
>
> >>
> >>
> >> the ruleset is very simple with tcp, http and udp filters. Nothing
> >> really spectacular.
> >> I wouldn't expect the ruleset to be a problem because CPU load is very
> >> very low. (even on the 130Mbps IDS it's only at 150-180% of the 800%
> >> available)
> >>
> >>
> >> I'll re-read what Victor said and will continue hunting for the cause.
> >> Thanks for all these fast replies !
> >>
> >> Christophe
> >>
> >> >
> >> > thank you
> >> >
> >> >
> >> > On Thu, Dec 6, 2012 at 11:40 AM, Christophe Vandeplas
> >> > <christophe at vandeplas.com> wrote:
> >> >>
> >> >> On Thu, Dec 6, 2012 at 11:21 AM, Peter Manev <petermanev at gmail.com>
> >> >> wrote:
> >> >> > Hi,
> >> >> >
> >> >> > what (how much) traffic do you average?
> >> >>
> >> >> Hello Peter,
> >> >>
> >> >> That was written in my mail, one of the IDSses sees only 15Mbps
> during
> >> >> the day on average. Spikes up to 40Mbps (but very short spikes 4
> times
> >> >> a day). That should certainly be feasible with such a system.
> >> >>
> >> >> Once I get that IDS working fine I'll finetune the settings of the
> >> >> others. (150 Mbps and 80 Mbps on average during the day)
> >> >>
> >> >>
> >> >> > On Thu, Dec 6, 2012 at 11:17 AM, Christophe Vandeplas
> >> >> > <christophe at vandeplas.com> wrote:
> >> >> >>
> >> >> >> Hello,
> >> >> >>
> >> >> >>
> >> >> >> Almost all my IDSses are having
> >> >> >> tcp.segment_memcap_drop
> >> >> >> tcp.reassembly_gap
> >> >> >>
> >> >> >> And some of them have
> >> >> >> tcp.ssn_memcap_drop
> >> >> >>
> >> >> >> I have been playing around with the memory settings in suricata,
> but
> >> >> >> I
> >> >> >> must admit it still looks very unclear to me, any help would
> really
> >> >> >> be
> >> >> >> appreciated.
> >> >> >>
> >> >> >> To attack this problem I'm now concentrating my efforts on the IDS
> >> >> >> dealing with the least traffic: during the day average of 15 Mbps.
> >> >> >> The IDS has 8 virtual-cores (4-core + ht = 8 ), and 8 GB of ram.
> And
> >> >> >> is sniffing using -i on a bond0 interface.
> >> >> >>
> >> >> >> The stats file is here: http://pastebin.com/kSVFDHRM
> >> >> >>
> >> >> >>
> >> >> >> Outputs that are on: fast, unified2, http, stats, syslog.
> >> >> >> I did not change anything in the threading section.
> >> >> >> Defrag is also default:
> >> >> >> defrag:
> >> >> >> max-frags: 65535
> >> >> >> prealloc: yes
> >> >> >> timeout: 60
> >> >> >>
> >> >> >> Raised flow:
> >> >> >> flow:
> >> >> >> memcap: 2gb
> >> >> >> hash-size: 65536
> >> >> >> prealloc: 10000
> >> >> >> emergency-recovery: 30
> >> >> >> prune-flows: 5
> >> >> >>
> >> >> >> Flow-timeouts are default, and I raised stream memcaps:
> >> >> >> stream:
> >> >> >> memcap: 2gb
> >> >> >> checksum-validation: yes # reject wrong csums
> >> >> >> inline: no # no inline mode
> >> >> >> reassembly:
> >> >> >> memcap: 1gb
> >> >> >> depth: 8mb # reassemble 1mb into a stream
> >> >> >> toserver-chunk-size: 2560
> >> >> >> toclient-chunk-size: 2560
> >> >> >>
> >> >> >>
> >> >> >> Any advice to further finetune is welcome !
> >> >> >>
> >> >> >> Thanks a lot
> >> >> >> Christophe
> >> >> >> _______________________________________________
> >> >> >> Suricata IDS Users mailing list:
> >> >> >> oisf-users at openinfosecfoundation.org
> >> >> >> Site: http://suricata-ids.org | Support:
> >> >> >> http://suricata-ids.org/support/
> >> >> >> List:
> >> >> >>
> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> >> >> >> OISF: http://www.openinfosecfoundation.org/
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > Regards,
> >> >> > Peter Manev
> >> >> >
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > Regards,
> >> > Peter Manev
> >> >
> >
> >
> >
> >
> > --
> > Regards,
> > Peter Manev
> >
>
--
Regards,
Peter Manev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20121206/70c853d4/attachment-0002.html>
More information about the Oisf-users
mailing list