[Oisf-users] Suricata 2.1beta3 vs 2.0.7
Yasha Zislin
coolyasha at hotmail.com
Wed May 6 12:52:32 UTC 2015
These were over short period of time. Here are stats which are almost after one day of running.I monitor two span ports, once has 10% packet loss, second one is 3%.Same setup on 2.0.7 gives me packet loss of 0.05% for each span port monitored.
capture.kernel_packets | RxPFReth020 | 396154912capture.kernel_drops | RxPFReth020 | 39804970dns.memuse | RxPFReth020 | 4568927dns.memcap_state | RxPFReth020 | 0dns.memcap_global | RxPFReth020 | 0decoder.pkts | RxPFReth020 | 396154912decoder.bytes | RxPFReth020 | 243772693515decoder.invalid | RxPFReth020 | 34decoder.ipv4 | RxPFReth020 | 396115296decoder.ipv6 | RxPFReth020 | 42982decoder.ethernet | RxPFReth020 | 396154912decoder.raw | RxPFReth020 | 0decoder.sll | RxPFReth020 | 0decoder.tcp | RxPFReth020 | 352247594decoder.udp | RxPFReth020 | 43360111decoder.sctp | RxPFReth020 | 36decoder.icmpv4 | RxPFReth020 | 120913decoder.icmpv6 | RxPFReth020 | 24441decoder.ppp | RxPFReth020 | 0decoder.pppoe | RxPFReth020 | 0decoder.gre | RxPFReth020 | 0decoder.vlan | RxPFReth020 | 0decoder.vlan_qinq | RxPFReth020 | 0decoder.teredo | RxPFReth020 | 1063decoder.ipv4_in_ipv6 | RxPFReth020 | 0decoder.ipv6_in_ipv6 | RxPFReth020 | 0decoder.mpls | RxPFReth020 | 0decoder.avg_pkt_size | RxPFReth020 | 615decoder.max_pkt_size | RxPFReth020 | 1514defrag.ipv4.fragments | RxPFReth020 | 4815defrag.ipv4.reassembled | RxPFReth020 | 2303defrag.ipv4.timeouts | RxPFReth020 | 0defrag.ipv6.fragments | RxPFReth020 | 0defrag.ipv6.reassembled | RxPFReth020 | 0defrag.ipv6.timeouts | RxPFReth020 | 0defrag.max_frag_hits | RxPFReth020 | 0tcp.sessions | RxPFReth020 | 2374258tcp.ssn_memcap_drop | RxPFReth020 | 0tcp.pseudo | RxPFReth020 | 582718tcp.pseudo_failed | RxPFReth020 | 0tcp.invalid_checksum | RxPFReth020 | 0tcp.no_flow | RxPFReth020 | 0tcp.reused_ssn | RxPFReth020 | 505tcp.memuse | RxPFReth020 | 20649552tcp.syn | RxPFReth020 | 2491251tcp.synack | RxPFReth020 | 1892253tcp.rst | RxPFReth020 | 1079891tcp.segment_memcap_drop | RxPFReth020 | 0tcp.stream_depth_reached | RxPFReth020 | 6691tcp.reassembly_memuse | RxPFReth020 | 40392320000tcp.reassembly_gap | RxPFReth020 | 46171http.memuse | RxPFReth020 | 865185241http.memcap | RxPFReth020 | 0detect.alert | RxPFReth020 | 9562flow_mgr.closed_pruned | FlowManagerThread | 206743007flow_mgr.new_pruned | FlowManagerThread | 28953165flow_mgr.est_pruned | FlowManagerThread | 38698267flow.memuse | FlowManagerThread | 5586600240flow.spare | FlowManagerThread | 16007979flow.emerg_mode_entered | FlowManagerThread | 0flow.emerg_mode_over | FlowManagerThread | 0
> Date: Tue, 5 May 2015 23:49:13 +0200
> Subject: Re: [Oisf-users] Suricata 2.1beta3 vs 2.0.7
> From: petermanev at gmail.com
> To: coolyasha at hotmail.com
> CC: modversion at gmail.com; oisf-users at lists.openinfosecfoundation.org
>
> On Tue, May 5, 2015 at 4:26 PM, Yasha Zislin <coolyasha at hotmail.com> wrote:
> > Here is an example of one of the threads:
> >
> > capture.kernel_packets | RxPFReth220 | 4438207
> > capture.kernel_drops | RxPFReth220 | 466880
> > dns.memuse | RxPFReth220 | 3908544
> > dns.memcap_state | RxPFReth220 | 0
> > dns.memcap_global | RxPFReth220 | 0
> > decoder.pkts | RxPFReth220 | 4438207
> > decoder.bytes | RxPFReth220 | 3216813731
> > decoder.invalid | RxPFReth220 | 0
> > decoder.ipv4 | RxPFReth220 | 4438207
> > decoder.ipv6 | RxPFReth220 | 38
> > decoder.ethernet | RxPFReth220 | 4438207
> > decoder.raw | RxPFReth220 | 0
> > decoder.sll | RxPFReth220 | 0
> > decoder.tcp | RxPFReth220 | 4229782
> > decoder.udp | RxPFReth220 | 205264
> > decoder.sctp | RxPFReth220 | 0
> > decoder.icmpv4 | RxPFReth220 | 3161
> > decoder.icmpv6 | RxPFReth220 | 0
> > decoder.ppp | RxPFReth220 | 0
> > decoder.pppoe | RxPFReth220 | 0
> > decoder.gre | RxPFReth220 | 0
> > decoder.vlan | RxPFReth220 | 0
> > decoder.vlan_qinq | RxPFReth220 | 0
> > decoder.teredo | RxPFReth220 | 38
> > decoder.ipv4_in_ipv6 | RxPFReth220 | 0
> > decoder.ipv6_in_ipv6 | RxPFReth220 | 0
> > decoder.mpls | RxPFReth220 | 0
> > decoder.avg_pkt_size | RxPFReth220 | 724
> > decoder.max_pkt_size | RxPFReth220 | 1514
> > defrag.ipv4.fragments | RxPFReth220 | 0
> > defrag.ipv4.reassembled | RxPFReth220 | 0
> > defrag.ipv4.timeouts | RxPFReth220 | 0
> > defrag.ipv6.fragments | RxPFReth220 | 0
> > defrag.ipv6.reassembled | RxPFReth220 | 0
> > defrag.ipv6.timeouts | RxPFReth220 | 0
> > defrag.max_frag_hits | RxPFReth220 | 0
> > tcp.sessions | RxPFReth220 | 34053
> > tcp.ssn_memcap_drop | RxPFReth220 | 0
> > tcp.pseudo | RxPFReth220 | 11290
> > tcp.pseudo_failed | RxPFReth220 | 0
> > tcp.invalid_checksum | RxPFReth220 | 0
> > tcp.no_flow | RxPFReth220 | 0
> > tcp.reused_ssn | RxPFReth220 | 7
> > tcp.memuse | RxPFReth220 | 21511360
> > tcp.syn | RxPFReth220 | 37423
> > tcp.synack | RxPFReth220 | 34159
> > tcp.rst | RxPFReth220 | 19061
> > tcp.segment_memcap_drop | RxPFReth220 | 0
> > tcp.stream_depth_reached | RxPFReth220 | 100
> > tcp.reassembly_memuse | RxPFReth220 | 40392320000
> > tcp.reassembly_gap | RxPFReth220 | 3348
> > http.memuse | RxPFReth220 | 868151492
> > http.memcap | RxPFReth220 | 0
> > detect.alert | RxPFReth220 | 352
> > flow_mgr.closed_pruned | FlowManagerThread | 3978049
> > flow_mgr.new_pruned | FlowManagerThread | 217874
> > flow_mgr.est_pruned | FlowManagerThread | 407013
> > flow.memuse | FlowManagerThread | 5589481392
> > flow.spare | FlowManagerThread | 16000950
> > flow.emerg_mode_entered | FlowManagerThread | 0
> > flow.emerg_mode_over | FlowManagerThread | 0
> >
>
> Over what period of time are those stats for? (5 min/3hrs ?)
>
> >
> >> Date: Mon, 4 May 2015 10:13:23 +0200
> >
> >> Subject: Re: [Oisf-users] Suricata 2.1beta3 vs 2.0.7
> >> From: petermanev at gmail.com
> >> To: coolyasha at hotmail.com
> >> CC: modversion at gmail.com; oisf-users at lists.openinfosecfoundation.org
> >>
> >> On Fri, May 1, 2015 at 9:24 PM, Yasha Zislin <coolyasha at hotmail.com>
> >> wrote:
> >> > I think I've done that before and it was less that 96% of my RAM.
> >> >
> >> > All memcaps together equal to 58 gigs (I have 140gigs total RAM).
> >> > Also PFRING utilizes some RAM. When 2.0.7 starts it is using 50% of RAM.
> >> > After couple of days it gets to 96% and stays there.
> >>
> >> Ok. Anything unusual in the stats.log - decoder invalid counters,
> >> memcaps reached, tcp gaps, emergency mode entered .. ?
> >>
> >> >
> >> >> Date: Fri, 1 May 2015 15:15:31 +0200
> >> >
> >> >> Subject: Re: [Oisf-users] Suricata 2.1beta3 vs 2.0.7
> >> >> From: petermanev at gmail.com
> >> >> To: coolyasha at hotmail.com
> >> >> CC: modversion at gmail.com; oisf-users at lists.openinfosecfoundation.org
> >> >>
> >> >> On Fri, May 1, 2015 at 3:05 PM, Yasha Zislin <coolyasha at hotmail.com>
> >> >> wrote:
> >> >> > Correct.
> >> >> >
> >> >> > I've also tried a slight different version of the config to add
> >> >> > MODBUS
> >> >> > functionality and change toserver to dp for the ports in application
> >> >> > layer
> >> >> > detection section of the config file. I've basically compared config
> >> >> > that
> >> >> > came with the beta version to make sure things are correct and I am
> >> >> > no
> >> >> > using
> >> >> > depricated stuff. Either way, the same result.
> >> >> >
> >> >> > It feels like something changed with memory. beta version is only
> >> >> > using
> >> >> > about 40% of RAM but 2.0.7 is using 96%. It could be the reason for
> >> >> > the
> >> >> > packet loss on beta.
> >> >>
> >> >> So is your memcap sum total in your yaml equal to that 40% or to the
> >> >> 96% you are mentioning? (or that is irrelevant?)
> >> >>
> >> >> > Just thinking out loud.
> >> >> >
> >> >> > Thanks.
> >> >> >
> >> >> >> Date: Fri, 1 May 2015 12:10:40 +0200
> >> >> >> Subject: Re: [Oisf-users] Suricata 2.1beta3 vs 2.0.7
> >> >> >> From: petermanev at gmail.com
> >> >> >> To: coolyasha at hotmail.com
> >> >> >> CC: modversion at gmail.com; oisf-users at lists.openinfosecfoundation.org
> >> >> >
> >> >> >>
> >> >> >> On Thu, Apr 30, 2015 at 5:13 PM, Yasha Zislin
> >> >> >> <coolyasha at hotmail.com>
> >> >> >> wrote:
> >> >> >> > I am inspecting two span ports. Each has about 15 million packets
> >> >> >> > per
> >> >> >> > minute, mostly HTTP. Bandwidth is about 2 Gbps on each.
> >> >> >> >
> >> >> >> > I've noticed one new message on startup with beta version.
> >> >> >> > VLAN disabled, setting cluster type to CLUSTER_FLOW_5_TUPLE
> >> >> >> >
> >> >> >> > Not sure if this has any effect.
> >> >> >> >
> >> >> >> >
> >> >> >> > ________________________________
> >> >> >> > Date: Thu, 30 Apr 2015 23:10:09 +0800
> >> >> >> > Subject: Re: [Oisf-users] Suricata 2.1beta3 vs 2.0.7
> >> >> >> > From: modversion at gmail.com
> >> >> >> > To: coolyasha at hotmail.com
> >> >> >> > CC: oisf-users at lists.openinfosecfoundation.org
> >> >> >> >
> >> >> >> >
> >> >> >> > It seems that 2.0.7 work better than 2.1beta3.
> >> >> >> > What's the bandwidth you protect by suricata ? 10Gbps or 20Gbps ?
> >> >> >> >
> >> >> >> > 2015-04-30 23:00 GMT+08:00 Yasha Zislin <coolyasha at hotmail.com>:
> >> >> >> >
> >> >> >> > I have tweaked my configuration to have Suricata 2.0.7 run with
> >> >> >> > minimal
> >> >> >> > packet loss less than 0.01%. This set up does use a ton of RAM 95%
> >> >> >> > of
> >> >> >> > 140GB.
> >> >> >> > As soon as I switch to Suricata 2.1beta3 and run it with the same
> >> >> >> > config, I
> >> >> >> > get 50% packet loss but RAM utilization stays around 50%.
> >> >> >> >
> >> >> >> > What was changed to have such a big impact?
> >> >> >>
> >> >> >> Just to confirm - you are running the same Suricata config the only
> >> >> >> thing you have changed is suricata from 2.0.7 to 2.1beta3, correct?
> >> >> >> (nothing else)
> >> >> >>
> >> >> >> >
> >> >> >> > P.S. I am using PF_RING.
> >> >> >> >
> >> >> >> > Thanks.
> >> >> >> >
> >> >> >> > _______________________________________________
> >> >> >> > Suricata IDS Users mailing list:
> >> >> >> > oisf-users at openinfosecfoundation.org
> >> >> >> > Site: http://suricata-ids.org | Support:
> >> >> >> > http://suricata-ids.org/support/
> >> >> >> > List:
> >> >> >> >
> >> >> >> > https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> >> >> >> > Suricata User Conference November 4 & 5 in Barcelona:
> >> >> >> > http://oisfevents.net
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > _______________________________________________
> >> >> >> > Suricata IDS Users mailing list:
> >> >> >> > oisf-users at openinfosecfoundation.org
> >> >> >> > Site: http://suricata-ids.org | Support:
> >> >> >> > http://suricata-ids.org/support/
> >> >> >> > List:
> >> >> >> >
> >> >> >> > https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> >> >> >> > Suricata User Conference November 4 & 5 in Barcelona:
> >> >> >> > http://oisfevents.net
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >> Regards,
> >> >> >> Peter Manev
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Regards,
> >> >> Peter Manev
> >>
> >>
> >>
> >> --
> >> Regards,
> >> Peter Manev
>
>
>
> --
> Regards,
> Peter Manev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20150506/a1aa91ea/attachment-0002.html>
More information about the Oisf-users
mailing list