[Oisf-users] Suricata using 35% cpu with no load?
Peter Manev
petermanev at gmail.com
Wed Aug 12 18:34:50 UTC 2015
On Wed, Aug 12, 2015 at 8:22 PM, Duane Howard <duane.security at gmail.com> wrote:
> 4x3.2Ghz Xeon cores
>
That is a VM, correct?
> On Wed, Aug 12, 2015 at 8:32 AM, Peter Manev <petermanev at gmail.com> wrote:
>>
>> On Tue, Aug 11, 2015 at 8:39 PM, Duane Howard <duane.security at gmail.com>
>> wrote:
>> > Drops to ~7-10% with that change. Sorry for the delayed response.
>>
>>
>> I would suggest (for your test set up) - start with the defaults, then
>> increase as needed (watch out for flow.emerg_mode_entered counters in
>> the stats.log).
>> How many CPUs do you have on the test VM ? (and what speed/type?)
>>
>> Thanks
>>
>> >
>> > On Sat, Aug 1, 2015 at 10:24 AM, Peter Manev <petermanev at gmail.com>
>> > wrote:
>> >>
>> >> On Fri, Jul 31, 2015 at 7:39 PM, Duane Howard
>> >> <duane.security at gmail.com>
>> >> wrote:
>> >> > Just build 2.1beta4 and it's the same thing.
>> >> >
>> >> > On Fri, Jul 31, 2015 at 10:22 AM, Duane Howard
>> >> > <duane.security at gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> yaml sent off list.
>> >>
>> >> From the privately shared yaml -
>> >> flow:
>> >> memcap: 1.5gb
>> >> hash-size: 15728640
>> >>
>> >> divide flow.memcap by 3 and flow.hash-size by 10 - reload and please
>> >> let us know if any difference.
>> >>
>> >> Thanks
>> >>
>> >> >>This is the only machine I've tried on right now, and
>> >> >> I can't use the other one that is similarly configured because it's
>> >> >> being
>> >> >> used for other stuff at the moment. I did pretty extensive
>> >> >> experimentation
>> >> >> with 2.0.6 a while back on hardware and didn't see this issue. I'll
>> >> >> try
>> >> >> installing 2.1 and playing with it later today to see if I can
>> >> >> reproduce
>> >> >> that way.
>> >> >>
>> >> >> On Fri, Jul 31, 2015 at 6:47 AM, Peter Manev <petermanev at gmail.com>
>> >> >> wrote:
>> >> >>>
>> >> >>> On Thu, Jul 30, 2015 at 11:27 PM, Duane Howard
>> >> >>> <duane.security at gmail.com>
>> >> >>> wrote:
>> >> >>> > strace only shows the process doing constant:
>> >> >>> > nanosleep({0, 10000000}, NULL) = 0
>> >> >>> > nanosleep({0, 10000000}, NULL) = 0
>> >> >>> > nanosleep({0, 10000000}, NULL) = 0
>> >> >>> > nanosleep({0, 10000000}, NULL) = 0
>> >> >>> > nanosleep({0, 10000000}, NULL) = 0
>> >> >>> > nanosleep({0, 10000000}, NULL) = 0
>> >> >>> > ....
>> >> >>> >
>> >> >>>
>> >> >>> Hmm strange. I have not experianced that before.
>> >> >>> Does this happen on any VM or just this one in particular?
>> >> >>> Can you switch to 2.1beta4 and see if any difference. Please feel
>> >> >>> free
>> >> >>> to send me your suricata.yaml if you would like - I can see if this
>> >> >>> is
>> >> >>> enough to reproduce the issue.
>> >> >>>
>> >> >>> thanks
>> >> >>>
>> >> >>>
>> >> >>> >
>> >> >>> > On Thu, Jul 30, 2015 at 2:17 PM, Duane Howard
>> >> >>> > <duane.security at gmail.com>
>> >> >>> > wrote:
>> >> >>> >>
>> >> >>> >> I can reproduce. I ran it on a single low volume real NIC with
>> >> >>> >> the
>> >> >>> >> same
>> >> >>> >> result. Nothing terribly interesting in suricata.log with -vv
>> >> >>> >> enabled:
>> >> >>> >> Here's the logs after rule and config loading:
>> >> >>> >>
>> >> >>> >> 30/7/2015 -- 21:14:06 - <Info> - Core dump size set to
>> >> >>> >> unlimited.
>> >> >>> >> 30/7/2015 -- 21:14:06 - <Info> - Unified2-alert initialized:
>> >> >>> >> filename
>> >> >>> >> suricata.u2, limit 128 MB
>> >> >>> >> 30/7/2015 -- 21:14:06 - <Info> - Syslog output initialized
>> >> >>> >> 30/7/2015 -- 21:14:06 - <Info> - Unable to find af-packet config
>> >> >>> >> for
>> >> >>> >> interface "eth0" or "default", using default value
>> >> >>> >> 30/7/2015 -- 21:14:06 - <Info> - Going to use 1 thread(s)
>> >> >>> >> 30/7/2015 -- 21:14:06 - <Info> - Enabling zero copy mode
>> >> >>> >> 30/7/2015 -- 21:14:06 - <Info> - RunModeIdsAFPWorkers
>> >> >>> >> initialised
>> >> >>> >> 30/7/2015 -- 21:14:06 - <Notice> - all 1 packet processing
>> >> >>> >> threads,
>> >> >>> >> 3
>> >> >>> >> management threads initialized, engine started.
>> >> >>> >> 30/7/2015 -- 21:14:06 - <Info> - Using interface 'eth0' via
>> >> >>> >> socket
>> >> >>> >> 7
>> >> >>> >> 30/7/2015 -- 21:14:06 - <Info> - All AFP capture threads are
>> >> >>> >> running.
>> >> >>> >> 30/7/2015 -- 21:14:06 - <Info> - Thread AFPacketeth01 using
>> >> >>> >> socket
>> >> >>> >> 7
>> >> >>> >> 30/7/2015 -- 21:14:06 - <Info> - Starting to read on
>> >> >>> >> AFPacketeth01
>> >> >>> >> ^C30/7/2015 -- 21:15:27 - <Notice> - Signal Received. Stopping
>> >> >>> >> engine.
>> >> >>> >> 30/7/2015 -- 21:15:27 - <Info> - 0 new flows, 0 established
>> >> >>> >> flows
>> >> >>> >> were
>> >> >>> >> timed out, 0 flows in closed state
>> >> >>> >> 30/7/2015 -- 21:15:28 - <Info> - time elapsed 81.925s
>> >> >>> >> 30/7/2015 -- 21:15:28 - <Info> - (AFPacketeth01) Kernel: Packets
>> >> >>> >> 3372,
>> >> >>> >> dropped 0
>> >> >>> >> 30/7/2015 -- 21:15:28 - <Info> - (AFPacketeth01) Packets 3345,
>> >> >>> >> bytes
>> >> >>> >> 1318862
>> >> >>> >> 30/7/2015 -- 21:15:28 - <Info> - Stream TCP processed 2723 TCP
>> >> >>> >> packets
>> >> >>> >> 30/7/2015 -- 21:15:28 - <Info> - (AFPacketeth01) Alerts 0
>> >> >>> >> 30/7/2015 -- 21:15:28 - <Info> - Alert unified2 module wrote 0
>> >> >>> >> alerts
>> >> >>> >> 30/7/2015 -- 21:15:28 - <Info> - host memory usage: 390144
>> >> >>> >> bytes,
>> >> >>> >> maximum:
>> >> >>> >> 16777216
>> >> >>> >> 30/7/2015 -- 21:15:28 - <Info> - cleaning up signature grouping
>> >> >>> >> structure... complete
>> >> >>> >> 30/7/2015 -- 21:15:28 - <Notice> - Stats for 'eth0': pkts:
>> >> >>> >> 3372,
>> >> >>> >> drop: 0
>> >> >>> >> (0.00%), invalid chksum: 0
>> >> >>> >>
>> >> >>> >>
>> >> >>> >> On Thu, Jul 30, 2015 at 2:02 PM, Peter Manev
>> >> >>> >> <petermanev at gmail.com>
>> >> >>> >> wrote:
>> >> >>> >>>
>> >> >>> >>>
>> >> >>> >>> On 30 jul 2015, at 21:50, Duane Howard
>> >> >>> >>> <duane.security at gmail.com>
>> >> >>> >>> wrote:
>> >> >>> >>>
>> >> >>> >>>
>> >> >>> >>> Can you reproduce this problem at will? Anything in verbose
>> >> >>> >>> mode
>> >> >>> >>> in
>> >> >>> >>> Suricata.log?
>> >> >>> >>>
>> >> >>> >>> Can you narrow it down to a relation of a single interface
>> >> >>> >>> configuration
>> >> >>> >>> (as you mention in your initial mail)?
>> >> >>> >>>
>> >> >>> >>> Thanks
>> >> >>> >>>
>> >> >>> >>>
>> >> >>> >>> This is Suricata version 2.0.8 RELEASE
>> >> >>> >>> Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 PF_RING
>> >> >>> >>> AF_PACKET
>> >> >>> >>> HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1
>> >> >>> >>> HAVE_HTP_URI_NORMALIZE_HOOK
>> >> >>> >>> SIMD support: none
>> >> >>> >>> Atomic intrisics: 1 2 4 8 byte(s)
>> >> >>> >>> 64-bits, Little-endian architecture
>> >> >>> >>> GCC version 4.6.3, C version 199901
>> >> >>> >>> compiled with -fstack-protector
>> >> >>> >>> compiled with _FORTIFY_SOURCE=2
>> >> >>> >>> L1 cache line size (CLS)=64
>> >> >>> >>> compiled with LibHTP v0.5.17, linked against LibHTP v0.5.17
>> >> >>> >>> Suricata Configuration:
>> >> >>> >>> AF_PACKET support: yes
>> >> >>> >>> PF_RING support: yes
>> >> >>> >>> NFQueue support: no
>> >> >>> >>> NFLOG support: no
>> >> >>> >>> IPFW support: no
>> >> >>> >>> DAG enabled: no
>> >> >>> >>> Napatech enabled: no
>> >> >>> >>> Unix socket enabled: no
>> >> >>> >>> Detection enabled: yes
>> >> >>> >>>
>> >> >>> >>> libnss support: no
>> >> >>> >>> libnspr support: no
>> >> >>> >>> libjansson support: no
>> >> >>> >>> Prelude support: no
>> >> >>> >>> PCRE jit: no
>> >> >>> >>> LUA support: no
>> >> >>> >>> libluajit: no
>> >> >>> >>> libgeoip: yes
>> >> >>> >>> Non-bundled htp: yes
>> >> >>> >>> Old barnyard2 support: no
>> >> >>> >>> CUDA enabled: no
>> >> >>> >>>
>> >> >>> >>> Suricatasc install: yes
>> >> >>> >>>
>> >> >>> >>> Unit tests enabled: no
>> >> >>> >>> Debug output enabled: no
>> >> >>> >>> Debug validation enabled: no
>> >> >>> >>> Profiling enabled: no
>> >> >>> >>> Profiling locks enabled: no
>> >> >>> >>> Coccinelle / spatch: no
>> >> >>> >>>
>> >> >>> >>> Generic build parameters:
>> >> >>> >>> Installation prefix (--prefix): /usr
>> >> >>> >>> Configuration directory (--sysconfdir): /etc/suricata/
>> >> >>> >>> Log directory (--localstatedir) : /var/log/suricata/
>> >> >>> >>>
>> >> >>> >>> Host: x86_64-pc-linux-gnu
>> >> >>> >>> GCC binary: gcc
>> >> >>> >>> GCC Protect enabled: no
>> >> >>> >>> GCC march native enabled: no
>> >> >>> >>> GCC Profile enabled: no
>> >> >>> >>>
>> >> >>> >>>
>> >> >>> >>> On Thu, Jul 30, 2015 at 1:41 PM, Peter Manev
>> >> >>> >>> <petermanev at gmail.com>
>> >> >>> >>> wrote:
>> >> >>> >>>>
>> >> >>> >>>>
>> >> >>> >>>>
>> >> >>> >>>> On 30 jul 2015, at 20:58, Duane Howard
>> >> >>> >>>> <duane.security at gmail.com>
>> >> >>> >>>> wrote:
>> >> >>> >>>>
>> >> >>> >>>> When closing suri it reports 0 packets for the interface I'm
>> >> >>> >>>> mainly
>> >> >>> >>>> concerned with. There's also nothing in fast or other logs to
>> >> >>> >>>> indicate
>> >> >>> >>>> traffic.
>> >> >>> >>>>
>> >> >>> >>>> 30/7/2015 -- 19:58:41 - <Notice> - Stats for 'bond0': pkts:
>> >> >>> >>>> 0,
>> >> >>> >>>> drop: 0
>> >> >>> >>>> (-nan%), invalid chksum: 0
>> >> >>> >>>>
>> >> >>> >>>>
>> >> >>> >>>> What is the output of suricata --build-info ?
>> >> >>> >>>>
>> >> >>> >>>> If you have perf tools installed have a look at what in
>> >> >>> >>>> Suricata
>> >> >>> >>>> uses
>> >> >>> >>>> the most CPU -
>> >> >>> >>>> perf top -p pidofsuri
>> >> >>> >>>>
>> >> >>> >>>> Thanks
>> >> >>> >>>>
>> >> >>> >>>>
>> >> >>> >>>> On Thu, Jul 30, 2015 at 12:57 PM, Alan Wanderley dos Santos
>> >> >>> >>>> <alan.santos at rnp.br> wrote:
>> >> >>> >>>>>
>> >> >>> >>>>> Didi you see the fast.log and others logs?
>> >> >>> >>>>>
>> >> >>> >>>>> Maybe there are some traffic (icmp or broadcast for example)
>> >> >>> >>>>> coming
>> >> >>> >>>>> to
>> >> >>> >>>>> virtual machine, even little being data, can generated a lot
>> >> >>> >>>>> of
>> >> >>> >>>>> logs and
>> >> >>> >>>>> degree the performance.
>> >> >>> >>>>>
>> >> >>> >>>>> I had a similar situation on a VM of testing.
>> >> >>> >>>>>
>> >> >>> >>>>> Just a shot into darkness rsrs
>> >> >>> >>>>>
>> >> >>> >>>>> Regards,
>> >> >>> >>>>>
>> >> >>> >>>>> -----------------------------------------------
>> >> >>> >>>>> Alan Santos
>> >> >>> >>>>> Analista de Segurança
>> >> >>> >>>>> Centro de Atendimento a Incidentes de Segurança (CAIS)
>> >> >>> >>>>> Rede Nacional de Ensino e Pesquisa (RNP)
>> >> >>> >>>>> (19) 3787-3314 | alan.santos at rnp.br
>> >> >>> >>>>>
>> >> >>> >>>>> ________________________________
>> >> >>> >>>>> De: "Duane Howard" <duane.security at gmail.com>
>> >> >>> >>>>> Para: "oisf-users" <oisf-users at openinfosecfoundation.org>
>> >> >>> >>>>> Enviadas: Quinta-feira, 30 de julho de 2015 16:50:51
>> >> >>> >>>>> Assunto: [Oisf-users] Suricata using 35% cpu with no load?
>> >> >>> >>>>>
>> >> >>> >>>>> I've got a random virtual testing machine, and I'm seeing
>> >> >>> >>>>> Suricata
>> >> >>> >>>>> sitting at about 35% CPU load, even though there's absolutely
>> >> >>> >>>>> no
>> >> >>> >>>>> traffic
>> >> >>> >>>>> heading to it at the moment. Is there an easy way to get
>> >> >>> >>>>> Suricata
>> >> >>> >>>>> to tell me
>> >> >>> >>>>> what it's doing that would cause this? It occurs on real
>> >> >>> >>>>> interfaces
>> >> >>> >>>>> with low
>> >> >>> >>>>> traffic, loopback, as well as bonds where there's no trafic.
>> >> >>> >>>>> ./d
>> >> >>> >>>>>
>> >> >>> >>>>> _______________________________________________
>> >> >>> >>>>> Suricata IDS Users mailing list:
>> >> >>> >>>>> oisf-users at openinfosecfoundation.org
>> >> >>> >>>>> Site: http://suricata-ids.org | Support:
>> >> >>> >>>>> http://suricata-ids.org/support/
>> >> >>> >>>>> List:
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>> >> >>> >>>>> Suricata User Conference November 4 & 5 in Barcelona:
>> >> >>> >>>>> http://oisfevents.net
>> >> >>> >>>>
>> >> >>> >>>>
>> >> >>> >>>> _______________________________________________
>> >> >>> >>>> Suricata IDS Users mailing list:
>> >> >>> >>>> oisf-users at openinfosecfoundation.org
>> >> >>> >>>> Site: http://suricata-ids.org | Support:
>> >> >>> >>>> http://suricata-ids.org/support/
>> >> >>> >>>> List:
>> >> >>> >>>>
>> >> >>> >>>>
>> >> >>> >>>> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>> >> >>> >>>> Suricata User Conference November 4 & 5 in Barcelona:
>> >> >>> >>>> http://oisfevents.net
>> >> >>> >>>
>> >> >>> >>>
>> >> >>> >>
>> >> >>> >
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> --
>> >> >>> Regards,
>> >> >>> Peter Manev
>> >> >>
>> >> >>
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Regards,
>> >> Peter Manev
>> >
>> >
>>
>>
>>
>> --
>> Regards,
>> Peter Manev
>
>
--
Regards,
Peter Manev
More information about the Oisf-users
mailing list