<div dir="ltr">4x3.2Ghz Xeon cores</div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Aug 12, 2015 at 8:32 AM, Peter Manev <span dir="ltr"><<a href="mailto:petermanev@gmail.com" target="_blank">petermanev@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Tue, Aug 11, 2015 at 8:39 PM, Duane Howard <<a href="mailto:duane.security@gmail.com">duane.security@gmail.com</a>> wrote:<br>
> Drops to ~7-10% with that change. Sorry for the delayed response.<br>
<br>
<br>
</span>I would suggest (for your test set up) - start with the defaults, then<br>
increase as needed (watch out for flow.emerg_mode_entered counters in<br>
the stats.log).<br>
How many CPUs do you have on the test VM ? (and what speed/type?)<br>
<br>
Thanks<br>
<div class="HOEnZb"><div class="h5"><br>
><br>
> On Sat, Aug 1, 2015 at 10:24 AM, Peter Manev <<a href="mailto:petermanev@gmail.com">petermanev@gmail.com</a>> wrote:<br>
>><br>
>> On Fri, Jul 31, 2015 at 7:39 PM, Duane Howard <<a href="mailto:duane.security@gmail.com">duane.security@gmail.com</a>><br>
>> wrote:<br>
>> > Just build 2.1beta4 and it's the same thing.<br>
>> ><br>
>> > On Fri, Jul 31, 2015 at 10:22 AM, Duane Howard<br>
>> > <<a href="mailto:duane.security@gmail.com">duane.security@gmail.com</a>><br>
>> > wrote:<br>
>> >><br>
>> >> yaml sent off list.<br>
>><br>
>> From the privately shared yaml -<br>
>> flow:<br>
>> memcap: 1.5gb<br>
>> hash-size: 15728640<br>
>><br>
>> divide flow.memcap by 3 and flow.hash-size by 10 - reload and please<br>
>> let us know if any difference.<br>
>><br>
>> Thanks<br>
>><br>
>> >>This is the only machine I've tried on right now, and<br>
>> >> I can't use the other one that is similarly configured because it's<br>
>> >> being<br>
>> >> used for other stuff at the moment. I did pretty extensive<br>
>> >> experimentation<br>
>> >> with 2.0.6 a while back on hardware and didn't see this issue. I'll try<br>
>> >> installing 2.1 and playing with it later today to see if I can<br>
>> >> reproduce<br>
>> >> that way.<br>
>> >><br>
>> >> On Fri, Jul 31, 2015 at 6:47 AM, Peter Manev <<a href="mailto:petermanev@gmail.com">petermanev@gmail.com</a>><br>
>> >> wrote:<br>
>> >>><br>
>> >>> On Thu, Jul 30, 2015 at 11:27 PM, Duane Howard<br>
>> >>> <<a href="mailto:duane.security@gmail.com">duane.security@gmail.com</a>><br>
>> >>> wrote:<br>
>> >>> > strace only shows the process doing constant:<br>
>> >>> > nanosleep({0, 10000000}, NULL) = 0<br>
>> >>> > nanosleep({0, 10000000}, NULL) = 0<br>
>> >>> > nanosleep({0, 10000000}, NULL) = 0<br>
>> >>> > nanosleep({0, 10000000}, NULL) = 0<br>
>> >>> > nanosleep({0, 10000000}, NULL) = 0<br>
>> >>> > nanosleep({0, 10000000}, NULL) = 0<br>
>> >>> > ....<br>
>> >>> ><br>
>> >>><br>
>> >>> Hmm strange. I have not experianced that before.<br>
>> >>> Does this happen on any VM or just this one in particular?<br>
>> >>> Can you switch to 2.1beta4 and see if any difference. Please feel free<br>
>> >>> to send me your suricata.yaml if you would like - I can see if this is<br>
>> >>> enough to reproduce the issue.<br>
>> >>><br>
>> >>> thanks<br>
>> >>><br>
>> >>><br>
>> >>> ><br>
>> >>> > On Thu, Jul 30, 2015 at 2:17 PM, Duane Howard<br>
>> >>> > <<a href="mailto:duane.security@gmail.com">duane.security@gmail.com</a>><br>
>> >>> > wrote:<br>
>> >>> >><br>
>> >>> >> I can reproduce. I ran it on a single low volume real NIC with the<br>
>> >>> >> same<br>
>> >>> >> result. Nothing terribly interesting in suricata.log with -vv<br>
>> >>> >> enabled:<br>
>> >>> >> Here's the logs after rule and config loading:<br>
>> >>> >><br>
>> >>> >> 30/7/2015 -- 21:14:06 - <Info> - Core dump size set to unlimited.<br>
>> >>> >> 30/7/2015 -- 21:14:06 - <Info> - Unified2-alert initialized:<br>
>> >>> >> filename<br>
>> >>> >> suricata.u2, limit 128 MB<br>
>> >>> >> 30/7/2015 -- 21:14:06 - <Info> - Syslog output initialized<br>
>> >>> >> 30/7/2015 -- 21:14:06 - <Info> - Unable to find af-packet config<br>
>> >>> >> for<br>
>> >>> >> interface "eth0" or "default", using default value<br>
>> >>> >> 30/7/2015 -- 21:14:06 - <Info> - Going to use 1 thread(s)<br>
>> >>> >> 30/7/2015 -- 21:14:06 - <Info> - Enabling zero copy mode<br>
>> >>> >> 30/7/2015 -- 21:14:06 - <Info> - RunModeIdsAFPWorkers initialised<br>
>> >>> >> 30/7/2015 -- 21:14:06 - <Notice> - all 1 packet processing threads,<br>
>> >>> >> 3<br>
>> >>> >> management threads initialized, engine started.<br>
>> >>> >> 30/7/2015 -- 21:14:06 - <Info> - Using interface 'eth0' via socket<br>
>> >>> >> 7<br>
>> >>> >> 30/7/2015 -- 21:14:06 - <Info> - All AFP capture threads are<br>
>> >>> >> running.<br>
>> >>> >> 30/7/2015 -- 21:14:06 - <Info> - Thread AFPacketeth01 using socket<br>
>> >>> >> 7<br>
>> >>> >> 30/7/2015 -- 21:14:06 - <Info> - Starting to read on AFPacketeth01<br>
>> >>> >> ^C30/7/2015 -- 21:15:27 - <Notice> - Signal Received. Stopping<br>
>> >>> >> engine.<br>
>> >>> >> 30/7/2015 -- 21:15:27 - <Info> - 0 new flows, 0 established flows<br>
>> >>> >> were<br>
>> >>> >> timed out, 0 flows in closed state<br>
>> >>> >> 30/7/2015 -- 21:15:28 - <Info> - time elapsed 81.925s<br>
>> >>> >> 30/7/2015 -- 21:15:28 - <Info> - (AFPacketeth01) Kernel: Packets<br>
>> >>> >> 3372,<br>
>> >>> >> dropped 0<br>
>> >>> >> 30/7/2015 -- 21:15:28 - <Info> - (AFPacketeth01) Packets 3345,<br>
>> >>> >> bytes<br>
>> >>> >> 1318862<br>
>> >>> >> 30/7/2015 -- 21:15:28 - <Info> - Stream TCP processed 2723 TCP<br>
>> >>> >> packets<br>
>> >>> >> 30/7/2015 -- 21:15:28 - <Info> - (AFPacketeth01) Alerts 0<br>
>> >>> >> 30/7/2015 -- 21:15:28 - <Info> - Alert unified2 module wrote 0<br>
>> >>> >> alerts<br>
>> >>> >> 30/7/2015 -- 21:15:28 - <Info> - host memory usage: 390144 bytes,<br>
>> >>> >> maximum:<br>
>> >>> >> 16777216<br>
>> >>> >> 30/7/2015 -- 21:15:28 - <Info> - cleaning up signature grouping<br>
>> >>> >> structure... complete<br>
>> >>> >> 30/7/2015 -- 21:15:28 - <Notice> - Stats for 'eth0': pkts: 3372,<br>
>> >>> >> drop: 0<br>
>> >>> >> (0.00%), invalid chksum: 0<br>
>> >>> >><br>
>> >>> >><br>
>> >>> >> On Thu, Jul 30, 2015 at 2:02 PM, Peter Manev <<a href="mailto:petermanev@gmail.com">petermanev@gmail.com</a>><br>
>> >>> >> wrote:<br>
>> >>> >>><br>
>> >>> >>><br>
>> >>> >>> On 30 jul 2015, at 21:50, Duane Howard <<a href="mailto:duane.security@gmail.com">duane.security@gmail.com</a>><br>
>> >>> >>> wrote:<br>
>> >>> >>><br>
>> >>> >>><br>
>> >>> >>> Can you reproduce this problem at will? Anything in verbose mode<br>
>> >>> >>> in<br>
>> >>> >>> Suricata.log?<br>
>> >>> >>><br>
>> >>> >>> Can you narrow it down to a relation of a single interface<br>
>> >>> >>> configuration<br>
>> >>> >>> (as you mention in your initial mail)?<br>
>> >>> >>><br>
>> >>> >>> Thanks<br>
>> >>> >>><br>
>> >>> >>><br>
>> >>> >>> This is Suricata version 2.0.8 RELEASE<br>
>> >>> >>> Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 PF_RING AF_PACKET<br>
>> >>> >>> HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK<br>
>> >>> >>> SIMD support: none<br>
>> >>> >>> Atomic intrisics: 1 2 4 8 byte(s)<br>
>> >>> >>> 64-bits, Little-endian architecture<br>
>> >>> >>> GCC version 4.6.3, C version 199901<br>
>> >>> >>> compiled with -fstack-protector<br>
>> >>> >>> compiled with _FORTIFY_SOURCE=2<br>
>> >>> >>> L1 cache line size (CLS)=64<br>
>> >>> >>> compiled with LibHTP v0.5.17, linked against LibHTP v0.5.17<br>
>> >>> >>> Suricata Configuration:<br>
>> >>> >>> AF_PACKET support: yes<br>
>> >>> >>> PF_RING support: yes<br>
>> >>> >>> NFQueue support: no<br>
>> >>> >>> NFLOG support: no<br>
>> >>> >>> IPFW support: no<br>
>> >>> >>> DAG enabled: no<br>
>> >>> >>> Napatech enabled: no<br>
>> >>> >>> Unix socket enabled: no<br>
>> >>> >>> Detection enabled: yes<br>
>> >>> >>><br>
>> >>> >>> libnss support: no<br>
>> >>> >>> libnspr support: no<br>
>> >>> >>> libjansson support: no<br>
>> >>> >>> Prelude support: no<br>
>> >>> >>> PCRE jit: no<br>
>> >>> >>> LUA support: no<br>
>> >>> >>> libluajit: no<br>
>> >>> >>> libgeoip: yes<br>
>> >>> >>> Non-bundled htp: yes<br>
>> >>> >>> Old barnyard2 support: no<br>
>> >>> >>> CUDA enabled: no<br>
>> >>> >>><br>
>> >>> >>> Suricatasc install: yes<br>
>> >>> >>><br>
>> >>> >>> Unit tests enabled: no<br>
>> >>> >>> Debug output enabled: no<br>
>> >>> >>> Debug validation enabled: no<br>
>> >>> >>> Profiling enabled: no<br>
>> >>> >>> Profiling locks enabled: no<br>
>> >>> >>> Coccinelle / spatch: no<br>
>> >>> >>><br>
>> >>> >>> Generic build parameters:<br>
>> >>> >>> Installation prefix (--prefix): /usr<br>
>> >>> >>> Configuration directory (--sysconfdir): /etc/suricata/<br>
>> >>> >>> Log directory (--localstatedir) : /var/log/suricata/<br>
>> >>> >>><br>
>> >>> >>> Host: x86_64-pc-linux-gnu<br>
>> >>> >>> GCC binary: gcc<br>
>> >>> >>> GCC Protect enabled: no<br>
>> >>> >>> GCC march native enabled: no<br>
>> >>> >>> GCC Profile enabled: no<br>
>> >>> >>><br>
>> >>> >>><br>
>> >>> >>> On Thu, Jul 30, 2015 at 1:41 PM, Peter Manev<br>
>> >>> >>> <<a href="mailto:petermanev@gmail.com">petermanev@gmail.com</a>><br>
>> >>> >>> wrote:<br>
>> >>> >>>><br>
>> >>> >>>><br>
>> >>> >>>><br>
>> >>> >>>> On 30 jul 2015, at 20:58, Duane Howard <<a href="mailto:duane.security@gmail.com">duane.security@gmail.com</a>><br>
>> >>> >>>> wrote:<br>
>> >>> >>>><br>
>> >>> >>>> When closing suri it reports 0 packets for the interface I'm<br>
>> >>> >>>> mainly<br>
>> >>> >>>> concerned with. There's also nothing in fast or other logs to<br>
>> >>> >>>> indicate<br>
>> >>> >>>> traffic.<br>
>> >>> >>>><br>
>> >>> >>>> 30/7/2015 -- 19:58:41 - <Notice> - Stats for 'bond0': pkts: 0,<br>
>> >>> >>>> drop: 0<br>
>> >>> >>>> (-nan%), invalid chksum: 0<br>
>> >>> >>>><br>
>> >>> >>>><br>
>> >>> >>>> What is the output of suricata --build-info ?<br>
>> >>> >>>><br>
>> >>> >>>> If you have perf tools installed have a look at what in Suricata<br>
>> >>> >>>> uses<br>
>> >>> >>>> the most CPU -<br>
>> >>> >>>> perf top -p pidofsuri<br>
>> >>> >>>><br>
>> >>> >>>> Thanks<br>
>> >>> >>>><br>
>> >>> >>>><br>
>> >>> >>>> On Thu, Jul 30, 2015 at 12:57 PM, Alan Wanderley dos Santos<br>
>> >>> >>>> <<a href="mailto:alan.santos@rnp.br">alan.santos@rnp.br</a>> wrote:<br>
>> >>> >>>>><br>
>> >>> >>>>> Didi you see the fast.log and others logs?<br>
>> >>> >>>>><br>
>> >>> >>>>> Maybe there are some traffic (icmp or broadcast for example)<br>
>> >>> >>>>> coming<br>
>> >>> >>>>> to<br>
>> >>> >>>>> virtual machine, even little being data, can generated a lot of<br>
>> >>> >>>>> logs and<br>
>> >>> >>>>> degree the performance.<br>
>> >>> >>>>><br>
>> >>> >>>>> I had a similar situation on a VM of testing.<br>
>> >>> >>>>><br>
>> >>> >>>>> Just a shot into darkness rsrs<br>
>> >>> >>>>><br>
>> >>> >>>>> Regards,<br>
>> >>> >>>>><br>
>> >>> >>>>> -----------------------------------------------<br>
>> >>> >>>>> Alan Santos<br>
>> >>> >>>>> Analista de Segurança<br>
>> >>> >>>>> Centro de Atendimento a Incidentes de Segurança (CAIS)<br>
>> >>> >>>>> Rede Nacional de Ensino e Pesquisa (RNP)<br>
>> >>> >>>>> (19) 3787-3314 | <a href="mailto:alan.santos@rnp.br">alan.santos@rnp.br</a><br>
>> >>> >>>>><br>
>> >>> >>>>> ________________________________<br>
>> >>> >>>>> De: "Duane Howard" <<a href="mailto:duane.security@gmail.com">duane.security@gmail.com</a>><br>
>> >>> >>>>> Para: "oisf-users" <<a href="mailto:oisf-users@openinfosecfoundation.org">oisf-users@openinfosecfoundation.org</a>><br>
>> >>> >>>>> Enviadas: Quinta-feira, 30 de julho de 2015 16:50:51<br>
>> >>> >>>>> Assunto: [Oisf-users] Suricata using 35% cpu with no load?<br>
>> >>> >>>>><br>
>> >>> >>>>> I've got a random virtual testing machine, and I'm seeing<br>
>> >>> >>>>> Suricata<br>
>> >>> >>>>> sitting at about 35% CPU load, even though there's absolutely no<br>
>> >>> >>>>> traffic<br>
>> >>> >>>>> heading to it at the moment. Is there an easy way to get<br>
>> >>> >>>>> Suricata<br>
>> >>> >>>>> to tell me<br>
>> >>> >>>>> what it's doing that would cause this? It occurs on real<br>
>> >>> >>>>> interfaces<br>
>> >>> >>>>> with low<br>
>> >>> >>>>> traffic, loopback, as well as bonds where there's no trafic.<br>
>> >>> >>>>> ./d<br>
>> >>> >>>>><br>
>> >>> >>>>> _______________________________________________<br>
>> >>> >>>>> Suricata IDS Users mailing list:<br>
>> >>> >>>>> <a href="mailto:oisf-users@openinfosecfoundation.org">oisf-users@openinfosecfoundation.org</a><br>
>> >>> >>>>> Site: <a href="http://suricata-ids.org" rel="noreferrer" target="_blank">http://suricata-ids.org</a> | Support:<br>
>> >>> >>>>> <a href="http://suricata-ids.org/support/" rel="noreferrer" target="_blank">http://suricata-ids.org/support/</a><br>
>> >>> >>>>> List:<br>
>> >>> >>>>><br>
>> >>> >>>>> <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" rel="noreferrer" target="_blank">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
>> >>> >>>>> Suricata User Conference November 4 & 5 in Barcelona:<br>
>> >>> >>>>> <a href="http://oisfevents.net" rel="noreferrer" target="_blank">http://oisfevents.net</a><br>
>> >>> >>>><br>
>> >>> >>>><br>
>> >>> >>>> _______________________________________________<br>
>> >>> >>>> Suricata IDS Users mailing list:<br>
>> >>> >>>> <a href="mailto:oisf-users@openinfosecfoundation.org">oisf-users@openinfosecfoundation.org</a><br>
>> >>> >>>> Site: <a href="http://suricata-ids.org" rel="noreferrer" target="_blank">http://suricata-ids.org</a> | Support:<br>
>> >>> >>>> <a href="http://suricata-ids.org/support/" rel="noreferrer" target="_blank">http://suricata-ids.org/support/</a><br>
>> >>> >>>> List:<br>
>> >>> >>>><br>
>> >>> >>>> <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" rel="noreferrer" target="_blank">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
>> >>> >>>> Suricata User Conference November 4 & 5 in Barcelona:<br>
>> >>> >>>> <a href="http://oisfevents.net" rel="noreferrer" target="_blank">http://oisfevents.net</a><br>
>> >>> >>><br>
>> >>> >>><br>
>> >>> >><br>
>> >>> ><br>
>> >>><br>
>> >>><br>
>> >>><br>
>> >>> --<br>
>> >>> Regards,<br>
>> >>> Peter Manev<br>
>> >><br>
>> >><br>
>> ><br>
>><br>
>><br>
>><br>
>> --<br>
>> Regards,<br>
>> Peter Manev<br>
><br>
><br>
<br>
<br>
<br>
</div></div><span class="HOEnZb"><font color="#888888">--<br>
Regards,<br>
Peter Manev<br>
</font></span></blockquote></div><br></div>