<div dir="ltr">hello,<div><br><div>We use default CPU affinity</div><div><br></div><div><div>threading:</div><div> set-cpu-affinity: no</div></div><div><br></div><div>Whole configuration available via pastebin (<a href="http://pastebin.com/CriMdqJP">http://pastebin.com/CriMdqJP</a>)</div><div><br></div><div>Regards,</div><div>Michal </div></div></div><div class="gmail_extra"><br><div class="gmail_quote">2016-09-28 8:58 GMT+02:00 Peter Manev <span dir="ltr"><<a href="mailto:petermanev@gmail.com" target="_blank">petermanev@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Tue, Sep 27, 2016 at 2:32 PM, Michał D <<a href="mailto:michu162@gmail.com">michu162@gmail.com</a>> wrote:<br>
> Hello,<br>
><br>
> In 3.1.2 everything looks the same. Still only two cores with 100%<br>
> utilization for both af-packet and pcap mode.<br>
<br>
</span>Do you use any cpu affinity settings or they are the defaults?<br>
<div><div class="h5"><br>
><br>
> Regards,<br>
> Michal<br>
><br>
> 2016-09-27 12:38 GMT+02:00 Victor Julien <<a href="mailto:lists@inliniac.net">lists@inliniac.net</a>>:<br>
>><br>
>> On 27-09-16 12:30, Michał D wrote:<br>
>> > Logs from starting and stopping suricata in af-packet mode<br>
>> ><br>
>> > # /usr/bin/suricata -c /etc/suricata/suricata.yaml --disable-detection<br>
>> > --pidfile /var/run/suricata.pid --af-packet -D -vvv -F<br>
>> > /var/log/suricata/bpf_filter.<wbr>txt<br>
>> > 27/9/2016 -- 12:17:18 - <Info> - detection engine disabled<br>
>> > 27/9/2016 -- 12:17:18 - <Notice> - This is Suricata version 3.1 RELEASE<br>
>><br>
>> Before trying anything else, upgrade to 3.1.2. We've fixed many issues<br>
>> since 3.1.<br>
>><br>
>> Cheers,<br>
>> Victor<br>
>><br>
>><br>
>> > 27/9/2016 -- 12:17:18 - <Info> - CPUs/cores online: 16<br>
>> > 27/9/2016 -- 12:17:18 - <Config> - Adding interface p2p1 from config<br>
>> > file<br>
>> > 27/9/2016 -- 12:17:18 - <Config> - Adding interface p2p2 from config<br>
>> > file<br>
>> > 27/9/2016 -- 12:17:18 - <Config> - 'default' server has<br>
>> > 'request-body-minimal-inspect-<wbr>size' set to 33882 and<br>
>> > 'request-body-inspect-window' set to 4053 after randomization.<br>
>> > 27/9/2016 -- 12:17:18 - <Config> - 'default' server has<br>
>> > 'response-body-minimal-<wbr>inspect-size' set to 42119 and<br>
>> > 'response-body-inspect-window' set to 16872 after randomization.<br>
>> > 27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser disabled<br>
>> > for tls protocol<br>
>> > 27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser disabled<br>
>> > for smb protocol.<br>
>> > 27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser disabled<br>
>> > for dcerpc protocol.<br>
>> > 27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser disabled<br>
>> > for dcerpc protocol.<br>
>> > 27/9/2016 -- 12:17:18 - <Info> - Parsed disabled for ftp protocol.<br>
>> > Protocol detectionstill on.<br>
>> > 27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser disabled<br>
>> > for smtp protocol.<br>
>> > 27/9/2016 -- 12:17:18 - <Config> - DNS request flood protection level:<br>
>> > 500<br>
>> > 27/9/2016 -- 12:17:18 - <Config> - DNS per flow memcap (state-memcap):<br>
>> > 524288<br>
>> > 27/9/2016 -- 12:17:18 - <Config> - DNS global memcap: 16777216<br>
>> > 27/9/2016 -- 12:17:18 - <Config> - Protocol detection and parser<br>
>> > disabled for modbus protocol.<br>
>> > 27/9/2016 -- 12:17:18 - <Info> - Found an MTU of 1500 for 'p2p1'<br>
>> > 27/9/2016 -- 12:17:18 - <Info> - Found an MTU of 1500 for 'p2p2'<br>
>> > 27/9/2016 -- 12:17:18 - <Config> - allocated 3670016 bytes of memory for<br>
>> > the defrag hash... 65536 buckets of size 56<br>
>> > 27/9/2016 -- 12:17:18 - <Config> - preallocated 65535 defrag trackers of<br>
>> > size 168<br>
>> > 27/9/2016 -- 12:17:18 - <Config> - defrag memory usage: 14679896 bytes,<br>
>> > maximum: 536870912<br>
>> > 27/9/2016 -- 12:17:18 - <Info> - eve-log output device (regular)<br>
>> > initialized: http.json<br>
>> > 27/9/2016 -- 12:17:18 - <Info> - eve-log output device (regular)<br>
>> > initialized: dns.json<br>
>> > 27/9/2016 -- 12:17:18 - <Info> - stats output device (regular)<br>
>> > initialized: stats.log<br>
>> > 27/9/2016 -- 12:17:18 - <Info> - NIC offloading on p2p1: GRO: unset,<br>
>> > LRO: unset<br>
>> > 27/9/2016 -- 12:17:18 - <Info> - Going to use 3 thread(s)<br>
>> > 27/9/2016 -- 12:17:18 - <Info> - NIC offloading on p2p2: GRO: unset,<br>
>> > LRO: unset<br>
>> > 27/9/2016 -- 12:17:18 - <Info> - Going to use 3 thread(s)<br>
>> > 27/9/2016 -- 12:17:19 - <Notice> - all 6 packet processing threads, 4<br>
>> > management threads initialized, engine started.<br>
>> > 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p1'<br>
>> > 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p1'<br>
>> > 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p1'<br>
>> > 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p2'<br>
>> > 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p2'<br>
>> > 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p2'<br>
>> > 27/9/2016 -- 12:17:19 - <Info> - All AFP capture threads are running.<br>
>> ><br>
>> > 27/9/2016 -- 12:25:24 - <Notice> - Signal Received. Stopping engine.<br>
>> > 27/9/2016 -- 12:25:24 - <Info> - time elapsed 486.360s<br>
>> > 27/9/2016 -- 12:25:27 - <Notice> - Stats for 'p2p1': pkts: 20029605,<br>
>> > drop: 0 (0.00%), invalid chksum: 0<br>
>> > 27/9/2016 -- 12:25:27 - <Notice> - Stats for 'p2p2': pkts: 20202957,<br>
>> > drop: 0 (0.00%), invalid chksum: 0<br>
>> ><br>
>> > Drop:0, but in files not so many logs.<br>
>> ><br>
>> > 2016-09-27 12:01 GMT+02:00 Peter Manev <<a href="mailto:petermanev@gmail.com">petermanev@gmail.com</a><br>
>> > <mailto:<a href="mailto:petermanev@gmail.com">petermanev@gmail.com</a>>><wbr>:<br>
>> ><br>
>> > On Tue, Sep 27, 2016 at 11:47 AM, Peter Manev <<a href="mailto:petermanev@gmail.com">petermanev@gmail.com</a><br>
>> > <mailto:<a href="mailto:petermanev@gmail.com">petermanev@gmail.com</a>>> wrote:<br>
>> > > On Tue, Sep 27, 2016 at 11:13 AM, Michał D <<a href="mailto:michu162@gmail.com">michu162@gmail.com</a><br>
>> > <mailto:<a href="mailto:michu162@gmail.com">michu162@gmail.com</a>>> wrote:<br>
>> > >> In af-packet mode (/usr/bin/suricata -c<br>
>> > /etc/suricata/suricata.yaml<br>
>> > >> --disable-detection --pidfile /var/run/suricata.pid --af-packet<br>
>> > -D -vvv -F<br>
>> > >> /var/log/suricata/bpf_filter.<wbr>txt ) suricata still utilise only<br>
>> > two cores.<br>
>> > ><br>
>> > > In the pastebin info provided (your previous mails) - it seems you<br>
>> > have -<br>
>> > > Detection enabled: yes<br>
>> > ><br>
>> > > You need to compile it first (./configure --disable-detection &&<br>
>> > make<br>
>> > > clean && make && make install) - as opposed to pass it to the<br>
>> > run<br>
>> > > line.<br>
>> > ><br>
>> ><br>
>> > Correction - it should work just as you have it as well -<br>
>> ><br>
>> > /opt/suricataqa/nodetection/<wbr>bin/suricata -c<br>
>> > /etc/suricata/suricata.yaml --af-packet=eth0 -vvv --set<br>
>> > "af-packet.0.threads=2" --disable-detection<br>
>> > [19553] 27/9/2016 -- 11:58:39 - (suricata.c:1529) <Info><br>
>> > (ParseCommandLine) -- detection engine disabled<br>
>> > [19553] 27/9/2016 -- 11:58:39 - (suricata.c:1005) <Notice><br>
>> > (SCPrintVersion) -- This is Suricata version 3.2dev (rev 398489e)<br>
>> > ....<br>
>> ><br>
>> > Can you share your suricata.log?<br>
>> ><br>
>> > Thank you<br>
>> ><br>
>> ><br>
>> > ><br>
>> > ><br>
>> > >> Additionally in log file I can see much less entries per second.<br>
>> > >><br>
>> > >> 2016-09-27 10:51 GMT+02:00 Peter Manev <<a href="mailto:petermanev@gmail.com">petermanev@gmail.com</a><br>
>> > <mailto:<a href="mailto:petermanev@gmail.com">petermanev@gmail.com</a>>><wbr>:<br>
>> > >>><br>
>> > >>> On Tue, Sep 27, 2016 at 10:18 AM, Michał D <<a href="mailto:michu162@gmail.com">michu162@gmail.com</a><br>
>> > <mailto:<a href="mailto:michu162@gmail.com">michu162@gmail.com</a>>> wrote:<br>
>> > >>> > Currently I use "--disable-detection" when I'm running<br>
>> > suricata and I<br>
>> > >>> > sill<br>
>> > >>> > have problems with high CPU usage of only two cores and packet<br>
>> > drops in<br>
>> > >>> > peaks.<br>
>> > >>><br>
>> > >>> Try af-packet and see if any diff.<br>
>> > >>><br>
>> > >>> ><br>
>> > >>> > 2016-09-27 9:56 GMT+02:00 Peter Manev <<a href="mailto:petermanev@gmail.com">petermanev@gmail.com</a><br>
>> > <mailto:<a href="mailto:petermanev@gmail.com">petermanev@gmail.com</a>>><wbr>:<br>
>> > >>> >><br>
>> > >>> >> On Tue, Sep 27, 2016 at 8:43 AM, Michał D <<a href="mailto:michu162@gmail.com">michu162@gmail.com</a><br>
>> > <mailto:<a href="mailto:michu162@gmail.com">michu162@gmail.com</a>>> wrote:<br>
>> > >>> >> > Hello,<br>
>> > >>> >> ><br>
>> > >>> >> > I would like to use suricata only to log incoming http<br>
>> > requests and<br>
>> > >>> >> > save<br>
>> > >>> >> > them as json into file (http.json).<br>
>> > >>> >><br>
>> > >>> >> If this is the only thing you need to do - log http request<br>
>> > only - no<br>
>> > >>> >> inspection, no alerts.<br>
>> > >>> >> You can try the nsm mode (./configure --disable-detection<br>
>> > .....) and<br>
>> > >>> >> enable only http logs in the eve-log section of<br>
>> > suricata.yaml.<br>
>> > >>> >><br>
>> > >>> >> > I have server with two 10G interfaces where I'm receiving<br>
>> > mirrored<br>
>> > >>> >> > traffic,<br>
>> > >>> >> > 48GB of RAM and Intel(R) Xeon(R) CPU E5540 2.53GHz with 16<br>
>> > cores<br>
>> > >>> >> > Configuration of suricata and build-info you can find here:<br>
>> > >>> >> > <a href="http://pastebin.com/CriMdqJP" rel="noreferrer" target="_blank">http://pastebin.com/CriMdqJP</a><br>
>> > >>> >> ><br>
>> > >>> >> > Currently it works in PCAP mode, but I can see 100% usage<br>
>> > only of 2<br>
>> > >>> >> > CPU<br>
>> > >>> >> > cores and a lot of drops.<br>
>> > >>> >> > (/usr/bin/suricata -c /etc/suricata/suricata.yaml<br>
>> > --disable-detection<br>
>> > >>> >> > --pidfile /var/run/suricata.pid --pcap=p2p1 --pcap=p2p2 -D<br>
>> > -vvv -F<br>
>> > >>> >> > /etc/suricata/bpf_filter.txt)<br>
>> > >>> >> ><br>
>> > >>> >> > How should I configure & run suricata to have no drops and<br>
>> > use all<br>
>> > >>> >> > cores?<br>
>> > >>> >> ><br>
>> > >>> >> > Regards<br>
>> > >>> >> > Michal<br>
>> > >>> >> ><br>
>> > >>> >> > ______________________________<wbr>_________________<br>
>> > >>> >> > Suricata IDS Users mailing list:<br>
>> > <a href="mailto:oisf-users@openinfosecfoundation.org">oisf-users@<wbr>openinfosecfoundation.org</a><br>
>> > <mailto:<a href="mailto:oisf-users@openinfosecfoundation.org">oisf-users@<wbr>openinfosecfoundation.org</a>><br>
>> > >>> >> > Site: <a href="http://suricata-ids.org" rel="noreferrer" target="_blank">http://suricata-ids.org</a> | Support:<br>
>> > >>> >> > <a href="http://suricata-ids.org/support/" rel="noreferrer" target="_blank">http://suricata-ids.org/<wbr>support/</a><br>
>> > <<a href="http://suricata-ids.org/support/" rel="noreferrer" target="_blank">http://suricata-ids.org/<wbr>support/</a>><br>
>> > >>> >> > List:<br>
>> > >>> >> ><br>
>> > <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" rel="noreferrer" target="_blank">https://lists.<wbr>openinfosecfoundation.org/<wbr>mailman/listinfo/oisf-users</a><br>
>> ><br>
>> > <<a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" rel="noreferrer" target="_blank">https://lists.<wbr>openinfosecfoundation.org/<wbr>mailman/listinfo/oisf-users</a>><br>
>> > >>> >> > Suricata User Conference November 9-11 in Washington, DC:<br>
>> > >>> >> > <a href="http://suricon.net" rel="noreferrer" target="_blank">http://suricon.net</a><br>
>> > >>> >><br>
>> > >>> >><br>
>> > >>> >><br>
>> > >>> >> --<br>
>> > >>> >> Regards,<br>
>> > >>> >> Peter Manev<br>
>> > >>> ><br>
>> > >>> ><br>
>> > >>><br>
>> > >>><br>
>> > >>><br>
>> > >>> --<br>
>> > >>> Regards,<br>
>> > >>> Peter Manev<br>
>> > >><br>
>> > >><br>
>> > ><br>
>> > ><br>
>> > ><br>
>> > > --<br>
>> > > Regards,<br>
>> > > Peter Manev<br>
>> ><br>
>> ><br>
>> ><br>
>> > --<br>
>> > Regards,<br>
>> > Peter Manev<br>
>> ><br>
>> ><br>
>> ><br>
>> ><br>
>> > ______________________________<wbr>_________________<br>
>> > Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org">oisf-users@<wbr>openinfosecfoundation.org</a><br>
>> > Site: <a href="http://suricata-ids.org" rel="noreferrer" target="_blank">http://suricata-ids.org</a> | Support:<br>
>> > <a href="http://suricata-ids.org/support/" rel="noreferrer" target="_blank">http://suricata-ids.org/<wbr>support/</a><br>
>> > List:<br>
>> > <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" rel="noreferrer" target="_blank">https://lists.<wbr>openinfosecfoundation.org/<wbr>mailman/listinfo/oisf-users</a><br>
>> > Suricata User Conference November 9-11 in Washington, DC:<br>
>> > <a href="http://suricon.net" rel="noreferrer" target="_blank">http://suricon.net</a><br>
>> ><br>
>><br>
>><br>
>> --<br>
>> ------------------------------<wbr>---------------<br>
>> Victor Julien<br>
>> <a href="http://www.inliniac.net/" rel="noreferrer" target="_blank">http://www.inliniac.net/</a><br>
>> PGP: <a href="http://www.inliniac.net/victorjulien.asc" rel="noreferrer" target="_blank">http://www.inliniac.net/<wbr>victorjulien.asc</a><br>
>> ------------------------------<wbr>---------------<br>
>><br>
>> ______________________________<wbr>_________________<br>
>> Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org">oisf-users@<wbr>openinfosecfoundation.org</a><br>
>> Site: <a href="http://suricata-ids.org" rel="noreferrer" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" rel="noreferrer" target="_blank">http://suricata-ids.org/<wbr>support/</a><br>
>> List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" rel="noreferrer" target="_blank">https://lists.<wbr>openinfosecfoundation.org/<wbr>mailman/listinfo/oisf-users</a><br>
>> Suricata User Conference November 9-11 in Washington, DC:<br>
>> <a href="http://suricon.net" rel="noreferrer" target="_blank">http://suricon.net</a><br>
><br>
><br>
><br>
> ______________________________<wbr>_________________<br>
> Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org">oisf-users@<wbr>openinfosecfoundation.org</a><br>
> Site: <a href="http://suricata-ids.org" rel="noreferrer" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" rel="noreferrer" target="_blank">http://suricata-ids.org/<wbr>support/</a><br>
> List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" rel="noreferrer" target="_blank">https://lists.<wbr>openinfosecfoundation.org/<wbr>mailman/listinfo/oisf-users</a><br>
> Suricata User Conference November 9-11 in Washington, DC: <a href="http://suricon.net" rel="noreferrer" target="_blank">http://suricon.net</a><br>
<br>
<br>
<br>
--<br>
</div></div>Regards,<br>
Peter Manev<br>
</blockquote></div><br></div>