<div dir="ltr">Hello,<div><br></div><div>In 3.1.2 everything looks the same. Still only two cores with 100% utilization for both af-packet and pcap mode. </div><div><br></div><div>Regards,</div><div>Michal</div></div><div class="gmail_extra"><br><div class="gmail_quote">2016-09-27 12:38 GMT+02:00 Victor Julien <span dir="ltr"><<a href="mailto:lists@inliniac.net" target="_blank">lists@inliniac.net</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 27-09-16 12:30, Michał D wrote:<br>
> Logs from starting and stopping suricata in af-packet mode<br>
><br>
> # /usr/bin/suricata -c /etc/suricata/suricata.yaml --disable-detection<br>
> --pidfile /var/run/suricata.pid --af-packet -D -vvv -F<br>
> /var/log/suricata/bpf_filter.<wbr>txt<br>
> 27/9/2016 -- 12:17:18 - <Info> - detection engine disabled<br>
> 27/9/2016 -- 12:17:18 - <Notice> - This is Suricata version 3.1 RELEASE<br>
<br>
</span>Before trying anything else, upgrade to 3.1.2. We've fixed many issues<br>
since 3.1.<br>
<br>
Cheers,<br>
Victor<br>
<div><div class="h5"><br>
<br>
> 27/9/2016 -- 12:17:18 - <Info> - CPUs/cores online: 16<br>
> 27/9/2016 -- 12:17:18 - <Config> - Adding interface p2p1 from config file<br>
> 27/9/2016 -- 12:17:18 - <Config> - Adding interface p2p2 from config file<br>
> 27/9/2016 -- 12:17:18 - <Config> - 'default' server has<br>
> 'request-body-minimal-inspect-<wbr>size' set to 33882 and<br>
> 'request-body-inspect-window' set to 4053 after randomization.<br>
> 27/9/2016 -- 12:17:18 - <Config> - 'default' server has<br>
> 'response-body-minimal-<wbr>inspect-size' set to 42119 and<br>
> 'response-body-inspect-window' set to 16872 after randomization.<br>
> 27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser disabled<br>
> for tls protocol<br>
> 27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser disabled<br>
> for smb protocol.<br>
> 27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser disabled<br>
> for dcerpc protocol.<br>
> 27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser disabled<br>
> for dcerpc protocol.<br>
> 27/9/2016 -- 12:17:18 - <Info> - Parsed disabled for ftp protocol.<br>
> Protocol detectionstill on.<br>
> 27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser disabled<br>
> for smtp protocol.<br>
> 27/9/2016 -- 12:17:18 - <Config> - DNS request flood protection level: 500<br>
> 27/9/2016 -- 12:17:18 - <Config> - DNS per flow memcap (state-memcap):<br>
> 524288<br>
> 27/9/2016 -- 12:17:18 - <Config> - DNS global memcap: 16777216<br>
> 27/9/2016 -- 12:17:18 - <Config> - Protocol detection and parser<br>
> disabled for modbus protocol.<br>
> 27/9/2016 -- 12:17:18 - <Info> - Found an MTU of 1500 for 'p2p1'<br>
> 27/9/2016 -- 12:17:18 - <Info> - Found an MTU of 1500 for 'p2p2'<br>
> 27/9/2016 -- 12:17:18 - <Config> - allocated 3670016 bytes of memory for<br>
> the defrag hash... 65536 buckets of size 56<br>
> 27/9/2016 -- 12:17:18 - <Config> - preallocated 65535 defrag trackers of<br>
> size 168<br>
> 27/9/2016 -- 12:17:18 - <Config> - defrag memory usage: 14679896 bytes,<br>
> maximum: 536870912<br>
> 27/9/2016 -- 12:17:18 - <Info> - eve-log output device (regular)<br>
> initialized: http.json<br>
> 27/9/2016 -- 12:17:18 - <Info> - eve-log output device (regular)<br>
> initialized: dns.json<br>
> 27/9/2016 -- 12:17:18 - <Info> - stats output device (regular)<br>
> initialized: stats.log<br>
> 27/9/2016 -- 12:17:18 - <Info> - NIC offloading on p2p1: GRO: unset,<br>
> LRO: unset<br>
> 27/9/2016 -- 12:17:18 - <Info> - Going to use 3 thread(s)<br>
> 27/9/2016 -- 12:17:18 - <Info> - NIC offloading on p2p2: GRO: unset,<br>
> LRO: unset<br>
> 27/9/2016 -- 12:17:18 - <Info> - Going to use 3 thread(s)<br>
> 27/9/2016 -- 12:17:19 - <Notice> - all 6 packet processing threads, 4<br>
> management threads initialized, engine started.<br>
> 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p1'<br>
> 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p1'<br>
> 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p1'<br>
> 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p2'<br>
> 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p2'<br>
> 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p2'<br>
> 27/9/2016 -- 12:17:19 - <Info> - All AFP capture threads are running.<br>
><br>
> 27/9/2016 -- 12:25:24 - <Notice> - Signal Received. Stopping engine.<br>
> 27/9/2016 -- 12:25:24 - <Info> - time elapsed 486.360s<br>
> 27/9/2016 -- 12:25:27 - <Notice> - Stats for 'p2p1': pkts: 20029605,<br>
> drop: 0 (0.00%), invalid chksum: 0<br>
> 27/9/2016 -- 12:25:27 - <Notice> - Stats for 'p2p2': pkts: 20202957,<br>
> drop: 0 (0.00%), invalid chksum: 0<br>
><br>
> Drop:0, but in files not so many logs.<br>
><br>
> 2016-09-27 12:01 GMT+02:00 Peter Manev <<a href="mailto:petermanev@gmail.com">petermanev@gmail.com</a><br>
</div></div>> <mailto:<a href="mailto:petermanev@gmail.com">petermanev@gmail.com</a>>><wbr>:<br>
<span class="">><br>
> On Tue, Sep 27, 2016 at 11:47 AM, Peter Manev <<a href="mailto:petermanev@gmail.com">petermanev@gmail.com</a><br>
</span>> <mailto:<a href="mailto:petermanev@gmail.com">petermanev@gmail.com</a>>> wrote:<br>
<span class="">> > On Tue, Sep 27, 2016 at 11:13 AM, Michał D <<a href="mailto:michu162@gmail.com">michu162@gmail.com</a> <mailto:<a href="mailto:michu162@gmail.com">michu162@gmail.com</a>>> wrote:<br>
> >> In af-packet mode (/usr/bin/suricata -c /etc/suricata/suricata.yaml<br>
> >> --disable-detection --pidfile /var/run/suricata.pid --af-packet -D -vvv -F<br>
> >> /var/log/suricata/bpf_filter.<wbr>txt ) suricata still utilise only two cores.<br>
> ><br>
> > In the pastebin info provided (your previous mails) - it seems you have -<br>
> > Detection enabled: yes<br>
> ><br>
> > You need to compile it first (./configure --disable-detection && make<br>
> > clean && make && make install) - as opposed to pass it to the run<br>
> > line.<br>
> ><br>
><br>
> Correction - it should work just as you have it as well -<br>
><br>
> /opt/suricataqa/nodetection/<wbr>bin/suricata -c<br>
> /etc/suricata/suricata.yaml --af-packet=eth0 -vvv --set<br>
> "af-packet.0.threads=2" --disable-detection<br>
> [19553] 27/9/2016 -- 11:58:39 - (suricata.c:1529) <Info><br>
> (ParseCommandLine) -- detection engine disabled<br>
> [19553] 27/9/2016 -- 11:58:39 - (suricata.c:1005) <Notice><br>
> (SCPrintVersion) -- This is Suricata version 3.2dev (rev 398489e)<br>
> ....<br>
><br>
> Can you share your suricata.log?<br>
><br>
> Thank you<br>
><br>
><br>
> ><br>
> ><br>
> >> Additionally in log file I can see much less entries per second.<br>
> >><br>
> >> 2016-09-27 10:51 GMT+02:00 Peter Manev <<a href="mailto:petermanev@gmail.com">petermanev@gmail.com</a><br>
</span>> <mailto:<a href="mailto:petermanev@gmail.com">petermanev@gmail.com</a>>><wbr>:<br>
<span class="">> >>><br>
> >>> On Tue, Sep 27, 2016 at 10:18 AM, Michał D <<a href="mailto:michu162@gmail.com">michu162@gmail.com</a><br>
</span><span class="">> <mailto:<a href="mailto:michu162@gmail.com">michu162@gmail.com</a>>> wrote:<br>
> >>> > Currently I use "--disable-detection" when I'm running<br>
> suricata and I<br>
> >>> > sill<br>
> >>> > have problems with high CPU usage of only two cores and packet<br>
> drops in<br>
> >>> > peaks.<br>
> >>><br>
> >>> Try af-packet and see if any diff.<br>
> >>><br>
> >>> ><br>
> >>> > 2016-09-27 9:56 GMT+02:00 Peter Manev <<a href="mailto:petermanev@gmail.com">petermanev@gmail.com</a><br>
</span>> <mailto:<a href="mailto:petermanev@gmail.com">petermanev@gmail.com</a>>><wbr>:<br>
<span class="">> >>> >><br>
> >>> >> On Tue, Sep 27, 2016 at 8:43 AM, Michał D <<a href="mailto:michu162@gmail.com">michu162@gmail.com</a><br>
</span><div><div class="h5">> <mailto:<a href="mailto:michu162@gmail.com">michu162@gmail.com</a>>> wrote:<br>
> >>> >> > Hello,<br>
> >>> >> ><br>
> >>> >> > I would like to use suricata only to log incoming http<br>
> requests and<br>
> >>> >> > save<br>
> >>> >> > them as json into file (http.json).<br>
> >>> >><br>
> >>> >> If this is the only thing you need to do - log http request<br>
> only - no<br>
> >>> >> inspection, no alerts.<br>
> >>> >> You can try the nsm mode (./configure --disable-detection<br>
> .....) and<br>
> >>> >> enable only http logs in the eve-log section of suricata.yaml.<br>
> >>> >><br>
> >>> >> > I have server with two 10G interfaces where I'm receiving<br>
> mirrored<br>
> >>> >> > traffic,<br>
> >>> >> > 48GB of RAM and Intel(R) Xeon(R) CPU E5540 2.53GHz with 16<br>
> cores<br>
> >>> >> > Configuration of suricata and build-info you can find here:<br>
> >>> >> > <a href="http://pastebin.com/CriMdqJP" rel="noreferrer" target="_blank">http://pastebin.com/CriMdqJP</a><br>
> >>> >> ><br>
> >>> >> > Currently it works in PCAP mode, but I can see 100% usage<br>
> only of 2<br>
> >>> >> > CPU<br>
> >>> >> > cores and a lot of drops.<br>
> >>> >> > (/usr/bin/suricata -c /etc/suricata/suricata.yaml<br>
> --disable-detection<br>
> >>> >> > --pidfile /var/run/suricata.pid --pcap=p2p1 --pcap=p2p2 -D<br>
> -vvv -F<br>
> >>> >> > /etc/suricata/bpf_filter.txt)<br>
> >>> >> ><br>
> >>> >> > How should I configure & run suricata to have no drops and<br>
> use all<br>
> >>> >> > cores?<br>
> >>> >> ><br>
> >>> >> > Regards<br>
> >>> >> > Michal<br>
> >>> >> ><br>
> >>> >> > ______________________________<wbr>_________________<br>
> >>> >> > Suricata IDS Users mailing list:<br>
> <a href="mailto:oisf-users@openinfosecfoundation.org">oisf-users@<wbr>openinfosecfoundation.org</a><br>
</div></div>> <mailto:<a href="mailto:oisf-users@openinfosecfoundation.org">oisf-users@<wbr>openinfosecfoundation.org</a>><br>
<div><div class="h5">> >>> >> > Site: <a href="http://suricata-ids.org" rel="noreferrer" target="_blank">http://suricata-ids.org</a> | Support:<br>
> >>> >> > <a href="http://suricata-ids.org/support/" rel="noreferrer" target="_blank">http://suricata-ids.org/<wbr>support/</a><br>
> <<a href="http://suricata-ids.org/support/" rel="noreferrer" target="_blank">http://suricata-ids.org/<wbr>support/</a>><br>
> >>> >> > List:<br>
> >>> >> ><br>
> <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" rel="noreferrer" target="_blank">https://lists.<wbr>openinfosecfoundation.org/<wbr>mailman/listinfo/oisf-users</a><br>
> <<a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" rel="noreferrer" target="_blank">https://lists.<wbr>openinfosecfoundation.org/<wbr>mailman/listinfo/oisf-users</a>><br>
> >>> >> > Suricata User Conference November 9-11 in Washington, DC:<br>
> >>> >> > <a href="http://suricon.net" rel="noreferrer" target="_blank">http://suricon.net</a><br>
> >>> >><br>
> >>> >><br>
> >>> >><br>
> >>> >> --<br>
> >>> >> Regards,<br>
> >>> >> Peter Manev<br>
> >>> ><br>
> >>> ><br>
> >>><br>
> >>><br>
> >>><br>
> >>> --<br>
> >>> Regards,<br>
> >>> Peter Manev<br>
> >><br>
> >><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > Regards,<br>
> > Peter Manev<br>
><br>
><br>
><br>
> --<br>
> Regards,<br>
> Peter Manev<br>
><br>
><br>
><br>
><br>
> ______________________________<wbr>_________________<br>
> Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org">oisf-users@<wbr>openinfosecfoundation.org</a><br>
> Site: <a href="http://suricata-ids.org" rel="noreferrer" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" rel="noreferrer" target="_blank">http://suricata-ids.org/<wbr>support/</a><br>
> List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" rel="noreferrer" target="_blank">https://lists.<wbr>openinfosecfoundation.org/<wbr>mailman/listinfo/oisf-users</a><br>
> Suricata User Conference November 9-11 in Washington, DC: <a href="http://suricon.net" rel="noreferrer" target="_blank">http://suricon.net</a><br>
><br>
<br>
<br>
--<br>
</div></div>------------------------------<wbr>---------------<br>
Victor Julien<br>
<a href="http://www.inliniac.net/" rel="noreferrer" target="_blank">http://www.inliniac.net/</a><br>
PGP: <a href="http://www.inliniac.net/victorjulien.asc" rel="noreferrer" target="_blank">http://www.inliniac.net/<wbr>victorjulien.asc</a><br>
------------------------------<wbr>---------------<br>
<div class="HOEnZb"><div class="h5"><br>
______________________________<wbr>_________________<br>
Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org">oisf-users@<wbr>openinfosecfoundation.org</a><br>
Site: <a href="http://suricata-ids.org" rel="noreferrer" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" rel="noreferrer" target="_blank">http://suricata-ids.org/<wbr>support/</a><br>
List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" rel="noreferrer" target="_blank">https://lists.<wbr>openinfosecfoundation.org/<wbr>mailman/listinfo/oisf-users</a><br>
Suricata User Conference November 9-11 in Washington, DC: <a href="http://suricon.net" rel="noreferrer" target="_blank">http://suricon.net</a></div></div></blockquote></div><br></div>