[Oisf-users] multicore http requests logging
Michał D
michu162 at gmail.com
Wed Sep 28 12:59:42 UTC 2016
hello,
We use default CPU affinity
threading:
set-cpu-affinity: no
Whole configuration available via pastebin (http://pastebin.com/CriMdqJP)
Regards,
Michal
2016-09-28 8:58 GMT+02:00 Peter Manev <petermanev at gmail.com>:
> On Tue, Sep 27, 2016 at 2:32 PM, Michał D <michu162 at gmail.com> wrote:
> > Hello,
> >
> > In 3.1.2 everything looks the same. Still only two cores with 100%
> > utilization for both af-packet and pcap mode.
>
> Do you use any cpu affinity settings or they are the defaults?
>
> >
> > Regards,
> > Michal
> >
> > 2016-09-27 12:38 GMT+02:00 Victor Julien <lists at inliniac.net>:
> >>
> >> On 27-09-16 12:30, Michał D wrote:
> >> > Logs from starting and stopping suricata in af-packet mode
> >> >
> >> > # /usr/bin/suricata -c /etc/suricata/suricata.yaml --disable-detection
> >> > --pidfile /var/run/suricata.pid --af-packet -D -vvv -F
> >> > /var/log/suricata/bpf_filter.txt
> >> > 27/9/2016 -- 12:17:18 - <Info> - detection engine disabled
> >> > 27/9/2016 -- 12:17:18 - <Notice> - This is Suricata version 3.1
> RELEASE
> >>
> >> Before trying anything else, upgrade to 3.1.2. We've fixed many issues
> >> since 3.1.
> >>
> >> Cheers,
> >> Victor
> >>
> >>
> >> > 27/9/2016 -- 12:17:18 - <Info> - CPUs/cores online: 16
> >> > 27/9/2016 -- 12:17:18 - <Config> - Adding interface p2p1 from config
> >> > file
> >> > 27/9/2016 -- 12:17:18 - <Config> - Adding interface p2p2 from config
> >> > file
> >> > 27/9/2016 -- 12:17:18 - <Config> - 'default' server has
> >> > 'request-body-minimal-inspect-size' set to 33882 and
> >> > 'request-body-inspect-window' set to 4053 after randomization.
> >> > 27/9/2016 -- 12:17:18 - <Config> - 'default' server has
> >> > 'response-body-minimal-inspect-size' set to 42119 and
> >> > 'response-body-inspect-window' set to 16872 after randomization.
> >> > 27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser
> disabled
> >> > for tls protocol
> >> > 27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser
> disabled
> >> > for smb protocol.
> >> > 27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser
> disabled
> >> > for dcerpc protocol.
> >> > 27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser
> disabled
> >> > for dcerpc protocol.
> >> > 27/9/2016 -- 12:17:18 - <Info> - Parsed disabled for ftp protocol.
> >> > Protocol detectionstill on.
> >> > 27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser
> disabled
> >> > for smtp protocol.
> >> > 27/9/2016 -- 12:17:18 - <Config> - DNS request flood protection level:
> >> > 500
> >> > 27/9/2016 -- 12:17:18 - <Config> - DNS per flow memcap (state-memcap):
> >> > 524288
> >> > 27/9/2016 -- 12:17:18 - <Config> - DNS global memcap: 16777216
> >> > 27/9/2016 -- 12:17:18 - <Config> - Protocol detection and parser
> >> > disabled for modbus protocol.
> >> > 27/9/2016 -- 12:17:18 - <Info> - Found an MTU of 1500 for 'p2p1'
> >> > 27/9/2016 -- 12:17:18 - <Info> - Found an MTU of 1500 for 'p2p2'
> >> > 27/9/2016 -- 12:17:18 - <Config> - allocated 3670016 bytes of memory
> for
> >> > the defrag hash... 65536 buckets of size 56
> >> > 27/9/2016 -- 12:17:18 - <Config> - preallocated 65535 defrag trackers
> of
> >> > size 168
> >> > 27/9/2016 -- 12:17:18 - <Config> - defrag memory usage: 14679896
> bytes,
> >> > maximum: 536870912
> >> > 27/9/2016 -- 12:17:18 - <Info> - eve-log output device (regular)
> >> > initialized: http.json
> >> > 27/9/2016 -- 12:17:18 - <Info> - eve-log output device (regular)
> >> > initialized: dns.json
> >> > 27/9/2016 -- 12:17:18 - <Info> - stats output device (regular)
> >> > initialized: stats.log
> >> > 27/9/2016 -- 12:17:18 - <Info> - NIC offloading on p2p1: GRO: unset,
> >> > LRO: unset
> >> > 27/9/2016 -- 12:17:18 - <Info> - Going to use 3 thread(s)
> >> > 27/9/2016 -- 12:17:18 - <Info> - NIC offloading on p2p2: GRO: unset,
> >> > LRO: unset
> >> > 27/9/2016 -- 12:17:18 - <Info> - Going to use 3 thread(s)
> >> > 27/9/2016 -- 12:17:19 - <Notice> - all 6 packet processing threads, 4
> >> > management threads initialized, engine started.
> >> > 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p1'
> >> > 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p1'
> >> > 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p1'
> >> > 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p2'
> >> > 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p2'
> >> > 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p2'
> >> > 27/9/2016 -- 12:17:19 - <Info> - All AFP capture threads are running.
> >> >
> >> > 27/9/2016 -- 12:25:24 - <Notice> - Signal Received. Stopping engine.
> >> > 27/9/2016 -- 12:25:24 - <Info> - time elapsed 486.360s
> >> > 27/9/2016 -- 12:25:27 - <Notice> - Stats for 'p2p1': pkts: 20029605,
> >> > drop: 0 (0.00%), invalid chksum: 0
> >> > 27/9/2016 -- 12:25:27 - <Notice> - Stats for 'p2p2': pkts: 20202957,
> >> > drop: 0 (0.00%), invalid chksum: 0
> >> >
> >> > Drop:0, but in files not so many logs.
> >> >
> >> > 2016-09-27 12:01 GMT+02:00 Peter Manev <petermanev at gmail.com
> >> > <mailto:petermanev at gmail.com>>:
> >> >
> >> > On Tue, Sep 27, 2016 at 11:47 AM, Peter Manev <
> petermanev at gmail.com
> >> > <mailto:petermanev at gmail.com>> wrote:
> >> > > On Tue, Sep 27, 2016 at 11:13 AM, Michał D <michu162 at gmail.com
> >> > <mailto:michu162 at gmail.com>> wrote:
> >> > >> In af-packet mode (/usr/bin/suricata -c
> >> > /etc/suricata/suricata.yaml
> >> > >> --disable-detection --pidfile /var/run/suricata.pid --af-packet
> >> > -D -vvv -F
> >> > >> /var/log/suricata/bpf_filter.txt ) suricata still utilise only
> >> > two cores.
> >> > >
> >> > > In the pastebin info provided (your previous mails) - it seems
> you
> >> > have -
> >> > > Detection enabled: yes
> >> > >
> >> > > You need to compile it first (./configure --disable-detection &&
> >> > make
> >> > > clean && make && make install) - as opposed to pass it to the
> >> > run
> >> > > line.
> >> > >
> >> >
> >> > Correction - it should work just as you have it as well -
> >> >
> >> > /opt/suricataqa/nodetection/bin/suricata -c
> >> > /etc/suricata/suricata.yaml --af-packet=eth0 -vvv --set
> >> > "af-packet.0.threads=2" --disable-detection
> >> > [19553] 27/9/2016 -- 11:58:39 - (suricata.c:1529) <Info>
> >> > (ParseCommandLine) -- detection engine disabled
> >> > [19553] 27/9/2016 -- 11:58:39 - (suricata.c:1005) <Notice>
> >> > (SCPrintVersion) -- This is Suricata version 3.2dev (rev 398489e)
> >> > ....
> >> >
> >> > Can you share your suricata.log?
> >> >
> >> > Thank you
> >> >
> >> >
> >> > >
> >> > >
> >> > >> Additionally in log file I can see much less entries per
> second.
> >> > >>
> >> > >> 2016-09-27 10:51 GMT+02:00 Peter Manev <petermanev at gmail.com
> >> > <mailto:petermanev at gmail.com>>:
> >> > >>>
> >> > >>> On Tue, Sep 27, 2016 at 10:18 AM, Michał D <
> michu162 at gmail.com
> >> > <mailto:michu162 at gmail.com>> wrote:
> >> > >>> > Currently I use "--disable-detection" when I'm running
> >> > suricata and I
> >> > >>> > sill
> >> > >>> > have problems with high CPU usage of only two cores and
> packet
> >> > drops in
> >> > >>> > peaks.
> >> > >>>
> >> > >>> Try af-packet and see if any diff.
> >> > >>>
> >> > >>> >
> >> > >>> > 2016-09-27 9:56 GMT+02:00 Peter Manev <petermanev at gmail.com
> >> > <mailto:petermanev at gmail.com>>:
> >> > >>> >>
> >> > >>> >> On Tue, Sep 27, 2016 at 8:43 AM, Michał D <
> michu162 at gmail.com
> >> > <mailto:michu162 at gmail.com>> wrote:
> >> > >>> >> > Hello,
> >> > >>> >> >
> >> > >>> >> > I would like to use suricata only to log incoming http
> >> > requests and
> >> > >>> >> > save
> >> > >>> >> > them as json into file (http.json).
> >> > >>> >>
> >> > >>> >> If this is the only thing you need to do - log http
> request
> >> > only - no
> >> > >>> >> inspection, no alerts.
> >> > >>> >> You can try the nsm mode (./configure --disable-detection
> >> > .....) and
> >> > >>> >> enable only http logs in the eve-log section of
> >> > suricata.yaml.
> >> > >>> >>
> >> > >>> >> > I have server with two 10G interfaces where I'm receiving
> >> > mirrored
> >> > >>> >> > traffic,
> >> > >>> >> > 48GB of RAM and Intel(R) Xeon(R) CPU E5540 2.53GHz with
> 16
> >> > cores
> >> > >>> >> > Configuration of suricata and build-info you can find
> here:
> >> > >>> >> > http://pastebin.com/CriMdqJP
> >> > >>> >> >
> >> > >>> >> > Currently it works in PCAP mode, but I can see 100% usage
> >> > only of 2
> >> > >>> >> > CPU
> >> > >>> >> > cores and a lot of drops.
> >> > >>> >> > (/usr/bin/suricata -c /etc/suricata/suricata.yaml
> >> > --disable-detection
> >> > >>> >> > --pidfile /var/run/suricata.pid --pcap=p2p1 --pcap=p2p2
> -D
> >> > -vvv -F
> >> > >>> >> > /etc/suricata/bpf_filter.txt)
> >> > >>> >> >
> >> > >>> >> > How should I configure & run suricata to have no drops
> and
> >> > use all
> >> > >>> >> > cores?
> >> > >>> >> >
> >> > >>> >> > Regards
> >> > >>> >> > Michal
> >> > >>> >> >
> >> > >>> >> > _______________________________________________
> >> > >>> >> > Suricata IDS Users mailing list:
> >> > oisf-users at openinfosecfoundation.org
> >> > <mailto:oisf-users at openinfosecfoundation.org>
> >> > >>> >> > Site: http://suricata-ids.org | Support:
> >> > >>> >> > http://suricata-ids.org/support/
> >> > <http://suricata-ids.org/support/>
> >> > >>> >> > List:
> >> > >>> >> >
> >> > https://lists.openinfosecfoundation.org/
> mailman/listinfo/oisf-users
> >> >
> >> > <https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users>
> >> > >>> >> > Suricata User Conference November 9-11 in Washington, DC:
> >> > >>> >> > http://suricon.net
> >> > >>> >>
> >> > >>> >>
> >> > >>> >>
> >> > >>> >> --
> >> > >>> >> Regards,
> >> > >>> >> Peter Manev
> >> > >>> >
> >> > >>> >
> >> > >>>
> >> > >>>
> >> > >>>
> >> > >>> --
> >> > >>> Regards,
> >> > >>> Peter Manev
> >> > >>
> >> > >>
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > Regards,
> >> > > Peter Manev
> >> >
> >> >
> >> >
> >> > --
> >> > Regards,
> >> > Peter Manev
> >> >
> >> >
> >> >
> >> >
> >> > _______________________________________________
> >> > Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> >> > Site: http://suricata-ids.org | Support:
> >> > http://suricata-ids.org/support/
> >> > List:
> >> > https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> >> > Suricata User Conference November 9-11 in Washington, DC:
> >> > http://suricon.net
> >> >
> >>
> >>
> >> --
> >> ---------------------------------------------
> >> Victor Julien
> >> http://www.inliniac.net/
> >> PGP: http://www.inliniac.net/victorjulien.asc
> >> ---------------------------------------------
> >>
> >> _______________________________________________
> >> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> >> Site: http://suricata-ids.org | Support: http://suricata-ids.org/
> support/
> >> List: https://lists.openinfosecfoundation.org/
> mailman/listinfo/oisf-users
> >> Suricata User Conference November 9-11 in Washington, DC:
> >> http://suricon.net
> >
> >
> >
> > _______________________________________________
> > Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> > Site: http://suricata-ids.org | Support: http://suricata-ids.org/
> support/
> > List: https://lists.openinfosecfoundation.org/
> mailman/listinfo/oisf-users
> > Suricata User Conference November 9-11 in Washington, DC:
> http://suricon.net
>
>
>
> --
> Regards,
> Peter Manev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20160928/04e912d9/attachment-0002.html>
More information about the Oisf-users
mailing list