[Oisf-users] multicore http requests logging
Michał D
michu162 at gmail.com
Tue Sep 27 12:32:26 UTC 2016
Hello,
In 3.1.2 everything looks the same. Still only two cores with 100%
utilization for both af-packet and pcap mode.
Regards,
Michal
2016-09-27 12:38 GMT+02:00 Victor Julien <lists at inliniac.net>:
> On 27-09-16 12:30, Michał D wrote:
> > Logs from starting and stopping suricata in af-packet mode
> >
> > # /usr/bin/suricata -c /etc/suricata/suricata.yaml --disable-detection
> > --pidfile /var/run/suricata.pid --af-packet -D -vvv -F
> > /var/log/suricata/bpf_filter.txt
> > 27/9/2016 -- 12:17:18 - <Info> - detection engine disabled
> > 27/9/2016 -- 12:17:18 - <Notice> - This is Suricata version 3.1 RELEASE
>
> Before trying anything else, upgrade to 3.1.2. We've fixed many issues
> since 3.1.
>
> Cheers,
> Victor
>
>
> > 27/9/2016 -- 12:17:18 - <Info> - CPUs/cores online: 16
> > 27/9/2016 -- 12:17:18 - <Config> - Adding interface p2p1 from config file
> > 27/9/2016 -- 12:17:18 - <Config> - Adding interface p2p2 from config file
> > 27/9/2016 -- 12:17:18 - <Config> - 'default' server has
> > 'request-body-minimal-inspect-size' set to 33882 and
> > 'request-body-inspect-window' set to 4053 after randomization.
> > 27/9/2016 -- 12:17:18 - <Config> - 'default' server has
> > 'response-body-minimal-inspect-size' set to 42119 and
> > 'response-body-inspect-window' set to 16872 after randomization.
> > 27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser disabled
> > for tls protocol
> > 27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser disabled
> > for smb protocol.
> > 27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser disabled
> > for dcerpc protocol.
> > 27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser disabled
> > for dcerpc protocol.
> > 27/9/2016 -- 12:17:18 - <Info> - Parsed disabled for ftp protocol.
> > Protocol detectionstill on.
> > 27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser disabled
> > for smtp protocol.
> > 27/9/2016 -- 12:17:18 - <Config> - DNS request flood protection level:
> 500
> > 27/9/2016 -- 12:17:18 - <Config> - DNS per flow memcap (state-memcap):
> > 524288
> > 27/9/2016 -- 12:17:18 - <Config> - DNS global memcap: 16777216
> > 27/9/2016 -- 12:17:18 - <Config> - Protocol detection and parser
> > disabled for modbus protocol.
> > 27/9/2016 -- 12:17:18 - <Info> - Found an MTU of 1500 for 'p2p1'
> > 27/9/2016 -- 12:17:18 - <Info> - Found an MTU of 1500 for 'p2p2'
> > 27/9/2016 -- 12:17:18 - <Config> - allocated 3670016 bytes of memory for
> > the defrag hash... 65536 buckets of size 56
> > 27/9/2016 -- 12:17:18 - <Config> - preallocated 65535 defrag trackers of
> > size 168
> > 27/9/2016 -- 12:17:18 - <Config> - defrag memory usage: 14679896 bytes,
> > maximum: 536870912
> > 27/9/2016 -- 12:17:18 - <Info> - eve-log output device (regular)
> > initialized: http.json
> > 27/9/2016 -- 12:17:18 - <Info> - eve-log output device (regular)
> > initialized: dns.json
> > 27/9/2016 -- 12:17:18 - <Info> - stats output device (regular)
> > initialized: stats.log
> > 27/9/2016 -- 12:17:18 - <Info> - NIC offloading on p2p1: GRO: unset,
> > LRO: unset
> > 27/9/2016 -- 12:17:18 - <Info> - Going to use 3 thread(s)
> > 27/9/2016 -- 12:17:18 - <Info> - NIC offloading on p2p2: GRO: unset,
> > LRO: unset
> > 27/9/2016 -- 12:17:18 - <Info> - Going to use 3 thread(s)
> > 27/9/2016 -- 12:17:19 - <Notice> - all 6 packet processing threads, 4
> > management threads initialized, engine started.
> > 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p1'
> > 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p1'
> > 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p1'
> > 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p2'
> > 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p2'
> > 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p2'
> > 27/9/2016 -- 12:17:19 - <Info> - All AFP capture threads are running.
> >
> > 27/9/2016 -- 12:25:24 - <Notice> - Signal Received. Stopping engine.
> > 27/9/2016 -- 12:25:24 - <Info> - time elapsed 486.360s
> > 27/9/2016 -- 12:25:27 - <Notice> - Stats for 'p2p1': pkts: 20029605,
> > drop: 0 (0.00%), invalid chksum: 0
> > 27/9/2016 -- 12:25:27 - <Notice> - Stats for 'p2p2': pkts: 20202957,
> > drop: 0 (0.00%), invalid chksum: 0
> >
> > Drop:0, but in files not so many logs.
> >
> > 2016-09-27 12:01 GMT+02:00 Peter Manev <petermanev at gmail.com
> > <mailto:petermanev at gmail.com>>:
> >
> > On Tue, Sep 27, 2016 at 11:47 AM, Peter Manev <petermanev at gmail.com
> > <mailto:petermanev at gmail.com>> wrote:
> > > On Tue, Sep 27, 2016 at 11:13 AM, Michał D <michu162 at gmail.com
> <mailto:michu162 at gmail.com>> wrote:
> > >> In af-packet mode (/usr/bin/suricata -c
> /etc/suricata/suricata.yaml
> > >> --disable-detection --pidfile /var/run/suricata.pid --af-packet
> -D -vvv -F
> > >> /var/log/suricata/bpf_filter.txt ) suricata still utilise only
> two cores.
> > >
> > > In the pastebin info provided (your previous mails) - it seems you
> have -
> > > Detection enabled: yes
> > >
> > > You need to compile it first (./configure --disable-detection &&
> make
> > > clean && make && make install) - as opposed to pass it to the run
> > > line.
> > >
> >
> > Correction - it should work just as you have it as well -
> >
> > /opt/suricataqa/nodetection/bin/suricata -c
> > /etc/suricata/suricata.yaml --af-packet=eth0 -vvv --set
> > "af-packet.0.threads=2" --disable-detection
> > [19553] 27/9/2016 -- 11:58:39 - (suricata.c:1529) <Info>
> > (ParseCommandLine) -- detection engine disabled
> > [19553] 27/9/2016 -- 11:58:39 - (suricata.c:1005) <Notice>
> > (SCPrintVersion) -- This is Suricata version 3.2dev (rev 398489e)
> > ....
> >
> > Can you share your suricata.log?
> >
> > Thank you
> >
> >
> > >
> > >
> > >> Additionally in log file I can see much less entries per second.
> > >>
> > >> 2016-09-27 10:51 GMT+02:00 Peter Manev <petermanev at gmail.com
> > <mailto:petermanev at gmail.com>>:
> > >>>
> > >>> On Tue, Sep 27, 2016 at 10:18 AM, Michał D <michu162 at gmail.com
> > <mailto:michu162 at gmail.com>> wrote:
> > >>> > Currently I use "--disable-detection" when I'm running
> > suricata and I
> > >>> > sill
> > >>> > have problems with high CPU usage of only two cores and packet
> > drops in
> > >>> > peaks.
> > >>>
> > >>> Try af-packet and see if any diff.
> > >>>
> > >>> >
> > >>> > 2016-09-27 9:56 GMT+02:00 Peter Manev <petermanev at gmail.com
> > <mailto:petermanev at gmail.com>>:
> > >>> >>
> > >>> >> On Tue, Sep 27, 2016 at 8:43 AM, Michał D <michu162 at gmail.com
> > <mailto:michu162 at gmail.com>> wrote:
> > >>> >> > Hello,
> > >>> >> >
> > >>> >> > I would like to use suricata only to log incoming http
> > requests and
> > >>> >> > save
> > >>> >> > them as json into file (http.json).
> > >>> >>
> > >>> >> If this is the only thing you need to do - log http request
> > only - no
> > >>> >> inspection, no alerts.
> > >>> >> You can try the nsm mode (./configure --disable-detection
> > .....) and
> > >>> >> enable only http logs in the eve-log section of suricata.yaml.
> > >>> >>
> > >>> >> > I have server with two 10G interfaces where I'm receiving
> > mirrored
> > >>> >> > traffic,
> > >>> >> > 48GB of RAM and Intel(R) Xeon(R) CPU E5540 2.53GHz with 16
> > cores
> > >>> >> > Configuration of suricata and build-info you can find here:
> > >>> >> > http://pastebin.com/CriMdqJP
> > >>> >> >
> > >>> >> > Currently it works in PCAP mode, but I can see 100% usage
> > only of 2
> > >>> >> > CPU
> > >>> >> > cores and a lot of drops.
> > >>> >> > (/usr/bin/suricata -c /etc/suricata/suricata.yaml
> > --disable-detection
> > >>> >> > --pidfile /var/run/suricata.pid --pcap=p2p1 --pcap=p2p2 -D
> > -vvv -F
> > >>> >> > /etc/suricata/bpf_filter.txt)
> > >>> >> >
> > >>> >> > How should I configure & run suricata to have no drops and
> > use all
> > >>> >> > cores?
> > >>> >> >
> > >>> >> > Regards
> > >>> >> > Michal
> > >>> >> >
> > >>> >> > _______________________________________________
> > >>> >> > Suricata IDS Users mailing list:
> > oisf-users at openinfosecfoundation.org
> > <mailto:oisf-users at openinfosecfoundation.org>
> > >>> >> > Site: http://suricata-ids.org | Support:
> > >>> >> > http://suricata-ids.org/support/
> > <http://suricata-ids.org/support/>
> > >>> >> > List:
> > >>> >> >
> > https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> > <https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> >
> > >>> >> > Suricata User Conference November 9-11 in Washington, DC:
> > >>> >> > http://suricon.net
> > >>> >>
> > >>> >>
> > >>> >>
> > >>> >> --
> > >>> >> Regards,
> > >>> >> Peter Manev
> > >>> >
> > >>> >
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Regards,
> > >>> Peter Manev
> > >>
> > >>
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Peter Manev
> >
> >
> >
> > --
> > Regards,
> > Peter Manev
> >
> >
> >
> >
> > _______________________________________________
> > Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> > Site: http://suricata-ids.org | Support: http://suricata-ids.org/
> support/
> > List: https://lists.openinfosecfoundation.org/
> mailman/listinfo/oisf-users
> > Suricata User Conference November 9-11 in Washington, DC:
> http://suricon.net
> >
>
>
> --
> ---------------------------------------------
> Victor Julien
> http://www.inliniac.net/
> PGP: http://www.inliniac.net/victorjulien.asc
> ---------------------------------------------
>
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> Suricata User Conference November 9-11 in Washington, DC:
> http://suricon.net
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20160927/cdd50959/attachment-0002.html>
More information about the Oisf-users
mailing list