[Oisf-users] multicore http requests logging

Tue Sep 27 10:30:36 UTC 2016

Logs from starting and stopping suricata in af-packet mode

# /usr/bin/suricata -c /etc/suricata/suricata.yaml --disable-detection
--pidfile /var/run/suricata.pid --af-packet -D -vvv -F
/var/log/suricata/bpf_filter.txt
27/9/2016 -- 12:17:18 - <Info> - detection engine disabled
27/9/2016 -- 12:17:18 - <Notice> - This is Suricata version 3.1 RELEASE
27/9/2016 -- 12:17:18 - <Info> - CPUs/cores online: 16
27/9/2016 -- 12:17:18 - <Config> - Adding interface p2p1 from config file
27/9/2016 -- 12:17:18 - <Config> - Adding interface p2p2 from config file
27/9/2016 -- 12:17:18 - <Config> - 'default' server has
'request-body-minimal-inspect-size' set to 33882 and
'request-body-inspect-window' set to 4053 after randomization.
27/9/2016 -- 12:17:18 - <Config> - 'default' server has
'response-body-minimal-inspect-size' set to 42119 and
'response-body-inspect-window' set to 16872 after randomization.
27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser disabled for
tls protocol
27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser disabled for
smb protocol.
27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser disabled for
dcerpc protocol.
27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser disabled for
dcerpc protocol.
27/9/2016 -- 12:17:18 - <Info> - Parsed disabled for ftp protocol. Protocol
detectionstill on.
27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser disabled for
smtp protocol.
27/9/2016 -- 12:17:18 - <Config> - DNS request flood protection level: 500
27/9/2016 -- 12:17:18 - <Config> - DNS per flow memcap (state-memcap):
524288
27/9/2016 -- 12:17:18 - <Config> - DNS global memcap: 16777216
27/9/2016 -- 12:17:18 - <Config> - Protocol detection and parser disabled
for modbus protocol.
27/9/2016 -- 12:17:18 - <Info> - Found an MTU of 1500 for 'p2p1'
27/9/2016 -- 12:17:18 - <Info> - Found an MTU of 1500 for 'p2p2'
27/9/2016 -- 12:17:18 - <Config> - allocated 3670016 bytes of memory for
the defrag hash... 65536 buckets of size 56
27/9/2016 -- 12:17:18 - <Config> - preallocated 65535 defrag trackers of
size 168
27/9/2016 -- 12:17:18 - <Config> - defrag memory usage: 14679896 bytes,
maximum: 536870912
27/9/2016 -- 12:17:18 - <Info> - eve-log output device (regular)
initialized: http.json
27/9/2016 -- 12:17:18 - <Info> - eve-log output device (regular)
initialized: dns.json
27/9/2016 -- 12:17:18 - <Info> - stats output device (regular) initialized:
stats.log
27/9/2016 -- 12:17:18 - <Info> - NIC offloading on p2p1: GRO: unset, LRO:
unset
27/9/2016 -- 12:17:18 - <Info> - Going to use 3 thread(s)
27/9/2016 -- 12:17:18 - <Info> - NIC offloading on p2p2: GRO: unset, LRO:
unset
27/9/2016 -- 12:17:18 - <Info> - Going to use 3 thread(s)
27/9/2016 -- 12:17:19 - <Notice> - all 6 packet processing threads, 4
management threads initialized, engine started.
27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p1'
27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p1'
27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p1'
27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p2'
27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p2'
27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p2'
27/9/2016 -- 12:17:19 - <Info> - All AFP capture threads are running.

27/9/2016 -- 12:25:24 - <Notice> - Signal Received.  Stopping engine.
27/9/2016 -- 12:25:24 - <Info> - time elapsed 486.360s
27/9/2016 -- 12:25:27 - <Notice> - Stats for 'p2p1':  pkts: 20029605, drop:
0 (0.00%), invalid chksum: 0
27/9/2016 -- 12:25:27 - <Notice> - Stats for 'p2p2':  pkts: 20202957, drop:
0 (0.00%), invalid chksum: 0

Drop:0, but in files not so many logs.

2016-09-27 12:01 GMT+02:00 Peter Manev <petermanev at gmail.com>:

> On Tue, Sep 27, 2016 at 11:47 AM, Peter Manev <petermanev at gmail.com>
> wrote:
> > On Tue, Sep 27, 2016 at 11:13 AM, Michał D <michu162 at gmail.com> wrote:
> >> In af-packet mode (/usr/bin/suricata -c /etc/suricata/suricata.yaml
> >> --disable-detection --pidfile /var/run/suricata.pid --af-packet -D -vvv
> -F
> >> /var/log/suricata/bpf_filter.txt ) suricata still utilise only two
> cores.
> >
> > In the pastebin info provided (your previous mails) - it seems you have
> -
> >   Detection enabled:                      yes
> >
> > You need to compile it first (./configure --disable-detection && make
> > clean && make && make install)  - as opposed to  pass it to the run
> > line.
> >
>
> Correction - it should work just as you have it as well -
>
> /opt/suricataqa/nodetection/bin/suricata -c
> /etc/suricata/suricata.yaml --af-packet=eth0 -vvv --set
> "af-packet.0.threads=2" --disable-detection
> [19553] 27/9/2016 -- 11:58:39 - (suricata.c:1529) <Info>
> (ParseCommandLine) -- detection engine disabled
> [19553] 27/9/2016 -- 11:58:39 - (suricata.c:1005) <Notice>
> (SCPrintVersion) -- This is Suricata version 3.2dev (rev 398489e)
> ....
>
> Can you share your suricata.log?
>
> Thank you
>
>
> >
> >
> >> Additionally in log file I can see much less entries per second.
> >>
> >> 2016-09-27 10:51 GMT+02:00 Peter Manev <petermanev at gmail.com>:
> >>>
> >>> On Tue, Sep 27, 2016 at 10:18 AM, Michał D <michu162 at gmail.com> wrote:
> >>> > Currently I use "--disable-detection" when I'm running suricata and I
> >>> > sill
> >>> > have problems with high CPU usage of only two cores and packet drops
> in
> >>> > peaks.
> >>>
> >>> Try af-packet and see if any diff.
> >>>
> >>> >
> >>> > 2016-09-27 9:56 GMT+02:00 Peter Manev <petermanev at gmail.com>:
> >>> >>
> >>> >> On Tue, Sep 27, 2016 at 8:43 AM, Michał D <michu162 at gmail.com>
> wrote:
> >>> >> > Hello,
> >>> >> >
> >>> >> > I would like to use suricata only to log incoming http requests
> and
> >>> >> > save
> >>> >> > them as json into file (http.json).
> >>> >>
> >>> >> If this is the only thing you need to do  - log http request only -
> no
> >>> >> inspection, no alerts.
> >>> >> You can try the nsm mode (./configure --disable-detection .....) and
> >>> >> enable only http logs in the eve-log section of suricata.yaml.
> >>> >>
> >>> >> > I have server with two 10G interfaces where I'm receiving mirrored
> >>> >> > traffic,
> >>> >> > 48GB of RAM and Intel(R) Xeon(R) CPU E5540 2.53GHz with 16 cores
> >>> >> > Configuration of suricata and build-info you can find here:
> >>> >> > http://pastebin.com/CriMdqJP
> >>> >> >
> >>> >> > Currently it works in PCAP mode, but I can see 100% usage only of
> 2
> >>> >> > CPU
> >>> >> > cores and a lot of drops.
> >>> >> > (/usr/bin/suricata -c /etc/suricata/suricata.yaml
> --disable-detection
> >>> >> > --pidfile /var/run/suricata.pid --pcap=p2p1 --pcap=p2p2 -D -vvv -F
> >>> >> > /etc/suricata/bpf_filter.txt)
> >>> >> >
> >>> >> > How should I configure & run suricata to have no drops and use all
> >>> >> > cores?
> >>> >> >
> >>> >> > Regards
> >>> >> > Michal
> >>> >> >
> >>> >> > _______________________________________________
> >>> >> > Suricata IDS Users mailing list: oisf-users@
> openinfosecfoundation.org
> >>> >> > Site: http://suricata-ids.org | Support:
> >>> >> > http://suricata-ids.org/support/
> >>> >> > List:
> >>> >> > https://lists.openinfosecfoundation.org/
> mailman/listinfo/oisf-users
> >>> >> > Suricata User Conference November 9-11 in Washington, DC:
> >>> >> > http://suricon.net
> >>> >>
> >>> >>
> >>> >>
> >>> >> --
> >>> >> Regards,
> >>> >> Peter Manev
> >>> >
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Regards,
> >>> Peter Manev
> >>
> >>
> >
> >
> >
> > --
> > Regards,
> > Peter Manev
>
>
>
> --
> Regards,
> Peter Manev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20160927/31e0e2db/attachment-0002.html>