[Oisf-users] multicore http requests logging

Tue Sep 27 10:38:00 UTC 2016

On 27-09-16 12:30, Michał D wrote:
> Logs from starting and stopping suricata in af-packet mode 
> 
> # /usr/bin/suricata -c /etc/suricata/suricata.yaml --disable-detection
> --pidfile /var/run/suricata.pid --af-packet -D -vvv -F
> /var/log/suricata/bpf_filter.txt
> 27/9/2016 -- 12:17:18 - <Info> - detection engine disabled
> 27/9/2016 -- 12:17:18 - <Notice> - This is Suricata version 3.1 RELEASE

Before trying anything else, upgrade to 3.1.2. We've fixed many issues
since 3.1.

Cheers,
Victor

> 27/9/2016 -- 12:17:18 - <Info> - CPUs/cores online: 16
> 27/9/2016 -- 12:17:18 - <Config> - Adding interface p2p1 from config file
> 27/9/2016 -- 12:17:18 - <Config> - Adding interface p2p2 from config file
> 27/9/2016 -- 12:17:18 - <Config> - 'default' server has
> 'request-body-minimal-inspect-size' set to 33882 and
> 'request-body-inspect-window' set to 4053 after randomization.
> 27/9/2016 -- 12:17:18 - <Config> - 'default' server has
> 'response-body-minimal-inspect-size' set to 42119 and
> 'response-body-inspect-window' set to 16872 after randomization.
> 27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser disabled
> for tls protocol
> 27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser disabled
> for smb protocol.
> 27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser disabled
> for dcerpc protocol.
> 27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser disabled
> for dcerpc protocol.
> 27/9/2016 -- 12:17:18 - <Info> - Parsed disabled for ftp protocol.
> Protocol detectionstill on.
> 27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser disabled
> for smtp protocol.
> 27/9/2016 -- 12:17:18 - <Config> - DNS request flood protection level: 500
> 27/9/2016 -- 12:17:18 - <Config> - DNS per flow memcap (state-memcap):
> 524288
> 27/9/2016 -- 12:17:18 - <Config> - DNS global memcap: 16777216
> 27/9/2016 -- 12:17:18 - <Config> - Protocol detection and parser
> disabled for modbus protocol.
> 27/9/2016 -- 12:17:18 - <Info> - Found an MTU of 1500 for 'p2p1'
> 27/9/2016 -- 12:17:18 - <Info> - Found an MTU of 1500 for 'p2p2'
> 27/9/2016 -- 12:17:18 - <Config> - allocated 3670016 bytes of memory for
> the defrag hash... 65536 buckets of size 56
> 27/9/2016 -- 12:17:18 - <Config> - preallocated 65535 defrag trackers of
> size 168
> 27/9/2016 -- 12:17:18 - <Config> - defrag memory usage: 14679896 bytes,
> maximum: 536870912
> 27/9/2016 -- 12:17:18 - <Info> - eve-log output device (regular)
> initialized: http.json
> 27/9/2016 -- 12:17:18 - <Info> - eve-log output device (regular)
> initialized: dns.json
> 27/9/2016 -- 12:17:18 - <Info> - stats output device (regular)
> initialized: stats.log
> 27/9/2016 -- 12:17:18 - <Info> - NIC offloading on p2p1: GRO: unset,
> LRO: unset
> 27/9/2016 -- 12:17:18 - <Info> - Going to use 3 thread(s)
> 27/9/2016 -- 12:17:18 - <Info> - NIC offloading on p2p2: GRO: unset,
> LRO: unset
> 27/9/2016 -- 12:17:18 - <Info> - Going to use 3 thread(s)
> 27/9/2016 -- 12:17:19 - <Notice> - all 6 packet processing threads, 4
> management threads initialized, engine started.
> 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p1'
> 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p1'
> 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p1'
> 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p2'
> 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p2'
> 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p2'
> 27/9/2016 -- 12:17:19 - <Info> - All AFP capture threads are running.
> 
> 27/9/2016 -- 12:25:24 - <Notice> - Signal Received.  Stopping engine.
> 27/9/2016 -- 12:25:24 - <Info> - time elapsed 486.360s
> 27/9/2016 -- 12:25:27 - <Notice> - Stats for 'p2p1':  pkts: 20029605,
> drop: 0 (0.00%), invalid chksum: 0
> 27/9/2016 -- 12:25:27 - <Notice> - Stats for 'p2p2':  pkts: 20202957,
> drop: 0 (0.00%), invalid chksum: 0
> 
> Drop:0, but in files not so many logs. 
> 
> 2016-09-27 12:01 GMT+02:00 Peter Manev <petermanev at gmail.com
> <mailto:petermanev at gmail.com>>:
> 
>     On Tue, Sep 27, 2016 at 11:47 AM, Peter Manev <petermanev at gmail.com
>     <mailto:petermanev at gmail.com>> wrote:
>     > On Tue, Sep 27, 2016 at 11:13 AM, Michał D <michu162 at gmail.com <mailto:michu162 at gmail.com>> wrote:
>     >> In af-packet mode (/usr/bin/suricata -c /etc/suricata/suricata.yaml
>     >> --disable-detection --pidfile /var/run/suricata.pid --af-packet -D -vvv -F
>     >> /var/log/suricata/bpf_filter.txt ) suricata still utilise only two cores.
>     >
>     > In the pastebin info provided (your previous mails) - it seems you have  -
>     >   Detection enabled:                      yes
>     >
>     > You need to compile it first (./configure --disable-detection && make
>     > clean && make && make install)  - as opposed to  pass it to the run
>     > line.
>     >
> 
>     Correction - it should work just as you have it as well -
> 
>     /opt/suricataqa/nodetection/bin/suricata -c
>     /etc/suricata/suricata.yaml --af-packet=eth0 -vvv --set
>     "af-packet.0.threads=2" --disable-detection
>     [19553] 27/9/2016 -- 11:58:39 - (suricata.c:1529) <Info>
>     (ParseCommandLine) -- detection engine disabled
>     [19553] 27/9/2016 -- 11:58:39 - (suricata.c:1005) <Notice>
>     (SCPrintVersion) -- This is Suricata version 3.2dev (rev 398489e)
>     ....
> 
>     Can you share your suricata.log?
> 
>     Thank you
> 
> 
>     >
>     >
>     >> Additionally in log file I can see much less entries per second.
>     >>
>     >> 2016-09-27 10:51 GMT+02:00 Peter Manev <petermanev at gmail.com
>     <mailto:petermanev at gmail.com>>:
>     >>>
>     >>> On Tue, Sep 27, 2016 at 10:18 AM, Michał D <michu162 at gmail.com
>     <mailto:michu162 at gmail.com>> wrote:
>     >>> > Currently I use "--disable-detection" when I'm running
>     suricata and I
>     >>> > sill
>     >>> > have problems with high CPU usage of only two cores and packet
>     drops in
>     >>> > peaks.
>     >>>
>     >>> Try af-packet and see if any diff.
>     >>>
>     >>> >
>     >>> > 2016-09-27 9:56 GMT+02:00 Peter Manev <petermanev at gmail.com
>     <mailto:petermanev at gmail.com>>:
>     >>> >>
>     >>> >> On Tue, Sep 27, 2016 at 8:43 AM, Michał D <michu162 at gmail.com
>     <mailto:michu162 at gmail.com>> wrote:
>     >>> >> > Hello,
>     >>> >> >
>     >>> >> > I would like to use suricata only to log incoming http
>     requests and
>     >>> >> > save
>     >>> >> > them as json into file (http.json).
>     >>> >>
>     >>> >> If this is the only thing you need to do  - log http request
>     only - no
>     >>> >> inspection, no alerts.
>     >>> >> You can try the nsm mode (./configure --disable-detection
>     .....) and
>     >>> >> enable only http logs in the eve-log section of suricata.yaml.
>     >>> >>
>     >>> >> > I have server with two 10G interfaces where I'm receiving
>     mirrored
>     >>> >> > traffic,
>     >>> >> > 48GB of RAM and Intel(R) Xeon(R) CPU E5540 2.53GHz with 16
>     cores
>     >>> >> > Configuration of suricata and build-info you can find here:
>     >>> >> > http://pastebin.com/CriMdqJP
>     >>> >> >
>     >>> >> > Currently it works in PCAP mode, but I can see 100% usage
>     only of 2
>     >>> >> > CPU
>     >>> >> > cores and a lot of drops.
>     >>> >> > (/usr/bin/suricata -c /etc/suricata/suricata.yaml
>     --disable-detection
>     >>> >> > --pidfile /var/run/suricata.pid --pcap=p2p1 --pcap=p2p2 -D
>     -vvv -F
>     >>> >> > /etc/suricata/bpf_filter.txt)
>     >>> >> >
>     >>> >> > How should I configure & run suricata to have no drops and
>     use all
>     >>> >> > cores?
>     >>> >> >
>     >>> >> > Regards
>     >>> >> > Michal
>     >>> >> >
>     >>> >> > _______________________________________________
>     >>> >> > Suricata IDS Users mailing list:
>     oisf-users at openinfosecfoundation.org
>     <mailto:oisf-users at openinfosecfoundation.org>
>     >>> >> > Site: http://suricata-ids.org | Support:
>     >>> >> > http://suricata-ids.org/support/
>     <http://suricata-ids.org/support/>
>     >>> >> > List:
>     >>> >> >
>     https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>     <https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users>
>     >>> >> > Suricata User Conference November 9-11 in Washington, DC:
>     >>> >> > http://suricon.net
>     >>> >>
>     >>> >>
>     >>> >>
>     >>> >> --
>     >>> >> Regards,
>     >>> >> Peter Manev
>     >>> >
>     >>> >
>     >>>
>     >>>
>     >>>
>     >>> --
>     >>> Regards,
>     >>> Peter Manev
>     >>
>     >>
>     >
>     >
>     >
>     > --
>     > Regards,
>     > Peter Manev
> 
> 
> 
>     --
>     Regards,
>     Peter Manev
> 
> 
> 
> 
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> Suricata User Conference November 9-11 in Washington, DC: http://suricon.net
> 

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------