[Oisf-users] multicore http requests logging
Peter Manev
petermanev at gmail.com
Wed Sep 28 06:58:20 UTC 2016
On Tue, Sep 27, 2016 at 2:32 PM, Michał D <michu162 at gmail.com> wrote:
> Hello,
>
> In 3.1.2 everything looks the same. Still only two cores with 100%
> utilization for both af-packet and pcap mode.
Do you use any cpu affinity settings or they are the defaults?
>
> Regards,
> Michal
>
> 2016-09-27 12:38 GMT+02:00 Victor Julien <lists at inliniac.net>:
>>
>> On 27-09-16 12:30, Michał D wrote:
>> > Logs from starting and stopping suricata in af-packet mode
>> >
>> > # /usr/bin/suricata -c /etc/suricata/suricata.yaml --disable-detection
>> > --pidfile /var/run/suricata.pid --af-packet -D -vvv -F
>> > /var/log/suricata/bpf_filter.txt
>> > 27/9/2016 -- 12:17:18 - <Info> - detection engine disabled
>> > 27/9/2016 -- 12:17:18 - <Notice> - This is Suricata version 3.1 RELEASE
>>
>> Before trying anything else, upgrade to 3.1.2. We've fixed many issues
>> since 3.1.
>>
>> Cheers,
>> Victor
>>
>>
>> > 27/9/2016 -- 12:17:18 - <Info> - CPUs/cores online: 16
>> > 27/9/2016 -- 12:17:18 - <Config> - Adding interface p2p1 from config
>> > file
>> > 27/9/2016 -- 12:17:18 - <Config> - Adding interface p2p2 from config
>> > file
>> > 27/9/2016 -- 12:17:18 - <Config> - 'default' server has
>> > 'request-body-minimal-inspect-size' set to 33882 and
>> > 'request-body-inspect-window' set to 4053 after randomization.
>> > 27/9/2016 -- 12:17:18 - <Config> - 'default' server has
>> > 'response-body-minimal-inspect-size' set to 42119 and
>> > 'response-body-inspect-window' set to 16872 after randomization.
>> > 27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser disabled
>> > for tls protocol
>> > 27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser disabled
>> > for smb protocol.
>> > 27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser disabled
>> > for dcerpc protocol.
>> > 27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser disabled
>> > for dcerpc protocol.
>> > 27/9/2016 -- 12:17:18 - <Info> - Parsed disabled for ftp protocol.
>> > Protocol detectionstill on.
>> > 27/9/2016 -- 12:17:18 - <Info> - Protocol detection and parser disabled
>> > for smtp protocol.
>> > 27/9/2016 -- 12:17:18 - <Config> - DNS request flood protection level:
>> > 500
>> > 27/9/2016 -- 12:17:18 - <Config> - DNS per flow memcap (state-memcap):
>> > 524288
>> > 27/9/2016 -- 12:17:18 - <Config> - DNS global memcap: 16777216
>> > 27/9/2016 -- 12:17:18 - <Config> - Protocol detection and parser
>> > disabled for modbus protocol.
>> > 27/9/2016 -- 12:17:18 - <Info> - Found an MTU of 1500 for 'p2p1'
>> > 27/9/2016 -- 12:17:18 - <Info> - Found an MTU of 1500 for 'p2p2'
>> > 27/9/2016 -- 12:17:18 - <Config> - allocated 3670016 bytes of memory for
>> > the defrag hash... 65536 buckets of size 56
>> > 27/9/2016 -- 12:17:18 - <Config> - preallocated 65535 defrag trackers of
>> > size 168
>> > 27/9/2016 -- 12:17:18 - <Config> - defrag memory usage: 14679896 bytes,
>> > maximum: 536870912
>> > 27/9/2016 -- 12:17:18 - <Info> - eve-log output device (regular)
>> > initialized: http.json
>> > 27/9/2016 -- 12:17:18 - <Info> - eve-log output device (regular)
>> > initialized: dns.json
>> > 27/9/2016 -- 12:17:18 - <Info> - stats output device (regular)
>> > initialized: stats.log
>> > 27/9/2016 -- 12:17:18 - <Info> - NIC offloading on p2p1: GRO: unset,
>> > LRO: unset
>> > 27/9/2016 -- 12:17:18 - <Info> - Going to use 3 thread(s)
>> > 27/9/2016 -- 12:17:18 - <Info> - NIC offloading on p2p2: GRO: unset,
>> > LRO: unset
>> > 27/9/2016 -- 12:17:18 - <Info> - Going to use 3 thread(s)
>> > 27/9/2016 -- 12:17:19 - <Notice> - all 6 packet processing threads, 4
>> > management threads initialized, engine started.
>> > 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p1'
>> > 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p1'
>> > 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p1'
>> > 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p2'
>> > 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p2'
>> > 27/9/2016 -- 12:17:19 - <Info> - Using BPF '( ... ) ' on iface 'p2p2'
>> > 27/9/2016 -- 12:17:19 - <Info> - All AFP capture threads are running.
>> >
>> > 27/9/2016 -- 12:25:24 - <Notice> - Signal Received. Stopping engine.
>> > 27/9/2016 -- 12:25:24 - <Info> - time elapsed 486.360s
>> > 27/9/2016 -- 12:25:27 - <Notice> - Stats for 'p2p1': pkts: 20029605,
>> > drop: 0 (0.00%), invalid chksum: 0
>> > 27/9/2016 -- 12:25:27 - <Notice> - Stats for 'p2p2': pkts: 20202957,
>> > drop: 0 (0.00%), invalid chksum: 0
>> >
>> > Drop:0, but in files not so many logs.
>> >
>> > 2016-09-27 12:01 GMT+02:00 Peter Manev <petermanev at gmail.com
>> > <mailto:petermanev at gmail.com>>:
>> >
>> > On Tue, Sep 27, 2016 at 11:47 AM, Peter Manev <petermanev at gmail.com
>> > <mailto:petermanev at gmail.com>> wrote:
>> > > On Tue, Sep 27, 2016 at 11:13 AM, Michał D <michu162 at gmail.com
>> > <mailto:michu162 at gmail.com>> wrote:
>> > >> In af-packet mode (/usr/bin/suricata -c
>> > /etc/suricata/suricata.yaml
>> > >> --disable-detection --pidfile /var/run/suricata.pid --af-packet
>> > -D -vvv -F
>> > >> /var/log/suricata/bpf_filter.txt ) suricata still utilise only
>> > two cores.
>> > >
>> > > In the pastebin info provided (your previous mails) - it seems you
>> > have -
>> > > Detection enabled: yes
>> > >
>> > > You need to compile it first (./configure --disable-detection &&
>> > make
>> > > clean && make && make install) - as opposed to pass it to the
>> > run
>> > > line.
>> > >
>> >
>> > Correction - it should work just as you have it as well -
>> >
>> > /opt/suricataqa/nodetection/bin/suricata -c
>> > /etc/suricata/suricata.yaml --af-packet=eth0 -vvv --set
>> > "af-packet.0.threads=2" --disable-detection
>> > [19553] 27/9/2016 -- 11:58:39 - (suricata.c:1529) <Info>
>> > (ParseCommandLine) -- detection engine disabled
>> > [19553] 27/9/2016 -- 11:58:39 - (suricata.c:1005) <Notice>
>> > (SCPrintVersion) -- This is Suricata version 3.2dev (rev 398489e)
>> > ....
>> >
>> > Can you share your suricata.log?
>> >
>> > Thank you
>> >
>> >
>> > >
>> > >
>> > >> Additionally in log file I can see much less entries per second.
>> > >>
>> > >> 2016-09-27 10:51 GMT+02:00 Peter Manev <petermanev at gmail.com
>> > <mailto:petermanev at gmail.com>>:
>> > >>>
>> > >>> On Tue, Sep 27, 2016 at 10:18 AM, Michał D <michu162 at gmail.com
>> > <mailto:michu162 at gmail.com>> wrote:
>> > >>> > Currently I use "--disable-detection" when I'm running
>> > suricata and I
>> > >>> > sill
>> > >>> > have problems with high CPU usage of only two cores and packet
>> > drops in
>> > >>> > peaks.
>> > >>>
>> > >>> Try af-packet and see if any diff.
>> > >>>
>> > >>> >
>> > >>> > 2016-09-27 9:56 GMT+02:00 Peter Manev <petermanev at gmail.com
>> > <mailto:petermanev at gmail.com>>:
>> > >>> >>
>> > >>> >> On Tue, Sep 27, 2016 at 8:43 AM, Michał D <michu162 at gmail.com
>> > <mailto:michu162 at gmail.com>> wrote:
>> > >>> >> > Hello,
>> > >>> >> >
>> > >>> >> > I would like to use suricata only to log incoming http
>> > requests and
>> > >>> >> > save
>> > >>> >> > them as json into file (http.json).
>> > >>> >>
>> > >>> >> If this is the only thing you need to do - log http request
>> > only - no
>> > >>> >> inspection, no alerts.
>> > >>> >> You can try the nsm mode (./configure --disable-detection
>> > .....) and
>> > >>> >> enable only http logs in the eve-log section of
>> > suricata.yaml.
>> > >>> >>
>> > >>> >> > I have server with two 10G interfaces where I'm receiving
>> > mirrored
>> > >>> >> > traffic,
>> > >>> >> > 48GB of RAM and Intel(R) Xeon(R) CPU E5540 2.53GHz with 16
>> > cores
>> > >>> >> > Configuration of suricata and build-info you can find here:
>> > >>> >> > http://pastebin.com/CriMdqJP
>> > >>> >> >
>> > >>> >> > Currently it works in PCAP mode, but I can see 100% usage
>> > only of 2
>> > >>> >> > CPU
>> > >>> >> > cores and a lot of drops.
>> > >>> >> > (/usr/bin/suricata -c /etc/suricata/suricata.yaml
>> > --disable-detection
>> > >>> >> > --pidfile /var/run/suricata.pid --pcap=p2p1 --pcap=p2p2 -D
>> > -vvv -F
>> > >>> >> > /etc/suricata/bpf_filter.txt)
>> > >>> >> >
>> > >>> >> > How should I configure & run suricata to have no drops and
>> > use all
>> > >>> >> > cores?
>> > >>> >> >
>> > >>> >> > Regards
>> > >>> >> > Michal
>> > >>> >> >
>> > >>> >> > _______________________________________________
>> > >>> >> > Suricata IDS Users mailing list:
>> > oisf-users at openinfosecfoundation.org
>> > <mailto:oisf-users at openinfosecfoundation.org>
>> > >>> >> > Site: http://suricata-ids.org | Support:
>> > >>> >> > http://suricata-ids.org/support/
>> > <http://suricata-ids.org/support/>
>> > >>> >> > List:
>> > >>> >> >
>> > https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>> >
>> > <https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users>
>> > >>> >> > Suricata User Conference November 9-11 in Washington, DC:
>> > >>> >> > http://suricon.net
>> > >>> >>
>> > >>> >>
>> > >>> >>
>> > >>> >> --
>> > >>> >> Regards,
>> > >>> >> Peter Manev
>> > >>> >
>> > >>> >
>> > >>>
>> > >>>
>> > >>>
>> > >>> --
>> > >>> Regards,
>> > >>> Peter Manev
>> > >>
>> > >>
>> > >
>> > >
>> > >
>> > > --
>> > > Regards,
>> > > Peter Manev
>> >
>> >
>> >
>> > --
>> > Regards,
>> > Peter Manev
>> >
>> >
>> >
>> >
>> > _______________________________________________
>> > Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>> > Site: http://suricata-ids.org | Support:
>> > http://suricata-ids.org/support/
>> > List:
>> > https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>> > Suricata User Conference November 9-11 in Washington, DC:
>> > http://suricon.net
>> >
>>
>>
>> --
>> ---------------------------------------------
>> Victor Julien
>> http://www.inliniac.net/
>> PGP: http://www.inliniac.net/victorjulien.asc
>> ---------------------------------------------
>>
>> _______________________________________________
>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
>> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>> Suricata User Conference November 9-11 in Washington, DC:
>> http://suricon.net
>
>
>
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> Suricata User Conference November 9-11 in Washington, DC: http://suricon.net
--
Regards,
Peter Manev
More information about the Oisf-users
mailing list