[Oisf-users] HTTP Logging Update

Adnan Baykal abaykal at gmail.com
Thu Jun 5 20:12:15 UTC 2014


just turned on eve.json and am using the new config.  no packet drops, but
no http whatsoever. this is so weird



On Thu, Jun 5, 2014 at 3:42 PM, Peter Manev <petermanev at gmail.com> wrote:

> On Thu, Jun 5, 2014 at 9:14 PM, Adnan Baykal <abaykal at gmail.com> wrote:
> > I am using Suricata 2.0 but I will update the config and try again.
> >
>
> If I remember correctly 2.0 had a bug where you needed to have both
> eve.json (http) and http.log enabled in order to get logs written.
> 2.0.1 has that fixed.
>
> P.S.
> click "reply all" :)
>
>
> >
> >
> > On Thu, Jun 5, 2014 at 2:57 PM, Peter Manev <petermanev at gmail.com>
> wrote:
> >>
> >> On Thu, Jun 5, 2014 at 8:15 PM, Adnan Baykal <abaykal at gmail.com> wrote:
> >> > here is some more info:
> >> >
> >> > - http-log:
> >> >
> >> >       enabled: yes
> >> >
> >> >       filename: http.log
> >> >
> >> >       append: yes
> >> >
> >> >       #extended: yes     # enable this for extended logging
> information
> >> >
> >> >       custom: yes       # enabled the custom logging format (defined
> by
> >> > customformat)
> >> >
> >> >       customformat: "%a [**]  %{User-agent}i"
> >> >
> >> >       #filetype: regular # 'regular', 'unix_stream' or 'unix_dgram'
> >> >
> >> >
> >> > detect-engine:
> >> >
> >> >   - profile: custom
> >> >
> >> >   - custom-values:
> >> >
> >> >       toclient-src-groups: 200
> >> >
> >> >       toclient-dst-groups: 200
> >> >
> >> >       toclient-sp-groups: 200
> >> >
> >> >       toclient-dp-groups: 300
> >> >
> >> >       toserver-src-groups: 200
> >> >
> >> >       toserver-dst-groups: 400
> >> >
> >> >       toserver-sp-groups: 200
> >> >
> >> >       toserver-dp-groups: 250
> >> >
> >> >   - sgh-mpm-context: auto
> >> >
> >> >   - inspection-recursion-limit: 3000
> >> >
> >> >
> >> > flow:
> >> >
> >> >   memcap: 1gb
> >> >
> >> >   hash-size: 1048576
> >> >
> >> >   prealloc: 1048576
> >> >
> >> >   emergency-recovery: 30
> >> >
> >> >
> >> > # This option controls the use of vlan ids in the flow (and defrag)
> >> >
> >> > # hashing. Normally this should be enabled, but in some (broken)
> >> >
> >> > # setups where both sides of a flow are not tagged with the same vlan
> >> >
> >> > # tag, we can ignore the vlan id's in the flow hashing.
> >> >
> >> > vlan:
> >> >
> >> >   use-for-tracking: false
> >> >
> >> >
> >> > flow-timeouts:
> >> >
> >> >
> >> >   default:
> >> >
> >> >     new: 5
> >> >
> >> >     established: 5
> >> >
> >> >     closed: 0
> >> >
> >> >     emergency-new: 5
> >> >
> >> >     emergency-established: 5
> >> >
> >> >     emergency-closed: 0
> >> >
> >> >   tcp:
> >> >
> >> >     new: 5
> >> >
> >> >     established: 100
> >> >
> >> >     closed: 10
> >> >
> >> >     emergency-new: 1
> >> >
> >> >     emergency-established: 5
> >> >
> >> >     emergency-closed: 5
> >> >
> >> >   udp:
> >> >
> >> >     new: 5
> >> >
> >> >     established: 5
> >> >
> >> >     emergency-new: 5
> >> >
> >> >     emergency-established: 5
> >> >
> >> >   icmp:
> >> >
> >> >     new: 5
> >> >
> >> >     established: 5
> >> >
> >> >     emergency-new: 5
> >> >
> >> >     emergency-established: 5
> >> >
> >> >
> >> >
> >> >
> >> > stream:
> >> >
> >> >   memcap: 4gb
> >> >
> >> >   checksum-validation: no      # reject wrong csums
> >> >
> >> >   inline: no                   # auto will use inline mode in IPS
> mode,
> >> > yes
> >> > or no set it statically
> >>
> >> ####
> >>
> >> >
> >> >   max-sessions: 20000000
> >> >
> >> >   prealloc-sessions: 10000000
> >>
> >> ####
> >>
> >> Judging by this above, you are using an old config or old version of
> >> Suricata.
> >> Could you try the latest config and Suricata and enable eve.json (only
> >> http) and see if it makes a difference?
> >>
> >>
> >> >
> >> >   #midstream: true
> >> >
> >> >   #asyn-oneside: true
> >> >
> >> > reassembly:
> >> >
> >> >     memcap: 12gb
> >> >
> >> >     depth: 1mb                  # reassemble 1mb into a stream
> >> >
> >> >     toserver-chunk-size: 2560
> >> >
> >> >     toclient-chunk-size: 2560
> >> >
> >> >     randomize-chunk-size: yes
> >> >
> >> >     #randomize-chunk-range: 10
> >> >
> >> >     #raw: yes
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > Here is the portion of the stat file:
> >> >
> >> > capture.kernel_packets    | RxPFRp2p17                | 6073789
> >> >
> >> > capture.kernel_drops      | RxPFRp2p17                | 0
> >> >
> >> > dns.memuse                | RxPFRp2p17                | 848709
> >> >
> >> > dns.memcap_state          | RxPFRp2p17                | 0
> >> >
> >> > dns.memcap_global         | RxPFRp2p17                | 0
> >> >
> >> > decoder.pkts              | RxPFRp2p17                | 6073790
> >> >
> >> > decoder.bytes             | RxPFRp2p17                | 2412501092
> >> >
> >> > decoder.invalid           | RxPFRp2p17                | 0
> >> >
> >> > decoder.ipv4              | RxPFRp2p17                | 6074122
> >> >
> >> > decoder.ipv6              | RxPFRp2p17                | 276
> >> >
> >> > decoder.ethernet          | RxPFRp2p17                | 6073790
> >> >
> >> > decoder.raw               | RxPFRp2p17                | 0
> >> >
> >> > decoder.sll               | RxPFRp2p17                | 0
> >> >
> >> > decoder.tcp               | RxPFRp2p17                | 4933100
> >> >
> >> > decoder.udp               | RxPFRp2p17                | 595580
> >> >
> >> > decoder.sctp              | RxPFRp2p17                | 0
> >> >
> >> > decoder.icmpv4            | RxPFRp2p17                | 2734
> >> >
> >> > decoder.icmpv6            | RxPFRp2p17                | 218
> >> >
> >> > decoder.ppp               | RxPFRp2p17                | 0
> >> >
> >> > decoder.pppoe             | RxPFRp2p17                | 0
> >> >
> >> > decoder.gre               | RxPFRp2p17                | 354
> >> >
> >> > decoder.vlan              | RxPFRp2p17                | 0
> >> >
> >> > decoder.vlan_qinq         | RxPFRp2p17                | 0
> >> >
> >> > decoder.teredo            | RxPFRp2p17                | 61
> >> >
> >> > decoder.ipv4_in_ipv6      | RxPFRp2p17                | 0
> >> >
> >> > decoder.ipv6_in_ipv6      | RxPFRp2p17                | 0
> >> >
> >> > decoder.avg_pkt_size      | RxPFRp2p17                | 397
> >> >
> >> > decoder.max_pkt_size      | RxPFRp2p17                | 1514
> >> >
> >> > defrag.ipv4.fragments     | RxPFRp2p17                | 18078
> >> >
> >> > defrag.ipv4.reassembled   | RxPFRp2p17                | 0
> >> >
> >> > defrag.ipv4.timeouts      | RxPFRp2p17                | 0
> >> >
> >> > defrag.ipv6.fragments     | RxPFRp2p17                | 0
> >> >
> >> > defrag.ipv6.reassembled   | RxPFRp2p17                | 0
> >> >
> >> > defrag.ipv6.timeouts      | RxPFRp2p17                | 0
> >> >
> >> > defrag.max_frag_hits      | RxPFRp2p17                | 0
> >> >
> >> > tcp.sessions              | RxPFRp2p17                | 110314
> >> >
> >> > tcp.ssn_memcap_drop       | RxPFRp2p17                | 0
> >> >
> >> > tcp.pseudo                | RxPFRp2p17                | 0
> >> >
> >> > tcp.invalid_checksum      | RxPFRp2p17                | 0
> >> >
> >> > tcp.no_flow               | RxPFRp2p17                | 0
> >> >
> >> > tcp.reused_ssn            | RxPFRp2p17                | 0
> >> >
> >> > tcp.memuse                | RxPFRp2p17                | 5348928
> >> >
> >> > tcp.syn                   | RxPFRp2p17                | 112183
> >> >
> >> > tcp.synack                | RxPFRp2p17                | 15048
> >> >
> >> > tcp.rst                   | RxPFRp2p17                | 34856
> >> >
> >> > tcp.segment_memcap_drop   | RxPFRp2p17                | 0
> >> >
> >> > tcp.stream_depth_reached  | RxPFRp2p17                | 0
> >> >
> >> > tcp.reassembly_memuse     | RxPFRp2p17                | 0
> >> >
> >> > tcp.reassembly_gap        | RxPFRp2p17                | 0
> >> >
> >> > http.memuse               | RxPFRp2p17                | 0
> >> >
> >> > http.memcap               | RxPFRp2p17                | 0
> >> >
> >> > detect.alert              | RxPFRp2p17                | 0
> >> >
> >> > capture.kernel_packets    | RxPFRp2p18                | 5863379
> >> >
> >> > capture.kernel_drops      | RxPFRp2p18                | 0
> >> >
> >> > dns.memuse                | RxPFRp2p18                | 849275
> >> >
> >> > dns.memcap_state          | RxPFRp2p18                | 0
> >> >
> >> > dns.memcap_global         | RxPFRp2p18                | 0
> >> >
> >> > decoder.pkts              | RxPFRp2p18                | 5863380
> >> >
> >> > decoder.bytes             | RxPFRp2p18                | 2460791034
> >> >
> >> > decoder.invalid           | RxPFRp2p18                | 0
> >> >
> >> > decoder.ipv4              | RxPFRp2p18                | 5863428
> >> >
> >> > decoder.ipv6              | RxPFRp2p18                | 236
> >> >
> >> > decoder.ethernet          | RxPFRp2p18                | 5863380
> >> >
> >> > decoder.raw               | RxPFRp2p18                | 0
> >> >
> >> > decoder.sll               | RxPFRp2p18                | 0
> >> >
> >> > decoder.tcp               | RxPFRp2p18                | 4880656
> >> >
> >> > decoder.udp               | RxPFRp2p18                | 538940
> >> >
> >> > decoder.sctp              | RxPFRp2p18                | 0
> >> >
> >> > decoder.icmpv4            | RxPFRp2p18                | 2987
> >> >
> >> > decoder.icmpv6            | RxPFRp2p18                | 201
> >> >
> >> > decoder.ppp               | RxPFRp2p18                | 0
> >> >
> >> > decoder.pppoe             | RxPFRp2p18                | 0
> >> >
> >> > decoder.gre               | RxPFRp2p18                | 48
> >> >
> >> > decoder.vlan              | RxPFRp2p18                | 0
> >> >
> >> > decoder.vlan_qinq         | RxPFRp2p18                | 0
> >> >
> >> > decoder.teredo            | RxPFRp2p18                | 35
> >> >
> >> > decoder.ipv4_in_ipv6      | RxPFRp2p18                | 0
> >> >
> >> > decoder.ipv6_in_ipv6      | RxPFRp2p18                | 0
> >> >
> >> > decoder.avg_pkt_size      | RxPFRp2p18                | 419
> >> >
> >> > decoder.max_pkt_size      | RxPFRp2p18                | 1514
> >> >
> >> > defrag.ipv4.fragments     | RxPFRp2p18                | 17064
> >> >
> >> > defrag.ipv4.reassembled   | RxPFRp2p18                | 0
> >> >
> >> > defrag.ipv4.timeouts      | RxPFRp2p18                | 0
> >> >
> >> > defrag.ipv6.fragments     | RxPFRp2p18                | 0
> >> >
> >> > defrag.ipv6.reassembled   | RxPFRp2p18                | 0
> >> >
> >> > defrag.ipv6.timeouts      | RxPFRp2p18                | 0
> >> >
> >> > defrag.max_frag_hits      | RxPFRp2p18                | 0
> >> >
> >> > tcp.sessions              | RxPFRp2p18                | 110186
> >> >
> >> > tcp.ssn_memcap_drop       | RxPFRp2p18                | 0
> >> >
> >> > tcp.pseudo                | RxPFRp2p18                | 0
> >> >
> >> > tcp.invalid_checksum      | RxPFRp2p18                | 0
> >> >
> >> > tcp.no_flow               | RxPFRp2p18                | 0
> >> >
> >> > tcp.reused_ssn            | RxPFRp2p18                | 0
> >> >
> >> > tcp.memuse                | RxPFRp2p18                | 5348928
> >> >
> >> > tcp.syn                   | RxPFRp2p18                | 112081
> >> >
> >> > tcp.synack                | RxPFRp2p18                | 15365
> >> >
> >> > tcp.rst                   | RxPFRp2p18                | 34375
> >> >
> >> > tcp.segment_memcap_drop   | RxPFRp2p18                | 0
> >> >
> >> > tcp.stream_depth_reached  | RxPFRp2p18                | 0
> >> >
> >> > tcp.reassembly_memuse     | RxPFRp2p18                | 0
> >> >
> >> > tcp.reassembly_gap        | RxPFRp2p18                | 0
> >> >
> >> > http.memuse               | RxPFRp2p18                | 0
> >> >
> >> > http.memcap               | RxPFRp2p18                | 0
> >> >
> >> > detect.alert              | RxPFRp2p18                | 0
> >> >
> >> > flow_mgr.closed_pruned    | FlowManagerThread         | 1151189
> >> >
> >> > flow_mgr.new_pruned       | FlowManagerThread         | 1106070
> >> >
> >> > flow_mgr.est_pruned       | FlowManagerThread         | 0
> >> >
> >> > flow.memuse               | FlowManagerThread         | 400679248
> >> >
> >> > flow.spare                | FlowManagerThread         | 1058085
> >> >
> >> > flow.emerg_mode_entered   | FlowManagerThread         | 0
> >> >
> >> > flow.emerg_mode_over      | FlowManagerThread         | 0
> >> >
> >> >
> >> >
> >> >
> >> > On Thu, Jun 5, 2014 at 2:14 PM, Adnan Baykal <abaykal at gmail.com>
> wrote:
> >> >>
> >> >> Checksums are disabled.
> >> >>
> >> >>
> >> >> On Thu, Jun 5, 2014 at 2:10 PM, Peter Manev <petermanev at gmail.com>
> >> >> wrote:
> >> >>>
> >> >>> On Thu, Jun 5, 2014 at 8:06 PM, Adnan Baykal <abaykal at gmail.com>
> >> >>> wrote:
> >> >>> > disabled vlan as tracking as well.
> >> >>> >
> >> >>> >
> >> >>>
> >> >>> What is your setting for checksums in suricata.yaml - enabled or
> >> >>> disabled?
> >> >>>
> >> >>>
> >> >>> >
> >> >>> > On Thu, Jun 5, 2014 at 2:04 PM, Peter Manev <petermanev at gmail.com
> >
> >> >>> > wrote:
> >> >>> >>
> >> >>> >> On Thu, Jun 5, 2014 at 5:30 PM, Adnan Baykal <abaykal at gmail.com>
> >> >>> >> wrote:
> >> >>> >> > Here is what I did. I found a top talker - video streaming -
> and
> >> >>> >> > put
> >> >>> >> > a
> >> >>> >> > bpf
> >> >>> >> > filter to filter it out (not (host 1.2.3.4)). I am not dropping
> >> >>> >> > as
> >> >>> >> > many
> >> >>> >> > packets any more (about 3%-4%).
> >> >>> >> >
> >> >>> >> > however, I still see extremely low number of http entries in
> the
> >> >>> >> > http
> >> >>> >> > log
> >> >>> >> > and I dont see anything when I take out the midstream and async
> >> >>> >> > entries
> >> >>> >> > from
> >> >>> >> > the yaml file.
> >> >>> >> >
> >> >>> >>
> >> >>> >> Do you have VLANs on the mirror port?
> >> >>> >>
> >> >>> >>
> >> >>> >> >
> >> >>> >> >
> >> >>> >> > On Wed, Jun 4, 2014 at 8:14 PM, Adnan Baykal <
> abaykal at gmail.com>
> >> >>> >> > wrote:
> >> >>> >> >>
> >> >>> >> >> Mbit
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >> On Wed, Jun 4, 2014 at 4:38 PM, Peter Manev
> >> >>> >> >> <petermanev at gmail.com>
> >> >>> >> >> wrote:
> >> >>> >> >>>
> >> >>> >> >>> On Wed, Jun 4, 2014 at 10:33 PM, Adnan Baykal
> >> >>> >> >>> <abaykal at gmail.com>
> >> >>> >> >>> wrote:
> >> >>> >> >>> > I do load about 7K rules.  I need to go back to my sensor
> but
> >> >>> >> >>> > it
> >> >>> >> >>> > is
> >> >>> >> >>> > probably
> >> >>> >> >>> > around 800MB/s
> >> >>> >> >>> >
> >> >>> >> >>> >
> >> >>> >> >>>
> >> >>> >> >>> Just to confirm - is that 800 Mbit or MByte?
> >> >>> >> >>>
> >> >>> >> >>>
> >> >>> >> >>> > On Wed, Jun 4, 2014 at 4:17 PM, Peter Manev
> >> >>> >> >>> > <petermanev at gmail.com>
> >> >>> >> >>> > wrote:
> >> >>> >> >>> >>
> >> >>> >> >>> >> On Wed, Jun 4, 2014 at 10:08 PM, Adnan Baykal
> >> >>> >> >>> >> <abaykal at gmail.com>
> >> >>> >> >>> >> wrote:
> >> >>> >> >>> >> > I have been having no HTTP logging at all on one of my
> >> >>> >> >>> >> > sensors. I
> >> >>> >> >>> >> > have
> >> >>> >> >>> >> > posted several questions to this blog. Mind you that
> this
> >> >>> >> >>> >> > sensor
> >> >>> >> >>> >> > does
> >> >>> >> >>> >> > drop
> >> >>> >> >>> >> > significant amount of data (about 50%) and I do
> understand
> >> >>> >> >>> >> > that
> >> >>> >> >>> >> > there
> >> >>> >> >>> >> > will
> >> >>> >> >>> >> > be a lot of http traffic missed due to drops but not
> >> >>> >> >>> >> > having
> >> >>> >> >>> >> > any
> >> >>> >> >>> >> > entry in
> >> >>> >> >>> >> > the
> >> >>> >> >>> >> > http.log file was concerning. I thought I would at least
> >> >>> >> >>> >> > see
> >> >>> >> >>> >> > some
> >> >>> >> >>> >> > entries.
> >> >>> >> >>> >> >
> >> >>> >> >>> >> > This morning, I found a setting:
> >> >>> >> >>> >> >
> >> >>> >> >>> >> >   midstream: true             # do not allow midstream
> >> >>> >> >>> >> > session
> >> >>> >> >>> >> > pickups
> >> >>> >> >>> >> >   async_oneside: true         # do not enable async
> stream
> >> >>> >> >>> >> > handling
> >> >>> >> >>> >> >
> >> >>> >> >>> >> > When above setting is applied to the stream, I get
> limited
> >> >>> >> >>> >> > HTTP
> >> >>> >> >>> >> > log.
> >> >>> >> >>> >> > My
> >> >>> >> >>> >> > question is "can this change in behavior be explained by
> >> >>> >> >>> >> > dropped
> >> >>> >> >>> >> > packets"?
> >> >>> >> >>> >> > does this change further support the theory that this
> box
> >> >>> >> >>> >> > is
> >> >>> >> >>> >> > significantly
> >> >>> >> >>> >> > undersized and that the bigger box would operate
> normally
> >> >>> >> >>> >> > with
> >> >>> >> >>> >> > full
> >> >>> >> >>> >> > http
> >> >>> >> >>> >> > traffic?
> >> >>> >> >>> >> >
> >> >>> >> >>> >> > I am in the process of upgrading this sensor to a 32GB
> 20
> >> >>> >> >>> >> > Core
> >> >>> >> >>> >> > system
> >> >>> >> >>> >> > (it is
> >> >>> >> >>> >> > currently 16GB 8 Core).
> >> >>> >> >>> >> >
> >> >>> >> >>> >> > Thanks,
> >> >>> >> >>> >> >
> >> >>> >> >>> >> > --Adnan
> >> >>> >> >>> >> >
> >> >>> >> >>> >> >
> >> >>> >> >>> >> > _______________________________________________
> >> >>> >> >>> >> > Suricata IDS Users mailing list:
> >> >>> >> >>> >> > oisf-users at openinfosecfoundation.org
> >> >>> >> >>> >> > Site: http://suricata-ids.org | Support:
> >> >>> >> >>> >> > http://suricata-ids.org/support/
> >> >>> >> >>> >> > List:
> >> >>> >> >>> >> >
> >> >>> >> >>> >> >
> >> >>> >> >>> >> >
> >> >>> >> >>> >> >
> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> >> >>> >> >>> >> > OISF: http://www.openinfosecfoundation.org/
> >> >>> >> >>> >>
> >> >>> >> >>> >> In general if you have significant % of drops  - you will
> be
> >> >>> >> >>> >> missing a
> >> >>> >> >>> >> lot of logs.
> >> >>> >> >>> >> How much traffic do you inspect with that set up? (and how
> >> >>> >> >>> >> many
> >> >>> >> >>> >> rules
> >> >>> >> >>> >> do you load?)
> >> >>> >> >>> >>
> >> >>> >> >>> >>
> >> >>> >> >>> >> --
> >> >>> >> >>> >> Regards,
> >> >>> >> >>> >> Peter Manev
> >> >>> >> >>> >
> >> >>> >> >>> >
> >> >>> >> >>>
> >> >>> >> >>>
> >> >>> >> >>>
> >> >>> >> >>> --
> >> >>> >> >>> Regards,
> >> >>> >> >>> Peter Manev
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >> --
> >> >>> >> Regards,
> >> >>> >> Peter Manev
> >> >>> >
> >> >>> >
> >> >>>
> >> >>>
> >> >>>
> >> >>> --
> >> >>> Regards,
> >> >>> Peter Manev
> >> >>
> >> >>
> >> >
> >>
> >>
> >>
> >> --
> >> Regards,
> >> Peter Manev
> >
> >
>
>
>
> --
> Regards,
> Peter Manev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20140605/cdcc15ae/attachment-0002.html>


More information about the Oisf-users mailing list