[Oisf-users] HTTP Logging Update
Peter Manev
petermanev at gmail.com
Thu Jun 5 18:57:50 UTC 2014
On Thu, Jun 5, 2014 at 8:15 PM, Adnan Baykal <abaykal at gmail.com> wrote:
> here is some more info:
>
> - http-log:
>
> enabled: yes
>
> filename: http.log
>
> append: yes
>
> #extended: yes # enable this for extended logging information
>
> custom: yes # enabled the custom logging format (defined by
> customformat)
>
> customformat: "%a [**] %{User-agent}i"
>
> #filetype: regular # 'regular', 'unix_stream' or 'unix_dgram'
>
>
> detect-engine:
>
> - profile: custom
>
> - custom-values:
>
> toclient-src-groups: 200
>
> toclient-dst-groups: 200
>
> toclient-sp-groups: 200
>
> toclient-dp-groups: 300
>
> toserver-src-groups: 200
>
> toserver-dst-groups: 400
>
> toserver-sp-groups: 200
>
> toserver-dp-groups: 250
>
> - sgh-mpm-context: auto
>
> - inspection-recursion-limit: 3000
>
>
> flow:
>
> memcap: 1gb
>
> hash-size: 1048576
>
> prealloc: 1048576
>
> emergency-recovery: 30
>
>
> # This option controls the use of vlan ids in the flow (and defrag)
>
> # hashing. Normally this should be enabled, but in some (broken)
>
> # setups where both sides of a flow are not tagged with the same vlan
>
> # tag, we can ignore the vlan id's in the flow hashing.
>
> vlan:
>
> use-for-tracking: false
>
>
> flow-timeouts:
>
>
> default:
>
> new: 5
>
> established: 5
>
> closed: 0
>
> emergency-new: 5
>
> emergency-established: 5
>
> emergency-closed: 0
>
> tcp:
>
> new: 5
>
> established: 100
>
> closed: 10
>
> emergency-new: 1
>
> emergency-established: 5
>
> emergency-closed: 5
>
> udp:
>
> new: 5
>
> established: 5
>
> emergency-new: 5
>
> emergency-established: 5
>
> icmp:
>
> new: 5
>
> established: 5
>
> emergency-new: 5
>
> emergency-established: 5
>
>
>
>
> stream:
>
> memcap: 4gb
>
> checksum-validation: no # reject wrong csums
>
> inline: no # auto will use inline mode in IPS mode, yes
> or no set it statically
####
>
> max-sessions: 20000000
>
> prealloc-sessions: 10000000
####
Judging by this above, you are using an old config or old version of Suricata.
Could you try the latest config and Suricata and enable eve.json (only
http) and see if it makes a difference?
>
> #midstream: true
>
> #asyn-oneside: true
>
> reassembly:
>
> memcap: 12gb
>
> depth: 1mb # reassemble 1mb into a stream
>
> toserver-chunk-size: 2560
>
> toclient-chunk-size: 2560
>
> randomize-chunk-size: yes
>
> #randomize-chunk-range: 10
>
> #raw: yes
>
>
>
>
>
>
> Here is the portion of the stat file:
>
> capture.kernel_packets | RxPFRp2p17 | 6073789
>
> capture.kernel_drops | RxPFRp2p17 | 0
>
> dns.memuse | RxPFRp2p17 | 848709
>
> dns.memcap_state | RxPFRp2p17 | 0
>
> dns.memcap_global | RxPFRp2p17 | 0
>
> decoder.pkts | RxPFRp2p17 | 6073790
>
> decoder.bytes | RxPFRp2p17 | 2412501092
>
> decoder.invalid | RxPFRp2p17 | 0
>
> decoder.ipv4 | RxPFRp2p17 | 6074122
>
> decoder.ipv6 | RxPFRp2p17 | 276
>
> decoder.ethernet | RxPFRp2p17 | 6073790
>
> decoder.raw | RxPFRp2p17 | 0
>
> decoder.sll | RxPFRp2p17 | 0
>
> decoder.tcp | RxPFRp2p17 | 4933100
>
> decoder.udp | RxPFRp2p17 | 595580
>
> decoder.sctp | RxPFRp2p17 | 0
>
> decoder.icmpv4 | RxPFRp2p17 | 2734
>
> decoder.icmpv6 | RxPFRp2p17 | 218
>
> decoder.ppp | RxPFRp2p17 | 0
>
> decoder.pppoe | RxPFRp2p17 | 0
>
> decoder.gre | RxPFRp2p17 | 354
>
> decoder.vlan | RxPFRp2p17 | 0
>
> decoder.vlan_qinq | RxPFRp2p17 | 0
>
> decoder.teredo | RxPFRp2p17 | 61
>
> decoder.ipv4_in_ipv6 | RxPFRp2p17 | 0
>
> decoder.ipv6_in_ipv6 | RxPFRp2p17 | 0
>
> decoder.avg_pkt_size | RxPFRp2p17 | 397
>
> decoder.max_pkt_size | RxPFRp2p17 | 1514
>
> defrag.ipv4.fragments | RxPFRp2p17 | 18078
>
> defrag.ipv4.reassembled | RxPFRp2p17 | 0
>
> defrag.ipv4.timeouts | RxPFRp2p17 | 0
>
> defrag.ipv6.fragments | RxPFRp2p17 | 0
>
> defrag.ipv6.reassembled | RxPFRp2p17 | 0
>
> defrag.ipv6.timeouts | RxPFRp2p17 | 0
>
> defrag.max_frag_hits | RxPFRp2p17 | 0
>
> tcp.sessions | RxPFRp2p17 | 110314
>
> tcp.ssn_memcap_drop | RxPFRp2p17 | 0
>
> tcp.pseudo | RxPFRp2p17 | 0
>
> tcp.invalid_checksum | RxPFRp2p17 | 0
>
> tcp.no_flow | RxPFRp2p17 | 0
>
> tcp.reused_ssn | RxPFRp2p17 | 0
>
> tcp.memuse | RxPFRp2p17 | 5348928
>
> tcp.syn | RxPFRp2p17 | 112183
>
> tcp.synack | RxPFRp2p17 | 15048
>
> tcp.rst | RxPFRp2p17 | 34856
>
> tcp.segment_memcap_drop | RxPFRp2p17 | 0
>
> tcp.stream_depth_reached | RxPFRp2p17 | 0
>
> tcp.reassembly_memuse | RxPFRp2p17 | 0
>
> tcp.reassembly_gap | RxPFRp2p17 | 0
>
> http.memuse | RxPFRp2p17 | 0
>
> http.memcap | RxPFRp2p17 | 0
>
> detect.alert | RxPFRp2p17 | 0
>
> capture.kernel_packets | RxPFRp2p18 | 5863379
>
> capture.kernel_drops | RxPFRp2p18 | 0
>
> dns.memuse | RxPFRp2p18 | 849275
>
> dns.memcap_state | RxPFRp2p18 | 0
>
> dns.memcap_global | RxPFRp2p18 | 0
>
> decoder.pkts | RxPFRp2p18 | 5863380
>
> decoder.bytes | RxPFRp2p18 | 2460791034
>
> decoder.invalid | RxPFRp2p18 | 0
>
> decoder.ipv4 | RxPFRp2p18 | 5863428
>
> decoder.ipv6 | RxPFRp2p18 | 236
>
> decoder.ethernet | RxPFRp2p18 | 5863380
>
> decoder.raw | RxPFRp2p18 | 0
>
> decoder.sll | RxPFRp2p18 | 0
>
> decoder.tcp | RxPFRp2p18 | 4880656
>
> decoder.udp | RxPFRp2p18 | 538940
>
> decoder.sctp | RxPFRp2p18 | 0
>
> decoder.icmpv4 | RxPFRp2p18 | 2987
>
> decoder.icmpv6 | RxPFRp2p18 | 201
>
> decoder.ppp | RxPFRp2p18 | 0
>
> decoder.pppoe | RxPFRp2p18 | 0
>
> decoder.gre | RxPFRp2p18 | 48
>
> decoder.vlan | RxPFRp2p18 | 0
>
> decoder.vlan_qinq | RxPFRp2p18 | 0
>
> decoder.teredo | RxPFRp2p18 | 35
>
> decoder.ipv4_in_ipv6 | RxPFRp2p18 | 0
>
> decoder.ipv6_in_ipv6 | RxPFRp2p18 | 0
>
> decoder.avg_pkt_size | RxPFRp2p18 | 419
>
> decoder.max_pkt_size | RxPFRp2p18 | 1514
>
> defrag.ipv4.fragments | RxPFRp2p18 | 17064
>
> defrag.ipv4.reassembled | RxPFRp2p18 | 0
>
> defrag.ipv4.timeouts | RxPFRp2p18 | 0
>
> defrag.ipv6.fragments | RxPFRp2p18 | 0
>
> defrag.ipv6.reassembled | RxPFRp2p18 | 0
>
> defrag.ipv6.timeouts | RxPFRp2p18 | 0
>
> defrag.max_frag_hits | RxPFRp2p18 | 0
>
> tcp.sessions | RxPFRp2p18 | 110186
>
> tcp.ssn_memcap_drop | RxPFRp2p18 | 0
>
> tcp.pseudo | RxPFRp2p18 | 0
>
> tcp.invalid_checksum | RxPFRp2p18 | 0
>
> tcp.no_flow | RxPFRp2p18 | 0
>
> tcp.reused_ssn | RxPFRp2p18 | 0
>
> tcp.memuse | RxPFRp2p18 | 5348928
>
> tcp.syn | RxPFRp2p18 | 112081
>
> tcp.synack | RxPFRp2p18 | 15365
>
> tcp.rst | RxPFRp2p18 | 34375
>
> tcp.segment_memcap_drop | RxPFRp2p18 | 0
>
> tcp.stream_depth_reached | RxPFRp2p18 | 0
>
> tcp.reassembly_memuse | RxPFRp2p18 | 0
>
> tcp.reassembly_gap | RxPFRp2p18 | 0
>
> http.memuse | RxPFRp2p18 | 0
>
> http.memcap | RxPFRp2p18 | 0
>
> detect.alert | RxPFRp2p18 | 0
>
> flow_mgr.closed_pruned | FlowManagerThread | 1151189
>
> flow_mgr.new_pruned | FlowManagerThread | 1106070
>
> flow_mgr.est_pruned | FlowManagerThread | 0
>
> flow.memuse | FlowManagerThread | 400679248
>
> flow.spare | FlowManagerThread | 1058085
>
> flow.emerg_mode_entered | FlowManagerThread | 0
>
> flow.emerg_mode_over | FlowManagerThread | 0
>
>
>
>
> On Thu, Jun 5, 2014 at 2:14 PM, Adnan Baykal <abaykal at gmail.com> wrote:
>>
>> Checksums are disabled.
>>
>>
>> On Thu, Jun 5, 2014 at 2:10 PM, Peter Manev <petermanev at gmail.com> wrote:
>>>
>>> On Thu, Jun 5, 2014 at 8:06 PM, Adnan Baykal <abaykal at gmail.com> wrote:
>>> > disabled vlan as tracking as well.
>>> >
>>> >
>>>
>>> What is your setting for checksums in suricata.yaml - enabled or
>>> disabled?
>>>
>>>
>>> >
>>> > On Thu, Jun 5, 2014 at 2:04 PM, Peter Manev <petermanev at gmail.com>
>>> > wrote:
>>> >>
>>> >> On Thu, Jun 5, 2014 at 5:30 PM, Adnan Baykal <abaykal at gmail.com>
>>> >> wrote:
>>> >> > Here is what I did. I found a top talker - video streaming - and put
>>> >> > a
>>> >> > bpf
>>> >> > filter to filter it out (not (host 1.2.3.4)). I am not dropping as
>>> >> > many
>>> >> > packets any more (about 3%-4%).
>>> >> >
>>> >> > however, I still see extremely low number of http entries in the
>>> >> > http
>>> >> > log
>>> >> > and I dont see anything when I take out the midstream and async
>>> >> > entries
>>> >> > from
>>> >> > the yaml file.
>>> >> >
>>> >>
>>> >> Do you have VLANs on the mirror port?
>>> >>
>>> >>
>>> >> >
>>> >> >
>>> >> > On Wed, Jun 4, 2014 at 8:14 PM, Adnan Baykal <abaykal at gmail.com>
>>> >> > wrote:
>>> >> >>
>>> >> >> Mbit
>>> >> >>
>>> >> >>
>>> >> >> On Wed, Jun 4, 2014 at 4:38 PM, Peter Manev <petermanev at gmail.com>
>>> >> >> wrote:
>>> >> >>>
>>> >> >>> On Wed, Jun 4, 2014 at 10:33 PM, Adnan Baykal <abaykal at gmail.com>
>>> >> >>> wrote:
>>> >> >>> > I do load about 7K rules. I need to go back to my sensor but it
>>> >> >>> > is
>>> >> >>> > probably
>>> >> >>> > around 800MB/s
>>> >> >>> >
>>> >> >>> >
>>> >> >>>
>>> >> >>> Just to confirm - is that 800 Mbit or MByte?
>>> >> >>>
>>> >> >>>
>>> >> >>> > On Wed, Jun 4, 2014 at 4:17 PM, Peter Manev
>>> >> >>> > <petermanev at gmail.com>
>>> >> >>> > wrote:
>>> >> >>> >>
>>> >> >>> >> On Wed, Jun 4, 2014 at 10:08 PM, Adnan Baykal
>>> >> >>> >> <abaykal at gmail.com>
>>> >> >>> >> wrote:
>>> >> >>> >> > I have been having no HTTP logging at all on one of my
>>> >> >>> >> > sensors. I
>>> >> >>> >> > have
>>> >> >>> >> > posted several questions to this blog. Mind you that this
>>> >> >>> >> > sensor
>>> >> >>> >> > does
>>> >> >>> >> > drop
>>> >> >>> >> > significant amount of data (about 50%) and I do understand
>>> >> >>> >> > that
>>> >> >>> >> > there
>>> >> >>> >> > will
>>> >> >>> >> > be a lot of http traffic missed due to drops but not having
>>> >> >>> >> > any
>>> >> >>> >> > entry in
>>> >> >>> >> > the
>>> >> >>> >> > http.log file was concerning. I thought I would at least see
>>> >> >>> >> > some
>>> >> >>> >> > entries.
>>> >> >>> >> >
>>> >> >>> >> > This morning, I found a setting:
>>> >> >>> >> >
>>> >> >>> >> > midstream: true # do not allow midstream
>>> >> >>> >> > session
>>> >> >>> >> > pickups
>>> >> >>> >> > async_oneside: true # do not enable async stream
>>> >> >>> >> > handling
>>> >> >>> >> >
>>> >> >>> >> > When above setting is applied to the stream, I get limited
>>> >> >>> >> > HTTP
>>> >> >>> >> > log.
>>> >> >>> >> > My
>>> >> >>> >> > question is "can this change in behavior be explained by
>>> >> >>> >> > dropped
>>> >> >>> >> > packets"?
>>> >> >>> >> > does this change further support the theory that this box is
>>> >> >>> >> > significantly
>>> >> >>> >> > undersized and that the bigger box would operate normally
>>> >> >>> >> > with
>>> >> >>> >> > full
>>> >> >>> >> > http
>>> >> >>> >> > traffic?
>>> >> >>> >> >
>>> >> >>> >> > I am in the process of upgrading this sensor to a 32GB 20
>>> >> >>> >> > Core
>>> >> >>> >> > system
>>> >> >>> >> > (it is
>>> >> >>> >> > currently 16GB 8 Core).
>>> >> >>> >> >
>>> >> >>> >> > Thanks,
>>> >> >>> >> >
>>> >> >>> >> > --Adnan
>>> >> >>> >> >
>>> >> >>> >> >
>>> >> >>> >> > _______________________________________________
>>> >> >>> >> > Suricata IDS Users mailing list:
>>> >> >>> >> > oisf-users at openinfosecfoundation.org
>>> >> >>> >> > Site: http://suricata-ids.org | Support:
>>> >> >>> >> > http://suricata-ids.org/support/
>>> >> >>> >> > List:
>>> >> >>> >> >
>>> >> >>> >> >
>>> >> >>> >> > https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>> >> >>> >> > OISF: http://www.openinfosecfoundation.org/
>>> >> >>> >>
>>> >> >>> >> In general if you have significant % of drops - you will be
>>> >> >>> >> missing a
>>> >> >>> >> lot of logs.
>>> >> >>> >> How much traffic do you inspect with that set up? (and how many
>>> >> >>> >> rules
>>> >> >>> >> do you load?)
>>> >> >>> >>
>>> >> >>> >>
>>> >> >>> >> --
>>> >> >>> >> Regards,
>>> >> >>> >> Peter Manev
>>> >> >>> >
>>> >> >>> >
>>> >> >>>
>>> >> >>>
>>> >> >>>
>>> >> >>> --
>>> >> >>> Regards,
>>> >> >>> Peter Manev
>>> >> >>
>>> >> >>
>>> >> >
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Regards,
>>> >> Peter Manev
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Peter Manev
>>
>>
>
--
Regards,
Peter Manev
More information about the Oisf-users
mailing list