[Oisf-users] Suricata 2Gbit/s traffic drops on AWS
Shell_Xu
xuh881026 at gmail.com
Sat Aug 24 14:07:27 UTC 2019
Thank you for your clarification!
I have disabled '# rollover: yes'. Next I will use 'placement group',
hoping to reduce the packet loss rate.
Peter Manev <petermanev at gmail.com> 于2019年8月24日周六 下午7:08写道：
>
>
> --
>
> Regards,
>
> Peter Manev
>
>
> On 24 Aug 2019, at 06:54, Tiago Faria <tiago.faria.backups at gmail.com>
> wrote:
>
> Rollover can help with packet losses by sending packets to a new socket
> when current one is full. As per the documentation, this can help with
> packet loss on single intensive flows, even though there are cons about
> using this (and it might even result in you missing alerts since Suricata
> might not be able to analyze all the traffic).
>
>
> Just for info “rollover” should not be used -
>
>
> https://github.com/OISF/suricata/commit/5d76f0897cc862a7096749355d261e2b3d130e0d
> It causes tracking issues.
>
>
> You don't _have to_ do the placement group, especially if you're not
> planning on doing 10/15Gbps+.
>
> On Sat, Aug 24, 2019 at 11:38 AM Shell_Xu <xuh881026 at gmail.com> wrote:
>
>> New problem, I tried to add 'rollover:yes' to the configuration file, I
>> found that the packet loss rate has dropped.The 5.0dev version does not
>> have this configuration by default. Is 'rollover:yes' obsolete?
>> In the test, I found that the packet loss rate dropped significantly, but
>> it was not stable.Why is this? What is the role of this configuration?
>> This result only adds configuration parameters, I did not add EC2 to the
>> Placement Group.
>>
>> The verification results are as follows
>> <image.png>
>>
>> Tiago Faria <tiago.faria.backups at gmail.com> 于2019年8月24日周六 下午3:49写道：
>>
>>> You can have mirror sessions as you want, including between AWS
>>> accounts. To get the best performance, however, placing them in the same
>>> placement group will help substantially.
>>>
>>> I’d first check if this helps in the problem you’re having though.
>>>
>>> On Sat, 24 Aug 2019 at 01:46, Shell_Xu <xuh881026 at gmail.com> wrote:
>>>
>>>> HI:
>>>> Thank you for your help!
>>>> 'What I recommend is the creation of a Placement Group of type
>>>> Cluster and deploy the EC2 instances inside that Placement Group. '
>>>> Does this mean that servers I monitor need to be deployed in the
>>>> Placement Group?
>>>> e.g:
>>>> Sruicata、Web Server、DB Server、Redis Cluster...
>>>>
>>>> Tiago Faria <tiago.faria.backups at gmail.com> 于2019年8月24日周六 上午1:38写道：
>>>>
>>>>> Hi,
>>>>>
>>>>> It can be fixed, yes, but it requires deployment of the EC2 instances
>>>>> (or re-deployment). What I recommend is the creation of a Placement Group
>>>>> of type Cluster and deploy the EC2 instances inside that Placement Group.
>>>>>
>>>>> On Fri, Aug 23, 2019 at 5:48 PM Shell_Xu <xuh881026 at gmail.com> wrote:
>>>>>
>>>>>> I am not sure if I use Placement Groups. If not used, can this
>>>>>> problem still be solved?
>>>>>>
>>>>>> Tiago Faria <tiago.faria.backups at gmail.com> 于2019年8月23日周五 下午11:06写道：
>>>>>>
>>>>>>> Are you using EC2 Placement Groups? Ideally you would use Cluster as
>>>>>>> much as possible exactly to prevent underlying hardware performance issues.
>>>>>>>
>>>>>>> It is also the recommended configuration for HPC applications, and
>>>>>>> Suricata would greatly benefit from that.
>>>>>>>
>>>>>>> On Fri, 23 Aug 2019 at 15:54, 徐慧 <xuh881026 at gmail.com> wrote:
>>>>>>>
>>>>>>>> hi, again:
>>>>>>>> Yes, I am using Elastic Network Adapter (ENA)
>>>>>>>> Since the EC2 instance is a shared underlying hardware, many
>>>>>>>> network interface hardware settings are not available.
>>>>>>>> I don't know how to optimize Suricata on EC2, can you help me?
>>>>>>>>
>>>>>>>> $ modinfo ena
>>>>>>>>
>>>>>>>> filename:
>>>>>>>> /lib/modules/4.15.0-1044-aws/kernel/drivers/net/ethernet/amazon/ena/ena.ko
>>>>>>>> version: 2.0.3K
>>>>>>>> license: GPL
>>>>>>>> description: Elastic Network Adapter (ENA)
>>>>>>>> author: Amazon.com, Inc. or its affiliates
>>>>>>>> srcversion: 1980993534E135DFC7933C4
>>>>>>>> alias: pci:v00001D0Fd0000EC21sv*sd*bc*sc*i*
>>>>>>>> alias: pci:v00001D0Fd0000EC20sv*sd*bc*sc*i*
>>>>>>>> alias: pci:v00001D0Fd00001EC2sv*sd*bc*sc*i*
>>>>>>>> alias: pci:v00001D0Fd00000EC2sv*sd*bc*sc*i*
>>>>>>>> depends:
>>>>>>>> retpoline: Y
>>>>>>>> intree: Y
>>>>>>>> name: ena
>>>>>>>> vermagic: 4.15.0-1044-aws SMP mod_unload
>>>>>>>> signat: PKCS#7
>>>>>>>> signer:
>>>>>>>> sig_key:
>>>>>>>> sig_hashalgo: md4
>>>>>>>> parm: debug:Debug level (0=none,...,16=all) (int)
>>>>>>>>
>>>>>>>> Tiago Faria <tiago.faria.backups at gmail.com> 于2019年8月23日周五 下午6:51写道：
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> Based on the instance type and interface name, you're most likely
>>>>>>>>> using enhanced networking, but, to be on the safe side, can you confirm?
>>>>>>>>>
>>>>>>>>> $ modinfo ena
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Aug 23, 2019 at 3:07 AM 徐慧 <xuh881026 at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> hi, team:
>>>>>>>>>> Since AWS traffic mirroring uses a VxLAN tunnel, I have to
>>>>>>>>>> use the 5.0dev version. i deployed Sruicata on AWS, but recently noticed
>>>>>>>>>> that 'capture. Kernel_drops' appears in stats.log when traffic reaches
>>>>>>>>>> 2Gbit/s. I tried rsync a large file, 'capture. Kernel_drops' appears in
>>>>>>>>>> stats.log. default loading ET rules.
>>>>>>>>>> I hope anyone can help me, any advice is good! Guys, I need
>>>>>>>>>> your help very much.
>>>>>>>>>>
>>>>>>>>>> # Client rsync files
>>>>>>>>>> $ rsync -trovpgP xxx.tgz /usr/local/data/xxx.tgz
>>>>>>>>>> sending incremental file list
>>>>>>>>>> xxx.tgz
>>>>>>>>>> 3,361,243,136 51% 114.14MB/s 0:00:27
>>>>>>>>>>
>>>>>>>>>> # Suricata Server:
>>>>>>>>>> $ suricata --af-packet -c /etc/suricata/suricata.yaml
>>>>>>>>>> [24073] 23/8/2019 -- 01:51:19 - (tm-threads.c:2145) <Notice>
>>>>>>>>>> (TmThreadWaitOnThreadInit) -- all 14 packet processing threads, 4
>>>>>>>>>> management threads initialized, engine started.
>>>>>>>>>> [24073] 23/8/2019 -- 01:53:58 - (suricata.c:2851) <Notice>
>>>>>>>>>> (SuricataMainLoop) -- Signal Received. Stopping engine.
>>>>>>>>>> [24073] 23/8/2019 -- 01:54:01 - (util-device.c:317) <Notice>
>>>>>>>>>> (LiveDeviceListClean) -- Stats for 'ens5': pkts: 11270384, drop: 2046365
>>>>>>>>>> (18.16%), invalid chksum: 0
>>>>>>>>>>
>>>>>>>>>> According to the official documentation, I made some
>>>>>>>>>> optimizations.
>>>>>>>>>>
>>>>>>>>>> https://suricata.readthedocs.io/en/latest/performance/packet-capture.html#rss
>>>>>>>>>> But I can't set RSS queues to 1
>>>>>>>>>> ethtool -L ens5 combined 1
>>>>>>>>>> Cannot set device channel parameters: Operation not supported
>>>>>>>>>>
>>>>>>>>>> Amazon EC2 C5
>>>>>>>>>> EC2 Hardware:
>>>>>>>>>> RAM: 32G
>>>>>>>>>> CPU(single): 16 Core (Intel(R) Xeon(R) Platinum 8124M CPU @
>>>>>>>>>> 3.00GHz)
>>>>>>>>>> NIC:
>>>>>>>>>> ethtool -l ens5
>>>>>>>>>> Channel parameters for ens5:
>>>>>>>>>> Pre-set maximums:
>>>>>>>>>> RX: 8
>>>>>>>>>> TX: 8
>>>>>>>>>> Other: 0
>>>>>>>>>> Combined: 0
>>>>>>>>>> Current hardware settings:
>>>>>>>>>> RX: 8
>>>>>>>>>> TX: 8
>>>>>>>>>> Other: 0
>>>>>>>>>> Combined: 0
>>>>>>>>>>
>>>>>>>>>> ethtool -i ens5
>>>>>>>>>> driver: ena
>>>>>>>>>> version: 2.0.3K
>>>>>>>>>> firmware-version:
>>>>>>>>>> expansion-rom-version:
>>>>>>>>>> bus-info: 0000:00:05.0
>>>>>>>>>> supports-statistics: yes
>>>>>>>>>> supports-test: no
>>>>>>>>>> supports-eeprom-access: no
>>>>>>>>>> supports-register-dump: no
>>>>>>>>>> supports-priv-flags: no
>>>>>>>>>>
>>>>>>>>>> Suricata Version: 5.0.0-dev (3a912446a 2019-07-22)
>>>>>>>>>> Suricata Config:
>>>>>>>>>> af-packet:
>>>>>>>>>> - interface: ens5
>>>>>>>>>> threads: 14
>>>>>>>>>> cluster-id: 99
>>>>>>>>>> cluster-type: cluster_flow
>>>>>>>>>> defrag: yes # Default AF_PACKET cluster type.
>>>>>>>>>> AF_PACKET can load balance per flow or per hash.
>>>>>>>>>> use-mmap: yes
>>>>>>>>>> mmap-locked: yes
>>>>>>>>>> tpacket-v3: yes
>>>>>>>>>> ring-size: 400000
>>>>>>>>>> block-size: 393216
>>>>>>>>>> #block-timeout: 10
>>>>>>>>>> #use-emergency-flush: yes
>>>>>>>>>> # buffer-size: 32768
>>>>>>>>>> # disable-promisc: no
>>>>>>>>>> #checksum-checks: kernel
>>>>>>>>>> #bpf-filter: port 80 or udp
>>>>>>>>>> #copy-mode: ips
>>>>>>>>>> #copy-iface: eth1
>>>>>>>>>>
>>>>>>>>>> - interface: default
>>>>>>>>>> threads: auto
>>>>>>>>>> use-mmap: yes
>>>>>>>>>> tpacket-v3: yes
>>>>>>>>>>
>>>>>>>>>> max-pending-packets: 1024
>>>>>>>>>> runmode: workers
>>>>>>>>>> default-packet-size: 1522
>>>>>>>>>>
>>>>>>>>>> defrag:
>>>>>>>>>> memcap: 4gb
>>>>>>>>>> hash-size: 65536
>>>>>>>>>> trackers: 65535 # number of defragmented flows to
>>>>>>>>>> follow
>>>>>>>>>> max-frags: 65535 # number of fragments to keep
>>>>>>>>>> (higher than trackers)
>>>>>>>>>> prealloc: yes
>>>>>>>>>> timeout: 60
>>>>>>>>>>
>>>>>>>>>> flow:
>>>>>>>>>> memcap: 4gb
>>>>>>>>>> hash-size: 1048576
>>>>>>>>>> prealloc: 1048576
>>>>>>>>>> emergency-recovery: 30
>>>>>>>>>>
>>>>>>>>>> stream:
>>>>>>>>>> memcap: 4gb
>>>>>>>>>> checksum-validation: no
>>>>>>>>>> inline: no
>>>>>>>>>> bypass: yes
>>>>>>>>>> reassembly:
>>>>>>>>>> memcap: 8gb
>>>>>>>>>> depth: 1mb
>>>>>>>>>> toserver-chunk-size: 2560
>>>>>>>>>> toclient-chunk-size: 2560
>>>>>>>>>> randomize-chunk-size: yes
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> detect:
>>>>>>>>>> profile: custom
>>>>>>>>>> custom-values:
>>>>>>>>>> toclient-groups: 200
>>>>>>>>>> toserver-groups: 200
>>>>>>>>>> sgh-mpm-context: auto
>>>>>>>>>> inspection-recursion-limit: 3000
>>>>>>>>>>
>>>>>>>>>> mpm-algo: hs
>>>>>>>>>> spm-algo: hs
>>>>>>>>>>
>>>>>>>>>> threading:
>>>>>>>>>> set-cpu-affinity: yes
>>>>>>>>>> cpu-affinity:
>>>>>>>>>> - management-cpu-set:
>>>>>>>>>> cpu: [ "0-1" ]
>>>>>>>>>> mode: "balanced"
>>>>>>>>>> prio:
>>>>>>>>>> default: "medium"
>>>>>>>>>> - worker-cpu-set:
>>>>>>>>>> cpu: [ "2-15" ]
>>>>>>>>>> mode: "exclusive"
>>>>>>>>>> prio:
>>>>>>>>>> default: "high"
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Suricata IDS Users mailing list:
>>>>>>>>>> oisf-users at openinfosecfoundation.org
>>>>>>>>>> Site: http://suricata-ids.org | Support:
>>>>>>>>>> http://suricata-ids.org/support/
>>>>>>>>>> List:
>>>>>>>>>> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>>>>>>>>
>>>>>>>>>> Conference: https://suricon.net
>>>>>>>>>> Trainings: https://suricata-ids.org/training/
>>>>>>>>>
>>>>>>>>> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>
> Conference: https://suricon.net
> Trainings: https://suricata-ids.org/training/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20190824/8405c393/attachment-0001.html>
More information about the Oisf-users
mailing list