[Oisf-users] Suricata 2Gbit/s traffic drops on AWS

Shell_Xu xuh881026 at gmail.com
Fri Aug 23 16:48:07 UTC 2019


I am not sure if I use Placement Groups. If not used, can this problem
still be solved?

Tiago Faria <tiago.faria.backups at gmail.com> 于2019年8月23日周五 下午11:06写道:

> Are you using EC2 Placement Groups? Ideally you would use Cluster as much
> as possible exactly to prevent underlying hardware performance issues.
>
> It is also the recommended configuration for HPC applications, and
> Suricata would greatly benefit from that.
>
> On Fri, 23 Aug 2019 at 15:54, 徐慧 <xuh881026 at gmail.com> wrote:
>
>> hi, again:
>>     Yes, I am using Elastic Network Adapter (ENA)
>>     Since the EC2 instance is a shared underlying hardware, many network
>> interface hardware settings are not available.
>>     I don't know how to optimize Suricata on EC2, can you help me?
>>
>>      $ modinfo ena
>>
>>     filename:
>> /lib/modules/4.15.0-1044-aws/kernel/drivers/net/ethernet/amazon/ena/ena.ko
>>     version:        2.0.3K
>>     license:        GPL
>>     description:    Elastic Network Adapter (ENA)
>>     author:         Amazon.com, Inc. or its affiliates
>>     srcversion:     1980993534E135DFC7933C4
>>     alias:          pci:v00001D0Fd0000EC21sv*sd*bc*sc*i*
>>     alias:          pci:v00001D0Fd0000EC20sv*sd*bc*sc*i*
>>     alias:          pci:v00001D0Fd00001EC2sv*sd*bc*sc*i*
>>     alias:          pci:v00001D0Fd00000EC2sv*sd*bc*sc*i*
>>     depends:
>>     retpoline:      Y
>>     intree:         Y
>>     name:           ena
>>     vermagic:       4.15.0-1044-aws SMP mod_unload
>>     signat:         PKCS#7
>>     signer:
>>     sig_key:
>>     sig_hashalgo:   md4
>>     parm:           debug:Debug level (0=none,...,16=all) (int)
>>
>> Tiago Faria <tiago.faria.backups at gmail.com> 于2019年8月23日周五 下午6:51写道:
>>
>>> Hi,
>>>
>>> Based on the instance type and interface name, you're most likely using
>>> enhanced networking, but, to be on the safe side, can you confirm?
>>>
>>> $ modinfo ena
>>>
>>>
>>>
>>> On Fri, Aug 23, 2019 at 3:07 AM 徐慧 <xuh881026 at gmail.com> wrote:
>>>
>>>> hi, team:
>>>>      Since AWS traffic mirroring uses a VxLAN tunnel, I have to use the
>>>> 5.0dev version. i deployed Sruicata on AWS, but recently noticed that
>>>> 'capture. Kernel_drops' appears in stats.log when traffic reaches 2Gbit/s.
>>>> I tried rsync a large file, 'capture. Kernel_drops' appears in stats.log.
>>>> default loading ET rules.
>>>>      I hope anyone can help me, any advice is good! Guys, I need your
>>>> help very much.
>>>>
>>>>     # Client rsync files
>>>>     $ rsync -trovpgP xxx.tgz /usr/local/data/xxx.tgz
>>>>     sending incremental file list
>>>>     xxx.tgz
>>>>     3,361,243,136  51%  114.14MB/s    0:00:27
>>>>
>>>>     # Suricata Server:
>>>>     $ suricata --af-packet -c /etc/suricata/suricata.yaml
>>>>     [24073] 23/8/2019 -- 01:51:19 - (tm-threads.c:2145) <Notice>
>>>> (TmThreadWaitOnThreadInit) -- all 14 packet processing threads, 4
>>>> management threads initialized, engine started.
>>>>     [24073] 23/8/2019 -- 01:53:58 - (suricata.c:2851) <Notice>
>>>> (SuricataMainLoop) -- Signal Received.  Stopping engine.
>>>>     [24073] 23/8/2019 -- 01:54:01 - (util-device.c:317) <Notice>
>>>> (LiveDeviceListClean) -- Stats for 'ens5':  pkts: 11270384, drop: 2046365
>>>> (18.16%), invalid chksum: 0
>>>>
>>>>     According to the official documentation, I made some optimizations.
>>>>
>>>> https://suricata.readthedocs.io/en/latest/performance/packet-capture.html#rss
>>>>     But I can't set RSS queues to 1
>>>>     ethtool -L ens5 combined 1
>>>>     Cannot set device channel parameters: Operation not supported
>>>>
>>>>     Amazon EC2 C5
>>>>     EC2 Hardware:
>>>>     RAM: 32G
>>>>     CPU(single): 16 Core (Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz)
>>>>     NIC:
>>>>         ethtool -l ens5
>>>>         Channel parameters for ens5:
>>>>         Pre-set maximums:
>>>>         RX: 8
>>>>         TX: 8
>>>>         Other: 0
>>>>         Combined: 0
>>>>         Current hardware settings:
>>>>         RX: 8
>>>>         TX: 8
>>>>         Other: 0
>>>>         Combined: 0
>>>>
>>>>         ethtool -i ens5
>>>>         driver: ena
>>>>         version: 2.0.3K
>>>>         firmware-version:
>>>>         expansion-rom-version:
>>>>         bus-info: 0000:00:05.0
>>>>         supports-statistics: yes
>>>>         supports-test: no
>>>>         supports-eeprom-access: no
>>>>         supports-register-dump: no
>>>>         supports-priv-flags: no
>>>>
>>>>     Suricata Version: 5.0.0-dev (3a912446a 2019-07-22)
>>>>     Suricata Config:
>>>>         af-packet:
>>>>         - interface: ens5
>>>>             threads: 14
>>>>             cluster-id: 99
>>>>             cluster-type: cluster_flow
>>>>             defrag: yes    # Default AF_PACKET cluster type. AF_PACKET
>>>> can load balance per flow or per hash.
>>>>             use-mmap: yes
>>>>             mmap-locked: yes
>>>>             tpacket-v3: yes
>>>>             ring-size: 400000
>>>>             block-size: 393216
>>>>             #block-timeout: 10
>>>>             #use-emergency-flush: yes
>>>>             # buffer-size: 32768
>>>>             # disable-promisc: no
>>>>             #checksum-checks: kernel
>>>>             #bpf-filter: port 80 or udp
>>>>             #copy-mode: ips
>>>>             #copy-iface: eth1
>>>>
>>>>         - interface: default
>>>>             threads: auto
>>>>             use-mmap: yes
>>>>             tpacket-v3: yes
>>>>
>>>>         max-pending-packets: 1024
>>>>         runmode: workers
>>>>         default-packet-size: 1522
>>>>
>>>>         defrag:
>>>>             memcap: 4gb
>>>>             hash-size: 65536
>>>>             trackers: 65535 # number of defragmented flows to follow
>>>>             max-frags: 65535 # number of fragments to keep (higher than
>>>> trackers)
>>>>             prealloc: yes
>>>>             timeout: 60
>>>>
>>>>         flow:
>>>>             memcap: 4gb
>>>>             hash-size: 1048576
>>>>             prealloc: 1048576
>>>>             emergency-recovery: 30
>>>>
>>>>         stream:
>>>>         memcap: 4gb
>>>>         checksum-validation: no
>>>>         inline: no
>>>>         bypass: yes
>>>>         reassembly:
>>>>             memcap: 8gb
>>>>             depth: 1mb
>>>>             toserver-chunk-size: 2560
>>>>             toclient-chunk-size: 2560
>>>>             randomize-chunk-size: yes
>>>>
>>>>
>>>>         detect:
>>>>             profile: custom
>>>>             custom-values:
>>>>                 toclient-groups: 200
>>>>                 toserver-groups: 200
>>>>             sgh-mpm-context: auto
>>>>             inspection-recursion-limit: 3000
>>>>
>>>>         mpm-algo: hs
>>>>         spm-algo: hs
>>>>
>>>>         threading:
>>>>         set-cpu-affinity: yes
>>>>         cpu-affinity:
>>>>             - management-cpu-set:
>>>>                 cpu: [ "0-1" ]
>>>>                 mode: "balanced"
>>>>                 prio:
>>>>                 default: "medium"
>>>>             - worker-cpu-set:
>>>>                 cpu: [ "2-15" ]
>>>>                 mode: "exclusive"
>>>>                 prio:
>>>>                 default: "high"
>>>> _______________________________________________
>>>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>>>> Site: http://suricata-ids.org | Support:
>>>> http://suricata-ids.org/support/
>>>> List:
>>>> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>>
>>>> Conference: https://suricon.net
>>>> Trainings: https://suricata-ids.org/training/
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20190824/0c00a490/attachment.html>


More information about the Oisf-users mailing list