[Oisf-users] Suricata 2Gbit/s traffic drops on AWS
Tiago Faria
tiago.faria.backups at gmail.com
Fri Aug 23 17:39:03 UTC 2019
Hi,
It can be fixed, yes, but it requires deployment of the EC2 instances (or
re-deployment). What I recommend is the creation of a Placement Group of
type Cluster and deploy the EC2 instances inside that Placement Group.
On Fri, Aug 23, 2019 at 5:48 PM Shell_Xu <xuh881026 at gmail.com> wrote:
> I am not sure if I use Placement Groups. If not used, can this problem
> still be solved?
>
> Tiago Faria <tiago.faria.backups at gmail.com> 于2019年8月23日周五 下午11:06写道：
>
>> Are you using EC2 Placement Groups? Ideally you would use Cluster as much
>> as possible exactly to prevent underlying hardware performance issues.
>>
>> It is also the recommended configuration for HPC applications, and
>> Suricata would greatly benefit from that.
>>
>> On Fri, 23 Aug 2019 at 15:54, 徐慧 <xuh881026 at gmail.com> wrote:
>>
>>> hi, again:
>>> Yes, I am using Elastic Network Adapter (ENA)
>>> Since the EC2 instance is a shared underlying hardware, many network
>>> interface hardware settings are not available.
>>> I don't know how to optimize Suricata on EC2, can you help me?
>>>
>>> $ modinfo ena
>>>
>>> filename:
>>> /lib/modules/4.15.0-1044-aws/kernel/drivers/net/ethernet/amazon/ena/ena.ko
>>> version: 2.0.3K
>>> license: GPL
>>> description: Elastic Network Adapter (ENA)
>>> author: Amazon.com, Inc. or its affiliates
>>> srcversion: 1980993534E135DFC7933C4
>>> alias: pci:v00001D0Fd0000EC21sv*sd*bc*sc*i*
>>> alias: pci:v00001D0Fd0000EC20sv*sd*bc*sc*i*
>>> alias: pci:v00001D0Fd00001EC2sv*sd*bc*sc*i*
>>> alias: pci:v00001D0Fd00000EC2sv*sd*bc*sc*i*
>>> depends:
>>> retpoline: Y
>>> intree: Y
>>> name: ena
>>> vermagic: 4.15.0-1044-aws SMP mod_unload
>>> signat: PKCS#7
>>> signer:
>>> sig_key:
>>> sig_hashalgo: md4
>>> parm: debug:Debug level (0=none,...,16=all) (int)
>>>
>>> Tiago Faria <tiago.faria.backups at gmail.com> 于2019年8月23日周五 下午6:51写道：
>>>
>>>> Hi,
>>>>
>>>> Based on the instance type and interface name, you're most likely using
>>>> enhanced networking, but, to be on the safe side, can you confirm?
>>>>
>>>> $ modinfo ena
>>>>
>>>>
>>>>
>>>> On Fri, Aug 23, 2019 at 3:07 AM 徐慧 <xuh881026 at gmail.com> wrote:
>>>>
>>>>> hi, team:
>>>>> Since AWS traffic mirroring uses a VxLAN tunnel, I have to use
>>>>> the 5.0dev version. i deployed Sruicata on AWS, but recently noticed that
>>>>> 'capture. Kernel_drops' appears in stats.log when traffic reaches 2Gbit/s.
>>>>> I tried rsync a large file, 'capture. Kernel_drops' appears in stats.log.
>>>>> default loading ET rules.
>>>>> I hope anyone can help me, any advice is good! Guys, I need your
>>>>> help very much.
>>>>>
>>>>> # Client rsync files
>>>>> $ rsync -trovpgP xxx.tgz /usr/local/data/xxx.tgz
>>>>> sending incremental file list
>>>>> xxx.tgz
>>>>> 3,361,243,136 51% 114.14MB/s 0:00:27
>>>>>
>>>>> # Suricata Server:
>>>>> $ suricata --af-packet -c /etc/suricata/suricata.yaml
>>>>> [24073] 23/8/2019 -- 01:51:19 - (tm-threads.c:2145) <Notice>
>>>>> (TmThreadWaitOnThreadInit) -- all 14 packet processing threads, 4
>>>>> management threads initialized, engine started.
>>>>> [24073] 23/8/2019 -- 01:53:58 - (suricata.c:2851) <Notice>
>>>>> (SuricataMainLoop) -- Signal Received. Stopping engine.
>>>>> [24073] 23/8/2019 -- 01:54:01 - (util-device.c:317) <Notice>
>>>>> (LiveDeviceListClean) -- Stats for 'ens5': pkts: 11270384, drop: 2046365
>>>>> (18.16%), invalid chksum: 0
>>>>>
>>>>> According to the official documentation, I made some optimizations.
>>>>>
>>>>> https://suricata.readthedocs.io/en/latest/performance/packet-capture.html#rss
>>>>> But I can't set RSS queues to 1
>>>>> ethtool -L ens5 combined 1
>>>>> Cannot set device channel parameters: Operation not supported
>>>>>
>>>>> Amazon EC2 C5
>>>>> EC2 Hardware:
>>>>> RAM: 32G
>>>>> CPU(single): 16 Core (Intel(R) Xeon(R) Platinum 8124M CPU @
>>>>> 3.00GHz)
>>>>> NIC:
>>>>> ethtool -l ens5
>>>>> Channel parameters for ens5:
>>>>> Pre-set maximums:
>>>>> RX: 8
>>>>> TX: 8
>>>>> Other: 0
>>>>> Combined: 0
>>>>> Current hardware settings:
>>>>> RX: 8
>>>>> TX: 8
>>>>> Other: 0
>>>>> Combined: 0
>>>>>
>>>>> ethtool -i ens5
>>>>> driver: ena
>>>>> version: 2.0.3K
>>>>> firmware-version:
>>>>> expansion-rom-version:
>>>>> bus-info: 0000:00:05.0
>>>>> supports-statistics: yes
>>>>> supports-test: no
>>>>> supports-eeprom-access: no
>>>>> supports-register-dump: no
>>>>> supports-priv-flags: no
>>>>>
>>>>> Suricata Version: 5.0.0-dev (3a912446a 2019-07-22)
>>>>> Suricata Config:
>>>>> af-packet:
>>>>> - interface: ens5
>>>>> threads: 14
>>>>> cluster-id: 99
>>>>> cluster-type: cluster_flow
>>>>> defrag: yes # Default AF_PACKET cluster type. AF_PACKET
>>>>> can load balance per flow or per hash.
>>>>> use-mmap: yes
>>>>> mmap-locked: yes
>>>>> tpacket-v3: yes
>>>>> ring-size: 400000
>>>>> block-size: 393216
>>>>> #block-timeout: 10
>>>>> #use-emergency-flush: yes
>>>>> # buffer-size: 32768
>>>>> # disable-promisc: no
>>>>> #checksum-checks: kernel
>>>>> #bpf-filter: port 80 or udp
>>>>> #copy-mode: ips
>>>>> #copy-iface: eth1
>>>>>
>>>>> - interface: default
>>>>> threads: auto
>>>>> use-mmap: yes
>>>>> tpacket-v3: yes
>>>>>
>>>>> max-pending-packets: 1024
>>>>> runmode: workers
>>>>> default-packet-size: 1522
>>>>>
>>>>> defrag:
>>>>> memcap: 4gb
>>>>> hash-size: 65536
>>>>> trackers: 65535 # number of defragmented flows to follow
>>>>> max-frags: 65535 # number of fragments to keep (higher
>>>>> than trackers)
>>>>> prealloc: yes
>>>>> timeout: 60
>>>>>
>>>>> flow:
>>>>> memcap: 4gb
>>>>> hash-size: 1048576
>>>>> prealloc: 1048576
>>>>> emergency-recovery: 30
>>>>>
>>>>> stream:
>>>>> memcap: 4gb
>>>>> checksum-validation: no
>>>>> inline: no
>>>>> bypass: yes
>>>>> reassembly:
>>>>> memcap: 8gb
>>>>> depth: 1mb
>>>>> toserver-chunk-size: 2560
>>>>> toclient-chunk-size: 2560
>>>>> randomize-chunk-size: yes
>>>>>
>>>>>
>>>>> detect:
>>>>> profile: custom
>>>>> custom-values:
>>>>> toclient-groups: 200
>>>>> toserver-groups: 200
>>>>> sgh-mpm-context: auto
>>>>> inspection-recursion-limit: 3000
>>>>>
>>>>> mpm-algo: hs
>>>>> spm-algo: hs
>>>>>
>>>>> threading:
>>>>> set-cpu-affinity: yes
>>>>> cpu-affinity:
>>>>> - management-cpu-set:
>>>>> cpu: [ "0-1" ]
>>>>> mode: "balanced"
>>>>> prio:
>>>>> default: "medium"
>>>>> - worker-cpu-set:
>>>>> cpu: [ "2-15" ]
>>>>> mode: "exclusive"
>>>>> prio:
>>>>> default: "high"
>>>>> _______________________________________________
>>>>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>>>>> Site: http://suricata-ids.org | Support:
>>>>> http://suricata-ids.org/support/
>>>>> List:
>>>>> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>>>
>>>>> Conference: https://suricon.net
>>>>> Trainings: https://suricata-ids.org/training/
>>>>
>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20190823/323dd1e1/attachment-0001.html>
More information about the Oisf-users
mailing list