[Oisf-users] Suricata 2Gbit/s traffic drops on AWS
Shell_Xu
xuh881026 at gmail.com
Sat Aug 31 03:50:42 UTC 2019
I am sorry for late reply. I think I have solved the problem of packet
drop, but I am not using the 'Placement Groups' solution.
I found the reason because I enabled '--enable-profiling' when compiling. I
don't understand why this feature causes me to lose a lot of packets.I read
the documentation and didn't mention any impact on performance? anyone tell
me?
Tiago Faria <tiago.faria.backups at gmail.com> 于2019年8月26日周一 下午4:53写道：
> You don't have to migrate your entire fleet of EC2 instances. I suggest
> trying it with just 2. Either create new instances of create a snapshot,
> AMI, and launch the instance in the newly created PG.
>
> On Mon, Aug 26, 2019 at 9:18 AM Shell_Xu <xuh881026 at gmail.com> wrote:
>
>> We have too many EC2s on AWS, so I have no way to put them all in
>> Placement Groups. Do you have any other suggestions?Thanks again!
>>
>> Tiago Faria <tiago.faria.backups at gmail.com> 于2019年8月23日周五 下午11:06写道：
>>
>>> Are you using EC2 Placement Groups? Ideally you would use Cluster as
>>> much as possible exactly to prevent underlying hardware performance issues.
>>>
>>> It is also the recommended configuration for HPC applications, and
>>> Suricata would greatly benefit from that.
>>>
>>> On Fri, 23 Aug 2019 at 15:54, 徐慧 <xuh881026 at gmail.com> wrote:
>>>
>>>> hi, again:
>>>> Yes, I am using Elastic Network Adapter (ENA)
>>>> Since the EC2 instance is a shared underlying hardware, many
>>>> network interface hardware settings are not available.
>>>> I don't know how to optimize Suricata on EC2, can you help me?
>>>>
>>>> $ modinfo ena
>>>>
>>>> filename:
>>>> /lib/modules/4.15.0-1044-aws/kernel/drivers/net/ethernet/amazon/ena/ena.ko
>>>> version: 2.0.3K
>>>> license: GPL
>>>> description: Elastic Network Adapter (ENA)
>>>> author: Amazon.com, Inc. or its affiliates
>>>> srcversion: 1980993534E135DFC7933C4
>>>> alias: pci:v00001D0Fd0000EC21sv*sd*bc*sc*i*
>>>> alias: pci:v00001D0Fd0000EC20sv*sd*bc*sc*i*
>>>> alias: pci:v00001D0Fd00001EC2sv*sd*bc*sc*i*
>>>> alias: pci:v00001D0Fd00000EC2sv*sd*bc*sc*i*
>>>> depends:
>>>> retpoline: Y
>>>> intree: Y
>>>> name: ena
>>>> vermagic: 4.15.0-1044-aws SMP mod_unload
>>>> signat: PKCS#7
>>>> signer:
>>>> sig_key:
>>>> sig_hashalgo: md4
>>>> parm: debug:Debug level (0=none,...,16=all) (int)
>>>>
>>>> Tiago Faria <tiago.faria.backups at gmail.com> 于2019年8月23日周五 下午6:51写道：
>>>>
>>>>> Hi,
>>>>>
>>>>> Based on the instance type and interface name, you're most likely
>>>>> using enhanced networking, but, to be on the safe side, can you confirm?
>>>>>
>>>>> $ modinfo ena
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Aug 23, 2019 at 3:07 AM 徐慧 <xuh881026 at gmail.com> wrote:
>>>>>
>>>>>> hi, team:
>>>>>> Since AWS traffic mirroring uses a VxLAN tunnel, I have to use
>>>>>> the 5.0dev version. i deployed Sruicata on AWS, but recently noticed that
>>>>>> 'capture. Kernel_drops' appears in stats.log when traffic reaches 2Gbit/s.
>>>>>> I tried rsync a large file, 'capture. Kernel_drops' appears in stats.log.
>>>>>> default loading ET rules.
>>>>>> I hope anyone can help me, any advice is good! Guys, I need your
>>>>>> help very much.
>>>>>>
>>>>>> # Client rsync files
>>>>>> $ rsync -trovpgP xxx.tgz /usr/local/data/xxx.tgz
>>>>>> sending incremental file list
>>>>>> xxx.tgz
>>>>>> 3,361,243,136 51% 114.14MB/s 0:00:27
>>>>>>
>>>>>> # Suricata Server:
>>>>>> $ suricata --af-packet -c /etc/suricata/suricata.yaml
>>>>>> [24073] 23/8/2019 -- 01:51:19 - (tm-threads.c:2145) <Notice>
>>>>>> (TmThreadWaitOnThreadInit) -- all 14 packet processing threads, 4
>>>>>> management threads initialized, engine started.
>>>>>> [24073] 23/8/2019 -- 01:53:58 - (suricata.c:2851) <Notice>
>>>>>> (SuricataMainLoop) -- Signal Received. Stopping engine.
>>>>>> [24073] 23/8/2019 -- 01:54:01 - (util-device.c:317) <Notice>
>>>>>> (LiveDeviceListClean) -- Stats for 'ens5': pkts: 11270384, drop: 2046365
>>>>>> (18.16%), invalid chksum: 0
>>>>>>
>>>>>> According to the official documentation, I made some
>>>>>> optimizations.
>>>>>>
>>>>>> https://suricata.readthedocs.io/en/latest/performance/packet-capture.html#rss
>>>>>> But I can't set RSS queues to 1
>>>>>> ethtool -L ens5 combined 1
>>>>>> Cannot set device channel parameters: Operation not supported
>>>>>>
>>>>>> Amazon EC2 C5
>>>>>> EC2 Hardware:
>>>>>> RAM: 32G
>>>>>> CPU(single): 16 Core (Intel(R) Xeon(R) Platinum 8124M CPU @
>>>>>> 3.00GHz)
>>>>>> NIC:
>>>>>> ethtool -l ens5
>>>>>> Channel parameters for ens5:
>>>>>> Pre-set maximums:
>>>>>> RX: 8
>>>>>> TX: 8
>>>>>> Other: 0
>>>>>> Combined: 0
>>>>>> Current hardware settings:
>>>>>> RX: 8
>>>>>> TX: 8
>>>>>> Other: 0
>>>>>> Combined: 0
>>>>>>
>>>>>> ethtool -i ens5
>>>>>> driver: ena
>>>>>> version: 2.0.3K
>>>>>> firmware-version:
>>>>>> expansion-rom-version:
>>>>>> bus-info: 0000:00:05.0
>>>>>> supports-statistics: yes
>>>>>> supports-test: no
>>>>>> supports-eeprom-access: no
>>>>>> supports-register-dump: no
>>>>>> supports-priv-flags: no
>>>>>>
>>>>>> Suricata Version: 5.0.0-dev (3a912446a 2019-07-22)
>>>>>> Suricata Config:
>>>>>> af-packet:
>>>>>> - interface: ens5
>>>>>> threads: 14
>>>>>> cluster-id: 99
>>>>>> cluster-type: cluster_flow
>>>>>> defrag: yes # Default AF_PACKET cluster type.
>>>>>> AF_PACKET can load balance per flow or per hash.
>>>>>> use-mmap: yes
>>>>>> mmap-locked: yes
>>>>>> tpacket-v3: yes
>>>>>> ring-size: 400000
>>>>>> block-size: 393216
>>>>>> #block-timeout: 10
>>>>>> #use-emergency-flush: yes
>>>>>> # buffer-size: 32768
>>>>>> # disable-promisc: no
>>>>>> #checksum-checks: kernel
>>>>>> #bpf-filter: port 80 or udp
>>>>>> #copy-mode: ips
>>>>>> #copy-iface: eth1
>>>>>>
>>>>>> - interface: default
>>>>>> threads: auto
>>>>>> use-mmap: yes
>>>>>> tpacket-v3: yes
>>>>>>
>>>>>> max-pending-packets: 1024
>>>>>> runmode: workers
>>>>>> default-packet-size: 1522
>>>>>>
>>>>>> defrag:
>>>>>> memcap: 4gb
>>>>>> hash-size: 65536
>>>>>> trackers: 65535 # number of defragmented flows to follow
>>>>>> max-frags: 65535 # number of fragments to keep (higher
>>>>>> than trackers)
>>>>>> prealloc: yes
>>>>>> timeout: 60
>>>>>>
>>>>>> flow:
>>>>>> memcap: 4gb
>>>>>> hash-size: 1048576
>>>>>> prealloc: 1048576
>>>>>> emergency-recovery: 30
>>>>>>
>>>>>> stream:
>>>>>> memcap: 4gb
>>>>>> checksum-validation: no
>>>>>> inline: no
>>>>>> bypass: yes
>>>>>> reassembly:
>>>>>> memcap: 8gb
>>>>>> depth: 1mb
>>>>>> toserver-chunk-size: 2560
>>>>>> toclient-chunk-size: 2560
>>>>>> randomize-chunk-size: yes
>>>>>>
>>>>>>
>>>>>> detect:
>>>>>> profile: custom
>>>>>> custom-values:
>>>>>> toclient-groups: 200
>>>>>> toserver-groups: 200
>>>>>> sgh-mpm-context: auto
>>>>>> inspection-recursion-limit: 3000
>>>>>>
>>>>>> mpm-algo: hs
>>>>>> spm-algo: hs
>>>>>>
>>>>>> threading:
>>>>>> set-cpu-affinity: yes
>>>>>> cpu-affinity:
>>>>>> - management-cpu-set:
>>>>>> cpu: [ "0-1" ]
>>>>>> mode: "balanced"
>>>>>> prio:
>>>>>> default: "medium"
>>>>>> - worker-cpu-set:
>>>>>> cpu: [ "2-15" ]
>>>>>> mode: "exclusive"
>>>>>> prio:
>>>>>> default: "high"
>>>>>> _______________________________________________
>>>>>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>>>>>> Site: http://suricata-ids.org | Support:
>>>>>> http://suricata-ids.org/support/
>>>>>> List:
>>>>>> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>>>>
>>>>>> Conference: https://suricon.net
>>>>>> Trainings: https://suricata-ids.org/training/
>>>>>
>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20190831/a5ae280f/attachment-0001.html>
More information about the Oisf-users
mailing list