[Oisf-users] Suricata 2Gbit/s traffic drops on AWS
Tiago Faria
tiago.faria.backups at gmail.com
Sat Aug 24 07:49:27 UTC 2019
You can have mirror sessions as you want, including between AWS accounts.
To get the best performance, however, placing them in the same placement
group will help substantially.
I’d first check if this helps in the problem you’re having though.
On Sat, 24 Aug 2019 at 01:46, Shell_Xu <xuh881026 at gmail.com> wrote:
> HI:
> Thank you for your help!
> 'What I recommend is the creation of a Placement Group of type Cluster
> and deploy the EC2 instances inside that Placement Group. '
> Does this mean that servers I monitor need to be deployed in the
> Placement Group?
> e.g:
> Sruicata、Web Server、DB Server、Redis Cluster...
>
> Tiago Faria <tiago.faria.backups at gmail.com> 于2019年8月24日周六 上午1:38写道：
>
>> Hi,
>>
>> It can be fixed, yes, but it requires deployment of the EC2 instances (or
>> re-deployment). What I recommend is the creation of a Placement Group of
>> type Cluster and deploy the EC2 instances inside that Placement Group.
>>
>> On Fri, Aug 23, 2019 at 5:48 PM Shell_Xu <xuh881026 at gmail.com> wrote:
>>
>>> I am not sure if I use Placement Groups. If not used, can this problem
>>> still be solved?
>>>
>>> Tiago Faria <tiago.faria.backups at gmail.com> 于2019年8月23日周五 下午11:06写道：
>>>
>>>> Are you using EC2 Placement Groups? Ideally you would use Cluster as
>>>> much as possible exactly to prevent underlying hardware performance issues.
>>>>
>>>> It is also the recommended configuration for HPC applications, and
>>>> Suricata would greatly benefit from that.
>>>>
>>>> On Fri, 23 Aug 2019 at 15:54, 徐慧 <xuh881026 at gmail.com> wrote:
>>>>
>>>>> hi, again:
>>>>> Yes, I am using Elastic Network Adapter (ENA)
>>>>> Since the EC2 instance is a shared underlying hardware, many
>>>>> network interface hardware settings are not available.
>>>>> I don't know how to optimize Suricata on EC2, can you help me?
>>>>>
>>>>> $ modinfo ena
>>>>>
>>>>> filename:
>>>>> /lib/modules/4.15.0-1044-aws/kernel/drivers/net/ethernet/amazon/ena/ena.ko
>>>>> version: 2.0.3K
>>>>> license: GPL
>>>>> description: Elastic Network Adapter (ENA)
>>>>> author: Amazon.com, Inc. or its affiliates
>>>>> srcversion: 1980993534E135DFC7933C4
>>>>> alias: pci:v00001D0Fd0000EC21sv*sd*bc*sc*i*
>>>>> alias: pci:v00001D0Fd0000EC20sv*sd*bc*sc*i*
>>>>> alias: pci:v00001D0Fd00001EC2sv*sd*bc*sc*i*
>>>>> alias: pci:v00001D0Fd00000EC2sv*sd*bc*sc*i*
>>>>> depends:
>>>>> retpoline: Y
>>>>> intree: Y
>>>>> name: ena
>>>>> vermagic: 4.15.0-1044-aws SMP mod_unload
>>>>> signat: PKCS#7
>>>>> signer:
>>>>> sig_key:
>>>>> sig_hashalgo: md4
>>>>> parm: debug:Debug level (0=none,...,16=all) (int)
>>>>>
>>>>> Tiago Faria <tiago.faria.backups at gmail.com> 于2019年8月23日周五 下午6:51写道：
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Based on the instance type and interface name, you're most likely
>>>>>> using enhanced networking, but, to be on the safe side, can you confirm?
>>>>>>
>>>>>> $ modinfo ena
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Aug 23, 2019 at 3:07 AM 徐慧 <xuh881026 at gmail.com> wrote:
>>>>>>
>>>>>>> hi, team:
>>>>>>> Since AWS traffic mirroring uses a VxLAN tunnel, I have to use
>>>>>>> the 5.0dev version. i deployed Sruicata on AWS, but recently noticed that
>>>>>>> 'capture. Kernel_drops' appears in stats.log when traffic reaches 2Gbit/s.
>>>>>>> I tried rsync a large file, 'capture. Kernel_drops' appears in stats.log.
>>>>>>> default loading ET rules.
>>>>>>> I hope anyone can help me, any advice is good! Guys, I need
>>>>>>> your help very much.
>>>>>>>
>>>>>>> # Client rsync files
>>>>>>> $ rsync -trovpgP xxx.tgz /usr/local/data/xxx.tgz
>>>>>>> sending incremental file list
>>>>>>> xxx.tgz
>>>>>>> 3,361,243,136 51% 114.14MB/s 0:00:27
>>>>>>>
>>>>>>> # Suricata Server:
>>>>>>> $ suricata --af-packet -c /etc/suricata/suricata.yaml
>>>>>>> [24073] 23/8/2019 -- 01:51:19 - (tm-threads.c:2145) <Notice>
>>>>>>> (TmThreadWaitOnThreadInit) -- all 14 packet processing threads, 4
>>>>>>> management threads initialized, engine started.
>>>>>>> [24073] 23/8/2019 -- 01:53:58 - (suricata.c:2851) <Notice>
>>>>>>> (SuricataMainLoop) -- Signal Received. Stopping engine.
>>>>>>> [24073] 23/8/2019 -- 01:54:01 - (util-device.c:317) <Notice>
>>>>>>> (LiveDeviceListClean) -- Stats for 'ens5': pkts: 11270384, drop: 2046365
>>>>>>> (18.16%), invalid chksum: 0
>>>>>>>
>>>>>>> According to the official documentation, I made some
>>>>>>> optimizations.
>>>>>>>
>>>>>>> https://suricata.readthedocs.io/en/latest/performance/packet-capture.html#rss
>>>>>>> But I can't set RSS queues to 1
>>>>>>> ethtool -L ens5 combined 1
>>>>>>> Cannot set device channel parameters: Operation not supported
>>>>>>>
>>>>>>> Amazon EC2 C5
>>>>>>> EC2 Hardware:
>>>>>>> RAM: 32G
>>>>>>> CPU(single): 16 Core (Intel(R) Xeon(R) Platinum 8124M CPU @
>>>>>>> 3.00GHz)
>>>>>>> NIC:
>>>>>>> ethtool -l ens5
>>>>>>> Channel parameters for ens5:
>>>>>>> Pre-set maximums:
>>>>>>> RX: 8
>>>>>>> TX: 8
>>>>>>> Other: 0
>>>>>>> Combined: 0
>>>>>>> Current hardware settings:
>>>>>>> RX: 8
>>>>>>> TX: 8
>>>>>>> Other: 0
>>>>>>> Combined: 0
>>>>>>>
>>>>>>> ethtool -i ens5
>>>>>>> driver: ena
>>>>>>> version: 2.0.3K
>>>>>>> firmware-version:
>>>>>>> expansion-rom-version:
>>>>>>> bus-info: 0000:00:05.0
>>>>>>> supports-statistics: yes
>>>>>>> supports-test: no
>>>>>>> supports-eeprom-access: no
>>>>>>> supports-register-dump: no
>>>>>>> supports-priv-flags: no
>>>>>>>
>>>>>>> Suricata Version: 5.0.0-dev (3a912446a 2019-07-22)
>>>>>>> Suricata Config:
>>>>>>> af-packet:
>>>>>>> - interface: ens5
>>>>>>> threads: 14
>>>>>>> cluster-id: 99
>>>>>>> cluster-type: cluster_flow
>>>>>>> defrag: yes # Default AF_PACKET cluster type.
>>>>>>> AF_PACKET can load balance per flow or per hash.
>>>>>>> use-mmap: yes
>>>>>>> mmap-locked: yes
>>>>>>> tpacket-v3: yes
>>>>>>> ring-size: 400000
>>>>>>> block-size: 393216
>>>>>>> #block-timeout: 10
>>>>>>> #use-emergency-flush: yes
>>>>>>> # buffer-size: 32768
>>>>>>> # disable-promisc: no
>>>>>>> #checksum-checks: kernel
>>>>>>> #bpf-filter: port 80 or udp
>>>>>>> #copy-mode: ips
>>>>>>> #copy-iface: eth1
>>>>>>>
>>>>>>> - interface: default
>>>>>>> threads: auto
>>>>>>> use-mmap: yes
>>>>>>> tpacket-v3: yes
>>>>>>>
>>>>>>> max-pending-packets: 1024
>>>>>>> runmode: workers
>>>>>>> default-packet-size: 1522
>>>>>>>
>>>>>>> defrag:
>>>>>>> memcap: 4gb
>>>>>>> hash-size: 65536
>>>>>>> trackers: 65535 # number of defragmented flows to follow
>>>>>>> max-frags: 65535 # number of fragments to keep (higher
>>>>>>> than trackers)
>>>>>>> prealloc: yes
>>>>>>> timeout: 60
>>>>>>>
>>>>>>> flow:
>>>>>>> memcap: 4gb
>>>>>>> hash-size: 1048576
>>>>>>> prealloc: 1048576
>>>>>>> emergency-recovery: 30
>>>>>>>
>>>>>>> stream:
>>>>>>> memcap: 4gb
>>>>>>> checksum-validation: no
>>>>>>> inline: no
>>>>>>> bypass: yes
>>>>>>> reassembly:
>>>>>>> memcap: 8gb
>>>>>>> depth: 1mb
>>>>>>> toserver-chunk-size: 2560
>>>>>>> toclient-chunk-size: 2560
>>>>>>> randomize-chunk-size: yes
>>>>>>>
>>>>>>>
>>>>>>> detect:
>>>>>>> profile: custom
>>>>>>> custom-values:
>>>>>>> toclient-groups: 200
>>>>>>> toserver-groups: 200
>>>>>>> sgh-mpm-context: auto
>>>>>>> inspection-recursion-limit: 3000
>>>>>>>
>>>>>>> mpm-algo: hs
>>>>>>> spm-algo: hs
>>>>>>>
>>>>>>> threading:
>>>>>>> set-cpu-affinity: yes
>>>>>>> cpu-affinity:
>>>>>>> - management-cpu-set:
>>>>>>> cpu: [ "0-1" ]
>>>>>>> mode: "balanced"
>>>>>>> prio:
>>>>>>> default: "medium"
>>>>>>> - worker-cpu-set:
>>>>>>> cpu: [ "2-15" ]
>>>>>>> mode: "exclusive"
>>>>>>> prio:
>>>>>>> default: "high"
>>>>>>> _______________________________________________
>>>>>>> Suricata IDS Users mailing list:
>>>>>>> oisf-users at openinfosecfoundation.org
>>>>>>> Site: http://suricata-ids.org | Support:
>>>>>>> http://suricata-ids.org/support/
>>>>>>> List:
>>>>>>> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>>>>>
>>>>>>> Conference: https://suricon.net
>>>>>>> Trainings: https://suricata-ids.org/training/
>>>>>>
>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20190824/7819e339/attachment-0001.html>
More information about the Oisf-users
mailing list