[Oisf-users] Suricata Myricom and 10Gbit
Erich Lerch
erich.lerch at gmail.com
Tue Apr 1 07:59:48 UTC 2014
I just asked Myricom about it the RING SIZE variables. That's what they say:
# ---
For Sniffer10G v2 releases, you should generally keep the ratio of the
sizes of the data and descriptor rings the same no matter how many
rings you intend to use. For example, if you double the size of the
data ring from the default value, you should double the size of the
descriptor ring from its default value.
By default, SNF_DATARING_SIZE = 256MB, and SNF_DESCRING_SIZE = 64MB.
# ---
So I'm leaving SNF_DATARING_SIZE at 32GB and will set
SNF_DESCRING_SIZE to 8GB. Indeed, SNF_DATARING_SIZE is the memory
allocated for all rings, just once.
As for the *-groups settings, I'm not sure about calculating memory
requirements. I have 132 GB of RAM, and found these settings on the
web...
--
erich
2014-03-31 17:04 GMT+02:00 Michał Purzyński <michalpurzynski1 at gmail.com>:
> Thanks, I'll try the settings in a few minutes. Related question - I
> remember trying to increase the toclient-*-groups toserver-*-groups and
> Suricata ate 64GB of memory dying in the process before it was able to
> start. How do I calculate the memory needed for such a high settings?
>
> Also, I will need SNF_DATARING_SIZE x number of workers of memory? Asking
> because you have it at 32GB, so does it mean it's actually allocated only
> once?
>
>
> On Mon, Mar 31, 2014 at 4:52 PM, Erich Lerch <erich.lerch at gmail.com> wrote:
>>
>> Michał,
>>
>> We have a similar setup, also with the Myricom 10gb interface.
>>
>> The following values give us fairly good results, although our traffic
>> does not exceed 2Gbit at the moment.
>> Performance also depends on the ruleset.
>> If you have profiling compiled in, try without.
>>
>> SNF settings:
>> SNF_NUM_RINGS=16 SNF_DATARING_SIZE=34359738368
>> SNF_DESCRING_SIZE=2147483648
>>
>> Suricata settings:
>>
>> ...
>> max-pending-packets: 2048
>> ...
>> # very important:
>> detect-engine:
>> - profile: custom
>> - custom-values:
>> toclient-src-groups: 200
>> toclient-dst-groups: 200
>> toclient-sp-groups: 200
>> toclient-dp-groups: 300
>> toserver-src-groups: 200
>> toserver-dst-groups: 400
>> toserver-sp-groups: 200
>> toserver-dp-groups: 200
>> - sgh-mpm-context: single
>> - inspection-recursion-limit: 3000
>> - rule-reload: true
>> ...
>> threading:
>> set-cpu-affinity: yes
>> [definition of cpu sets]
>> ...
>> defrag:
>> memcap: 512mb
>> hash-size: 65536
>> trackers: 65535
>> max-frags: 65535
>> prealloc: yes
>> timeout: 10
>> ...
>> flow:
>> memcap: 256mb
>> hash-size: 262144
>> prealloc: 300000
>> emergency-recovery: 30
>> ...
>> flow-timeouts:
>> [a lot more aggressive than the default!!!]
>> ...
>> stream:
>> memcap: 12gb
>> checksum-validation: no
>> inline: no
>> prealloc-sessions: 10000000
>> reassembly:
>> memcap: 16gb
>> depth: 6mb # reassemble 1mb into a stream
>> toserver-chunk-size: 2560
>> toclient-chunk-size: 2560
>> randomize-chunk-size: yes
>> ...
>>
>> pcap:
>> - interface: snf0
>> threads: 16
>> buffer-size: 512mb
>> checksum-checks: no
>> promisc: no
>> ...
>>
>> Also set the app-layer values much higher than the defaults.
>>
>> cheers,
>> erich
>>
>>
>> 2014-03-31 13:27 GMT+02:00 Michał Purzyński <michalpurzynski1 at gmail.com>:
>> > Hello.
>> >
>> > I'm trying to tune Suricata to handle up to 10Gbit/sec of traffic
>> > (that's a
>> > peak, jumps like crazy from 2.5 - 4.5 - 6 and up). So far my results
>> > were
>> > quite bad, so I'm seeking help - must be missing something obvious here
>> > judging by the numbers of articles where everyone seems to use Suricata
>> > on
>> > 10Gbit traffic.
>> >
>> > Server:
>> >
>> > 2 x Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz (16 physical cores)
>> > 64GB RAM
>> >
>> > NIC - Myricom 10Gb 10G-PCIE-8B-S with the Sniffer software loaded and
>> > activated
>> >
>> > Software:
>> >
>> > This is Suricata version 2.0rc2 RELEASE
>> >
>> > Command line:
>> >
>> > SNF_NUM_RINGS=16 SNF_FLAGS=0x1 SNF_DESCRING_SIZE=1073741824
>> > SNF_DATARING_SIZE=1073741824 SNF_DEBUG_MASK=0x3 suricata -c
>> > /etc/nsm/nsm11-eth4/suricata.yaml -i eth4 --runmode=workers
>> >
>> > (16 threads, 1GB for each buffer)
>> >
>> > The Myricom debug output seems fine.
>> >
>> > Config file - pretty standard, most important things:
>> >
>> > max-pending-packets: 5000
>> > runmode: workers
>> >
>> > detect-engine:
>> > - profile: medium
>> >
>> > Did not touch parameters here.
>> >
>> > set-cpu-affinity: no
>> >
>> > Also default settings here.
>> >
>> > detect-thread-ratio: 1.5
>> >
>> > (should not it be 1.0?)
>> >
>> > defrag:
>> > memcap: 512mb
>> > trackers: 65535 # number of defragmented flows to follow
>> > max-frags: 65535 # number of fragments to keep (higher than trackers)
>> > prealloc: yes
>> > timeout: 60
>> >
>> > flow:
>> > memcap: 32mb
>> > hash-size: 65536
>> > prealloc: 10000
>> > emergency-recovery: 30
>> >
>> > stream:
>> > memcap: 16gb
>> > max-sessions: 20000000
>> > prealloc-sessions: 10000000
>> > checksum-validation: yes # reject wrong csums
>> > inline: no # no inline mode
>> > reassembly:
>> > memcap: 14gb
>> > depth: 6mb # reassemble 1mb into a stream
>> > toserver-chunk-size: 2560
>> > toclient-chunk-size: 2560
>> >
>> > pcap:
>> > - interface: eth4
>> > threads: 16
>> > buffer-size: 512kb
>> > checksum-checks: no
>> >
>> > The myricom tools show a high packet loss
>> >
>> > SNF recv pkts: 634485790
>> > SNF drop ring full: 137774061
>> > Interrupts: 12053363
>> > Net bad PHY/CRC32 drop: 32092
>> > Net overflow drop: 219656
>> >
>> > Also note that it reports quite a few interrupts, which there should be
>> > almost none.
>> >
>> > What is the direction I should go here? I know that tuning a high
>> > capacity
>> > Suricata isn't exactly a single afternoon task, but I need to advise
>> > what to
>> > do now, how to proceed, etc.
>> >
>> > Looking for clues.
>> >
>> > --
>> > Michał Purzyński
>> >
>> > _______________________________________________
>> > Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>> > Site: http://suricata-ids.org | Support:
>> > http://suricata-ids.org/support/
>> > List:
>> > https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>> > OISF: http://www.openinfosecfoundation.org/
>> _______________________________________________
>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
>> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>> OISF: http://www.openinfosecfoundation.org/
>
>
>
>
> --
> Michał Purzyński
More information about the Oisf-users
mailing list