[Oisf-users] suricata at 40Gb
Brandon Lattin
latt0050 at umn.edu
Tue Feb 10 16:39:52 UTC 2015
Erich,
After making some significant changes, we're at 0% packetloss for the last
12 hours. We'll hit network peak at around 1PM, so I'll update if that
changes.
I've swapped to pinning to 18 cores as well as 2-tuple load balancing
across 3 boxes. Additionally, I've increased max-pending-packets to 60k
from ~16k and swapped to your values for the ring buffers:
SNF_DATARING_SIZE=34359738368 SNF_DESCRING_SIZE=8589934592
Here's the relevant config:
set-cpu-affinity: yes
cpu-affinity:
- management-cpu-set:
cpu: [ 0-1 ] # include only these cpus in affinity settings
mode: "balanced"
prio:
default: "low"
- receive-cpu-set:
cpu: [ 0-1 ] # include only these cpus in affinity settings
- decode-cpu-set:
cpu: [ 0-1 ]
mode: "balanced"
- stream-cpu-set:
cpu: [ 0-1 ]
- detect-cpu-set:
cpu: [ 2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36 ]
mode: "exclusive" # run detect threads in these cpus
prio:
default: "high"
- verdict-cpu-set:
cpu: [ 0-1 ]
prio:
default: "high"
- reject-cpu-set:
cpu: [ 0-1 ]
prio:
default: "low"
- output-cpu-set:
cpu: [ 0-1 ]
prio:
default: "medium"
Thanks for providing some insight into improvements!
On Mon, Feb 9, 2015 at 1:28 PM, Brandon Lattin <latt0050 at umn.edu> wrote:
> Erich,
>
> Your response got me thinking. The 32 ring buffers was a remnant from an
> old config where I was seeing significantly better performance _not_
> pinning directly to cores on an older test box (with a much lower clock
> speed). I'm about to reconfigure for 16 threads on our dev box. I'll let
> you know how it goes.
>
> Here are my ring buffer settings: SNF_DATARING_SIZE=268435456
> SNF_DESCRING_SIZE=67108864
> And yours: SNF_DATARING_SIZE=34359738368 SNF_DESCRING_SIZE=8589934592
>
> I do notice that if I drop mine at all, packetloss starts to creep. The
> boxes have RAM to spare, so I'll crank up mine to match your setting of
> 32GB after testing out pinning to 16 physical cores.
>
> Right now, we see spikes of high packetloss on one of the boxes. The
> others hover under 1%. This is probably due to the type of traffic we have
> on that interface. Whitelisting a few IPs, such as the Debian mirror we
> run, should clear up the burst packetloss.
>
> [image: Inline image 1]
>
> On Fri, Feb 6, 2015 at 3:59 PM, Erich Lerch <erich.lerch at gmail.com> wrote:
>
>> Very intersting, Brandon, thanks for sharing!
>>
>> We have a somewhat similar setup (less traffic though, only one suricata
>> box), with Myricom Sniffer10G NICs.
>>
>> May I ask how you configured the ring buffers for suricata?
>>
>> We use 16 rings and set
>> SNF_DATARING_SIZE=34359738368
>> SNF_DESCRING_SIZE=8589934592
>>
>> This is a lot, but we kept having higher packet loss with lower settings.
>> That's why I'd be interested in others' experiences with ring number and
>> ring mem settings.
>>
>> Do you observe significant packet loss with your settings?
>>
>> I'd have thought that having 32 threads on a box with 20 physical cores
>> would not be optimal. Do you get better results than with only one thread
>> per core?
>>
>> Thanks
>> erich
>>
>>
>>
>> On 05.02.2015 21:30, Brandon Lattin wrote:
>>
>>> We are currently running a cluster of six Suricata (IDS mode) boxes fed
>>> from a pair of Arista 7150. Each Arista is fed by a regeneration tap
>>> (which also feeds a Cisco NGA netflow appliance). We have two active
>>> borders. Each has 10GB up and 10GB down, for an effective 40GB max
>>> (though we're not saturating the links). Not quite what you're
>>> architecting from the sounds of it, but not too far off.
>>>
>>> Three of the sensor boxes are production and each has a redundant pair,
>>> in case we loose power at an edge node. Fiber crossovers from each data
>>> center ensure each Arista 7150 aggregates a full set of traffic from
>>> both borders.
>>>
>>> Each server (Dell R620) has 2x 10 physical core Xeons at 3.0GHz, 128GB
>>> of RAM, has has 1x Myricom 10G-PCIE2-8C2-2S NIC with the Sniffer10G
>>> firmware. Each server runs a single 32 thread Suricata instance (maxium
>>> number of ring buffers per Myricom port). We currently see around
>>> 250-500MBps on each port and run around 15,000 Emerging Threats Pro
>>> rules.
>>>
>>> We use EVE JSON output with the printable_packet option. Logs are
>>> ingested by a Splunk Universal Forwarder agent on each box which feeds
>>> our Splunk indexer where analysis takes place. We see around a million
>>> events per day, which ends up being about 1-2GB of Splunk indexing
>>> volume per day.
>>>
>>> Additionally, we feed 3x Dell R620 running Bro (in testing), and we have
>>> a spare R620 for staging/testing.
>>>
>>> Things just went into production, so I'm still working out a few kinks.
>>>
>>> Hope this helps.
>>>
>>> On Thu, Feb 5, 2015 at 1:57 PM, Peter Manev <petermanev at gmail.com
>>> <mailto:petermanev at gmail.com>> wrote:
>>>
>>> Hi Douglas,
>>>
>>> I have not - but I am willing to help you out in that every step of
>>> the way - should you consider the offer.
>>>
>>> There would be a lot of considerations that will need to be taken
>>> into
>>> account prior getting the HW (i suspect)
>>>
>>> An initial question if I may:
>>> You mention IDS/IPS - is one or the other or is it both ?
>>>
>>> Thank you
>>>
>>>
>>> On Thu, Feb 5, 2015 at 1:59 PM, Duckworth, Douglas C
>>> <duckd at tulane.edu <mailto:duckd at tulane.edu>> wrote:
>>> > -----BEGIN PGP SIGNED MESSAGE-----
>>> > Hash: SHA1
>>> >
>>> > Hello
>>> >
>>> > We are developing a new high-speed network and are looking into
>>> IDS /
>>> > IPS solutions.
>>> >
>>> > Has anyone ran suricata at 40Gb?
>>> >
>>> > I found Tilera as one hardware vendor but appreciate any
>>> > recommendations for others.
>>> >
>>> >
>>> http://www.openinfosecfoundation.org/index.php/download-suricata/
>>> 173-oisf-welcomes-tilera-as-a-gold-level-consortium-member
>>> >
>>> >
>>> http://www.tilera.com/sites/default/files/productbriefs/
>>> TILExtreme-Gx-PB040-02_web.pdfhttp://www.tilera.com/sites/default/files/
>>> productbriefs/TILExtreme-Gx-PB040-02_web.pdf
>>> >
>>> > Thanks
>>> > Doug
>>> >
>>> > - --
>>> > Thanks
>>> >
>>> > Douglas Charles Duckworth
>>> > Unix Administrator
>>> > Tulane University
>>> > Technology Services
>>> > 1555 Poydras Ave
>>> > NOLA -- 70112
>>> >
>>> > E: duckd at tulane.edu <mailto:duckd at tulane.edu>
>>> > O: 504-988-9341 <tel:504-988-9341>
>>> > F: 504-988-8505 <tel:504-988-8505>
>>> > -----BEGIN PGP SIGNATURE-----
>>> > Version: GnuPG v1
>>> >
>>> > iQIcBAEBAgAGBQJU072YAAoJEP/Xbmk1axQptuEP/ilJ7Rt0Ep7ApdypLxnfKuV1
>>> > NzFRqkusjwV6SdOQ7pC0vKiFASwSVyivkSbNG8NfdU565qHj8uuXRt+Qm0zDMRkN
>>> > /IbKMcs5zxtPVA0OYdm8VcyUFO7AmDrOqZj7Du3o7RjmDn3JRTfICZDrDNTadxXX
>>> > GA/e8aXZx7a6EDU1basILY+71hueu9D8STto2EWdbNuZPtIvQHt5UygzdPg+N/64
>>> > XW0TT+f8TAqxw6UZvhw3EHeI+UyoB+JQWbUypbp7+XXMlnj/xpHoAb0JQwCq/Zjd
>>> > BL3scTTvU3LKAmMGkG3a20xORsn9Tm/3yTRnOzrhQOpkfXBgJuUncrA7Nar0K9AM
>>> > 6TkJzRhd2MpdPP4RtnYCO4z+KVhkcL1w8UfZlEFf2R/AUUHXNPI2kn/pI1z39qxI
>>> > qFyBeVt5N+ntFfd+wNAwqwKOmYDJVPBQixIo+U0jg2b2SLoaKFHEeyLaEXBYAlRU
>>> > 1IsWHeJz1Uci0ob63JS9CjO2gLyewqbTSQjo6L5jZayDCPEGlMF4XN95FCy5et0s
>>> > L/NJ6roHGq6RoFkV6xjfqPZBUGvZgjQJcq60n659r7QNlbeudI6S1cTRLb5TdkOH
>>> > /9hk5mWmhRtFtoUZr6UOM2U0Xc6Bhy+Cch4ThhfaOyaBRDkrktd5dNW6WbXVYCAa
>>> > tQXjcGH0XRyeT55FgHP2
>>> > =lILb
>>> > -----END PGP SIGNATURE-----
>>> > _______________________________________________
>>> > Suricata IDS Users mailing list:
>>> oisf-users at openinfosecfoundation.org
>>> <mailto:oisf-users at openinfosecfoundation.org>
>>> > Site: http://suricata-ids.org | Support:
>>> http://suricata-ids.org/support/
>>> > List:
>>> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>> > Training now available: http://suricata-ids.org/training/
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Peter Manev
>>> _______________________________________________
>>> Suricata IDS Users mailing list:
>>> oisf-users at openinfosecfoundation.org
>>> <mailto:oisf-users at openinfosecfoundation.org>
>>> Site: http://suricata-ids.org | Support:
>>> http://suricata-ids.org/support/
>>> List:
>>> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>> Training now available: http://suricata-ids.org/training/
>>>
>>>
>>>
>>>
>>> --
>>> Brandon Lattin
>>> Security Analyst
>>> University of Minnesota - University Information Security
>>> Office: 612-626-6672
>>>
>>>
>>> _______________________________________________
>>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>>> Site: http://suricata-ids.org | Support: http://suricata-ids.org/
>>> support/
>>> List: https://lists.openinfosecfoundation.org/
>>> mailman/listinfo/oisf-users
>>> Training now available: http://suricata-ids.org/training/
>>>
>>>
>
>
> --
> Brandon Lattin
> Security Analyst
> University of Minnesota - University Information Security
> Office: 612-626-6672
>
--
Brandon Lattin
Security Analyst
University of Minnesota - University Information Security
Office: 612-626-6672
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20150210/10e2f2ac/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: packetloss.jpg
Type: image/jpeg
Size: 47162 bytes
Desc: not available
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20150210/10e2f2ac/attachment-0002.jpg>
More information about the Oisf-users
mailing list