[Oisf-users] suricata at 40Gb
Brandon Lattin
latt0050 at umn.edu
Mon Feb 9 19:28:27 UTC 2015
Erich,
Your response got me thinking. The 32 ring buffers was a remnant from an
old config where I was seeing significantly better performance _not_
pinning directly to cores on an older test box (with a much lower clock
speed). I'm about to reconfigure for 16 threads on our dev box. I'll let
you know how it goes.
Here are my ring buffer settings: SNF_DATARING_SIZE=268435456
SNF_DESCRING_SIZE=67108864
And yours: SNF_DATARING_SIZE=34359738368 SNF_DESCRING_SIZE=8589934592
I do notice that if I drop mine at all, packetloss starts to creep. The
boxes have RAM to spare, so I'll crank up mine to match your setting of
32GB after testing out pinning to 16 physical cores.
Right now, we see spikes of high packetloss on one of the boxes. The others
hover under 1%. This is probably due to the type of traffic we have on that
interface. Whitelisting a few IPs, such as the Debian mirror we run, should
clear up the burst packetloss.
[image: Inline image 1]
On Fri, Feb 6, 2015 at 3:59 PM, Erich Lerch <erich.lerch at gmail.com> wrote:
> Very intersting, Brandon, thanks for sharing!
>
> We have a somewhat similar setup (less traffic though, only one suricata
> box), with Myricom Sniffer10G NICs.
>
> May I ask how you configured the ring buffers for suricata?
>
> We use 16 rings and set
> SNF_DATARING_SIZE=34359738368
> SNF_DESCRING_SIZE=8589934592
>
> This is a lot, but we kept having higher packet loss with lower settings.
> That's why I'd be interested in others' experiences with ring number and
> ring mem settings.
>
> Do you observe significant packet loss with your settings?
>
> I'd have thought that having 32 threads on a box with 20 physical cores
> would not be optimal. Do you get better results than with only one thread
> per core?
>
> Thanks
> erich
>
>
>
> On 05.02.2015 21:30, Brandon Lattin wrote:
>
>> We are currently running a cluster of six Suricata (IDS mode) boxes fed
>> from a pair of Arista 7150. Each Arista is fed by a regeneration tap
>> (which also feeds a Cisco NGA netflow appliance). We have two active
>> borders. Each has 10GB up and 10GB down, for an effective 40GB max
>> (though we're not saturating the links). Not quite what you're
>> architecting from the sounds of it, but not too far off.
>>
>> Three of the sensor boxes are production and each has a redundant pair,
>> in case we loose power at an edge node. Fiber crossovers from each data
>> center ensure each Arista 7150 aggregates a full set of traffic from
>> both borders.
>>
>> Each server (Dell R620) has 2x 10 physical core Xeons at 3.0GHz, 128GB
>> of RAM, has has 1x Myricom 10G-PCIE2-8C2-2S NIC with the Sniffer10G
>> firmware. Each server runs a single 32 thread Suricata instance (maxium
>> number of ring buffers per Myricom port). We currently see around
>> 250-500MBps on each port and run around 15,000 Emerging Threats Pro rules.
>>
>> We use EVE JSON output with the printable_packet option. Logs are
>> ingested by a Splunk Universal Forwarder agent on each box which feeds
>> our Splunk indexer where analysis takes place. We see around a million
>> events per day, which ends up being about 1-2GB of Splunk indexing
>> volume per day.
>>
>> Additionally, we feed 3x Dell R620 running Bro (in testing), and we have
>> a spare R620 for staging/testing.
>>
>> Things just went into production, so I'm still working out a few kinks.
>>
>> Hope this helps.
>>
>> On Thu, Feb 5, 2015 at 1:57 PM, Peter Manev <petermanev at gmail.com
>> <mailto:petermanev at gmail.com>> wrote:
>>
>> Hi Douglas,
>>
>> I have not - but I am willing to help you out in that every step of
>> the way - should you consider the offer.
>>
>> There would be a lot of considerations that will need to be taken into
>> account prior getting the HW (i suspect)
>>
>> An initial question if I may:
>> You mention IDS/IPS - is one or the other or is it both ?
>>
>> Thank you
>>
>>
>> On Thu, Feb 5, 2015 at 1:59 PM, Duckworth, Douglas C
>> <duckd at tulane.edu <mailto:duckd at tulane.edu>> wrote:
>> > -----BEGIN PGP SIGNED MESSAGE-----
>> > Hash: SHA1
>> >
>> > Hello
>> >
>> > We are developing a new high-speed network and are looking into
>> IDS /
>> > IPS solutions.
>> >
>> > Has anyone ran suricata at 40Gb?
>> >
>> > I found Tilera as one hardware vendor but appreciate any
>> > recommendations for others.
>> >
>> >
>> http://www.openinfosecfoundation.org/index.php/download-suricata/
>> 173-oisf-welcomes-tilera-as-a-gold-level-consortium-member
>> >
>> >
>> http://www.tilera.com/sites/default/files/productbriefs/
>> TILExtreme-Gx-PB040-02_web.pdfhttp://www.tilera.com/sites/default/files/
>> productbriefs/TILExtreme-Gx-PB040-02_web.pdf
>> >
>> > Thanks
>> > Doug
>> >
>> > - --
>> > Thanks
>> >
>> > Douglas Charles Duckworth
>> > Unix Administrator
>> > Tulane University
>> > Technology Services
>> > 1555 Poydras Ave
>> > NOLA -- 70112
>> >
>> > E: duckd at tulane.edu <mailto:duckd at tulane.edu>
>> > O: 504-988-9341 <tel:504-988-9341>
>> > F: 504-988-8505 <tel:504-988-8505>
>> > -----BEGIN PGP SIGNATURE-----
>> > Version: GnuPG v1
>> >
>> > iQIcBAEBAgAGBQJU072YAAoJEP/Xbmk1axQptuEP/ilJ7Rt0Ep7ApdypLxnfKuV1
>> > NzFRqkusjwV6SdOQ7pC0vKiFASwSVyivkSbNG8NfdU565qHj8uuXRt+Qm0zDMRkN
>> > /IbKMcs5zxtPVA0OYdm8VcyUFO7AmDrOqZj7Du3o7RjmDn3JRTfICZDrDNTadxXX
>> > GA/e8aXZx7a6EDU1basILY+71hueu9D8STto2EWdbNuZPtIvQHt5UygzdPg+N/64
>> > XW0TT+f8TAqxw6UZvhw3EHeI+UyoB+JQWbUypbp7+XXMlnj/xpHoAb0JQwCq/Zjd
>> > BL3scTTvU3LKAmMGkG3a20xORsn9Tm/3yTRnOzrhQOpkfXBgJuUncrA7Nar0K9AM
>> > 6TkJzRhd2MpdPP4RtnYCO4z+KVhkcL1w8UfZlEFf2R/AUUHXNPI2kn/pI1z39qxI
>> > qFyBeVt5N+ntFfd+wNAwqwKOmYDJVPBQixIo+U0jg2b2SLoaKFHEeyLaEXBYAlRU
>> > 1IsWHeJz1Uci0ob63JS9CjO2gLyewqbTSQjo6L5jZayDCPEGlMF4XN95FCy5et0s
>> > L/NJ6roHGq6RoFkV6xjfqPZBUGvZgjQJcq60n659r7QNlbeudI6S1cTRLb5TdkOH
>> > /9hk5mWmhRtFtoUZr6UOM2U0Xc6Bhy+Cch4ThhfaOyaBRDkrktd5dNW6WbXVYCAa
>> > tQXjcGH0XRyeT55FgHP2
>> > =lILb
>> > -----END PGP SIGNATURE-----
>> > _______________________________________________
>> > Suricata IDS Users mailing list:
>> oisf-users at openinfosecfoundation.org
>> <mailto:oisf-users at openinfosecfoundation.org>
>> > Site: http://suricata-ids.org | Support:
>> http://suricata-ids.org/support/
>> > List:
>> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>> > Training now available: http://suricata-ids.org/training/
>>
>>
>>
>> --
>> Regards,
>> Peter Manev
>> _______________________________________________
>> Suricata IDS Users mailing list:
>> oisf-users at openinfosecfoundation.org
>> <mailto:oisf-users at openinfosecfoundation.org>
>> Site: http://suricata-ids.org | Support:
>> http://suricata-ids.org/support/
>> List:
>> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>> Training now available: http://suricata-ids.org/training/
>>
>>
>>
>>
>> --
>> Brandon Lattin
>> Security Analyst
>> University of Minnesota - University Information Security
>> Office: 612-626-6672
>>
>>
>> _______________________________________________
>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
>> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>> Training now available: http://suricata-ids.org/training/
>>
>>
--
Brandon Lattin
Security Analyst
University of Minnesota - University Information Security
Office: 612-626-6672
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20150209/94344d95/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: packetloss.jpg
Type: image/jpeg
Size: 47162 bytes
Desc: not available
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20150209/94344d95/attachment-0002.jpg>
More information about the Oisf-users
mailing list