<div dir="ltr"><div>Erich,</div><div><br></div>After making some significant changes, we're at 0% packetloss for the last 12 hours. We'll hit network peak at around 1PM, so I'll update if that changes.<div><br><div>I've swapped to pinning to 18 cores as well as 2-tuple load balancing across 3 boxes. Additionally, I've increased max-pending-packets to 60k from ~16k and swapped to your values for the ring buffers: SNF_DATARING_SIZE=34359738368 SNF_DESCRING_SIZE=8589934592</div><div><br></div><div>Here's the relevant config:</div><div><br></div><div><div> set-cpu-affinity: yes </div><div> cpu-affinity:</div><div> - management-cpu-set:</div><div> cpu: [ 0-1 ] # include only these cpus in affinity settings</div><div> mode: "balanced"</div><div> prio:</div><div> default: "low"</div><div> - receive-cpu-set:</div><div> cpu: [ 0-1 ] # include only these cpus in affinity settings</div><div> - decode-cpu-set:</div><div> cpu: [ 0-1 ]</div><div> mode: "balanced"</div><div> - stream-cpu-set:</div><div> cpu: [ 0-1 ]</div><div> - detect-cpu-set:</div><div> cpu: [ 2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36 ]</div><div> mode: "exclusive" # run detect threads in these cpus</div><div> prio:</div><div> default: "high"</div><div> - verdict-cpu-set:</div><div> cpu: [ 0-1 ]</div><div> prio:</div><div> default: "high"</div><div> - reject-cpu-set:</div><div> cpu: [ 0-1 ]</div><div> prio:</div><div> default: "low"</div><div> - output-cpu-set:</div><div> cpu: [ 0-1 ]</div><div> prio:</div><div> default: "medium"</div></div></div><div><br></div><div><br></div><div>Thanks for providing some insight into improvements!</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Feb 9, 2015 at 1:28 PM, Brandon Lattin <span dir="ltr"><<a href="mailto:latt0050@umn.edu" target="_blank">latt0050@umn.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Erich,<div><br></div><div>Your response got me thinking. The 32 ring buffers was a remnant from an old config where I was seeing significantly better performance _not_ pinning directly to cores on an older test box (with a much lower clock speed). I'm about to reconfigure for 16 threads on our dev box. I'll let you know how it goes.</div><div><br></div><div><div>Here are my ring buffer settings: SNF_DATARING_SIZE=268435456 SNF_DESCRING_SIZE=67108864</div><div>And yours: SNF_DATARING_SIZE=34359738368 SNF_DESCRING_SIZE=<a href="tel:8589934592" value="+18589934592" target="_blank">8589934592</a></div></div><div><br></div><div>I do notice that if I drop mine at all, packetloss starts to creep. The boxes have RAM to spare, so I'll crank up mine to match your setting of 32GB after testing out pinning to 16 physical cores.</div><div><br></div><div>Right now, we see spikes of high packetloss on one of the boxes. The others hover under 1%. This is probably due to the type of traffic we have on that interface. Whitelisting a few IPs, such as the Debian mirror we run, should clear up the burst packetloss. </div><div><br></div><div><img src="cid:ii_14b6fcfa819a0efa" alt="Inline image 1" width="540" height="87"> </div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Feb 6, 2015 at 3:59 PM, Erich Lerch <span dir="ltr"><<a href="mailto:erich.lerch@gmail.com" target="_blank">erich.lerch@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Very intersting, Brandon, thanks for sharing!<br>
<br>
We have a somewhat similar setup (less traffic though, only one suricata box), with Myricom Sniffer10G NICs.<br>
<br>
May I ask how you configured the ring buffers for suricata?<br>
<br>
We use 16 rings and set<br>
SNF_DATARING_SIZE=34359738368<br>
SNF_DESCRING_SIZE=<a href="tel:8589934592" value="+18589934592" target="_blank">8589934592</a><br>
<br>
This is a lot, but we kept having higher packet loss with lower settings.<br>
That's why I'd be interested in others' experiences with ring number and ring mem settings.<br>
<br>
Do you observe significant packet loss with your settings?<br>
<br>
I'd have thought that having 32 threads on a box with 20 physical cores would not be optimal. Do you get better results than with only one thread per core?<br>
<br>
Thanks<br>
erich<div><div><br>
<br>
<br>
On 05.02.2015 21:30, Brandon Lattin wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div>
We are currently running a cluster of six Suricata (IDS mode) boxes fed<br>
from a pair of Arista 7150. Each Arista is fed by a regeneration tap<br>
(which also feeds a Cisco NGA netflow appliance). We have two active<br>
borders. Each has 10GB up and 10GB down, for an effective 40GB max<br>
(though we're not saturating the links). Not quite what you're<br>
architecting from the sounds of it, but not too far off.<br>
<br>
Three of the sensor boxes are production and each has a redundant pair,<br>
in case we loose power at an edge node. Fiber crossovers from each data<br>
center ensure each Arista 7150 aggregates a full set of traffic from<br>
both borders.<br>
<br>
Each server (Dell R620) has 2x 10 physical core Xeons at 3.0GHz, 128GB<br>
of RAM, has has 1x Myricom 10G-PCIE2-8C2-2S NIC with the Sniffer10G<br>
firmware. Each server runs a single 32 thread Suricata instance (maxium<br>
number of ring buffers per Myricom port). We currently see around<br>
250-500MBps on each port and run around 15,000 Emerging Threats Pro rules.<br>
<br>
We use EVE JSON output with the printable_packet option. Logs are<br>
ingested by a Splunk Universal Forwarder agent on each box which feeds<br>
our Splunk indexer where analysis takes place. We see around a million<br>
events per day, which ends up being about 1-2GB of Splunk indexing<br>
volume per day.<br>
<br>
Additionally, we feed 3x Dell R620 running Bro (in testing), and we have<br>
a spare R620 for staging/testing.<br>
<br>
Things just went into production, so I'm still working out a few kinks.<br>
<br>
Hope this helps.<br>
<br>
On Thu, Feb 5, 2015 at 1:57 PM, Peter Manev <<a href="mailto:petermanev@gmail.com" target="_blank">petermanev@gmail.com</a><br></div></div><span>
<mailto:<a href="mailto:petermanev@gmail.com" target="_blank">petermanev@gmail.com</a>>> wrote:<br>
<br>
Hi Douglas,<br>
<br>
I have not - but I am willing to help you out in that every step of<br>
the way - should you consider the offer.<br>
<br>
There would be a lot of considerations that will need to be taken into<br>
account prior getting the HW (i suspect)<br>
<br>
An initial question if I may:<br>
You mention IDS/IPS - is one or the other or is it both ?<br>
<br>
Thank you<br>
<br>
<br>
On Thu, Feb 5, 2015 at 1:59 PM, Duckworth, Douglas C<br></span><span>
<<a href="mailto:duckd@tulane.edu" target="_blank">duckd@tulane.edu</a> <mailto:<a href="mailto:duckd@tulane.edu" target="_blank">duckd@tulane.edu</a>>> wrote:<br>
> -----BEGIN PGP SIGNED MESSAGE-----<br>
> Hash: SHA1<br>
><br>
> Hello<br>
><br>
> We are developing a new high-speed network and are looking into IDS /<br>
> IPS solutions.<br>
><br>
> Has anyone ran suricata at 40Gb?<br>
><br>
> I found Tilera as one hardware vendor but appreciate any<br>
> recommendations for others.<br>
><br>
><br>
<a href="http://www.openinfosecfoundation.org/index.php/download-suricata/173-oisf-welcomes-tilera-as-a-gold-level-consortium-member" target="_blank">http://www.<u></u>openinfosecfoundation.org/<u></u>index.php/download-suricata/<u></u>173-oisf-welcomes-tilera-as-a-<u></u>gold-level-consortium-member</a><br>
><br>
><br>
<a href="http://www.tilera.com/sites/default/files/productbriefs/TILExtreme-Gx-PB040-02_web.pdfhttp://www.tilera.com/sites/default/files/productbriefs/TILExtreme-Gx-PB040-02_web.pdf" target="_blank">http://www.tilera.com/sites/<u></u>default/files/productbriefs/<u></u>TILExtreme-Gx-PB040-02_web.<u></u>pdfhttp://www.tilera.com/<u></u>sites/default/files/<u></u>productbriefs/TILExtreme-Gx-<u></u>PB040-02_web.pdf</a><br>
><br>
> Thanks<br>
> Doug<br>
><br>
> - --<br>
> Thanks<br>
><br>
> Douglas Charles Duckworth<br>
> Unix Administrator<br>
> Tulane University<br>
> Technology Services<br>
> 1555 Poydras Ave<br>
> NOLA -- 70112<br>
><br></span>
> E: <a href="mailto:duckd@tulane.edu" target="_blank">duckd@tulane.edu</a> <mailto:<a href="mailto:duckd@tulane.edu" target="_blank">duckd@tulane.edu</a>><br>
> O: <a href="tel:504-988-9341" value="+15049889341" target="_blank">504-988-9341</a> <tel:<a href="tel:504-988-9341" value="+15049889341" target="_blank">504-988-9341</a>><br>
> F: <a href="tel:504-988-8505" value="+15049888505" target="_blank">504-988-8505</a> <tel:<a href="tel:504-988-8505" value="+15049888505" target="_blank">504-988-8505</a>><span><br>
> -----BEGIN PGP SIGNATURE-----<br>
> Version: GnuPG v1<br>
><br>
> iQIcBAEBAgAGBQJU072YAAoJEP/<u></u>Xbmk1axQptuEP/<u></u>ilJ7Rt0Ep7ApdypLxnfKuV1<br>
> NzFRqkusjwV6SdOQ7pC0vKiFASwSVy<u></u>ivkSbNG8NfdU565qHj8uuXRt+<u></u>Qm0zDMRkN<br>
> /<u></u>IbKMcs5zxtPVA0OYdm8VcyUFO7AmDr<u></u>OqZj7Du3o7RjmDn3JRTfICZDrDNTad<u></u>xXX<br>
> GA/e8aXZx7a6EDU1basILY+<u></u>71hueu9D8STto2EWdbNuZPtIvQHt5U<u></u>ygzdPg+N/64<br>
> XW0TT+f8TAqxw6UZvhw3EHeI+UyoB+<u></u>JQWbUypbp7+XXMlnj/<u></u>xpHoAb0JQwCq/Zjd<br>
> BL3scTTvU3LKAmMGkG3a20xORsn9Tm<u></u>/<u></u>3yTRnOzrhQOpkfXBgJuUncrA7Nar0K<u></u>9AM<br>
> 6TkJzRhd2MpdPP4RtnYCO4z+<u></u>KVhkcL1w8UfZlEFf2R/<u></u>AUUHXNPI2kn/pI1z39qxI<br>
> qFyBeVt5N+ntFfd+<u></u>wNAwqwKOmYDJVPBQixIo+<u></u>U0jg2b2SLoaKFHEeyLaEXBYAlRU<br>
> 1IsWHeJz1Uci0ob63JS9CjO2gLyewq<u></u>bTSQjo6L5jZayDCPEGlMF4XN95FCy5<u></u>et0s<br>
> L/<u></u>NJ6roHGq6RoFkV6xjfqPZBUGvZgjQJ<u></u>cq60n659r7QNlbeudI6S1cTRLb5Tdk<u></u>OH<br>
> /<u></u>9hk5mWmhRtFtoUZr6UOM2U0Xc6Bhy+<u></u>Cch4ThhfaOyaBRDkrktd5dNW6WbXVY<u></u>CAa<br>
> tQXjcGH0XRyeT55FgHP2<br>
> =lILb<br>
> -----END PGP SIGNATURE-----<br>
> ______________________________<u></u>_________________<br>
> Suricata IDS Users mailing list:<br>
<a href="mailto:oisf-users@openinfosecfoundation.org" target="_blank">oisf-users@<u></u>openinfosecfoundation.org</a><br></span>
<mailto:<a href="mailto:oisf-users@openinfosecfoundation.org" target="_blank">oisf-users@<u></u>openinfosecfoundation.org</a>><span><br>
> Site: <a href="http://suricata-ids.org" target="_blank">http://suricata-ids.org</a> | Support:<br>
<a href="http://suricata-ids.org/support/" target="_blank">http://suricata-ids.org/<u></u>support/</a><br>
> List:<br>
<a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" target="_blank">https://lists.<u></u>openinfosecfoundation.org/<u></u>mailman/listinfo/oisf-users</a><br>
> Training now available: <a href="http://suricata-ids.org/training/" target="_blank">http://suricata-ids.org/<u></u>training/</a><br>
<br>
<br>
<br>
--<br>
Regards,<br>
Peter Manev<br>
______________________________<u></u>_________________<br>
Suricata IDS Users mailing list:<br>
<a href="mailto:oisf-users@openinfosecfoundation.org" target="_blank">oisf-users@<u></u>openinfosecfoundation.org</a><br></span>
<mailto:<a href="mailto:oisf-users@openinfosecfoundation.org" target="_blank">oisf-users@<u></u>openinfosecfoundation.org</a>><span><br>
Site: <a href="http://suricata-ids.org" target="_blank">http://suricata-ids.org</a> | Support:<br>
<a href="http://suricata-ids.org/support/" target="_blank">http://suricata-ids.org/<u></u>support/</a><br>
List:<br>
<a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" target="_blank">https://lists.<u></u>openinfosecfoundation.org/<u></u>mailman/listinfo/oisf-users</a><br>
Training now available: <a href="http://suricata-ids.org/training/" target="_blank">http://suricata-ids.org/<u></u>training/</a><br>
<br>
<br>
<br>
<br>
--<br>
Brandon Lattin<br>
Security Analyst<br>
University of Minnesota - University Information Security<br>
Office: <a href="tel:612-626-6672" value="+16126266672" target="_blank">612-626-6672</a><br>
<br>
<br></span><span>
______________________________<u></u>_________________<br>
Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org" target="_blank">oisf-users@<u></u>openinfosecfoundation.org</a><br>
Site: <a href="http://suricata-ids.org" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" target="_blank">http://suricata-ids.org/<u></u>support/</a><br>
List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" target="_blank">https://lists.<u></u>openinfosecfoundation.org/<u></u>mailman/listinfo/oisf-users</a><br>
Training now available: <a href="http://suricata-ids.org/training/" target="_blank">http://suricata-ids.org/<u></u>training/</a><br>
<br>
</span></blockquote>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div><div dir="ltr">Brandon Lattin<div>Security Analyst<br><div>University of Minnesota - University Information Security<br>Office: <a href="tel:612-626-6672" value="+16126266672" target="_blank">612-626-6672</a></div></div></div></div>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr">Brandon Lattin<div>Security Analyst<br><div>University of Minnesota - University Information Security<br>Office: 612-626-6672</div></div></div></div>
</div>