[Oisf-users] Suricata with IRQBalance

[Shane Boissevain]

Just realized I only replied to Yasha directly. Resending this to the list.

I'm running something similar on Ubuntu 16.04, and while this is not on
CentOS, i'll share my findings. Since you're cat-ing /proc/interrupts, it
sounds like what you're asking is how to set the IRQ for the NIC at a
SYSTEM level. If you then pair the worker's threads via suricata's
set-cpu-affinity setting, you should see (or at least feel better?) about
that performance boost.

Firstly, I've found that irqbalance was giving me headaches and resetting
my values....so:

killall irqbalance
> systemctl disable irqbalance


Secondly, I needed to know what interrupts to set. I'm running an 8-core
CPU (4 pysical cores, hyperthreaded) so i should have 8 irq's to deal with
(This varies, but I believe it to be the smaller number between Rx-Tx
queues provided by the NIC and the number of cores you have - it looks like
you've got 4). Also, since you're asking about IRQs, I'm assuming you can
get the name of the NIC in question. Mine is enp1s0f0, looks like yours is
eth0.

# grep enp1s0f0 /proc/interrupts

     CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7
>  33:  300    0    0    0    0    0    0    0  IR-PCI-MSI 524288-edge
>  enp1s0f0-TxRx-0
>  34:    0  300    0    0    0    0    0    0  IR-PCI-MSI 524289-edge
>  enp1s0f0-TxRx-1
>  35:    0    0  300    0    0    0    0    0  IR-PCI-MSI 524290-edge
>  enp1s0f0-TxRx-2
>  36:    0    0    0  300    0    0    0    0  IR-PCI-MSI 524291-edge
>  enp1s0f0-TxRx-3
>  37:    0    0    0    0  300    0    0    0  IR-PCI-MSI 524292-edge
>  enp1s0f0-TxRx-4
>  38:    0    0    0    0    0  300    0    0  IR-PCI-MSI 524293-edge
> enp1s0f0-TxRx-5
>  39:    0    0    0    0    0    0  300    0  IR-PCI-MSI 524294-edge
> enp1s0f0-TxRx-6
>  40:    0    0    0    0    0    0    0  300  IR-PCI-MSI 524295-edge
> enp1s0f0-TxRx-7
>  41:    0    0    0    0    0    0    0    0  IR-PCI-MSI 524296-edge
> enp1s0f0


*NOTE: *Take notice of the 'diagonal'. I'm doing this on a box that has
already had the settings applied. After a reboot, yours should also have
zeros except for the pretty "diagonal" that shows it's working correctly
(although the numbers may not be all equal, it should be distributed about
evenly).

The above shows interrupts 33 through 40 are for enp1s0f0's Tx and Rx
queues, so I'm going to go ahead and set those via the bitmask as denoted
here:
https://github.com/torvalds/linux/blob/master/Documentation/IRQ-affinity.txt

echo 01 > /proc/irq/33/smp_affinity

echo 02 > /proc/irq/34/smp_affinity

echo 04 > /proc/irq/35/smp_affinity

echo 08 > /proc/irq/36/smp_affinity

echo 10 > /proc/irq/37/smp_affinity

echo 20 > /proc/irq/38/smp_affinity

echo 40 > /proc/irq/39/smp_affinity

echo 80 > /proc/irq/40/smp_affinity


And...that should do it!

Cheers,
~Shane

On Tue, May 31, 2016 at 10:47 AM, Cooper F. Nelson <cnelson at ucsd.edu> wrote:

> I don't see any difference and it works fine as-is.
>
> I may switch to the script when I start using multiple 10G interfaces,
> just I have precise control over what cores are used.
>
> -Coop
>
> On 5/31/2016 12:05 AM, Peter Manev wrote:
> > On Mon, May 30, 2016 at 6:44 PM, Cooper F. Nelson <cnelson at ucsd.edu>
> wrote:
> >> > Try reloading your NIC kernel module and restarting IRQ balance.
> >> >
> > So - you use irqbalance ? (I always disable it)
> > Have you seen any benefit contra using just the affinity script ?
> >
> >> > I know I've seen it get 'stuck' on occasion, so I have a script to do
> >> > the above prior to launching suricata.
> >> >
> >> > -Coop
>
>
> --
> Cooper Nelson
> Network Security Analyst
> UCSD ITS Security Team
> cnelson at ucsd.edu x41042
>
>
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> Suricata User Conference November 9-11 in Washington, DC:
> http://oisfevents.net
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20160531/90f5452f/attachment-0002.html>