[Oisf-users] Question about cpu-affinity

[Eric Leblond]

Hi,

On Fri, 2018-03-02 at 11:51 +0100, Victor Julien wrote:
> On 02-03-18 11:48, Victor Julien wrote:
> > On 02-03-18 11:35, Eric Leblond wrote:
> > > Hi Cooper,
> > > 
> > > On Thu, 2018-03-01 at 08:12 -0800, Cooper F. Nelson wrote:
> > > > Hi Eric,
> > > > 
> > > > I hope you don't mind me copying you directly, as I saw you
> > > > were the
> > > > author of the threading/cpu-affinity module.
> > > 
> > > That was long ago ;)
> > > 
> > > > I'm wondering if it is currently possible when using multiple
> > > > interfaces
> > > > to 'interleave' the threads so that they are evenly distributed
> > > > across
> > > > all cores.  For example, consider a system with 16 cores and
> > > > two
> > > > interfaces, configured with 8 threads each.
> > > > 
> > > > Can you do something like this?
> > > > 
> > > >   - detect-cpu-set
> > > > 
> > > >         cpu: [ 0, 2, 4, 6, 8, 10, 12, 14, 1, 3, 5, 7, 9, 11,
> > > > 13, 15 ]
> > > > 
> > > >         mode: "exclusive"
> > > > 
> > > > The idea is we want the first 8 threads allocated to 0-14, even
> > > > cores
> > > > only.  The next 8 threads being allocated to 1-16, odd cores
> > > > only. 
> > > > Lets
> > > > assume we are using RSS in this example.
> > > > 
> > > > I had a look at the code and while it appears that this how you
> > > > have
> > > > implemented it, I'm not 100% certain, so I wanted to verify.
> > > 
> > > I don't think this will work. The CPU set is an unordered.
> > > Suricata
> > > build the CPU set as a map then it start by 0, get next CPU in
> > > the set
> > > (testing if 0 is in then switching to 1 and checking if it is in
> > > the
> > > set).
> > > 
> > > So with current code, the only way to do what you want is to have
> > > a
> > > ["all"] CPU set in exclusive mode in the affinity and an ugly af-
> > > packet 
> > > configuration like:
> > > 
> > >   af-packet:
> > >     - interface: eth0
> > >       threads: 1
> > >       cluster-id: 99
> > >     - interface: eth1
> > >       threads: 1
> > >       cluster-id: 98
> > >     - interface: eth0
> > >       threads: 1
> > >       cluster-id: 99
> > >     - interface: eth1
> > >       threads: 1
> > >       cluster-id: 98
> > >     ...
> > >     -
> > > default:
> > >       #set every global variable here
> > > 
> > 
> > Wonder how hard it would be to hide this kind of detail. E.g.
> > something
> > like:
> > 
> > af-packet:
> >   - interface: eth0
> >     cluster-id: 99
> >     numa-node: 0
> >   - interface: eth1
> >     cluster-id: 98
> >     numa-node: 1
> > 
> > It would immediately bring up the issue of how to exclude cores,
> > etc. So
> > might not be trivial.
> > 
> 
> Or maybe allow defining named cpu sets and allow assigning those to
> af-packet interface configs:
> 
> - cpu-set
>   name: af-packet-eth0
>   cpu: [ 0, 2, 4, 6, 8, 10, 12, 14]
>   mode: "exclusive"
> - cpu-set
>   name: af-packet-eth1
>   cpu: [1, 3, 5, 7, 9, 11, 13, 15 ]
>   mode: "exclusive"
> 
> 
> 
> af-packet:
>   - interface: eth0
>     cluster-id: 99
>     cpu-set: "af-packet-eth0"
>   - interface: eth1
>     cluster-id: 98
>     cpu-set: "af-packet-eth1"

I like this second proposal better. From what I've seen a few packet
capture APIs are using the numa node in the capture params, maybe we
could combined both approach.

++
-- 
Eric Leblond <eric at regit.org>