[Oisf-devel] per thread pools

[Victor Julien]

We have several tickets for per thread pools [1,2,3]. Mostly for the
stream engine, but there will be a need for other parts of the engine as
well, like in my DNS parser work and Anoop's streaming parser API.

The challenge is that the per-thread pools should be accessible by other
threads in some cases. Most importantly here is that our Flow manager
thread cleans up Flows, including TCP ssn, app layer, etc. It will have
to be able to return things to the proper pools.

Below is a simple proposal for a API.

In summary, we'd just use an array of Pools and an array of locks per
pool. The index to the arrays will be a thread id. We can't use the OS
thread id obviously as it would make the arrays much bigger than necessary.

Since most activity will be in the thread itself, contention should be
low, certainly much lower than the current global pools. We still need
the locking for the cases where another thread does access a pool.


1. We need a per thread idx, 0 - threadnum

2. Thread pool defined as:

struct ThreadPool {
    Pool tpools[threadnum];
    Lock tlocks[threadnum];
};

3. A normal ThreadPoolGet(...)

void *ThreadPoolGet(ThreadPool *p) {
    int tid = ThreadId();
    Lock(p->tlocks[tid]);
    void *ptr = PoolGet(tpools[tid]);
    Unlock(p->tlocks[tid]);
    return ptr;
}

4. A normal ThreadPoolReturn(...)

void ThreadPoolReturn(ThreadPool *p, void *ptr) {
    int tid = ThreadId();
    Lock(p->tlocks[tid]);
    PoolReturn(tpools[tid], ptr);
    Unlock(p->tlocks[tid]);
}


5a. A ThreadPoolReturn(...) by another thread

void ThreadPoolReturnForId(ThreadPool *p, int tid, void *ptr) {
    Lock(p->tlocks[tid]);
    PoolReturn(tpools[tid], ptr);
    Unlock(p->tlocks[tid]);
}

An obvious problem is how in this last case the thread id should be
stored. In stream-tcp-reassemble.c we use a variable in the TcpSegment
structure (pool_size) to map the segment to a pool (we have pools for
several segment sizes).

5b. Maybe we can force the users of the pools to reserve 4 bytes at the
start of their data:

struct TcpSegment {
    ThreadPoolReserved res;

    ... segment vars...
}

struct ThreadPoolReserved {
    int id;
}

void ThreadPoolReturnProbeId(ThreadPool *p, void *ptr) {
    ThreadPoolReserved *r = ptr;
    int tid = r->id;
    Lock(p->tlocks[tid]);
    PoolReturn(tpools[tid], ptr);
    Unlock(p->tlocks[tid]);
}

Thoughts?

Cheers,
Victor


[1] https://redmine.openinfosecfoundation.org/issues/519
[2] https://redmine.openinfosecfoundation.org/issues/520
[3] https://redmine.openinfosecfoundation.org/issues/521
-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------