[Oisf-devel] [PATCH 2/2] Flow: Increase refresh timer

[Eric Leblond]

Hi,

First of all: Happy New Year to all and long life to Suricata.

Le vendredi 31 décembre 2010 à 17:52 +0100, Victor Julien a écrit :
> On 12/31/2010 01:15 PM, Eric Leblond wrote:
> > FlowManagerThread function was running its check after a 10
> > microseconds sleep. nanosleep system call were causing a
> > consequent CPU load on my computer running running no trafic.
> > This patch modifies the timer to a default of 10 ms.
> > 
> > It also suppress a usleep(1) used in pcap file mode. This delay
> > seems to short to really bring something.
> 
> This part of the patch seems problematic. In testing it seems almost 25%
> slower when processing pcaps.

I agree with the result of your tests. Mine give same result on bigger
pcap.

After more thinking, it seems to me there is something weird with the
FlowManagerThread(): activity of the flow is mainly linked with packet
rate and should thus flow manager should be triggered each certain
amount of packets.

A solution to implement something like that could be to do something
like that:

 TmEcode DecodePcapFile(ThreadVars *tv, Packet *p, void *data, PacketQueue *pq, PacketQueue *postpq)
 {
     SCEnter();
@@ -308,6 +312,12 @@ TmEcode DecodePcapFile(ThreadVars *tv, Packet *p, void *data, PacketQueue *pq, P
     SCPerfCounterIncr(dtv->counter_pkts, tv->sc_perf_pca);
     SCPerfCounterIncr(dtv->counter_pkts_per_sec, tv->sc_perf_pca);
 
+    if (gruik_count >= PACKET_FLOW_INTERVAL) {
+        gruik_count = 0;
+        FlowManagerTasks();
+    } else
+        gruik_count++;
+
     SCPerfCounterAddUI64(dtv->counter_bytes, tv->sc_perf_pca, GET_PKT_LEN(p));

We update a counter (here named gruik_count because this is not
thread-safe) and after some iteration we start the Flow management task
(pruning or allocating) in a serial way. This will stop packet
processing for a moment but this will be short and may be better than
being out of flow. In the case of pca file input, this seems efficient.
I've implemented this and performance are a very little bit better than
usleep(1) version.

On other solution is to signal the flow manager thread when the amount
of Packet has been reached. This provide a more multithread way to do
things but we have to aasume the flow manager task will be run as soon
as wanted (which can not be guaranted in case of high load). I've tried
a implementation of this solution but this was not a success in term of
performance (but this has great chance to be due to the quality of the
code).

What do you think ?

BR,
-- 
Eric Leblond <eric at regit.org>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-devel/attachments/20110103/327eb238/attachment.sig>