[Oisf-users] Performance on multiple CPUs

[Dave Remien]

On Fri, Aug 26, 2011 at 1:54 AM, Victor Julien <victor at inliniac.net> wrote:

> On 08/25/2011 08:18 PM, Gene Albin wrote:
> > Victor,
> >   Thanks for the explanation.  So do I understand correctly that for
> reading
> > pcap files I'm limited to only a single packet reader thread?  So at some
> > point my performance bottleneck would become the packet reading thread,
> > regardless of where the pcap file is stored?  If that's the case I could
> > probably use tcpreplay to resend the traffic through the interface and
> > pfring (thought I haven't begun to work with pfring yet).
>
> Correct. I don't think there is a sane way to have more pcap file
> readers. Replaying seems to be your best option.
>
>
I'm going to crawl out on a limb here.... Feel free to join me 8-).

I'll venture that reading a pcap is, for all practical purposes, limited by
the I/O to the pcap file. If you're reading the pcap from memory (either
from the cache or a RAM disk), you can probably read it at 1+gigabytes/sec,
assuming a modern machine with modern memory.

FWIW, the following is from a fairly elderly 8 cpu 3GHz Core2 machine, with
a 12 disk RAID6 array on a 4 port, 3Gbit/sec SAS backplane and modern LSI
PCiE controller:

space2:/y/pcaps # time dd if=tcpdump-2007-02-16-08-02 of=/dev/null bs=256k
2861036+1 records in
2861036+1 records out
750003609600 bytes (750 GB) copied, 994.034 s, 755 MB/s

real    16m34.051s
user    0m0.720s
sys     8m12.135s
space2:/y/pcaps # time tcpdump -nnr tcpdump-2007-02-16-08-02 -s0 -w
/dev/null
reading from file tcpdump-2007-02-16-08-02, link-type EN10MB (Ethernet)
tcpdump: pcap_loop: truncated dump file; tried to read 106 captured bytes,
only got 93

real    19m34.546s
user    5m46.591s
sys     9m31.326s
space2:/y/pcaps # ll tcpdump-2007-02-16-08-02
-rw-r--r-- 1 root root 750003609600 Mar 26  2007 tcpdump-2007-02-16-08-02

 So, this system can read a pcap from disk and write it (pcap formatted) to
/dev/null (tcpdump is single threaded) at
638MBytes/sec. Which is about 5Gbits/sec.

Chances are pretty good that a RAM drive would do better, but I don't have
enough memory to do that with this pcap (yet).

I haven't seen either snort or suricata process this pcap at anything
approaching this rate on this machine yet (using an actual fairly modern
rule set, I mean).

If Gene is still reading from a RAM drive, I wouldn't think that he's
"limited to only a single packet reader thread", pcap-read-performance-wise,
unless his pcap library was compiled without optimization.

YMMV...
(And welcome back from vacation, Victor!)

Dave

-- 
"Of course, someone who knows more about this will correct me if I'm
wron g, and someone who knows less will correct me if I'm right."
David Palmer (palmer at tybalt.caltech.edu)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20110826/b303b745/attachment-0002.html>