[Oisf-users] FW: Performance of pcap-log output

[Victor Julien]

On 10/23/2012 10:47 PM, Jake Gionet wrote:
> Suricata isn't actually reporting that its dropping the packets, but
> I'll admit that I wouldn't know where to look if it isn't as apparent
> as reading the stats.log values that include "drop" in the name.  When
> I go back and read the pcaps that have been written, tcpdump reports
> large gaps in the data.  I'm currently writing 1000 MB files that when
> re-read report ~1500 MB of missing data.  If I set up a ring buffer
> with tcpdump it appears to capture more data (the files roll almost
> twice as fast) and it does not report any missing data while
> re-reading the pcaps.
> 
> 
> On Tue, Oct 23, 2012 at 2:39 PM, Martin Holste <mcholste at gmail.com> wrote:
>> Jake,
>>
>> That seems very low indeed, though I don't use Suricata for collecting pcap.
>>
>> At anything less than a Gb/sec, the hard drive speeds will almost
>> certainly not be a factor.  HD's write and read at rates not seen on
>> normal networks during normal operation.  Modern SATA will
>> theoretically write at 6 GB/sec, which is 48 Gb/sec.  Until you're
>> pushing more than 1 Gb/sec, your bottleneck will be the pcap interface
>> at all times, even if you have very slow disk.
>>
>> Do you get drops when you have disk writing turned off?
>>
> 
> 
> I'm seeing peaks of ~500 Mb/s but for the most part is stays under 120
> Mb/s with an average of 85 Mb/s.  I'm using version 1.4beta2 of
> Suricata.
> I had actually planned on using that article as a guideline for this
> device once I had packet capturing covered.  Unfortunately my NIC
> isn't hashing the flows properly for RSS and it cannot be manually
> configured to do it, otherwise I would have set it up almost
> identically to that system.

The current implementation of the pcap recording is far from optimal. We
just use a single dumper with a big lock to ensure thread safety.

How much performance is possible with it I don't know, but I would
recommend tuning suricata without the pcap logging first, then after you
have it running properly try to enable it again.

Cheers,
Victor

> 
> On Tue, Oct 23, 2012 at 2:27 PM, Peter Manev <petermanev at gmail.com> wrote:
>> Hi,
>>
>> The speeds that you are achieving are very low, almost impossible :).
>> Please have a look here (although it uses advanced techniques for your
>> network card drivers and such, it will be helpful to set up your
>> suricata.yaml):
>> https://home.regit.org/2012/07/suricata-to-10gbps-and-beyond/
>>
>> What speeds are you looking at .. on your network interface?
>> Which version of Suricata are you using?
>>
>> I am not sure about pcap-log throughput, but i am assuming it will be hugely
>> dependent on your HDD speed as well.
>>
>> thanks
>>
>>
>> Regards,
>> Peter Manev
>>
> _______________________________________________
> Oisf-users mailing list
> Oisf-users at openinfosecfoundation.org
> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> 


-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------