[Oisf-users] Questions about stats and packet drops

[Jose Vila]

I've noticed the tcp.reassembly_memuse values I'm getting are very close to
2^64, which is the maximum value the variable can get per its type
(UINT64).

Maybe there's a problem with this counter and it doesn't get successfully
decreased? I tried to look for problems in the source code but I'm not an
expert and couldn't find anything ...

On Wed, Jan 7, 2015 at 2:44 PM, Jose Vila <jovimon at gmail.com> wrote:

> Thanks Cooper for your reply.
>
> I've added more cores, reducing the drop rate below 2%. Can't add BPF
> filters as the network is heterogeneous and I want to catch as much traffic
> as possible, despite its src/dst port (I have detected some webservices in
> weird ports).
>
> I still have the same questions I posted in my first mail:
>
> * What does exactly "tcp.reassembly_memuse" mean and in which units it is
> measured? If it's measured in bytes I'm getting more than 18 Exabytes of
> memory usage !!!
>
> * I believe "tcp.segment_memcap_drop" means packets received by suricata
> (thus counted in "capture.kernel_packets") but couldn't get to the (stream
> or reassembly?) processor for further treatment. Which processor is the
> right one? How can I reduce its value?
>
> * I believe "tcp.stream_depth_reached" gets incremented each time the
> "stream.reassembly.depth" is reached, but no packets are dropped here, they
> are passed to other processors for further inspection without being
> reassembled. Is this right?
>
> * What does exactly "tcp.reassembly_gap" mean?
>
> Thank you very much,
>
> Regards,
>
> Jose Vila.
>
>
> On Sun, Jan 4, 2015 at 4:57 PM, Cooper F. Nelson <cnelson at ucsd.edu> wrote:
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Couple things you could try.
>>
>> 1.  Use all available cores (12 workers threads).
>>
>> 2.  Use a bpf filter to only monitor ports 80 and 53
>>
>> On 12/24/2014 12:37 AM, Jose Vila wrote:
>> > Hi,
>> >
>> > I'm playing around with Suricata, and want to reduce the number of
>> drops.
>> >
>> > I have 1000Mbits/s traffic and a server with 12 cores and 12GB of RAM.
>> > The objective of this sensor is to get HTTP and DNS logging and it only
>> > has a bunch of very simple rules for file extraction.
>> >
>> > I'm using PF_RING, and recently switched to "workers" runmode, which
>> > reduced my packer drop rate (capture.kernel_drop statistic) to around
>> > 5-6% with 6 worker threads.
>> >
>> > My memcaps are:
>> > defrag.memcap = 32mb
>> > flow.memcap = 256mb
>> > stream.memcap = 7gb
>> > stream.reassembly.memcap = 3gb
>> > stream.reassembly.depth = 8mb
>> >
>>
>>
>> - --
>> Cooper Nelson
>> Network Security Analyst
>> UCSD ACT Security Team
>> cnelson at ucsd.edu x41042
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v2.0.17 (MingW32)
>>
>> iQEcBAEBAgAGBQJUqWMAAAoJEKIFRYQsa8FWX30IAKTZvJbYsQLmMAXnr7z+yWhl
>> FfcXyBkwOB5SddbAQUoBEWunqAjU2VNAVyh8w/gf5kK8mGYA87iIdGYxfz1XqNK2
>> TEKqgHeYkAjCCQxtiUtYwrSHoul5slMt5HKvJg2JtVP6QchT6SwJ/srnL2n6+PSM
>> FB5q3pr4oQpqwGiwQTwQlcWYVFWOpnMXKy9w9tenbDpGmx78YJZhoZ1z7cxIbAEu
>> LyIImTu4Iou61a7i7b1o0LQiwxLViW7Ouw3QthIcl5OnKXIzD0xL3VGSuZLP/RY0
>> uv9lA1sYdZDtRsBVS1skEc/cX3akmrADbY73Inc8em4rq9Gao0F+4Cs50LUeDJc=
>> =W2mu
>> -----END PGP SIGNATURE-----
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20150107/21c1c82a/attachment-0002.html>