[Oisf-users] Clarification on dropped packet counters

[Will Metcalf]

Gene,

Well there is an upside and a down side... upside is with this patch
you should get accurate drop counters in pcap mode. downside is that
the logic was wrong so you will more than likely see an increase in
drops. You can try fiddling with --pcap-buffer-size option (pcap
buffer size in bytes), also try increasing the max-pending-packets
option in the suricata.yaml from 50 to 500 or maybe 5000.

with regards to tcp.segment_memcap_drop counter, you have two choices,
decrease the amount of data you are performing reassembly on by
turning down depth, or by turning up the reassembly memcap in the
section of the suricata.yaml displayed below.


stream:
  memcap: 33554432              # 32mb
  checksum_validation: yes      # reject wrong csums
  inline: no                    # no inline mode
  reassembly:
    memcap: 67108864            # 64mb for reassembly
    depth: 1048576              # reassemble 1mb into a stream
    toserver_chunk_size: 2560
    toclient_chunk_size: 2560

Regards,

Will

On Tue, Aug 9, 2011 at 10:15 PM, Will Metcalf <william.metcalf at gmail.com> wrote:
> Hmmmm.... it looks like once upon a time I screwed pcap stats
> calculation... Patch on the way...
>
> Regards,
>
> Will
>
> On Tue, Aug 9, 2011 at 8:37 PM, Gene Albin <gene.albin at gmail.com> wrote:
>> So it turns out that my CentOS 5.6 server with the default kernel network
>> settings is not optimal for an IDS connected to a high speed network.  One
>> of my problems was that the kernel couldn't keep up with the flow of
>> traffic.  So I made the following changes to my kernel:
>>
>> sysctl -w net.core.netdev_max_backlog=10000
>>
>> sysctl -w net.core.rmem_devault=16777216
>>
>> sysctl -w net.core.rmem_max=33554432
>>
>> sysctl -w net.ipv4.tcp_mem=’194688 259584 389376’
>>
>> sysctl -w net.ipv4.tcp_rmem=’1048576 4194304 33554432’
>>
>> sysctl -w net.ipv4.tcp_no_metrics_save=1
>>
>> Now when I run tcpdump I get 0 dropped packets after several minutes, and
>> after running Suricata for about 15 minutes my suricata.log drops were down
>> to 3.9%. Much better than the 27% I had been seeing.
>> Further, looking at the stats.log file my tcp.ssn_memcap_drop number is at 0
>> for the same run.  Unfortunately the tcp.segment_memcap_drop number is still
>> high at 2938343 (out of 14754737 packets)
>> So even though I've minimized my drops, I'm still uncertain about the
>> metrics listed in my original post.
>> Gene
>> On Tue, Aug 9, 2011 at 2:38 PM, Gene Albin <gene.albin at gmail.com> wrote:
>>>
>>> I'm trying to make sense out of the various packet metrics in the
>>> suricata.log and stats.log files.  Can anyone shed light on what
>>> specifically each of these counters is measuring?
>>> suricata.log:
>>> [4947] 8/8/2011 -- 14:50:31 - (source-pcap.c:561) <Info>
>>> (ReceivePcapThreadExitStats) -- (ReceivePcap) Packets 238097983, bytes
>>> 182382168249
>>> [4947] 8/8/2011 -- 14:50:31 - (source-pcap.c:569) <Info>
>>> (ReceivePcapThreadExitStats) -- (ReceivePcap) Pcap Total:539841804
>>> Recv:388969943 Drop:150871861 (27.9%).
>>> Looking at these two lines from suricata.log it looks like the pcap engine
>>> received a total of 238 million packets AND 388 million packets.  Also,
>>> notice how the difference between 539M and 388M is 150M AND the difference
>>> between 388M and 238M is also 150M.  I checked another set of suricata.log
>>> and stats.log files I have and found that this relationship between Recv and
>>> Drop, and Packets and Drop appears the be the same in that file.
>>> What specifically are each of these metrics measuring and from where are
>>> the measurements taken (nic, pcap, suricata)?
>>> What is the relationship between these numbers?
>>> Stats.log:
>>> decoder.pkts              | Decode & Stream   | 238097982
>>> tcp.ssn_memcap_drop       | Decode & Stream   | 299435
>>> tcp.segment_memcap_drop   | Decode & Stream   | 31445861
>>> In stats.log the decoder.pkts line matches up with the (ReceivePcap)
>>> Packets line in the suricata.log file.  What about these memcap drop lines?
>>>  They don't seem to match up with the drop counter in suricata.log leading
>>> me to believe that these are packets dropped by Suricata and are independent
>>> of the ones in the suricata.log file.
>>> Sure would appreciate any insight into the differences between these
>>> metrics.  I'm just a bit confused.
>>> Thanks,
>>> --
>>> Gene Albin
>>> gene.albin at gmail.com
>>>
>>
>>
>>
>> --
>> Gene Albin
>> gene.albin at gmail.com
>>
>>
>> _______________________________________________
>> Oisf-users mailing list
>> Oisf-users at openinfosecfoundation.org
>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>
>>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Fix-for-silly-pcap-counters-mistake-made-by-me.-ps_r.patch
Type: application/octet-stream
Size: 2008 bytes
Desc: not available
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20110810/61f117a9/attachment.obj>