Will, I have a question. Is the number in dropped packets registered wher Suricata stops is independent of the number of packets drops by memcap_drops in stats.log?<br><br><div>Cheers, </div><div><br></div><div>Fernando<br>
<div class="gmail_quote"><br></div><div class="gmail_quote">2011/8/10 Will Metcalf <span dir="ltr"><<a href="mailto:william.metcalf@gmail.com">william.metcalf@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Gene,<br>
<br>
Well there is an upside and a down side... upside is with this patch<br>
you should get accurate drop counters in pcap mode. downside is that<br>
the logic was wrong so you will more than likely see an increase in<br>
drops. You can try fiddling with --pcap-buffer-size option (pcap<br>
buffer size in bytes), also try increasing the max-pending-packets<br>
option in the suricata.yaml from 50 to 500 or maybe 5000.<br>
<br>
with regards to tcp.segment_memcap_drop counter, you have two choices,<br>
decrease the amount of data you are performing reassembly on by<br>
turning down depth, or by turning up the reassembly memcap in the<br>
section of the suricata.yaml displayed below.<br>
<br>
<br>
stream:<br>
memcap: 33554432 # 32mb<br>
checksum_validation: yes # reject wrong csums<br>
inline: no # no inline mode<br>
reassembly:<br>
memcap: 67108864 # 64mb for reassembly<br>
depth: 1048576 # reassemble 1mb into a stream<br>
toserver_chunk_size: 2560<br>
toclient_chunk_size: 2560<br>
<br>
Regards,<br>
<font color="#888888"><br>
Will<br>
</font><div><div></div><div class="h5"><br>
On Tue, Aug 9, 2011 at 10:15 PM, Will Metcalf <<a href="mailto:william.metcalf@gmail.com">william.metcalf@gmail.com</a>> wrote:<br>
> Hmmmm.... it looks like once upon a time I screwed pcap stats<br>
> calculation... Patch on the way...<br>
><br>
> Regards,<br>
><br>
> Will<br>
><br>
> On Tue, Aug 9, 2011 at 8:37 PM, Gene Albin <<a href="mailto:gene.albin@gmail.com">gene.albin@gmail.com</a>> wrote:<br>
>> So it turns out that my CentOS 5.6 server with the default kernel network<br>
>> settings is not optimal for an IDS connected to a high speed network. One<br>
>> of my problems was that the kernel couldn't keep up with the flow of<br>
>> traffic. So I made the following changes to my kernel:<br>
>><br>
>> sysctl -w net.core.netdev_max_backlog=10000<br>
>><br>
>> sysctl -w net.core.rmem_devault=16777216<br>
>><br>
>> sysctl -w net.core.rmem_max=33554432<br>
>><br>
>> sysctl -w net.ipv4.tcp_mem=’194688 259584 389376’<br>
>><br>
>> sysctl -w net.ipv4.tcp_rmem=’1048576 4194304 33554432’<br>
>><br>
>> sysctl -w net.ipv4.tcp_no_metrics_save=1<br>
>><br>
>> Now when I run tcpdump I get 0 dropped packets after several minutes, and<br>
>> after running Suricata for about 15 minutes my suricata.log drops were down<br>
>> to 3.9%. Much better than the 27% I had been seeing.<br>
>> Further, looking at the stats.log file my tcp.ssn_memcap_drop number is at 0<br>
>> for the same run. Unfortunately the tcp.segment_memcap_drop number is still<br>
>> high at 2938343 (out of 14754737 packets)<br>
>> So even though I've minimized my drops, I'm still uncertain about the<br>
>> metrics listed in my original post.<br>
>> Gene<br>
>> On Tue, Aug 9, 2011 at 2:38 PM, Gene Albin <<a href="mailto:gene.albin@gmail.com">gene.albin@gmail.com</a>> wrote:<br>
>>><br>
>>> I'm trying to make sense out of the various packet metrics in the<br>
>>> suricata.log and stats.log files. Can anyone shed light on what<br>
>>> specifically each of these counters is measuring?<br>
>>> suricata.log:<br>
>>> [4947] 8/8/2011 -- 14:50:31 - (source-pcap.c:561) <Info><br>
>>> (ReceivePcapThreadExitStats) -- (ReceivePcap) Packets 238097983, bytes<br>
>>> 182382168249<br>
>>> [4947] 8/8/2011 -- 14:50:31 - (source-pcap.c:569) <Info><br>
>>> (ReceivePcapThreadExitStats) -- (ReceivePcap) Pcap Total:539841804<br>
>>> Recv:388969943 Drop:150871861 (27.9%).<br>
>>> Looking at these two lines from suricata.log it looks like the pcap engine<br>
>>> received a total of 238 million packets AND 388 million packets. Also,<br>
>>> notice how the difference between 539M and 388M is 150M AND the difference<br>
>>> between 388M and 238M is also 150M. I checked another set of suricata.log<br>
>>> and stats.log files I have and found that this relationship between Recv and<br>
>>> Drop, and Packets and Drop appears the be the same in that file.<br>
>>> What specifically are each of these metrics measuring and from where are<br>
>>> the measurements taken (nic, pcap, suricata)?<br>
>>> What is the relationship between these numbers?<br>
>>> Stats.log:<br>
>>> decoder.pkts | Decode & Stream | 238097982<br>
>>> tcp.ssn_memcap_drop | Decode & Stream | 299435<br>
>>> tcp.segment_memcap_drop | Decode & Stream | 31445861<br>
>>> In stats.log the decoder.pkts line matches up with the (ReceivePcap)<br>
>>> Packets line in the suricata.log file. What about these memcap drop lines?<br>
>>> They don't seem to match up with the drop counter in suricata.log leading<br>
>>> me to believe that these are packets dropped by Suricata and are independent<br>
>>> of the ones in the suricata.log file.<br>
>>> Sure would appreciate any insight into the differences between these<br>
>>> metrics. I'm just a bit confused.<br>
>>> Thanks,<br>
>>> --<br>
>>> Gene Albin<br>
>>> <a href="mailto:gene.albin@gmail.com">gene.albin@gmail.com</a><br>
>>><br>
>><br>
>><br>
>><br>
>> --<br>
>> Gene Albin<br>
>> <a href="mailto:gene.albin@gmail.com">gene.albin@gmail.com</a><br>
>><br>
>><br>
>> _______________________________________________<br>
>> Oisf-users mailing list<br>
>> <a href="mailto:Oisf-users@openinfosecfoundation.org">Oisf-users@openinfosecfoundation.org</a><br>
>> <a href="http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" target="_blank">http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
>><br>
>><br>
><br>
</div></div><br>_______________________________________________<br>
Oisf-users mailing list<br>
<a href="mailto:Oisf-users@openinfosecfoundation.org">Oisf-users@openinfosecfoundation.org</a><br>
<a href="http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" target="_blank">http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br>Fernando Ortiz <br>Twitter: <a href="http://twitter.com/FernandOrtizF">http://twitter.com/FernandOrtizF</a><br> <br>
</div>