Intesting problem just occurred while trying to detemine the max memory utilization.<div><br></div><div>I received a segment fault after ~9 minutes of runtime on live traffic. Since I was watching the memory utilization I can report that it used about 1GB during startup and once it started reading packets the memory utilization just continued to increase by about 1.5MB/sec until it reached ~2GB then segment fault. I thought it might have been a fluke so I ran suricata again with the same results. Seg falut at ~2GB of memory utilization. According to top the suricata process at it's peak was using about 11% of the system memory. I have 16GB allocated to the VM with 4 processors. CPU utilization was humming along at about 200% (50% on each core).</div>
<div><br></div><div>For comparison I ran suricata against a 6GB pcap file I had on hand and watch the memory increase by about 1GB as well with top reporting 11.5% memory utilization, however on the pcap file it did NOT seg fault. </div>
<div><br></div><div><meta http-equiv="content-type" content="text/html; charset=utf-8">Also of note, at around 8 and a half minutes into the live run (not the pcap one) my segment_memcap_drop counter started to increase by about 200-300 packets every 8 minutes. </div>
<div><br></div><div>Any suggestions on what may have caused the segment fault? Attached is the suricata.yaml file I used during this run.</div><div><br></div><div>Thanks,</div><div>Gene<br><br><div class="gmail_quote">On Mon, Aug 1, 2011 at 2:10 PM, Fernando Ortiz <span dir="ltr"><<a href="mailto:fernando.ortiz.f@gmail.com">fernando.ortiz.f@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div>I once asked something similar:</div><a href="http://lists.openinfosecfoundation.org/pipermail/oisf-users/2011-June/000658.html" target="_blank">http://lists.openinfosecfoundation.org/pipermail/oisf-users/2011-June/000658.html</a><div>
<br></div><div>Just for curiosity. What is your maximum consumption of RAM while running suricata? </div><div><br><br><div class="gmail_quote"><div><div></div><div class="h5">2011/8/1 Gene Albin <span dir="ltr"><<a href="mailto:gene.albin@gmail.com" target="_blank">gene.albin@gmail.com</a>></span><br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div></div><div class="h5">
So it looks like increasing the stream and flow memcap variables to 1 and 2 GB seems to have fixed the segment_memcap_drop numbers:<div><br></div><div><div><font face="'courier new', monospace">tcp.sessions | Decode & Stream | 62179</font></div>
<div>
<div><font face="'courier new', monospace">tcp.ssn_memcap_drop | Decode & Stream | 0</font></div></div><div><font face="'courier new', monospace">tcp.pseudo | Decode & Stream | 10873</font></div>
<div><font face="'courier new', monospace">tcp.segment_memcap_drop | Decode & Stream | 0</font></div><div><font face="'courier new', monospace">tcp.stream_depth_reached | Decode & Stream | 347</font></div>
<div><font face="'courier new', monospace">detect.alert | Detect | 715</font></div></div><div><br></div><div>But according to (ReceivePcapThreadExitStats) I'm still losing about 20% of my packets. Any ideas on why this may be? Below is a cut from the suricata.log file showing the packet drops after I increased the memcap values.</div>
<div><br></div><div><div><font face="'courier new', monospace">Increased Flow memcap from 32MB to 1GB</font></div><div><font face="'courier new', monospace">No change:</font></div>
<div><font face="'courier new', monospace"><br></font></div><div><font face="'courier new', monospace">[11736] 1/8/2011 -- 13:27:07 - (source-pcap.c:561) <Info> (ReceivePcapThreadExitStats) -- (ReceivePcap) Packets 1784959, bytes 1318154313</font></div>
<div><font face="'courier new', monospace">[11736] 1/8/2011 -- 13:27:07 - (source-pcap.c:569) <Info> (ReceivePcapThreadExitStats) -- (ReceivePcap) Pcap Total:3865595 Recv:2825319 Drop:1040276 (26.9%).</font></div>
<div><font face="'courier new', monospace"><br></font></div><div><font face="'courier new', monospace">Increased Stream memcap from 32MB to 1GB</font></div>
<div><font face="'courier new', monospace">Increased Stream reassembly memcap from 64MB to 2GB</font></div><div><font face="'courier new', monospace">No change:</font></div>
<div><font face="'courier new', monospace"><br></font></div><div><font face="'courier new', monospace">[11955] 1/8/2011 -- 13:34:38 - (source-pcap.c:561) <Info> (ReceivePcapThreadExitStats) -- (ReceivePcap) Packets 2906643, bytes 1977212962</font></div>
<div><font face="'courier new', monospace">[11955] 1/8/2011 -- 13:34:38 - (source-pcap.c:569) <Info> (ReceivePcapThreadExitStats) -- (ReceivePcap) Pcap Total:5634300 Recv:4270513 Drop:1363787 (24.2%).</font></div>
<div><br></div><font color="#888888"><div>Gene</div></font><div><div></div><div><div><br></div><br><div class="gmail_quote">On Fri, Jul 29, 2011 at 8:17 PM, Gene Albin <span dir="ltr"><<a href="mailto:gene.albin@gmail.com" target="_blank">gene.albin@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">What causes the tcp.segment_memcap_drop and the tcp.ssn_memcap_drop counters to increment in the stats.log file? I haven't found much of a description or suggestions on what I can do to reduce the number. Here is a cut from my stats.log file:<br>
<br><span style="font-family:courier new,monospace">tcp.sessions | Decode & Stream | 569818</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace">tcp.ssn_memcap_drop | Decode & Stream | 0</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">tcp.pseudo | Decode & Stream | 94588</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace">tcp.segment_memcap_drop | Decode & Stream | 11204200</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">tcp.stream_depth_reached | Decode & Stream | 14</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace">detect.alert | Detect | 13239</span><br style="font-family:courier new,monospace">
<br>Thanks for any suggestions.<br><br>Gene<br><font color="#888888"><br>-- <br>Gene Albin<br><a href="mailto:gene.albin@gmail.com" target="_blank">gene.albin@gmail.com</a><br><br>
</font></blockquote></div><br><br clear="all"><br>-- <br>Gene Albin<br><a href="mailto:gene.albin@gmail.com" target="_blank">gene.albin@gmail.com</a><br><br>
</div></div></div>
<br></div></div><div class="im">_______________________________________________<br>
Oisf-users mailing list<br>
<a href="mailto:Oisf-users@openinfosecfoundation.org" target="_blank">Oisf-users@openinfosecfoundation.org</a><br>
<a href="http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" target="_blank">http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
<br></div></blockquote></div><br><br clear="all"><br>
</div>
</blockquote></div><br><br clear="all"><br>-- <br>Gene Albin<br><a href="mailto:gene.albin@gmail.com" target="_blank">gene.albin@gmail.com</a><br><br>
</div>