Rmkml,<div> Original suricata.yaml file (original memcap settings) works fine. In fact, I was able to double them (64MB, etc) and get it to run for several hours without crashing. While watching the memory during these runs it appears that the memory utilization was increasing throughout the entire run. That leads me to believe that I'm pushing the 4GB limit imposed by the 32-bit OS. Since I don't have the time left to restart with a 64-bit OS I think I'm going to just get by with lower memcap values and deal for now. </div>
<div><br></div><div> I haven't tried to disable all sigs, but I'd imagine that would work too since it loads them into memory. I might try that if I get some more time. Thanks for the suggestions.</div><div><br>
</div><div>Gene<br><br><div class="gmail_quote">On Mon, Aug 1, 2011 at 11:29 PM, rmkml <span dir="ltr"><<a href="mailto:rmkml@yahoo.fr">rmkml@yahoo.fr</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Hi Gene,<br>
Sorry I didn't help, but Im curious if you have same pb with:<br>
-revert to original conf suricata.yaml please? (principally revert stream/memcap and stream/reassembly/memcap default value)<br>
-comment/disable all sigs<br>
Regards<br><font color="#888888">
Rmkml</font><div><div></div><div class="h5"><br>
<br>
<br>
On Mon, 1 Aug 2011, Gene Albin wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Intesting problem just occurred while trying to detemine the max memory utilization.<br>
I received a segment fault after ~9 minutes of runtime on live traffic. Since I was watching the memory utilization I can report that it used about 1GB during startup and once it started reading packets the memory utilization just<br>
continued to increase by about 1.5MB/sec until it reached ~2GB then segment fault. I thought it might have been a fluke so I ran suricata again with the same results. Seg falut at ~2GB of memory utilization. According to top the<br>
suricata process at it's peak was using about 11% of the system memory. I have 16GB allocated to the VM with 4 processors. CPU utilization was humming along at about 200% (50% on each core).<br>
<br>
For comparison I ran suricata against a 6GB pcap file I had on hand and watch the memory increase by about 1GB as well with top reporting 11.5% memory utilization, however on the pcap file it did NOT seg fault. <br>
<br>
Also of note, at around 8 and a half minutes into the live run (not the pcap one) my segment_memcap_drop counter started to increase by about 200-300 packets every 8 minutes. <br>
<br>
Any suggestions on what may have caused the segment fault? Attached is the suricata.yaml file I used during this run.<br>
<br>
Thanks,<br>
Gene<br>
<br>
On Mon, Aug 1, 2011 at 2:10 PM, Fernando Ortiz <<a href="mailto:fernando.ortiz.f@gmail.com" target="_blank">fernando.ortiz.f@gmail.com</a>> wrote:<br>
I once asked something similar:<br>
<a href="http://lists.openinfosecfoundation.org/pipermail/oisf-users/2011-June/000658.html" target="_blank">http://lists.<u></u>openinfosecfoundation.org/<u></u>pipermail/oisf-users/2011-<u></u>June/000658.html</a><br>
Just for curiosity. What is your maximum consumption of RAM while running suricata? <br>
<br>
<br>
2011/8/1 Gene Albin <<a href="mailto:gene.albin@gmail.com" target="_blank">gene.albin@gmail.com</a>><br>
So it looks like increasing the stream and flow memcap variables to 1 and 2 GB seems to have fixed the segment_memcap_drop numbers:<br>
tcp.sessions | Decode & Stream | 62179<br>
tcp.ssn_memcap_drop | Decode & Stream | 0<br>
tcp.pseudo | Decode & Stream | 10873<br>
tcp.segment_memcap_drop | Decode & Stream | 0<br>
tcp.stream_depth_reached | Decode & Stream | 347<br>
detect.alert | Detect | 715<br>
<br>
But according to (ReceivePcapThreadExitStats) I'm still losing about 20% of my packets. Any ideas on why this may be? Below is a cut from the suricata.log file showing the packet drops after I increased the memcap<br>
values.<br>
<br>
Increased Flow memcap from 32MB to 1GB<br>
No change:<br>
<br>
[11736] 1/8/2011 -- 13:27:07 - (source-pcap.c:561) <Info> (ReceivePcapThreadExitStats) -- (ReceivePcap) Packets 1784959, bytes 1318154313<br>
[11736] 1/8/2011 -- 13:27:07 - (source-pcap.c:569) <Info> (ReceivePcapThreadExitStats) -- (ReceivePcap) Pcap Total:3865595 Recv:2825319 Drop:1040276 (26.9%).<br>
<br>
Increased Stream memcap from 32MB to 1GB<br>
Increased Stream reassembly memcap from 64MB to 2GB<br>
No change:<br>
<br>
[11955] 1/8/2011 -- 13:34:38 - (source-pcap.c:561) <Info> (ReceivePcapThreadExitStats) -- (ReceivePcap) Packets 2906643, bytes 1977212962<br>
[11955] 1/8/2011 -- 13:34:38 - (source-pcap.c:569) <Info> (ReceivePcapThreadExitStats) -- (ReceivePcap) Pcap Total:5634300 Recv:4270513 Drop:1363787 (24.2%).<br>
<br>
Gene<br>
<br>
<br>
On Fri, Jul 29, 2011 at 8:17 PM, Gene Albin <<a href="mailto:gene.albin@gmail.com" target="_blank">gene.albin@gmail.com</a>> wrote:<br>
What causes the tcp.segment_memcap_drop and the tcp.ssn_memcap_drop counters to increment in the stats.log file? I haven't found much of a description or suggestions on what I can do to reduce the number. <br>
Here is a cut from my stats.log file:<br>
<br>
tcp.sessions | Decode & Stream | 569818<br>
tcp.ssn_memcap_drop | Decode & Stream | 0<br>
tcp.pseudo | Decode & Stream | 94588<br>
tcp.segment_memcap_drop | Decode & Stream | 11204200<br>
tcp.stream_depth_reached | Decode & Stream | 14<br>
detect.alert | Detect | 13239<br>
<br>
Thanks for any suggestions.<br>
<br>
Gene<br>
<br>
--<br>
Gene Albin<br>
<a href="mailto:gene.albin@gmail.com" target="_blank">gene.albin@gmail.com</a><br>
<br>
<br>
<br>
<br>
--<br>
Gene Albin<br>
<a href="mailto:gene.albin@gmail.com" target="_blank">gene.albin@gmail.com</a><br>
<br>
<br>
______________________________<u></u>_________________<br>
Oisf-users mailing list<br>
<a href="mailto:Oisf-users@openinfosecfoundation.org" target="_blank">Oisf-users@<u></u>openinfosecfoundation.org</a><br>
<a href="http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" target="_blank">http://lists.<u></u>openinfosecfoundation.org/<u></u>mailman/listinfo/oisf-users</a><br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
--<br>
Gene Albin<br>
<a href="mailto:gene.albin@gmail.com" target="_blank">gene.albin@gmail.com</a><br>
<br>
<br>
</blockquote>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>Gene Albin<br><a href="mailto:gene.albin@gmail.com" target="_blank">gene.albin@gmail.com</a><br><br>
</div>