<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Sun, Jan 4, 2015 at 4:10 AM, Peter Manev <span dir="ltr"><<a href="mailto:petermanev@gmail.com" target="_blank">petermanev@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span>On Fri, Jan 2, 2015 at 2:48 PM, Jay M. <<a href="mailto:jskier@gmail.com" target="_blank">jskier@gmail.com</a>> wrote:<br>
> On Thu, Jan 1, 2015 at 10:15 AM, Peter Manev <<a href="mailto:petermanev@gmail.com" target="_blank">petermanev@gmail.com</a>> wrote:<br>
>> On Wed, Dec 31, 2014 at 4:13 PM, Jay M. <<a href="mailto:jskier@gmail.com" target="_blank">jskier@gmail.com</a>> wrote:<br>
>>> I've been playing around a little with a geoip rule and noticed only<br>
>>> when the sole one is enabled, ram is gobbled up quickly (about an<br>
>>> hour) and eats into the swap with 16 gigs of ram.<br>
>>><br>
>><br>
>> What is the sum total of all your mem settings in suricata.yaml?<br>
><br>
> About 16.3 GB if the host memcap is kilobytes. Everything else is<br>
> commented out / default. I am hashing all and do store some files,<br>
> usually a handful a day.<br>
><br>
<br>
</span>Ok - so you are using default yaml, correct? You have not changed<br>
anything else except maybe the HOME_NET values ?<br>
(just so that I can get a better idea of the set up)<br></blockquote><div><br></div><div>Mostly default, I upped the memcaps a little to enable hashing and file store, and am outputting everything to eve.log and have rule alert debugging and stats turned on. I'm also running suricata as it's own user and a specific pid file; perhaps this could impact memory management somehow? </div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<span><br>
> degrag memcap: 32mb<br>
> flow memcap: 64mb<br>
> stream memcap: 64mb<br>
> stream reassembly: 128 mb<br>
> host memcap: 16777216 (16 GB?)<br>
<br>
</span>The value is in bytes - if not otherwise specified - aka 1000mb.<br>
<span><br>
><br>
> I have mitigated the eating in to swap problem for now by changing my<br>
> rule update script to run every 6 hours and restart the daemon as<br>
> opposed to reloading it (see the other caveat below). I read in the<br>
> wiki that rule reloading is still in a delicate state, so this makes<br>
> sense.<br>
><br>
>><br>
>>> So, I've added more RAM to the VM, from 16 to 24 gigs, I'll see what<br>
>>> that does (up to 15 gigs allocated after starting 40 minutes ago).<br>
>>><br>
>>> It does not appear to be dropping packets and the rule is working, as<br>
>>> well as the ETPRO set. I'm wondering if others using geo rules are<br>
>>> also seeing this behavior? I'm not ready to call it a memory leak just<br>
>>> yet...<br>
>><br>
<br>
</span>You are loading a full ETPro ruleset, correct?<br></blockquote><div><br></div><div>Correct, full ETPro ruleset.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<span><br>
>> What amount of traffic are you inspecting?<br>
>> Is this reproducible only (and every time) when you enable geoip?<br>
><br>
> I am inspecting a 100 meg pipe using rspan, and am monitoring only. On<br>
> my virtual host box in VMware 11, I passthru a poor man receiver so to<br>
> speak, which is a 1 gig USB3 dongle. Not the most ideal setup I know,<br>
> but it actually works fairly well and should hold me off until erspan<br>
> span gets implemented in suricata.<br>
><br>
<br>
</span>Is that 100Mb/s or 100MB/s?<br></blockquote><div> </div><div>Megabits per second. <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<span><br>
> RAM consumption is quickly reproducible with the one geoip rule<br>
> (basically if not US, alert) although there is another gothca I'm<br>
> looking into. I noticed my script to reload the rules every four hours<br>
> by invoking the kill command (as noted in the wiki) via a systemd unit<br>
> also will eat up a lot of RAM (usually 3~4 gig chunks per reload),<br>
<br>
</span>Live rule reload needs twice the memory to do the rule reload (twice<br>
the memory to do the reload procedure for the rulsets)<br></blockquote><div> </div><div>Good to know. But, should it incrementally keep growing upon each reload? <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<span><br>
> albeit noticeably fewer volume gobbled in time than the geoip rule. I<br>
> noticed after a weekend before the geoip rule was deployed this<br>
> basically killed suricata because it it ate up all the ram and swap<br>
> when I was at 16/8 ram/swap respectively.<br>
<br>
</span>Can you please share the output of :<br>
suricata --build-info?<br></blockquote><div><br></div><div>This is at the bottom, second to last. Note this is after recompiling with your next suggestion.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
Since it is a virtual machine you might want to try adding<br>
"--disable-gccmarch-native"to the configure line when compiling<br>
Suricata.<br></blockquote><div><br></div><div>Done.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
What are the last stats in stats.log when it goes into swap?<br></blockquote><div><br></div><div>You may find this at the very bottom.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
Thanks<br>
<div><div><br>
><br>
>>><br>
>>> Additionally, running 64-bit, ArchLinux 3.17.6 kernel.<br>
>>><br>
>>> --<br>
>>> Jay<br>
>>> <a href="mailto:jskier@gmail.com" target="_blank">jskier@gmail.com</a><br>
>>> _______________________________________________<br>
>>> Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org" target="_blank">oisf-users@openinfosecfoundation.org</a><br>
>>> Site: <a href="http://suricata-ids.org" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" target="_blank">http://suricata-ids.org/support/</a><br>
>>> List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" target="_blank">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
>>> Training now available: <a href="http://suricata-ids.org/training/" target="_blank">http://suricata-ids.org/training/</a><br>
>><br>
>><br>
>><br>
>> --<br>
>> Regards,<br>
>> Peter Manev<br>
><br>
> --<br>
> Jay<br>
> <a href="mailto:jskier@gmail.com" target="_blank">jskier@gmail.com</a><br>
<br>
<br>
<br>
</div></div><span><font color="#888888">--<br>
Regards,<br>
Peter Manev</font></span></blockquote></div></div><div class="gmail_extra"><br></div><div class="gmail_extra">*****************************************************************************<br></div><div class="gmail_extra">Build info:</div><div class="gmail_extra"><div class="gmail_extra">This is Suricata version 2.1beta2 RELEASE</div><div class="gmail_extra">Features: NFQ PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LIBJANSSON </div><div class="gmail_extra">SIMD support: none</div><div class="gmail_extra">Atomic intrisics: 1 2 4 8 byte(s)</div><div class="gmail_extra">64-bits, Little-endian architecture</div><div class="gmail_extra">GCC version 4.9.2 20141224 (prerelease), C version 199901</div><div class="gmail_extra">compiled with _FORTIFY_SOURCE=2</div><div class="gmail_extra">L1 cache line size (CLS)=64</div><div class="gmail_extra">compiled with LibHTP v0.5.15, linked against LibHTP v0.5.15</div><div class="gmail_extra">Suricata Configuration:</div><div class="gmail_extra"> AF_PACKET support: yes</div><div class="gmail_extra"> PF_RING support: no</div><div class="gmail_extra"> NFQueue support: yes</div><div class="gmail_extra"> NFLOG support: no</div><div class="gmail_extra"> IPFW support: no</div><div class="gmail_extra"> DAG enabled: no</div><div class="gmail_extra"> Napatech enabled: no</div><div class="gmail_extra"> Unix socket enabled: yes</div><div class="gmail_extra"> Detection enabled: yes</div><div class="gmail_extra"><br></div><div class="gmail_extra"> libnss support: yes</div><div class="gmail_extra"> libnspr support: yes</div><div class="gmail_extra"> libjansson support: yes</div><div class="gmail_extra"> Prelude support: no</div><div class="gmail_extra"> PCRE jit: yes</div><div class="gmail_extra"> LUA support: no</div><div class="gmail_extra"> libluajit: no</div><div class="gmail_extra"> libgeoip: yes</div><div class="gmail_extra"> Non-bundled htp: no</div><div class="gmail_extra"> Old barnyard2 support: no</div><div class="gmail_extra"> CUDA enabled: no</div><div class="gmail_extra"><br></div><div class="gmail_extra"> Suricatasc install: no</div><div class="gmail_extra"><br></div><div class="gmail_extra"> Unit tests enabled: no</div><div class="gmail_extra"> Debug output enabled: no</div><div class="gmail_extra"> Debug validation enabled: no</div><div class="gmail_extra"> Profiling enabled: no</div><div class="gmail_extra"> Profiling locks enabled: no</div><div class="gmail_extra"> Coccinelle / spatch: no</div><div class="gmail_extra"><br></div><div class="gmail_extra">Generic build parameters:</div><div class="gmail_extra"> Installation prefix (--prefix): /usr</div><div class="gmail_extra"> Configuration directory (--sysconfdir): /etc/suricata/</div><div class="gmail_extra"> Log directory (--localstatedir) : /var/log/suricata/</div><div class="gmail_extra"><br></div><div class="gmail_extra"> Host: x86_64-unknown-linux-gnu</div><div class="gmail_extra"> GCC binary: gcc</div><div class="gmail_extra"> GCC Protect enabled: no</div><div class="gmail_extra"> GCC march native enabled: no</div><div class="gmail_extra"> GCC Profile enabled: no</div></div><div class="gmail_extra"><br></div><div class="gmail_extra">*****************************************************************************</div><div class="gmail_extra">stats.log</div><div class="gmail_extra"><br></div><div class="gmail_extra"><div class="gmail_extra">-------------------------------------------------------------------</div><div class="gmail_extra">Date: 12/29/2014 -- 08:47:16 (uptime: 5d, 22h 11m 16s)</div><div class="gmail_extra">-------------------------------------------------------------------</div><div class="gmail_extra">Counter | TM Name | Value</div><div class="gmail_extra">-------------------------------------------------------------------</div><div class="gmail_extra">capture.kernel_packets | RxPcaprspan01 | 189319344</div><div class="gmail_extra">capture.kernel_drops | RxPcaprspan01 | 34155</div><div class="gmail_extra">capture.kernel_ifdrops | RxPcaprspan01 | 0</div><div class="gmail_extra">dns.memuse | RxPcaprspan01 | 238516</div><div class="gmail_extra">dns.memcap_state | RxPcaprspan01 | 0</div><div class="gmail_extra">dns.memcap_global | RxPcaprspan01 | 0</div><div class="gmail_extra">decoder.pkts | RxPcaprspan01 | 189284875</div><div class="gmail_extra">decoder.bytes | RxPcaprspan01 | 67868253003</div><div class="gmail_extra">decoder.invalid | RxPcaprspan01 | 8</div><div class="gmail_extra">decoder.ipv4 | RxPcaprspan01 | 189290229</div><div class="gmail_extra">decoder.ipv6 | RxPcaprspan01 | 2988</div><div class="gmail_extra">decoder.ethernet | RxPcaprspan01 | 189284875</div><div class="gmail_extra">decoder.raw | RxPcaprspan01 | 0</div><div class="gmail_extra">decoder.sll | RxPcaprspan01 | 0</div><div class="gmail_extra">decoder.tcp | RxPcaprspan01 | 57549996</div><div class="gmail_extra">decoder.udp | RxPcaprspan01 | 124080607</div><div class="gmail_extra">decoder.sctp | RxPcaprspan01 | 0</div><div class="gmail_extra">decoder.icmpv4 | RxPcaprspan01 | 153021</div><div class="gmail_extra">decoder.icmpv6 | RxPcaprspan01 | 36</div><div class="gmail_extra">decoder.ppp | RxPcaprspan01 | 0</div><div class="gmail_extra">decoder.pppoe | RxPcaprspan01 | 0</div><div class="gmail_extra">decoder.gre | RxPcaprspan01 | 0</div><div class="gmail_extra">decoder.vlan | RxPcaprspan01 | 0</div><div class="gmail_extra">decoder.vlan_qinq | RxPcaprspan01 | 0</div><div class="gmail_extra">decoder.teredo | RxPcaprspan01 | 832</div><div class="gmail_extra">decoder.ipv4_in_ipv6 | RxPcaprspan01 | 0</div><div class="gmail_extra">decoder.ipv6_in_ipv6 | RxPcaprspan01 | 0</div><div class="gmail_extra">decoder.mpls | RxPcaprspan01 | 0</div><div class="gmail_extra">decoder.avg_pkt_size | RxPcaprspan01 | 358</div><div class="gmail_extra">decoder.max_pkt_size | RxPcaprspan01 | 1516</div><div class="gmail_extra">defrag.ipv4.fragments | RxPcaprspan01 | 21739</div><div class="gmail_extra">defrag.ipv4.reassembled | RxPcaprspan01 | 10857</div><div class="gmail_extra">defrag.ipv4.timeouts | RxPcaprspan01 | 0</div><div class="gmail_extra">defrag.ipv6.fragments | RxPcaprspan01 | 0</div><div class="gmail_extra">defrag.ipv6.reassembled | RxPcaprspan01 | 0</div><div class="gmail_extra">defrag.ipv6.timeouts | RxPcaprspan01 | 0</div><div class="gmail_extra">defrag.max_frag_hits | RxPcaprspan01 | 0</div><div class="gmail_extra">tcp.sessions | Detect | 544723</div><div class="gmail_extra">tcp.ssn_memcap_drop | Detect | 0</div><div class="gmail_extra">tcp.pseudo | Detect | 192120</div><div class="gmail_extra">tcp.pseudo_failed | Detect | 0</div><div class="gmail_extra">tcp.invalid_checksum | Detect | 0</div><div class="gmail_extra">tcp.no_flow | Detect | 0</div><div class="gmail_extra">tcp.reused_ssn | Detect | 124</div><div class="gmail_extra">tcp.memuse | Detect | 379008</div><div class="gmail_extra">tcp.syn | Detect | 566080</div><div class="gmail_extra">tcp.synack | Detect | 510273</div><div class="gmail_extra">tcp.rst | Detect | 210377</div><div class="gmail_extra">dns.memuse | Detect | 303480</div><div class="gmail_extra">dns.memcap_state | Detect | 0</div><div class="gmail_extra">dns.memcap_global | Detect | 0</div><div class="gmail_extra">tcp.segment_memcap_drop | Detect | 0</div><div class="gmail_extra">tcp.stream_depth_reached | Detect | 0</div><div class="gmail_extra">tcp.reassembly_memuse | Detect | 74263464</div><div class="gmail_extra">tcp.reassembly_gap | Detect | 104</div><div class="gmail_extra">http.memuse | Detect | 548522868</div><div class="gmail_extra">http.memcap | Detect | 0</div><div class="gmail_extra">detect.alert | Detect | 11032</div><div class="gmail_extra">flow_mgr.closed_pruned | FlowManagerThread | 503125</div><div class="gmail_extra">flow_mgr.new_pruned | FlowManagerThread | 53352</div><div class="gmail_extra">flow_mgr.est_pruned | FlowManagerThread | 336649</div><div class="gmail_extra">flow.memuse | FlowManagerThread | 12900272</div><div class="gmail_extra">flow.spare | FlowManagerThread | 10000</div><div class="gmail_extra">flow.emerg_mode_entered | FlowManagerThread | 0</div><div class="gmail_extra">flow.emerg_mode_over | FlowManagerThread | 0</div><div><br></div></div><div class="gmail_extra"><br clear="all"><div><div><span style="font-family:verdana,sans-serif">--</span><br style="font-family:verdana,sans-serif"><span style="font-family:verdana,sans-serif">Jay</span><br style="font-family:verdana,sans-serif"><a href="mailto:jskier@gmail.com" style="font-family:verdana,sans-serif" target="_blank">jskier@gmail.com</a></div></div></div></div>