[Oisf-users] Suricata v2.1beta2 with geoip and high ram consumption
Jay M.
jskier at gmail.com
Tue Jan 6 13:54:14 UTC 2015
Correction, I meant inline IDS not IPS. I'll try that for science
anyway. What I meant was I am doing out of band monitoring only with
suricata, not using it as an inline IDS, so any blocking would be
irrelevant.
FYI, I'm up to 20 gigabytes of allocated ram this morning after
turning on the timer to reload every two hours and testing some custom
rules I did yesterday.
--
Jay
jskier at gmail.com
On Tue, Jan 6, 2015 at 2:52 AM, Peter Manev <petermanev at gmail.com> wrote:
> On Mon, Jan 5, 2015 at 5:02 PM, Jay M. <jskier at gmail.com> wrote:
>> I'm a little confused, af-packet mode is for IPS mode, correct? I'm
>> only monitoring an rspan... If I turn on af-packet, wouldn't I need a
>> second interface to forward to?
>
> You can use af_packet for IDS, no problem. Just configure the
> listening interface in suricata.yaml and then you can do:
> suricata -c /etc/suricata/suricata.yaml --af-packet=eth1 -v
>
>
> thanks
>
>>
>> Here is the config line I use:
>> ./configure --prefix=/usr --disable-gccmarch-native
>> --with-libnss-libraries=/usr/lib
>> --with-libnss-includes=/usr/include/nss/
>> --with-libnspr-libraries=/usr/lib
>> --with-libnspr-includes=/usr/include/nspr --enable-geoip
>> --with-libcap_ng-libraries=/usr/lib --sysconfdir=/etc
>> --localstatedir=/var
>>
>> --
>> Jay
>> jskier at gmail.com
>>
>>
>> On Mon, Jan 5, 2015 at 9:49 AM, Peter Manev <petermanev at gmail.com> wrote:
>>> On Mon, Jan 5, 2015 at 4:13 PM, Jay M. <jskier at gmail.com> wrote:
>>>> Whoops, not sure why I had nfqueue compiled in; I disabled that as well.
>>>>
>>>> With the two changes, I'm still at 6 - 8 GB allocated RAM out the
>>>> gate. I turned the updates back to every two hours and set timer unit
>>>> to reload instead of restart to see if I can reproduce the problem
>>>> some more.
>>>> --
>>>
>>> Hi,
>>>
>>> Some more questions/suggestions if i may:
>>> What is your configure line exactly?
>>>
>>> I see you are using pcap mode. Can you please try af_packet (with workers)?
>>> In suricata.yaml:
>>>
>>> runmode: workers
>>> Then enable af_packet in suricata.yaml for that particular listening
>>> interface and please make sure you change your starting line
>>> accordingly.
>>> You might want to adjust the number of threads in the af_packet
>>> section (depending on how many cpus you have on the VM)
>>>
>>> Would those changes have any different effect?
>>>
>>> Thanks
>>>
>>>
>>>
>>>
>>>
>>>> Jay
>>>> jskier at gmail.com
>>>>
>>>>
>>>> On Mon, Jan 5, 2015 at 7:50 AM, Jay M. <jskier at gmail.com> wrote:
>>>>> On Sun, Jan 4, 2015 at 4:10 AM, Peter Manev <petermanev at gmail.com> wrote:
>>>>>>
>>>>>> On Fri, Jan 2, 2015 at 2:48 PM, Jay M. <jskier at gmail.com> wrote:
>>>>>> > On Thu, Jan 1, 2015 at 10:15 AM, Peter Manev <petermanev at gmail.com>
>>>>>> > wrote:
>>>>>> >> On Wed, Dec 31, 2014 at 4:13 PM, Jay M. <jskier at gmail.com> wrote:
>>>>>> >>> I've been playing around a little with a geoip rule and noticed only
>>>>>> >>> when the sole one is enabled, ram is gobbled up quickly (about an
>>>>>> >>> hour) and eats into the swap with 16 gigs of ram.
>>>>>> >>>
>>>>>> >>
>>>>>> >> What is the sum total of all your mem settings in suricata.yaml?
>>>>>> >
>>>>>> > About 16.3 GB if the host memcap is kilobytes. Everything else is
>>>>>> > commented out / default. I am hashing all and do store some files,
>>>>>> > usually a handful a day.
>>>>>> >
>>>>>>
>>>>>> Ok - so you are using default yaml, correct? You have not changed
>>>>>> anything else except maybe the HOME_NET values ?
>>>>>> (just so that I can get a better idea of the set up)
>>>>>
>>>>>
>>>>> Mostly default, I upped the memcaps a little to enable hashing and file
>>>>> store, and am outputting everything to eve.log and have rule alert debugging
>>>>> and stats turned on. I'm also running suricata as it's own user and a
>>>>> specific pid file; perhaps this could impact memory management somehow?
>>>>>
>>>>>>
>>>>>>
>>>>>> > degrag memcap: 32mb
>>>>>> > flow memcap: 64mb
>>>>>> > stream memcap: 64mb
>>>>>> > stream reassembly: 128 mb
>>>>>> > host memcap: 16777216 (16 GB?)
>>>>>>
>>>>>> The value is in bytes - if not otherwise specified - aka 1000mb.
>>>>>>
>>>>>> >
>>>>>> > I have mitigated the eating in to swap problem for now by changing my
>>>>>> > rule update script to run every 6 hours and restart the daemon as
>>>>>> > opposed to reloading it (see the other caveat below). I read in the
>>>>>> > wiki that rule reloading is still in a delicate state, so this makes
>>>>>> > sense.
>>>>>> >
>>>>>> >>
>>>>>> >>> So, I've added more RAM to the VM, from 16 to 24 gigs, I'll see what
>>>>>> >>> that does (up to 15 gigs allocated after starting 40 minutes ago).
>>>>>> >>>
>>>>>> >>> It does not appear to be dropping packets and the rule is working, as
>>>>>> >>> well as the ETPRO set. I'm wondering if others using geo rules are
>>>>>> >>> also seeing this behavior? I'm not ready to call it a memory leak just
>>>>>> >>> yet...
>>>>>> >>
>>>>>>
>>>>>> You are loading a full ETPro ruleset, correct?
>>>>>
>>>>>
>>>>> Correct, full ETPro ruleset.
>>>>>
>>>>>>
>>>>>>
>>>>>> >> What amount of traffic are you inspecting?
>>>>>> >> Is this reproducible only (and every time) when you enable geoip?
>>>>>> >
>>>>>> > I am inspecting a 100 meg pipe using rspan, and am monitoring only. On
>>>>>> > my virtual host box in VMware 11, I passthru a poor man receiver so to
>>>>>> > speak, which is a 1 gig USB3 dongle. Not the most ideal setup I know,
>>>>>> > but it actually works fairly well and should hold me off until erspan
>>>>>> > span gets implemented in suricata.
>>>>>> >
>>>>>>
>>>>>> Is that 100Mb/s or 100MB/s?
>>>>>
>>>>>
>>>>> Megabits per second.
>>>>>>
>>>>>>
>>>>>> > RAM consumption is quickly reproducible with the one geoip rule
>>>>>> > (basically if not US, alert) although there is another gothca I'm
>>>>>> > looking into. I noticed my script to reload the rules every four hours
>>>>>> > by invoking the kill command (as noted in the wiki) via a systemd unit
>>>>>> > also will eat up a lot of RAM (usually 3~4 gig chunks per reload),
>>>>>>
>>>>>> Live rule reload needs twice the memory to do the rule reload (twice
>>>>>> the memory to do the reload procedure for the rulsets)
>>>>>
>>>>>
>>>>> Good to know. But, should it incrementally keep growing upon each reload?
>>>>>>
>>>>>>
>>>>>> > albeit noticeably fewer volume gobbled in time than the geoip rule. I
>>>>>> > noticed after a weekend before the geoip rule was deployed this
>>>>>> > basically killed suricata because it it ate up all the ram and swap
>>>>>> > when I was at 16/8 ram/swap respectively.
>>>>>>
>>>>>> Can you please share the output of :
>>>>>> suricata --build-info?
>>>>>
>>>>>
>>>>> This is at the bottom, second to last. Note this is after recompiling with
>>>>> your next suggestion.
>>>>>
>>>>>>
>>>>>> Since it is a virtual machine you might want to try adding
>>>>>> "--disable-gccmarch-native"to the configure line when compiling
>>>>>> Suricata.
>>>>>
>>>>>
>>>>> Done.
>>>>>
>>>>>>
>>>>>> What are the last stats in stats.log when it goes into swap?
>>>>>
>>>>>
>>>>> You may find this at the very bottom.
>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> >
>>>>>> >>>
>>>>>> >>> Additionally, running 64-bit, ArchLinux 3.17.6 kernel.
>>>>>> >>>
>>>>>> >>> --
>>>>>> >>> Jay
>>>>>> >>> jskier at gmail.com
>>>>>> >>> _______________________________________________
>>>>>> >>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>>>>>> >>> Site: http://suricata-ids.org | Support:
>>>>>> >>> http://suricata-ids.org/support/
>>>>>> >>> List:
>>>>>> >>> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>>>> >>> Training now available: http://suricata-ids.org/training/
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >> --
>>>>>> >> Regards,
>>>>>> >> Peter Manev
>>>>>> >
>>>>>> > --
>>>>>> > Jay
>>>>>> > jskier at gmail.com
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Peter Manev
>>>>>
>>>>>
>>>>> *****************************************************************************
>>>>> Build info:
>>>>> This is Suricata version 2.1beta2 RELEASE
>>>>> Features: NFQ PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET
>>>>> HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT
>>>>> HAVE_NSS HAVE_LIBJANSSON
>>>>> SIMD support: none
>>>>> Atomic intrisics: 1 2 4 8 byte(s)
>>>>> 64-bits, Little-endian architecture
>>>>> GCC version 4.9.2 20141224 (prerelease), C version 199901
>>>>> compiled with _FORTIFY_SOURCE=2
>>>>> L1 cache line size (CLS)=64
>>>>> compiled with LibHTP v0.5.15, linked against LibHTP v0.5.15
>>>>> Suricata Configuration:
>>>>> AF_PACKET support: yes
>>>>> PF_RING support: no
>>>>> NFQueue support: yes
>>>>> NFLOG support: no
>>>>> IPFW support: no
>>>>> DAG enabled: no
>>>>> Napatech enabled: no
>>>>> Unix socket enabled: yes
>>>>> Detection enabled: yes
>>>>>
>>>>> libnss support: yes
>>>>> libnspr support: yes
>>>>> libjansson support: yes
>>>>> Prelude support: no
>>>>> PCRE jit: yes
>>>>> LUA support: no
>>>>> libluajit: no
>>>>> libgeoip: yes
>>>>> Non-bundled htp: no
>>>>> Old barnyard2 support: no
>>>>> CUDA enabled: no
>>>>>
>>>>> Suricatasc install: no
>>>>>
>>>>> Unit tests enabled: no
>>>>> Debug output enabled: no
>>>>> Debug validation enabled: no
>>>>> Profiling enabled: no
>>>>> Profiling locks enabled: no
>>>>> Coccinelle / spatch: no
>>>>>
>>>>> Generic build parameters:
>>>>> Installation prefix (--prefix): /usr
>>>>> Configuration directory (--sysconfdir): /etc/suricata/
>>>>> Log directory (--localstatedir) : /var/log/suricata/
>>>>>
>>>>> Host: x86_64-unknown-linux-gnu
>>>>> GCC binary: gcc
>>>>> GCC Protect enabled: no
>>>>> GCC march native enabled: no
>>>>> GCC Profile enabled: no
>>>>>
>>>>> *****************************************************************************
>>>>> stats.log
>>>>>
>>>>> -------------------------------------------------------------------
>>>>> Date: 12/29/2014 -- 08:47:16 (uptime: 5d, 22h 11m 16s)
>>>>> -------------------------------------------------------------------
>>>>> Counter | TM Name | Value
>>>>> -------------------------------------------------------------------
>>>>> capture.kernel_packets | RxPcaprspan01 | 189319344
>>>>> capture.kernel_drops | RxPcaprspan01 | 34155
>>>>> capture.kernel_ifdrops | RxPcaprspan01 | 0
>>>>> dns.memuse | RxPcaprspan01 | 238516
>>>>> dns.memcap_state | RxPcaprspan01 | 0
>>>>> dns.memcap_global | RxPcaprspan01 | 0
>>>>> decoder.pkts | RxPcaprspan01 | 189284875
>>>>> decoder.bytes | RxPcaprspan01 | 67868253003
>>>>> decoder.invalid | RxPcaprspan01 | 8
>>>>> decoder.ipv4 | RxPcaprspan01 | 189290229
>>>>> decoder.ipv6 | RxPcaprspan01 | 2988
>>>>> decoder.ethernet | RxPcaprspan01 | 189284875
>>>>> decoder.raw | RxPcaprspan01 | 0
>>>>> decoder.sll | RxPcaprspan01 | 0
>>>>> decoder.tcp | RxPcaprspan01 | 57549996
>>>>> decoder.udp | RxPcaprspan01 | 124080607
>>>>> decoder.sctp | RxPcaprspan01 | 0
>>>>> decoder.icmpv4 | RxPcaprspan01 | 153021
>>>>> decoder.icmpv6 | RxPcaprspan01 | 36
>>>>> decoder.ppp | RxPcaprspan01 | 0
>>>>> decoder.pppoe | RxPcaprspan01 | 0
>>>>> decoder.gre | RxPcaprspan01 | 0
>>>>> decoder.vlan | RxPcaprspan01 | 0
>>>>> decoder.vlan_qinq | RxPcaprspan01 | 0
>>>>> decoder.teredo | RxPcaprspan01 | 832
>>>>> decoder.ipv4_in_ipv6 | RxPcaprspan01 | 0
>>>>> decoder.ipv6_in_ipv6 | RxPcaprspan01 | 0
>>>>> decoder.mpls | RxPcaprspan01 | 0
>>>>> decoder.avg_pkt_size | RxPcaprspan01 | 358
>>>>> decoder.max_pkt_size | RxPcaprspan01 | 1516
>>>>> defrag.ipv4.fragments | RxPcaprspan01 | 21739
>>>>> defrag.ipv4.reassembled | RxPcaprspan01 | 10857
>>>>> defrag.ipv4.timeouts | RxPcaprspan01 | 0
>>>>> defrag.ipv6.fragments | RxPcaprspan01 | 0
>>>>> defrag.ipv6.reassembled | RxPcaprspan01 | 0
>>>>> defrag.ipv6.timeouts | RxPcaprspan01 | 0
>>>>> defrag.max_frag_hits | RxPcaprspan01 | 0
>>>>> tcp.sessions | Detect | 544723
>>>>> tcp.ssn_memcap_drop | Detect | 0
>>>>> tcp.pseudo | Detect | 192120
>>>>> tcp.pseudo_failed | Detect | 0
>>>>> tcp.invalid_checksum | Detect | 0
>>>>> tcp.no_flow | Detect | 0
>>>>> tcp.reused_ssn | Detect | 124
>>>>> tcp.memuse | Detect | 379008
>>>>> tcp.syn | Detect | 566080
>>>>> tcp.synack | Detect | 510273
>>>>> tcp.rst | Detect | 210377
>>>>> dns.memuse | Detect | 303480
>>>>> dns.memcap_state | Detect | 0
>>>>> dns.memcap_global | Detect | 0
>>>>> tcp.segment_memcap_drop | Detect | 0
>>>>> tcp.stream_depth_reached | Detect | 0
>>>>> tcp.reassembly_memuse | Detect | 74263464
>>>>> tcp.reassembly_gap | Detect | 104
>>>>> http.memuse | Detect | 548522868
>>>>> http.memcap | Detect | 0
>>>>> detect.alert | Detect | 11032
>>>>> flow_mgr.closed_pruned | FlowManagerThread | 503125
>>>>> flow_mgr.new_pruned | FlowManagerThread | 53352
>>>>> flow_mgr.est_pruned | FlowManagerThread | 336649
>>>>> flow.memuse | FlowManagerThread | 12900272
>>>>> flow.spare | FlowManagerThread | 10000
>>>>> flow.emerg_mode_entered | FlowManagerThread | 0
>>>>> flow.emerg_mode_over | FlowManagerThread | 0
>>>>>
>>>>>
>>>>> --
>>>>> Jay
>>>>> jskier at gmail.com
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Peter Manev
>
>
>
> --
> Regards,
> Peter Manev
More information about the Oisf-users
mailing list