[Oisf-users] threshold.conf with rate_limit or drop rules

Jeff Dyke jeff.dyke at gmail.com
Wed Sep 20 15:11:26 UTC 2017 

Closing the loop on this, since rate_filter indeed won't work with
threshold rules set in the *.rules files. I started to notice that
everything i wanted to apply as a drop rather than an alert were pretty
well contained in a fairly small set of rule files, so i used
oinkmaster.conf to change the entire rule file(s) to drop rather than alert
via the modifysids directive.  The one drawback of course is that you can't
put a time on this, so a brute force ssh attack is dropped when it hits the
threshold, but then is allowed to start over again a few seconds later,
which was the most appealing part of rate_filter for me.  The upside is
this is where my HIDS kicks in b/c once that IP attempts more than N times,
it is added to iptables at a increasing timeout.

Personally i think rate_filter overriding the thresholds in signatures
would be a great feature and seemingly not hard to implement (i'm not a
great C programmer), so given that fact and that it's been open for many
years, and incrementally improved, leads me to believe that there is some
complexity i'm missing in, or after, the rule building code namely
util-threshold-config.c  I can think of some complexities that would need
to be monitored by the admin employing rate_filters as i was trying to use
them, but that comes with most of this type of work anyway.

Appreciate the input/feedback, both here and on IRC, i definitely have a
greater understanding of the code/process as i've trolled through quite a
bit of the relevant parts that apply to this problem.  And most importantly
have working solution to my problem.

Jeff

On Sat, Sep 16, 2017 at 1:01 PM, Jeff Dyke <jeff.dyke at gmail.com> wrote:

> Thanks, i'm trying that now, i've upped my threshold to be an  hour and
> removed it from the rule, i'll report back.  good stuff, didn't realize i
> could remove that section from the rules, will have to brush up on that
> apparently.
>
> On Sat, Sep 16, 2017 at 12:16 PM, Peter Manev <petermanev at gmail.com>
> wrote:
>
>> On Sat, Sep 16, 2017 at 5:07 PM, Jeff Dyke <jeff.dyke at gmail.com> wrote:
>> > please all, don't get me wrong, I have had wonderful success, and
>> respect
>> > for the folks that contribute, on many mailing lists starting with
>> apache,
>> > mysql in the 90's and contributing as possible.  I like to follow OSS
>> > project mailing lists that i have an active interest in, i won't list
>> them
>> > here.  That is the main reason i asked if there was another avenue, b/c
>> it
>> > seemed like a simple question, perhaps i underestimated, my apologies
>> if i
>> > was terse.
>> >
>> > I found there was a PR merged in 2016 (current memory) which updated the
>> > documentation for rate_filter, I've also been looking through the
>> source and
>> > while i'm not a C programmer, it looked well supported.  Also I now have
>> > confirmation from one of the developers, via IRC, that it is indeed
>> > supported other than bug 425.  Now to dig more into the code and see if
>> more
>> > logging can be enabled for it, as it looks functional from the stats.log
>> > output, but waiting on confirmation on that.  Hopefully i'll be able to
>> > report back some further information.  If its enough to suggest a doc
>> update
>> > i'll be sure to do that as it looks like a great feature for this
>> product
>> > and understanding it fully could help folks trying to solve similar
>> issues.
>>
>> Jeff,
>>
>> Can you try the following as a test  -
>>
>> for example for one of the ET Open rules that you had in mind -
>> sid:2019876 - remove the threshold part like so:
>> alert ssh $EXTERNAL_NET any -> $HOME_NET any (msg:"ET SCAN SSH
>> BruteForce Tool with fake PUTTY version"; flow:established,to_server;
>> ssh.softwareversion:"PUTTY"; classtype:network-scan; sid:2019876;
>> rev:4; metadata:created_at 2014_12_05, updated_at 2014_12_05;)
>>
>> and use the rate_filter in your threshold.com as you have originally
>> specified:
>> rate_filter gen_id 1, sig_id 2019876, track by_rule, count 3, seconds
>> 120, new_action drop, timeout 14400
>>
>> and confirm if it behaves as expected?
>>
>> Thank you
>>
>> >
>> > Best,
>> > Jeff
>> >
>> > On Fri, Sep 15, 2017 at 7:43 PM, Chris Boley <ilgtech75 at gmail.com>
>> wrote:
>> >>
>> >> Jeff, adding to Amar's comments; I was doing some unorthodox inline
>> >> scanning over a tagged vlan link in router-on-a-stick topology once
>> upon a
>> >> time with hsrp configurations that traversed the transparent Linux
>> bridge. I
>> >> asked about others experiences with similar scenarios and got crickets.
>> >> Sometimes we're pioneers. Sometimes the venture ends in frustration.
>> Overall
>> >> I'm thankful for this list. I love seeing everyone's ideas.
>> >> Chris
>> >>
>> >> On Fri, Sep 15, 2017 at 6:27 AM amar countersnipe.com
>> >> <amar at countersnipe.com> wrote:
>> >>>
>> >>> Jeff! Are you saying we are not good enough on this list :-)
>> >>>
>> >>> On a serious note, I do understand your point. There are many users of
>> >>> Suricata, but perhaps we could do with more contribution from some of
>> them
>> >>> in response to the questions that are posted here...just to ensure
>> that this
>> >>> list is done some justice.
>> >>>
>> >>> The whole concept of Open Source is there so that as people use the
>> >>> software, try out some new features, they can share their knowledge
>> with
>> >>> others. I think you might just be one of the leaders for rate_filter
>> trials
>> >>> :-)
>> >>>
>> >>> I do believe that documentation is open for anyone to update.
>> >>>
>> >>> Amar
>> >>>
>> >>> On September 14, 2017 at 7:25 PM Jeff Dyke <jeff.dyke at gmail.com>
>> wrote:
>> >>>
>> >>> Is there a better mailing list for suricata users only?  I feel like
>> this
>> >>> information is not covered in the documents and is glazed over in
>> both the
>> >>> rate_limit and IPS/Inline mode, sections.  and should be pretty
>> standard in
>> >>> using suricata as an IPS.  It appears not.  If you're not running an
>> >>> interface on top of all of this, like securityonion or running in the
>> cloud
>> >>> with headless servers, there are some missing portions.  I'd love to
>> help
>> >>> others out, help document, but i first need to understand.
>> >>>
>> >>> as always, thank you,
>> >>> Jeff
>> >>>
>> >>> On Thu, Sep 14, 2017 at 4:53 PM, Jeff Dyke <jeff.dyke at gmail.com>
>> wrote:
>> >>>
>> >>> Hi, i have started to notice one thing on restarts.  the RX-QN
>> notices,
>> >>>
>> >>> $> grep "(RX" /var/log/suricata/suricata.log
>> >>> 14/9/2017 -- 16:24:49 - <Notice> - (RX-Q0) Treated: Pkts 788124, Bytes
>> >>> 297444677, Errors 0
>> >>> 14/9/2017 -- 16:24:49 - <Notice> - (RX-Q0) Verdict: Accepted 787983,
>> >>> Dropped 141, Replaced 0
>> >>> 14/9/2017 -- 16:32:29 - <Notice> - (RX-Q0) Treated: Pkts 5552, Bytes
>> >>> 1534474, Errors 0
>> >>> 14/9/2017 -- 16:32:29 - <Notice> - (RX-Q0) Verdict: Accepted 5422,
>> >>> Dropped 129, Replaced 0
>> >>> 14/9/2017 -- 16:34:18 - <Notice> - (RX-Q0) Treated: Pkts 1658, Bytes
>> >>> 438573, Errors 0
>> >>> 14/9/2017 -- 16:34:18 - <Notice> - (RX-Q0) Verdict: Accepted 1633,
>> >>> Dropped 24, Replaced 0
>> >>> 14/9/2017 -- 20:45:51 - <Notice> - (RX-Q0) Treated: Pkts 176445, Bytes
>> >>> 62762358, Errors 0
>> >>> 14/9/2017 -- 20:45:51 - <Notice> - (RX-Q0) Verdict: Accepted 176220,
>> >>> Dropped 223, Replaced 0
>> >>>
>> >>> Are these indeed properly dropped packets, hopefully from my
>> rate_filter
>> >>> rules?  If so, i'm not seeing logs of what has been dropped to
>> confirm i'm
>> >>> dropping the correct packets, how would i enabled this.  I have drop
>> enabled
>> >>> in eve-ips.json, but since these are not drop rules i assume they
>> would go
>> >>> elsewhere???
>> >>>
>> >>> Thanks,
>> >>> Jeff
>> >>>
>> >>> On Wed, Sep 13, 2017 at 6:45 PM, Jeff Dyke <jeff.dyke at gmail.com>
>> wrote:
>> >>>
>> >>> Thanks Amar.  I could do this in sshd, but those rules are OK by me
>> b/c i
>> >>> don't allow passwords, nor do i allow root login, you MUST have a
>> valid key,
>> >>> so the defaults are OK, but you're correct i could reduce them to say
>> 2 (at
>> >>> home, I accidentally ssh'd with the wrong/default key, then let me
>> pass -i
>> >>> identity.foo for login).  To me writing the iptables rules is the
>> same as
>> >>> leaving drops in OSSEC b/c it will learn from its sid's and auth.log
>> and
>> >>> prevent it from happening again, and do so on my time schedule
>> (currently
>> >>> 4h, 12h, 1d, 1 week).  I'm actually OK leaving them in OSSEC, but
>> when i get
>> >>> a single ip trying to login and fail over a long period of time, i'd
>> like to
>> >>> drop them, and my idea was to do so with suricata.
>> >>>
>> >>> While this is not my question, i'm seeing these as event_type: "ssh",
>> not
>> >>> event_type: "alert", my desire is to make this learn from repeated
>> failed
>> >>> attempts that are outside the bounds of the alert sids, that i have
>> tried to
>> >>> override in threshold.config.  again, not my question, i can open a
>> new
>> >>> thread.
>> >>>
>> >>> Appreciate the comments, i hope the response points to what i am
>> trying
>> >>> to accomplish with suricata, which i've really appreciated since
>> taking the
>> >>> time to learn how to do most of it correctly.   Its the drops and
>> overrides
>> >>> i continue to struggle with.
>> >>>
>> >>> Jeff
>> >>>
>> >>> On Wed, Sep 13, 2017 at 6:02 PM, amar countersnipe.com
>> >>> <amar at countersnipe.com> wrote:
>> >>>
>> >>> Hi Jeff
>> >>>
>> >>> Just an initial thought. Most of the services can be controlled via
>> the
>> >>> service daemons. For example ssh login frequency can be controlled via
>> >>> sshd_config using
>> >>>
>> >>> MaxAuthTries X(number you are happy with) or using MaxStartups
>> >>>
>> >>> Alternatively, since you are using iptables, an even better approach
>> >>> would be to use iptables with something like:
>> >>>
>> >>> iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent
>> >>> --update --seconds X --hitcount Y --rttl --name SSH -j DROP
>> >>>
>> >>> Once you have eliminated the basic problem then you could get
>> Suricata to
>> >>> inspect deeper.
>> >>>
>> >>> Hope I didn't just miss the requirement in the first place.
>> >>>
>> >>>
>> >>> regards
>> >>>
>> >>> Amar Rathore
>> >>> CounterSnipe - Suricata based IDS/IPS with so much more.
>> >>>
>> >>> On September 13, 2017 at 3:57 PM Jeff Dyke <jeff.dyke at gmail.com>
>> wrote:
>> >>>
>> >>> I should have stated that i'm successfully attached to NFQUEUE in
>> >>> inline/IPS mode.  <Info> - NFQ running in standard ACCEPT/DROP mode.
>> >>>
>> >>> On Wed, Sep 13, 2017 at 3:53 PM, Jeff Dyke <jeff.dyke at gmail.com>
>> wrote:
>> >>>
>> >>> i am running an array of servers on aws (EC2 instances), one server in
>> >>> both the staging and production environments has SSH open and 2 have
>> 443/80
>> >>> open (active/passive HAProxy instances)
>> >>>
>> >>> I've been using OSSEC with active-response to block malicious ssh
>> >>> attacks, and while i like the software and the other things that it
>> finds, I
>> >>> would like to move this type of logic to the edge servers, using
>> suricata.
>> >>> i'll concentrate on SSH for now, from there i can apply my knowledge
>> or
>> >>> other protocols.
>> >>>
>> >>> If i'm understanding correctly (likely not) i could add a rate_filter
>> >>> into threshold.conf, or i could add drop rules.  What is the best
>> practice
>> >>> in this instance.  I know the threshold.config is getting parsed as i
>> see
>> >>> the warning
>> >>> [ERRCODE: SC_ERR_EVENT_ENGINE(210)] - signature sid:2019876 has a
>> >>> threshold set. The signature event var is given precedence over the
>> >>> threshold.conf one. Bug #425.
>> >>>
>> >>> I'm running suricata 4.0.0 RELEASE
>> >>>
>> >>> Thanks, for any pointers.  If rate_filter is correct, how do i
>> convert it
>> >>> to a drop event when threshold is hit?  The docs are great, but i
>> seemed to
>> >>> have missed this piece.
>> >>>
>> >>> Jeff
>> >>>
>> >>> my 4 threshold.config entries.
>> >>> rate_filter gen_id 1, sig_id 2019876, track by_rule, count 3, seconds
>> >>> 120, new_action drop, timeout 14400
>> >>> rate_filter gen_id 1, sig_id 2101638, track by_rule, count 3, seconds
>> >>> 120, new_action drop, timeout 14400
>> >>> rate_filter gen_id 1, sig_id 2001219, track by_rule, count 3, seconds
>> >>> 120, new_action drop, timeout 14400
>> >>> rate_filter gen_id 1, sig_id 2006546, track by_rule, count 3, seconds
>> >>> 120, new_action drop, timeout 14400
>> >>> suppress gen_id 1, sig_id 2221002, track by_src, ip 10.0.0.0/16
>> >>>
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>> >>> Site: http://suricata-ids.org | Support:
>> http://suricata-ids.org/support/
>> >>> List: https://lists.openinfosecfoundation.org/mailman/listinfo/
>> oisf-users
>> >>>
>> >>> Conference: https://suricon.net
>> >>> Trainings: https://suricata-ids.org/training/
>> >>>
>> >>>
>> >>> Kind regards
>> >>>
>> >>> Amar Rathore
>> >>>
>> >>> CounterSnipe Systems LLC
>> >>> Tel: +1 617 701 7213
>> >>> Mobile: +44 (0) 7876 233333
>> >>> Skype ID: amarrathore
>> >>> Web: www.countersnipe.com <http://www.countersnipe.com/>
>> >>>
>> >>> This message contains confidential information and is intended only
>> for
>> >>> the individual named. If you are not the named addressee you should
>> not
>> >>> disseminate, distribute or copy this e-mail. Please notify the sender
>> >>> immediately by e-mail if you have received this e-mail by mistake and
>> delete
>> >>> this e-mail from your system.
>> >>>
>> >>> E-mail transmission cannot be guaranteed to be secure or error-free as
>> >>> information could be intercepted, corrupted, lost, destroyed, arrive
>> late or
>> >>> incomplete, or contain viruses. The sender therefore does not accept
>> >>> liability for any errors or omissions.
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>> >>> Site: http://suricata-ids.org | Support:
>> http://suricata-ids.org/support/
>> >>> List: https://lists.openinfosecfoundation.org/mailman/listinfo/
>> oisf-users
>> >>>
>> >>> Conference: https://suricon.net
>> >>> Trainings: https://suricata-ids.org/training/
>> >>>
>> >>>
>> >>> Kind regards
>> >>>
>> >>> Amar Rathore
>> >>>
>> >>> CounterSnipe Systems LLC
>> >>> Tel: +1 617 701 7213
>> >>> Mobile: +44 (0) 7876 233333
>> >>> Skype ID: amarrathore
>> >>> Web: www.countersnipe.com <http://www.countersnipe.com/>
>> >>>
>> >>> This message contains confidential information and is intended only
>> for
>> >>> the individual named. If you are not the named addressee you should
>> not
>> >>> disseminate, distribute or copy this e-mail. Please notify the sender
>> >>> immediately by e-mail if you have received this e-mail by mistake and
>> delete
>> >>> this e-mail from your system.
>> >>>
>> >>> E-mail transmission cannot be guaranteed to be secure or error-free as
>> >>> information could be intercepted, corrupted, lost, destroyed, arrive
>> late or
>> >>> incomplete, or contain viruses. The sender therefore does not accept
>> >>> liability for any errors or omissions.
>> >>>
>> >>> _______________________________________________
>> >>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>> >>> Site: http://suricata-ids.org | Support:
>> http://suricata-ids.org/support/
>> >>> List: https://lists.openinfosecfoundation.org/mailman/listinfo/
>> oisf-users
>> >>>
>> >>> Conference: https://suricon.net
>> >>> Trainings: https://suricata-ids.org/training/
>> >
>> >
>> >
>> > _______________________________________________
>> > Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>> > Site: http://suricata-ids.org | Support: http://suricata-ids.org/suppor
>> t/
>> > List: https://lists.openinfosecfoundation.org/mailman/listinfo/
>> oisf-users
>> >
>> > Conference: https://suricon.net
>> > Trainings: https://suricata-ids.org/training/
>>
>>
>>
>> --
>> Regards,
>> Peter Manev
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20170920/e84c6b0b/attachment-0002.html>

More information about the Oisf-users mailing list