[Oisf-users] Issue with Profiling in Suricata (Seen both in 2.0.11 and 3.0)

Victor Julien lists at inliniac.net
Mon Mar 7 13:19:43 UTC 2016


On 29-02-16 16:21, John Rett wrote:
> Sorry for the delay. After a lot of testing and reading, I'm fairly sure
> that this is expected behavior. There was quite a few different things
> going on, and it took me awhile to peal through all the layers.
> 
> Things I learned:
> 1) The profiler will only profile a rule if it matches the MPM. Only
> then will it pass through to the signature evaluation and be profiled. 

Correct.

> 2) The MPM also works on HOME_NET etc matching.

Yeah, although we call the whole process 'prefilter' and the pattern
matching part of it MPM.

> 3) My PCAP contained no packets that should match anything. But one
> stream which did match a fast_pattern.
> 
> So overall what happened is that my packet that matched the
> fast_pattern, did NOT match the directionality. So if the only rule that
> was included was rule B, then it wouldn't match the MPM because the
> directionally didn't match. If I included rule C, then the MPM would
> match the MPM because of the inclusion of $HOME_NET any -> $EXTERNAL_NET
> $HTTP_PORTS. After the MPM match, the signature match would then fail.
> 
> Does anyone know how to "disable" the MPM? I'd like to get profiling
> stats for all my rules, but with the MPM I have to create a pcap that
> (almost) matches every single rule.

It's not possible at this time.

Cheers,
Victor

> 
> Thanks!
> -JR
> 
> On Thu, Feb 25, 2016 at 7:07 PM, Peter Manev <petermanev at gmail.com
> <mailto:petermanev at gmail.com>> wrote:
> 
>     On Wed, Feb 17, 2016 at 9:45 AM, John Rett <johnarett at gmail.com
>     <mailto:johnarett at gmail.com>> wrote:
>     > Yes.
> 
>     Anything reproducible you can share? (offline if you would like)
> 
>     >
>     > On Wed, Feb 17, 2016 at 3:45 AM, Peter Manev <petermanev at gmail.com
>     <mailto:petermanev at gmail.com>> wrote:
>     >>
>     >> On Mon, Feb 8, 2016 at 9:53 PM, John Rett <johnarett at gmail.com
>     <mailto:johnarett at gmail.com>> wrote:
>     >> > I'm seeing some weird behavior from the profiling results, and I'm
>     >> > trying to
>     >> > understand if what I'm seeing is a bug, some issue with my rules (I
>     >> > doubt
>     >> > this), or some behavior that I don't understand.
>     >> >
>     >> > I have configured and built suricata with profiling
>     successfully. I'm
>     >> > getting output in my rule_perf.log.
>     >> >
>     >> > I'm running the default yaml:
>     >> > /data/suricata-3.0/src/suricata -vv -c
>     /data/suricata-3.0/suricata.yaml
>     >> > -r
>     >> > /data/my.pcap -S /data/rules_file.txt
>     >> >
>     >> > Say I have rule A, B, and C in my rules file.
>     >> > Rule A is http://doc.emergingthreats.net/2006588
>     >> > Rule B is http://doc.emergingthreats.net/2005568
>     >> > Rule C is an boring ETpro rule (Let me know if there is a
>     proper way to
>     >> > share this.)
>     >> >>
>     >> >> alert tcp $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS etc...
>     >> >
>     >> >
>     >> > If I run this rules file though a couple huge (7G and 23G)
>     pcaps of real
>     >> > large network I would expect these rules to have many "ticks"
>     and many
>     >> > "checks". But instead I get one "check" for Rule B, ~2488
>     ticks. Only
>     >> > one
>     >> > single "check" out of everything.
>     >>
>     >> If you re-run with --runmode=single would the stats be similar ?
>     >>
>     >> >
>     >> > This happens for both text output:
>     >> > http://pastebin.com/XbXMyw5J
>     >> >
>     >> > And JSON output:
>     >> >>
>     >> >>
>     >> >>
>     {"timestamp":"2016-02-08T20:09:52.066377+0000","rules":[{"signature_id":
>     >> >>
>     >> >>
>     2005568,"gid":1,"rev":5,"checks":1,"matches":0,"ticks_total":2376,"ticks_max":2376,"ticks_avg":2376,"ticks_avg_match":0,"ticks_avg_nomatch":2376,"percent":100}]}
>     >> >
>     >> >
>     >> > How could a rules file with three rules run against a huge
>     pcaps, only
>     >> > have
>     >> > a single "check" for only one of the rules?
>     >> >
>     >> > Second question/issue, maybe related, maybe not. If I reorder
>     the rules,
>     >> > I
>     >> > get the same result (expected.) If I remove rule A from the
>     list, I get
>     >> > the
>     >> > same result (expected). If I remove rule C, I get a different
>     result.
>     >> > Profiling will return nothing, aka no "check" or "ticks" for
>     any rules
>     >> > (not
>     >> > expected).
>     >> >
>     >> > For the record this happens in larger rule files too. But as I
>     add more
>     >> > rules, some of them will get checked a lot, whereas some of
>     them won't
>     >> > be
>     >> > checked at all.
>     >> >
>     >> > Let me know if I can include any other information that would be
>     >> > helpful.
>     >> >
>     >> > Many thanks for any and all help!
>     >> > -JR
>     >> >
>     >> >
>     >> > _______________________________________________
>     >> > Suricata IDS Users mailing list:
>     oisf-users at openinfosecfoundation.org
>     <mailto:oisf-users at openinfosecfoundation.org>
>     >> > Site: http://suricata-ids.org | Support:
>     >> > http://suricata-ids.org/support/
>     >> > List:
>     >> > https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>     >> > Suricata User Conference November 9-11 in Washington, DC:
>     >> > http://oisfevents.net
>     >>
>     >>
>     >>
>     >> --
>     >> Regards,
>     >> Peter Manev
>     >
>     >
> 
> 
> 
>     --
>     Regards,
>     Peter Manev
> 
> 
> 
> 
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> Suricata User Conference November 9-11 in Washington, DC: http://oisfevents.net
> 


-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------



More information about the Oisf-users mailing list