[Oisf-users] Issue with Profiling in Suricata (Seen both in 2.0.11 and 3.0)
Victor Julien
lists at inliniac.net
Mon Mar 7 13:19:43 UTC 2016
On 29-02-16 16:21, John Rett wrote:
> Sorry for the delay. After a lot of testing and reading, I'm fairly sure
> that this is expected behavior. There was quite a few different things
> going on, and it took me awhile to peal through all the layers.
>
> Things I learned:
> 1) The profiler will only profile a rule if it matches the MPM. Only
> then will it pass through to the signature evaluation and be profiled.
Correct.
> 2) The MPM also works on HOME_NET etc matching.
Yeah, although we call the whole process 'prefilter' and the pattern
matching part of it MPM.
> 3) My PCAP contained no packets that should match anything. But one
> stream which did match a fast_pattern.
>
> So overall what happened is that my packet that matched the
> fast_pattern, did NOT match the directionality. So if the only rule that
> was included was rule B, then it wouldn't match the MPM because the
> directionally didn't match. If I included rule C, then the MPM would
> match the MPM because of the inclusion of $HOME_NET any -> $EXTERNAL_NET
> $HTTP_PORTS. After the MPM match, the signature match would then fail.
>
> Does anyone know how to "disable" the MPM? I'd like to get profiling
> stats for all my rules, but with the MPM I have to create a pcap that
> (almost) matches every single rule.
It's not possible at this time.
Cheers,
Victor
>
> Thanks!
> -JR
>
> On Thu, Feb 25, 2016 at 7:07 PM, Peter Manev <petermanev at gmail.com
> <mailto:petermanev at gmail.com>> wrote:
>
> On Wed, Feb 17, 2016 at 9:45 AM, John Rett <johnarett at gmail.com
> <mailto:johnarett at gmail.com>> wrote:
> > Yes.
>
> Anything reproducible you can share? (offline if you would like)
>
> >
> > On Wed, Feb 17, 2016 at 3:45 AM, Peter Manev <petermanev at gmail.com
> <mailto:petermanev at gmail.com>> wrote:
> >>
> >> On Mon, Feb 8, 2016 at 9:53 PM, John Rett <johnarett at gmail.com
> <mailto:johnarett at gmail.com>> wrote:
> >> > I'm seeing some weird behavior from the profiling results, and I'm
> >> > trying to
> >> > understand if what I'm seeing is a bug, some issue with my rules (I
> >> > doubt
> >> > this), or some behavior that I don't understand.
> >> >
> >> > I have configured and built suricata with profiling
> successfully. I'm
> >> > getting output in my rule_perf.log.
> >> >
> >> > I'm running the default yaml:
> >> > /data/suricata-3.0/src/suricata -vv -c
> /data/suricata-3.0/suricata.yaml
> >> > -r
> >> > /data/my.pcap -S /data/rules_file.txt
> >> >
> >> > Say I have rule A, B, and C in my rules file.
> >> > Rule A is http://doc.emergingthreats.net/2006588
> >> > Rule B is http://doc.emergingthreats.net/2005568
> >> > Rule C is an boring ETpro rule (Let me know if there is a
> proper way to
> >> > share this.)
> >> >>
> >> >> alert tcp $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS etc...
> >> >
> >> >
> >> > If I run this rules file though a couple huge (7G and 23G)
> pcaps of real
> >> > large network I would expect these rules to have many "ticks"
> and many
> >> > "checks". But instead I get one "check" for Rule B, ~2488
> ticks. Only
> >> > one
> >> > single "check" out of everything.
> >>
> >> If you re-run with --runmode=single would the stats be similar ?
> >>
> >> >
> >> > This happens for both text output:
> >> > http://pastebin.com/XbXMyw5J
> >> >
> >> > And JSON output:
> >> >>
> >> >>
> >> >>
> {"timestamp":"2016-02-08T20:09:52.066377+0000","rules":[{"signature_id":
> >> >>
> >> >>
> 2005568,"gid":1,"rev":5,"checks":1,"matches":0,"ticks_total":2376,"ticks_max":2376,"ticks_avg":2376,"ticks_avg_match":0,"ticks_avg_nomatch":2376,"percent":100}]}
> >> >
> >> >
> >> > How could a rules file with three rules run against a huge
> pcaps, only
> >> > have
> >> > a single "check" for only one of the rules?
> >> >
> >> > Second question/issue, maybe related, maybe not. If I reorder
> the rules,
> >> > I
> >> > get the same result (expected.) If I remove rule A from the
> list, I get
> >> > the
> >> > same result (expected). If I remove rule C, I get a different
> result.
> >> > Profiling will return nothing, aka no "check" or "ticks" for
> any rules
> >> > (not
> >> > expected).
> >> >
> >> > For the record this happens in larger rule files too. But as I
> add more
> >> > rules, some of them will get checked a lot, whereas some of
> them won't
> >> > be
> >> > checked at all.
> >> >
> >> > Let me know if I can include any other information that would be
> >> > helpful.
> >> >
> >> > Many thanks for any and all help!
> >> > -JR
> >> >
> >> >
> >> > _______________________________________________
> >> > Suricata IDS Users mailing list:
> oisf-users at openinfosecfoundation.org
> <mailto:oisf-users at openinfosecfoundation.org>
> >> > Site: http://suricata-ids.org | Support:
> >> > http://suricata-ids.org/support/
> >> > List:
> >> > https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> >> > Suricata User Conference November 9-11 in Washington, DC:
> >> > http://oisfevents.net
> >>
> >>
> >>
> >> --
> >> Regards,
> >> Peter Manev
> >
> >
>
>
>
> --
> Regards,
> Peter Manev
>
>
>
>
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> Suricata User Conference November 9-11 in Washington, DC: http://oisfevents.net
>
--
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------
More information about the Oisf-users
mailing list