[Oisf-users] Issue with Profiling in Suricata (Seen both in 2.0.11 and 3.0)

[John Rett]

Sorry for the delay. After a lot of testing and reading, I'm fairly sure
that this is expected behavior. There was quite a few different things
going on, and it took me awhile to peal through all the layers.

Things I learned:
1) The profiler will only profile a rule if it matches the MPM. Only then
will it pass through to the signature evaluation and be profiled.
2) The MPM also works on HOME_NET etc matching.
3) My PCAP contained no packets that should match anything. But one stream
which did match a fast_pattern.

So overall what happened is that my packet that matched the fast_pattern,
did NOT match the directionality. So if the only rule that was included was
rule B, then it wouldn't match the MPM because the directionally didn't
match. If I included rule C, then the MPM would match the MPM because of
the inclusion of $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS. After the MPM
match, the signature match would then fail.

Does anyone know how to "disable" the MPM? I'd like to get profiling stats
for all my rules, but with the MPM I have to create a pcap that (almost)
matches every single rule.

Thanks!
-JR

On Thu, Feb 25, 2016 at 7:07 PM, Peter Manev <petermanev at gmail.com> wrote:

> On Wed, Feb 17, 2016 at 9:45 AM, John Rett <johnarett at gmail.com> wrote:
> > Yes.
>
> Anything reproducible you can share? (offline if you would like)
>
> >
> > On Wed, Feb 17, 2016 at 3:45 AM, Peter Manev <petermanev at gmail.com>
> wrote:
> >>
> >> On Mon, Feb 8, 2016 at 9:53 PM, John Rett <johnarett at gmail.com> wrote:
> >> > I'm seeing some weird behavior from the profiling results, and I'm
> >> > trying to
> >> > understand if what I'm seeing is a bug, some issue with my rules (I
> >> > doubt
> >> > this), or some behavior that I don't understand.
> >> >
> >> > I have configured and built suricata with profiling successfully. I'm
> >> > getting output in my rule_perf.log.
> >> >
> >> > I'm running the default yaml:
> >> > /data/suricata-3.0/src/suricata -vv -c
> /data/suricata-3.0/suricata.yaml
> >> > -r
> >> > /data/my.pcap -S /data/rules_file.txt
> >> >
> >> > Say I have rule A, B, and C in my rules file.
> >> > Rule A is http://doc.emergingthreats.net/2006588
> >> > Rule B is http://doc.emergingthreats.net/2005568
> >> > Rule C is an boring ETpro rule (Let me know if there is a proper way
> to
> >> > share this.)
> >> >>
> >> >> alert tcp $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS etc...
> >> >
> >> >
> >> > If I run this rules file though a couple huge (7G and 23G) pcaps of
> real
> >> > large network I would expect these rules to have many "ticks" and many
> >> > "checks". But instead I get one "check" for Rule B, ~2488 ticks. Only
> >> > one
> >> > single "check" out of everything.
> >>
> >> If you re-run with --runmode=single would the stats be similar ?
> >>
> >> >
> >> > This happens for both text output:
> >> > http://pastebin.com/XbXMyw5J
> >> >
> >> > And JSON output:
> >> >>
> >> >>
> >> >>
> {"timestamp":"2016-02-08T20:09:52.066377+0000","rules":[{"signature_id":
> >> >>
> >> >>
> 2005568,"gid":1,"rev":5,"checks":1,"matches":0,"ticks_total":2376,"ticks_max":2376,"ticks_avg":2376,"ticks_avg_match":0,"ticks_avg_nomatch":2376,"percent":100}]}
> >> >
> >> >
> >> > How could a rules file with three rules run against a huge pcaps, only
> >> > have
> >> > a single "check" for only one of the rules?
> >> >
> >> > Second question/issue, maybe related, maybe not. If I reorder the
> rules,
> >> > I
> >> > get the same result (expected.) If I remove rule A from the list, I
> get
> >> > the
> >> > same result (expected). If I remove rule C, I get a different result.
> >> > Profiling will return nothing, aka no "check" or "ticks" for any rules
> >> > (not
> >> > expected).
> >> >
> >> > For the record this happens in larger rule files too. But as I add
> more
> >> > rules, some of them will get checked a lot, whereas some of them won't
> >> > be
> >> > checked at all.
> >> >
> >> > Let me know if I can include any other information that would be
> >> > helpful.
> >> >
> >> > Many thanks for any and all help!
> >> > -JR
> >> >
> >> >
> >> > _______________________________________________
> >> > Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> >> > Site: http://suricata-ids.org | Support:
> >> > http://suricata-ids.org/support/
> >> > List:
> >> > https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> >> > Suricata User Conference November 9-11 in Washington, DC:
> >> > http://oisfevents.net
> >>
> >>
> >>
> >> --
> >> Regards,
> >> Peter Manev
> >
> >
>
>
>
> --
> Regards,
> Peter Manev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20160229/1b8370af/attachment-0002.html>