[Oisf-users] real time alert on tcp stream and flowint
Peter Manev
petermanev at gmail.com
Wed Feb 15 10:39:51 UTC 2012
On Wed, Feb 15, 2012 at 6:52 AM, Anoop Saldanha <anoopsaldanha at gmail.com>wrote:
> On Wed, Feb 15, 2012 at 11:04 AM, Nikolay Denev <ndenev at gmail.com> wrote:
> >
> > On Feb 15, 2012, at 6:51 AM, Anoop Saldanha wrote:
> >
> >> On Tue, Feb 14, 2012 at 2:59 PM, Peter Manev <petermanev at gmail.com>
> wrote:
> >>>
> >>>
> >>> On Tue, Feb 14, 2012 at 10:21 AM, Victor Julien <victor at inliniac.net>
> wrote:
> >>>>
> >>>> On 02/12/2012 08:15 AM, Nikolay Denev wrote:
> >>>>>
> >>>>> On Feb 11, 2012, at 10:11 PM, Peter Manev wrote:
> >>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Sat, Feb 11, 2012 at 8:27 PM, Nikolay Denev <ndenev at gmail.com
> >>>>>> <mailto:ndenev at gmail.com>> wrote:
> >>>>>>
> >>>>>>
> >>>>>> On Feb 11, 2012, at 9:14 PM, Peter Manev wrote:
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Sat, Feb 11, 2012 at 7:42 PM, Nikolay Denev <
> ndenev at gmail.com
> >>>>>>> <mailto:ndenev at gmail.com>> wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>> On Feb 11, 2012, at 7:52 PM, Peter Manev wrote:
> >>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Sat, Feb 11, 2012 at 4:31 PM, Nikolay Denev
> >>>>>>>> <ndenev at gmail.com <mailto:ndenev at gmail.com>> wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Feb 11, 2012, at 12:11 PM, Peter Manev wrote:
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Fri, Feb 10, 2012 at 6:43 AM, Nikolay Denev
> >>>>>>>>> <ndenev at gmail.com <mailto:ndenev at gmail.com>> wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Feb 9, 2012, at 10:04 PM, Nikolay Denev wrote:
> >>>>>>>>>
> >>>>>>>>> > On Feb 9, 2012, at 10:03 PM, Nikolay Denev
> wrote:
> >>>>>>>>> >
> >>>>>>>>> >> Hi all,
> >>>>>>>>> >>
> >>>>>>>>> >> It's probably stupid question and I'm missing
> >>>>>>>>> something but I don't seem to be able
> >>>>>>>>> >> to generate alert immediately when for
> example a
> >>>>>>>>> given string is found inside a TCP stream.
> >>>>>>>>> >> When the TCP connection closes, suricata
> >>>>>>>>> immediately prints the alert in fast.log.
> >>>>>>>>> >> How can I make the alert be generated
> >>>>>>>>> immediately when the rule condition is matched?
> >>>>>>>>> >>
> >>>>>>>>> >> Also I don't know if its because of this I
> don't
> >>>>>>>>> seem to be able to trigger the rule to match
> >>>>>>>>> several times on the same stream,
> >>>>>>>>> >> while I have the string that should fire the
> >>>>>>>>> alert several times in the stream.
> >>>>>>>>> >>
> >>>>>>>>> >> Here's an example :
> >>>>>>>>> >>
> >>>>>>>>> >> alert tcp $HOME_NET 6666 -> any any \
> >>>>>>>>> >> (msg:"got one"; content:"something";
> >>>>>>>>> flowint:something,notset; flowint:something,=,1;
> >>>>>>>>> sid:10;)
> >>>>>>>>> >>
> >>>>>>>>> >> alert tcp $HOME_NET 6666 -> any any \
> >>>>>>>>> >> (msg:"got five or more";
> >>>>>>>>> content:"something"; flowint:something,isset;
> >>>>>>>>> flowint:something,+,1; flowint:something,>,5;
> >>>>>>>>> sid:11;)
> >>>>>>>>> >>
> >>>>>>>>> >> This never works, I just have the first rule
> >>>>>>>>> fire once when the TCP session is terminated.
> >>>>>>>>> >>
> >>>>>>>>> >>
> >>>>>>>>> >> P.S.: As a side note the wiki should be
> updated
> >>>>>>>>> to include probably "sid"s for the rules, as
> >>>>>>>>> currently when I try to run the examples
> >>>>>>>>> >> suricata complains about duplicated rules.
> >>>>>>>>> >>
> >>>>>>>>> >> Thanks,
> >>>>>>>>> >>
> >>>>>>>>> >
> >>>>>>>>> > I'm running 1.2.1 RELEASE on
> FreeBSD-9.0-STABLE.
> >>>>>>>>>
> >>>>>>>>> This seems to work :
> >>>>>>>>>
> >>>>>>>>> alert tcp $HOME_NET 6666 -> any any \
> >>>>>>>>> (msg:"got one"; content:"something";
> >>>>>>>>> flowint:something,notset; flowint:something,=,1;
> >>>>>>>>> noalert; sid:10; priority: 1;)
> >>>>>>>>>
> >>>>>>>>> alert tcp $HOME_NET 6666 -> any any \
> >>>>>>>>> (msg:"got more"; content:"something";
> >>>>>>>>> flowint:something,isset; flowint:something,+,1;
> >>>>>>>>> noalert; sid:11; priority: 2;)
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> alert tcp $HOME_NET 6666 -> any any \
> >>>>>>>>> (msg:"got too many"; content:"something";
> >>>>>>>>> flowint:something,isset; flowint:something,>,2;
> >>>>>>>>> sid:12; priority: 3;)
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> _______________________________________________
> >>>>>>>>> Oisf-users mailing list
> >>>>>>>>> Oisf-users at openinfosecfoundation.org
> >>>>>>>>> <mailto:Oisf-users at openinfosecfoundation.org>
> >>>>>>>>>
> >>>>>>>>>
> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Hi Nikolay,
> >>>>>>>>> I think this is the way it is supposed to work. (last
> >>>>>>>>> example, by you).
> >>>>>>>>>
> >>>>>>>>> When you take out "noalert" form sid 11 - does it
> fire ?
> >>>>>>>>>
> >>>>>>>>> And are these the only rules that are loaded in terms
> >>>>>>>>> of flowint or you have others before that?
> >>>>>>>>>
> >>>>>>>>> thanks
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Peter Manev
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Yes, It fires, the problem I have is that it doesn't
> >>>>>>>> fire for each occurence of "content".
> >>>>>>>> Is alert supposed to fire once per packet if it
> matches,
> >>>>>>>> or for each match in the stream?
> >>>>>>>>
> >>>>>>>> For example now I'm using these rules to catch if
> there
> >>>>>>>> are more than some defined amount of email addresses
> in
> >>>>>>>> a given stream :
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> alert tcp $HOME_NET 80 -> any any \
> >>>>>>>> (msg:"got one email addr"; content:"|40|";
> >>>>>>>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \
> >>>>>>>> flow:established,from_server;
> >>>>>>>> flowint:something,notset; flowint:something,=,1;
> sid:10;
> >>>>>>>> priority:3; noalert;)
> >>>>>>>>
> >>>>>>>> alert tcp $HOME_NET 80 -> any any \
> >>>>>>>> (msg:"got more email addrs"; content:"|40|";
> >>>>>>>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \
> >>>>>>>> flow:established,from_server;
> >>>>>>>> flowint:something,isset; flowint:something,+,1;
> sid:11;
> >>>>>>>> priority:2; noalert;)
> >>>>>>>>
> >>>>>>>> alert tcp $HOME_NET 80 -> any any \
> >>>>>>>> (msg:"Got too many email addrs!";
> >>>>>>>> content:"|40|";
> >>>>>>>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \
> >>>>>>>> flow:established,from_server;
> >>>>>>>> flowint:something,isset; flowint:something,>,10;
> sid:12;
> >>>>>>>> priority:1; classtype:policy-violation;)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> This for example works, but would not match for a
> simple
> >>>>>>>> plain text file with 10 email adresses, I need to have
> >>>>>>>> maybe 40-50 or more for this to match.
> >>>>>>>> Maybe I'm missing something…
> >>>>>>>>
> >>>>>>>> And yes, these are my only rules that I'm testing
> with.
> >>>>>>>> No other rules with or without flowint whatsoever.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Hi ,
> >>>>>>>> Just so I understand you correctly - you have a text file
> >>>>>>>> (in the stream) and in that text file you have 10 e-mail
> >>>>>>>> addresses and it wold not fire. correct ?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> thanks
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Peter Manev
> >>>>>>>
> >>>>>>> Exactly.
> >>>>>>>
> >>>>>>> For example if I try to fetch the file emails.txt via http
> >>>>>>> which has the following content :
> >>>>>>>
> >>>>>>> # cat emails.txt
> >>>>>>> edin at email.com <mailto:edin at email.com>
> >>>>>>> edin at email.com <mailto:edin at email.com>
> >>>>>>> edin at email.com <mailto:edin at email.com>
> >>>>>>> edin at email.com <mailto:edin at email.com>
> >>>>>>> edin at email.com <mailto:edin at email.com>
> >>>>>>> edin at email.com <mailto:edin at email.com>
> >>>>>>> edin at email.com <mailto:edin at email.com>
> >>>>>>> edin at email.com <mailto:edin at email.com>
> >>>>>>> edin at email.com <mailto:edin at email.com>
> >>>>>>> edin at email.com <mailto:edin at email.com>
> >>>>>>>
> >>>>>>> $ curl http://testserver/emails.txt
> >>>>>>> edin at email.com <mailto:edin at email.com>
> >>>>>>> edin at email.com <mailto:edin at email.com>
> >>>>>>> edin at email.com <mailto:edin at email.com>
> >>>>>>> edin at email.com <mailto:edin at email.com>
> >>>>>>> edin at email.com <mailto:edin at email.com>
> >>>>>>> edin at email.com <mailto:edin at email.com>
> >>>>>>> edin at email.com <mailto:edin at email.com>
> >>>>>>> edin at email.com <mailto:edin at email.com>
> >>>>>>> edin at email.com <mailto:edin at email.com>
> >>>>>>> edin at email.com <mailto:edin at email.com>
> >>>>>>> $
> >>>>>>>
> >>>>>>> And I also remove the "noalert" option from the rules, this
> >>>>>>> is what I get in fast.log :
> >>>>>>>
> >>>>>>> 02/11/2012-20:37:23.988271 [**] [1:10:0] got one email
> addr
> >>>>>>> [**] [Classification: (null)] [Priority: 3] {TCP}
> X.X.X.X:80
> >>>>>>> -> Y.Y.Y.Y:57923
> >>>>>>> 02/11/2012-20:37:23.988271 [**] [1:11:0] got more email
> >>>>>>> addrs [**] [Classification: (null)] [Priority: 2] {TCP}
> >>>>>>> X.X.X.X:80 -> Y.Y.Y.Y:57923
> >>>>>>>
> >>>>>>>
> >>>>>>> If I change the third rule to fire if the flowint var is
> more
> >>>>>>> than 1, it is being triggered.
> >>>>>>>
> >>>>>>> If I insert some random data between the email addresses in
> >>>>>>> the text file, then I get 4 maybe 5 matches. Doesn't it
> have
> >>>>>>> to match all 10 of them?
> >>>>>>>
> >>>>>>>
> >>>>>>> 1. What happens if you take out the PCRE expressions from all
> >>>>>>> the rules ?
> >>>>>>> 2. sid:12 - should not fire because you have >10 , and there
> are
> >>>>>>> exactly 10 e-mails in the file
> >>>>>>> 3. how big is the stream itself? i think it is below 2KB,
> correct?
> >>>>>>> 4. is the PCRE matching the e-mails, under the unix shell ?
> >>>>>>> 5. yes i think you should get more sid:11 alerts - but first
> lets
> >>>>>>> investigate the above 4.
> >>>>>>>
> >>>>>>> thanks
> >>>>>>>
> >>>>>>> --
> >>>>>>> Peter Manev
> >>>>>>
> >>>>>> The file with only the 10 emails is 160 bytes. Even without
> pcre I
> >>>>>> get the same result :
> >>>>>>
> >>>>>> alert tcp $HOME_NET 80 -> any any \
> >>>>>> (msg:"got one email addr"; content:"|40|"; \
> >>>>>> flow:established,from_server; flowint:something,notset;
> >>>>>> flowint:something,=,1; sid:10; priority:3;)
> >>>>>>
> >>>>>> alert tcp $HOME_NET 80 -> any any \
> >>>>>> (msg:"got more email addrs"; content:"|40|"; \
> >>>>>> flow:established,from_server; flowint:something,isset;
> >>>>>> flowint:something,+,1; sid:11; priority:2;)
> >>>>>>
> >>>>>> alert tcp $HOME_NET 80 -> any any \
> >>>>>> (msg:"Got too many email addrs!"; content:"|40|"; \
> >>>>>> flow:established,from_server; flowint:something,isset;
> >>>>>> flowint:something,>,9; sid:12; priority:1;
> >>>>>> classtype:policy-violation;)
> >>>>>>
> >>>>>>
> >>>>>> alerts I get :
> >>>>>>
> >>>>>> 02/11/2012-21:23:14.567194 [**] [1:10:0] got one email addr
> [**]
> >>>>>> [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 ->
> >>>>>> Y.Y.Y.Y:58158
> >>>>>> 02/11/2012-21:23:14.567194 [**] [1:11:0] got more email addrs
> >>>>>> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 ->
> >>>>>> Y.Y.Y.Y:58158
> >>>>>>
> >>>>>> If I put some '#' symbols between the emails in the file so that
> >>>>>> it gets about 9K big and I fetch it I get these alerts :
> >>>>>>
> >>>>>> 02/11/2012-21:25:37.755214 [**] [1:10:0] got one email addr
> [**]
> >>>>>> [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 ->
> >>>>>> Y.Y.Y.Y:58166
> >>>>>> 02/11/2012-21:25:37.755214 [**] [1:11:0] got more email addrs
> >>>>>> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 ->
> >>>>>> Y.Y.Y.Y:58166
> >>>>>> 02/11/2012-21:25:37.761077 [**] [1:11:0] got more email addrs
> >>>>>> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 ->
> >>>>>> Y.Y.Y.Y:58166
> >>>>>> 02/11/2012-21:25:37.764451 [**] [1:11:0] got more email addrs
> >>>>>> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 ->
> >>>>>> Y.Y.Y.Y:58166
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Hi Nikolay,
> >>>>>>
> >>>>>>
> >>>>>> Can you please post this as a bug - please be detailed (as you were
> in
> >>>>>> your 2 previous e-mails).
> >>>>>> Personally i think here sid 11 is the problem , may be it does not
> >>>>>> count/increment correctly....
> >>>>>> thanks
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Peter Manev
> >>>>>
> >>>>> Yes I will post this as a bug. But I've just found a much simpler
> case.
> >>>>>
> >>>>> Let's for example have only this rule in suricata :
> >>>>>
> >>>>> alert tcp $HOME_NET 6666 -> any any (msg:"match"; content:"|40|";)
> >>>>>
> >>>>> Then on a monitored machine from the $HOME_NET range I do :
> >>>>>
> >>>>> echo "@ @ @ @ @ @ @ @ @" | nc -l 6666
> >>>>>
> >>>>> And on different host I do :
> >>>>>
> >>>>> nc testserver 6666
> >>>>>
> >>>>> This gets the ten @ chars transferred, and I get only one alert.
> >>>>> But for example if I echo more @ chars, like 5000 or something, I get
> >>>>> 3-6 alerts.
> >>>>> I have to check what is actually the number of packets with payload,
> >>>>> probably the rule
> >>>>> is matched once per packet? But this could not explain that I get
> >>>>> different number of alerts on different runs.
> >>>>
> >>>> The behavior is by design. TCP data by default is inspected in the
> >>>> stream context, which means the "@ @ @ @ @ @ @ @ @" buffer is
> inspected
> >>>> at once.
> >>>
> >>>
> >>>>
> >>>> Suricata will not try to find every possible match in a
> >>>> payload, but just one.
> >>>
> >>> That's good to know - clears out a few questions of mine....
> >>> but then a PCRE (matching on 10 "@") should match all of them -
> correct?
> >>> having in mind they are in the same "chunk".
> >>>
> >>
> >> If I have understood your question right, no! Pcre works just like
> >> content on the first match it finds. So alerts wise or match wise it
> >> should work the same as using content
> >>
> >
> > So this means that there is no way to count the total number of
> occurrences of a
> > given string or pattern in a flow, and alert if some predefined number
> is reached?
> > i.e. no matter the number I will get one alert per chunk?
> >
> > Something like 'g' (global match) flag for pcre? This will definitely be
> very expensive, but looks interesting as feature.
> >
>
> You won't be able to alert for every pattern in the flow, but you can
> alert once by counting the no of patterns present in the stream using
> pcre. You don't need 'g' or such feature to do so.
>
> Something like
>
> pcre:"/(kaboom.*){3}/";
>
That's what i had in mind ...
>
> should do the trick by alerting if the pattern kaboom is present in
> the string thrice.
>
> But I'd suggest avoiding pcre under most circumstances.
>
> >>>
> >>>>
> >>>>
> >>>> The reason you get more alerts if you increase the payload
> >>>> significantly, is that the stream is inspected in chunks. The size of
> >>>> those chunks is determined by your stream toserver_chunk_size setting.
> >>>>
> >>>> --
> >>>> ---------------------------------------------
> >>>> Victor Julien
> >>>> http://www.inliniac.net/
> >>>> PGP: http://www.inliniac.net/victorjulien.asc
> >>>> ---------------------------------------------
> >>>>
> >>>> _______________________________________________
> >>>> Oisf-users mailing list
> >>>> Oisf-users at openinfosecfoundation.org
> >>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> Peter Manev
> >>>
> >>> _______________________________________________
> >>> Oisf-users mailing list
> >>> Oisf-users at openinfosecfoundation.org
> >>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> >>>
> >>
> >>
> >>
> >> --
> >> Anoop Saldanha
> >> _______________________________________________
> >> Oisf-users mailing list
> >> Oisf-users at openinfosecfoundation.org
> >> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> >
>
>
>
> --
> Anoop Saldanha
>
--
Peter Manev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120215/1e430fb4/attachment-0002.html>
More information about the Oisf-users
mailing list