[Oisf-users] real time alert on tcp stream and flowint
Anoop Saldanha
anoopsaldanha at gmail.com
Wed Feb 15 05:52:25 UTC 2012
On Wed, Feb 15, 2012 at 11:04 AM, Nikolay Denev <ndenev at gmail.com> wrote:
>
> On Feb 15, 2012, at 6:51 AM, Anoop Saldanha wrote:
>
>> On Tue, Feb 14, 2012 at 2:59 PM, Peter Manev <petermanev at gmail.com> wrote:
>>>
>>>
>>> On Tue, Feb 14, 2012 at 10:21 AM, Victor Julien <victor at inliniac.net> wrote:
>>>>
>>>> On 02/12/2012 08:15 AM, Nikolay Denev wrote:
>>>>>
>>>>> On Feb 11, 2012, at 10:11 PM, Peter Manev wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Sat, Feb 11, 2012 at 8:27 PM, Nikolay Denev <ndenev at gmail.com
>>>>>> <mailto:ndenev at gmail.com>> wrote:
>>>>>>
>>>>>>
>>>>>> On Feb 11, 2012, at 9:14 PM, Peter Manev wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Sat, Feb 11, 2012 at 7:42 PM, Nikolay Denev <ndenev at gmail.com
>>>>>>> <mailto:ndenev at gmail.com>> wrote:
>>>>>>>
>>>>>>>
>>>>>>> On Feb 11, 2012, at 7:52 PM, Peter Manev wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sat, Feb 11, 2012 at 4:31 PM, Nikolay Denev
>>>>>>>> <ndenev at gmail.com <mailto:ndenev at gmail.com>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On Feb 11, 2012, at 12:11 PM, Peter Manev wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Feb 10, 2012 at 6:43 AM, Nikolay Denev
>>>>>>>>> <ndenev at gmail.com <mailto:ndenev at gmail.com>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Feb 9, 2012, at 10:04 PM, Nikolay Denev wrote:
>>>>>>>>>
>>>>>>>>> > On Feb 9, 2012, at 10:03 PM, Nikolay Denev wrote:
>>>>>>>>> >
>>>>>>>>> >> Hi all,
>>>>>>>>> >>
>>>>>>>>> >> It's probably stupid question and I'm missing
>>>>>>>>> something but I don't seem to be able
>>>>>>>>> >> to generate alert immediately when for example a
>>>>>>>>> given string is found inside a TCP stream.
>>>>>>>>> >> When the TCP connection closes, suricata
>>>>>>>>> immediately prints the alert in fast.log.
>>>>>>>>> >> How can I make the alert be generated
>>>>>>>>> immediately when the rule condition is matched?
>>>>>>>>> >>
>>>>>>>>> >> Also I don't know if its because of this I don't
>>>>>>>>> seem to be able to trigger the rule to match
>>>>>>>>> several times on the same stream,
>>>>>>>>> >> while I have the string that should fire the
>>>>>>>>> alert several times in the stream.
>>>>>>>>> >>
>>>>>>>>> >> Here's an example :
>>>>>>>>> >>
>>>>>>>>> >> alert tcp $HOME_NET 6666 -> any any \
>>>>>>>>> >> (msg:"got one"; content:"something";
>>>>>>>>> flowint:something,notset; flowint:something,=,1;
>>>>>>>>> sid:10;)
>>>>>>>>> >>
>>>>>>>>> >> alert tcp $HOME_NET 6666 -> any any \
>>>>>>>>> >> (msg:"got five or more";
>>>>>>>>> content:"something"; flowint:something,isset;
>>>>>>>>> flowint:something,+,1; flowint:something,>,5;
>>>>>>>>> sid:11;)
>>>>>>>>> >>
>>>>>>>>> >> This never works, I just have the first rule
>>>>>>>>> fire once when the TCP session is terminated.
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >> P.S.: As a side note the wiki should be updated
>>>>>>>>> to include probably "sid"s for the rules, as
>>>>>>>>> currently when I try to run the examples
>>>>>>>>> >> suricata complains about duplicated rules.
>>>>>>>>> >>
>>>>>>>>> >> Thanks,
>>>>>>>>> >>
>>>>>>>>> >
>>>>>>>>> > I'm running 1.2.1 RELEASE on FreeBSD-9.0-STABLE.
>>>>>>>>>
>>>>>>>>> This seems to work :
>>>>>>>>>
>>>>>>>>> alert tcp $HOME_NET 6666 -> any any \
>>>>>>>>> (msg:"got one"; content:"something";
>>>>>>>>> flowint:something,notset; flowint:something,=,1;
>>>>>>>>> noalert; sid:10; priority: 1;)
>>>>>>>>>
>>>>>>>>> alert tcp $HOME_NET 6666 -> any any \
>>>>>>>>> (msg:"got more"; content:"something";
>>>>>>>>> flowint:something,isset; flowint:something,+,1;
>>>>>>>>> noalert; sid:11; priority: 2;)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> alert tcp $HOME_NET 6666 -> any any \
>>>>>>>>> (msg:"got too many"; content:"something";
>>>>>>>>> flowint:something,isset; flowint:something,>,2;
>>>>>>>>> sid:12; priority: 3;)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Oisf-users mailing list
>>>>>>>>> Oisf-users at openinfosecfoundation.org
>>>>>>>>> <mailto:Oisf-users at openinfosecfoundation.org>
>>>>>>>>>
>>>>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Nikolay,
>>>>>>>>> I think this is the way it is supposed to work. (last
>>>>>>>>> example, by you).
>>>>>>>>>
>>>>>>>>> When you take out "noalert" form sid 11 - does it fire ?
>>>>>>>>>
>>>>>>>>> And are these the only rules that are loaded in terms
>>>>>>>>> of flowint or you have others before that?
>>>>>>>>>
>>>>>>>>> thanks
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Peter Manev
>>>>>>>>
>>>>>>>>
>>>>>>>> Yes, It fires, the problem I have is that it doesn't
>>>>>>>> fire for each occurence of "content".
>>>>>>>> Is alert supposed to fire once per packet if it matches,
>>>>>>>> or for each match in the stream?
>>>>>>>>
>>>>>>>> For example now I'm using these rules to catch if there
>>>>>>>> are more than some defined amount of email addresses in
>>>>>>>> a given stream :
>>>>>>>>
>>>>>>>>
>>>>>>>> alert tcp $HOME_NET 80 -> any any \
>>>>>>>> (msg:"got one email addr"; content:"|40|";
>>>>>>>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \
>>>>>>>> flow:established,from_server;
>>>>>>>> flowint:something,notset; flowint:something,=,1; sid:10;
>>>>>>>> priority:3; noalert;)
>>>>>>>>
>>>>>>>> alert tcp $HOME_NET 80 -> any any \
>>>>>>>> (msg:"got more email addrs"; content:"|40|";
>>>>>>>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \
>>>>>>>> flow:established,from_server;
>>>>>>>> flowint:something,isset; flowint:something,+,1; sid:11;
>>>>>>>> priority:2; noalert;)
>>>>>>>>
>>>>>>>> alert tcp $HOME_NET 80 -> any any \
>>>>>>>> (msg:"Got too many email addrs!";
>>>>>>>> content:"|40|";
>>>>>>>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \
>>>>>>>> flow:established,from_server;
>>>>>>>> flowint:something,isset; flowint:something,>,10; sid:12;
>>>>>>>> priority:1; classtype:policy-violation;)
>>>>>>>>
>>>>>>>>
>>>>>>>> This for example works, but would not match for a simple
>>>>>>>> plain text file with 10 email adresses, I need to have
>>>>>>>> maybe 40-50 or more for this to match.
>>>>>>>> Maybe I'm missing something…
>>>>>>>>
>>>>>>>> And yes, these are my only rules that I'm testing with.
>>>>>>>> No other rules with or without flowint whatsoever.
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi ,
>>>>>>>> Just so I understand you correctly - you have a text file
>>>>>>>> (in the stream) and in that text file you have 10 e-mail
>>>>>>>> addresses and it wold not fire. correct ?
>>>>>>>>
>>>>>>>>
>>>>>>>> thanks
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Peter Manev
>>>>>>>
>>>>>>> Exactly.
>>>>>>>
>>>>>>> For example if I try to fetch the file emails.txt via http
>>>>>>> which has the following content :
>>>>>>>
>>>>>>> # cat emails.txt
>>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>>>
>>>>>>> $ curl http://testserver/emails.txt
>>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>>> $
>>>>>>>
>>>>>>> And I also remove the "noalert" option from the rules, this
>>>>>>> is what I get in fast.log :
>>>>>>>
>>>>>>> 02/11/2012-20:37:23.988271 [**] [1:10:0] got one email addr
>>>>>>> [**] [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80
>>>>>>> -> Y.Y.Y.Y:57923
>>>>>>> 02/11/2012-20:37:23.988271 [**] [1:11:0] got more email
>>>>>>> addrs [**] [Classification: (null)] [Priority: 2] {TCP}
>>>>>>> X.X.X.X:80 -> Y.Y.Y.Y:57923
>>>>>>>
>>>>>>>
>>>>>>> If I change the third rule to fire if the flowint var is more
>>>>>>> than 1, it is being triggered.
>>>>>>>
>>>>>>> If I insert some random data between the email addresses in
>>>>>>> the text file, then I get 4 maybe 5 matches. Doesn't it have
>>>>>>> to match all 10 of them?
>>>>>>>
>>>>>>>
>>>>>>> 1. What happens if you take out the PCRE expressions from all
>>>>>>> the rules ?
>>>>>>> 2. sid:12 - should not fire because you have >10 , and there are
>>>>>>> exactly 10 e-mails in the file
>>>>>>> 3. how big is the stream itself? i think it is below 2KB, correct?
>>>>>>> 4. is the PCRE matching the e-mails, under the unix shell ?
>>>>>>> 5. yes i think you should get more sid:11 alerts - but first lets
>>>>>>> investigate the above 4.
>>>>>>>
>>>>>>> thanks
>>>>>>>
>>>>>>> --
>>>>>>> Peter Manev
>>>>>>
>>>>>> The file with only the 10 emails is 160 bytes. Even without pcre I
>>>>>> get the same result :
>>>>>>
>>>>>> alert tcp $HOME_NET 80 -> any any \
>>>>>> (msg:"got one email addr"; content:"|40|"; \
>>>>>> flow:established,from_server; flowint:something,notset;
>>>>>> flowint:something,=,1; sid:10; priority:3;)
>>>>>>
>>>>>> alert tcp $HOME_NET 80 -> any any \
>>>>>> (msg:"got more email addrs"; content:"|40|"; \
>>>>>> flow:established,from_server; flowint:something,isset;
>>>>>> flowint:something,+,1; sid:11; priority:2;)
>>>>>>
>>>>>> alert tcp $HOME_NET 80 -> any any \
>>>>>> (msg:"Got too many email addrs!"; content:"|40|"; \
>>>>>> flow:established,from_server; flowint:something,isset;
>>>>>> flowint:something,>,9; sid:12; priority:1;
>>>>>> classtype:policy-violation;)
>>>>>>
>>>>>>
>>>>>> alerts I get :
>>>>>>
>>>>>> 02/11/2012-21:23:14.567194 [**] [1:10:0] got one email addr [**]
>>>>>> [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 ->
>>>>>> Y.Y.Y.Y:58158
>>>>>> 02/11/2012-21:23:14.567194 [**] [1:11:0] got more email addrs
>>>>>> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 ->
>>>>>> Y.Y.Y.Y:58158
>>>>>>
>>>>>> If I put some '#' symbols between the emails in the file so that
>>>>>> it gets about 9K big and I fetch it I get these alerts :
>>>>>>
>>>>>> 02/11/2012-21:25:37.755214 [**] [1:10:0] got one email addr [**]
>>>>>> [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 ->
>>>>>> Y.Y.Y.Y:58166
>>>>>> 02/11/2012-21:25:37.755214 [**] [1:11:0] got more email addrs
>>>>>> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 ->
>>>>>> Y.Y.Y.Y:58166
>>>>>> 02/11/2012-21:25:37.761077 [**] [1:11:0] got more email addrs
>>>>>> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 ->
>>>>>> Y.Y.Y.Y:58166
>>>>>> 02/11/2012-21:25:37.764451 [**] [1:11:0] got more email addrs
>>>>>> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 ->
>>>>>> Y.Y.Y.Y:58166
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hi Nikolay,
>>>>>>
>>>>>>
>>>>>> Can you please post this as a bug - please be detailed (as you were in
>>>>>> your 2 previous e-mails).
>>>>>> Personally i think here sid 11 is the problem , may be it does not
>>>>>> count/increment correctly....
>>>>>> thanks
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Peter Manev
>>>>>
>>>>> Yes I will post this as a bug. But I've just found a much simpler case.
>>>>>
>>>>> Let's for example have only this rule in suricata :
>>>>>
>>>>> alert tcp $HOME_NET 6666 -> any any (msg:"match"; content:"|40|";)
>>>>>
>>>>> Then on a monitored machine from the $HOME_NET range I do :
>>>>>
>>>>> echo "@ @ @ @ @ @ @ @ @" | nc -l 6666
>>>>>
>>>>> And on different host I do :
>>>>>
>>>>> nc testserver 6666
>>>>>
>>>>> This gets the ten @ chars transferred, and I get only one alert.
>>>>> But for example if I echo more @ chars, like 5000 or something, I get
>>>>> 3-6 alerts.
>>>>> I have to check what is actually the number of packets with payload,
>>>>> probably the rule
>>>>> is matched once per packet? But this could not explain that I get
>>>>> different number of alerts on different runs.
>>>>
>>>> The behavior is by design. TCP data by default is inspected in the
>>>> stream context, which means the "@ @ @ @ @ @ @ @ @" buffer is inspected
>>>> at once.
>>>
>>>
>>>>
>>>> Suricata will not try to find every possible match in a
>>>> payload, but just one.
>>>
>>> That's good to know - clears out a few questions of mine....
>>> but then a PCRE (matching on 10 "@") should match all of them - correct?
>>> having in mind they are in the same "chunk".
>>>
>>
>> If I have understood your question right, no! Pcre works just like
>> content on the first match it finds. So alerts wise or match wise it
>> should work the same as using content
>>
>
> So this means that there is no way to count the total number of occurrences of a
> given string or pattern in a flow, and alert if some predefined number is reached?
> i.e. no matter the number I will get one alert per chunk?
>
> Something like 'g' (global match) flag for pcre? This will definitely be very expensive, but looks interesting as feature.
>
You won't be able to alert for every pattern in the flow, but you can
alert once by counting the no of patterns present in the stream using
pcre. You don't need 'g' or such feature to do so.
Something like
pcre:"/(kaboom.*){3}/";
should do the trick by alerting if the pattern kaboom is present in
the string thrice.
But I'd suggest avoiding pcre under most circumstances.
>>>
>>>>
>>>>
>>>> The reason you get more alerts if you increase the payload
>>>> significantly, is that the stream is inspected in chunks. The size of
>>>> those chunks is determined by your stream toserver_chunk_size setting.
>>>>
>>>> --
>>>> ---------------------------------------------
>>>> Victor Julien
>>>> http://www.inliniac.net/
>>>> PGP: http://www.inliniac.net/victorjulien.asc
>>>> ---------------------------------------------
>>>>
>>>> _______________________________________________
>>>> Oisf-users mailing list
>>>> Oisf-users at openinfosecfoundation.org
>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>
>>>
>>>
>>>
>>> --
>>> Peter Manev
>>>
>>> _______________________________________________
>>> Oisf-users mailing list
>>> Oisf-users at openinfosecfoundation.org
>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>
>>
>>
>>
>> --
>> Anoop Saldanha
>> _______________________________________________
>> Oisf-users mailing list
>> Oisf-users at openinfosecfoundation.org
>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>
--
Anoop Saldanha
More information about the Oisf-users
mailing list