[Oisf-users] real time alert on tcp stream and flowint
Nikolay Denev
ndenev at gmail.com
Tue Feb 14 13:49:24 UTC 2012
On Feb 14, 2012, at 11:21 AM, Victor Julien wrote:
> On 02/12/2012 08:15 AM, Nikolay Denev wrote:
>>
>> On Feb 11, 2012, at 10:11 PM, Peter Manev wrote:
>>
>>>
>>>
>>> On Sat, Feb 11, 2012 at 8:27 PM, Nikolay Denev <ndenev at gmail.com
>>> <mailto:ndenev at gmail.com>> wrote:
>>>
>>>
>>> On Feb 11, 2012, at 9:14 PM, Peter Manev wrote:
>>>
>>>>
>>>>
>>>> On Sat, Feb 11, 2012 at 7:42 PM, Nikolay Denev <ndenev at gmail.com
>>>> <mailto:ndenev at gmail.com>> wrote:
>>>>
>>>>
>>>> On Feb 11, 2012, at 7:52 PM, Peter Manev wrote:
>>>>
>>>>>
>>>>>
>>>>> On Sat, Feb 11, 2012 at 4:31 PM, Nikolay Denev
>>>>> <ndenev at gmail.com <mailto:ndenev at gmail.com>> wrote:
>>>>>
>>>>>
>>>>> On Feb 11, 2012, at 12:11 PM, Peter Manev wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Feb 10, 2012 at 6:43 AM, Nikolay Denev
>>>>>> <ndenev at gmail.com <mailto:ndenev at gmail.com>> wrote:
>>>>>>
>>>>>>
>>>>>> On Feb 9, 2012, at 10:04 PM, Nikolay Denev wrote:
>>>>>>
>>>>>>> On Feb 9, 2012, at 10:03 PM, Nikolay Denev wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> It's probably stupid question and I'm missing
>>>>>> something but I don't seem to be able
>>>>>>>> to generate alert immediately when for example a
>>>>>> given string is found inside a TCP stream.
>>>>>>>> When the TCP connection closes, suricata
>>>>>> immediately prints the alert in fast.log.
>>>>>>>> How can I make the alert be generated
>>>>>> immediately when the rule condition is matched?
>>>>>>>>
>>>>>>>> Also I don't know if its because of this I don't
>>>>>> seem to be able to trigger the rule to match
>>>>>> several times on the same stream,
>>>>>>>> while I have the string that should fire the
>>>>>> alert several times in the stream.
>>>>>>>>
>>>>>>>> Here's an example :
>>>>>>>>
>>>>>>>> alert tcp $HOME_NET 6666 -> any any \
>>>>>>>> (msg:"got one"; content:"something";
>>>>>> flowint:something,notset; flowint:something,=,1;
>>>>>> sid:10;)
>>>>>>>>
>>>>>>>> alert tcp $HOME_NET 6666 -> any any \
>>>>>>>> (msg:"got five or more";
>>>>>> content:"something"; flowint:something,isset;
>>>>>> flowint:something,+,1; flowint:something,>,5; sid:11;)
>>>>>>>>
>>>>>>>> This never works, I just have the first rule
>>>>>> fire once when the TCP session is terminated.
>>>>>>>>
>>>>>>>>
>>>>>>>> P.S.: As a side note the wiki should be updated
>>>>>> to include probably "sid"s for the rules, as
>>>>>> currently when I try to run the examples
>>>>>>>> suricata complains about duplicated rules.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>
>>>>>>> I'm running 1.2.1 RELEASE on FreeBSD-9.0-STABLE.
>>>>>>
>>>>>> This seems to work :
>>>>>>
>>>>>> alert tcp $HOME_NET 6666 -> any any \
>>>>>> (msg:"got one"; content:"something";
>>>>>> flowint:something,notset; flowint:something,=,1;
>>>>>> noalert; sid:10; priority: 1;)
>>>>>>
>>>>>> alert tcp $HOME_NET 6666 -> any any \
>>>>>> (msg:"got more"; content:"something";
>>>>>> flowint:something,isset; flowint:something,+,1;
>>>>>> noalert; sid:11; priority: 2;)
>>>>>>
>>>>>>
>>>>>> alert tcp $HOME_NET 6666 -> any any \
>>>>>> (msg:"got too many"; content:"something";
>>>>>> flowint:something,isset; flowint:something,>,2;
>>>>>> sid:12; priority: 3;)
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Oisf-users mailing list
>>>>>> Oisf-users at openinfosecfoundation.org
>>>>>> <mailto:Oisf-users at openinfosecfoundation.org>
>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hi Nikolay,
>>>>>> I think this is the way it is supposed to work. (last
>>>>>> example, by you).
>>>>>>
>>>>>> When you take out "noalert" form sid 11 - does it fire ?
>>>>>>
>>>>>> And are these the only rules that are loaded in terms
>>>>>> of flowint or you have others before that?
>>>>>>
>>>>>> thanks
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Peter Manev
>>>>>
>>>>>
>>>>> Yes, It fires, the problem I have is that it doesn't
>>>>> fire for each occurence of "content".
>>>>> Is alert supposed to fire once per packet if it matches,
>>>>> or for each match in the stream?
>>>>>
>>>>> For example now I'm using these rules to catch if there
>>>>> are more than some defined amount of email addresses in
>>>>> a given stream :
>>>>>
>>>>>
>>>>> alert tcp $HOME_NET 80 -> any any \
>>>>> (msg:"got one email addr"; content:"|40|";
>>>>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \
>>>>> flow:established,from_server;
>>>>> flowint:something,notset; flowint:something,=,1; sid:10;
>>>>> priority:3; noalert;)
>>>>>
>>>>> alert tcp $HOME_NET 80 -> any any \
>>>>> (msg:"got more email addrs"; content:"|40|";
>>>>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \
>>>>> flow:established,from_server;
>>>>> flowint:something,isset; flowint:something,+,1; sid:11;
>>>>> priority:2; noalert;)
>>>>>
>>>>> alert tcp $HOME_NET 80 -> any any \
>>>>> (msg:"Got too many email addrs!";
>>>>> content:"|40|";
>>>>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \
>>>>> flow:established,from_server;
>>>>> flowint:something,isset; flowint:something,>,10; sid:12;
>>>>> priority:1; classtype:policy-violation;)
>>>>>
>>>>>
>>>>> This for example works, but would not match for a simple
>>>>> plain text file with 10 email adresses, I need to have
>>>>> maybe 40-50 or more for this to match.
>>>>> Maybe I'm missing something…
>>>>>
>>>>> And yes, these are my only rules that I'm testing with.
>>>>> No other rules with or without flowint whatsoever.
>>>>>
>>>>>
>>>>> Hi ,
>>>>> Just so I understand you correctly - you have a text file
>>>>> (in the stream) and in that text file you have 10 e-mail
>>>>> addresses and it wold not fire. correct ?
>>>>>
>>>>>
>>>>> thanks
>>>>>
>>>>>
>>>>> --
>>>>> Peter Manev
>>>>
>>>> Exactly.
>>>>
>>>> For example if I try to fetch the file emails.txt via http
>>>> which has the following content :
>>>>
>>>> # cat emails.txt
>>>> edin at email.com <mailto:edin at email.com>
>>>> edin at email.com <mailto:edin at email.com>
>>>> edin at email.com <mailto:edin at email.com>
>>>> edin at email.com <mailto:edin at email.com>
>>>> edin at email.com <mailto:edin at email.com>
>>>> edin at email.com <mailto:edin at email.com>
>>>> edin at email.com <mailto:edin at email.com>
>>>> edin at email.com <mailto:edin at email.com>
>>>> edin at email.com <mailto:edin at email.com>
>>>> edin at email.com <mailto:edin at email.com>
>>>>
>>>> $ curl http://testserver/emails.txt
>>>> edin at email.com <mailto:edin at email.com>
>>>> edin at email.com <mailto:edin at email.com>
>>>> edin at email.com <mailto:edin at email.com>
>>>> edin at email.com <mailto:edin at email.com>
>>>> edin at email.com <mailto:edin at email.com>
>>>> edin at email.com <mailto:edin at email.com>
>>>> edin at email.com <mailto:edin at email.com>
>>>> edin at email.com <mailto:edin at email.com>
>>>> edin at email.com <mailto:edin at email.com>
>>>> edin at email.com <mailto:edin at email.com>
>>>> $
>>>>
>>>> And I also remove the "noalert" option from the rules, this
>>>> is what I get in fast.log :
>>>>
>>>> 02/11/2012-20:37:23.988271 [**] [1:10:0] got one email addr
>>>> [**] [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80
>>>> -> Y.Y.Y.Y:57923
>>>> 02/11/2012-20:37:23.988271 [**] [1:11:0] got more email
>>>> addrs [**] [Classification: (null)] [Priority: 2] {TCP}
>>>> X.X.X.X:80 -> Y.Y.Y.Y:57923
>>>>
>>>>
>>>> If I change the third rule to fire if the flowint var is more
>>>> than 1, it is being triggered.
>>>>
>>>> If I insert some random data between the email addresses in
>>>> the text file, then I get 4 maybe 5 matches. Doesn't it have
>>>> to match all 10 of them?
>>>>
>>>>
>>>> 1. What happens if you take out the PCRE expressions from all
>>>> the rules ?
>>>> 2. sid:12 - should not fire because you have >10 , and there are
>>>> exactly 10 e-mails in the file
>>>> 3. how big is the stream itself? i think it is below 2KB, correct?
>>>> 4. is the PCRE matching the e-mails, under the unix shell ?
>>>> 5. yes i think you should get more sid:11 alerts - but first lets
>>>> investigate the above 4.
>>>>
>>>> thanks
>>>>
>>>> --
>>>> Peter Manev
>>>
>>> The file with only the 10 emails is 160 bytes. Even without pcre I
>>> get the same result :
>>>
>>> alert tcp $HOME_NET 80 -> any any \
>>> (msg:"got one email addr"; content:"|40|"; \
>>> flow:established,from_server; flowint:something,notset;
>>> flowint:something,=,1; sid:10; priority:3;)
>>>
>>> alert tcp $HOME_NET 80 -> any any \
>>> (msg:"got more email addrs"; content:"|40|"; \
>>> flow:established,from_server; flowint:something,isset;
>>> flowint:something,+,1; sid:11; priority:2;)
>>>
>>> alert tcp $HOME_NET 80 -> any any \
>>> (msg:"Got too many email addrs!"; content:"|40|"; \
>>> flow:established,from_server; flowint:something,isset;
>>> flowint:something,>,9; sid:12; priority:1;
>>> classtype:policy-violation;)
>>>
>>>
>>> alerts I get :
>>>
>>> 02/11/2012-21:23:14.567194 [**] [1:10:0] got one email addr [**]
>>> [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 ->
>>> Y.Y.Y.Y:58158
>>> 02/11/2012-21:23:14.567194 [**] [1:11:0] got more email addrs
>>> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 ->
>>> Y.Y.Y.Y:58158
>>>
>>> If I put some '#' symbols between the emails in the file so that
>>> it gets about 9K big and I fetch it I get these alerts :
>>>
>>> 02/11/2012-21:25:37.755214 [**] [1:10:0] got one email addr [**]
>>> [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 ->
>>> Y.Y.Y.Y:58166
>>> 02/11/2012-21:25:37.755214 [**] [1:11:0] got more email addrs
>>> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 ->
>>> Y.Y.Y.Y:58166
>>> 02/11/2012-21:25:37.761077 [**] [1:11:0] got more email addrs
>>> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 ->
>>> Y.Y.Y.Y:58166
>>> 02/11/2012-21:25:37.764451 [**] [1:11:0] got more email addrs
>>> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 ->
>>> Y.Y.Y.Y:58166
>>>
>>>
>>>
>>> Hi Nikolay,
>>>
>>>
>>> Can you please post this as a bug - please be detailed (as you were in
>>> your 2 previous e-mails).
>>> Personally i think here sid 11 is the problem , may be it does not
>>> count/increment correctly....
>>> thanks
>>>
>>>
>>> --
>>> Peter Manev
>>
>> Yes I will post this as a bug. But I've just found a much simpler case.
>>
>> Let's for example have only this rule in suricata :
>>
>> alert tcp $HOME_NET 6666 -> any any (msg:"match"; content:"|40|";)
>>
>> Then on a monitored machine from the $HOME_NET range I do :
>>
>> echo "@ @ @ @ @ @ @ @ @" | nc -l 6666
>>
>> And on different host I do :
>>
>> nc testserver 6666
>>
>> This gets the ten @ chars transferred, and I get only one alert.
>> But for example if I echo more @ chars, like 5000 or something, I get
>> 3-6 alerts.
>> I have to check what is actually the number of packets with payload,
>> probably the rule
>> is matched once per packet? But this could not explain that I get
>> different number of alerts on different runs.
>
> The behavior is by design. TCP data by default is inspected in the
> stream context, which means the "@ @ @ @ @ @ @ @ @" buffer is inspected
> at once. Suricata will not try to find every possible match in a
> payload, but just one.
>
> The reason you get more alerts if you increase the payload
> significantly, is that the stream is inspected in chunks. The size of
> those chunks is determined by your stream toserver_chunk_size setting.
>
> --
> ---------------------------------------------
> Victor Julien
> http://www.inliniac.net/
> PGP: http://www.inliniac.net/victorjulien.asc
> ---------------------------------------------
>
>
I see now.
I was suspecting something like this, and it makes sense performance wise.
I was trying to create a rule that will fire only if more than let's say 50 email addresses are contained
in the stream, maybe I'll try with lower threshold.
Thanks,
More information about the Oisf-users
mailing list