[Oisf-users] real time alert on tcp stream and flowint

Peter Manev petermanev at gmail.com
Tue Feb 14 21:30:04 UTC 2012


On 2/14/2012 2:49 PM, Nikolay Denev wrote:
> On Feb 14, 2012, at 11:21 AM, Victor Julien wrote:
>
>> On 02/12/2012 08:15 AM, Nikolay Denev wrote:
>>> On Feb 11, 2012, at 10:11 PM, Peter Manev wrote:
>>>
>>>>
>>>> On Sat, Feb 11, 2012 at 8:27 PM, Nikolay Denev<ndenev at gmail.com
>>>> <mailto:ndenev at gmail.com>>  wrote:
>>>>
>>>>
>>>>     On Feb 11, 2012, at 9:14 PM, Peter Manev wrote:
>>>>
>>>>>
>>>>>     On Sat, Feb 11, 2012 at 7:42 PM, Nikolay Denev<ndenev at gmail.com
>>>>>     <mailto:ndenev at gmail.com>>  wrote:
>>>>>
>>>>>
>>>>>         On Feb 11, 2012, at 7:52 PM, Peter Manev wrote:
>>>>>
>>>>>>
>>>>>>         On Sat, Feb 11, 2012 at 4:31 PM, Nikolay Denev
>>>>>>         <ndenev at gmail.com<mailto:ndenev at gmail.com>>  wrote:
>>>>>>
>>>>>>
>>>>>>             On Feb 11, 2012, at 12:11 PM, Peter Manev wrote:
>>>>>>
>>>>>>>
>>>>>>>             On Fri, Feb 10, 2012 at 6:43 AM, Nikolay Denev
>>>>>>>             <ndenev at gmail.com<mailto:ndenev at gmail.com>>  wrote:
>>>>>>>
>>>>>>>
>>>>>>>                 On Feb 9, 2012, at 10:04 PM, Nikolay Denev wrote:
>>>>>>>
>>>>>>>> On Feb 9, 2012, at 10:03 PM, Nikolay Denev wrote:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> It's probably stupid question and I'm missing
>>>>>>>                 something but I don't seem to be able
>>>>>>>>> to generate alert immediately when for example a
>>>>>>>                 given string is found inside a TCP stream.
>>>>>>>>> When the TCP connection closes, suricata
>>>>>>>                 immediately prints the alert in fast.log.
>>>>>>>>> How can I make the alert be generated
>>>>>>>                 immediately when the rule condition is matched?
>>>>>>>>> Also I don't know if its because of this I don't
>>>>>>>                 seem to be able to trigger the rule to match
>>>>>>>                 several times on the same stream,
>>>>>>>>> while I have the string that should fire the
>>>>>>>                 alert several times in the stream.
>>>>>>>>> Here's an example :
>>>>>>>>>
>>>>>>>>> alert tcp $HOME_NET 6666 ->  any any \
>>>>>>>>>       (msg:"got one"; content:"something";
>>>>>>>                 flowint:something,notset; flowint:something,=,1;
>>>>>>>                 sid:10;)
>>>>>>>>> alert tcp $HOME_NET 6666 ->  any any \
>>>>>>>>>       (msg:"got five or more";
>>>>>>>                 content:"something"; flowint:something,isset;
>>>>>>>                 flowint:something,+,1; flowint:something,>,5; sid:11;)
>>>>>>>>> This never works, I just have the first rule
>>>>>>>                 fire once when the TCP session is terminated.
>>>>>>>>>
>>>>>>>>> P.S.: As a side note the wiki should be updated
>>>>>>>                 to include probably "sid"s for the rules, as
>>>>>>>                 currently when I try to run the examples
>>>>>>>>> suricata complains about duplicated rules.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>> I'm running 1.2.1 RELEASE on FreeBSD-9.0-STABLE.
>>>>>>>                 This seems to work :
>>>>>>>
>>>>>>>                 alert tcp $HOME_NET 6666 ->  any any \
>>>>>>>                        (msg:"got one"; content:"something";
>>>>>>>                 flowint:something,notset; flowint:something,=,1;
>>>>>>>                 noalert; sid:10; priority: 1;)
>>>>>>>
>>>>>>>                 alert tcp $HOME_NET 6666 ->  any any \
>>>>>>>                        (msg:"got more"; content:"something";
>>>>>>>                 flowint:something,isset; flowint:something,+,1;
>>>>>>>                 noalert; sid:11; priority: 2;)
>>>>>>>
>>>>>>>
>>>>>>>                 alert tcp $HOME_NET 6666 ->  any any \
>>>>>>>                        (msg:"got too many"; content:"something";
>>>>>>>                 flowint:something,isset; flowint:something,>,2;
>>>>>>>                 sid:12; priority: 3;)
>>>>>>>
>>>>>>>
>>>>>>>                 _______________________________________________
>>>>>>>                 Oisf-users mailing list
>>>>>>>                 Oisf-users at openinfosecfoundation.org
>>>>>>>                 <mailto:Oisf-users at openinfosecfoundation.org>
>>>>>>>                 http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>             Hi Nikolay,
>>>>>>>             I think this is the way it is supposed to work. (last
>>>>>>>             example, by you).
>>>>>>>
>>>>>>>             When you take out "noalert" form sid 11 - does it fire ?
>>>>>>>
>>>>>>>             And are these the only rules that are loaded in terms
>>>>>>>             of flowint or you have others before that?
>>>>>>>
>>>>>>>             thanks
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>             --
>>>>>>>             Peter Manev
>>>>>>
>>>>>>             Yes, It fires, the problem I have is that it doesn't
>>>>>>             fire for each occurence of "content".
>>>>>>             Is alert supposed to fire once per packet if it matches,
>>>>>>             or for each match in the stream?
>>>>>>
>>>>>>             For example now I'm using these rules to catch if there
>>>>>>             are more than some defined amount of email addresses in
>>>>>>             a given stream :
>>>>>>
>>>>>>
>>>>>>             alert tcp $HOME_NET 80 ->  any any \
>>>>>>                     (msg:"got one email addr"; content:"|40|";
>>>>>>             pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \
>>>>>>                     flow:established,from_server;
>>>>>>             flowint:something,notset; flowint:something,=,1; sid:10;
>>>>>>             priority:3; noalert;)
>>>>>>
>>>>>>             alert tcp $HOME_NET 80 ->  any any \
>>>>>>                     (msg:"got more email addrs"; content:"|40|";
>>>>>>             pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \
>>>>>>                     flow:established,from_server;
>>>>>>             flowint:something,isset; flowint:something,+,1; sid:11;
>>>>>>             priority:2; noalert;)
>>>>>>
>>>>>>             alert tcp $HOME_NET 80 ->  any any \
>>>>>>                     (msg:"Got too many email addrs!";
>>>>>>             content:"|40|";
>>>>>>             pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \
>>>>>>                     flow:established,from_server;
>>>>>>             flowint:something,isset; flowint:something,>,10; sid:12;
>>>>>>             priority:1; classtype:policy-violation;)
>>>>>>
>>>>>>
>>>>>>             This for example works, but would not match for a simple
>>>>>>             plain text file with 10 email adresses, I need to have
>>>>>>             maybe 40-50 or more for this to match.
>>>>>>             Maybe I'm missing something…
>>>>>>
>>>>>>             And yes, these are my only rules that I'm testing with.
>>>>>>             No other rules with or without flowint whatsoever.
>>>>>>
>>>>>>
>>>>>>         Hi ,
>>>>>>         Just so I understand you correctly - you have a text file
>>>>>>         (in the stream) and in that text file you have 10 e-mail
>>>>>>         addresses and it wold not fire. correct ?
>>>>>>
>>>>>>
>>>>>>         thanks
>>>>>>
>>>>>>
>>>>>>         --
>>>>>>         Peter Manev
>>>>>         Exactly.
>>>>>
>>>>>         For example if I try to fetch the file emails.txt via http
>>>>>         which has the following content :
>>>>>
>>>>>         # cat emails.txt
>>>>>         edin at email.com<mailto:edin at email.com>
>>>>>         edin at email.com<mailto:edin at email.com>
>>>>>         edin at email.com<mailto:edin at email.com>
>>>>>         edin at email.com<mailto:edin at email.com>
>>>>>         edin at email.com<mailto:edin at email.com>
>>>>>         edin at email.com<mailto:edin at email.com>
>>>>>         edin at email.com<mailto:edin at email.com>
>>>>>         edin at email.com<mailto:edin at email.com>
>>>>>         edin at email.com<mailto:edin at email.com>
>>>>>         edin at email.com<mailto:edin at email.com>
>>>>>
>>>>>         $ curl http://testserver/emails.txt
>>>>>         edin at email.com<mailto:edin at email.com>
>>>>>         edin at email.com<mailto:edin at email.com>
>>>>>         edin at email.com<mailto:edin at email.com>
>>>>>         edin at email.com<mailto:edin at email.com>
>>>>>         edin at email.com<mailto:edin at email.com>
>>>>>         edin at email.com<mailto:edin at email.com>
>>>>>         edin at email.com<mailto:edin at email.com>
>>>>>         edin at email.com<mailto:edin at email.com>
>>>>>         edin at email.com<mailto:edin at email.com>
>>>>>         edin at email.com<mailto:edin at email.com>
>>>>>         $
>>>>>
>>>>>         And I also remove the "noalert" option from the rules, this
>>>>>         is what I get in fast.log :
>>>>>
>>>>>         02/11/2012-20:37:23.988271  [**] [1:10:0] got one email addr
>>>>>         [**] [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80
>>>>>         ->  Y.Y.Y.Y:57923
>>>>>         02/11/2012-20:37:23.988271  [**] [1:11:0] got more email
>>>>>         addrs [**] [Classification: (null)] [Priority: 2] {TCP}
>>>>>         X.X.X.X:80 ->  Y.Y.Y.Y:57923
>>>>>
>>>>>
>>>>>         If I change the third rule to fire if the flowint var is more
>>>>>         than 1, it is being triggered.
>>>>>
>>>>>         If I insert some random data between the email addresses in
>>>>>         the text file, then I get 4 maybe 5 matches. Doesn't it have
>>>>>         to match all 10 of them?
>>>>>
>>>>>
>>>>>     1. What happens if you take out  the PCRE expressions from all
>>>>>     the rules ?
>>>>>     2. sid:12 - should not fire because you have>10 , and there are
>>>>>     exactly 10 e-mails in the file
>>>>>     3. how big is the stream itself? i think it is below 2KB, correct?
>>>>>     4. is the PCRE matching the e-mails, under the unix shell ?
>>>>>     5. yes i think you should get more sid:11 alerts - but first lets
>>>>>     investigate the above 4.
>>>>>
>>>>>     thanks
>>>>>
>>>>>     --
>>>>>     Peter Manev
>>>>     The file with only the 10 emails is 160 bytes. Even without pcre I
>>>>     get the same result :
>>>>
>>>>     alert tcp $HOME_NET 80 ->  any any \
>>>>             (msg:"got one email addr"; content:"|40|"; \
>>>>             flow:established,from_server; flowint:something,notset;
>>>>     flowint:something,=,1; sid:10; priority:3;)
>>>>
>>>>     alert tcp $HOME_NET 80 ->  any any \
>>>>             (msg:"got more email addrs"; content:"|40|"; \
>>>>             flow:established,from_server; flowint:something,isset;
>>>>     flowint:something,+,1; sid:11; priority:2;)
>>>>
>>>>     alert tcp $HOME_NET 80 ->  any any \
>>>>             (msg:"Got too many email addrs!"; content:"|40|"; \
>>>>             flow:established,from_server; flowint:something,isset;
>>>>     flowint:something,>,9; sid:12; priority:1;
>>>>     classtype:policy-violation;)
>>>>
>>>>
>>>>     alerts I get :
>>>>
>>>>     02/11/2012-21:23:14.567194  [**] [1:10:0] got one email addr [**]
>>>>     [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 ->
>>>>     Y.Y.Y.Y:58158
>>>>     02/11/2012-21:23:14.567194  [**] [1:11:0] got more email addrs
>>>>     [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 ->
>>>>     Y.Y.Y.Y:58158
>>>>
>>>>     If I put some '#' symbols between the emails in the file so that
>>>>     it gets about 9K big and I fetch it I get these alerts :
>>>>
>>>>     02/11/2012-21:25:37.755214  [**] [1:10:0] got one email addr [**]
>>>>     [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 ->
>>>>     Y.Y.Y.Y:58166
>>>>     02/11/2012-21:25:37.755214  [**] [1:11:0] got more email addrs
>>>>     [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 ->
>>>>     Y.Y.Y.Y:58166
>>>>     02/11/2012-21:25:37.761077  [**] [1:11:0] got more email addrs
>>>>     [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 ->
>>>>     Y.Y.Y.Y:58166
>>>>     02/11/2012-21:25:37.764451  [**] [1:11:0] got more email addrs
>>>>     [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 ->
>>>>     Y.Y.Y.Y:58166
>>>>
>>>>
>>>>
>>>> Hi Nikolay,
>>>>
>>>>
>>>> Can you please post this as a bug - please be detailed (as you were in
>>>> your 2 previous e-mails).
>>>> Personally i think here sid 11 is the problem , may be it does not
>>>> count/increment correctly....
>>>> thanks
>>>>
>>>>
>>>> -- 
>>>> Peter Manev
>>> Yes I will post this as a bug. But I've just found a much simpler case.
>>>
>>> Let's for example have only this rule in suricata :
>>>
>>>   alert tcp $HOME_NET 6666 ->  any any (msg:"match"; content:"|40|";)
>>>
>>> Then on a monitored machine from the $HOME_NET range I do :
>>>
>>>   echo "@ @ @ @ @ @ @ @ @" | nc -l 6666
>>>
>>> And on different host I do :
>>>
>>>   nc testserver 6666
>>>
>>> This gets the ten @ chars transferred, and I get only one alert.
>>> But for example if I echo more @ chars, like 5000 or something, I get
>>> 3-6 alerts.
>>> I have to check what is actually the number of packets with payload,
>>> probably the rule
>>> is matched once per packet? But this could not explain that I get
>>> different number of alerts on different runs.
>> The behavior is by design. TCP data by default is inspected in the
>> stream context, which means the "@ @ @ @ @ @ @ @ @" buffer is inspected
>> at once. Suricata will not try to find every possible match in a
>> payload, but just one.
>>
>> The reason you get more alerts if you increase the payload
>> significantly, is that the stream is inspected in chunks. The size of
>> those chunks is determined by your stream toserver_chunk_size setting.
>>
>> -- 
>> ---------------------------------------------
>> Victor Julien
>> http://www.inliniac.net/
>> PGP: http://www.inliniac.net/victorjulien.asc
>> ---------------------------------------------
>>
>>
> I see now.
> I was suspecting something like this, and it makes sense performance wise.
>
> I was trying to create a rule that will fire only if more than let's say 50 email addresses are contained
> in the stream, maybe I'll try with lower threshold.
>
> Thanks,
>
> _______________________________________________
> Oisf-users mailing list
> Oisf-users at openinfosecfoundation.org
> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
If you ...for a cheap trick ...try the PCRE - would it behave as expected?

-- 
Regards,
Peter Manev




More information about the Oisf-users mailing list