[Oisf-users] real time alert on tcp stream and flowint
Nikolay Denev
ndenev at gmail.com
Wed Feb 15 05:34:12 UTC 2012
On Feb 15, 2012, at 6:51 AM, Anoop Saldanha wrote:
> On Tue, Feb 14, 2012 at 2:59 PM, Peter Manev <petermanev at gmail.com> wrote:
>>
>>
>> On Tue, Feb 14, 2012 at 10:21 AM, Victor Julien <victor at inliniac.net> wrote:
>>>
>>> On 02/12/2012 08:15 AM, Nikolay Denev wrote:
>>>>
>>>> On Feb 11, 2012, at 10:11 PM, Peter Manev wrote:
>>>>
>>>>>
>>>>>
>>>>> On Sat, Feb 11, 2012 at 8:27 PM, Nikolay Denev <ndenev at gmail.com
>>>>> <mailto:ndenev at gmail.com>> wrote:
>>>>>
>>>>>
>>>>> On Feb 11, 2012, at 9:14 PM, Peter Manev wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Sat, Feb 11, 2012 at 7:42 PM, Nikolay Denev <ndenev at gmail.com
>>>>>> <mailto:ndenev at gmail.com>> wrote:
>>>>>>
>>>>>>
>>>>>> On Feb 11, 2012, at 7:52 PM, Peter Manev wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Sat, Feb 11, 2012 at 4:31 PM, Nikolay Denev
>>>>>>> <ndenev at gmail.com <mailto:ndenev at gmail.com>> wrote:
>>>>>>>
>>>>>>>
>>>>>>> On Feb 11, 2012, at 12:11 PM, Peter Manev wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Feb 10, 2012 at 6:43 AM, Nikolay Denev
>>>>>>>> <ndenev at gmail.com <mailto:ndenev at gmail.com>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On Feb 9, 2012, at 10:04 PM, Nikolay Denev wrote:
>>>>>>>>
>>>>>>>> > On Feb 9, 2012, at 10:03 PM, Nikolay Denev wrote:
>>>>>>>> >
>>>>>>>> >> Hi all,
>>>>>>>> >>
>>>>>>>> >> It's probably stupid question and I'm missing
>>>>>>>> something but I don't seem to be able
>>>>>>>> >> to generate alert immediately when for example a
>>>>>>>> given string is found inside a TCP stream.
>>>>>>>> >> When the TCP connection closes, suricata
>>>>>>>> immediately prints the alert in fast.log.
>>>>>>>> >> How can I make the alert be generated
>>>>>>>> immediately when the rule condition is matched?
>>>>>>>> >>
>>>>>>>> >> Also I don't know if its because of this I don't
>>>>>>>> seem to be able to trigger the rule to match
>>>>>>>> several times on the same stream,
>>>>>>>> >> while I have the string that should fire the
>>>>>>>> alert several times in the stream.
>>>>>>>> >>
>>>>>>>> >> Here's an example :
>>>>>>>> >>
>>>>>>>> >> alert tcp $HOME_NET 6666 -> any any \
>>>>>>>> >> (msg:"got one"; content:"something";
>>>>>>>> flowint:something,notset; flowint:something,=,1;
>>>>>>>> sid:10;)
>>>>>>>> >>
>>>>>>>> >> alert tcp $HOME_NET 6666 -> any any \
>>>>>>>> >> (msg:"got five or more";
>>>>>>>> content:"something"; flowint:something,isset;
>>>>>>>> flowint:something,+,1; flowint:something,>,5;
>>>>>>>> sid:11;)
>>>>>>>> >>
>>>>>>>> >> This never works, I just have the first rule
>>>>>>>> fire once when the TCP session is terminated.
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> P.S.: As a side note the wiki should be updated
>>>>>>>> to include probably "sid"s for the rules, as
>>>>>>>> currently when I try to run the examples
>>>>>>>> >> suricata complains about duplicated rules.
>>>>>>>> >>
>>>>>>>> >> Thanks,
>>>>>>>> >>
>>>>>>>> >
>>>>>>>> > I'm running 1.2.1 RELEASE on FreeBSD-9.0-STABLE.
>>>>>>>>
>>>>>>>> This seems to work :
>>>>>>>>
>>>>>>>> alert tcp $HOME_NET 6666 -> any any \
>>>>>>>> (msg:"got one"; content:"something";
>>>>>>>> flowint:something,notset; flowint:something,=,1;
>>>>>>>> noalert; sid:10; priority: 1;)
>>>>>>>>
>>>>>>>> alert tcp $HOME_NET 6666 -> any any \
>>>>>>>> (msg:"got more"; content:"something";
>>>>>>>> flowint:something,isset; flowint:something,+,1;
>>>>>>>> noalert; sid:11; priority: 2;)
>>>>>>>>
>>>>>>>>
>>>>>>>> alert tcp $HOME_NET 6666 -> any any \
>>>>>>>> (msg:"got too many"; content:"something";
>>>>>>>> flowint:something,isset; flowint:something,>,2;
>>>>>>>> sid:12; priority: 3;)
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Oisf-users mailing list
>>>>>>>> Oisf-users at openinfosecfoundation.org
>>>>>>>> <mailto:Oisf-users at openinfosecfoundation.org>
>>>>>>>>
>>>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi Nikolay,
>>>>>>>> I think this is the way it is supposed to work. (last
>>>>>>>> example, by you).
>>>>>>>>
>>>>>>>> When you take out "noalert" form sid 11 - does it fire ?
>>>>>>>>
>>>>>>>> And are these the only rules that are loaded in terms
>>>>>>>> of flowint or you have others before that?
>>>>>>>>
>>>>>>>> thanks
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Peter Manev
>>>>>>>
>>>>>>>
>>>>>>> Yes, It fires, the problem I have is that it doesn't
>>>>>>> fire for each occurence of "content".
>>>>>>> Is alert supposed to fire once per packet if it matches,
>>>>>>> or for each match in the stream?
>>>>>>>
>>>>>>> For example now I'm using these rules to catch if there
>>>>>>> are more than some defined amount of email addresses in
>>>>>>> a given stream :
>>>>>>>
>>>>>>>
>>>>>>> alert tcp $HOME_NET 80 -> any any \
>>>>>>> (msg:"got one email addr"; content:"|40|";
>>>>>>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \
>>>>>>> flow:established,from_server;
>>>>>>> flowint:something,notset; flowint:something,=,1; sid:10;
>>>>>>> priority:3; noalert;)
>>>>>>>
>>>>>>> alert tcp $HOME_NET 80 -> any any \
>>>>>>> (msg:"got more email addrs"; content:"|40|";
>>>>>>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \
>>>>>>> flow:established,from_server;
>>>>>>> flowint:something,isset; flowint:something,+,1; sid:11;
>>>>>>> priority:2; noalert;)
>>>>>>>
>>>>>>> alert tcp $HOME_NET 80 -> any any \
>>>>>>> (msg:"Got too many email addrs!";
>>>>>>> content:"|40|";
>>>>>>> pcre:"/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i"; \
>>>>>>> flow:established,from_server;
>>>>>>> flowint:something,isset; flowint:something,>,10; sid:12;
>>>>>>> priority:1; classtype:policy-violation;)
>>>>>>>
>>>>>>>
>>>>>>> This for example works, but would not match for a simple
>>>>>>> plain text file with 10 email adresses, I need to have
>>>>>>> maybe 40-50 or more for this to match.
>>>>>>> Maybe I'm missing something…
>>>>>>>
>>>>>>> And yes, these are my only rules that I'm testing with.
>>>>>>> No other rules with or without flowint whatsoever.
>>>>>>>
>>>>>>>
>>>>>>> Hi ,
>>>>>>> Just so I understand you correctly - you have a text file
>>>>>>> (in the stream) and in that text file you have 10 e-mail
>>>>>>> addresses and it wold not fire. correct ?
>>>>>>>
>>>>>>>
>>>>>>> thanks
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Peter Manev
>>>>>>
>>>>>> Exactly.
>>>>>>
>>>>>> For example if I try to fetch the file emails.txt via http
>>>>>> which has the following content :
>>>>>>
>>>>>> # cat emails.txt
>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>>
>>>>>> $ curl http://testserver/emails.txt
>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>> edin at email.com <mailto:edin at email.com>
>>>>>> $
>>>>>>
>>>>>> And I also remove the "noalert" option from the rules, this
>>>>>> is what I get in fast.log :
>>>>>>
>>>>>> 02/11/2012-20:37:23.988271 [**] [1:10:0] got one email addr
>>>>>> [**] [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80
>>>>>> -> Y.Y.Y.Y:57923
>>>>>> 02/11/2012-20:37:23.988271 [**] [1:11:0] got more email
>>>>>> addrs [**] [Classification: (null)] [Priority: 2] {TCP}
>>>>>> X.X.X.X:80 -> Y.Y.Y.Y:57923
>>>>>>
>>>>>>
>>>>>> If I change the third rule to fire if the flowint var is more
>>>>>> than 1, it is being triggered.
>>>>>>
>>>>>> If I insert some random data between the email addresses in
>>>>>> the text file, then I get 4 maybe 5 matches. Doesn't it have
>>>>>> to match all 10 of them?
>>>>>>
>>>>>>
>>>>>> 1. What happens if you take out the PCRE expressions from all
>>>>>> the rules ?
>>>>>> 2. sid:12 - should not fire because you have >10 , and there are
>>>>>> exactly 10 e-mails in the file
>>>>>> 3. how big is the stream itself? i think it is below 2KB, correct?
>>>>>> 4. is the PCRE matching the e-mails, under the unix shell ?
>>>>>> 5. yes i think you should get more sid:11 alerts - but first lets
>>>>>> investigate the above 4.
>>>>>>
>>>>>> thanks
>>>>>>
>>>>>> --
>>>>>> Peter Manev
>>>>>
>>>>> The file with only the 10 emails is 160 bytes. Even without pcre I
>>>>> get the same result :
>>>>>
>>>>> alert tcp $HOME_NET 80 -> any any \
>>>>> (msg:"got one email addr"; content:"|40|"; \
>>>>> flow:established,from_server; flowint:something,notset;
>>>>> flowint:something,=,1; sid:10; priority:3;)
>>>>>
>>>>> alert tcp $HOME_NET 80 -> any any \
>>>>> (msg:"got more email addrs"; content:"|40|"; \
>>>>> flow:established,from_server; flowint:something,isset;
>>>>> flowint:something,+,1; sid:11; priority:2;)
>>>>>
>>>>> alert tcp $HOME_NET 80 -> any any \
>>>>> (msg:"Got too many email addrs!"; content:"|40|"; \
>>>>> flow:established,from_server; flowint:something,isset;
>>>>> flowint:something,>,9; sid:12; priority:1;
>>>>> classtype:policy-violation;)
>>>>>
>>>>>
>>>>> alerts I get :
>>>>>
>>>>> 02/11/2012-21:23:14.567194 [**] [1:10:0] got one email addr [**]
>>>>> [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 ->
>>>>> Y.Y.Y.Y:58158
>>>>> 02/11/2012-21:23:14.567194 [**] [1:11:0] got more email addrs
>>>>> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 ->
>>>>> Y.Y.Y.Y:58158
>>>>>
>>>>> If I put some '#' symbols between the emails in the file so that
>>>>> it gets about 9K big and I fetch it I get these alerts :
>>>>>
>>>>> 02/11/2012-21:25:37.755214 [**] [1:10:0] got one email addr [**]
>>>>> [Classification: (null)] [Priority: 3] {TCP} X.X.X.X:80 ->
>>>>> Y.Y.Y.Y:58166
>>>>> 02/11/2012-21:25:37.755214 [**] [1:11:0] got more email addrs
>>>>> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 ->
>>>>> Y.Y.Y.Y:58166
>>>>> 02/11/2012-21:25:37.761077 [**] [1:11:0] got more email addrs
>>>>> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 ->
>>>>> Y.Y.Y.Y:58166
>>>>> 02/11/2012-21:25:37.764451 [**] [1:11:0] got more email addrs
>>>>> [**] [Classification: (null)] [Priority: 2] {TCP} X.X.X.X:80 ->
>>>>> Y.Y.Y.Y:58166
>>>>>
>>>>>
>>>>>
>>>>> Hi Nikolay,
>>>>>
>>>>>
>>>>> Can you please post this as a bug - please be detailed (as you were in
>>>>> your 2 previous e-mails).
>>>>> Personally i think here sid 11 is the problem , may be it does not
>>>>> count/increment correctly....
>>>>> thanks
>>>>>
>>>>>
>>>>> --
>>>>> Peter Manev
>>>>
>>>> Yes I will post this as a bug. But I've just found a much simpler case.
>>>>
>>>> Let's for example have only this rule in suricata :
>>>>
>>>> alert tcp $HOME_NET 6666 -> any any (msg:"match"; content:"|40|";)
>>>>
>>>> Then on a monitored machine from the $HOME_NET range I do :
>>>>
>>>> echo "@ @ @ @ @ @ @ @ @" | nc -l 6666
>>>>
>>>> And on different host I do :
>>>>
>>>> nc testserver 6666
>>>>
>>>> This gets the ten @ chars transferred, and I get only one alert.
>>>> But for example if I echo more @ chars, like 5000 or something, I get
>>>> 3-6 alerts.
>>>> I have to check what is actually the number of packets with payload,
>>>> probably the rule
>>>> is matched once per packet? But this could not explain that I get
>>>> different number of alerts on different runs.
>>>
>>> The behavior is by design. TCP data by default is inspected in the
>>> stream context, which means the "@ @ @ @ @ @ @ @ @" buffer is inspected
>>> at once.
>>
>>
>>>
>>> Suricata will not try to find every possible match in a
>>> payload, but just one.
>>
>> That's good to know - clears out a few questions of mine....
>> but then a PCRE (matching on 10 "@") should match all of them - correct?
>> having in mind they are in the same "chunk".
>>
>
> If I have understood your question right, no! Pcre works just like
> content on the first match it finds. So alerts wise or match wise it
> should work the same as using content
>
So this means that there is no way to count the total number of occurrences of a
given string or pattern in a flow, and alert if some predefined number is reached?
i.e. no matter the number I will get one alert per chunk?
Something like 'g' (global match) flag for pcre? This will definitely be very expensive, but looks interesting as feature.
>>
>>>
>>>
>>> The reason you get more alerts if you increase the payload
>>> significantly, is that the stream is inspected in chunks. The size of
>>> those chunks is determined by your stream toserver_chunk_size setting.
>>>
>>> --
>>> ---------------------------------------------
>>> Victor Julien
>>> http://www.inliniac.net/
>>> PGP: http://www.inliniac.net/victorjulien.asc
>>> ---------------------------------------------
>>>
>>> _______________________________________________
>>> Oisf-users mailing list
>>> Oisf-users at openinfosecfoundation.org
>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>
>>
>>
>>
>> --
>> Peter Manev
>>
>> _______________________________________________
>> Oisf-users mailing list
>> Oisf-users at openinfosecfoundation.org
>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>
>
>
>
> --
> Anoop Saldanha
> _______________________________________________
> Oisf-users mailing list
> Oisf-users at openinfosecfoundation.org
> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
More information about the Oisf-users
mailing list