[Oisf-users] For those who haven't read this

[Robert Vineyard]

Interesting, I did not realize that was something that Suricata could do.

Although, it does seem like that would preclude it from running in IPS mode 
since real-time analysis is essential for that kind of setup. However, for 
IDS mode, being able to fork off a separate thread/process to handle 
querying against an IP reputation database would be handy. I'm presently 
looking at ways to post-process the output of <insert favorite IDS here> for 
correlation against those types of data sources, but being able to do it 
natively inside the engine would be a big plus.

I'm not trying to criticize here folks, just wanting to stir up the 
discussion a bit particularly in light of the flame war that seems to be 
starting over snort vs. suricata. I think OISF has the right idea and of 
course an all-star cast of sponsors and developers (I'm especially excited 
about the HTP library's application-layer inspection capabilities). Also I 
wasn't trying to espouse the theory of single-threading vs. multi-threading 
(I'm quite familiar with the implications of both design methodologies), but 
having worked with real-time bandwidth-intensive streaming video 
applications in the past in a parallel processing aspect, keeping everything 
synchronized without having to wait on one or two pieces of the puzzle is 
crucial to achieving real-time performance on these types of applications. I 
suppose I should take a closer look at the threading model implemented by 
Suricata, I was just curious if you guys had a writeup comparable to Marty's 
article on the subject. I can definitely see merit in both approaches, but 
parallel computing is clearly the way of the future now that CPU vendors are 
focusing more on core count than MHz...

Just my 2c :-)

--
[ Robert Vineyard | RHCE, Security+ ]    [ robert.vineyard at oit.gatech.edu  ]
[ Information Security Engineer III ]    [ 404.385.6900 | FAX 404.894.4690 ]
[Finding a needle in a haystack isn't hard when every straw is computerized]

On 7/21/2010 5:27 PM, Brian Rectanus wrote:
> On Wed, Jul 21, 2010 at 1:00 PM, Matt Jonkman<jonkman at jonkmans.com>  wrote:
>> On 7/21/10 2:58 PM, Robert Vineyard wrote:
>>> http://vrt-sourcefire.blogspot.com/2010/06/single-threaded-data-processing.html
>>>
>>> Here's another link (a writeup by Marty Roesch of Sourcefire) referenced
>>> from the article you mentioned.
>>>
>>> In light of this discussion, does OISF / Suricata have a response to
>>> Sourcefire's critique of a multi-threaded engine model that uses several
>>> threads to process the same data simultaneously? It seems to me that the
>>> most efficient way to do things would be to have a front-end load-balancer
>>> that could distribute the traffic to multiple back-end threads or processes
>>> that would each operate on independent data streams. This is the strategy
>>> employed by Endace and others to accomplish high-throughput IDS inspection.
>>
>> I'll leave this for Victor and Will to answer in detail, but that's
>> about where we're going.
>>
>> Now nobody is saying that 4 cores gives you a 4-fold performance
>> increase. It's something less than that. But it's our only way forward.
>> The processors aren't going to get faster.
>
> Besides performance, one thing you cannot do with snort is send the
> data off to be analyzed externally without blocking until you get a
> response.  This is important in L7 if you want to send the data off to
> a WAF module, or maybe send the data to a virus checker, etc.
>
> So multi-threaded is not all about performance, but also about being
> able to do things that snort just cannot do.
>
> And comparing<  1yr old code to code as old as snort is a bit unreasonable.
>
>>
>>> On a related note, are there any plans to implement native acceleration
>>> support for other vendors besides Endace (in particular Napatech / nPulse)?
>>>
>>
>> Yes! As Randy mentioned they're building it now. We'll have news about
>> this very shortly as far as membership and support!
>>
>> Matt
>>
>>> Thanks!
>>>
>>> --
>>> [ Robert Vineyard | RHCE, Security+ ]    [ robert.vineyard at oit.gatech.edu  ]
>>> [ Information Security Engineer III ]    [ 404.385.6900 | FAX 404.894.9548 ]
>>> [Finding a needle in a haystack isn't hard when every straw is computerized]
>>>
>>>
>>> On 07/21/2010 07:05 AM, Kevin Ross wrote:
>>>> http://vrt-sourcefire.blogspot.com/2010/07/innovation-you-keep-using-that-word.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+Vrt+(Sourcefire+VRT+-+Vulnerability+Research%2C+Snort+Rules+and+Explosions)
>>>> <http://vrt-sourcefire.blogspot.com/2010/07/innovation-you-keep-using-that-word.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+Vrt+(Sourcefire+VRT+-+Vulnerability+Research%2C+Snort+Rules+and+Explosions)>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Oisf-users mailing list
>>>> Oisf-users at openinfosecfoundation.org
>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>> _______________________________________________
>>> Oisf-users mailing list
>>> Oisf-users at openinfosecfoundation.org
>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>
>> --
>>
>> ----------------------------------------------------
>> Matthew Jonkman
>> Emerging Threats
>> Open Information Security Foundation (OISF)
>> Phone 765-429-0398
>> Fax 312-264-0205
>> http://www.emergingthreats.net
>> http://www.openinfosecfoundation.org
>> ----------------------------------------------------
>>
>> PGP: http://www.jonkmans.com/mattjonkman.asc
>> _______________________________________________
>> Oisf-users mailing list
>> Oisf-users at openinfosecfoundation.org
>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>