[Oisf-users] Hardware specs for monitoring 100GB

[Peter Manev]

On Tue, Nov 5, 2019 at 7:18 PM Nelson, Cooper <cnelson at ucsd.edu> wrote:

> Indeed, we are running into associated IO and licensing bottlenecks with
> the torrent of metadata that is produced.  I had to write an asynchronous
> spooler to copy stored files from a tmpfs partition to long-term storage,
> for example.  Our JSON logging is to a tmpfs partition as well.
>
>

I think there are a couple of bottle neck spots that can be hit in such
intense traffic volumes- disk speed, write locks, logoutput in general, bus
speed in some cases too, NUMA cross talks.
In general limiting what you need to look at is always a good step - ex
flush out streaming/video traffic etc. Perf top is your friend :) , "run on
empty" see what the performance is without loading any rules that would
pinpoint any non inspection related bottlenecks too.

I am trying to find a measurable,consistent, repetitive way of easy
figuring out if the system bus becomes a bottle neck and when on huge
speeds. Any suggestions or pointers are welcome :)


> -Coop
>
> -----Original Message-----
> From: Peter Manev <petermanev at gmail.com>
> Sent: Tuesday, November 5, 2019 12:15 AM
> To: Nelson, Cooper <cnelson at ucsd.edu>
> Cc: Michał Purzyński <michalpurzynski1 at gmail.com>; Drew Dixon <
> dwdixon at umich.edu>; Daniel Wallmeyer <Daniel.Wallmeyer at cisecurity.org>;
> oisf-users at lists.openinfosecfoundation.org
> Subject: Re: [Oisf-users] Hardware specs for monitoring 100GB
>
>
> We have recently experimented with AFPv2 IPS set up and Trex and were able
> to achieve 40Gbps throughput (Intel based CPU/NIC), (doc reminder for me)
> It is not always trivial esp at 100Gbps as it becomes a major single point
> of failure as well so there are a lot of caveats to consider and
> test(HA/Fail over/log writing/shipping etc..)
>
>
> --
> Regards,
> Peter Manev
>


-- 
Regards,
Peter Manev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20191227/c3ac23d7/attachment.html>