[Oisf-users] 40GB inspection and I/O write speed concerns

[Peter Manev]

On Mon, Jul 22, 2019 at 7:20 PM Nelson, Cooper <cnelson at ucsd.edu> wrote:
>
> Btw I’ve done 20Gbs w/full EVE logging to a SAS 10K RAID5 container, no problem.  Filesystem was btrfs with lzop compression enabled.
>

Whats your eps in that case? (out of curiosity :) ) and did it reflect
on the drops (full vs non full logging ) ?


>
>
> -Coop
>
>
>
> From: Oisf-users <oisf-users-bounces at lists.openinfosecfoundation.org> On Behalf Of Jeremy A. Grove
> Sent: Monday, July 22, 2019 9:49 AM
> To: Peter Manev <petermanev at gmail.com>
> Cc: oisf-users <oisf-users at lists.openinfosecfoundation.org>
> Subject: Re: [Oisf-users] 40GB inspection and I/O write speed concerns
>
>
>
> Thanks to both Peter and Cooper!
>
>
>
> I do split out by event type already and I can certainly trim out some of this if needed. Ill let you know how it goes.
>
>
>
> Regards,
>
>
>
> Jeremy Grove, SSCP
> Security Engineer
> Quadrant Information Security
> o: (904)296-9100 x100
> t: (800) 538-9357 x100
> e: soc at quadrantsec.com
>
> Learn more= about our managed SIEM people + product
>
>
>
> ________________________________
>
> From: "Peter Manev" <petermanev at gmail.com>
> To: "Jeremy A. Grove" <jgrove at quadrantsec.com>
> Cc: "oisf-users" <oisf-users at lists.openinfosecfoundation.org>
> Sent: Monday, July 22, 2019 11:13:46 AM
> Subject: Re: [Oisf-users] 40GB inspection and I/O write speed concerns
>
>
>
>
>
> On 22 Jul 2019, at 08:34, Jeremy A. Grove <jgrove at quadrantsec.com> wrote:
>
> Hi All,
>
>
>
> I am looking for advice. We are working on setting up a machine for potential 40GB a second on inspection. Our concern comes in the write speed of the I/O to disk as the meta-data that Suricata creates is important to us. Does anyone have experience with this? I have listed some of our set up below. Are there any suggestions or known issues that I should be aware of.
>
>
>
>
>
> My five cents to start with:
>
> I would split the logging per event_type and only log what is needed not just everything as some logs depending on the traffic could be very very verbose- like dns/fileinfo on university network etc..
>
> Also filter out if possible with a bpf(or on the mirror /tap)  what is not relevant too.
>
>
>
> DL360 Gen 10
>
> P408I-A (Raid Card)
>
> 4 x 2TB SSD in RAID 10 (Part number 877788-B21)
>
> Mixed use SSDs
>
>
>
> Thanks!
>
>
>
> Jeremy Grove, SSCP
> Security Engineer
> Quadrant Information Security
>
>



-- 
Regards,
Peter Manev