[Oisf-users] 40GB inspection and I/O write speed concerns

[Nelson, Cooper]

Btw I’ve done 20Gbs w/full EVE logging to a SAS 10K RAID5 container, no problem.  Filesystem was btrfs with lzop compression enabled.

-Coop

From: Oisf-users <oisf-users-bounces at lists.openinfosecfoundation.org> On Behalf Of Jeremy A. Grove
Sent: Monday, July 22, 2019 9:49 AM
To: Peter Manev <petermanev at gmail.com>
Cc: oisf-users <oisf-users at lists.openinfosecfoundation.org>
Subject: Re: [Oisf-users] 40GB inspection and I/O write speed concerns

Thanks to both Peter and Cooper!

I do split out by event type already and I can certainly trim out some of this if needed. Ill let you know how it goes.

Regards,

Jeremy Grove, SSCP
Security Engineer
Quadrant Information Security
o: (904)296-9100<callto:(904)296-9100> x100
t: (800) 538-9357<callto:(800)%20538-9357> x100
e: soc at quadrantsec.com<mailto:soc at quadrantsec.com>

Learn more= about our managed SIEM people + product<https://a.quadrantsec.com/3D%22https:/quadrantsec.com/SaganMSSP%22>


________________________________
From: "Peter Manev" <petermanev at gmail.com<mailto:petermanev at gmail.com>>
To: "Jeremy A. Grove" <jgrove at quadrantsec.com<mailto:jgrove at quadrantsec.com>>
Cc: "oisf-users" <oisf-users at lists.openinfosecfoundation.org<mailto:oisf-users at lists.openinfosecfoundation.org>>
Sent: Monday, July 22, 2019 11:13:46 AM
Subject: Re: [Oisf-users] 40GB inspection and I/O write speed concerns


On 22 Jul 2019, at 08:34, Jeremy A. Grove <jgrove at quadrantsec.com<mailto:jgrove at quadrantsec.com>> wrote:
Hi All,

I am looking for advice. We are working on setting up a machine for potential 40GB a second on inspection. Our concern comes in the write speed of the I/O to disk as the meta-data that Suricata creates is important to us. Does anyone have experience with this? I have listed some of our set up below. Are there any suggestions or known issues that I should be aware of.


My five cents to start with:
I would split the logging per event_type and only log what is needed not just everything as some logs depending on the traffic could be very very verbose- like dns/fileinfo on university network etc..
Also filter out if possible with a bpf(or on the mirror /tap)  what is not relevant too.


DL360 Gen 10
P408I-A (Raid Card)
4 x 2TB SSD in RAID 10 (Part number 877788-B21)
Mixed use SSDs

Thanks!

Jeremy Grove, SSCP
Security Engineer
Quadrant Information Security

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20190722/74e6434a/attachment-0001.html>