<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 12pt;
font-family:Calibri
}
--></style></head>
<body class='hmmessage'><div dir='ltr'>I have a lot of RAM to work with (132GB). Besides tweaking libhtp, I've increased stream and flow buffers. In addition, I have two interfaces with 20 detection threads for each interface. Plus I have 20k ruleset. My memory consumption is big. It starts with around 80gb and grows to 105gb over time.<div>I've been trying to tweak and tune my config but I dont want to have packet loss.</div><div><br></div><div>It seems that after increasing libhtp buffers, my alert count increased.</div><div><br><div><hr id="stopSpelling">Date: Thu, 6 Nov 2014 18:07:26 +0100<br>From: michal@rsbac.org<br>To: oisf-users@lists.openinfosecfoundation.org<br>Subject: Re: [Oisf-users] Occasional burst of packet loss<br><br>
body-limit 1GB? That's huge, isn't it? How does the performance look
like?<br>
<br>
I'm running with something around 20MB+<br>
<br>
<div class="ecxmoz-cite-prefix">On 06/11/14 16:20, Yasha Zislin wrote:<br>
</div>
<blockquote cite="mid:COL127-W19390629A9608981C2F94AA2840@phx.gbl">
<style><!--
.ExternalClass .ecxhmmessage P {
padding:0px;
}
.ExternalClass body.ecxhmmessage {
font-size:12pt;
font-family:Calibri;
}
--></style>
<div dir="ltr">I've tried af-packet mode in the beginning and had
a high packet loss. Most likely I didnt configure it right but
I've gotten so used to PF_RING, I think I can make it work just
have to tune suricata config.
<div><br>
</div>
<div>I've increased libhtp request-body-limit and
response-body-limit values to 1gb. It seems to be holding up
without any loss.</div>
<div>I've also changed rx-usecs to 1. In addition, I've reduced
pf_ring ring slots. So maybe this will just work. </div>
<div><br>
</div>
<div>Thanks for all of the information.</div>
<div><br>
</div>
<div><br>
<br>
<div>> Date: Wed, 5 Nov 2014 09:30:13 -0800<br>
> From: <a class="ecxmoz-txt-link-abbreviated" href="mailto:cnelson@ucsd.edu">cnelson@ucsd.edu</a><br>
> To: <a class="ecxmoz-txt-link-abbreviated" href="mailto:coolyasha@hotmail.com">coolyasha@hotmail.com</a>; <a class="ecxmoz-txt-link-abbreviated" href="mailto:petermanev@gmail.com">petermanev@gmail.com</a><br>
> CC: <a class="ecxmoz-txt-link-abbreviated" href="mailto:oisf-users@lists.openinfosecfoundation.org">oisf-users@lists.openinfosecfoundation.org</a><br>
> Subject: Re: [Oisf-users] Occasional burst of packet
loss<br>
> <br>
> -----BEGIN PGP SIGNED MESSAGE-----<br>
> Hash: SHA1<br>
> <br>
> My config is a little different.<br>
> <br>
> I'm running Suricata 2.1dev, built from the git
sources. I usually<br>
> update it weekly.<br>
> <br>
> I'm running af-packet/mmap mode; which has an option to
set a socket<br>
> buffer per-thread.<br>
> <br>
> In my case, increasing the socket buffer size resulted
in less (but not<br>
> zero) packet drops during bursty traffic.<br>
> <br>
> I also admit that I'm not familiar with the inner
workings of PF_RING,<br>
> but it may be that the addition of a socket-buffer in
af-packet/mmap<br>
> mode can mitigate packet drops due to periods of
extremely high packet<br>
> rates. The linux kernerl and PF_RING are not magic and
if anywhere in<br>
> the networking stack you are pushing packets faster
than the relevant<br>
> FIFO can process them you will get packet drops.
Buffered IO can<br>
> alleviate this to a certain extent.<br>
> <br>
> So, if you have the time, I would suggest trying a test
with the latest<br>
> git release, fresh kernel/drivers and af-packet/mmap
mode with at least<br>
> a megabyte of socket buffers.<br>
> <br>
> - -Coop<br>
> <br>
> On 11/5/2014 7:28 AM, Yasha Zislin wrote:<br>
> > I am using latest Suricata release 2.0.4. BTW,
I've had 2.0.1 version<br>
> > running recently and I had a drastic drop in
number of alerts after like<br>
> > 5 hours of starting Suricata. 2.0.4 fixed that.<br>
> > <br>
> > Actually, I managed to get pf_ring number of slots
way higher than<br>
> > suggested default.<br>
> > I found out that maximum number for me was 400000.
Originally during<br>
> > config tweaking, I've noticed that higher number
of slots did improve<br>
> > packet loss. <br>
> > Recently, I've started testing with lower number
of slots (200000), same<br>
> > behavior as far as occasional burst of packet
loss.<br>
> > <br>
> <br>
> - -- <br>
> Cooper Nelson<br>
> Network Security Analyst<br>
> UCSD ACT Security Team<br>
> <a class="ecxmoz-txt-link-abbreviated" href="mailto:cnelson@ucsd.edu">cnelson@ucsd.edu</a> x41042<br>
> -----BEGIN PGP SIGNATURE-----<br>
> Version: GnuPG v2.0.17 (MingW32)<br>
> <br>
>
iQEcBAEBAgAGBQJUWl6lAAoJEKIFRYQsa8FW8WAH/0NN7NogZ4B+KlbQla4EBZOC<br>
>
TBqv7IsjW7/tmS+u+k6VpRvP/1BbmMEdWbbOfz66uSaxFMMaZZFAC0PB9DXfROAL<br>
>
njdOQiCrienEsJD5xhIZTjZ+Q+brv9WicUAr0YtLKZ25/Y9jPD/crXQ21aBWa+yp<br>
>
IKIuhluclLBC0brd9nHGweKwd9BGc7e4NOUFu2gIGWVn3053OiZu1lyuqzrE3Fcw<br>
>
FP0sUJ+afhO8COrND+jehHoVTuLRde0+wbCav1srq3EcMGuctOhKBbqhvJS9iF4n<br>
>
+fvDTmeItxvZSOfDuMxyMfhT07Vt7GS4/T7EY+udaQhmiPTiJy1fkmuyAxTLtP0=<br>
> =DxQR<br>
> -----END PGP SIGNATURE-----<br>
</div>
</div>
</div>
<br>
<fieldset class="ecxmimeAttachmentHeader"></fieldset>
<br>
<pre>_______________________________________________
Suricata IDS Users mailing list: <a class="ecxmoz-txt-link-abbreviated" href="mailto:oisf-users@openinfosecfoundation.org">oisf-users@openinfosecfoundation.org</a>
Site: <a class="ecxmoz-txt-link-freetext" href="http://suricata-ids.org" target="_blank">http://suricata-ids.org</a> | Support: <a class="ecxmoz-txt-link-freetext" href="http://suricata-ids.org/support/" target="_blank">http://suricata-ids.org/support/</a>
List: <a class="ecxmoz-txt-link-freetext" href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" target="_blank">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a>
Training now available: <a class="ecxmoz-txt-link-freetext" href="http://suricata-ids.org/training/" target="_blank">http://suricata-ids.org/training/</a></pre>
</blockquote>
<br>
<br>_______________________________________________
Suricata IDS Users mailing list: oisf-users@openinfosecfoundation.org
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
Training now available: http://suricata-ids.org/training/</div></div> </div></body>
</html>