[Oisf-users] packet loss troubleshooting

[Yasha Zislin]

I use PF_RING.
Changing these net.core buffers actually made it worse. Packet loss is instant with 30%.These are what my defaults are:net.core.wmem_default = 124928net.core.rmem_default = 124928net.core.netdev_max_backlog = 1000
I have 10 gig NIC as well. Not that busy pipe. About 1 million packets a minute.
> Subject: Re: [Oisf-users] packet loss troubleshooting
> To: coolyasha at hotmail.com; oisf-users at lists.openinfosecfoundation.org
> From: cnelson at ucsd.edu
> Date: Wed, 9 Dec 2015 09:03:33 -0800
> 
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> I use AF_PACKET + mmap mode, as described here:
> 
> > https://home.regit.org/2012/07/suricata-to-10gbps-and-beyond/
> 
> This is my config for a 16 core server.  Using af-packet mode and large
> buffers is the best way to mitigate packet drops in my experience.
> 
> > af-packet:
> >   - interface: eth2
> >     threads: 16
> >     cluster-id: 99
> >     cluster-type: cluster_flow
> >     defrag: yes
> >     use-mmap: yes
> >     ring-size: 500000
> >     use-emergency-flush: yes
> >     buffer-size: 1048576
> >     checksum-checks: kernel
> 
> To answer your question, you can drop packets anywhere.  I believe if
> you see kernel drops that means you could be losing packets from the NIC
> - -> kernel or from the kernel -> suricata.  Increasing the sysctl
> parameters mitigates the former.  Increasing suricata's buffers
> mitigates the latter.
> 
> Here is my config for a heavily utilized 10Gbit tap:
> 
> > net.core.netdev_max_backlog = 8000000
> > net.core.rmem_default = 1073741824
> > net.core.rmem_max = 1073741824
> 
> Make sure you make them permanent with 'sysctl -p' if you change them.
> 
> - -Coop
> 
> On 12/9/2015 5:36 AM, Yasha Zislin wrote:
> > I am at about 10% now. So this is not good.
> > So whenever I see capture.kernel_drops this is always OS or NIC problem?
> > Suricata itself has nothing to do with it, right?
> > I guess once I start seeing kernel drops, reassembly gaps start to
> > increase. Is that correct as well?
> > 
> > I am not an expert on net.core.* buffers. Can you advise on which ones i
> > need to increase or how to find out which ones I need to increase?
> > 
> > Thank you.
> 
> 
> - -- 
> Cooper Nelson
> Network Security Analyst
> UCSD ACT Security Team
> cnelson at ucsd.edu x41042
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (MingW32)
> 
> iQEcBAEBAgAGBQJWaF7lAAoJEKIFRYQsa8FWwbcH/2fBw4xfEM+etnn77y+Hqc6c
> PEHtZarvr4HZRQlnZ2NICt9seCfw60sEOIeduiv7nOOAjw4nU/B/xndeyj9b+Ls4
> 02+GUh9q4M4Sh0VV0KhMI/eaYMHMnBpjDOkZPzRBpcjcua/KHQEZFe6tCA5v7KWT
> qiFYSoxDtmm+jUETBtt07rWX+WM3Bdp8M4MnCT1rA6zqIUBqqjiylFisBWcAJjuN
> vCd6jD/XhcTHUSjrAkz5F9p+59isztip0H0XsO91JsTTCka/1+5mG0nIUpaRR/3c
> DTclbxfCz3joJ1DLFSUpDmr/R09TTkgAIKBSh3NHrdEhEPmbGJywAzqGpRoLpfE=
> =o07w
> -----END PGP SIGNATURE-----
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20151209/8df19eaf/attachment-0002.html>