<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoPlainText">Thanks - that is a handy way to find top talkers. In the past I've also used Darkstat which is a bit old but works
<a href="https://unix4lyfe.org/darkstat/">https://unix4lyfe.org/darkstat/</a> <o:p>
</o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">In my case though, the drops don't usually happen during peak traffic times and often there are much higher peaks within a day or two when no packets are dropped. This is specifically about The Zabbix graphs I use rarely show a sloped
line, but a flat line at zero for days before and after. The chart below is a perfect example. Looking at the graph on the far right - at some point on Feb. 6 packet drops went from zero to 15 million w/in a 10 minute span and then continued with no further
drops. Looking at the chart on the far left shows the normal rhythm of daily traffic and no abnormal spike and one peak over 2Gbps. The following week shows a significant increase in traffic including a spike over 3Gbps but no additional drops during that
time. That is why I don’t think it is volume which is causing this. If it were I would increase the rx buffers on the NIC from the 512 I have it set to right now.
<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><img border="0" width="1898" height="185" style="width:19.7708in;height:1.927in" id="Picture_x0020_1" src="cid:image001.png@01D4CDBD.B87F23C0"><o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">I did check into the monitoring potential but it seems that nobody is aware of jumbo packets in use on monitored LAN segments nor in testing tools. If that was happening, would is show up in the stats under decoder.max_pkt_size ? Maybe
I need to add that stat and monitor. To date the largest packet size has been 6520.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">-----Original Message-----<br>
From: Nelson, Cooper <cnelson@ucsd.edu> <br>
Sent: Friday, February 22, 2019 2:02 PM<br>
To: Peter Manev <petermanev@gmail.com><br>
Cc: Cloherty, Sean E <scloherty@mitre.org>; Open Information Security Foundation <oisf-users@lists.openinfosecfoundation.org><br>
Subject: RE: [Oisf-users] [EXT] Re: Packet loss and increased resource consumption after upgrade to 4.1.2 with Rust support</p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">What I ended up doing was just putting some windows on my desktop showing htop (all the threads would get pegged) and the packet drops from stats.log , since this was happening several times a day for extended periods.
<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">I then wrote a script to just grab a million packets off the wire and show the top talkers. Found it pretty easily.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">$ cat bin/top_flows.sh<o:p></o:p></p>
<p class="MsoPlainText">#!/bin/bash<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">sudo tcpdump -tnn -c 100000 -i any 2>/dev/null | awk '{print $2,$3,$4,$5}' | sort | uniq -c | sort -nr | head<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">You could automate something like to watch for packet drops in stats.log and then run the above script (I would recommend several times).<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">I think I've suggested in the past if suri could dump the ring buffer to a file when the 'emergency flush' condition is triggered you could just run the above script on the resulting pcap to find what caused it.
<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">-Coop<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">-----Original Message-----<o:p></o:p></p>
<p class="MsoPlainText">From: Peter Manev <<a href="mailto:petermanev@gmail.com"><span style="color:windowtext;text-decoration:none">petermanev@gmail.com</span></a>>
<o:p></o:p></p>
<p class="MsoPlainText">Sent: Friday, February 22, 2019 9:43 AM<o:p></o:p></p>
<p class="MsoPlainText">To: Nelson, Cooper <<a href="mailto:cnelson@ucsd.edu"><span style="color:windowtext;text-decoration:none">cnelson@ucsd.edu</span></a>><o:p></o:p></p>
<p class="MsoPlainText">Cc: Cloherty, Sean E <<a href="mailto:scloherty@mitre.org"><span style="color:windowtext;text-decoration:none">scloherty@mitre.org</span></a>>; Open Information Security Foundation <<a href="mailto:oisf-users@lists.openinfosecfoundation.org"><span style="color:windowtext;text-decoration:none">oisf-users@lists.openinfosecfoundation.org</span></a>><o:p></o:p></p>
<p class="MsoPlainText">Subject: Re: [Oisf-users] [EXT] Re: Packet loss and increased resource consumption after upgrade to 4.1.2 with Rust support<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Could be related indeed.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">@Sean Could you try the following and give me some feedback please.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
</div>
</body>
</html>