<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri",sans-serif;}
span.EmailStyle20
{mso-style-type:personal-compose;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:689334615;
mso-list-type:hybrid;
mso-list-template-ids:999315104 -1392327698 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:5;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Calibri",sans-serif;
mso-fareast-font-family:Calibri;
mso-bidi-font-family:"Times New Roman";}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoPlainText">Our Suricata installation went from normal to completely haywire overnight Tuesday. It was cruising along with very low packet loss (0.002%) when suddenly between 2:24 and 2:29 AM it began to grow extremely rapidly.
<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">So far ‘ve checked and <o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText" style="margin-left:.5in;text-indent:-.25in;mso-list:l0 level1 lfo2">
<![if !supportLists]><span style="mso-list:Ignore">-<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>NIC stats for errors or drops are very few(at bottom of email)<o:p></o:p></p>
<p class="MsoPlainText" style="margin-left:.5in;text-indent:-.25in;mso-list:l0 level1 lfo2">
<![if !supportLists]><span style="mso-list:Ignore">-<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>There were no changes to server Tuesday AM to account for this<o:p></o:p></p>
<p class="MsoPlainText" style="margin-left:.5in;text-indent:-.25in;mso-list:l0 level1 lfo2">
<![if !supportLists]><span style="mso-list:Ignore">-<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>Network traffic just before and after exhibited no major change of volume.
<o:p></o:p></p>
<p class="MsoPlainText" style="margin-left:.5in;text-indent:-.25in;mso-list:l0 level1 lfo2">
<![if !supportLists]><span style="mso-list:Ignore">-<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>No errors are visible in the messages file, or Suricata logs that appear out of the ordinary.<o:p></o:p></p>
<p class="MsoPlainText" style="margin-left:.5in;text-indent:-.25in;mso-list:l0 level1 lfo2">
<![if !supportLists]><span style="mso-list:Ignore">-<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>Since that time RAM usage and CPU utilization is much higher (no surprise)
<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">The most pertinent data is below or attached. Any input at all would be helpful to say the least . . .
<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Host info – <o:p></o:p></p>
<p class="MsoPlainText">CentOS 7.2.1511 64bit, 128G RAM, 32 cores Xeon E5-2640 v3 @ 2.60GHz, hundreds of gigs of free space<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Suricata – 3.0 – <o:p></o:p></p>
<p class="MsoPlainText">/usr/bin/suricata -c /etc/suricata/suricata.yaml --user=suri --group=suri -v --af-packet=ens1f1 --runmode=workers –D<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas">top - 11:24:38 up 28 days, 21:16, 1 user, load average: 7.63, 9.83, 9.49<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas">Tasks: 396 total, 1 running, 395 sleeping, 0 stopped, 0 zombie<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas">%Cpu(s): 19.3 us, 0.1 sy, 0.0 ni, 80.1 id, 0.0 wa, 0.0 hi, 0.5 si, 0.0 st<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas">KiB Mem : 13175409+total, 11530432+free, 14969692 used, 1480088 buff/cache<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas">KiB Swap: 4194300 total, 4194300 free, 0 used. 11620499+avail Mem<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas">25685 suri 20 0 14.600g 0.013t 36420 S 631.8 10.6 21749:31 Suricata-Main<o:p></o:p></span></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Entries from the stats.log just before and after the start of the problem. 1<sup>st</sup> is the last “normal” entry. Attached spreadsheet showing the delta % of changes which vary from 90% to 20,000% over a short period of time has
more detail (attached). <o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><b>PRIOR TO JUMP:<o:p></o:p></b></p>
<p class="MsoPlainText">capture.kernel_drops 2,042<o:p></o:p></p>
<p class="MsoPlainText">flow.memcap 0<o:p></o:p></p>
<p class="MsoPlainText">tcp.no_flow 0<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><b>5 mins AFTER JUMP:<o:p></o:p></b></p>
<p class="MsoPlainText">capture.kernel_drops 420,209<o:p></o:p></p>
<p class="MsoPlainText">flow.memcap 22,445<o:p></o:p></p>
<p class="MsoPlainText">tcp.no_flow 19,584<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><b>10 mins AFTER JUMP:<o:p></o:p></b></p>
<p class="MsoPlainText">capture.kernel_drops 16,651,570<o:p></o:p></p>
<p class="MsoPlainText">flow.memcap 308,876<o:p></o:p></p>
<p class="MsoPlainText">tcp.no_flow 273,039<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><b>CURRENT:<o:p></o:p></b></p>
<p class="MsoPlainText">capture.kernel_drops 36,169,877,508<o:p></o:p></p>
<p class="MsoPlainText">flow.memcap 240,736,337<o:p></o:p></p>
<p class="MsoPlainText">tcp.no_flow 182,273,322<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><b>CURRENT (full detail):<o:p></o:p></b></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><span style="font-family:Consolas">-------------------------------------------------------------------<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">Date: 3/23/2016 -- 08:47:55 (uptime: 5d, 17h 02m 59s)<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">-------------------------------------------------------------------<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">Counter | TM Name | Value<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">-------------------------------------------------------------------<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">capture.kernel_packets | Total | 36169877508<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">capture.kernel_drops | Total | 8610389158<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">decoder.pkts | Total | 27103400573<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">decoder.bytes | Total | 17260791141325<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">decoder.invalid | Total | 201<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">decoder.ipv4 | Total | 27113873640<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">decoder.ipv6 | Total | 450013<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">decoder.ethernet | Total | 27103400573<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">decoder.tcp | Total | 17460031447<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">decoder.udp | Total | 9390824301<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">decoder.sctp | Total | 572<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">decoder.icmpv4 | Total | 14544707<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">decoder.icmpv6 | Total | 54284<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">decoder.gre | Total | 58888<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">decoder.teredo | Total | 278019<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">decoder.avg_pkt_size | Total | 636<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">decoder.max_pkt_size | Total | 1514<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">flow.memcap | Total | 240736337<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">defrag.ipv4.fragments | Total | 23421551<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">defrag.ipv4.reassembled | Total | 11686108<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">defrag.ipv6.fragments | Total | 686<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">tcp.sessions | Total | 91335846<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">tcp.pseudo | Total | 15993656<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">tcp.invalid_checksum | Total | 5377<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">tcp.no_flow | Total | 182273322<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">tcp.syn | Total | 156088653<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">tcp.synack | Total | 60140655<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">tcp.rst | Total | 93730134<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">tcp.stream_depth_reached | Total | 138960<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">tcp.reassembly_gap | Total | 5070871<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">detect.alert | Total | 391<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">flow_mgr.closed_pruned | Total | 64297464<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">flow_mgr.new_pruned | Total | 11662246<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">flow_mgr.est_pruned | Total | 36603185<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">flow.spare | Total | 20626<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">flow.tcp_reuse | Total | 6278669<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">tcp.memuse | Total | 280650800<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">tcp.reassembly_memuse | Total | 1539438177<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">dns.memuse | Total | 25028081<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">dns.memcap_state | Total | 53614<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">dns.memcap_global | Total | 56782651<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">http.memuse | Total | 1175360548<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-family:Consolas">flow.memuse | Total | 536870752<o:p></o:p></span></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">NIC STATS -<o:p></o:p></p>
<p class="MsoPlainText">Ethtool stats from the NIC<o:p></o:p></p>
<p class="MsoPlainText"> rx_errors: 141<o:p></o:p></p>
<p class="MsoPlainText"> tx_errors: 0<o:p></o:p></p>
<p class="MsoPlainText"> rx_dropped: 0<o:p></o:p></p>
<p class="MsoPlainText"> tx_dropped: 0<o:p></o:p></p>
<p class="MsoPlainText"> rx_over_errors: 0<o:p></o:p></p>
<p class="MsoPlainText"> rx_crc_errors: 0<o:p></o:p></p>
<p class="MsoPlainText"> rx_frame_errors: 0<o:p></o:p></p>
<p class="MsoPlainText"> rx_fifo_errors: 0<o:p></o:p></p>
<p class="MsoPlainText"> rx_missed_errors: 0<o:p></o:p></p>
<p class="MsoPlainText"> tx_aborted_errors: 0<o:p></o:p></p>
<p class="MsoPlainText"> tx_carrier_errors: 0<o:p></o:p></p>
<p class="MsoPlainText"> tx_fifo_errors: 0<o:p></o:p></p>
<p class="MsoPlainText"> tx_heartbeat_errors: 0<o:p></o:p></p>
<p class="MsoPlainText"> rx_long_length_errors: 0<o:p></o:p></p>
<p class="MsoPlainText"> rx_short_length_errors: 0<o:p></o:p></p>
<p class="MsoPlainText"> rx_csum_offload_errors: 249<o:p></o:p></p>
<p class="MsoPlainText"> rx_fcoe_dropped: 0<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
</div>
</body>
</html>