<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 3.0cm 70.85pt 3.0cm;}
div.WordSection1
{page:WordSection1;}
--></style></head><body lang=PT link=blue vlink="#954F72"><div class=WordSection1><p class=MsoNormal>Hi Sean,</p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>To debug such situations what I do is:</p><p class=MsoNormal>- install Suricata debug symbols</p><p class=MsoNormal>- install gdb</p><p class=MsoNormal>- launch Suricata and attach gdb</p><p class=MsoNormal>- when the error occurs, I look at the call trace and stack to determine where the problem is and maybe the why.</p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>To help reproduce the error, I would have tcpdump creating a network packet dump so that I could replay traffic to Suricata.</p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Cheers,</p><p class=MsoNormal>Duarte</p><p class=MsoNormal><o:p> </o:p></p><div style='mso-element:para-border-div;border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal style='border:none;padding:0cm'><b>De: </b><a href="mailto:scloherty@mitre.org">Cloherty, Sean E</a><br><b>Enviado: </b>12 de abril de 2017 18:18<br><b>Para: </b><a href="mailto:oisf-users@lists.openinfosecfoundation.org">oisf-users@lists.openinfosecfoundation.org</a><br><b>Assunto: </b>[Oisf-users] Battling segfaults on 3.2.1</p></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span lang=EN-US>I am running 3.2.1 on 4 identical servers. Two of them started having segfaults and traps. <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>Troubleshooting - Compared yamls amd found an extra 0 (making the tracker 10x larger) in the SMTP mime section for inspected-tracker for file data keyword. Also, one system had 2gb vs. 4gb for the http memcap in the app layer protocol config. I changed the yamls to match the less problematic server. I also took the opportunity to recompile Suricata with Hyperscan (Thank you Derek Spransy and Justin Viiret!).<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>On one box I’ve had no segfaults since the April 7<sup>th </sup>(following the changes). The other one continues to have the problem 2-3 times a day at random hours – mid-morning, early evening, sometimes after midnight. Messages in the system log only include the actual fault message and nothing else. The fault always points to a worker thread and the numbers vary W#01-ensf1 or W#15-ens1f1 etc. Two types of errors come up from segfaults<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal style='text-indent:36.0pt'><span lang=EN-US>error 4 in suricata[400000+242000] or <o:p></o:p></span></p><p class=MsoNormal style='text-indent:36.0pt'><span lang=EN-US>error 5 in suricata[400000+242000]<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>Trap messages seem to have stopped on April 7<sup>th</sup> (following the changes), but also had error messages with the same info in the brackets –<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal style='text-indent:36.0pt'><span lang=EN-US>error:0 in suricata[400000+242000]<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>I’ve attached a zip file of the startup script, suricata.yaml, the suricata.log, stats.log, a copy of the faults listed in the /var/log/messages, and a textfule with the time and date of crashes. The server details follow:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>GENERAL SERVER INFO :<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>- CentOS Linux release 7.3.1611 (Core) 3.10.0-514.10.2.el7.x86_64 #1 SMP Fri Mar 3 00:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>- Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz - 16 cores / 32 threads <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>- 128GB of RAM<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>- Capture NIC is a dual port Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>- NIC Driver is Intel(R) 10GbE PCI Express Linux Network Driver - version 4.6.4<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>- Max traffic seen on the interface in the last 4 months has been 1.2 Gb/s, but usually mid-day peaks are around 1.1 Gb/s<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>Any suggestions of what to check next?<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>Sean<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>