<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoPlainText">Here is the build info:<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas">This is Suricata version 3.2.1 RELEASE<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas">Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LUA HAVE_LIBJANSSON TLS MAGIC<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas">SIMD support: SSE_4_2 SSE_4_1 SSE_3<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas">Atomic intrisics: 1 2 4 8 16 byte(s)<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas">64-bits, Little-endian architecture<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas">GCC version 4.8.5 20150623 (Red Hat 4.8.5-11), C version 199901<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas">compiled with _FORTIFY_SOURCE=0<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas">L1 cache line size (CLS)=64<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas">thread local storage method: __thread<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas">compiled with LibHTP v0.5.23, linked against LibHTP v0.5.23<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas">Suricata Configuration:<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> AF_PACKET support: yes<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> PF_RING support: no<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> NFQueue support: no<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> NFLOG support: no<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> IPFW support: no<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> Netmap support: no<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> DAG enabled: no<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> Napatech enabled: no<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> Unix socket enabled: yes<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> Detection enabled: yes<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> Libmagic support: yes<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> libnss support: yes<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> libnspr support: yes<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> libjansson support: yes<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> hiredis support: no<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> Prelude support: no<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> PCRE jit: yes<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> LUA support: yes<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> libluajit: no<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> libgeoip: yes<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> Non-bundled htp: no<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> Old barnyard2 support: no<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> CUDA enabled: no<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> Hyperscan support: yes<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> Libnet support: yes<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> Suricatasc install: yes<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> Profiling enabled: no<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> Profiling locks enabled: no<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas">Development settings:<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> Coccinelle / spatch: no<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> Unit tests enabled: no<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> Debug output enabled: no<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> Debug validation enabled: no<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas">Generic build parameters:<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> Installation prefix: /usr<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> Configuration directory: /etc/suricata/<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> Log directory: /var/log/suricata/<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> --prefix /usr<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> --sysconfdir /etc<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> --localstatedir /var<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> Host: x86_64-pc-linux-gnu<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> Compiler: gcc (exec name) / gcc (real)<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> GCC Protect enabled: no<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> GCC march native enabled: yes<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> GCC Profile enabled: no<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> Position Independent Executable enabled: no<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> CFLAGS -g -O2 -march=native<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> PCAP_CFLAGS<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="font-size:10.0pt;font-family:Consolas"> SECCFLAGS<o:p></o:p></span></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">-----Original Message-----<br>
From: jason taylor [mailto:jtfas90@gmail.com] <br>
Sent: Thursday, April 13, 2017 07:11 AM<br>
To: Duarte Silva <duarte.silva@serializing.me>; Cloherty, Sean E <scloherty@mitre.org>; oisf-users@lists.openinfosecfoundation.org<br>
Subject: Re: [Oisf-users] Battling segfaults on 3.2.1</p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Hi Sean,<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Can you also provide a suricata --build-info?<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">barnyard appears to be segfaulting as well which is curious.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">As Duarte mentioned, gdb is going to be the best bet.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">More than likely there is a shared library in there somewhere causing the problem but gdb will help point in the right direction.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">JT<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">On Thu, 2017-04-13 at 06:52 +0200, Duarte Silva wrote:<o:p></o:p></p>
<p class="MsoPlainText">> Hi Sean,<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> To debug such situations what I do is:<o:p></o:p></p>
<p class="MsoPlainText">> - install Suricata debug symbols<o:p></o:p></p>
<p class="MsoPlainText">> - install gdb<o:p></o:p></p>
<p class="MsoPlainText">> - launch Suricata and attach gdb<o:p></o:p></p>
<p class="MsoPlainText">> - when the error occurs, I look at the call trace and stack to
<o:p></o:p></p>
<p class="MsoPlainText">> determine where the problem is and maybe the why.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> To help reproduce the error, I would have tcpdump creating a network
<o:p></o:p></p>
<p class="MsoPlainText">> packet dump so that I could replay traffic to Suricata.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Cheers,<o:p></o:p></p>
<p class="MsoPlainText">> Duarte<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> De: Cloherty, Sean E<o:p></o:p></p>
<p class="MsoPlainText">> Enviado: 12 de abril de 2017 18:18<o:p></o:p></p>
<p class="MsoPlainText">> Para: <a href="mailto:oisf-users@lists.openinfosecfoundation.org">
<span style="color:windowtext;text-decoration:none">oisf-users@lists.openinfosecfoundation.org</span></a><o:p></o:p></p>
<p class="MsoPlainText">> Assunto: [Oisf-users] Battling segfaults on 3.2.1<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> I am running 3.2.1 on 4 identical servers. Two of them started having
<o:p></o:p></p>
<p class="MsoPlainText">> segfaults and traps.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Troubleshooting - Compared yamls amd found an extra 0 (making the
<o:p></o:p></p>
<p class="MsoPlainText">> tracker 10x larger) in the SMTP mime section for inspected-tracker for
<o:p></o:p></p>
<p class="MsoPlainText">> file data keyword. Also, one system had 2gb vs. 4gb for the http
<o:p></o:p></p>
<p class="MsoPlainText">> memcap in the app layer protocol config. I changed the yamls to match
<o:p></o:p></p>
<p class="MsoPlainText">> the less problematic server. I also took the opportunity to recompile
<o:p></o:p></p>
<p class="MsoPlainText">> Suricata with Hyperscan (Thank you Derek Spransy and Justin Viiret!).<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> On one box I’ve had no segfaults since the April 7th (following the
<o:p></o:p></p>
<p class="MsoPlainText">> changes). The other one continues to have the problem 2-3 times a day
<o:p></o:p></p>
<p class="MsoPlainText">> at random hours – mid-morning, early evening, sometimes after
<o:p></o:p></p>
<p class="MsoPlainText">> midnight. Messages in the system log only include the actual fault
<o:p></o:p></p>
<p class="MsoPlainText">> message and nothing else. The fault always points to a worker thread
<o:p></o:p></p>
<p class="MsoPlainText">> and the numbers vary W#01-ensf1 or W#15-ens1f1 etc. Two types of
<o:p></o:p></p>
<p class="MsoPlainText">> errors come up from segfaults<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> error 4 in suricata[400000+242000] or<o:p></o:p></p>
<p class="MsoPlainText">> error 5 in suricata[400000+242000]<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Trap messages seem to have stopped on April 7th (following the
<o:p></o:p></p>
<p class="MsoPlainText">> changes), but also had error messages with the same info in the
<o:p></o:p></p>
<p class="MsoPlainText">> brackets –<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> error:0 in suricata[400000+242000]<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> I’ve attached a zip file of the startup script, suricata.yaml, the
<o:p></o:p></p>
<p class="MsoPlainText">> suricata.log, stats.log, a copy of the faults listed in the
<o:p></o:p></p>
<p class="MsoPlainText">> /var/log/messages, and a textfule with the time and date of crashes.<o:p></o:p></p>
<p class="MsoPlainText">> The server details follow:<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> GENERAL SERVER INFO :<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> - CentOS Linux release 7.3.1611 (Core) 3.10.0-514.10.2.el7.x86_64 #1
<o:p></o:p></p>
<p class="MsoPlainText">> SMP Fri Mar 3 00:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux<o:p></o:p></p>
<p class="MsoPlainText">> - Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz - 16 cores / 32 threads<o:p></o:p></p>
<p class="MsoPlainText">> - 128GB of RAM<o:p></o:p></p>
<p class="MsoPlainText">> - Capture NIC is a dual port Intel Corporation 82599ES 10-Gigabit
<o:p></o:p></p>
<p class="MsoPlainText">> SFI/SFP+ Network Connection (rev 01)<o:p></o:p></p>
<p class="MsoPlainText">> - NIC Driver is Intel(R) 10GbE PCI Express Linux Network Driver -
<o:p></o:p></p>
<p class="MsoPlainText">> version 4.6.4<o:p></o:p></p>
<p class="MsoPlainText">> - Max traffic seen on the interface in the last 4 months has been 1.2
<o:p></o:p></p>
<p class="MsoPlainText">> Gb/s, but usually mid-day peaks are around 1.1 Gb/s<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Any suggestions of what to check next?<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Sean<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> _______________________________________________<o:p></o:p></p>
<p class="MsoPlainText">> Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org">
<span style="color:windowtext;text-decoration:none">oisf-users@openinfosecfoundation.org</span></a><o:p></o:p></p>
<p class="MsoPlainText">> Site: <a href="http://suricata-ids.org"><span style="color:windowtext;text-decoration:none">http://suricata-ids.org</span></a> | Support:
<a href="http://suricata-"><span style="color:windowtext;text-decoration:none">http://suricata-</span></a>
<o:p></o:p></p>
<p class="MsoPlainText">> ids.org/support/<o:p></o:p></p>
<p class="MsoPlainText">> List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-u">
<span style="color:windowtext;text-decoration:none">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-u</span></a><o:p></o:p></p>
<p class="MsoPlainText">> sers<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
</div>
</body>
</html>