[Oisf-users] Battling segfaults on 3.2.1
Cloherty, Sean E
scloherty at mitre.org
Thu Apr 13 13:25:32 UTC 2017
Here is the build info:
This is Suricata version 3.2.1 RELEASE
Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LUA HAVE_LIBJANSSON TLS MAGIC
SIMD support: SSE_4_2 SSE_4_1 SSE_3
Atomic intrisics: 1 2 4 8 16 byte(s)
64-bits, Little-endian architecture
GCC version 4.8.5 20150623 (Red Hat 4.8.5-11), C version 199901
compiled with _FORTIFY_SOURCE=0
L1 cache line size (CLS)=64
thread local storage method: __thread
compiled with LibHTP v0.5.23, linked against LibHTP v0.5.23
Suricata Configuration:
AF_PACKET support: yes
PF_RING support: no
NFQueue support: no
NFLOG support: no
IPFW support: no
Netmap support: no
DAG enabled: no
Napatech enabled: no
Unix socket enabled: yes
Detection enabled: yes
Libmagic support: yes
libnss support: yes
libnspr support: yes
libjansson support: yes
hiredis support: no
Prelude support: no
PCRE jit: yes
LUA support: yes
libluajit: no
libgeoip: yes
Non-bundled htp: no
Old barnyard2 support: no
CUDA enabled: no
Hyperscan support: yes
Libnet support: yes
Suricatasc install: yes
Profiling enabled: no
Profiling locks enabled: no
Development settings:
Coccinelle / spatch: no
Unit tests enabled: no
Debug output enabled: no
Debug validation enabled: no
Generic build parameters:
Installation prefix: /usr
Configuration directory: /etc/suricata/
Log directory: /var/log/suricata/
--prefix /usr
--sysconfdir /etc
--localstatedir /var
Host: x86_64-pc-linux-gnu
Compiler: gcc (exec name) / gcc (real)
GCC Protect enabled: no
GCC march native enabled: yes
GCC Profile enabled: no
Position Independent Executable enabled: no
CFLAGS -g -O2 -march=native
PCAP_CFLAGS
SECCFLAGS
-----Original Message-----
From: jason taylor [mailto:jtfas90 at gmail.com]
Sent: Thursday, April 13, 2017 07:11 AM
To: Duarte Silva <duarte.silva at serializing.me>; Cloherty, Sean E <scloherty at mitre.org>; oisf-users at lists.openinfosecfoundation.org
Subject: Re: [Oisf-users] Battling segfaults on 3.2.1
Hi Sean,
Can you also provide a suricata --build-info?
barnyard appears to be segfaulting as well which is curious.
As Duarte mentioned, gdb is going to be the best bet.
More than likely there is a shared library in there somewhere causing the problem but gdb will help point in the right direction.
JT
On Thu, 2017-04-13 at 06:52 +0200, Duarte Silva wrote:
> Hi Sean,
>
> To debug such situations what I do is:
> - install Suricata debug symbols
> - install gdb
> - launch Suricata and attach gdb
> - when the error occurs, I look at the call trace and stack to
> determine where the problem is and maybe the why.
>
> To help reproduce the error, I would have tcpdump creating a network
> packet dump so that I could replay traffic to Suricata.
>
> Cheers,
> Duarte
>
> De: Cloherty, Sean E
> Enviado: 12 de abril de 2017 18:18
> Para: oisf-users at lists.openinfosecfoundation.org<mailto:oisf-users at lists.openinfosecfoundation.org>
> Assunto: [Oisf-users] Battling segfaults on 3.2.1
>
> I am running 3.2.1 on 4 identical servers. Two of them started having
> segfaults and traps.
>
> Troubleshooting - Compared yamls amd found an extra 0 (making the
> tracker 10x larger) in the SMTP mime section for inspected-tracker for
> file data keyword. Also, one system had 2gb vs. 4gb for the http
> memcap in the app layer protocol config. I changed the yamls to match
> the less problematic server. I also took the opportunity to recompile
> Suricata with Hyperscan (Thank you Derek Spransy and Justin Viiret!).
>
> On one box I’ve had no segfaults since the April 7th (following the
> changes). The other one continues to have the problem 2-3 times a day
> at random hours – mid-morning, early evening, sometimes after
> midnight. Messages in the system log only include the actual fault
> message and nothing else. The fault always points to a worker thread
> and the numbers vary W#01-ensf1 or W#15-ens1f1 etc. Two types of
> errors come up from segfaults
>
> error 4 in suricata[400000+242000] or
> error 5 in suricata[400000+242000]
>
> Trap messages seem to have stopped on April 7th (following the
> changes), but also had error messages with the same info in the
> brackets –
>
> error:0 in suricata[400000+242000]
>
>
> I’ve attached a zip file of the startup script, suricata.yaml, the
> suricata.log, stats.log, a copy of the faults listed in the
> /var/log/messages, and a textfule with the time and date of crashes.
> The server details follow:
>
> GENERAL SERVER INFO :
>
> - CentOS Linux release 7.3.1611 (Core) 3.10.0-514.10.2.el7.x86_64 #1
> SMP Fri Mar 3 00:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
> - Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz - 16 cores / 32 threads
> - 128GB of RAM
> - Capture NIC is a dual port Intel Corporation 82599ES 10-Gigabit
> SFI/SFP+ Network Connection (rev 01)
> - NIC Driver is Intel(R) 10GbE PCI Express Linux Network Driver -
> version 4.6.4
> - Max traffic seen on the interface in the last 4 months has been 1.2
> Gb/s, but usually mid-day peaks are around 1.1 Gb/s
>
>
> Any suggestions of what to check next?
>
> Sean
>
>
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org<mailto:oisf-users at openinfosecfoundation.org>
> Site: http://suricata-ids.org | Support: http://suricata-
> ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-u
> sers
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20170413/fe3d7739/attachment-0002.html>
More information about the Oisf-users
mailing list