[Oisf-users] Battling segfaults on 3.2.1

Cloherty, Sean E scloherty at mitre.org
Thu Apr 13 13:25:32 UTC 2017


Here is the build info:



This is Suricata version 3.2.1 RELEASE

Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LUA HAVE_LIBJANSSON TLS MAGIC

SIMD support: SSE_4_2 SSE_4_1 SSE_3

Atomic intrisics: 1 2 4 8 16 byte(s)

64-bits, Little-endian architecture

GCC version 4.8.5 20150623 (Red Hat 4.8.5-11), C version 199901

compiled with _FORTIFY_SOURCE=0

L1 cache line size (CLS)=64

thread local storage method: __thread

compiled with LibHTP v0.5.23, linked against LibHTP v0.5.23



Suricata Configuration:

  AF_PACKET support:                       yes

  PF_RING support:                         no

  NFQueue support:                         no

  NFLOG support:                           no

  IPFW support:                            no

  Netmap support:                          no

  DAG enabled:                             no

  Napatech enabled:                        no



  Unix socket enabled:                     yes

  Detection enabled:                       yes



  Libmagic support:                        yes

  libnss support:                          yes

  libnspr support:                         yes

  libjansson support:                      yes

  hiredis support:                         no

  Prelude support:                         no

  PCRE jit:                                yes

  LUA support:                             yes

  libluajit:                               no

  libgeoip:                                yes

  Non-bundled htp:                         no

  Old barnyard2 support:                   no

  CUDA enabled:                            no

  Hyperscan support:                       yes

  Libnet support:                          yes



  Suricatasc install:                      yes



  Profiling enabled:                       no

  Profiling locks enabled:                 no



Development settings:

  Coccinelle / spatch:                     no

  Unit tests enabled:                      no

  Debug output enabled:                    no

  Debug validation enabled:                no



Generic build parameters:

  Installation prefix:                     /usr

  Configuration directory:                 /etc/suricata/

  Log directory:                           /var/log/suricata/



  --prefix                                 /usr

  --sysconfdir                             /etc

  --localstatedir                          /var



  Host:                                    x86_64-pc-linux-gnu

  Compiler:                                gcc (exec name) / gcc (real)

  GCC Protect enabled:                     no

  GCC march native enabled:                yes

  GCC Profile enabled:                     no

  Position Independent Executable enabled: no

  CFLAGS                                   -g -O2 -march=native

  PCAP_CFLAGS

  SECCFLAGS





-----Original Message-----
From: jason taylor [mailto:jtfas90 at gmail.com]
Sent: Thursday, April 13, 2017 07:11 AM
To: Duarte Silva <duarte.silva at serializing.me>; Cloherty, Sean E <scloherty at mitre.org>; oisf-users at lists.openinfosecfoundation.org
Subject: Re: [Oisf-users] Battling segfaults on 3.2.1



Hi Sean,



Can you also provide a suricata --build-info?



barnyard appears to be segfaulting as well which is curious.





As Duarte mentioned, gdb is going to be the best bet.



More than likely there is a shared library in there somewhere causing the problem but gdb will help point in the right direction.



JT



On Thu, 2017-04-13 at 06:52 +0200, Duarte Silva wrote:

> Hi Sean,

>

> To debug such situations what I do is:

> - install Suricata debug symbols

> - install gdb

> - launch Suricata and attach gdb

> - when the error occurs, I look at the call trace and stack to

> determine where the problem is and maybe the why.

>

> To help reproduce the error, I would have tcpdump creating a network

> packet dump so that I could replay traffic to Suricata.

>

> Cheers,

> Duarte

>

> De: Cloherty, Sean E

> Enviado: 12 de abril de 2017 18:18

> Para: oisf-users at lists.openinfosecfoundation.org<mailto:oisf-users at lists.openinfosecfoundation.org>

> Assunto: [Oisf-users] Battling segfaults on 3.2.1

>

> I am running 3.2.1 on 4 identical servers.  Two of them started having

> segfaults and traps.

>

> Troubleshooting - Compared yamls amd found an extra 0 (making the

> tracker 10x larger) in the SMTP mime section for inspected-tracker for

> file data keyword.  Also, one system had 2gb vs. 4gb for the http

> memcap in the app layer protocol config.  I changed the yamls to match

> the less problematic server.  I also took the opportunity to recompile

> Suricata with Hyperscan (Thank you Derek Spransy and Justin Viiret!).

>

> On one box I’ve had no segfaults since the April 7th (following the

> changes). The other one continues to have the problem 2-3 times a day

> at random hours – mid-morning, early evening, sometimes after

> midnight. Messages in the system log only include the actual fault

> message and nothing else. The fault always points to a worker thread

> and the numbers vary W#01-ensf1 or  W#15-ens1f1 etc.   Two types of

> errors come up from segfaults

>

> error 4 in suricata[400000+242000] or

> error 5 in suricata[400000+242000]

>

> Trap messages seem to have stopped on April 7th (following the

> changes), but also had error messages with the same info in the

> brackets –

>

> error:0 in suricata[400000+242000]

>

>

> I’ve attached a zip file of the startup script, suricata.yaml, the

> suricata.log, stats.log, a copy of the faults listed in the

> /var/log/messages, and a textfule with the time and date of crashes.

> The server details follow:

>

> GENERAL SERVER INFO :

>

> - CentOS Linux release 7.3.1611 (Core) 3.10.0-514.10.2.el7.x86_64 #1

> SMP Fri Mar 3 00:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

> - Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz - 16 cores / 32 threads

> - 128GB of RAM

> - Capture NIC is a dual port Intel Corporation 82599ES 10-Gigabit

> SFI/SFP+ Network Connection (rev 01)

> - NIC Driver is Intel(R) 10GbE PCI Express Linux Network Driver -

> version 4.6.4

> - Max traffic seen on the interface in the last 4 months has been 1.2

> Gb/s, but usually mid-day peaks are around 1.1 Gb/s

>

>

> Any suggestions of what to check next?

>

> Sean

>

>

> _______________________________________________

> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org<mailto:oisf-users at openinfosecfoundation.org>

> Site: http://suricata-ids.org | Support: http://suricata-

> ids.org/support/

> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-u

> sers


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20170413/fe3d7739/attachment-0002.html>


More information about the Oisf-users mailing list