<html><body><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; color: #000000"><div><br></div><div><br></div><div data-marker="__SIG_PRE__"><div><span style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;">Jeremy Grove, SSCP</span><br style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;"><span style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;">Security Engineer</span><br style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;"><span style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;">Quadrant Information Security</span><br style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;"><span style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;">o: </span><span class="Object" id="OBJ_PREFIX_DWT146_com_zimbra_phone" style="color: #005a95; cursor: pointer; font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;"><a href="callto:(904)296-9100" style="color: #005a95; text-decoration: none; cursor: pointer;" target="_blank">(904)296-9100</a></span><span style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;"> x100</span><br style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;"><span style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;">t: </span><span class="Object" id="OBJ_PREFIX_DWT147_com_zimbra_phone" style="color: #005a95; cursor: pointer; font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;"><a href="callto:(800) 538-9357" style="color: #005a95; text-decoration: none; cursor: pointer;" target="_blank">(800) 538-9357</a></span><span style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;"> x100</span><br style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;"><span style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;">e:</span><span style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;"> </span><span class="Object" id="OBJ_PREFIX_DWT148_ZmEmailObjectHandler" style="color: #005a95; cursor: pointer; font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;"><a class="moz-txt-link-abbreviated" href="mailto:soc@quadrantsec.com" target="_blank" style="color: #005a95; text-decoration: none; cursor: pointer;">soc@quadrantsec.com</a></span><br style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;"><br style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;"><span style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;">Learn more= about our managed SIEM <span class="Object" id="OBJ_PREFIX_DWT149_com_zimbra_url" style="color: #005a95; cursor: pointer;"><a href="https://a.quadrantsec.com/3D%22https://quadrantsec.com/SaganMSSP%22" target="_blank" style="color: #005a95; text-decoration: none; cursor: pointer;">people + product</a></span></span><br><br><br></div></div><div><br></div><hr id="zwchr" data-marker="__DIVIDER__"><div data-marker="__HEADERS__"><b>From: </b>"Peter Manev" <petermanev@gmail.com><br><b>To: </b>"Jeremy A. Grove" <jgrove@quadrantsec.com><br><b>Cc: </b>"oisf-users" <oisf-users@lists.openinfosecfoundation.org><br><b>Sent: </b>Thursday, November 29, 2018 12:10:46 PM<br><b>Subject: </b>Re: [Oisf-users] meta-data crashes<br></div><div><br></div><div data-marker="__QUOTED_TEXT__"><br><br><div id="AppleMailSignature" dir="ltr"><blockquote><span style="background-color: ;">-- <br></span></blockquote><blockquote><span style="background-color: ;">Regards,<br></span></blockquote><blockquote><span style="background-color: ;">Peter Manev </span></blockquote></div><div dir="ltr"><br>On 29 Nov 2018, at 17:52, Jeremy A. Grove <<a href="mailto:jgrove@quadrantsec.com" target="_blank">jgrove@quadrantsec.com</a>> wrote:<br><br></div><div dir="ltr"><span></span></div><blockquote><div dir="ltr"><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; color: #000000"><div>Hello All,</div><br><div>We use Suricata in a variety of situations with varying amounts of data input. The version that is currently being used is Suricata version 4.1.0-beta1 and we have upgraded a portion to Suricata version 4.1.0. In both versions I am running into an issue with the meta data where it will stop logging entirely or it will only log some of the protocols. This is while Suricata is still running. </div><br></div></div></blockquote><br><div>Hi,</div><br><div>Some info questions :</div><div>What is the output of “suricata—build-info” of the boxes that are experiencing the issue with 4.1?</div><div><br data-mce-bogus="1"></div><div>Build info for both examples:</div><div><br data-mce-bogus="1"></div><div><div>This is Suricata version 4.1.0-beta1 RELEASE</div><div>Features: PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LIBJANSSON TLS MAGIC </div><div>SIMD support: SSE_4_2 SSE_4_1 SSE_3 </div><div>Atomic intrisics: 1 2 4 8 16 byte(s)</div><div>64-bits, Little-endian architecture</div><div>GCC version 4.9.2, C version 199901</div><div>compiled with _FORTIFY_SOURCE=0</div><div>L1 cache line size (CLS)=64</div><div>thread local storage method: __thread</div><div>compiled with LibHTP v0.5.26, linked against LibHTP v0.5.26</div><div><br></div><div>Suricata Configuration:</div><div> AF_PACKET support: yes</div><div> eBPF support: no</div><div> XDP support: </div><div> PF_RING support: no</div><div> NFQueue support: no</div><div> NFLOG support: no</div><div> IPFW support: no</div><div> Netmap support: no</div><div> DAG enabled: no</div><div> Napatech enabled: no</div><div><br></div><div> Unix socket enabled: yes</div><div> Detection enabled: yes</div><div><br></div><div> Libmagic support: yes</div><div> libnss support: yes</div><div> libnspr support: yes</div><div> libjansson support: yes</div><div> liblzma support: yes</div><div> hiredis support: no</div><div> hiredis async with libevent: no</div><div> Prelude support: no</div><div> PCRE jit: yes</div><div> LUA support: no</div><div> libluajit: no</div><div> libgeoip: no</div><div> Non-bundled htp: no</div><div> Old barnyard2 support: no</div><div> Hyperscan support: yes</div><div> Libnet support: yes</div><div><br></div><div> Rust support (experimental): no</div><div> Experimental Rust parsers: no</div><div> Rust strict mode: no</div><div> Rust debug mode: no</div><div><br></div><div> Suricatasc install: yes</div><div><br></div><div> Profiling enabled: no</div><div> Profiling locks enabled: no</div><div><br></div><div>Development settings:</div><div> Coccinelle / spatch: no</div><div> Unit tests enabled: no</div><div> Debug output enabled: no</div><div> Debug validation enabled: no</div><div><br></div><div>Generic build parameters:</div><div> Installation prefix: /usr</div><div> Configuration directory: /etc/suricata/</div><div> Log directory: /var/log/suricata/</div><div><br></div><div> --prefix /usr</div><div> --sysconfdir /etc</div><div> --localstatedir /var</div><div><br></div><div> Host: x86_64-pc-linux-gnu</div><div> Compiler: gcc (exec name) / gcc (real)</div><div> GCC Protect enabled: no</div><div> GCC march native enabled: yes</div><div> GCC Profile enabled: no</div><div> Position Independent Executable enabled: no</div><div> CFLAGS -g -O2 -march=native</div><div> PCAP_CFLAGS -I/usr/include</div><div> SECCFLAGS </div></div><div>____________________</div><div><br data-mce-bogus="1"></div><div><div>This is Suricata version 4.1.0 RELEASE</div><div>Features: PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LIBJANSSON TLS MAGIC RUST </div><div>SIMD support: SSE_4_2 SSE_4_1 SSE_3 </div><div>Atomic intrisics: 1 2 4 8 16 byte(s)</div><div>64-bits, Little-endian architecture</div><div>GCC version 6.3.0 20170516, C version 199901</div><div>compiled with _FORTIFY_SOURCE=0</div><div>L1 cache line size (CLS)=64</div><div>thread local storage method: __thread</div><div>compiled with LibHTP v0.5.28, linked against LibHTP v0.5.25</div><div><br></div><div>Suricata Configuration:</div><div> AF_PACKET support: yes</div><div> eBPF support: no</div><div> XDP support: no</div><div> PF_RING support: no</div><div> NFQueue support: no</div><div> NFLOG support: no</div><div> IPFW support: no</div><div> Netmap support: no</div><div> DAG enabled: no</div><div> Napatech enabled: no</div><div> WinDivert enabled: no</div><div><br></div><div> Unix socket enabled: yes</div><div> Detection enabled: yes</div><div><br></div><div> Libmagic support: yes</div><div> libnss support: yes</div><div> libnspr support: yes</div><div> libjansson support: yes</div><div> liblzma support: no</div><div> hiredis support: no</div><div> hiredis async with libevent: no</div><div> Prelude support: no</div><div> PCRE jit: yes</div><div> LUA support: no</div><div> libluajit: no</div><div> libgeoip: no</div><div> Non-bundled htp: no</div><div> Old barnyard2 support: no</div><div> Hyperscan support: yes</div><div> Libnet support: yes</div><div> liblz4 support: yes</div><div><br></div><div> Rust support: yes (default)</div><div> Rust strict mode: no</div><div> Rust debug mode: no</div><div> Rust compiler: rustc 1.30.0 (da5f414c2 2018-10-24)</div><div> Rust cargo: cargo 1.30.0 (36d96825d 2018-10-24)</div><div><br></div><div> Suricatasc install: yes</div><div><br></div><div> Profiling enabled: no</div><div> Profiling locks enabled: no</div><div><br></div><div>Development settings:</div><div> Coccinelle / spatch: no</div><div> Unit tests enabled: no</div><div> Debug output enabled: no</div><div> Debug validation enabled: no</div><div><br></div><div>Generic build parameters:</div><div> Installation prefix: /usr</div><div> Configuration directory: /etc/suricata/</div><div> Log directory: /var/log/suricata/</div><div><br></div><div> --prefix /usr</div><div> --sysconfdir /etc</div><div> --localstatedir /var</div><div><br></div><div> Host: x86_64-pc-linux-gnu</div><div> Compiler: gcc (exec name) / gcc (real)</div><div> GCC Protect enabled: no</div><div> GCC march native enabled: yes</div><div> GCC Profile enabled: no</div><div> Position Independent Executable enabled: no</div><div> CFLAGS -g -O2 -march=native -I${srcdir}/../rust/gen/c-headers</div><div> PCAP_CFLAGS -I/usr/include</div><div> SECCFLAGS </div></div><br><div>How do you start/use Suricata ?(cmd line for example)</div><div><br data-mce-bogus="1"></div><div>I wrap the start line in a shell script and inittab ensures that it is up.</div><div><br data-mce-bogus="1"></div><div>/usr/bin/suricata -vvv -c /etc/suricata/suricata.yaml -F /etc/suricata/bpf.conf --pidfile /var/log/suricata/suricata.pid --af-packet --user=suricata --group=suricata<br data-mce-bogus="1"></div><br><div>Does it happen randomly or you can reproduce it at will ?</div><div><br data-mce-bogus="1"></div><div>It happens randomly.... Sometimes it goes for days and sometimes hours.</div><br><br><blockquote><div dir="ltr"><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; color: #000000"><div>An idea of the configuration:</div><br><div><div> - eve-log:</div><div> enabled: yes</div><div> filetype: regular #regular|syslog|unix_dgram|unix_stream|redis</div><div> filename: /var/log/suricata/flows/current/dns.json</div><div> pcap-file: false</div><div> types:</div><div> - dns:</div><div> version: 2 </div><div> - eve-log:</div><div> enabled: yes</div><div> filetype: regular #regular|syslog|unix_dgram|unix_stream|redis</div><div> filename: /var/log/suricata/flows/current/tls.json</div><div> pcap-file: false</div><div> types:</div><div> - tls:</div><div> extended: yes # enable this for extended logging information</div><div> - eve-log:</div><div> enabled: yes</div><div> filetype: regular #regular|syslog|unix_dgram|unix_stream|redis</div><div> filename: /var/log/suricata/flows/current/files.json</div><div> pcap-file: false</div><div> types:</div><div> - files:</div><div> force-magic: no # force logging magic on all logged files</div><div> force-hash: [md5] # force logging of md5 checksums</div><div> - eve-log:</div><div> enabled: yes</div><div> filetype: regular #regular|syslog|unix_dgram|unix_stream|redis</div><div> filename: /var/log/suricata/flows/current/http.json</div><div> pcap-file: false</div><div> types: </div><div> - http:</div><div> extended: yes # enable this for extended logging information</div><div> # custom allows additional http fields to be included in eve-log</div><div> # the example below adds three additional fields when uncommented</div><div> custom: [accept, accept-charset, accept-encoding, accept-language,</div><div> accept-datetime, authorization, cache-control, cookie, from,</div><div> max-forwards, origin, pragma, proxy-authorization, range, te, via,</div><div> x-requested-with, dnt, x-forwarded-proto, accept-range, age,</div><div> allow, connection, content-encoding, content-language,</div><div> content-length, content-location, content-md5, content-range,</div><div> content-type, date, etags, last-modified, link, location,</div><div> proxy-authenticate, referrer, refresh, retry-after, server,</div><div> set-cookie, trailer, transfer-encoding, upgrade, vary, warning,</div><div> www-authenticate, x-flash-version, x-authenticated-user]</div><div> - eve-log:</div><div> enabled: yes</div><div> filetype: regular #regular|syslog|unix_dgram|unix_stream|redis</div><div> filename: /var/log/suricata/flows/current/ssh.json</div><div> pcap-file: false</div><div> types:</div><div> - ssh</div><div> - eve-log:</div><div> enabled: yes</div><div> filetype: regular #regular|syslog|unix_dgram|unix_stream|redis</div><div> filename: /var/log/suricata/flows/current/smtp.json</div><div> pcap-file: false</div><div> types:</div><div> - smtp:</div><div> extended: yes # enable this for extended logging information</div><div> custom: [received, x-mailer, x-originating-ip, relays, reply-to, bcc]</div><div> - eve-log:</div><div> enabled: yes</div><div> filetype: regular #regular|syslog|unix_dgram|unix_stream|redis</div><div> filename: /var/log/suricata/flows/current/flow.json</div><div> pcap-file: false</div><div> types:</div><div> - flow</div><div> - eve-log:</div><div> enabled: yes</div><div> filetype: regular #regular|syslog|unix_dgram|unix_stream|redis</div><div> filename: /var/log/suricata/flows/current/nfs.json</div><div> pcap-file: false</div><div> types:</div><div> - nfs</div><div> - eve-log:</div><div> enabled: yes</div><div> filetype: regular #regular|syslog|unix_dgram|unix_stream|redis</div><div> filename: /var/log/suricata/flows/current/smb.json</div><div> pcap-file: false</div><div> types:</div><div> - smb</div><div> - eve-log:</div><div> enabled: yes</div><div> filetype: regular #regular|syslog|unix_dgram|unix_stream|redis</div><div> filename: /var/log/suricata/flows/current/tftp.json</div><div> pcap-file: false</div><div> types:</div><div> - tftp</div><div> - eve-log:</div><div> enabled: yes</div><div> filetype: regular #regular|syslog|unix_dgram|unix_stream|redis</div><div> filename: /var/log/suricata/flows/current/ikev2.json</div><div> pcap-file: false</div><div> types:</div><div> - ikev2</div><div> - eve-log:</div><div> enabled: yes</div><div> filetype: regular #regular|syslog|unix_dgram|unix_stream|redis</div><div> filename: /var/log/suricata/flows/current/dhcp.json</div><div> pcap-file: false</div><div> types:</div><div> - dhcp</div></div><br><div>I am happy to provide more detail to anyone willing to give an opinion on how this may be addressed but I am not sure exactly what is useful to know. </div><br><div>Has anyone else seen this behavior? This is a critical piece for us and I will need to switch back to Bro/Zeek until I can get this resolved.</div><br><div>Regards,</div><br><div><div><span style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;">Jeremy Grove, SSCP</span><br style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;"><span style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;">Security Engineer</span><br style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;"><span style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;">Quadrant Information Security</span><br style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;"><br style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;"><br style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;"><span style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;">Learn more= about our managed SIEM <span class="Object" id="OBJ_PREFIX_DWT149_com_zimbra_url" style="color: #005a95; cursor: pointer;"><a href="https://a.quadrantsec.com/3D%22https://quadrantsec.com/SaganMSSP%22" target="_blank" style="color: #005a95; text-decoration: none; cursor: pointer;">people + product</a></span></span></div></div></div></div></blockquote><br></div></div></body></html>