[Oisf-users] meta-data crashes
Jeremy A. Grove
jgrove at quadrantsec.com
Thu Nov 29 18:20:17 UTC 2018
Jeremy Grove, SSCP
Security Engineer
Quadrant Information Security
o: [ callto:(904)296-9100 | (904)296-9100 ] x100
t: [ callto:(800) 538-9357 | (800) 538-9357 ] x100
e: [ mailto:soc at quadrantsec.com | soc at quadrantsec.com ]
Learn more= about our managed SIEM [ https://a.quadrantsec.com/3D%22https://quadrantsec.com/SaganMSSP%22 | people + product ]
From: "Peter Manev" <petermanev at gmail.com>
To: "Jeremy A. Grove" <jgrove at quadrantsec.com>
Cc: "oisf-users" <oisf-users at lists.openinfosecfoundation.org>
Sent: Thursday, November 29, 2018 12:10:46 PM
Subject: Re: [Oisf-users] meta-data crashes
--
BQ_BEGIN
Regards,
BQ_END
BQ_BEGIN
Peter Manev
BQ_END
On 29 Nov 2018, at 17:52, Jeremy A. Grove < [ mailto:jgrove at quadrantsec.com | jgrove at quadrantsec.com ] > wrote:
BQ_BEGIN
Hello All,
We use Suricata in a variety of situations with varying amounts of data input. The version that is currently being used is Suricata version 4.1.0-beta1 and we have upgraded a portion to Suricata version 4.1.0. In both versions I am running into an issue with the meta data where it will stop logging entirely or it will only log some of the protocols. This is while Suricata is still running.
BQ_END
Hi,
Some info questions :
What is the output of “suricata—build-info” of the boxes that are experiencing the issue with 4.1?
Build info for both examples:
This is Suricata version 4.1.0-beta1 RELEASE
Features: PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LIBJANSSON TLS MAGIC
SIMD support: SSE_4_2 SSE_4_1 SSE_3
Atomic intrisics: 1 2 4 8 16 byte(s)
64-bits, Little-endian architecture
GCC version 4.9.2, C version 199901
compiled with _FORTIFY_SOURCE=0
L1 cache line size (CLS)=64
thread local storage method: __thread
compiled with LibHTP v0.5.26, linked against LibHTP v0.5.26
Suricata Configuration:
AF_PACKET support: yes
eBPF support: no
XDP support:
PF_RING support: no
NFQueue support: no
NFLOG support: no
IPFW support: no
Netmap support: no
DAG enabled: no
Napatech enabled: no
Unix socket enabled: yes
Detection enabled: yes
Libmagic support: yes
libnss support: yes
libnspr support: yes
libjansson support: yes
liblzma support: yes
hiredis support: no
hiredis async with libevent: no
Prelude support: no
PCRE jit: yes
LUA support: no
libluajit: no
libgeoip: no
Non-bundled htp: no
Old barnyard2 support: no
Hyperscan support: yes
Libnet support: yes
Rust support (experimental): no
Experimental Rust parsers: no
Rust strict mode: no
Rust debug mode: no
Suricatasc install: yes
Profiling enabled: no
Profiling locks enabled: no
Development settings:
Coccinelle / spatch: no
Unit tests enabled: no
Debug output enabled: no
Debug validation enabled: no
Generic build parameters:
Installation prefix: /usr
Configuration directory: /etc/suricata/
Log directory: /var/log/suricata/
--prefix /usr
--sysconfdir /etc
--localstatedir /var
Host: x86_64-pc-linux-gnu
Compiler: gcc (exec name) / gcc (real)
GCC Protect enabled: no
GCC march native enabled: yes
GCC Profile enabled: no
Position Independent Executable enabled: no
CFLAGS -g -O2 -march=native
PCAP_CFLAGS -I/usr/include
SECCFLAGS
____________________
This is Suricata version 4.1.0 RELEASE
Features: PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LIBJANSSON TLS MAGIC RUST
SIMD support: SSE_4_2 SSE_4_1 SSE_3
Atomic intrisics: 1 2 4 8 16 byte(s)
64-bits, Little-endian architecture
GCC version 6.3.0 20170516, C version 199901
compiled with _FORTIFY_SOURCE=0
L1 cache line size (CLS)=64
thread local storage method: __thread
compiled with LibHTP v0.5.28, linked against LibHTP v0.5.25
Suricata Configuration:
AF_PACKET support: yes
eBPF support: no
XDP support: no
PF_RING support: no
NFQueue support: no
NFLOG support: no
IPFW support: no
Netmap support: no
DAG enabled: no
Napatech enabled: no
WinDivert enabled: no
Unix socket enabled: yes
Detection enabled: yes
Libmagic support: yes
libnss support: yes
libnspr support: yes
libjansson support: yes
liblzma support: no
hiredis support: no
hiredis async with libevent: no
Prelude support: no
PCRE jit: yes
LUA support: no
libluajit: no
libgeoip: no
Non-bundled htp: no
Old barnyard2 support: no
Hyperscan support: yes
Libnet support: yes
liblz4 support: yes
Rust support: yes (default)
Rust strict mode: no
Rust debug mode: no
Rust compiler: rustc 1.30.0 (da5f414c2 2018-10-24)
Rust cargo: cargo 1.30.0 (36d96825d 2018-10-24)
Suricatasc install: yes
Profiling enabled: no
Profiling locks enabled: no
Development settings:
Coccinelle / spatch: no
Unit tests enabled: no
Debug output enabled: no
Debug validation enabled: no
Generic build parameters:
Installation prefix: /usr
Configuration directory: /etc/suricata/
Log directory: /var/log/suricata/
--prefix /usr
--sysconfdir /etc
--localstatedir /var
Host: x86_64-pc-linux-gnu
Compiler: gcc (exec name) / gcc (real)
GCC Protect enabled: no
GCC march native enabled: yes
GCC Profile enabled: no
Position Independent Executable enabled: no
CFLAGS -g -O2 -march=native -I${srcdir}/../rust/gen/c-headers
PCAP_CFLAGS -I/usr/include
SECCFLAGS
How do you start/use Suricata ?(cmd line for example)
I wrap the start line in a shell script and inittab ensures that it is up.
/usr/bin/suricata -vvv -c /etc/suricata/suricata.yaml -F /etc/suricata/bpf.conf --pidfile /var/log/suricata/suricata.pid --af-packet --user=suricata --group=suricata
Does it happen randomly or you can reproduce it at will ?
It happens randomly.... Sometimes it goes for days and sometimes hours.
BQ_BEGIN
An idea of the configuration:
- eve-log:
enabled: yes
filetype: regular #regular|syslog|unix_dgram|unix_stream|redis
filename: /var/log/suricata/flows/current/dns.json
pcap-file: false
types:
- dns:
version: 2
- eve-log:
enabled: yes
filetype: regular #regular|syslog|unix_dgram|unix_stream|redis
filename: /var/log/suricata/flows/current/tls.json
pcap-file: false
types:
- tls:
extended: yes # enable this for extended logging information
- eve-log:
enabled: yes
filetype: regular #regular|syslog|unix_dgram|unix_stream|redis
filename: /var/log/suricata/flows/current/files.json
pcap-file: false
types:
- files:
force-magic: no # force logging magic on all logged files
force-hash: [md5] # force logging of md5 checksums
- eve-log:
enabled: yes
filetype: regular #regular|syslog|unix_dgram|unix_stream|redis
filename: /var/log/suricata/flows/current/http.json
pcap-file: false
types:
- http:
extended: yes # enable this for extended logging information
# custom allows additional http fields to be included in eve-log
# the example below adds three additional fields when uncommented
custom: [accept, accept-charset, accept-encoding, accept-language,
accept-datetime, authorization, cache-control, cookie, from,
max-forwards, origin, pragma, proxy-authorization, range, te, via,
x-requested-with, dnt, x-forwarded-proto, accept-range, age,
allow, connection, content-encoding, content-language,
content-length, content-location, content-md5, content-range,
content-type, date, etags, last-modified, link, location,
proxy-authenticate, referrer, refresh, retry-after, server,
set-cookie, trailer, transfer-encoding, upgrade, vary, warning,
www-authenticate, x-flash-version, x-authenticated-user]
- eve-log:
enabled: yes
filetype: regular #regular|syslog|unix_dgram|unix_stream|redis
filename: /var/log/suricata/flows/current/ssh.json
pcap-file: false
types:
- ssh
- eve-log:
enabled: yes
filetype: regular #regular|syslog|unix_dgram|unix_stream|redis
filename: /var/log/suricata/flows/current/smtp.json
pcap-file: false
types:
- smtp:
extended: yes # enable this for extended logging information
custom: [received, x-mailer, x-originating-ip, relays, reply-to, bcc]
- eve-log:
enabled: yes
filetype: regular #regular|syslog|unix_dgram|unix_stream|redis
filename: /var/log/suricata/flows/current/flow.json
pcap-file: false
types:
- flow
- eve-log:
enabled: yes
filetype: regular #regular|syslog|unix_dgram|unix_stream|redis
filename: /var/log/suricata/flows/current/nfs.json
pcap-file: false
types:
- nfs
- eve-log:
enabled: yes
filetype: regular #regular|syslog|unix_dgram|unix_stream|redis
filename: /var/log/suricata/flows/current/smb.json
pcap-file: false
types:
- smb
- eve-log:
enabled: yes
filetype: regular #regular|syslog|unix_dgram|unix_stream|redis
filename: /var/log/suricata/flows/current/tftp.json
pcap-file: false
types:
- tftp
- eve-log:
enabled: yes
filetype: regular #regular|syslog|unix_dgram|unix_stream|redis
filename: /var/log/suricata/flows/current/ikev2.json
pcap-file: false
types:
- ikev2
- eve-log:
enabled: yes
filetype: regular #regular|syslog|unix_dgram|unix_stream|redis
filename: /var/log/suricata/flows/current/dhcp.json
pcap-file: false
types:
- dhcp
I am happy to provide more detail to anyone willing to give an opinion on how this may be addressed but I am not sure exactly what is useful to know.
Has anyone else seen this behavior? This is a critical piece for us and I will need to switch back to Bro/Zeek until I can get this resolved.
Regards,
Jeremy Grove, SSCP
Security Engineer
Quadrant Information Security
Learn more= about our managed SIEM [ https://a.quadrantsec.com/3D%22https://quadrantsec.com/SaganMSSP%22 | people + product ]
BQ_END
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20181129/c9c4a30c/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2131 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20181129/c9c4a30c/attachment-0001.bin>
More information about the Oisf-users
mailing list