[Oisf-users] Suffering Simultaneous Suricata Segfaults
Cloherty, Sean E
scloherty at mitre.org
Fri Oct 19 17:38:53 UTC 2018
I've finally got the PCAP from that time, ran it through Suricata using the -r file.pcap command and got a satisfying segfault message but no core dump. I've attached the --buildiinfo output and this was the command I used when recompiling:
./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var --enable-lua --enable-unix-socket --enable-geoip --with-libhs-includes=/usr/local/include/hs/ --with-libhs-libraries=/usr/local/lib/ --with-liblzma-includes=/usr/include/lzma/ --with-liblzma-libraries=/usr/lib64/ --enable-profiling --enable-debug --enable-debug-validation CFLAGS="-ggdb -O0"
Below are the messages that went to the console - I guess that the LIBHTP error was because I used the yaml from a 4.1 server. The other SC_ERR_THREAD_NICE_PRIO must be related to worker threads which don’t get spawned in offline pcap reading ? My affinity settings only have 9 (11-19) so that is odd.
[root at idstestmclean suricata]# suricata -c /etc/suricata/test1/suricata.yaml -r /tmp/Sean2.pcap -l /etc/suricata/test1
19/10/2018 -- 13:05:23 - <Info> - Including configuration file vars.yaml.
19/10/2018 -- 13:05:23 - <Notice> - This is Suricata version 4.0.5 RELEASE
19/10/2018 -- 13:05:23 - <Warning> - [ERRCODE: SC_ERR_UNKNOWN_VALUE(129)] - LIBHTP Ignoring unknown default config: swf-decompression
19/10/2018 -- 13:05:27 - <Error> - [ERRCODE: SC_ERR_THREAD_NICE_PRIO(47)] - Error setting nice value -2 for thread W#01: Operation not permitted
19/10/2018 -- 13:05:27 - <Error> - [ERRCODE: SC_ERR_THREAD_NICE_PRIO(47)] - Error setting nice value -2 for thread W#02: Operation not permitted
19/10/2018 -- 13:05:27 - <Error> - [ERRCODE: SC_ERR_THREAD_NICE_PRIO(47)] - Error setting nice value -2 for thread W#03: Operation not permitted
19/10/2018 -- 13:05:27 - <Error> - [ERRCODE: SC_ERR_THREAD_NICE_PRIO(47)] - Error setting nice value -2 for thread W#04: Operation not permitted
19/10/2018 -- 13:05:27 - <Error> - [ERRCODE: SC_ERR_THREAD_NICE_PRIO(47)] - Error setting nice value -2 for thread W#05: Operation not permitted
19/10/2018 -- 13:05:27 - <Error> - [ERRCODE: SC_ERR_THREAD_NICE_PRIO(47)] - Error setting nice value -2 for thread W#06: Operation not permitted
19/10/2018 -- 13:05:27 - <Error> - [ERRCODE: SC_ERR_THREAD_NICE_PRIO(47)] - Error setting nice value -2 for thread W#07: Operation not permitted
19/10/2018 -- 13:05:27 - <Error> - [ERRCODE: SC_ERR_THREAD_NICE_PRIO(47)] - Error setting nice value -2 for thread W#08: Operation not permitted
19/10/2018 -- 13:05:27 - <Error> - [ERRCODE: SC_ERR_THREAD_NICE_PRIO(47)] - Error setting nice value -2 for thread W#09: Operation not permitted
19/10/2018 -- 13:05:27 - <Error> - [ERRCODE: SC_ERR_THREAD_NICE_PRIO(47)] - Error setting nice value -2 for thread W#10: Operation not permitted
19/10/2018 -- 13:05:27 - <Error> - [ERRCODE: SC_ERR_THREAD_NICE_PRIO(47)] - Error setting nice value -2 for thread W#11: Operation not permitted
19/10/2018 -- 13:05:27 - <Error> - [ERRCODE: SC_ERR_THREAD_NICE_PRIO(47)] - Error setting nice value -2 for thread W#12: Operation not permitted
19/10/2018 -- 13:05:27 - <Error> - [ERRCODE: SC_ERR_THREAD_NICE_PRIO(47)] - Error setting nice value -2 for thread W#13: Operation not permitted
19/10/2018 -- 13:05:27 - <Error> - [ERRCODE: SC_ERR_THREAD_NICE_PRIO(47)] - Error setting nice value -2 for thread W#14: Operation not permitted
19/10/2018 -- 13:05:27 - <Error> - [ERRCODE: SC_ERR_THREAD_NICE_PRIO(47)] - Error setting nice value -2 for thread W#15: Operation not permitted
19/10/2018 -- 13:05:27 - <Error> - [ERRCODE: SC_ERR_THREAD_NICE_PRIO(47)] - Error setting nice value -2 for thread W#16: Operation not permitted
19/10/2018 -- 13:05:27 - <Error> - [ERRCODE: SC_ERR_THREAD_NICE_PRIO(47)] - Error setting nice value -2 for thread W#17: Operation not permitted
19/10/2018 -- 13:05:27 - <Error> - [ERRCODE: SC_ERR_THREAD_NICE_PRIO(47)] - Error setting nice value -2 for thread W#18: Operation not permitted
19/10/2018 -- 13:05:27 - <Error> - [ERRCODE: SC_ERR_THREAD_NICE_PRIO(47)] - Error setting nice value -2 for thread W#19: Operation not permitted
19/10/2018 -- 13:05:27 - <Error> - [ERRCODE: SC_ERR_THREAD_NICE_PRIO(47)] - Error setting nice value -2 for thread W#20: Operation not permitted
19/10/2018 -- 13:05:27 - <Notice> - all 21 packet processing threads, 4 management threads initialized, engine started.
Segmentation fault
-----Original Message-----
From: Greg Grasmehr <greg.grasmehr at caltech.edu>
Sent: Friday, October 5, 2018 2:36 PM
To: Cloherty, Sean E <scloherty at mitre.org>
Cc: lists at inliniac.net; oisf-users at lists.openinfosecfoundation.org
Subject: Re: [Oisf-users] Suffering Simultaneous Suricata Segfaults
Yes I will check; there are many groups on campus that run their own mail servers, if it's one of them we won't have info. Should be able to get back to you regarding this by Monday.
Greg
On 10/05/18 17:21:42, Cloherty, Sean E wrote:
> Greg & Cooper
>
> Just a shot in the dark here --- Do you have email logs from that time period? I was wondering if you might see email that has a to or from each other's domains. Or that our domains are in the same TO:
>
> That might help narrow the scope of where to look.
>
> Sean
>
> -----Original Message-----
> From: Greg Grasmehr <greg.grasmehr at caltech.edu>
> Sent: Thursday, September 27, 2018 1:36 PM
> To: lists at inliniac.net
> Cc: Cloherty, Sean E <scloherty at mitre.org>;
> oisf-users at lists.openinfosecfoundation.org
> Subject: Re: [Oisf-users] Suffering Simultaneous Suricata Segfaults
>
> Hello,
>
> Having the same issue, pointer to the problem code is below
>
>
> addr2line -e /opt/suricata/bin/suricata 000000000055ae67
> /root/installers/suricata-4.0.5/src/util-decode-mime.c:2330
>
> /var/log/messages:Sep 25 15:28:19 is-pig3 kernel: W#45[31078]:
> segfault at 0 ip 000000000055ae67 sp 00007f9af7ffd8a0 error 4 in
> suricata[400000+1f7000] /var/log/messages:Sep 26 03:17:00 is-pig3
> kernel: W#53[11268]: segfault at 0 ip 000000000055ae67 sp
> 00007fbbf97f88a0 error 4 in suricata[400000+1f7000]
> /var/log/messages-20180909:Sep 7 03:29:12 is-pig3 kernel:
> W#33[51367]: segfault at 0 ip 000000000055ae67 sp 00007f8c817f88a0
> error 4 in suricata[400000+1f7000]
>
> Greg
>
> On 09/27/18 18:02:12, Victor Julien wrote:
> > On 27-09-18 15:51, Cloherty, Sean E wrote:
> > > Hello Victor -
> > >
> > > I am not sure if the actual fault messages came across in my previous email. Below is what I've got from syslog - (apologies if the tabs and spaces mess up the faux table). No core dump so I've gone back and reverted the two test servers to the settings that they had when they faulted, enabled. Now I need to puzzle through enabling cored dumps on CentOS 7.
> > >
> > > TIME HOST SURICATA SEGFAULT
> > > 9/25/2018 18:26 production host #1 4.04 kernel: W#14-ens1f1[29348]: segfault at 0 ip 0000000000597207 sp 00007f918b7fbef0 error 4 in suricata[400000+256000]
> > > 9/25/2018 18:26 test-host #1 4.1rc1 kernel: W#03-ens1f1[24471]: segfault at 0 ip 00000000005b7787 sp 00007f6650b27cb0 error 4 in suricata[400000+28c000]
> > > 9/25/2018 18:26 production host #3 4.04 kernel: W#06-ens1f1[24268]: segfault at 0 ip 0000000000597207 sp 00007f3a077fbef0 error 4 in suricata[400000+256000]
> > > 9/25/2018 18:26 test-host #2 4.05 kernel: W#01-ens1f1[4720]: segfault at 0 ip 000000000059b557 sp 00007efc6e69cde0 error 4 in suricata[400000+265000]
> > > 9/25/2018 18:27 test-host #2 4.05 kernel: W#07-ens1f1[4406]: segfault at 0 ip 000000000059b557 sp 00007fc4c2504de0 error 4 in suricata[400000+265000]
> >
> > Hi Sean, I had seen those The link I posted gives some hints on how
> > to extract info from these lines. Could you try that? It might help
> > with pinpointing where in the code the crashes happen.
> >
> > Regards,
> > Victor
> >
> >
> > > -----Original Message-----
> > > From: Oisf-users
> > > <oisf-users-bounces at lists.openinfosecfoundation.org> On Behalf Of
> > > Victor Julien
> > > Sent: Thursday, September 27, 2018 1:23 AM
> > > To: oisf-users at lists.openinfosecfoundation.org
> > > Subject: Re: [Oisf-users] Suffering Simultaneous Suricata
> > > Segfaults
> > >
> > > On 26-09-18 18:55, Cloherty, Sean E wrote:
> > >> I was troubleshooting instances of Suricata being down on
> > >> multiple hosts and I found that 2 production hosts running 4.04
> > >> and 2 test hosts running 4.05 and 4.1rc1 faulted at roughly the same time.
> > >> Strangely, 2 additional production hosts running 4.04 on
> > >> duplicate hardware have not had any issues to date. Below is the
> > >> outline of what I’ve been able to put together this morning.
> > >>
> > >>
> > >
> > > Did any of the instances dump a core file you can inspect?
> > >
> > > Another way to get more info based on the lines you posted is
> > > described
> > > here:
> > > https://stackoverflow.com/questions/2549214/interpreting-segfault-
> > > me ssages could you try to see if you can get more info about
> > > where in the code the crash happens?
> > >
> > >
> > >>
> > >> What is the same across all platforms faulting or not:
> > >>
> > >>
> > >>
> > >> All use tpacket v3 & AF-PACKET
> > >>
> > >> All use workers mode
> > >>
> > >> All are in IDS mode
> > >>
> > >> All ingest traffic from Gigamon taps
> > >>
> > >> All are running CentOS 7.5 64bit
> > >>
> > >> All use Intel(R) 10GbE PCI Express Linux Network Driver 5.3.7
> > >>
> > >> All use Intel Corporation 82599ES 10-Gigabit SFI/SFP+
> > >>
> > >> What is different:
> > >>
> > >>
> > >>
> > >> NO FAULT: #zero-copy-size: 128
> > >>
> > >> FAULT: zero-copy-size: 128
> > >
> > > This option is no longer used by any of the versions you are using.
> > >
> > >
> > >> NO FAULT:
> > >>
> > >> prio:
> > >>
> > >> # low: [ 0 ]
> > >>
> > >> # medium: [ "1-2" ]
> > >>
> > >> # high: [ 3 ]
> > >>
> > >> default: "high"
> > >>
> > >>
> > >>
> > >> FAULT:
> > >>
> > >> prio:
> > >>
> > >> low: [ 0 ]
> > >>
> > >> medium: [ "1-2" ]
> > >>
> > >> high: [ 3 ]
> > >>
> > >> default: "high"
> > > Would be weird if this did anything.
> > >
> > > --
> > > ---------------------------------------------
> > > Victor Julien
> > > http://www.inliniac.net/
> > > PGP: http://www.inliniac.net/victorjulien.asc
> > > ---------------------------------------------
> > >
> > > _______________________________________________
> > > Suricata IDS Users mailing list:
> > > oisf-users at openinfosecfoundation.org
> > > Site: http://suricata-ids.org | Support:
> > > http://suricata-ids.org/support/
> > > List:
> > > https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-user
> > > s
> > >
> > > Conference: https://suricon.net
> > > Trainings: https://suricata-ids.org/training/
> > >
> >
> >
> > --
> > ---------------------------------------------
> > Victor Julien
> > http://www.inliniac.net/
> > PGP: http://www.inliniac.net/victorjulien.asc
> > ---------------------------------------------
> >
> > _______________________________________________
> > Suricata IDS Users mailing list:
> > oisf-users at openinfosecfoundation.org
> > Site: http://suricata-ids.org | Support:
> > http://suricata-ids.org/support/
> > List:
> > https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> >
> > Conference: https://suricon.net
> > Trainings: https://suricata-ids.org/training/
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: buildinfo.txt
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20181019/f1e46d2d/attachment-0001.txt>
More information about the Oisf-users
mailing list