[Oisf-users] Suffering Simultaneous Suricata Segfaults

Greg Grasmehr greg.grasmehr at caltech.edu
Fri Oct 5 18:35:59 UTC 2018 

Yes I will check; there are many groups on campus that run their own
mail servers, if it's one of them we won't have info.  Should be able to
get back to you regarding this by Monday.

Greg

On 10/05/18 17:21:42, Cloherty, Sean E wrote:
> Greg & Cooper
> 
> Just a shot in the dark here --- Do you have email logs from that time period?  I was wondering if you might see email that has a to or from each other's domains.  Or that our domains are in the same TO: 
> 
> That might help narrow the scope of where to look.
> 
> Sean
> 
> -----Original Message-----
> From: Greg Grasmehr <greg.grasmehr at caltech.edu> 
> Sent: Thursday, September 27, 2018 1:36 PM
> To: lists at inliniac.net
> Cc: Cloherty, Sean E <scloherty at mitre.org>; oisf-users at lists.openinfosecfoundation.org
> Subject: Re: [Oisf-users] Suffering Simultaneous Suricata Segfaults
> 
> Hello,
> 
> Having the same issue, pointer to the problem code is below
> 
> 
>  addr2line -e /opt/suricata/bin/suricata 000000000055ae67
> /root/installers/suricata-4.0.5/src/util-decode-mime.c:2330
> 
> /var/log/messages:Sep 25 15:28:19 is-pig3 kernel: W#45[31078]: segfault at 0 ip 000000000055ae67 sp 00007f9af7ffd8a0 error 4 in suricata[400000+1f7000] /var/log/messages:Sep 26 03:17:00 is-pig3 kernel: W#53[11268]: segfault at 0 ip 000000000055ae67 sp 00007fbbf97f88a0 error 4 in suricata[400000+1f7000] /var/log/messages-20180909:Sep  7 03:29:12 is-pig3 kernel: W#33[51367]: segfault at 0 ip 000000000055ae67 sp 00007f8c817f88a0 error 4 in suricata[400000+1f7000]
> 
> Greg
> 
> On 09/27/18 18:02:12, Victor Julien wrote:
> > On 27-09-18 15:51, Cloherty, Sean E wrote:
> > > Hello Victor -
> > > 
> > > I am not sure if the actual fault messages came across in my previous email. Below is what I've got from syslog - (apologies if the tabs and spaces mess up the faux table).  No core dump so I've gone back and reverted the two test servers to the settings that they had when they faulted, enabled.  Now I need to puzzle through enabling cored dumps on CentOS 7.
> > > 
> > > TIME			HOST			SURICATA	SEGFAULT
> > > 9/25/2018 18:26	production host #1	4.04	 kernel: W#14-ens1f1[29348]: segfault at 0 ip 0000000000597207 sp 00007f918b7fbef0 error 4 in suricata[400000+256000]
> > > 9/25/2018 18:26	test-host #1		4.1rc1	 kernel: W#03-ens1f1[24471]: segfault at 0 ip 00000000005b7787 sp 00007f6650b27cb0 error 4 in suricata[400000+28c000]
> > > 9/25/2018 18:26	production host #3	4.04	 kernel: W#06-ens1f1[24268]: segfault at 0 ip 0000000000597207 sp 00007f3a077fbef0 error 4 in suricata[400000+256000]
> > > 9/25/2018 18:26	test-host #2		4.05	 kernel: W#01-ens1f1[4720]: segfault at 0 ip 000000000059b557 sp 00007efc6e69cde0 error 4 in suricata[400000+265000]
> > > 9/25/2018 18:27	test-host #2		4.05	 kernel: W#07-ens1f1[4406]: segfault at 0 ip 000000000059b557 sp 00007fc4c2504de0 error 4 in suricata[400000+265000]
> > 
> > Hi Sean, I had seen those The link I posted gives some hints on how to 
> > extract info from these lines. Could you try that? It might help with 
> > pinpointing where in the code the crashes happen.
> > 
> > Regards,
> > Victor
> > 
> > 
> > > -----Original Message-----
> > > From: Oisf-users 
> > > <oisf-users-bounces at lists.openinfosecfoundation.org> On Behalf Of 
> > > Victor Julien
> > > Sent: Thursday, September 27, 2018 1:23 AM
> > > To: oisf-users at lists.openinfosecfoundation.org
> > > Subject: Re: [Oisf-users] Suffering Simultaneous Suricata Segfaults
> > > 
> > > On 26-09-18 18:55, Cloherty, Sean E wrote:
> > >> I was troubleshooting instances of Suricata being down on multiple 
> > >> hosts and I found that 2 production hosts running 4.04 and 2 test 
> > >> hosts running 4.05 and 4.1rc1 faulted at roughly the same time.
> > >> Strangely,  2 additional production hosts running 4.04 on duplicate 
> > >> hardware have not had any issues to date.  Below is the outline of 
> > >> what I’ve been able to put together this morning.
> > >>
> > >>  
> > > 
> > > Did any of the instances dump a core file you can inspect?
> > > 
> > > Another way to get more info based on the lines you posted is 
> > > described
> > > here:
> > > https://stackoverflow.com/questions/2549214/interpreting-segfault-me
> > > ssages could you try to see if you can get more info about where in 
> > > the code the crash happens?
> > > 
> > > 
> > >>
> > >> What is the same across all platforms faulting or not:
> > >>
> > >>  
> > >>
> > >> All use tpacket v3 & AF-PACKET
> > >>
> > >> All use workers mode
> > >>
> > >> All are in IDS mode
> > >>
> > >> All ingest traffic from Gigamon taps
> > >>
> > >> All are running CentOS 7.5 64bit
> > >>
> > >> All use Intel(R) 10GbE PCI Express Linux Network Driver 5.3.7
> > >>
> > >> All use Intel Corporation 82599ES 10-Gigabit SFI/SFP+
> > >>
> > >> What is different:
> > >>
> > >>  
> > >>
> > >> NO FAULT:          #zero-copy-size: 128
> > >>
> > >> FAULT:                  zero-copy-size: 128
> > > 
> > > This option is no longer used by any of the versions you are using.
> > > 
> > > 
> > >> NO FAULT:
> > >>
> > >>         prio:
> > >>
> > >> #          low: [ 0 ]
> > >>
> > >> #          medium: [ "1-2" ]
> > >>
> > >> #          high: [ 3 ]
> > >>
> > >>           default: "high"
> > >>
> > >>  
> > >>
> > >> FAULT:
> > >>
> > >>         prio:
> > >>
> > >>           low: [ 0 ]
> > >>
> > >>           medium: [ "1-2" ]
> > >>
> > >>           high: [ 3 ]
> > >>
> > >>           default: "high"
> > > Would be weird if this did anything.
> > > 
> > > --
> > > ---------------------------------------------
> > > Victor Julien
> > > http://www.inliniac.net/
> > > PGP: http://www.inliniac.net/victorjulien.asc
> > > ---------------------------------------------
> > > 
> > > _______________________________________________
> > > Suricata IDS Users mailing list: 
> > > oisf-users at openinfosecfoundation.org
> > > Site: http://suricata-ids.org | Support: 
> > > http://suricata-ids.org/support/
> > > List: 
> > > https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> > > 
> > > Conference: https://suricon.net
> > > Trainings: https://suricata-ids.org/training/
> > > 
> > 
> > 
> > --
> > ---------------------------------------------
> > Victor Julien
> > http://www.inliniac.net/
> > PGP: http://www.inliniac.net/victorjulien.asc
> > ---------------------------------------------
> > 
> > _______________________________________________
> > Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> > Site: http://suricata-ids.org | Support: 
> > http://suricata-ids.org/support/
> > List: 
> > https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> > 
> > Conference: https://suricon.net
> > Trainings: https://suricata-ids.org/training/

More information about the Oisf-users mailing list