[Oisf-devel] Segmentation fault on the latest Suricata beta

[Duarte Silva]

Hi Anoop,

no. We know that a specific flow is being processed by two different CPUs
when the segmentation fault happens. I'm able to reproduce the issue but
unable to reduce the pcap size and pinpoint the exact flow since it crashes
unpredictably (sometimes at the beginning, other times after replaying the
traffic 3 times, etc.). I hadn't have time to investigate it further since
we've been doing some work intensive infrastructure upgrades.

I will see what i can do next week.

Cheers,
Duarte
On 28 Feb 2014 03:14, "Anoop Saldanha" <anoopsaldanha at gmail.com> wrote:

> On Fri, Feb 7, 2014 at 11:50 PM, Duarte Silva
> <duarte.silva at serializing.me> wrote:
> > On Friday 07 February 2014 18:40:31 Victor Julien wrote:
> >> On 02/07/2014 06:37 PM, Duarte Silva wrote:
> >> > On Friday 07 February 2014 18:23:55 Victor Julien wrote:
> >> >> On 02/07/2014 05:56 PM, Duarte Silva wrote:
> >> >>> Hi Victor,
> >> >>>
> >> >>> I have come to a conclusion about this. The fact that Suricata is
> >> >>> compiled
> >> >>> with stack protection and position independent code, doesn't seem to
> >> >>> have
> >> >>> anything to do with the issue in hand.
> >> >>>
> >> >>> Actually, that specific segmentation fault did not happened again
> >> >>> (another
> >> >>> one did, I didn't compare back traces correctly :/).
> >> >>>
> >> >>> Anyway, the new segmentation fault still happens on master, at
> least, up
> >> >>> to
> >> >>> commit 3967bd5517e7ab57fd82d7ce9b70dac16057b8f7. Issue opened at
> [1].
> >> >>
> >> >> This is very interesting. It looks like 2 packet processing threads
> have
> >> >> the same flow. This is not supposed to happen.
> >> >>
> >> >> You are using workers, correct?
> >> >
> >> > Yes, I'm.
> >> >
> >> >> In gdb, can you do:
> >> >>
> >> >> thread 4
> >> >> frame 4
> >> >> print *p
> >> >> print *p->flow
> >> >>
> >> >> and
> >> >>
> >> >> thread 5
> >> >> frame 5
> >> >> print *p
> >> >> print *p->flow
> >> >
> >> > I terminated the gdb session, I will reproduce the issue and
> hopefully the
> >> > condition repeats it self (I really need to start saving core dumps
> :D)
> >>
> >> Ah, too bad.
> >>
> >> In af-packet, are you using the cluster flow setting? What kernel
> >> version, NIC and driver do you have?
> >
> > Reproduced the segmentation fault, the issue of the flow being present
> in the
> > two threads happened again. Updated the issue. As I come to think about
> it,
> > one of the following conditions or even both could be related with this:
> >
> > 1- In the same machine, I'm running Suricata and tcpreplay on the same
> > interface. So Suricata is actually inspecting sent packets, instead of
> > received.
> >
> > 2- We do have multiple VLAN's on the traffic, the latests commits do
> have fixes
> > for VLAN identifier handling.
> >
> > Now the information required.
> >
> > # uname -a
> > Linux pidsma02 3.13.1-1.el6.x86_64 #1 SMP Fri Feb 7 12:01:59 UTC 2014
> x86_64
> > x86_64 x86_64 GNU/Linux
> >
> > (You probably ain't going to like this part :P)
> > # ethtool -i eth1
> > driver: vmxnet3
> > version: 1.2.0.0-k-NAPI
> > firmware-version:
> > bus-info: 0000:13:00.0
> > supports-statistics: yes
> > supports-test: no
> > supports-eeprom-access: no
> > supports-register-dump: yes
> > supports-priv-flags: no
> >
>
> Duarte,
>
> Were you able to confirm the cause for your segv?
>
> --
> -------------------------------
> Anoop Saldanha
> http://www.poona.me
> -------------------------------
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-devel/attachments/20140228/ca22ddc7/attachment-0002.html>