[Oisf-devel] Segmentation fault on the latest Suricata beta

[Anoop Saldanha]

cool.  Thanks.

On Fri, Feb 28, 2014 at 12:31 PM, Duarte Silva
<duarte.silva at serializing.me> wrote:
> Hi Anoop,
>
> no. We know that a specific flow is being processed by two different CPUs
> when the segmentation fault happens. I'm able to reproduce the issue but
> unable to reduce the pcap size and pinpoint the exact flow since it crashes
> unpredictably (sometimes at the beginning, other times after replaying the
> traffic 3 times, etc.). I hadn't have time to investigate it further since
> we've been doing some work intensive infrastructure upgrades.
>
> I will see what i can do next week.
>
> Cheers,
> Duarte
>
> On 28 Feb 2014 03:14, "Anoop Saldanha" <anoopsaldanha at gmail.com> wrote:
>>
>> On Fri, Feb 7, 2014 at 11:50 PM, Duarte Silva
>> <duarte.silva at serializing.me> wrote:
>> > On Friday 07 February 2014 18:40:31 Victor Julien wrote:
>> >> On 02/07/2014 06:37 PM, Duarte Silva wrote:
>> >> > On Friday 07 February 2014 18:23:55 Victor Julien wrote:
>> >> >> On 02/07/2014 05:56 PM, Duarte Silva wrote:
>> >> >>> Hi Victor,
>> >> >>>
>> >> >>> I have come to a conclusion about this. The fact that Suricata is
>> >> >>> compiled
>> >> >>> with stack protection and position independent code, doesn't seem
>> >> >>> to
>> >> >>> have
>> >> >>> anything to do with the issue in hand.
>> >> >>>
>> >> >>> Actually, that specific segmentation fault did not happened again
>> >> >>> (another
>> >> >>> one did, I didn't compare back traces correctly :/).
>> >> >>>
>> >> >>> Anyway, the new segmentation fault still happens on master, at
>> >> >>> least, up
>> >> >>> to
>> >> >>> commit 3967bd5517e7ab57fd82d7ce9b70dac16057b8f7. Issue opened at
>> >> >>> [1].
>> >> >>
>> >> >> This is very interesting. It looks like 2 packet processing threads
>> >> >> have
>> >> >> the same flow. This is not supposed to happen.
>> >> >>
>> >> >> You are using workers, correct?
>> >> >
>> >> > Yes, I'm.
>> >> >
>> >> >> In gdb, can you do:
>> >> >>
>> >> >> thread 4
>> >> >> frame 4
>> >> >> print *p
>> >> >> print *p->flow
>> >> >>
>> >> >> and
>> >> >>
>> >> >> thread 5
>> >> >> frame 5
>> >> >> print *p
>> >> >> print *p->flow
>> >> >
>> >> > I terminated the gdb session, I will reproduce the issue and
>> >> > hopefully the
>> >> > condition repeats it self (I really need to start saving core dumps
>> >> > :D)
>> >>
>> >> Ah, too bad.
>> >>
>> >> In af-packet, are you using the cluster flow setting? What kernel
>> >> version, NIC and driver do you have?
>> >
>> > Reproduced the segmentation fault, the issue of the flow being present
>> > in the
>> > two threads happened again. Updated the issue. As I come to think about
>> > it,
>> > one of the following conditions or even both could be related with this:
>> >
>> > 1- In the same machine, I'm running Suricata and tcpreplay on the same
>> > interface. So Suricata is actually inspecting sent packets, instead of
>> > received.
>> >
>> > 2- We do have multiple VLAN's on the traffic, the latests commits do
>> > have fixes
>> > for VLAN identifier handling.
>> >
>> > Now the information required.
>> >
>> > # uname -a
>> > Linux pidsma02 3.13.1-1.el6.x86_64 #1 SMP Fri Feb 7 12:01:59 UTC 2014
>> > x86_64
>> > x86_64 x86_64 GNU/Linux
>> >
>> > (You probably ain't going to like this part :P)
>> > # ethtool -i eth1
>> > driver: vmxnet3
>> > version: 1.2.0.0-k-NAPI
>> > firmware-version:
>> > bus-info: 0000:13:00.0
>> > supports-statistics: yes
>> > supports-test: no
>> > supports-eeprom-access: no
>> > supports-register-dump: yes
>> > supports-priv-flags: no
>> >
>>
>> Duarte,
>>
>> Were you able to confirm the cause for your segv?
>>
>> --
>> -------------------------------
>> Anoop Saldanha
>> http://www.poona.me
>> -------------------------------



-- 
-------------------------------
Anoop Saldanha
http://www.poona.me
-------------------------------