[Oisf-users] Discrepancies in Snort and Suricata alerts

Tue Oct 30 17:23:23 UTC 2018

Hi Everyone,

So finally after a lot of time spent on the troubleshooting from  Peter,
Michal, Victor and Travis , figured out what was the exact cause of missing
TCP alerts in suricata was.
It was because of the under assignment of memory for the "flow" and
"stream" sections in suricata.yaml, which were set to default values, and
suricata was assigning the memory at the run-time for them.

*Cause:*
It was caught while looking at the stats.log file, which was having pretty
high values for  tcp.ssn_memcap_drop, tcp.segment_memcap_drop,
tcp.reassembly_gap.
That indicated that the TCP sessions were not properly tracked and
processed due to the stream.memcap/stream.reassemble.memcap setting being
too low.

*Solution:*
So I changed these following settings to ~10x in suricata.yaml:

flow.memcap : 2048mb  # changing the default value from 128mb
flow.hash size : 655360  # changing the default value from 65536
stream.memcap: 1000mb  # changing the default value from 64mb
stream.reassembly.memcap: 4000mb  # changing the default value from 256mb

And the TCP alerts started getting triggered nicely!
I also had an increased capture loss because of this, which got resolved by
re-compiling suricata with disabled debugging and profiling options (Yes, I
had those enabled when I built suricata before).

BIG THANKS to the patience and co-operation of Peter, Michal, Victor and
Travis in helping me out to troubleshoot the issue!

Hope this helps to any future suricata newbie who is unaware of those
memcap default settings and how it's so crucial in configuring suricata
correctly for the given environment :)

Thanks everyone for the help and responses!

Fatema.

On Tue, Oct 16, 2018 at 7:59 PM Michał Purzyński <michalpurzynski1 at gmail.com>
wrote:

> The number of suricata threads should be indeed equal to what you have
> pfring configured to. Not sure if pfring requires configuration in two
> places, though.
>
> Some parts of the Septun tuning, like pinning workers to cores, correct
> memory settings and making sure you’re not allocating memory during runtime
> would still apply.
>
> Ethtool, etc - depending on what pfring says.
>
> They basically bypassed most of the Linux network stack and it’s hard to
> say what it is they did there.
>
> The only advantage I can find of pfring is that it should work on your
> kernel version.
>
> On Oct 16, 2018, at 11:53 AM, fatema bannatwala <
> fatema.bannatwala at gmail.com> wrote:
>
> Had a quick question regarding using pf_ring, I think I am doing something
> wrong:
> So in the suricata.yaml it says:
>
> "#Number of receive threads (>1 will enable experimental flow pinned
> runmode)"
>   #threads: 1
>
> And I am using 18 as the threads value here in pf_ring section. So should
> I change it to 1?
>
> Also, I am running kernel Linux 3.10 and for h/w NIC based RSS hashing it
> requires at least  Linux 3.14, so I can't use it unless I upgrade the
> kernel.
> But I think if I can set the pf_ring the correct way it should work.
> Do I have to undo the ethtool settings mentioned in Septune to use pf_ring
> over af_packet? I believe those settings were mentioned for using af_packet
> and shouldn't be needed/set for pf_ring?
>
> Thanks,
> Fatema
>
> On Tue, Oct 16, 2018 at 1:30 PM fatema bannatwala <
> fatema.bannatwala at gmail.com> wrote:
>
>> I tried with pf_ring, but couldn't get the "SearchProtect" (sid:2022813)
>> to trigger still.
>> Next I am going to try the hardware hashing and af_packet in qm mode and
>> see if there's any difference, will keep posted..
>>
>> On Fri, Oct 12, 2018 at 2:45 PM Michał Purzyński <
>> michalpurzynski1 at gmail.com> wrote:
>>
>>> After talking a bit more with Fatema, we learned there might be a
>>> problem with the kernel version on that server.
>>>
>>> Victor's patch shows 30% of packets sent to a wrong thread. Bro and
>>> Snort use pf_ring.
>>>
>>> Suggestions would be to use hardware hashing and af_packet in QM mode.
>>> Note - the XDP part is not needed for this setup to work.
>>>
>>> *https://github.com/pevma/SEPTun-Mark-II/blob/master/SEPTun-Mark-II.rst#setup-symmetric-hashing-on-the-nic
>>> <https://github.com/pevma/SEPTun-Mark-II/blob/master/SEPTun-Mark-II.rst#setup-symmetric-hashing-on-the-nic>*
>>>
>>> just do the symmetric hashing part and then change Suricata's
>>> configuration to use cluster_qm
>>>
>>>   cluster-type: cluster_qm # symmetric hashing  is a must!
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Oct 5, 2018 at 2:20 PM Michał Purzyński <
>>> michalpurzynski1 at gmail.com> wrote:
>>>
>>>> How about we make Suricata write us a pcap in afpacket workers mode?
>>>> I’m pretty sure a rule can do that.
>>>>
>>>> > On Oct 5, 2018, at 8:17 PM, fatema bannatwala <
>>>> fatema.bannatwala at gmail.com> wrote:
>>>> >
>>>> > Changing $HOME_NET to any in sid 2022813 didn't help though, still
>>>> not getting that alert fired.
>>>> > One difference I had in suricata.yaml when running in offline pcap
>>>> reading mode was, I set runmode to "single", while when suricata runs in
>>>> packet sniffing mode it's set to "workers".
>>>> >
>>>> > I tried to set it to "runmode:single" while on interface sniffing
>>>> mode but was hit by ~60% capture loss, which makes sense as single threaded
>>>> suricata can't handle the traffic flowing through the interface.
>>>> >
>>>> > The fact that alerts are fired when in offline single threaded mode
>>>> and same alerts are not fired when online packet sniffing multi-threaded
>>>> mode, makes me think it has to do with multi-threading vs single threaded
>>>> mode and how "workers" are capturing packets.
>>>> >
>>>> > I will keep looking.
>>>> >
>>>> > (The good thing is that Interrupt/IRQ pinning has helped to reduce
>>>> capture loss to 0%)
>>>> >
>>>> > Thanks,
>>>> > Fatema
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>>>> > Site: http://suricata-ids.org | Support:
>>>> http://suricata-ids.org/support/
>>>> > List:
>>>> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>> >
>>>> > Conference: https://suricon.net
>>>> > Trainings: https://suricata-ids.org/training/
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20181030/d3eb43bd/attachment.html>