[Oisf-users] Missing Alerts? May be the network

[Michał Purzyński]

Thanks for sharing. It is indeed important to monitor packet drops during
all stages of packet processing.
We actually wrote about it in the original SEPTun Mark I - see the section
about packet drops (point 1, SPAN ports)

https://github.com/pevma/SEPTun/blob/master/SEPTun.rst#packet-drops

Fortunately, most switches and Arista lets you query everything with SNMP
and that is something your monitoring should look at as well. You have a
tricky setup, so you should monitor both.

If you have Zeek running with the same upstream infrastructure and you can
enable the stats.log + capture_loss.log and if there are no drops seen in
stats.log but there's something in the capture_loss.log - it is time to
check the upstream. I used Zeek to troubleshoot the entire NSM stack a
couple of times ;)

Now (and developers can verify me) I think that if in Suricata's stats log
- kernel drops are 0
- memcap drops are 0
- tcp.reassembly_gap is bigger than 0

You can deduce from those you have an upstream problem.


On Sun, Mar 17, 2019 at 8:14 AM Greg Grasmehr <greg.grasmehr at caltech.edu>
wrote:

> Hello All,
>
> Some of you may remember in early-mid 2018 I was essentially pulling my
> hair out trying to figure out why Suricata was apparently missing alert
> traffic on our 10G wire.  Network claimed everything was hunky dory on
> their end, and I spent countless hours testing different configs and
> rule sets trying to determine what was going on.
>
> Fortunately I attended Zeekcon last October and one of the presentations
> got me looking at our Zeek data and then to thinking.
>
> Long story short - one of the SPANS feeding our Arista switch turned out
> to be saturated and dropping packets on the edge switch feeding the
> Arista.  Once that was rectified Suricata was finally receiving all
> network data and with Hyperscan enabled easily handling the traffic,
> even during micro bursts exceeding 10G, and this is with more than 57000
> rules enabled.  As far as I can tell it doesn't miss a thing now. w00t!
>
> --
> Sincerely,
>
> Greg Grasmehr
> Lead Information Security Analyst
> California Institute of Technology (Caltech)
> GPGMe: 38E2 F9BD A95E 9824 20AB  331A 9E29 D1A1 AAEE 5F42
> pgp.mit.edu:11371/pks/lookup?op=vindex&search=0x9E29D1A1AAEE5F42
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>
> Conference: https://suricon.net
> Trainings: https://suricata-ids.org/training/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20190318/154d8f92/attachment.html>