[Oisf-users] packet loss troubleshooting

[Cooper F. Nelson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Forgot to mention, a single client participating in DOS attack using
forged packets can easily crush a single suricata sensor.  Due entirely
to the flow tuple hashing you mention.

I use these sigs to detect SYN floods as they happen, however I've been
told they have a high impact on CPU utilization on some systems by the
ET folk:

> alert tcp $EXTERNAL_NET any -> $HOME_NET any (msg:"LOCAL DOS Unusually fast SYN packets inbound, Potential DOS"; flags: S,12; threshold: type both, track by_dst, count 5000, seconds 5; classtype:misc-activity; sid:5;)
> alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"LOCAL DOS Unusually fast SYN packets outbound, Potential DOS"; flags: S,12; threshold: type both, track by_dst, count 5000, seconds 5; classtype:misc-activity; sid:6;)

- -Coop

On 12/9/2015 1:13 PM, Brandon Lattin wrote:
> Keep in mind that large flows can induce bursty packetloss.
> 
> For instance, a perfSonar network monitoring device will test bandwidth
> by shoving many gigabytes of max MTU null padded packets through the
> pipe to a remote perfSonar box. This will result in the whole stream
> being buffered and fed to a single core due to tuple hashing. Chances
> are good that your buffer won't flush fast enough and you'll start
> dropping packets. 
> 
> Long story short. Know your traffic. See what netflow has to say.
> 
> On Wed, Dec 9, 2015 at 1:51 PM, Cooper F. Nelson <cnelson at ucsd.edu
> <mailto:cnelson at ucsd.edu>> wrote:
> 
> Let it run for a bit.  There is race condition somewhere that causes
> suricata to drop packets when its starting up and large buffers are
> enabled.  Or, at least there is on my config.
> 
> Aside from that, try running a "top-talkers" report to see if is any
> traffic you can filter out.  Just dropping our local Netflix/Youtube
> caches doubled our capacity.
> 
> -Coop
> 
> On 12/9/2015 11:32 AM, Yasha Zislin wrote:
>> I use PF_RING.
> 
>> Changing these net.core buffers actually made it worse. Packet loss is
>> instant with 30%.
>> These are what my defaults are:
>> net.core.wmem_default = 124928
>> net.core.rmem_default = 124928
>> net.core.netdev_max_backlog = 1000
> 
>> I have 10 gig NIC as well. Not that busy pipe. About 1 million packets a
>> minute.
> 
> 
>     _______________________________________________
>     Suricata IDS Users mailing list:
>     oisf-users at openinfosecfoundation.org
>     <mailto:oisf-users at openinfosecfoundation.org>
>     Site: http://suricata-ids.org | Support:
>     http://suricata-ids.org/support/
>     List:
>     https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>     Suricata User Conference November 4 & 5 in Barcelona:
>     http://oisfevents.net
> 
> 
> 
> 
> -- 
> Brandon Lattin
> Security Analyst
> University of Minnesota - University Information Security
> Office: 612-626-6672

- -- 
Cooper Nelson
Network Security Analyst
UCSD ACT Security Team
cnelson at ucsd.edu x41042
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)

iQEcBAEBAgAGBQJWaJ47AAoJEKIFRYQsa8FWfycH/Rvej+BNT0SYeVZV01QliRXW
wrBIum9H2C82W0ueMOjcVSElSUahqv+YEDbCnVqW1L4e8vtlj7QJY61v5HDN5AhO
ZqCOPyGwvErmaDLbb+dj+OvYPE+mEOHxFYKTkgfaczogaAGTKvdDqamrYgu7M3c+
QDN+5lVFu0/JfROs0T4wWnrHYyUEJgJoF1J3Te1NrS0jt+miVHNYprFG03T/9NkT
uh10CG3CGO5bHfnQkCRu+2VgiOLAQvVxtGZMtA7QH+yCIbVVK5k/hP5RbkXiL20F
l9/UzO/OxAyr22ECLENryelH+lXTNCio/LozOHUxf2uwax5+v0hPh4B09r0EEnQ=
=iKph
-----END PGP SIGNATURE-----