[Oisf-users] Drops: From none to gigantic in the blink of an eye

[Victor Julien]

On 23-03-16 17:12, Cloherty, Sean E wrote:
> Our Suricata installation went from normal to completely haywire
> overnight Tuesday.  It was cruising along with very low packet loss
> (0.002%) when suddenly between 2:24 and 2:29 AM it began to grow
> extremely rapidly. 
> 
>  
> 
> So far ‘ve checked and
> 
>  
> 
> -          NIC stats for errors or drops are very few(at bottom of email)
> 
> -          There were no changes to server Tuesday AM to account for this
> 
> -          Network traffic just before and after exhibited no major
> change of volume. 
> 
> -          No errors are visible in the messages file, or Suricata logs
> that appear out of the ordinary.
> 
> -          Since that time RAM usage and CPU utilization is much higher
> (no surprise)
> 
>  
> 
> The most pertinent data is below or attached. Any input at all would be
> helpful to say the least . . .
> 

>From the spreadsheet it looks like the flow engine is running out of
memory. Memory use is stable: 536870752 (almost exactly 512MB), but
flow.memcap counter increases (meaning flow engine could not get memory
because of memcap limit) and tcp packets w/o a flow (tcp.no_flow
counter) suddenly increases.

Is your flow memcap 512MB? I would suggest increasing it, or
alternatively lower flow timeout settings. Lower flow timeout settings
will lead to quicker eviction of flows, so that the memory is freed up.
If you have the memory, I would increase the flow memcap.

As to why it drops packets, there can be multiple explanations. It will
enter some slow paths like trying to walk the flow hash in search of a
flow to kill off and reuse.

Also the IP only rules inspection happens per flow, not per packet. But
if there are no flows, it will happen per packet, increasing the cost.

There may be other slow paths, not sure.

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------