[Oisf-users] Drops: From none to gigantic in the blink of an eye

[Cloherty, Sean E]

Not wanting to be too extreme, I cautiously raised the memcap to 2048mb.  Things seem back to normal but for the fact that the number of dropped packets is much higher than before.  Not at all up to the level of when the problem happened, but usually it runs for days before getting to the > 2000 drops.  Today it is at 282,140.

One odd behavior I am noticing is that despite setting the afpacket mode to workers, both in the yaml file and on the command line, the start messages always notes autofp mode - not sure what could cause that.  I am running in IDS mode if that makes a difference.

As this is a test server where I am piloting Suricata for a production rollout in April, do you think that using the 3.0.1RC1 code would be helpful ? I don't mind testing as long as it is fairly stable.  Don't want people here to get cold feet about moving our IDS to Suricata.

-----Original Message-----
From: Victor Julien [mailto:lists at inliniac.net] 
Sent: Thursday, March 24, 2016 11:51 AM
To: Cloherty, Sean E <scloherty at mitre.org>; oisf-users at lists.openinfosecfoundation.org
Subject: Re: [Oisf-users] Drops: From none to gigantic in the blink of an eye

On 23-03-16 20:22, Cloherty, Sean E wrote:
> Thank you so much for you quick response.
> 
> I wouldn't not want to decrease the flow timeout for just the reasons you stated.  The memcap is set exactly to 512 MB.  I will increase that and restart.  
> 
> Is there a limit to how much I should increase that number?  I could easily double that to be sure not to run into that again.

Technically you can go very high (it's a 64bit limit), but that won't be very useful. Don't think there will be side effects from just doubling it if you have the memory.

With other settings there may be side effects. If you increase the hash size for example, there will be an added cost for the hash walk we do in garbage collection. But here you can probably just increase the memcap.

Cheers,
Victor


> 
> Sean Cloherty
> 
> -----Original Message-----
> From: Oisf-users 
> [mailto:oisf-users-bounces at lists.openinfosecfoundation.org] On Behalf 
> Of Victor Julien
> Sent: Wednesday, March 23, 2016 12:50 PM
> To: oisf-users at lists.openinfosecfoundation.org
> Subject: Re: [Oisf-users] Drops: From none to gigantic in the blink of 
> an eye
> 
> On 23-03-16 17:12, Cloherty, Sean E wrote:
>> Our Suricata installation went from normal to completely haywire 
>> overnight Tuesday.  It was cruising along with very low packet loss
>> (0.002%) when suddenly between 2:24 and 2:29 AM it began to grow 
>> extremely rapidly.
>>
>>  
>>
>> So far ‘ve checked and
>>
>>  
>>
>> -          NIC stats for errors or drops are very few(at bottom of email)
>>
>> -          There were no changes to server Tuesday AM to account for this
>>
>> -          Network traffic just before and after exhibited no major
>> change of volume. 
>>
>> -          No errors are visible in the messages file, or Suricata logs
>> that appear out of the ordinary.
>>
>> -          Since that time RAM usage and CPU utilization is much higher
>> (no surprise)
>>
>>  
>>
>> The most pertinent data is below or attached. Any input at all would 
>> be helpful to say the least . . .
>>
> 
> From the spreadsheet it looks like the flow engine is running out of 
> memory. Memory use is stable: 536870752 (almost exactly 512MB), but 
> flow.memcap counter increases (meaning flow engine could not get 
> memory because of memcap limit) and tcp packets w/o a flow 
> (tcp.no_flow
> counter) suddenly increases.
> 
> Is your flow memcap 512MB? I would suggest increasing it, or alternatively lower flow timeout settings. Lower flow timeout settings will lead to quicker eviction of flows, so that the memory is freed up.
> If you have the memory, I would increase the flow memcap.
> 
> As to why it drops packets, there can be multiple explanations. It will enter some slow paths like trying to walk the flow hash in search of a flow to kill off and reuse.
> 
> Also the IP only rules inspection happens per flow, not per packet. But if there are no flows, it will happen per packet, increasing the cost.
> 
> There may be other slow paths, not sure.
> 
> --
> ---------------------------------------------
> Victor Julien
> http://www.inliniac.net/
> PGP: http://www.inliniac.net/victorjulien.asc
> ---------------------------------------------
> 
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: 
> http://suricata-ids.org/support/
> List: 
> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> Suricata User Conference November 9-11 in Washington, DC: 
> http://oisfevents.net
> 


--
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------