[Oisf-users] Drops: From none to gigantic in the blink of an eye

Thu Mar 24 17:33:37 UTC 2016

When I start up - the last line is as below.

24/3/2016 -- 13:32:30 - <Notice> - This is Suricata version 3.0 RELEASE
24/3/2016 -- 13:32:30 - <Info> - CPUs/cores online: 32
24/3/2016 -- 13:32:30 - <Info> - 'default' server has 'request-body-minimal-inspect-size' set to 33882 and 'request-body-inspect-window' set to 4053 after randomization.
24/3/2016 -- 13:32:30 - <Info> - 'default' server has 'response-body-minimal-inspect-size' set to 42119 and 'response-body-inspect-window' set to 16872 after randomization.
24/3/2016 -- 13:32:30 - <Info> - DNS request flood protection level: 500
24/3/2016 -- 13:32:30 - <Info> - DNS per flow memcap (state-memcap): 524288
24/3/2016 -- 13:32:30 - <Info> - DNS global memcap: 16777216
24/3/2016 -- 13:32:30 - <Info> - Protocol detection and parser disabled for modbus protocol.
24/3/2016 -- 13:32:30 - <Info> - Found an MTU of 1500 for 'ens1f1'
24/3/2016 -- 13:32:30 - <Info> - allocated 3670016 bytes of memory for the defrag hash... 65536 buckets of size 56
24/3/2016 -- 13:32:30 - <Info> - preallocated 65535 defrag trackers of size 168
24/3/2016 -- 13:32:30 - <Info> - defrag memory usage: 14679896 bytes, maximum: 2147483648
24/3/2016 -- 13:32:30 - <Info> - AutoFP mode using default "Active Packets" flow load balancer

-----Original Message-----
From: Cooper F. Nelson [mailto:cnelson at ucsd.edu] 
Sent: Thursday, March 24, 2016 13:30 PM
To: Cloherty, Sean E <scloherty at mitre.org>; Victor Julien <lists at inliniac.net>; oisf-users at lists.openinfosecfoundation.org
Subject: Re: [Oisf-users] Drops: From none to gigantic in the blink of an eye

It would be in the suricata.log file, the log entries look like this:

> [12105] 24/3/2016 -- 17:07:04 - (flow-manager.c:667) <Info> 
> (FlowManager) -- Flow emergency mode over, back to normal... unsetting 
> FLOW_EMERGENCY bit (ts.tv_sec: 1458839205, ts.tv_usec:62) flow_spare_q 
> status(): 123% flows at the queue

Are you following this article, configuration-wise?

https://home.regit.org/2012/07/suricata-to-10gbps-and-beyond/

What makes you think you are running in "autofp" mode?  If you want, you can send your suricata.log file to me and I'll see if I can figure out what the problem is.

-Coop

On 3/24/2016 10:22 AM, Cloherty, Sean E wrote:
> Grepping the stats.log file for 'emergency' came up blank.
> 
> -----Original Message-----
> From: Oisf-users 
> [mailto:oisf-users-bounces at lists.openinfosecfoundation.org] On Behalf 
> Of Victor Julien
> Sent: Thursday, March 24, 2016 11:52 AM
> To: oisf-users at lists.openinfosecfoundation.org
> Subject: Re: [Oisf-users] Drops: From none to gigantic in the blink of 
> an eye
> 
> On 23-03-16 20:33, Cooper F. Nelson wrote:
>> The thing about SYN floods is that don't generate that much traffic.
>> They are designed to DOS hosts, not networks.  And it only takes a 
>> few mbits of SYN packets from spoofed source addresses/ports to DOS a 
>> suricata sensor, due to flow hashing/tracking.
>>
>> If it's a memory leak you should be able to see that by running top 
>> and monitoring memory usage, which looks ok in your case.
>>
>> If it's an issue with the flow memory-cap I would think you be seeing 
>> lots of messages in the suricata log about flow-emergency mode.
> 
> Good point. The counters didn't show this, while they should have I think.
> 
> Sean, did you get flow 'emergency' mode messages in your log or to stdout?
> 
> Cheers,
> Victor
> 
>>
>> -Coop
>>
>> On 3/23/2016 12:26 PM, Cloherty, Sean E wrote:
>>> Thank you Cooper,
>>>
>>> I will give this a try.  Though I would assume that the SYN flood 
>>> would still show up as increased network traffic on the interface.
>>> This is a test machine, but I do have it integrated into our 
>>> production Zabbix monitor so I can keep an eye on it.
>>>
>>> Does anyone think it might be a symptom of a memory leak?  Would it 
>>> be worthwhile testing Victor's suggestion before trying the new RC 
>>> that was released?
>>
>>
>>
>>
>> _______________________________________________
>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>> Site: http://suricata-ids.org | Support: 
>> http://suricata-ids.org/support/
>> List: 
>> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>> Suricata User Conference November 9-11 in Washington, DC: 
>> http://oisfevents.net
>>
> 
> 
> --
> ---------------------------------------------
> Victor Julien
> http://www.inliniac.net/
> PGP: http://www.inliniac.net/victorjulien.asc
> ---------------------------------------------
> 
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: 
> http://suricata-ids.org/support/
> List: 
> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> Suricata User Conference November 9-11 in Washington, DC: 
> http://oisfevents.net _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: 
> http://suricata-ids.org/support/
> List: 
> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> Suricata User Conference November 9-11 in Washington, DC: 
> http://oisfevents.net
> 

--
Cooper Nelson
Network Security Analyst
UCSD ITS Security Team
cnelson at ucsd.edu x41042