[Oisf-users] Negative in the HOME_NET variable

[Anoop Saldanha]

On Fri, Jan 10, 2014 at 2:38 PM, Peter Manev <petermanev at gmail.com> wrote:
> On Fri, Jan 10, 2014 at 3:44 AM, Anoop Saldanha <anoopsaldanha at gmail.com> wrote:
>> On Fri, Jan 10, 2014 at 2:45 AM, Michael McAndrews <michaelm14 at gmail.com> wrote:
>>> Hello all,
>>>
>>> I'm new to the list so I apologize if this is already out there but
>>> nothing popped on Google....
>>>
>>> I'm running Suricata and encountered an issue when modifying the
>>> HOME_NET variable. I want to exclude a particular IP address from a CIDR
>>> defined network. For example, in HOME_NET I have defined 192.168.0.0/16.
>>> If I want to EXCLUDE the 192.168.14.0 subnet, the documentation I found
>>> said it would noted as follows:
>>>
>>> HOME_NET:       [192.168.0.0/16,!192.168.14.0/24]
>>>
>>> If I DO NOT have the negative in my Suricata.yaml file, it loads in seconds:
>>>
>>> --------------------------------------------------------------------
>>> Jan  9 20:33:04 IDS_GW suricata: 9/1/2014 -- 20:33:04 - <Info> - 8641
>>> signatures processed. 704 are IP-only rules, 3688 are inspecting packet
>>> payload, 4896 inspect application layer, 0 are decoder event only
>>> Jan  9 20:33:04 IDS_GW suricata: 9/1/2014 -- 20:33:04 - <Info> -
>>> building signature grouping structure, stage 1: adding signatures to
>>> signature source addresses... complete
>>> Jan  9 20:33:04 IDS_GW suricata: 9/1/2014 -- 20:33:04 - <Info> -
>>> building signature grouping structure, stage 2: building source address
>>> list... complete
>>> Jan  9 20:33:11 IDS_GW suricata: 9/1/2014 -- 20:33:11 - <Info> -
>>> building signature grouping structure, stage 3: building destination
>>> address lists... complete
>>> ---------------------------------------------------------------------
>>>
>>> The problem is, when I add the negative to the variable, it takes over
>>> 40 minutes for Suricata to load and start inspecting traffic.
>>>
>>> From the logs, I can tell it hangs after stage 1. Notice the times:
>>>
>>> ---------------------------------------------------------------------
>>> 9/1/2014 -- 20:23:29 - <Info> - building signature grouping structure,
>>> stage 1: adding signatures to signature source addresses... complete
>>> 9/1/2014 -- 20:53:39 - <Info> - building signature grouping structure,
>>> stage 2: building source address list... complete
>>> 9/1/2014 -- 21:10:09 - <Info> - building signature grouping structure,
>>> stage 3: building destination address lists... complete
>>> ---------------------------------------------------------------------
>>>
>>> Has anyone else seen this behavior or have a way to remove a subset of
>>> addresses from a network?
>>>
>>
>> I tried the above by loading a rule that uses the HOME_NET you
>> specificed, and it loaded instantly.
>>
>> Which version of suricata are you using?
>> How many rules are you loading?
>>
>
>
> I just run some tests.
> This is a valid bug with 2.0beta2 :
>
> TEST 1)
> when using in suricata.yaml -
> HOME_NET: "[192.168.0.0/16,10.0.0.0/8,172.16.0.0/12,!192.168.1.1/24]"
>
> 10/1/2014 -- 09:03:57 - <Error> - [ERRCODE:
> SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp
> $HOME_NET any -> $EXTERNAL_NET any (msg:"ET WORM W32/Njw0rm CnC
> Beacon"; flow:established,to_server; content:"lv0njxq80"; depth:9;
> content:"njxq80"; distance:0;
> reference:url,www.fireeye.com/blog/technical/malware-research/2013/08/njw0rm-brother-from-the-same-mother.html;
> reference:md5,4c60493b14c666c56db163203e819272;
> reference:md5,b0e1d20accd9a2ed29cdacb803e4a89d;
> classtype:trojan-activity; sid:2017404; rev:3;)" from file
> /etc/suricata/rules/emerging-worm.rules at line 187
> 10/1/2014 -- 09:03:57 - <Warning> - [ERRCODE: SC_ERR_NO_RULES(42)] -
> No rules loaded from /etc/suricata/rules/emerging-worm.rules
> 10/1/2014 -- 09:03:57 - <Info> - 48 rule files processed. 1602 rules
> successfully loaded, 12443 rules failed
> 10/1/2014 -- 09:03:57 - <Info> - 1602 signatures processed. 0 are
> IP-only rules, 249 are inspecting packet payload, 93 inspect
> application layer, 72 are decoder event only
> 10/1/2014 -- 09:03:57 - <Info> - building signature grouping
> structure, stage 1: preprocessing rules... complete
> Killed
>
> killed ... happens after 40 min.
>
> TEST 2)
> when using
> HOME_NET: "[192.168.0.0/16,10.0.0.0/8,172.16.0.0/12]"
>
> 10/1/2014 -- 09:02:24 - <Info> - 48 rule files processed. 14045 rules
> successfully loaded, 0 rules failed
> 10/1/2014 -- 09:02:24 - <Info> - 14053 signatures processed. 1136 are
> IP-only rules, 4310 are inspecting packet payload, 10513 inspect
> application layer, 72 are decoder event only
> 10/1/2014 -- 09:02:24 - <Info> - building signature grouping
> structure, stage 1: preprocessing rules... complete
> 10/1/2014 -- 09:02:24 - <Info> - building signature grouping
> structure, stage 2: building source address list... complete
> 10/1/2014 -- 09:02:27 - <Info> - building signature grouping
> structure, stage 3: building destination address lists... complete
>
> everything is fine, Suricata loads in 5 sec with 14K rules
>
> I think there are two bugs here:
> 1) is the time it takes to load the rules
> 2) the huge amount of rules failing to load (using the same yaml for both cases)
>
> @Michael - would you please post 2 bug reports. Please be detailed.
>

@Peter,

So the delay surfaces if we are loading a huge no of rules with the
specified HOME_NET?

-- 
-------------------------------
Anoop Saldanha
http://www.poona.me
-------------------------------