[Oisf-users] suricata-update appending env stuff to the test command?

Tue May 28 20:20:38 UTC 2019

> On 28/05/2019, at 6:47 PM, Peter Manev <petermanev at gmail.com> wrote:
> 
> On Tue, May 28, 2019 at 2:05 AM Russell Fulton <r.fulton at auckland.ac.nz> wrote:
>> 
>> Since I updated my suricata.yaml file for 4.1.4 update has been failing the rule test so rules have not been getting updated.
>> 
>> 28/5/2019 -- 11:37:17 - <Info> -- Testing with suricata -T.
>> 28/5/2019 -- 11:37:17 - <Debug> -- Running /usr/local/bin/suricata -T -l /tmp -c /usr/local/etc/suricata/suricata.yaml -S /usr/local/var/lib/suricata/rules/suricata.rules; env={'SC_LOG_FORMAT': '%t - <%d> -- ', 'SC_LOG_LEVEL': 'Warning', 'ASAN_OPTIONS': 'detect_leaks=0'}
>> 28/5/2019 -- 11:37:17 - <Error> -- Suricata test failed, aborting.
>> 28/5/2019 -- 11:37:17 - <Error> -- Restoring previous rules.
>> sensors at secmonprd11:~$
>> 
> 
> Hi Russel !
> What command do you run to reproduce that ? (can you provide the full
> output if more than that is available)
> How did you install/upgrade?

All the gory details ;)

Host OS is ubuntu 14.04 
suricata 4.1,4 built from source with hyperscan 5.0.0 and packaged with fpm.
suricata-update 1.0.5 installed via pip

sensors at secmonprd11:~$  suricata-update --sid-msg-map /home/sensors/Rules/sensor/sid-msg.map -v
28/5/2019 -- 11:36:35 - <Debug> -- This is suricata-update version 1.0.5 (rev: None); Python: 2.7.12 (default, Dec  4 2017, 14:50:18) - [GCC 5.4.0 20160609]
28/5/2019 -- 11:36:35 - <Info> -- Loading /usr/local/etc/suricata/update.yaml
28/5/2019 -- 11:36:35 - <Debug> -- Setting configuration value force -> False
28/5/2019 -- 11:36:35 - <Debug> -- Setting configuration value verbose -> True
28/5/2019 -- 11:36:35 - <Debug> -- Setting configuration value enable -> False
28/5/2019 -- 11:36:35 - <Debug> -- Setting configuration value sid-msg-map -> /home/sensors/Rules/sensor/sid-msg.map
28/5/2019 -- 11:36:35 - <Debug> -- Setting configuration value no-merge -> False
28/5/2019 -- 11:36:35 - <Debug> -- Setting configuration value version -> False
28/5/2019 -- 11:36:35 - <Debug> -- Setting configuration value dump-sample-configs -> False
28/5/2019 -- 11:36:35 - <Debug> -- Setting configuration value no-test -> False
28/5/2019 -- 11:36:35 - <Debug> -- Setting configuration value subcommand -> update
28/5/2019 -- 11:36:35 - <Debug> -- Setting configuration value modify -> False
28/5/2019 -- 11:36:35 - <Debug> -- Setting configuration value no-reload -> False
28/5/2019 -- 11:36:35 - <Debug> -- Setting configuration value no-ignore -> False
28/5/2019 -- 11:36:35 - <Debug> -- Setting configuration value disable -> False
28/5/2019 -- 11:36:35 - <Debug> -- Setting configuration value etopen -> False
28/5/2019 -- 11:36:35 - <Debug> -- Setting configuration value now -> False
28/5/2019 -- 11:36:35 - <Debug> -- Setting configuration value url -> []
28/5/2019 -- 11:36:35 - <Debug> -- Setting configuration value drop -> False
28/5/2019 -- 11:36:35 - <Debug> -- Found suricata at /usr/local/bin/suricata
28/5/2019 -- 11:36:35 - <Info> -- Using data-directory /usr/local/var/lib/suricata.
28/5/2019 -- 11:36:35 - <Info> -- Using Suricata configuration /usr/local/etc/suricata/suricata.yaml
28/5/2019 -- 11:36:35 - <Info> -- Using /usr/local/etc/suricata/rules for Suricata provided rules.
28/5/2019 -- 11:36:35 - <Info> -- Found Suricata version 4.1.4 at /usr/local/bin/suricata.
28/5/2019 -- 11:36:35 - <Info> -- Loading /home/sensors/conf/suricata/update/disable.conf.
28/5/2019 -- 11:36:35 - <Debug> -- Parsing group matcher: group:policy
28/5/2019 -- 11:36:35 - <Debug> -- Parsing group matcher: group:info
28/5/2019 -- 11:36:35 - <Debug> -- Parsing group matcher: group:chat
28/5/2019 -- 11:36:35 - <Debug> -- Parsing group matcher: group:malware
28/5/2019 -- 11:36:35 - <Debug> -- Parsing group matcher: group:user_agents
28/5/2019 -- 11:36:35 - <Debug> -- Parsing group matcher: group:exploit
28/5/2019 -- 11:36:35 - <Debug> -- Parsing regex matcher: re:TROJAN Suspicious user
28/5/2019 -- 11:36:35 - <Debug> -- Parsing regex matcher: re:TROJAN Possible Linux.Mirai Login Attempt
28/5/2019 -- 11:36:35 - <Debug> -- Parsing regex matcher: re:ET DNS Query to a *\.
28/5/2019 -- 11:36:35 - <Debug> -- Parsing regex matcher: re:ET DNS Query for \.
28/5/2019 -- 11:36:35 - <Info> -- Loading /home/sensors/conf/suricata/update/enable.conf.
28/5/2019 -- 11:36:35 - <Debug> -- Parsing regex matcher: re:ET\.http\.binary
28/5/2019 -- 11:36:35 - <Info> -- Loading /home/sensors/conf/suricata/update/modify.conf.
28/5/2019 -- 11:36:35 - <Debug> -- Parsing group matcher: group:web_server
28/5/2019 -- 11:36:35 - <Info> -- Loading /usr/local/etc/suricata/suricata.yaml
28/5/2019 -- 11:36:35 - <Info> -- Disabling rules with proto modbus
28/5/2019 -- 11:36:35 - <Info> -- Disabling rules with proto enip
28/5/2019 -- 11:36:35 - <Info> -- Disabling rules with proto dnp3
28/5/2019 -- 11:36:35 - <Debug> -- Adding source file:///home/sensors/Rules/raw/emerging-suri.rules.tar.gz.
28/5/2019 -- 11:36:35 - <Info> -- Last download less than 15 minutes ago. Not downloading file:///home/sensors/Rules/raw/emerging-suri.rules.tar.gz.
28/5/2019 -- 11:36:35 - <Info> -- Loading local file /usr/local/var/lib/suricata/rules/local.rules
28/5/2019 -- 11:36:35 - <Info> -- Ignoring file rules/shellcode.rules
28/5/2019 -- 11:36:35 - <Info> -- Ignoring file rules/imap.rules
<snip Ignoring lines >
28/5/2019 -- 11:36:35 - <Info> -- Ignoring file rules/malware.rules
28/5/2019 -- 11:36:35 - <Info> -- Ignoring file rules/ciarmy.rules
28/5/2019 -- 11:36:35 - <Info> -- Ignoring file rules/smtp.rules
< snip parsing lines >
28/5/2019 -- 11:36:38 - <Debug> -- Parsing rules/botcc.portgrouped.rules.
28/5/2019 -- 11:36:38 - <Debug> -- Parsing rules/info.rules.
28/5/2019 -- 11:36:38 - <Info> -- Loaded 36363 rules.
28/5/2019 -- 11:36:38 - <Debug> -- Disabling: [1:2013845] ET INFO DYNAMIC_DNS Query to a Suspicious *.ez-dns.com Domain
28/5/2019 -- 11:36:38 - <Debug> -- Disabling: [1:2811611] ETPRO POLICY DNS Query to .onion proxy Domain (paywelcomefor.com)
28/5/2019 -- 11:36:38 - <Debug> -- Disabling: [1:2020976] ET EXPLOIT Possible Redirect to SMB exploit attempt - 307

< snip *lots* of Disabling lines>

28/5/2019 -- 11:37:09 - <Debug> -- Disabling: [1:2810697] ETPRO POLICY DNS Query to .onion proxy Domain (toradvisor.com)
28/5/2019 -- 11:37:09 - <Debug> -- Disabling: [1:2833567] ETPRO EXPLOIT Possible Novidade EK Attempting Intranet Router Compromise M9 (Bruteforce)
28/5/2019 -- 11:37:12 - <Info> -- Disabled 3190 rules.
28/5/2019 -- 11:37:12 - <Info> -- Enabled 76 rules.
28/5/2019 -- 11:37:12 - <Info> -- Modified 18 rules.
28/5/2019 -- 11:37:12 - <Info> -- Dropped 0 rules.
28/5/2019 -- 11:37:13 - <Debug> -- Found 383 required flowbits.
28/5/2019 -- 11:37:13 - <Debug> -- Found 141 rules to enable to for flowbit requirements
28/5/2019 -- 11:37:13 - <Debug> -- Enabling previously disabled rule for flowbits: # [1:2014954] ET INFO Vulnerable iTunes Version 10.6.x
28/5/2019 -- 11:37:13 - <Debug> -- Enabling previously disabled rule for flowbits: # [1:2017226] ET INFO Obfuscated Split String (Single Q) 7
28/5/2019 -- 11:37:13 - <Debug> -- Enabling previously disabled rule for flowbits: # [1:2019834] ET INFO Microsoft Compact Office Document Format File Download

<snip quite a few "Enabling previously disabled rule for flowbits” lines >

28/5/2019 -- 11:37:13 - <Debug> -- Enabling previously disabled rule for flowbits: # [1:2022520] ET POLICY Possible HTA Application Download
28/5/2019 -- 11:37:13 - <Debug> -- Found 384 required flowbits.
28/5/2019 -- 11:37:13 - <Debug> -- Found 1 rules to enable to for flowbit requirements
28/5/2019 -- 11:37:13 - <Debug> -- Enabling previously disabled rule for flowbits: # [1:2008298] ET CHAT GaduGadu Chat Client Login Packet
28/5/2019 -- 11:37:13 - <Debug> -- Found 385 required flowbits.
28/5/2019 -- 11:37:13 - <Debug> -- Found 1 rules to enable to for flowbit requirements
28/5/2019 -- 11:37:13 - <Debug> -- Enabling previously disabled rule for flowbits: # [1:2008297] ET CHAT GaduGadu Chat Server Welcome Packet
28/5/2019 -- 11:37:13 - <Debug> -- Found 385 required flowbits.
28/5/2019 -- 11:37:13 - <Debug> -- Found 0 rules to enable to for flowbit requirements
28/5/2019 -- 11:37:13 - <Debug> -- All required rules enabled.
28/5/2019 -- 11:37:13 - <Info> -- Enabled 143 rules for flowbit dependencies.
28/5/2019 -- 11:37:13 - <Info> -- Backing up current rules.
28/5/2019 -- 11:37:13 - <Debug> -- Recording existing file /usr/local/var/lib/suricata/rules/suricata.rules with hash '6fc141c77de86dde58d3c313af48ae4c'.
28/5/2019 -- 11:37:16 - <Info> -- Writing rules to /usr/local/var/lib/suricata/rules/suricata.rules: total: 36363; enabled: 28972; added: 97; removed 0; modified: 306
28/5/2019 -- 11:37:16 - <Info> -- Writing /home/sensors/Rules/sensor/sid-msg.map.
28/5/2019 -- 11:37:17 - <Info> -- Testing with suricata -T.
28/5/2019 -- 11:37:17 - <Debug> -- Running /usr/local/bin/suricata -T -l /tmp -c /usr/local/etc/suricata/suricata.yaml -S /usr/local/var/lib/suricata/rules/suricata.rules; env={'SC_LOG_FORMAT': '%t - <%d> -- ', 'SC_LOG_LEVEL': 'Warning', 'ASAN_OPTIONS': 'detect_leaks=0'}
28/5/2019 -- 11:37:17 - <Error> -- Suricata test failed, aborting.
28/5/2019 -- 11:37:17 - <Error> -- Restoring previous rules.

Work fine when run with —no-test. (surprise)

This is happening on all (six) sensors which had pretty well identical setups.

If I get time today I will resort to the “real” documentation (much as I hate python ;).

I am currently working on upgrading the OS on my sensors to 18.04 so I can use Peter’s packages with hyperscan so this is not a big deal for me.

Russell

> Thanks !
> 
>> this is with a new install of 1.0.5 (as well as the original 1.0.0).
>> 
>> Any idea what is causing suricata-update to append that hash definition to the command line?
>> 
>> I also tried explicitly setting the testing command in the yaml.
>> 
>> I am running it with —no-test at the moment but it would be good to have the rules tested before getting loaded.
>> 
>> Russell
>> 
>> _______________________________________________
>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
>> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>> 
>> Conference: https://suricon.net
>> Trainings: https://suricata-ids.org/training/
> 
> 
> 
> -- 
> Regards,
> Peter Manev