[OISF/outreachy] [outreachy] Claiming Bug #2851

[Shivani Bhardwaj]

On Thu, Mar 14, 2019 at 3:35 PM Vidushi Agrawal <vidushi229 at gmail.com> wrote:
>
>
>
> On Thu, Mar 14, 2019 at 11:04 AM Shivani Bhardwaj <sbhardwaj at openinfosecfoundation.org> wrote:
>>
>> On Thu, Mar 14, 2019 at 12:42 AM Vidushi Agrawal <vidushi229 at gmail.com> wrote:
>> >
>> >
>> >
>> > On Wed, Mar 13, 2019 at 11:46 PM Shivani Bhardwaj <sbhardwaj at openinfosecfoundation.org> wrote:
>> >>
>> >> On Wed, Mar 13, 2019 at 4:18 PM Vidushi Agrawal <vidushi229 at gmail.com> wrote:
>> >> >
>> >> >
>> >> >
>> >> > On Wed, Mar 13, 2019 at 2:05 PM Shivani Bhardwaj <sbhardwaj at openinfosecfoundation.org> wrote:
>> >> >>
>> >> >> Hi, Vidushi!
>> >> >>
>> >> >> On Wed, Mar 13, 2019 at 1:28 AM Vidushi Agrawal via Outreachy
>> >> >> <outreachy at lists.openinfosecfoundation.org> wrote:
>> >> >> >
>> >> >> > Hi,
>> >> >> > I'd like to start working on Bug #2851. If none's assigned to it yet, can it be assigned to me?
>> >> >> Yes, please. Go ahead and start working on it. Do you mind making an
>> >> >> account on redmine? I shall assign the task on tracker to you.
>> >> >
>> >> > I've made an account on redmine.
>> >> >
>> >> >> > Also, could you please guide me how to reproduce the issue after changing update.yaml file?
>> >> >> All the explanation is assuming that you have a working installation
>> >> >> of suricata-update.
>> >> >
>> >> > Yes, now I  do have a working installation of suricata-update.
>> >> >
>> >> >> See below.
>> >> >> One good thing to do would be to use "-v" flag while running
>> >> >> suricata-update. You'll get to know about a lot of settings and
>> >> >> environment like where the different cofiguration and rule files
>> >> >> reside which suricata-update is trying to read/write from/to.
>> >> >>
>> >> >> For this particular issue, in the log after running "./bin/suricata-update -v",
>> >> >> Look for "Loading /path/to/update.yaml"
>> >> >
>> >> >
>> >> > On running  "./bin/suricata-update -v", I get the loading message
>> >> > 13/3/2019 -- 15:54:13 - <Info> -- Loading /usr/local/etc/suricata/suricata.yaml
>> >> >
>> >> > There's "Loading /path/to/suricata.yaml" not update.yaml
>> >> >
>> >> Could you please create one in /etc/suricata? See sample update.yaml
>> >> here: https://suricata-update.readthedocs.io/en/latest/update.html#example-configuration-file-etc-suricata-update-yaml
>> >
>> >
>> > I created one update.yaml in /etc/suricata and I checked the path in config.py. It still loads suricata.yaml instead of update.yaml. What could the possible fix be?
>>
>> Could you please give the entire log?
>
>
> On running ./bin/suricata-update -v I get the following output:
>
> 14/3/2019 -- 15:27:01 - <Debug> -- This is suricata-update version 1.1.0dev0 (rev: None); Python: 2.7.15rc1 (default, Nov 12 2018, 14:31:15) - [GCC 7.3.0]
> 14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value force -> False
> 14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value verbose -> True
> 14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value enable -> False
> 14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value no-merge -> False
> 14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value version -> False
> 14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value dump-sample-configs -> False
> 14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value no-test -> False
> 14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value subcommand -> update
> 14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value modify -> False
> 14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value no-reload -> False
> 14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value no-ignore -> False
> 14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value disable -> False
> 14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value etopen -> False
> 14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value now -> False
> 14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value url -> []
> 14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value drop -> False
> 14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value ignore -> []
> 14/3/2019 -- 15:27:01 - <Debug> -- Looking for suricata in /home/vidushi/bin
> 14/3/2019 -- 15:27:01 - <Debug> -- Looking for suricata in /home/vidushi/.local/bin
> 14/3/2019 -- 15:27:01 - <Debug> -- Looking for suricata in /usr/local/sbin
> 14/3/2019 -- 15:27:01 - <Debug> -- Looking for suricata in /usr/local/bin
> 14/3/2019 -- 15:27:01 - <Debug> -- Found /usr/local/bin/suricata.
> 14/3/2019 -- 15:27:01 - <Info> -- Using data-directory /usr/local/var/lib/suricata.
> 14/3/2019 -- 15:27:01 - <Info> -- Using Suricata configuration /usr/local/etc/suricata/suricata.yaml
> 14/3/2019 -- 15:27:01 - <Info> -- Found Suricata version 5.0.0-dev at /usr/local/bin/suricata.
> 14/3/2019 -- 15:27:01 - <Info> -- Loading /usr/local/etc/suricata/suricata.yaml
> 14/3/2019 -- 15:27:01 - <Info> -- Disabling rules with proto dhcp
> 14/3/2019 -- 15:27:01 - <Info> -- Disabling rules with proto tftp
> 14/3/2019 -- 15:27:01 - <Info> -- Disabling rules with proto krb5
> 14/3/2019 -- 15:27:01 - <Info> -- Disabling rules with proto ntp
> 14/3/2019 -- 15:27:01 - <Info> -- Disabling rules with proto modbus
> 14/3/2019 -- 15:27:01 - <Info> -- Disabling rules with proto enip
> 14/3/2019 -- 15:27:01 - <Info> -- Disabling rules with proto dnp3
> 14/3/2019 -- 15:27:01 - <Info> -- Disabling rules with proto nfs
> 14/3/2019 -- 15:27:01 - <Info> -- No sources configured, will use Emerging Threats Open
> 14/3/2019 -- 15:27:01 - <Info> -- Checking https://rules.emergingthreats.net/open/suricata-5.0.0/emerging.rules.tar.gz.md5.
> 14/3/2019 -- 15:27:01 - <Debug> -- Setting HTTP User-Agent to Suricata-Update/1.1.0dev0 (OS: Linux; CPU: x86_64; Python: 2.7.15rc1; Dist: Ubuntu/18.04; Suricata: 5.0.0-dev)
> 14/3/2019 -- 15:27:03 - <Debug> -- Local checksum=|71780cede70d4e28397745292843be1b|; remote checksum=|af5c3120d83827ba36a05d1c50a4fc9c|
> 14/3/2019 -- 15:27:03 - <Info> -- Fetching https://rules.emergingthreats.net/open/suricata-5.0.0/emerging.rules.tar.gz.
> 14/3/2019 -- 15:27:03 - <Debug> -- Setting HTTP User-Agent to Suricata-Update/1.1.0dev0 (OS: Linux; CPU: x86_64; Python: 2.7.15rc1; Dist: Ubuntu/18.04; Suricata: 5.0.0-dev)
>  100% - 2333940/2333940
> 14/3/2019 -- 15:27:25 - <Info> -- Done.
> 14/3/2019 -- 15:27:25 - <Warning> -- No distribution rule directory found.
> 14/3/2019 -- 15:27:25 - <Debug> -- Parsing rules/emerging-mobile_malware.rules.
> 14/3/2019 -- 15:27:25 - <Debug> -- Parsing rules/emerging-icmp.rules.
> 14/3/2019 -- 15:27:25 - <Debug> -- Parsing rules/tor.rules.
> 14/3/2019 -- 15:27:25 - <Debug> -- Parsing rules/emerging-activex.rules.
> 14/3/2019 -- 15:27:25 - <Debug> -- Parsing rules/emerging-icmp_info.rules.
> 14/3/2019 -- 15:27:25 - <Debug> -- Parsing rules/emerging-policy.rules.
> 14/3/2019 -- 15:27:26 - <Debug> -- Parsing rules/emerging-pop3.rules.
> 14/3/2019 -- 15:27:26 - <Debug> -- Parsing rules/emerging-shellcode.rules.
> 14/3/2019 -- 15:27:26 - <Debug> -- Parsing rules/emerging-attack_response.rules.
> 14/3/2019 -- 15:27:26 - <Debug> -- Parsing rules/emerging-trojan.rules.
> 14/3/2019 -- 15:27:26 - <Debug> -- Parsing rules/emerging-dns.rules.
> 14/3/2019 -- 15:27:26 - <Debug> -- Parsing rules/emerging-telnet.rules.
> 14/3/2019 -- 15:27:26 - <Debug> -- Parsing rules/emerging-scada.rules.
> 14/3/2019 -- 15:27:26 - <Debug> -- Parsing rules/emerging-misc.rules.
> 14/3/2019 -- 15:27:26 - <Debug> -- Parsing rules/dshield.rules.
> 14/3/2019 -- 15:27:26 - <Debug> -- Parsing rules/emerging-sql.rules.
> 14/3/2019 -- 15:27:26 - <Debug> -- Parsing rules/emerging-inappropriate.rules.
> 14/3/2019 -- 15:27:26 - <Debug> -- Parsing rules/emerging-web_server.rules.
> 14/3/2019 -- 15:27:26 - <Debug> -- Parsing rules/emerging-web_specific_apps.rules.
> 14/3/2019 -- 15:27:27 - <Debug> -- Parsing rules/emerging-user_agents.rules.
> 14/3/2019 -- 15:27:27 - <Debug> -- Parsing rules/emerging-exploit.rules.
> 14/3/2019 -- 15:27:27 - <Debug> -- Parsing rules/emerging-malware.rules.
> 14/3/2019 -- 15:27:27 - <Debug> -- Parsing rules/emerging-info.rules.
> 14/3/2019 -- 15:27:27 - <Debug> -- Parsing rules/botcc.rules.
> 14/3/2019 -- 15:27:27 - <Debug> -- Parsing rules/emerging-rpc.rules.
> 14/3/2019 -- 15:27:27 - <Debug> -- Parsing rules/compromised.rules.
> 14/3/2019 -- 15:27:27 - <Debug> -- Parsing rules/emerging-tftp.rules.
> 14/3/2019 -- 15:27:27 - <Debug> -- Parsing rules/emerging-ftp.rules.
> 14/3/2019 -- 15:27:27 - <Debug> -- Parsing rules/emerging-netbios.rules.
> 14/3/2019 -- 15:27:27 - <Debug> -- Parsing rules/emerging-current_events.rules.
> 14/3/2019 -- 15:27:28 - <Debug> -- Parsing rules/emerging-p2p.rules.
> 14/3/2019 -- 15:27:28 - <Debug> -- Parsing rules/drop.rules.
> 14/3/2019 -- 15:27:28 - <Debug> -- Parsing rules/emerging-scan.rules.
> 14/3/2019 -- 15:27:28 - <Debug> -- Parsing rules/emerging-games.rules.
> 14/3/2019 -- 15:27:28 - <Debug> -- Parsing rules/emerging-imap.rules.
> 14/3/2019 -- 15:27:28 - <Debug> -- Parsing rules/emerging-deleted.rules.
> 14/3/2019 -- 15:27:28 - <Debug> -- Parsing rules/emerging-chat.rules.
> 14/3/2019 -- 15:27:28 - <Debug> -- Parsing rules/emerging-web_client.rules.
> 14/3/2019 -- 15:27:28 - <Debug> -- Parsing rules/botcc.portgrouped.rules.
> 14/3/2019 -- 15:27:28 - <Debug> -- Parsing rules/emerging-smtp.rules.
> 14/3/2019 -- 15:27:28 - <Debug> -- Parsing rules/emerging-dos.rules.
> 14/3/2019 -- 15:27:28 - <Debug> -- Parsing rules/emerging-snmp.rules.
> 14/3/2019 -- 15:27:28 - <Debug> -- Parsing rules/ciarmy.rules.
> 14/3/2019 -- 15:27:28 - <Debug> -- Parsing rules/emerging-worm.rules.
> 14/3/2019 -- 15:27:28 - <Debug> -- Parsing rules/emerging-voip.rules.
> 14/3/2019 -- 15:27:28 - <Info> -- Loaded 26887 rules.
> 14/3/2019 -- 15:27:29 - <Info> -- Disabled 0 rules.
> 14/3/2019 -- 15:27:29 - <Info> -- Enabled 0 rules.
> 14/3/2019 -- 15:27:29 - <Info> -- Modified 0 rules.
> 14/3/2019 -- 15:27:29 - <Info> -- Dropped 0 rules.
> 14/3/2019 -- 15:27:29 - <Debug> -- Found 195 required flowbits.
> 14/3/2019 -- 15:27:29 - <Debug> -- Found 187 rules to enable to for flowbit requirements
> 14/3/2019 -- 15:27:29 - <Debug> -- Found 197 required flowbits.
> 14/3/2019 -- 15:27:29 - <Debug> -- Found 1 rules to enable to for flowbit requirements
> 14/3/2019 -- 15:27:29 - <Debug> -- Found 197 required flowbits.
> 14/3/2019 -- 15:27:29 - <Debug> -- Found 0 rules to enable to for flowbit requirements
> 14/3/2019 -- 15:27:29 - <Debug> -- All required rules enabled.
> 14/3/2019 -- 15:27:29 - <Info> -- Enabled 188 rules for flowbit dependencies.
> 14/3/2019 -- 15:27:29 - <Info> -- Backing up current rules.
> 14/3/2019 -- 15:27:29 - <Debug> -- Recording existing file /usr/local/var/lib/suricata/rules/suricata.rules with hash 'e5319f78798e445dce8219fc470f9c5c'.
> 14/3/2019 -- 15:27:32 - <Info> -- Writing rules to /usr/local/var/lib/suricata/rules/suricata.rules: total: 26887; enabled: 19533; added: 7; removed 3; modified: 1204
> 14/3/2019 -- 15:27:33 - <Info> -- Testing with suricata -T.
> 14/3/2019 -- 15:27:33 - <Debug> -- Running /usr/local/bin/suricata -T -l /tmp -c /usr/local/etc/suricata/suricata.yaml -S /usr/local/var/lib/suricata/rules/suricata.rules; env={'SC_LOG_FORMAT': '%t - <%d> -- ', 'SC_LOG_LEVEL': 'Warning', 'ASAN_OPTIONS': 'detect_leaks=0'}
> 14/3/2019 -- 15:27:37 - <Info> -- Done.
>
hmm this is weird. I actually just tried the exact installation
instructions and what I mentioned to you on a fresh Docker container,
works out. Check the permissions of /etc/suricata.
Anyway, for now, you could pass the path to update.yaml file with -c option.
Read about it here:
https://suricata-update.readthedocs.io/en/latest/update.html#cmdoption-c
>>
>>
>> >>
>> >>
>> >>
>> >> >> Then make the changes as described by the person (in the issue on
>> >> >> redmine) in update.yaml file on the path you just discovered from the
>> >> >> above mentioned log line.
>> >> >> On looking closely at the log, you will see a line "Parsing
>> >> >> rules/emerging-deleted.rules."
>> >> >
>> >> > Yes, I do see this line.
>> >> >
>> >> >> This is the problem that the person is defining. Despite defining in
>> >> >> the configuration for update to ignore any rule files with
>> >> >> "deleted.rules" in their name, a file with name *deleted.rules is
>> >> >> still being processed.
>> >> >
>> >> > I did understand the problem. But on running suricata-update with -v flag, it is loading suricata.yaml not update.yaml. Am I doing it wrong or is it a problem with the installation?
>> >> >
>> >> >> Now, apply the changes you think are ideal for this case. Run
>> >> >> suricata-update again with -v flag. Observe the output. The "Parsing
>> >> >> rules/emerging-deleted.rules." should no longer be in the log.
>> >> >>
>> >> >> You could store the logs in both the cases and then run a diff on them
>> >> >> to see if something changed as per your expectations.
>> >> >> Does this help?
>> >> >
>> >> > Thank you
>> >> >>
>> >> >> > Thanks,
>> >> >> > Vidushi
>> >> >> > _______________________________________________
>> >> >> > Outreachy mailing list
>> >> >> > Outreachy at lists.openinfosecfoundation.org
>> >> >> > https://lists.openinfosecfoundation.org/listinfo/outreachy
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Shivani
>> >>
>> >>
>> >>
>> >> --
>> >> Shivani
>>
>>
>>
>> --
>> Shivani



-- 
Shivani