[OISF/outreachy] [outreachy] Claiming Bug #2851

[Vidushi Agrawal]

On Thu, Mar 14, 2019 at 11:04 AM Shivani Bhardwaj <
sbhardwaj at openinfosecfoundation.org> wrote:

> On Thu, Mar 14, 2019 at 12:42 AM Vidushi Agrawal <vidushi229 at gmail.com>
> wrote:
> >
> >
> >
> > On Wed, Mar 13, 2019 at 11:46 PM Shivani Bhardwaj <
> sbhardwaj at openinfosecfoundation.org> wrote:
> >>
> >> On Wed, Mar 13, 2019 at 4:18 PM Vidushi Agrawal <vidushi229 at gmail.com>
> wrote:
> >> >
> >> >
> >> >
> >> > On Wed, Mar 13, 2019 at 2:05 PM Shivani Bhardwaj <
> sbhardwaj at openinfosecfoundation.org> wrote:
> >> >>
> >> >> Hi, Vidushi!
> >> >>
> >> >> On Wed, Mar 13, 2019 at 1:28 AM Vidushi Agrawal via Outreachy
> >> >> <outreachy at lists.openinfosecfoundation.org> wrote:
> >> >> >
> >> >> > Hi,
> >> >> > I'd like to start working on Bug #2851. If none's assigned to it
> yet, can it be assigned to me?
> >> >> Yes, please. Go ahead and start working on it. Do you mind making an
> >> >> account on redmine? I shall assign the task on tracker to you.
> >> >
> >> > I've made an account on redmine.
> >> >
> >> >> > Also, could you please guide me how to reproduce the issue after
> changing update.yaml file?
> >> >> All the explanation is assuming that you have a working installation
> >> >> of suricata-update.
> >> >
> >> > Yes, now I  do have a working installation of suricata-update.
> >> >
> >> >> See below.
> >> >> One good thing to do would be to use "-v" flag while running
> >> >> suricata-update. You'll get to know about a lot of settings and
> >> >> environment like where the different cofiguration and rule files
> >> >> reside which suricata-update is trying to read/write from/to.
> >> >>
> >> >> For this particular issue, in the log after running
> "./bin/suricata-update -v",
> >> >> Look for "Loading /path/to/update.yaml"
> >> >
> >> >
> >> > On running  "./bin/suricata-update -v", I get the loading message
> >> > 13/3/2019 -- 15:54:13 - <Info> -- Loading
> /usr/local/etc/suricata/suricata.yaml
> >> >
> >> > There's "Loading /path/to/suricata.yaml" not update.yaml
> >> >
> >> Could you please create one in /etc/suricata? See sample update.yaml
> >> here:
> https://suricata-update.readthedocs.io/en/latest/update.html#example-configuration-file-etc-suricata-update-yaml
> >
> >
> > I created one update.yaml in /etc/suricata and I checked the path in
> config.py. It still loads suricata.yaml instead of update.yaml. What could
> the possible fix be?
>
> Could you please give the entire log?
>

On running ./bin/suricata-update -v I get the following output:

14/3/2019 -- 15:27:01 - <Debug> -- This is suricata-update version
1.1.0dev0 (rev: None); Python: 2.7.15rc1 (default, Nov 12 2018, 14:31:15) -
[GCC 7.3.0]
14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value force ->
False
14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value verbose ->
True
14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value enable ->
False
14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value no-merge ->
False
14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value version ->
False
14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value
dump-sample-configs -> False
14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value no-test ->
False
14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value subcommand
-> update
14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value modify ->
False
14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value no-reload ->
False
14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value no-ignore ->
False
14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value disable ->
False
14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value etopen ->
False
14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value now -> False
14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value url -> []
14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value drop -> False
14/3/2019 -- 15:27:01 - <Debug> -- Setting configuration value ignore -> []
14/3/2019 -- 15:27:01 - <Debug> -- Looking for suricata in /home/vidushi/bin
14/3/2019 -- 15:27:01 - <Debug> -- Looking for suricata in
/home/vidushi/.local/bin
14/3/2019 -- 15:27:01 - <Debug> -- Looking for suricata in /usr/local/sbin
14/3/2019 -- 15:27:01 - <Debug> -- Looking for suricata in /usr/local/bin
14/3/2019 -- 15:27:01 - <Debug> -- Found /usr/local/bin/suricata.
14/3/2019 -- 15:27:01 - <Info> -- Using data-directory
/usr/local/var/lib/suricata.
14/3/2019 -- 15:27:01 - <Info> -- Using Suricata configuration
/usr/local/etc/suricata/suricata.yaml
14/3/2019 -- 15:27:01 - <Info> -- Found Suricata version 5.0.0-dev at
/usr/local/bin/suricata.
14/3/2019 -- 15:27:01 - <Info> -- Loading
/usr/local/etc/suricata/suricata.yaml
14/3/2019 -- 15:27:01 - <Info> -- Disabling rules with proto dhcp
14/3/2019 -- 15:27:01 - <Info> -- Disabling rules with proto tftp
14/3/2019 -- 15:27:01 - <Info> -- Disabling rules with proto krb5
14/3/2019 -- 15:27:01 - <Info> -- Disabling rules with proto ntp
14/3/2019 -- 15:27:01 - <Info> -- Disabling rules with proto modbus
14/3/2019 -- 15:27:01 - <Info> -- Disabling rules with proto enip
14/3/2019 -- 15:27:01 - <Info> -- Disabling rules with proto dnp3
14/3/2019 -- 15:27:01 - <Info> -- Disabling rules with proto nfs
14/3/2019 -- 15:27:01 - <Info> -- No sources configured, will use Emerging
Threats Open
14/3/2019 -- 15:27:01 - <Info> -- Checking
https://rules.emergingthreats.net/open/suricata-5.0.0/emerging.rules.tar.gz.md5
.
14/3/2019 -- 15:27:01 - <Debug> -- Setting HTTP User-Agent to
Suricata-Update/1.1.0dev0 (OS: Linux; CPU: x86_64; Python: 2.7.15rc1; Dist:
Ubuntu/18.04; Suricata: 5.0.0-dev)
14/3/2019 -- 15:27:03 - <Debug> -- Local
checksum=|71780cede70d4e28397745292843be1b|; remote
checksum=|af5c3120d83827ba36a05d1c50a4fc9c|
14/3/2019 -- 15:27:03 - <Info> -- Fetching
https://rules.emergingthreats.net/open/suricata-5.0.0/emerging.rules.tar.gz.
14/3/2019 -- 15:27:03 - <Debug> -- Setting HTTP User-Agent to
Suricata-Update/1.1.0dev0 (OS: Linux; CPU: x86_64; Python: 2.7.15rc1; Dist:
Ubuntu/18.04; Suricata: 5.0.0-dev)
 100% - 2333940/2333940
14/3/2019 -- 15:27:25 - <Info> -- Done.
14/3/2019 -- 15:27:25 - <Warning> -- No distribution rule directory found.
14/3/2019 -- 15:27:25 - <Debug> -- Parsing
rules/emerging-mobile_malware.rules.
14/3/2019 -- 15:27:25 - <Debug> -- Parsing rules/emerging-icmp.rules.
14/3/2019 -- 15:27:25 - <Debug> -- Parsing rules/tor.rules.
14/3/2019 -- 15:27:25 - <Debug> -- Parsing rules/emerging-activex.rules.
14/3/2019 -- 15:27:25 - <Debug> -- Parsing rules/emerging-icmp_info.rules.
14/3/2019 -- 15:27:25 - <Debug> -- Parsing rules/emerging-policy.rules.
14/3/2019 -- 15:27:26 - <Debug> -- Parsing rules/emerging-pop3.rules.
14/3/2019 -- 15:27:26 - <Debug> -- Parsing rules/emerging-shellcode.rules.
14/3/2019 -- 15:27:26 - <Debug> -- Parsing
rules/emerging-attack_response.rules.
14/3/2019 -- 15:27:26 - <Debug> -- Parsing rules/emerging-trojan.rules.
14/3/2019 -- 15:27:26 - <Debug> -- Parsing rules/emerging-dns.rules.
14/3/2019 -- 15:27:26 - <Debug> -- Parsing rules/emerging-telnet.rules.
14/3/2019 -- 15:27:26 - <Debug> -- Parsing rules/emerging-scada.rules.
14/3/2019 -- 15:27:26 - <Debug> -- Parsing rules/emerging-misc.rules.
14/3/2019 -- 15:27:26 - <Debug> -- Parsing rules/dshield.rules.
14/3/2019 -- 15:27:26 - <Debug> -- Parsing rules/emerging-sql.rules.
14/3/2019 -- 15:27:26 - <Debug> -- Parsing
rules/emerging-inappropriate.rules.
14/3/2019 -- 15:27:26 - <Debug> -- Parsing rules/emerging-web_server.rules.
14/3/2019 -- 15:27:26 - <Debug> -- Parsing
rules/emerging-web_specific_apps.rules.
14/3/2019 -- 15:27:27 - <Debug> -- Parsing rules/emerging-user_agents.rules.
14/3/2019 -- 15:27:27 - <Debug> -- Parsing rules/emerging-exploit.rules.
14/3/2019 -- 15:27:27 - <Debug> -- Parsing rules/emerging-malware.rules.
14/3/2019 -- 15:27:27 - <Debug> -- Parsing rules/emerging-info.rules.
14/3/2019 -- 15:27:27 - <Debug> -- Parsing rules/botcc.rules.
14/3/2019 -- 15:27:27 - <Debug> -- Parsing rules/emerging-rpc.rules.
14/3/2019 -- 15:27:27 - <Debug> -- Parsing rules/compromised.rules.
14/3/2019 -- 15:27:27 - <Debug> -- Parsing rules/emerging-tftp.rules.
14/3/2019 -- 15:27:27 - <Debug> -- Parsing rules/emerging-ftp.rules.
14/3/2019 -- 15:27:27 - <Debug> -- Parsing rules/emerging-netbios.rules.
14/3/2019 -- 15:27:27 - <Debug> -- Parsing
rules/emerging-current_events.rules.
14/3/2019 -- 15:27:28 - <Debug> -- Parsing rules/emerging-p2p.rules.
14/3/2019 -- 15:27:28 - <Debug> -- Parsing rules/drop.rules.
14/3/2019 -- 15:27:28 - <Debug> -- Parsing rules/emerging-scan.rules.
14/3/2019 -- 15:27:28 - <Debug> -- Parsing rules/emerging-games.rules.
14/3/2019 -- 15:27:28 - <Debug> -- Parsing rules/emerging-imap.rules.
14/3/2019 -- 15:27:28 - <Debug> -- Parsing rules/emerging-deleted.rules.
14/3/2019 -- 15:27:28 - <Debug> -- Parsing rules/emerging-chat.rules.
14/3/2019 -- 15:27:28 - <Debug> -- Parsing rules/emerging-web_client.rules.
14/3/2019 -- 15:27:28 - <Debug> -- Parsing rules/botcc.portgrouped.rules.
14/3/2019 -- 15:27:28 - <Debug> -- Parsing rules/emerging-smtp.rules.
14/3/2019 -- 15:27:28 - <Debug> -- Parsing rules/emerging-dos.rules.
14/3/2019 -- 15:27:28 - <Debug> -- Parsing rules/emerging-snmp.rules.
14/3/2019 -- 15:27:28 - <Debug> -- Parsing rules/ciarmy.rules.
14/3/2019 -- 15:27:28 - <Debug> -- Parsing rules/emerging-worm.rules.
14/3/2019 -- 15:27:28 - <Debug> -- Parsing rules/emerging-voip.rules.
14/3/2019 -- 15:27:28 - <Info> -- Loaded 26887 rules.
14/3/2019 -- 15:27:29 - <Info> -- Disabled 0 rules.
14/3/2019 -- 15:27:29 - <Info> -- Enabled 0 rules.
14/3/2019 -- 15:27:29 - <Info> -- Modified 0 rules.
14/3/2019 -- 15:27:29 - <Info> -- Dropped 0 rules.
14/3/2019 -- 15:27:29 - <Debug> -- Found 195 required flowbits.
14/3/2019 -- 15:27:29 - <Debug> -- Found 187 rules to enable to for flowbit
requirements
14/3/2019 -- 15:27:29 - <Debug> -- Found 197 required flowbits.
14/3/2019 -- 15:27:29 - <Debug> -- Found 1 rules to enable to for flowbit
requirements
14/3/2019 -- 15:27:29 - <Debug> -- Found 197 required flowbits.
14/3/2019 -- 15:27:29 - <Debug> -- Found 0 rules to enable to for flowbit
requirements
14/3/2019 -- 15:27:29 - <Debug> -- All required rules enabled.
14/3/2019 -- 15:27:29 - <Info> -- Enabled 188 rules for flowbit
dependencies.
14/3/2019 -- 15:27:29 - <Info> -- Backing up current rules.
14/3/2019 -- 15:27:29 - <Debug> -- Recording existing file
/usr/local/var/lib/suricata/rules/suricata.rules with hash
'e5319f78798e445dce8219fc470f9c5c'.
14/3/2019 -- 15:27:32 - <Info> -- Writing rules to
/usr/local/var/lib/suricata/rules/suricata.rules: total: 26887; enabled:
19533; added: 7; removed 3; modified: 1204
14/3/2019 -- 15:27:33 - <Info> -- Testing with suricata -T.
14/3/2019 -- 15:27:33 - <Debug> -- Running /usr/local/bin/suricata -T -l
/tmp -c /usr/local/etc/suricata/suricata.yaml -S
/usr/local/var/lib/suricata/rules/suricata.rules; env={'SC_LOG_FORMAT': '%t
- <%d> -- ', 'SC_LOG_LEVEL': 'Warning', 'ASAN_OPTIONS': 'detect_leaks=0'}
14/3/2019 -- 15:27:37 - <Info> -- Done.


>
> >>
> >>
> >>
> >> >> Then make the changes as described by the person (in the issue on
> >> >> redmine) in update.yaml file on the path you just discovered from the
> >> >> above mentioned log line.
> >> >> On looking closely at the log, you will see a line "Parsing
> >> >> rules/emerging-deleted.rules."
> >> >
> >> > Yes, I do see this line.
> >> >
> >> >> This is the problem that the person is defining. Despite defining in
> >> >> the configuration for update to ignore any rule files with
> >> >> "deleted.rules" in their name, a file with name *deleted.rules is
> >> >> still being processed.
> >> >
> >> > I did understand the problem. But on running suricata-update with -v
> flag, it is loading suricata.yaml not update.yaml. Am I doing it wrong or
> is it a problem with the installation?
> >> >
> >> >> Now, apply the changes you think are ideal for this case. Run
> >> >> suricata-update again with -v flag. Observe the output. The "Parsing
> >> >> rules/emerging-deleted.rules." should no longer be in the log.
> >> >>
> >> >> You could store the logs in both the cases and then run a diff on
> them
> >> >> to see if something changed as per your expectations.
> >> >> Does this help?
> >> >
> >> > Thank you
> >> >>
> >> >> > Thanks,
> >> >> > Vidushi
> >> >> > _______________________________________________
> >> >> > Outreachy mailing list
> >> >> > Outreachy at lists.openinfosecfoundation.org
> >> >> > https://lists.openinfosecfoundation.org/listinfo/outreachy
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Shivani
> >>
> >>
> >>
> >> --
> >> Shivani
>
>
>
> --
> Shivani
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/outreachy/attachments/20190314/da16f90d/attachment-0001.html>