[Oisf-users] App layer decoding of Eve.json alerts
SiNA
sina.rabbani at gmail.com
Tue Jul 12 22:33:30 UTC 2016
I think I know where my issue is, I am sending requests with an empty
body so there is really nothing for suricata decode. But when I send
an actual request with proper headers, I see the decoded entry in the
logs:
curl -v 104.155.198.240
Gives me:
{"timestamp":"2016-07-12T18:32:44.144781-0400","flow_id":4020961152,"in_iface":"eno1","event_type":"alert","src_ip":"104.155.198.240","src_port":80,"dest_ip":"172.16.1.254","dest_port":40420,"proto":"TCP","alert":{"action":"allowed","gid":1,"signature_id":1,"rev":1,"signature":"IPREP
High Value CnC","category":"","severity":3},"payload":"","stream":0}
{"timestamp":"2016-07-12T18:32:44.283854-0400","flow_id":4020961152,"in_iface":"eno1","event_type":"http","src_ip":"172.16.1.254","src_port":40420,"dest_ip":"104.155.198.240","dest_port":80,"proto":"TCP","tx_id":0,"http":{"hostname":"104.155.198.240","url":"\/","http_user_agent":"curl\/7.47.0","http_method":"GET","protocol":"HTTP\/1.1","length":0}}
Thanks again and sorry for the noise!
--
SiNA
PGP: 0x0B47D56D
On Tue, Jul 12, 2016 at 6:21 PM, SiNA <sina.rabbani at gmail.com> wrote:
> Here is the full experience, I'm wondering how to get more details
> about the alert that was generated other than manually decoding the
> payload from the alert log itself:
>
> reputation rules:
>
> root at mana:/redteam/var/log/suricata# grep "" /redteam/etc/suricata/rules/*
> /redteam/etc/suricata/rules/test_categories.txt:1,BadHosts,Known bad hosts
> /redteam/etc/suricata/rules/test_categories.txt:2,Google,Known google host
> /redteam/etc/suricata/rules/test_categories.txt:
> /redteam/etc/suricata/rules/test_ips.list:104.155.198.240,1,10
> /redteam/etc/suricata/rules/test_reputation.rules:alert ip any any ->
> any any (msg:"IPREP High Value CnC"; iprep:src,BadHosts,>,1; sid:1;
> rev:1;)
>
>
>
> Suricata start up:
> root at mana:/redteam/etc/suricata# /redteam/bin/suricata -c
> /redteam/etc/suricata/suricata.yaml -i eno1 -v
> 12/7/2016 -- 18:15:13 - <Notice> - This is Suricata version 3.1 RELEASE
> 12/7/2016 -- 18:15:13 - <Info> - CPUs/cores online: 24
> 12/7/2016 -- 18:15:13 - <Info> - Found an MTU of 1500 for 'eno1'
> 12/7/2016 -- 18:15:13 - <Info> - Loading reputation file:
> /redteam/etc/suricata/rules/test_ips.list
> 12/7/2016 -- 18:15:13 - <Info> - Loading rule file:
> /redteam/etc/suricata/rules/test_reputation.rules
> 12/7/2016 -- 18:15:13 - <Info> - 1 rule files processed. 1 rules
> successfully loaded, 0 rules failed
> 12/7/2016 -- 18:15:13 - <Info> - 1 signatures processed. 1 are IP-only
> rules, 0 are inspecting packet payload, 0 inspect application layer, 0
> are decoder event only
> 12/7/2016 -- 18:15:13 - <Info> - Threshold config parsed: 0 rule(s) found
> 12/7/2016 -- 18:15:13 - <Info> - fast output device (regular)
> initialized: fast.log
> 12/7/2016 -- 18:15:13 - <Info> - eve-log output device (regular)
> initialized: eve.json
> 12/7/2016 -- 18:15:13 - <Info> - http-log output device (regular)
> initialized: http.log
> 12/7/2016 -- 18:15:13 - <Info> - tls-log output device (regular)
> initialized: tls.log
> 12/7/2016 -- 18:15:13 - <Info> - dns-log output device (regular)
> initialized: dns.log
> 12/7/2016 -- 18:15:13 - <Info> - Using log dir /redteam/var/log/suricata/
> 12/7/2016 -- 18:15:13 - <Info> - using normal logging
> 12/7/2016 -- 18:15:13 - <Info> - NIC offloading on eno1: GRO: SET, LRO: unset
> 12/7/2016 -- 18:15:13 - <Warning> - [ERRCODE: SC_ERR_AFP_CREATE(190)]
> - Using AF_PACKET with GRO or LRO activated can lead to capture
> problems
> 12/7/2016 -- 18:15:13 - <Info> - Going to use 24 thread(s)
> 12/7/2016 -- 18:15:13 - <Notice> - all 24 packet processing threads, 2
> management threads initialized, engine started.
> 12/7/2016 -- 18:15:14 - <Info> - All AFP capture threads are running.
>
>
> suricata.yaml:
>
> <--snip-->
>
> default-rule-path: /redteam/etc/suricata/rules
> rule-files:
> - test_reputation.rules
>
> reputation-categories-file: /redteam/etc/suricata/rules/test_categories.txt
> default-reputation-path: /redteam/etc/suricata/rules/
>
> reputation-files:
> - test_ips.list
>
>
> classification-file: /redteam/etc/suricata/classification.config
> reference-config-file: /redteam/etc/suricata/reference.config
> # threshold-file: /redteam/etc/suricata/threshold.config
>
> <--snip-->
>
>
> # Extensible Event Format (nicknamed EVE) event log in JSON format
> - eve-log:
> enabled: yes
> filetype: regular #regular|syslog|unix_dgram|unix_stream|redis
> filename: eve.json
> #prefix: "@cee: " # prefix to prepend to each log entry
> # the following are valid when type: syslog above
> #identity: "suricata"
> #facility: local5
> #level: Info ## possible levels: Emergency, Alert, Critical,
> ## Error, Warning, Notice, Info, Debug
> #redis:
> # server: 127.0.0.1
> # port: 6379
> # mode: list ## possible values: list (default), channel
> # key: suricata ## key or channel to use (default to suricata)
> # Redis pipelining set up. This will enable to only do a query every
> # 'batch-size' events. This should lower the latency induced by network
> # connection at the cost of some memory. There is no flushing implemented
> # so this setting as to be reserved to high traffic suricata.
> # pipelining:
> # enabled: yes ## set enable to yes to enable query pipelining
> # batch-size: 10 ## number of entry to keep in buffer
> types:
> - alert:
> payload: yes # enable dumping payload in Base64
> # payload-buffer-size: 4kb # max size of payload buffer to
> output in eve-log
> # payload-printable: yes # enable dumping payload in
> printable (lossy) format
> # packet: yes # enable dumping of packet
> (without stream segments)
> http: yes # enable dumping of http fields
> tls: yes # enable dumping of tls fields
> ssh: yes # enable dumping of ssh fields
> smtp: yes # enable dumping of smtp fields
>
> # HTTP X-Forwarded-For support by adding an extra field or
> overwriting
> # the source or destination IP address (depending on flow direction)
> # with the one reported in the X-Forwarded-For HTTP header. This is
> # helpful when reviewing alerts for traffic that is being reverse
> # or forward proxied.
> xff:
> enabled: no
> # Two operation modes are available, "extra-data" and "overwrite".
> mode: extra-data
> # Two proxy deployments are supported, "reverse" and "forward". In
> # a "reverse" deployment the IP address used is the last one, in a
> # "forward" deployment the first IP address is used.
> deployment: reverse
> # Header name where the actual IP address will be
> reported, if more
> # than one IP address is present, the last IP address will be the
> # one taken into consideration.
> header: X-Forwarded-For
> - http:
> extended: yes # enable this for extended logging information
> # custom allows additional http fields to be included in eve-log
> # the example below adds three additional fields when uncommented
> #custom: [Accept-Encoding, Accept-Language, Authorization]
> - dns
> - tls:
> extended: yes # enable this for extended logging information
> - files:
> force-magic: no # force logging magic on all logged files
> force-md5: no # force logging of md5 checksums
> #- drop:
> # alerts: no # log alerts that caused drops
> - smtp:
> #extended: yes # enable this for extended logging information
> # this includes: bcc, message-id, subject, x_mailer, user-agent
> # custom fields logging from the list:
> # reply-to, bcc, message-id, subject, x-mailer,
> user-agent, received,
> # x-originating-ip, in-reply-to, references, importance, priority,
> # sensitivity, organization, content-md5, date
> #custom: [received, x-mailer, x-originating-ip, relays,
> reply-to, bcc]
> # output md5 of fields: body, subject
> # for the body you need to set
> app-layer.protocols.smtp.mime.body-md5
> # to yes
> #md5: [body, subject]
>
> - ssh
> # bi-directional flows
> #- flow
> # uni-directional flows
> #- netflow
> <--snip-->
>
> triggering an alert:
>
> architect at mana:~$ telnet 104.155.198.240 80
> Trying 104.155.198.240...
> Connected to 104.155.198.240.
> Escape character is '^]'.
>
>
> eve.json (tail)
>
> {"timestamp":"2016-07-12T18:19:57.759766-0400","flow_id":4093574929,"in_iface":"eno1","event_type":"alert","src_ip":"104.155.198.240","src_port":80,"dest_ip":"172.16.1.254","dest_port":40412,"proto":"TCP","alert":{"action":"allowed","gid":1,"signature_id":1,"rev":1,"signature":"IPREP
> High Value CnC","category":"","severity":3},"payload":"","stream":0}
> {"timestamp":"2016-07-12T18:19:58.000249-0400","event_type":"stats","stats":{"uptime":762253,"capture":{"kernel_packets":125141,"kernel_drops":0},"decoder":{"pkts":125123,"bytes":10637186,"invalid":0,"ipv4":34200,"ipv6":12173,"ethernet":125123,"raw":0,"null":0,"sll":0,"tcp":457,"udp":35489,"sctp":0,"icmpv4":43,"icmpv6":6805,"ppp":0,"pppoe":0,"gre":0,"vlan":0,"vlan_qinq":0,"teredo":0,"ipv4_in_ipv6":0,"ipv6_in_ipv6":0,"mpls":0,"avg_pkt_size":85,"max_pkt_size":1486,"erspan":0,"ipraw":{"invalid_ip_version":0},"ltnull":{"pkt_too_small":0,"unsupported_type":0}},"flow":{"memcap":0,"spare":10000,"emerg_mode_entered":0,"emerg_mode_over":0,"tcp_reuse":0,"memuse":7154304},"defrag":{"ipv4":{"fragments":0,"reassembled":0,"timeouts":0},"ipv6":{"fragments":40,"reassembled":20,"timeouts":0},"max_frag_hits":0},"stream":{"3whs_ack_in_wrong_dir":0,"3whs_async_wrong_seq":0,"3whs_right_seq_wrong_ack_evasion":0},"tcp":{"sessions":12,"ssn_memcap_drop":0,"pseudo":0,"pseudo_failed":0,"invalid_checksum":0,"no_flow":0,"syn":31,"synack":4,"rst":4,"segment_memcap_drop":0,"stream_depth_reached":0,"reassembly_gap":0,"memuse":9435072,"reassembly_memuse":12320544},"detect":{"alert":7},"flow_mgr":{"closed_pruned":0,"new_pruned":25936,"est_pruned":0},"dns":{"memuse":0,"memcap_state":0,"memcap_global":0},"http":{"memuse":0,"memcap":0}}}
> --
> SiNA
> PGP: 0x0B47D56D
>
>
> On Tue, Jul 12, 2016 at 5:52 PM, SiNA <sina.rabbani at gmail.com> wrote:
>> Thank you for your reply!
>>
>> When an alert is triggered from the IP Reputation rules, I only see 1
>> entry with even_type: alert in the log. I don't see another entry
>> which contains more details about the packet.
>> Am I missing something here? I'm cleaning up my configuration files to
>> post here shortly. Do I need to have flow logging enabled too?
>>
>> All the best,
>> Sina
>>
>>
>>
>> --
>> SiNA
>> PGP: 0x0B47D56D
>>
>>
>> On Tue, Jul 12, 2016 at 4:15 PM, Victor Julien <lists at inliniac.net> wrote:
>>> On 12-07-16 22:13, SiNA wrote:
>>>> When Suricata generates an alert based on ip reputation rules, the alert
>>>> json log doesn't include decoded application layer information. I see
>>>> the option of including the payload itself, which would require
>>>> additional processing by a third party scrip or tool. Is it possible to
>>>> configure Suricata to generste both an event and an alert in this case?
>>>
>>> IP only rules are generally inspected on the first packet of the flow
>>> only. For TCP that is normally the SYN packet, so there isn't much we
>>> can log then.
>>>
>>> Each event contains a flow_id field that you can use to correlate
>>> multiple events.
>>>
>>> --
>>> ---------------------------------------------
>>> Victor Julien
>>> http://www.inliniac.net/
>>> PGP: http://www.inliniac.net/victorjulien.asc
>>> ---------------------------------------------
>>>
>>> _______________________________________________
>>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>>> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
>>> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>> Suricata User Conference November 9-11 in Washington, DC: http://oisfevents.net
More information about the Oisf-users
mailing list