[Oisf-users] Question about file-store

["강지환"]

Hi, My name is Chi Hwan Kang.  I am trying to build an IPS using Suricata which is able to extract files.While testing file extraction, I have faced something I don't understand.  When replaying pcap that contains files, there generated files-json.log and extracted files. In the log, there are md5 values for some files, BUT NOT ALL.     // correct md5    { "id": 4, "timestamp": "09\/06\/2019-18:00:24.199874", "ipver": 4, "srcip": "100.100.200.254", "dstip": "100.100.200.1", "protocol": 6, "sp": 80, "dp": 42734, "http_uri": "\/yara-82\/CVE-2017-0072.otf", "http_host": "100.100.200.254", "http_referer": "<unknown>", "http_user_agent": "Wget\/1.19.4 (linux-gnu)", "filename": "\/yara-82\/CVE-2017-0072.otf", "magic": "OpenType font data", "state": "CLOSED", "md5": "84ce8712d1335b6c5547473543fb10f9", "stored": true, "size": 20128 }   // no md5    { "id": 5, "timestamp": "09\/06\/2019-18:00:24.213569", "ipver": 4, "srcip": "100.100.200.254", "dstip": "100.100.200.1", "protocol": 6, "sp": 80, "dp": 42736, "http_uri": "\/yara-82\/CVE-2017-0083.ttf", "http_host": "100.100.200.254", "http_referer": "<unknown>", "http_user_agent": "Wget\/1.19.4 (linux-gnu)", "filename": "\/yara-82\/CVE-2017-0083.ttf", "magic": "TrueType font data", "state": "TRUNCATED", "stored": true, "size": 102460 } I also tcpdump the replayed packets on the other side and the number of packets received matches the number of packets sent. And Suricata also generates the following logs (I am using Redis).    1567760569.851302 [0 172.18.0.1:40856] "LPUSH" "suricata" "{\"timestamp\": \"2019-09-06T18:02:49.837364+0900\", \"flow_id\": 91038813841778, \"in_iface\": \"ens785f0\", \"event_type\": \"alert\", \"src_ip\": \"100.100.200.1\", \"src_port\": 42772, \"dest_ip\": \"100.100.200.254\", \"dest_port\": 80, \"proto\": \"TCP\", \"metadata\": {\"flowbits\": [\"tcp.retransmission.alerted\"], \"flowints\": {\"tcp.retransmission.count\": 1268}}, \"community_id\": \"1:rpxRWVlwTXUCiNKKnUEcbnP3vt8=\", \"alert\": {\"action\": \"allowed\", \"gid\": 1, \"signature_id\": 3210044, \"rev\": 2, \"signature\": \"SURICATA STREAM Packet with invalid timestamp\", \"category\": \"Generic Protocol Command Decode\", \"severity\": 3}, \"app_proto\": \"http\", \"flow\": {\"pkts_toserver\": 706, \"pkts_toclient\": 1273, \"bytes_toserver\": 46946, \"bytes_toclient\": 1919664, \"start\": \"2019-09-06T18:00:24.518514+0900\"}, \"payload\": \"\", \"stream\": 0, \"host\": \"localhost.localdomain\"}"1567760569.851539 [0 172.18.0.1:40856] "LPUSH" "suricata" "{\"timestamp\": \"2019-09-06T18:02:49.838029+0900\", \"flow_id\": 91038813841778, \"in_iface\": \"ens785f0\", \"event_type\": \"alert\", \"src_ip\": \"100.100.200.254\", \"src_port\": 80, \"dest_ip\": \"100.100.200.1\", \"dest_port\": 42772, \"proto\": \"TCP\", \"metadata\": {\"flowbits\": [\"tcp.retransmission.alerted\"], \"flowints\": {\"tcp.retransmission.count\": 1268}}, \"community_id\": \"1:rpxRWVlwTXUCiNKKnUEcbnP3vt8=\", \"alert\": {\"action\": \"allowed\", \"gid\": 1, \"signature_id\": 3210016, \"rev\": 2, \"signature\": \"SURICATA STREAM CLOSEWAIT FIN out of window\", \"category\": \"Generic Protocol Command Decode\", \"severity\": 3}, \"app_proto\": \"http\", \"flow\": {\"pkts_toserver\": 707, \"pkts_toclient\": 1274, \"bytes_toserver\": 47012, \"bytes_toclient\": 1919730, \"start\": \"2019-09-06T18:00:24.518514+0900\"}, \"payload\": \"\", \"stream\": 0, \"host\": \"localhost.localdomain\"}" My questions are:    - Where can be the point that leads to md5 failure (or "state": "TRUNCATED")? Is it tcprelay's fault? NIC's fault? What could be the cause?   - Are "SURICATA STREAM Packet with invalid timestamp" and "SURICATA STREAM CLOSEWAIT FIN out of window" related to md5 failure?suricata.yaml attached. Thank you very much Chi Kang                  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20190906/316ccb4a/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: suricata.yaml
Type: application/octet-stream
Size: 66341 bytes
Desc: not available
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20190906/316ccb4a/attachment-0001.obj>