<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:#2E74B5;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Arial",sans-serif;
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Arial",sans-serif;
color:windowtext;}
span.EmailStyle20
{mso-style-type:personal-compose;
font-family:"Arial",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:2.0cm 42.5pt 2.0cm 3.0cm;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="RU" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">Suricata version: 4.1.3<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">Hello Suricata support,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">My name is Georgy, with my team we works on integrating threat intelligence hash feeds with Suricata using rules “filemd5” option.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">Our goal is to receive alerts when files, which md5s we consider as dangerous, passes through Suricata. To achieve this goal we does the following:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">1. Create a rule:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"TCP: FILE MD5 Found"; filemd5:md5_list.txt; filestore; sid:11111112; rev:1;)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">2. Create a file with set of hashes<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">1276481102f218c981e0324180bafd9f</span><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">f18208f33ca9f847dd2e348117e3bc54<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">cac7b533ba7abfb8591bb6a3f7a95eab<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">3. Include the rule to suricata.yaml and run Suricata<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">While testing the detection files using MD5-sum ('filemd5' keyword in rules) I faced to not clear behavior of Suricata: It looks like Suricata can’t
calculate the MD5-sum of files with size about 50 MB and more. In that case the string in event-json.log looks like:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">{ "timestamp": "05\/14\/2019-17:22:50.384309",<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">"ipver": 4,<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">"srcip": "10.16.159.190",<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">"dstip": "10.65.67.147",<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">"protocol": 6,<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">"sp": 80,<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">"dp": 41430,<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">"http_uri":<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">"\/dev\/INC000010408298\/test_file50mb.img",<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">"http_host": "ponybuntu.avp.ru",<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">"http_referer": "<unknown>",<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">"http_user_agent": "Wget\/1.14 (linux-gnu)",<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">"filename": "\/dev\/INC000010408298\/test_file50mb.img",<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">"state": "TRUNCATED",<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">"stored": false,<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">"size": 1013331 }<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">At the same time, in case of a small files everything works well:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">{ "id": 1,<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">"timestamp": "05\/14\/2019-17:22:42.507823",<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">"ipver": 4,<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">"srcip": "10.16.159.190",<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">"dstip": "10.65.67.147",<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">"protocol": 6,<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">"sp": 80,<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">"dp": 41428,<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">"http_uri": "\/dev\/INC000010408298\/test_file10kb.img",<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">"http_host": "ponybuntu.avp.ru",<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">"http_referer": "<unknown>",<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">"http_user_agent": "Wget\/1.14 (linux-gnu)",<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">"filename": "\/dev\/INC000010408298\/test_file10kb.img",<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">"state": "CLOSED",<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">"md5": "1276481102f218c981e0324180bafd9f",<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">"stored": true,<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:1.0cm"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">"size": 10240 }<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">I tried to understand why Suricata marks large files as Truncated. I followed Self-help diagram
<a href="https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Self_Help_Diagrams">
https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Self_Help_Diagrams</a>, but it isn’t help me. So, I turned on debug logging and I found that according to
<a href="https://github.com/OISF/suricata/blob/master/src/stream-tcp-reassemble.c">
https://github.com/OISF/suricata/blob/master/src/stream-tcp-reassemble.c</a> the while-loop in ReassembleUpdateAppLayer may breaks if CheckGap function returns “false”, and it seems to be so in case of Suricata finds a GAP.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">In that case Suricata stops to reassemble the tcp stream and doesn’t calculate the MD5-sum. It is proofed by debug logs – after CheckGap function
was called, file assembling stopped and its size will not be increased anymore. Nevertheless on a client side file loads fully and correct.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">Could you please give me any suggestions how to avoid such behavior and force Suricata calculate MD5-sums in large files with GAPs? Or this behavior
is by design and should not be fixed?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">Links:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext"><a href="https://box.kaspersky.com/f/a54978a4b2924b4eb2c9/?dl=1">Pcap-file</a><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext"><a href="https://box.kaspersky.com/f/12ef7ee147744ce592b7/?dl=1">Debug-log (nohup.out)</a><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext"><a href="https://box.kaspersky.com/f/07cb539222fb4e66b870/?dl=1">suricata.yaml</a><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext">Thank you in advance!<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext;mso-fareast-language:RU">Georgy Varlamov</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:windowtext;mso-fareast-language:RU">
<b>|</b> System analyst<b> |</b> Unix <b>| </b>Kaspersky Lab</span><span lang="EN-US" style="font-size:8.0pt;font-family:"Arial",sans-serif;color:windowtext;mso-fareast-language:RU"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:windowtext;mso-fareast-language:RU"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Arial",sans-serif;color:windowtext;mso-fareast-language:RU">Office: +7 495 797 87 00 x3601 | Mobile: +7 917 551 89 92 |
<a href="mailto:Georgy.Varlamov@kaspersky.com"><span style="color:blue">Georgy.Varlamov@kaspersky.com</span></a><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Arial",sans-serif;color:windowtext;mso-fareast-language:RU">39A/3 Leningradskoe Shosse, Moscow, 125212, Russia |
</span><span style="font-size:8.0pt;color:windowtext;mso-fareast-language:RU"><a href="http://www.kaspersky.com/"><span lang="EN-US" style="font-family:"Arial",sans-serif;color:blue">www.kaspersky.com</span></a></span><span lang="EN-US" style="font-size:8.0pt;color:windowtext;mso-fareast-language:RU">
| </span><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#1F497D;mso-fareast-language:RU"><a href="https://www.kaspersky.com/blog/"><span lang="EN-US" style="color:blue">https://www.kaspersky.com/blog/</span></a></span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D;mso-fareast-language:RU"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
</body>
</html>