<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Franklin Gothic Book";
panose-1:2 11 5 3 2 1 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link="#0563C1" vlink="#954F72"><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'>Yes, I have seen something similar with alerting when running against pcap files. I haven’t had a chance to dig further into it yet, but I flagged it as something to come back to.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'>In addition, I have also noticed this with luajit sigatures. As a sanity check, I ran Suricata against a pcap with 1 signature. The alert fired a number of times. Then I modified the signature to call a luajit script that simply returned 1 (indicating a match). I received a fraction the number of alerts (I did this a few weeks ago, so I don’t have the output handy, but I could recreate it if necessary).<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'>My first thoughts are that this has to do with timing – the engine is killed once the end of the pcap file is reached, before all alerts have fired. But, as I said, I haven’t had a chance to confirm this in the code.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'><o:p> </o:p></span></p><div><p class=MsoNormal><b><span style='font-family:"Franklin Gothic Book",sans-serif;color:#1F497D'>________________________<o:p></o:p></span></b></p><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Franklin Gothic Book",sans-serif;color:#1F497D'>Zach Rasmor<o:p></o:p></span></b></p><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Franklin Gothic Book",sans-serif;color:#1F497D'>Email: <a href="mailto:zachary.r.rasmor@lmco.com"><span style='color:blue'>zachary.r.rasmor@lmco.com</span></a></span><b><span style='font-size:10.0pt;font-family:"Franklin Gothic Book",sans-serif;color:#1F497D'><o:p></o:p></span></b></p><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Franklin Gothic Book",sans-serif;color:#1F497D'>Office: 301.240.6116<o:p></o:p></span></p></div><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'>From:</span></b><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'> Oisf-users [mailto:oisf-users-bounces@lists.openinfosecfoundation.org] <b>On Behalf Of </b>derek_smithg@yahoo.com<br><b>Sent:</b> Monday, February 22, 2016 11:30 AM<br><b>To:</b> oisf-users@lists.openinfosecfoundation.org<br><b>Subject:</b> EXTERNAL: [Oisf-users] Inconsistent Alerting<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><div><div id="yui_3_16_0_1_1456158282000_3487"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>Hello everyone,<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3491"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'> <o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3495"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>I am new to the mailing list. Sorry for the long email, I just wanted to include enough information in case anyone has come across a similar problem. <o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3499"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'> <o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3503"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>I have been running Suricata against several pcaps with different yaml configurations and am seeing the total count of alerts change from one run to another, or even with the same yaml but run at a different time. Has anyone come across anything similar before?<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3507"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>Below are details on the rules whose counts change and the yaml files I am using. <o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3511"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'> <o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3515"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>Suricata-2.0.11<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3519"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>CentOS 7.1.1503<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3523"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'> <o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3527"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>Yaml Files:<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3531"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'> - yaml1: very out of the box yaml, except moved address-groups and other locally defined variables to an include file. <o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3535"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'> - yaml2: changed max-pending-packets from 1024 to 645000, and detect-engine.profile from medium to high.<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3539"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'> - threading1 and threading2: turned on cpu-affinity but set them to use [ ‘all’ ] cpu’s, with detect-ratio set to .5 and 1 respectively. <o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3543"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>(all are only outputting eve.json)<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3547"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'> <o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3551"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>I ran them against 3 pcaps of sizes roughly 100GB, 200GB, and 400GB, and tallied the alert counts, outputting any that were not the same across the board. <o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3555"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'> <o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3559"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>100 GB pcap<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3563"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'> sid yaml1 yaml2 th1 th2<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3567"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>2001805 : 139 137 137 137<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3571"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>12037 : 1 - - -<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3575"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>2101633 : 132 129 129 129<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3579"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'> <o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3583"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>200 GB pcap<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3587"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>2009375 : 91134 91127 91119 91122<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3591"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>2010935 : 657 657 662 657<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3595"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'> <o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3599"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>400 GB pcap<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3603"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>2001805 : 96 96 96 99<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3607"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>2010935 : 818 818 880 818<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3611"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>2010937 : 160 160 192 160<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3615"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>2101633 : 138 136 138 137<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3619"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'> <o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3623"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>They are mostly the same rules in common misbehaving. They are listed below.<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3627"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'> <o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3631"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>emerging-chat.rules:alert tcp $HOME_NET any <> $EXTERNAL_NET any (msg:"ET CHAT ICQ Message"; flow: established; content:"|2A02|"; depth: 2; content:"|000400060000|"; offset: 6; depth: 6; reference:url,doc.emergingthreats.net/2001805; classtype:policy-violation; sid:2001805; rev:5;)<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3635"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'> <o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3639"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>content-replace.rules:alert tcp $EXTERNAL_NET any -> $HOME_NET any (msg:"CONTENT-REPLACE AIM deny in-bound file transfer attempts"; flow:to_client,established; content:"*|02|"; depth:2; content:"|00 04 00 07|"; within:8; distance:4; content:"|09|F|13|CL|7F 11 D1 82 22|DEST|00|"; content:"DEST"; distance:-5; replace:"XXXX"; byte_test:2,=,0,-24,relative; classtype:policy-violation; sid:12037; rev:3;)<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3643"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'> <o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3647"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>emerging-chat.rules:alert tcp $AIM_SERVERS any -> $HOME_NET any (msg:"GPL CHAT AIM receive message"; flow:to_client; content:"*|02|"; depth:2; content:"|00 04 00 07|"; depth:4; offset:6; classtype:policy-violation; sid:2101633; rev:7;)<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3651"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'> <o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3655"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>emerging-chat.rules:alert tcp $HOME_NET any <> $EXTERNAL_NET any (msg:"ET CHAT General MSN Chat Activity"; flow: established; content:"Content-Type|3A|"; http_header; content:"application/x-msn-messenger"; http_header; reference:url,www.hypothetic.org/docs/msn/general/http_examples.php; reference:url,doc.emergingthreats.net/2009375; classtype:policy-violation; sid:2009375; rev:3;)<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3659"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'> <o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3663"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>emerging-policy.rules:alert tcp $EXTERNAL_NET any -> $HOME_NET 1433 (msg:"ET POLICY Suspicious inbound to MSSQL port 1433"; flow:to_server; flags:S; threshold: type limit, count 5, seconds 60, track by_src; reference:url,doc.emergingthreats.net/2010935; classtype:bad-unknown; sid:2010935; rev:2;)<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3667"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'> <o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3671"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>emerging-policy.rules:alert tcp $EXTERNAL_NET any -> $HOME_NET 3306 (msg:"ET POLICY Suspicious inbound to mySQL port 3306"; flow:to_server; flags:S; threshold: type limit, count 5, seconds 60, track by_src; reference:url,doc.emergingthreats.net/2010937; classtype:bad-unknown; sid:2010937; rev:2;)<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3675"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'> <o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3679"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'> <o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3683"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>This may be a different issue, but I have looked into 12037, which is very similar to 2101633 but with added replace and byte_test keywords, and think it might be a false positive. From carving out the ip’s involved with it from the pcap and running Suricata on that alone it hits that one alert about 50% of the time. I ran it once with alert-debug output and found the packet it’s supposedly alerting on and cannot find the byte pattern that would match to it. <o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3687"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'> <o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3691"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>Debug output regarding sid 12037:<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3695"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>STREAM DATA:<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3699"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>. . .<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3703"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>0030 02 00 01 00 05 00 04 47 BB 2B AC 00 0D 00 50 09 .......G .+....P.<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3707"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'> 0040 46 13 43 4C 7F 11 D1 82 22 44 45 53 54 00 00 09 F.CL.... "DEST...<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3711"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>. . . <o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3715"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>0160 06 00 04 10 02 00 01 00 0D 00 50 09 46 13 43 4C ........ ..P.F.CL<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3719"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'> 0170 7F 11 D1 82 22 44 45 53 54 00 00 09 46 13 45 4C ...."DES T...F.EL<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3723"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'> <o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3727"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>It is my understanding that byte_test is looking for 00 00, 24 bytes before 53 54 (because of content match “DEST” and “relative” keyword), but there is 00 04 in both. <o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3731"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'> <o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3735"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>Please let me know if you have any insight as to what is going on. I would greatly appreciate it.<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3739"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'> <o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3743"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>Thank you,<o:p></o:p></span></p></div><div id="yui_3_16_0_1_1456158282000_3747"><p class=MsoNormal style='background:white'><span style='font-family:"Helvetica",sans-serif;color:black'>Derek<o:p></o:p></span></p></div></div></div></body></html>