<div dir="ltr">Yes, on both versions of filestore.</div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 21, 2018 at 5:03 PM, Peter Manev <span dir="ltr"><<a href="mailto:petermanev@gmail.com" target="_blank">petermanev@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Tue, Aug 21, 2018 at 2:57 PM Carl Rotenan <<a href="mailto:carlrotenan@gmail.com">carlrotenan@gmail.com</a>> wrote:<br>
><br>
> I'm getting the same behavior even if I created a Magic file that only knows about PDF files.<br>
> I'm seeing this behavior on both stable and the RC version.<br>
><br>
> alert http any any -> any any (msg:"FILE store all"; filemagic:"PDF"; filestore; sid:1; rev:1;)<br>
><br>
> If I do just a filestore all files are extracted.<br>
<br>
</span>Ok thank you for the feedback - is this with filestore v2 as well?<br>
(in my tests it was, i will open a bug report following that as well)<br>
<br>
Thanks<br>
<div class="HOEnZb"><div class="h5"><br>
><br>
><br>
><br>
> On Mon, Aug 20, 2018 at 8:17 PM, Peter Manev <<a href="mailto:petermanev@gmail.com">petermanev@gmail.com</a>> wrote:<br>
>><br>
>> On Thu, Aug 16, 2018 at 11:10 AM Carl Rotenan <<a href="mailto:carlrotenan@gmail.com">carlrotenan@gmail.com</a>> wrote:<br>
>> ><br>
>> > It appears that if the HTTP info (URI, HOST, REFERER, USER AGENT) aren't known the file gets stored.<br>
>> ><br>
>> > The info below comes from the file meta data files that are created for each capture.<br>
>> ><br>
>> > foo.cap<br>
>> ><br>
>> > magic: HTML document, UTF-8 Unicode text, with very long lines, with CRLF, LF line terminators<br>
>> > app proto: http<br>
>> > http uri: <unknown><br>
>> > http host: <unknown><br>
>> > http referer: <unknown><br>
>> > http user agent: <unknown><br>
>> ><br>
>> > magic: PDF document, version 1.4<br>
>> > app proto: http<br>
>> > http uri: /files/documents/2018/03/12/<wbr>dor-2017-inc-sch-hc.pdf<br>
>> > http host: <a href="http://www.mass.gov" rel="noreferrer" target="_blank">www.mass.gov</a><br>
>> > http referer: <a href="https://www.mass.gov/lists/2017-massachusetts-personal-income-tax-forms-and-instructions" rel="noreferrer" target="_blank">https://www.mass.gov/lists/<wbr>2017-massachusetts-personal-<wbr>income-tax-forms-and-<wbr>instructions</a><br>
>> > http user agent: Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko<br>
>> ><br>
>> > magic: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators<br>
>> > app proto: http<br>
>> > http uri: <unknown><br>
>> > http host: <unknown><br>
>> > http referer: <unknown><br>
>> > http user agent: <unknown><br>
>> ><br>
>> > magic: JPEG image data, JFIF standard 1.01<br>
>> > app proto: http<br>
>> > http uri: <unknown><br>
>> > http host: <unknown><br>
>> > http referer: <unknown><br>
>> > http user agent: <unknown><br>
>> ><br>
>> > magic: PDF document, version 1.6<br>
>> > app proto: http<br>
>> > http uri: /files/documents/2018/02/07/<wbr>dor-2017-inc-sch-xy.pdf<br>
>> > http host: <a href="http://www.mass.gov" rel="noreferrer" target="_blank">www.mass.gov</a><br>
>> > http referer: <a href="https://www.mass.gov/lists/2017-massachusetts-personal-income-tax-forms-and-instructions" rel="noreferrer" target="_blank">https://www.mass.gov/lists/<wbr>2017-massachusetts-personal-<wbr>income-tax-forms-and-<wbr>instructions</a><br>
>> > http user agent: Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko<br>
>> ><br>
>> > magic: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators<br>
>> > app proto: http<br>
>> > http uri: <unknown><br>
>> > http host: <unknown><br>
>> > http referer: <unknown><br>
>> > http user agent: <unknown><br>
>> ><br>
>> > magic: JPEG image data, JFIF standard 1.01<br>
>> > app proto: http<br>
>> > http uri: <unknown><br>
>> > http host: <unknown><br>
>> > http referer: <unknown><br>
>> > http user agent: <unknown><br>
>> ><br>
>> > magic: HTML document, ASCII text, with very long lines<br>
>> > app proto: http<br>
>> > http uri: <unknown><br>
>> > http host: <unknown><br>
>> > http referer: <unknown><br>
>> > http user agent: <unknown><br>
>> ><br>
>> > magic: HTML document, ASCII text, with very long lines<br>
>> > app proto: http<br>
>> > http uri: <unknown><br>
>> > http host: <unknown><br>
>> > http referer: <unknown><br>
>> > http user agent: <unknown><br>
>> ><br>
>> > magic: UTF-8 Unicode text, with very long lines<br>
>> > app proto: http<br>
>> > http uri: <unknown><br>
>> > http host: <unknown><br>
>> > http referer: <unknown><br>
>> > http user agent: <unknown><br>
>> ><br>
>> ><br>
>> > boo.cap<br>
>> ><br>
>> ><br>
>> > magic: PNG image data, 3996 x 80, 8-bit colormap, non-interlaced<br>
>> > app proto: http<br>
>> > http uri: <unknown><br>
>> > http host: <unknown><br>
>> > http referer: <unknown><br>
>> > http user agent: <unknown><br>
>> ><br>
>> > magic: PNG image data, 492 x 400, 8-bit/color RGB, non-interlaced<br>
>> > app proto: http<br>
>> > http uri: <unknown><br>
>> > http host: <unknown><br>
>> > http referer: <unknown><br>
>> > http user agent: <unknown><br>
>> ><br>
>> > magic: HTML document, UTF-8 Unicode text, with very long lines, with no line terminators<br>
>> > app proto: http<br>
>> > http uri: <unknown><br>
>> > http host: <unknown><br>
>> > http referer: <unknown><br>
>> > http user agent: <unknown><br>
>> ><br>
>> > magic: data<br>
>> > app proto: http<br>
>> > http uri: <unknown><br>
>> > http host: <unknown><br>
>> > http referer: <unknown><br>
>> > http user agent: <unknown><br>
>> ><br>
>> > magic: PNG image data, 310 x 440, 8-bit colormap, non-interlaced<br>
>> > app proto: http<br>
>> > http uri: <unknown><br>
>> > http host: <unknown><br>
>> > http referer: <unknown><br>
>> > http user agent: <unknown><br>
>> ><br>
>> > magic: ASCII text, with very long lines<br>
>> > app proto: http<br>
>> > http uri: <unknown><br>
>> > http host: <unknown><br>
>> > http referer: <unknown><br>
>> > http user agent: <unknown><br>
>> ><br>
>> > magic: ASCII text, with very long lines<br>
>> > app proto: http<br>
>> > http uri: <unknown><br>
>> > http host: <unknown><br>
>> > http referer: <unknown><br>
>> > http user agent: <unknown><br>
>> ><br>
>> > magic: PNG image data, 320 x 198, 8-bit colormap, non-interlaced<br>
>> > app proto: http<br>
>> > http uri: <unknown><br>
>> > http host: <unknown><br>
>> > http referer: <unknown><br>
>> > http user agent: <unknown><br>
>> ><br>
>> > magic: PDF document, version 1.4<br>
>> > app proto: http<br>
>> > http uri: /archive3/<wbr>GflUt00Q30KF03YzCLl43rm2po76/<wbr>D3400UM_SG(En)02.pdf<br>
>> > http host: <a href="http://download.nikonimglib.com" rel="noreferrer" target="_blank">download.nikonimglib.com</a><br>
>> > http referer: <a href="http://downloadcenter.nikonimglib.com/en/products/330/D3400.html" rel="noreferrer" target="_blank">http://downloadcenter.<wbr>nikonimglib.com/en/products/<wbr>330/D3400.html</a><br>
>> > http user agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36<br>
>> ><br>
>> > magic: PDF document, version 1.3<br>
>> > app proto: http<br>
>> > http uri: /biassets/bi/4128311.pdf<br>
>> > http host: <a href="http://www.lego.com" rel="noreferrer" target="_blank">www.lego.com</a><br>
>> > http referer: <unknown><br>
>> > http user agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36<br>
>> ><br>
>> > magic: PDF document, version 1.3<br>
>> > app proto: http<br>
>> > http uri: /biassets/bi/4128312.pdf<br>
>> > http host: <a href="http://www.lego.com" rel="noreferrer" target="_blank">www.lego.com</a><br>
>> > http referer: <unknown><br>
>> > http user agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36<br>
>> ><br>
>> > magic: JPEG image data, EXIF standard<br>
>> > app proto: http<br>
>> > http uri: <unknown><br>
>> > http host: <unknown><br>
>> > http referer: <unknown><br>
>> > http user agent: <unknown><br>
>> ><br>
>> > magic: PDF document, version 1.3<br>
>> > app proto: http<br>
>> > http uri: /biassets/bi/4132659.pdf<br>
>> > http host: <a href="http://www.lego.com" rel="noreferrer" target="_blank">www.lego.com</a><br>
>> > http referer: <unknown><br>
>> > http user agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36<br>
>> ><br>
>> > magic: UTF-8 Unicode text, with very long lines, with no line terminators<br>
>> > app proto: http<br>
>> > http uri: <unknown><br>
>> > http host: <unknown><br>
>> > http referer: <unknown><br>
>> > http user agent: <unknown><br>
>> ><br>
>> ><br>
>><br>
>><br>
>> I tried the latest gitmaster with filestore v2 - I observed the<br>
>> following - if you could confirm on your set up please as well with<br>
>> 4.1.0-rc1.<br>
>> If i use<br>
>> alert http any any -> any any (msg:"FILE magic"; filemagic:"PDF";<br>
>> filestore; sid:0; rev:1;)<br>
>> I get results like you with the pcap provided foo.pcap (partial html<br>
>> files present in the download)<br>
>><br>
>> If i use<br>
>> alert http any any -> any any (msg:"FILE magic"; filemagic:"PDF<br>
>> document"; filestore; sid:0; rev:1;)<br>
>> The only diff is filemagic:"PDF document" - i get 0 alerts and 0<br>
>> partial or full files stored.<br>
>><br>
>> Thank you<br>
>><br>
>> --<br>
>> Regards,<br>
>> Peter Manev<br>
><br>
><br>
<br>
<br>
</div></div><span class="HOEnZb"><font color="#888888">-- <br>
Regards,<br>
Peter Manev<br>
</font></span></blockquote></div><br></div>