<div><div dir="auto">There might be some truncated files in that capture, but I’d expect anything with a proper file magic signature of a PDF to be extracted. What I wasn’t expecting were the other non PDF files to be extracted. I can certainly do more testing with other PDF download captures, but I suspect the same behavior will be present. </div></div><div><br><div class="gmail_quote"><div dir="ltr">On Wed, Aug 22, 2018 at 10:56 PM Peter Manev <<a href="mailto:petermanev@gmail.com">petermanev@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On Tue, Aug 21, 2018 at 3:06 PM Carl Rotenan <<a href="mailto:carlrotenan@gmail.com" target="_blank">carlrotenan@gmail.com</a>> wrote:<br>
><br>
> Yes, on both versions of filestore.<br>
><br>
<br>
I had a second run/look of the pcap and i have a question -<br>
What/which PDF file do you expect to extract form that capture.<br>
I mean when you view it with wireshark there are some files wit names<br>
that have pdf extensions - but they all are 500-2500 bytes. Is that<br>
expected?<br>
<br>
Thank you<br>
<br>
> On Tue, Aug 21, 2018 at 5:03 PM, Peter Manev <<a href="mailto:petermanev@gmail.com" target="_blank">petermanev@gmail.com</a>> wrote:<br>
>><br>
>> On Tue, Aug 21, 2018 at 2:57 PM Carl Rotenan <<a href="mailto:carlrotenan@gmail.com" target="_blank">carlrotenan@gmail.com</a>> wrote:<br>
>> ><br>
>> > I'm getting the same behavior even if I created a Magic file that only knows about PDF files.<br>
>> > I'm seeing this behavior on both stable and the RC version.<br>
>> ><br>
>> > alert http any any -> any any (msg:"FILE store all"; filemagic:"PDF"; filestore; sid:1; rev:1;)<br>
>> ><br>
>> > If I do just a filestore all files are extracted.<br>
>><br>
>> Ok thank you for the feedback - is this with filestore v2 as well?<br>
>> (in my tests it was, i will open a bug report following that as well)<br>
>><br>
>> Thanks<br>
>><br>
>> ><br>
>> ><br>
>> ><br>
>> > On Mon, Aug 20, 2018 at 8:17 PM, Peter Manev <<a href="mailto:petermanev@gmail.com" target="_blank">petermanev@gmail.com</a>> wrote:<br>
>> >><br>
>> >> On Thu, Aug 16, 2018 at 11:10 AM Carl Rotenan <<a href="mailto:carlrotenan@gmail.com" target="_blank">carlrotenan@gmail.com</a>> wrote:<br>
>> >> ><br>
>> >> > It appears that if the HTTP info (URI, HOST, REFERER, USER AGENT) aren't known the file gets stored.<br>
>> >> ><br>
>> >> > The info below comes from the file meta data files that are created for each capture.<br>
>> >> ><br>
>> >> > foo.cap<br>
>> >> ><br>
>> >> > magic: HTML document, UTF-8 Unicode text, with very long lines, with CRLF, LF line terminators<br>
>> >> > app proto: http<br>
>> >> > http uri: <unknown><br>
>> >> > http host: <unknown><br>
>> >> > http referer: <unknown><br>
>> >> > http user agent: <unknown><br>
>> >> ><br>
>> >> > magic: PDF document, version 1.4<br>
>> >> > app proto: http<br>
>> >> > http uri: /files/documents/2018/03/12/dor-2017-inc-sch-hc.pdf<br>
>> >> > http host: <a href="http://www.mass.gov" rel="noreferrer" target="_blank">www.mass.gov</a><br>
>> >> > http referer: <a href="https://www.mass.gov/lists/2017-massachusetts-personal-income-tax-forms-and-instructions" rel="noreferrer" target="_blank">https://www.mass.gov/lists/2017-massachusetts-personal-income-tax-forms-and-instructions</a><br>
>> >> > http user agent: Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko<br>
>> >> ><br>
>> >> > magic: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators<br>
>> >> > app proto: http<br>
>> >> > http uri: <unknown><br>
>> >> > http host: <unknown><br>
>> >> > http referer: <unknown><br>
>> >> > http user agent: <unknown><br>
>> >> ><br>
>> >> > magic: JPEG image data, JFIF standard 1.01<br>
>> >> > app proto: http<br>
>> >> > http uri: <unknown><br>
>> >> > http host: <unknown><br>
>> >> > http referer: <unknown><br>
>> >> > http user agent: <unknown><br>
>> >> ><br>
>> >> > magic: PDF document, version 1.6<br>
>> >> > app proto: http<br>
>> >> > http uri: /files/documents/2018/02/07/dor-2017-inc-sch-xy.pdf<br>
>> >> > http host: <a href="http://www.mass.gov" rel="noreferrer" target="_blank">www.mass.gov</a><br>
>> >> > http referer: <a href="https://www.mass.gov/lists/2017-massachusetts-personal-income-tax-forms-and-instructions" rel="noreferrer" target="_blank">https://www.mass.gov/lists/2017-massachusetts-personal-income-tax-forms-and-instructions</a><br>
>> >> > http user agent: Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko<br>
>> >> ><br>
>> >> > magic: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators<br>
>> >> > app proto: http<br>
>> >> > http uri: <unknown><br>
>> >> > http host: <unknown><br>
>> >> > http referer: <unknown><br>
>> >> > http user agent: <unknown><br>
>> >> ><br>
>> >> > magic: JPEG image data, JFIF standard 1.01<br>
>> >> > app proto: http<br>
>> >> > http uri: <unknown><br>
>> >> > http host: <unknown><br>
>> >> > http referer: <unknown><br>
>> >> > http user agent: <unknown><br>
>> >> ><br>
>> >> > magic: HTML document, ASCII text, with very long lines<br>
>> >> > app proto: http<br>
>> >> > http uri: <unknown><br>
>> >> > http host: <unknown><br>
>> >> > http referer: <unknown><br>
>> >> > http user agent: <unknown><br>
>> >> ><br>
>> >> > magic: HTML document, ASCII text, with very long lines<br>
>> >> > app proto: http<br>
>> >> > http uri: <unknown><br>
>> >> > http host: <unknown><br>
>> >> > http referer: <unknown><br>
>> >> > http user agent: <unknown><br>
>> >> ><br>
>> >> > magic: UTF-8 Unicode text, with very long lines<br>
>> >> > app proto: http<br>
>> >> > http uri: <unknown><br>
>> >> > http host: <unknown><br>
>> >> > http referer: <unknown><br>
>> >> > http user agent: <unknown><br>
>> >> ><br>
>> >> ><br>
>> >> > boo.cap<br>
>> >> ><br>
>> >> ><br>
>> >> > magic: PNG image data, 3996 x 80, 8-bit colormap, non-interlaced<br>
>> >> > app proto: http<br>
>> >> > http uri: <unknown><br>
>> >> > http host: <unknown><br>
>> >> > http referer: <unknown><br>
>> >> > http user agent: <unknown><br>
>> >> ><br>
>> >> > magic: PNG image data, 492 x 400, 8-bit/color RGB, non-interlaced<br>
>> >> > app proto: http<br>
>> >> > http uri: <unknown><br>
>> >> > http host: <unknown><br>
>> >> > http referer: <unknown><br>
>> >> > http user agent: <unknown><br>
>> >> ><br>
>> >> > magic: HTML document, UTF-8 Unicode text, with very long lines, with no line terminators<br>
>> >> > app proto: http<br>
>> >> > http uri: <unknown><br>
>> >> > http host: <unknown><br>
>> >> > http referer: <unknown><br>
>> >> > http user agent: <unknown><br>
>> >> ><br>
>> >> > magic: data<br>
>> >> > app proto: http<br>
>> >> > http uri: <unknown><br>
>> >> > http host: <unknown><br>
>> >> > http referer: <unknown><br>
>> >> > http user agent: <unknown><br>
>> >> ><br>
>> >> > magic: PNG image data, 310 x 440, 8-bit colormap, non-interlaced<br>
>> >> > app proto: http<br>
>> >> > http uri: <unknown><br>
>> >> > http host: <unknown><br>
>> >> > http referer: <unknown><br>
>> >> > http user agent: <unknown><br>
>> >> ><br>
>> >> > magic: ASCII text, with very long lines<br>
>> >> > app proto: http<br>
>> >> > http uri: <unknown><br>
>> >> > http host: <unknown><br>
>> >> > http referer: <unknown><br>
>> >> > http user agent: <unknown><br>
>> >> ><br>
>> >> > magic: ASCII text, with very long lines<br>
>> >> > app proto: http<br>
>> >> > http uri: <unknown><br>
>> >> > http host: <unknown><br>
>> >> > http referer: <unknown><br>
>> >> > http user agent: <unknown><br>
>> >> ><br>
>> >> > magic: PNG image data, 320 x 198, 8-bit colormap, non-interlaced<br>
>> >> > app proto: http<br>
>> >> > http uri: <unknown><br>
>> >> > http host: <unknown><br>
>> >> > http referer: <unknown><br>
>> >> > http user agent: <unknown><br>
>> >> ><br>
>> >> > magic: PDF document, version 1.4<br>
>> >> > app proto: http<br>
>> >> > http uri: /archive3/GflUt00Q30KF03YzCLl43rm2po76/D3400UM_SG(En)02.pdf<br>
>> >> > http host: <a href="http://download.nikonimglib.com" rel="noreferrer" target="_blank">download.nikonimglib.com</a><br>
>> >> > http referer: <a href="http://downloadcenter.nikonimglib.com/en/products/330/D3400.html" rel="noreferrer" target="_blank">http://downloadcenter.nikonimglib.com/en/products/330/D3400.html</a><br>
>> >> > http user agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36<br>
>> >> ><br>
>> >> > magic: PDF document, version 1.3<br>
>> >> > app proto: http<br>
>> >> > http uri: /biassets/bi/4128311.pdf<br>
>> >> > http host: <a href="http://www.lego.com" rel="noreferrer" target="_blank">www.lego.com</a><br>
>> >> > http referer: <unknown><br>
>> >> > http user agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36<br>
>> >> ><br>
>> >> > magic: PDF document, version 1.3<br>
>> >> > app proto: http<br>
>> >> > http uri: /biassets/bi/4128312.pdf<br>
>> >> > http host: <a href="http://www.lego.com" rel="noreferrer" target="_blank">www.lego.com</a><br>
>> >> > http referer: <unknown><br>
>> >> > http user agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36<br>
>> >> ><br>
>> >> > magic: JPEG image data, EXIF standard<br>
>> >> > app proto: http<br>
>> >> > http uri: <unknown><br>
>> >> > http host: <unknown><br>
>> >> > http referer: <unknown><br>
>> >> > http user agent: <unknown><br>
>> >> ><br>
>> >> > magic: PDF document, version 1.3<br>
>> >> > app proto: http<br>
>> >> > http uri: /biassets/bi/4132659.pdf<br>
>> >> > http host: <a href="http://www.lego.com" rel="noreferrer" target="_blank">www.lego.com</a><br>
>> >> > http referer: <unknown><br>
>> >> > http user agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36<br>
>> >> ><br>
>> >> > magic: UTF-8 Unicode text, with very long lines, with no line terminators<br>
>> >> > app proto: http<br>
>> >> > http uri: <unknown><br>
>> >> > http host: <unknown><br>
>> >> > http referer: <unknown><br>
>> >> > http user agent: <unknown><br>
>> >> ><br>
>> >> ><br>
>> >><br>
>> >><br>
>> >> I tried the latest gitmaster with filestore v2 - I observed the<br>
>> >> following - if you could confirm on your set up please as well with<br>
>> >> 4.1.0-rc1.<br>
>> >> If i use<br>
>> >> alert http any any -> any any (msg:"FILE magic"; filemagic:"PDF";<br>
>> >> filestore; sid:0; rev:1;)<br>
>> >> I get results like you with the pcap provided foo.pcap (partial html<br>
>> >> files present in the download)<br>
>> >><br>
>> >> If i use<br>
>> >> alert http any any -> any any (msg:"FILE magic"; filemagic:"PDF<br>
>> >> document"; filestore; sid:0; rev:1;)<br>
>> >> The only diff is filemagic:"PDF document" - i get 0 alerts and 0<br>
>> >> partial or full files stored.<br>
>> >><br>
>> >> Thank you<br>
>> >><br>
>> >> --<br>
>> >> Regards,<br>
>> >> Peter Manev<br>
>> ><br>
>> ><br>
>><br>
>><br>
>> --<br>
>> Regards,<br>
>> Peter Manev<br>
><br>
><br>
<br>
<br>
-- <br>
Regards,<br>
Peter Manev<br>
</blockquote></div></div>