[Oisf-users] filestore version 2
Peter Manev
petermanev at gmail.com
Fri Aug 24 23:25:18 UTC 2018
On Wed, Aug 22, 2018 at 9:08 PM Carl Rotenan <carlrotenan at gmail.com> wrote:
>
> There might be some truncated files in that capture, but I’d expect anything with a proper file magic signature of a PDF to be extracted. What I wasn’t expecting were the other non PDF files to be extracted. I can certainly do more testing with other PDF download captures, but I suspect the same behavior will be present.
>
Took another look.
(using the foo.cap previously provided in this thread)
1)
Using 4.1.0-dev (rev 1f4cd75f) with filestorev2 and having
fileextraction unconditionally enabled (
https://github.com/OISF/suricata/blob/master/suricata.yaml.in#L443
un-commented )
I get the 2 PDFs -
locallog/filestore/41/41bb5056d7760a903bb2b5462fe7480aeb3d34cf15d0299195795b6194bcbaf1:
PDF document, version 1.6
locallog/filestore/c6/c6f1db059595d3ff29e58129adf47f94c0d55d0aa3efa26cecb24d21c8c20ffa:
PDF document, version 1.4
root at DonPedro:/home/pevma/Work/Suricata/QA/tmp2# ls -lh
locallog/filestore/41/41bb5056d7760a903bb2b5462fe7480aeb3d34cf15d0299195795b6194bcbaf1
-rw-r--r-- 1 root root 294K Aug 24 16:54
locallog/filestore/41/41bb5056d7760a903bb2b5462fe7480aeb3d34cf15d0299195795b6194bcbaf1
root at DonPedro:/home/pevma/Work/Suricata/QA/tmp2# ls -lh
locallog/filestore/c6/c6f1db059595d3ff29e58129adf47f94c0d55d0aa3efa26cecb24d21c8c20ffa
-rw-r--r-- 1 root root 94K Aug 24 16:54
locallog/filestore/c6/c6f1db059595d3ff29e58129adf47f94c0d55d0aa3efa26cecb24d21c8c20ffa
2)
Disabled fileextraction unconditionally (
https://github.com/OISF/suricata/blob/master/suricata.yaml.in#L443
commented back )
and using only this rule -
alert http any any -> any any (msg:"FILE magic"; filemagic:"PDF
document"; filestore; sid:777; rev:1;)
i get no PDF fiels extracted. (although i should)
Using only this rule however -
alert http any any -> any any (msg:"FILE magic"; filemagic:"PDF";
filestore; sid:666; rev:1;)
I get the two PDFs extracted.
so it seems the only difference is filemagic:"PDF document" and
filemagic:"PDF". (it didnt use to be like that before - you could
just specify filemagic:"PDF document" and that was working as
expected)
Could you please open a bug report if you dont mind with all detailed
info of how to reproduce and the pcap.
> On Wed, Aug 22, 2018 at 10:56 PM Peter Manev <petermanev at gmail.com> wrote:
>>
>> On Tue, Aug 21, 2018 at 3:06 PM Carl Rotenan <carlrotenan at gmail.com> wrote:
>> >
>> > Yes, on both versions of filestore.
>> >
>>
>> I had a second run/look of the pcap and i have a question -
>> What/which PDF file do you expect to extract form that capture.
>> I mean when you view it with wireshark there are some files wit names
>> that have pdf extensions - but they all are 500-2500 bytes. Is that
>> expected?
>>
>> Thank you
>>
>> > On Tue, Aug 21, 2018 at 5:03 PM, Peter Manev <petermanev at gmail.com> wrote:
>> >>
>> >> On Tue, Aug 21, 2018 at 2:57 PM Carl Rotenan <carlrotenan at gmail.com> wrote:
>> >> >
>> >> > I'm getting the same behavior even if I created a Magic file that only knows about PDF files.
>> >> > I'm seeing this behavior on both stable and the RC version.
>> >> >
>> >> > alert http any any -> any any (msg:"FILE store all"; filemagic:"PDF"; filestore; sid:1; rev:1;)
>> >> >
>> >> > If I do just a filestore all files are extracted.
>> >>
>> >> Ok thank you for the feedback - is this with filestore v2 as well?
>> >> (in my tests it was, i will open a bug report following that as well)
>> >>
>> >> Thanks
>> >>
>> >> >
>> >> >
>> >> >
>> >> > On Mon, Aug 20, 2018 at 8:17 PM, Peter Manev <petermanev at gmail.com> wrote:
>> >> >>
>> >> >> On Thu, Aug 16, 2018 at 11:10 AM Carl Rotenan <carlrotenan at gmail.com> wrote:
>> >> >> >
>> >> >> > It appears that if the HTTP info (URI, HOST, REFERER, USER AGENT) aren't known the file gets stored.
>> >> >> >
>> >> >> > The info below comes from the file meta data files that are created for each capture.
>> >> >> >
>> >> >> > foo.cap
>> >> >> >
>> >> >> > magic: HTML document, UTF-8 Unicode text, with very long lines, with CRLF, LF line terminators
>> >> >> > app proto: http
>> >> >> > http uri: <unknown>
>> >> >> > http host: <unknown>
>> >> >> > http referer: <unknown>
>> >> >> > http user agent: <unknown>
>> >> >> >
>> >> >> > magic: PDF document, version 1.4
>> >> >> > app proto: http
>> >> >> > http uri: /files/documents/2018/03/12/dor-2017-inc-sch-hc.pdf
>> >> >> > http host: www.mass.gov
>> >> >> > http referer: https://www.mass.gov/lists/2017-massachusetts-personal-income-tax-forms-and-instructions
>> >> >> > http user agent: Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko
>> >> >> >
>> >> >> > magic: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators
>> >> >> > app proto: http
>> >> >> > http uri: <unknown>
>> >> >> > http host: <unknown>
>> >> >> > http referer: <unknown>
>> >> >> > http user agent: <unknown>
>> >> >> >
>> >> >> > magic: JPEG image data, JFIF standard 1.01
>> >> >> > app proto: http
>> >> >> > http uri: <unknown>
>> >> >> > http host: <unknown>
>> >> >> > http referer: <unknown>
>> >> >> > http user agent: <unknown>
>> >> >> >
>> >> >> > magic: PDF document, version 1.6
>> >> >> > app proto: http
>> >> >> > http uri: /files/documents/2018/02/07/dor-2017-inc-sch-xy.pdf
>> >> >> > http host: www.mass.gov
>> >> >> > http referer: https://www.mass.gov/lists/2017-massachusetts-personal-income-tax-forms-and-instructions
>> >> >> > http user agent: Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko
>> >> >> >
>> >> >> > magic: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators
>> >> >> > app proto: http
>> >> >> > http uri: <unknown>
>> >> >> > http host: <unknown>
>> >> >> > http referer: <unknown>
>> >> >> > http user agent: <unknown>
>> >> >> >
>> >> >> > magic: JPEG image data, JFIF standard 1.01
>> >> >> > app proto: http
>> >> >> > http uri: <unknown>
>> >> >> > http host: <unknown>
>> >> >> > http referer: <unknown>
>> >> >> > http user agent: <unknown>
>> >> >> >
>> >> >> > magic: HTML document, ASCII text, with very long lines
>> >> >> > app proto: http
>> >> >> > http uri: <unknown>
>> >> >> > http host: <unknown>
>> >> >> > http referer: <unknown>
>> >> >> > http user agent: <unknown>
>> >> >> >
>> >> >> > magic: HTML document, ASCII text, with very long lines
>> >> >> > app proto: http
>> >> >> > http uri: <unknown>
>> >> >> > http host: <unknown>
>> >> >> > http referer: <unknown>
>> >> >> > http user agent: <unknown>
>> >> >> >
>> >> >> > magic: UTF-8 Unicode text, with very long lines
>> >> >> > app proto: http
>> >> >> > http uri: <unknown>
>> >> >> > http host: <unknown>
>> >> >> > http referer: <unknown>
>> >> >> > http user agent: <unknown>
>> >> >> >
>> >> >> >
>> >> >> > boo.cap
>> >> >> >
>> >> >> >
>> >> >> > magic: PNG image data, 3996 x 80, 8-bit colormap, non-interlaced
>> >> >> > app proto: http
>> >> >> > http uri: <unknown>
>> >> >> > http host: <unknown>
>> >> >> > http referer: <unknown>
>> >> >> > http user agent: <unknown>
>> >> >> >
>> >> >> > magic: PNG image data, 492 x 400, 8-bit/color RGB, non-interlaced
>> >> >> > app proto: http
>> >> >> > http uri: <unknown>
>> >> >> > http host: <unknown>
>> >> >> > http referer: <unknown>
>> >> >> > http user agent: <unknown>
>> >> >> >
>> >> >> > magic: HTML document, UTF-8 Unicode text, with very long lines, with no line terminators
>> >> >> > app proto: http
>> >> >> > http uri: <unknown>
>> >> >> > http host: <unknown>
>> >> >> > http referer: <unknown>
>> >> >> > http user agent: <unknown>
>> >> >> >
>> >> >> > magic: data
>> >> >> > app proto: http
>> >> >> > http uri: <unknown>
>> >> >> > http host: <unknown>
>> >> >> > http referer: <unknown>
>> >> >> > http user agent: <unknown>
>> >> >> >
>> >> >> > magic: PNG image data, 310 x 440, 8-bit colormap, non-interlaced
>> >> >> > app proto: http
>> >> >> > http uri: <unknown>
>> >> >> > http host: <unknown>
>> >> >> > http referer: <unknown>
>> >> >> > http user agent: <unknown>
>> >> >> >
>> >> >> > magic: ASCII text, with very long lines
>> >> >> > app proto: http
>> >> >> > http uri: <unknown>
>> >> >> > http host: <unknown>
>> >> >> > http referer: <unknown>
>> >> >> > http user agent: <unknown>
>> >> >> >
>> >> >> > magic: ASCII text, with very long lines
>> >> >> > app proto: http
>> >> >> > http uri: <unknown>
>> >> >> > http host: <unknown>
>> >> >> > http referer: <unknown>
>> >> >> > http user agent: <unknown>
>> >> >> >
>> >> >> > magic: PNG image data, 320 x 198, 8-bit colormap, non-interlaced
>> >> >> > app proto: http
>> >> >> > http uri: <unknown>
>> >> >> > http host: <unknown>
>> >> >> > http referer: <unknown>
>> >> >> > http user agent: <unknown>
>> >> >> >
>> >> >> > magic: PDF document, version 1.4
>> >> >> > app proto: http
>> >> >> > http uri: /archive3/GflUt00Q30KF03YzCLl43rm2po76/D3400UM_SG(En)02.pdf
>> >> >> > http host: download.nikonimglib.com
>> >> >> > http referer: http://downloadcenter.nikonimglib.com/en/products/330/D3400.html
>> >> >> > http user agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
>> >> >> >
>> >> >> > magic: PDF document, version 1.3
>> >> >> > app proto: http
>> >> >> > http uri: /biassets/bi/4128311.pdf
>> >> >> > http host: www.lego.com
>> >> >> > http referer: <unknown>
>> >> >> > http user agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
>> >> >> >
>> >> >> > magic: PDF document, version 1.3
>> >> >> > app proto: http
>> >> >> > http uri: /biassets/bi/4128312.pdf
>> >> >> > http host: www.lego.com
>> >> >> > http referer: <unknown>
>> >> >> > http user agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
>> >> >> >
>> >> >> > magic: JPEG image data, EXIF standard
>> >> >> > app proto: http
>> >> >> > http uri: <unknown>
>> >> >> > http host: <unknown>
>> >> >> > http referer: <unknown>
>> >> >> > http user agent: <unknown>
>> >> >> >
>> >> >> > magic: PDF document, version 1.3
>> >> >> > app proto: http
>> >> >> > http uri: /biassets/bi/4132659.pdf
>> >> >> > http host: www.lego.com
>> >> >> > http referer: <unknown>
>> >> >> > http user agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
>> >> >> >
>> >> >> > magic: UTF-8 Unicode text, with very long lines, with no line terminators
>> >> >> > app proto: http
>> >> >> > http uri: <unknown>
>> >> >> > http host: <unknown>
>> >> >> > http referer: <unknown>
>> >> >> > http user agent: <unknown>
>> >> >> >
>> >> >> >
>> >> >>
>> >> >>
>> >> >> I tried the latest gitmaster with filestore v2 - I observed the
>> >> >> following - if you could confirm on your set up please as well with
>> >> >> 4.1.0-rc1.
>> >> >> If i use
>> >> >> alert http any any -> any any (msg:"FILE magic"; filemagic:"PDF";
>> >> >> filestore; sid:0; rev:1;)
>> >> >> I get results like you with the pcap provided foo.pcap (partial html
>> >> >> files present in the download)
>> >> >>
>> >> >> If i use
>> >> >> alert http any any -> any any (msg:"FILE magic"; filemagic:"PDF
>> >> >> document"; filestore; sid:0; rev:1;)
>> >> >> The only diff is filemagic:"PDF document" - i get 0 alerts and 0
>> >> >> partial or full files stored.
>> >> >>
>> >> >> Thank you
>> >> >>
>> >> >> --
>> >> >> Regards,
>> >> >> Peter Manev
>> >> >
>> >> >
>> >>
>> >>
>> >> --
>> >> Regards,
>> >> Peter Manev
>> >
>> >
>>
>>
>> --
>> Regards,
>> Peter Manev
--
Regards,
Peter Manev
More information about the Oisf-users
mailing list