[Oisf-users] filestore version 2
Peter Manev
petermanev at gmail.com
Thu Aug 23 02:56:41 UTC 2018
On Tue, Aug 21, 2018 at 3:06 PM Carl Rotenan <carlrotenan at gmail.com> wrote:
>
> Yes, on both versions of filestore.
>
I had a second run/look of the pcap and i have a question -
What/which PDF file do you expect to extract form that capture.
I mean when you view it with wireshark there are some files wit names
that have pdf extensions - but they all are 500-2500 bytes. Is that
expected?
Thank you
> On Tue, Aug 21, 2018 at 5:03 PM, Peter Manev <petermanev at gmail.com> wrote:
>>
>> On Tue, Aug 21, 2018 at 2:57 PM Carl Rotenan <carlrotenan at gmail.com> wrote:
>> >
>> > I'm getting the same behavior even if I created a Magic file that only knows about PDF files.
>> > I'm seeing this behavior on both stable and the RC version.
>> >
>> > alert http any any -> any any (msg:"FILE store all"; filemagic:"PDF"; filestore; sid:1; rev:1;)
>> >
>> > If I do just a filestore all files are extracted.
>>
>> Ok thank you for the feedback - is this with filestore v2 as well?
>> (in my tests it was, i will open a bug report following that as well)
>>
>> Thanks
>>
>> >
>> >
>> >
>> > On Mon, Aug 20, 2018 at 8:17 PM, Peter Manev <petermanev at gmail.com> wrote:
>> >>
>> >> On Thu, Aug 16, 2018 at 11:10 AM Carl Rotenan <carlrotenan at gmail.com> wrote:
>> >> >
>> >> > It appears that if the HTTP info (URI, HOST, REFERER, USER AGENT) aren't known the file gets stored.
>> >> >
>> >> > The info below comes from the file meta data files that are created for each capture.
>> >> >
>> >> > foo.cap
>> >> >
>> >> > magic: HTML document, UTF-8 Unicode text, with very long lines, with CRLF, LF line terminators
>> >> > app proto: http
>> >> > http uri: <unknown>
>> >> > http host: <unknown>
>> >> > http referer: <unknown>
>> >> > http user agent: <unknown>
>> >> >
>> >> > magic: PDF document, version 1.4
>> >> > app proto: http
>> >> > http uri: /files/documents/2018/03/12/dor-2017-inc-sch-hc.pdf
>> >> > http host: www.mass.gov
>> >> > http referer: https://www.mass.gov/lists/2017-massachusetts-personal-income-tax-forms-and-instructions
>> >> > http user agent: Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko
>> >> >
>> >> > magic: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators
>> >> > app proto: http
>> >> > http uri: <unknown>
>> >> > http host: <unknown>
>> >> > http referer: <unknown>
>> >> > http user agent: <unknown>
>> >> >
>> >> > magic: JPEG image data, JFIF standard 1.01
>> >> > app proto: http
>> >> > http uri: <unknown>
>> >> > http host: <unknown>
>> >> > http referer: <unknown>
>> >> > http user agent: <unknown>
>> >> >
>> >> > magic: PDF document, version 1.6
>> >> > app proto: http
>> >> > http uri: /files/documents/2018/02/07/dor-2017-inc-sch-xy.pdf
>> >> > http host: www.mass.gov
>> >> > http referer: https://www.mass.gov/lists/2017-massachusetts-personal-income-tax-forms-and-instructions
>> >> > http user agent: Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko
>> >> >
>> >> > magic: HTML document, ASCII text, with very long lines, with CRLF, LF line terminators
>> >> > app proto: http
>> >> > http uri: <unknown>
>> >> > http host: <unknown>
>> >> > http referer: <unknown>
>> >> > http user agent: <unknown>
>> >> >
>> >> > magic: JPEG image data, JFIF standard 1.01
>> >> > app proto: http
>> >> > http uri: <unknown>
>> >> > http host: <unknown>
>> >> > http referer: <unknown>
>> >> > http user agent: <unknown>
>> >> >
>> >> > magic: HTML document, ASCII text, with very long lines
>> >> > app proto: http
>> >> > http uri: <unknown>
>> >> > http host: <unknown>
>> >> > http referer: <unknown>
>> >> > http user agent: <unknown>
>> >> >
>> >> > magic: HTML document, ASCII text, with very long lines
>> >> > app proto: http
>> >> > http uri: <unknown>
>> >> > http host: <unknown>
>> >> > http referer: <unknown>
>> >> > http user agent: <unknown>
>> >> >
>> >> > magic: UTF-8 Unicode text, with very long lines
>> >> > app proto: http
>> >> > http uri: <unknown>
>> >> > http host: <unknown>
>> >> > http referer: <unknown>
>> >> > http user agent: <unknown>
>> >> >
>> >> >
>> >> > boo.cap
>> >> >
>> >> >
>> >> > magic: PNG image data, 3996 x 80, 8-bit colormap, non-interlaced
>> >> > app proto: http
>> >> > http uri: <unknown>
>> >> > http host: <unknown>
>> >> > http referer: <unknown>
>> >> > http user agent: <unknown>
>> >> >
>> >> > magic: PNG image data, 492 x 400, 8-bit/color RGB, non-interlaced
>> >> > app proto: http
>> >> > http uri: <unknown>
>> >> > http host: <unknown>
>> >> > http referer: <unknown>
>> >> > http user agent: <unknown>
>> >> >
>> >> > magic: HTML document, UTF-8 Unicode text, with very long lines, with no line terminators
>> >> > app proto: http
>> >> > http uri: <unknown>
>> >> > http host: <unknown>
>> >> > http referer: <unknown>
>> >> > http user agent: <unknown>
>> >> >
>> >> > magic: data
>> >> > app proto: http
>> >> > http uri: <unknown>
>> >> > http host: <unknown>
>> >> > http referer: <unknown>
>> >> > http user agent: <unknown>
>> >> >
>> >> > magic: PNG image data, 310 x 440, 8-bit colormap, non-interlaced
>> >> > app proto: http
>> >> > http uri: <unknown>
>> >> > http host: <unknown>
>> >> > http referer: <unknown>
>> >> > http user agent: <unknown>
>> >> >
>> >> > magic: ASCII text, with very long lines
>> >> > app proto: http
>> >> > http uri: <unknown>
>> >> > http host: <unknown>
>> >> > http referer: <unknown>
>> >> > http user agent: <unknown>
>> >> >
>> >> > magic: ASCII text, with very long lines
>> >> > app proto: http
>> >> > http uri: <unknown>
>> >> > http host: <unknown>
>> >> > http referer: <unknown>
>> >> > http user agent: <unknown>
>> >> >
>> >> > magic: PNG image data, 320 x 198, 8-bit colormap, non-interlaced
>> >> > app proto: http
>> >> > http uri: <unknown>
>> >> > http host: <unknown>
>> >> > http referer: <unknown>
>> >> > http user agent: <unknown>
>> >> >
>> >> > magic: PDF document, version 1.4
>> >> > app proto: http
>> >> > http uri: /archive3/GflUt00Q30KF03YzCLl43rm2po76/D3400UM_SG(En)02.pdf
>> >> > http host: download.nikonimglib.com
>> >> > http referer: http://downloadcenter.nikonimglib.com/en/products/330/D3400.html
>> >> > http user agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
>> >> >
>> >> > magic: PDF document, version 1.3
>> >> > app proto: http
>> >> > http uri: /biassets/bi/4128311.pdf
>> >> > http host: www.lego.com
>> >> > http referer: <unknown>
>> >> > http user agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
>> >> >
>> >> > magic: PDF document, version 1.3
>> >> > app proto: http
>> >> > http uri: /biassets/bi/4128312.pdf
>> >> > http host: www.lego.com
>> >> > http referer: <unknown>
>> >> > http user agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
>> >> >
>> >> > magic: JPEG image data, EXIF standard
>> >> > app proto: http
>> >> > http uri: <unknown>
>> >> > http host: <unknown>
>> >> > http referer: <unknown>
>> >> > http user agent: <unknown>
>> >> >
>> >> > magic: PDF document, version 1.3
>> >> > app proto: http
>> >> > http uri: /biassets/bi/4132659.pdf
>> >> > http host: www.lego.com
>> >> > http referer: <unknown>
>> >> > http user agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
>> >> >
>> >> > magic: UTF-8 Unicode text, with very long lines, with no line terminators
>> >> > app proto: http
>> >> > http uri: <unknown>
>> >> > http host: <unknown>
>> >> > http referer: <unknown>
>> >> > http user agent: <unknown>
>> >> >
>> >> >
>> >>
>> >>
>> >> I tried the latest gitmaster with filestore v2 - I observed the
>> >> following - if you could confirm on your set up please as well with
>> >> 4.1.0-rc1.
>> >> If i use
>> >> alert http any any -> any any (msg:"FILE magic"; filemagic:"PDF";
>> >> filestore; sid:0; rev:1;)
>> >> I get results like you with the pcap provided foo.pcap (partial html
>> >> files present in the download)
>> >>
>> >> If i use
>> >> alert http any any -> any any (msg:"FILE magic"; filemagic:"PDF
>> >> document"; filestore; sid:0; rev:1;)
>> >> The only diff is filemagic:"PDF document" - i get 0 alerts and 0
>> >> partial or full files stored.
>> >>
>> >> Thank you
>> >>
>> >> --
>> >> Regards,
>> >> Peter Manev
>> >
>> >
>>
>>
>> --
>> Regards,
>> Peter Manev
>
>
--
Regards,
Peter Manev
More information about the Oisf-users
mailing list