[Oisf-users] filestore version 2

[Carl Rotenan]

Yes, on both versions of filestore.

On Tue, Aug 21, 2018 at 5:03 PM, Peter Manev <petermanev at gmail.com> wrote:

> On Tue, Aug 21, 2018 at 2:57 PM Carl Rotenan <carlrotenan at gmail.com>
> wrote:
> >
> > I'm getting the same behavior even if I created a Magic file that only
> knows about PDF files.
> > I'm seeing this behavior on both stable and the RC version.
> >
> > alert http any any -> any any (msg:"FILE store all"; filemagic:"PDF";
> filestore; sid:1; rev:1;)
> >
> > If I do just a filestore all files are extracted.
>
> Ok thank you for the feedback  - is this with filestore v2 as well?
> (in my tests it was, i will open a bug report following that as well)
>
> Thanks
>
> >
> >
> >
> > On Mon, Aug 20, 2018 at 8:17 PM, Peter Manev <petermanev at gmail.com>
> wrote:
> >>
> >> On Thu, Aug 16, 2018 at 11:10 AM Carl Rotenan <carlrotenan at gmail.com>
> wrote:
> >> >
> >> > It appears that if the HTTP info (URI, HOST, REFERER, USER AGENT)
> aren't known the file gets stored.
> >> >
> >> > The info below comes from the file meta data files that are created
> for each capture.
> >> >
> >> > foo.cap
> >> >
> >> > magic:            HTML document, UTF-8 Unicode text, with very long
> lines, with CRLF, LF line terminators
> >> > app proto:        http
> >> > http uri:         <unknown>
> >> > http host:        <unknown>
> >> > http referer:     <unknown>
> >> > http user agent:  <unknown>
> >> >
> >> > magic:            PDF document, version 1.4
> >> > app proto:        http
> >> > http uri:         /files/documents/2018/03/12/dor-2017-inc-sch-hc.pdf
> >> > http host:        www.mass.gov
> >> > http referer:     https://www.mass.gov/lists/
> 2017-massachusetts-personal-income-tax-forms-and-instructions
> >> > http user agent:  Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0)
> like Gecko
> >> >
> >> > magic:            HTML document, ASCII text, with very long lines,
> with CRLF, LF line terminators
> >> > app proto:        http
> >> > http uri:         <unknown>
> >> > http host:        <unknown>
> >> > http referer:     <unknown>
> >> > http user agent:  <unknown>
> >> >
> >> > magic:            JPEG image data, JFIF standard 1.01
> >> > app proto:        http
> >> > http uri:         <unknown>
> >> > http host:        <unknown>
> >> > http referer:     <unknown>
> >> > http user agent:  <unknown>
> >> >
> >> > magic:            PDF document, version 1.6
> >> > app proto:        http
> >> > http uri:         /files/documents/2018/02/07/dor-2017-inc-sch-xy.pdf
> >> > http host:        www.mass.gov
> >> > http referer:     https://www.mass.gov/lists/
> 2017-massachusetts-personal-income-tax-forms-and-instructions
> >> > http user agent:  Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0)
> like Gecko
> >> >
> >> > magic:            HTML document, ASCII text, with very long lines,
> with CRLF, LF line terminators
> >> > app proto:        http
> >> > http uri:         <unknown>
> >> > http host:        <unknown>
> >> > http referer:     <unknown>
> >> > http user agent:  <unknown>
> >> >
> >> > magic:            JPEG image data, JFIF standard 1.01
> >> > app proto:        http
> >> > http uri:         <unknown>
> >> > http host:        <unknown>
> >> > http referer:     <unknown>
> >> > http user agent:  <unknown>
> >> >
> >> > magic:            HTML document, ASCII text, with very long lines
> >> > app proto:        http
> >> > http uri:         <unknown>
> >> > http host:        <unknown>
> >> > http referer:     <unknown>
> >> > http user agent:  <unknown>
> >> >
> >> > magic:            HTML document, ASCII text, with very long lines
> >> > app proto:        http
> >> > http uri:         <unknown>
> >> > http host:        <unknown>
> >> > http referer:     <unknown>
> >> > http user agent:  <unknown>
> >> >
> >> > magic:            UTF-8 Unicode text, with very long lines
> >> > app proto:        http
> >> > http uri:         <unknown>
> >> > http host:        <unknown>
> >> > http referer:     <unknown>
> >> > http user agent:  <unknown>
> >> >
> >> >
> >> > boo.cap
> >> >
> >> >
> >> > magic:            PNG image data, 3996 x 80, 8-bit colormap,
> non-interlaced
> >> > app proto:        http
> >> > http uri:         <unknown>
> >> > http host:        <unknown>
> >> > http referer:     <unknown>
> >> > http user agent:  <unknown>
> >> >
> >> > magic:            PNG image data, 492 x 400, 8-bit/color RGB,
> non-interlaced
> >> > app proto:        http
> >> > http uri:         <unknown>
> >> > http host:        <unknown>
> >> > http referer:     <unknown>
> >> > http user agent:  <unknown>
> >> >
> >> > magic:            HTML document, UTF-8 Unicode text, with very long
> lines, with no line terminators
> >> > app proto:        http
> >> > http uri:         <unknown>
> >> > http host:        <unknown>
> >> > http referer:     <unknown>
> >> > http user agent:  <unknown>
> >> >
> >> > magic:            data
> >> > app proto:        http
> >> > http uri:         <unknown>
> >> > http host:        <unknown>
> >> > http referer:     <unknown>
> >> > http user agent:  <unknown>
> >> >
> >> > magic:            PNG image data, 310 x 440, 8-bit colormap,
> non-interlaced
> >> > app proto:        http
> >> > http uri:         <unknown>
> >> > http host:        <unknown>
> >> > http referer:     <unknown>
> >> > http user agent:  <unknown>
> >> >
> >> > magic:            ASCII text, with very long lines
> >> > app proto:        http
> >> > http uri:         <unknown>
> >> > http host:        <unknown>
> >> > http referer:     <unknown>
> >> > http user agent:  <unknown>
> >> >
> >> > magic:            ASCII text, with very long lines
> >> > app proto:        http
> >> > http uri:         <unknown>
> >> > http host:        <unknown>
> >> > http referer:     <unknown>
> >> > http user agent:  <unknown>
> >> >
> >> > magic:            PNG image data, 320 x 198, 8-bit colormap,
> non-interlaced
> >> > app proto:        http
> >> > http uri:         <unknown>
> >> > http host:        <unknown>
> >> > http referer:     <unknown>
> >> > http user agent:  <unknown>
> >> >
> >> > magic:            PDF document, version 1.4
> >> > app proto:        http
> >> > http uri:         /archive3/GflUt00Q30KF03YzCLl43rm2po76/
> D3400UM_SG(En)02.pdf
> >> > http host:        download.nikonimglib.com
> >> > http referer:     http://downloadcenter.nikonimglib.com/en/products/
> 330/D3400.html
> >> > http user agent:  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36
> (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
> >> >
> >> > magic:            PDF document, version 1.3
> >> > app proto:        http
> >> > http uri:         /biassets/bi/4128311.pdf
> >> > http host:        www.lego.com
> >> > http referer:     <unknown>
> >> > http user agent:  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36
> (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
> >> >
> >> > magic:            PDF document, version 1.3
> >> > app proto:        http
> >> > http uri:         /biassets/bi/4128312.pdf
> >> > http host:        www.lego.com
> >> > http referer:     <unknown>
> >> > http user agent:  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36
> (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
> >> >
> >> > magic:            JPEG image data, EXIF standard
> >> > app proto:        http
> >> > http uri:         <unknown>
> >> > http host:        <unknown>
> >> > http referer:     <unknown>
> >> > http user agent:  <unknown>
> >> >
> >> > magic:            PDF document, version 1.3
> >> > app proto:        http
> >> > http uri:         /biassets/bi/4132659.pdf
> >> > http host:        www.lego.com
> >> > http referer:     <unknown>
> >> > http user agent:  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36
> (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
> >> >
> >> > magic:            UTF-8 Unicode text, with very long lines, with no
> line terminators
> >> > app proto:        http
> >> > http uri:         <unknown>
> >> > http host:        <unknown>
> >> > http referer:     <unknown>
> >> > http user agent:  <unknown>
> >> >
> >> >
> >>
> >>
> >> I tried the latest gitmaster with filestore v2 - I observed the
> >> following  - if you could confirm on your set up please as well with
> >> 4.1.0-rc1.
> >> If i use
> >> alert http any any -> any any (msg:"FILE magic"; filemagic:"PDF";
> >> filestore; sid:0; rev:1;)
> >> I get results like you with the pcap provided foo.pcap (partial html
> >> files present in the download)
> >>
> >> If i use
> >> alert http any any -> any any (msg:"FILE magic"; filemagic:"PDF
> >> document"; filestore; sid:0; rev:1;)
> >> The only diff is filemagic:"PDF document" - i get 0 alerts and 0
> >> partial or full files stored.
> >>
> >> Thank you
> >>
> >> --
> >> Regards,
> >> Peter Manev
> >
> >
>
>
> --
> Regards,
> Peter Manev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20180821/35159429/attachment.html>