[Oisf-users] Filestore showing truncated files

[Bradley Mcalister]

Thanks for the suggestions, Peter.

Didn't notice the checksum-validation setting, I thought I had changed that
already. Just doing that didn't resolve the issue though.

ethtool showed everything as off, but after I updated it on the server it
gave me a couple of new settings that weren't there before, one of which
was on. After setting it off (the others were already), I have yet to get a
truncated file! Thank you so much for your assistance.


On Tue, Mar 11, 2014 at 3:55 PM, Peter Manev <petermanev at gmail.com> wrote:

> On Tue, Mar 11, 2014 at 8:47 PM, Bradley Mcalister <bmcalister at gmail.com>
> wrote:
> > Hello,
> >
> > I recently enabled filestore and am working on MD5 alerting. The issue I
> am
> > running into is that I am randomly getting files listed in
> files-json.log as
> > truncated (I am currently mostly interested in just Win32 executables),
> and
> > as such the hash is not generated. If I download an executable
> repeatedly,
> > it will sometimes generate the MD5 successfully, but I cannot seem to
> figure
> > out what specifically is the problem. Looking at the stats file, no
> packets
> > are being dropped. If anyone has any suggestions, it would be greatly
> > appreciated. Thanks.
> >
> >
> > Configuration settings that may or may not prove useful:
> >
> >  - file-store:
> >       enabled: yes       # set to yes to enable
> >       log-dir: files    # directory to store the files
> >       force-magic: yes   # force logging magic on all stored files
> >       force-md5: yes     # force logging of md5 checksums
> >       waldo: file.waldo # waldo file to store the file_id across runs
> >
> >   # output module to log files tracked in a easily parsable json format
> >   - file-log:
> >       enabled: yes
> >       filename: files-json.log
> >       append: no
> >       force-magic: yes   # force logging magic on all logged files
> >       force-md5: yes    # force logging of md5 checksums
> >
> >
> > af-packet:
> >   - interface: em3 em4
> >     threads: 8
> >     cluster-id: 99
> >     cluster-type: cluster_cpu
> >     defrag: yes
> >     use-mmap: yes
> >     checksum-checks: no
> >     threads: 1
> >     cluster-id: 98
> >     cluster-type: cluster_flow
> >     defrag: yes
> >
> > stream:
> >   memcap: 4gb
> >   max-sessions: 2000000
> >   prealloc-sessions: 1000000
> >   checksum-validation: yes      # reject wrong csums
> >   inline: no                  # auto will use inline mode in IPS mode,
> yes
> > or no set it statically
> >   reassembly:
> >     memcap: 8gb
> >     depth: 0                    # reassemble 1mb into a stream
> >     toserver-chunk-size: 2560
> >     toclient-chunk-size: 2560
> >
> > pfring:
> >   - interface: em3 em4
> >     threads: 8
> >     cluster-id: 99
> >     cluster-type: cluster_flow
> >     # bpf filter for this interface
> >     bpf-filter: tcp
> >     checksum-checks: no
> >
> > libhtp:
> >
> >    default-config:
> >      personality: IDS
> >
> >      request-body-limit: 0
> >      response-body-limit: 0
> >
> >      # inspection limits
> >      request-body-minimal-inspect-size: 32kb
> >      request-body-inspect-window: 4kb
> >      response-body-minimal-inspect-size: 32kb
> >      response-body-inspect-window: 4kb
> >
> >      # decoding
> >      double-decode-path: no
> >      double-decode-query: no
> >
> > _______________________________________________
> > Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> > Site: http://suricata-ids.org | Support:
> http://suricata-ids.org/support/
> > List:
> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> > OISF: http://www.openinfosecfoundation.org/
>
> Hi,
> I see you have checksum validation enabled - that could be one reason.
> Try with "checksum-validation: no"
>
> In my experience you should have all offloading on the NIC disabled (OFF).
> apt-get install ethtool
> ethtool -k eth0 - that will show you if any offloading is used on the
> interface itself, everything should be OFF
> to disbale (for example tcp-segmentation-offload  ) : ethtool -K tso eth0
>
> thanks
>
>
>
> --
> Regards,
> Peter Manev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20140311/80836883/attachment-0002.html>