[Oisf-users] Filestore showing truncated files

[Bradley Mcalister]

Hello,

I recently enabled filestore and am working on MD5 alerting. The issue I am
running into is that I am randomly getting files listed in files-json.log
as truncated (I am currently mostly interested in just Win32 executables),
and as such the hash is not generated. If I download an executable
repeatedly, it will sometimes generate the MD5 successfully, but I cannot
seem to figure out what specifically is the problem. Looking at the stats
file, no packets are being dropped. If anyone has any suggestions, it would
be greatly appreciated. Thanks.


Configuration settings that may or may not prove useful:

 - file-store:
      enabled: yes       # set to yes to enable
      log-dir: files    # directory to store the files
      force-magic: yes   # force logging magic on all stored files
      force-md5: yes     # force logging of md5 checksums
      waldo: file.waldo # waldo file to store the file_id across runs

  # output module to log files tracked in a easily parsable json format
  - file-log:
      enabled: yes
      filename: files-json.log
      append: no
      force-magic: yes   # force logging magic on all logged files
      force-md5: yes    # force logging of md5 checksums


af-packet:
  - interface: em3 em4
    threads: 8
    cluster-id: 99
    cluster-type: cluster_cpu
    defrag: yes
    use-mmap: yes
    checksum-checks: no
    threads: 1
    cluster-id: 98
    cluster-type: cluster_flow
    defrag: yes

stream:
  memcap: 4gb
  max-sessions: 2000000
  prealloc-sessions: 1000000
  checksum-validation: yes      # reject wrong csums
  inline: no                  # auto will use inline mode in IPS mode, yes
or no set it statically
  reassembly:
    memcap: 8gb
    depth: 0                    # reassemble 1mb into a stream
    toserver-chunk-size: 2560
    toclient-chunk-size: 2560

pfring:
  - interface: em3 em4
    threads: 8
    cluster-id: 99
    cluster-type: cluster_flow
    # bpf filter for this interface
    bpf-filter: tcp
    checksum-checks: no

libhtp:

   default-config:
     personality: IDS

     request-body-limit: 0
     response-body-limit: 0

     # inspection limits
     request-body-minimal-inspect-size: 32kb
     request-body-inspect-window: 4kb
     response-body-minimal-inspect-size: 32kb
     response-body-inspect-window: 4kb

     # decoding
     double-decode-path: no
     double-decode-query: no
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20140311/bbdead46/attachment.html>