[Oisf-users] Filestore showing truncated files

[Peter Manev]

On Tue, Mar 11, 2014 at 8:47 PM, Bradley Mcalister <bmcalister at gmail.com> wrote:
> Hello,
>
> I recently enabled filestore and am working on MD5 alerting. The issue I am
> running into is that I am randomly getting files listed in files-json.log as
> truncated (I am currently mostly interested in just Win32 executables), and
> as such the hash is not generated. If I download an executable repeatedly,
> it will sometimes generate the MD5 successfully, but I cannot seem to figure
> out what specifically is the problem. Looking at the stats file, no packets
> are being dropped. If anyone has any suggestions, it would be greatly
> appreciated. Thanks.
>
>
> Configuration settings that may or may not prove useful:
>
>  - file-store:
>       enabled: yes       # set to yes to enable
>       log-dir: files    # directory to store the files
>       force-magic: yes   # force logging magic on all stored files
>       force-md5: yes     # force logging of md5 checksums
>       waldo: file.waldo # waldo file to store the file_id across runs
>
>   # output module to log files tracked in a easily parsable json format
>   - file-log:
>       enabled: yes
>       filename: files-json.log
>       append: no
>       force-magic: yes   # force logging magic on all logged files
>       force-md5: yes    # force logging of md5 checksums
>
>
> af-packet:
>   - interface: em3 em4
>     threads: 8
>     cluster-id: 99
>     cluster-type: cluster_cpu
>     defrag: yes
>     use-mmap: yes
>     checksum-checks: no
>     threads: 1
>     cluster-id: 98
>     cluster-type: cluster_flow
>     defrag: yes
>
> stream:
>   memcap: 4gb
>   max-sessions: 2000000
>   prealloc-sessions: 1000000
>   checksum-validation: yes      # reject wrong csums
>   inline: no                  # auto will use inline mode in IPS mode, yes
> or no set it statically
>   reassembly:
>     memcap: 8gb
>     depth: 0                    # reassemble 1mb into a stream
>     toserver-chunk-size: 2560
>     toclient-chunk-size: 2560
>
> pfring:
>   - interface: em3 em4
>     threads: 8
>     cluster-id: 99
>     cluster-type: cluster_flow
>     # bpf filter for this interface
>     bpf-filter: tcp
>     checksum-checks: no
>
> libhtp:
>
>    default-config:
>      personality: IDS
>
>      request-body-limit: 0
>      response-body-limit: 0
>
>      # inspection limits
>      request-body-minimal-inspect-size: 32kb
>      request-body-inspect-window: 4kb
>      response-body-minimal-inspect-size: 32kb
>      response-body-inspect-window: 4kb
>
>      # decoding
>      double-decode-path: no
>      double-decode-query: no
>
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> OISF: http://www.openinfosecfoundation.org/

Hi,
I see you have checksum validation enabled - that could be one reason.
Try with "checksum-validation: no"

In my experience you should have all offloading on the NIC disabled (OFF).
apt-get install ethtool
ethtool -k eth0 - that will show you if any offloading is used on the
interface itself, everything should be OFF
to disbale (for example tcp-segmentation-offload  ) : ethtool -K tso eth0

thanks



-- 
Regards,
Peter Manev