[Oisf-users] MD5 hashing of files not correct most of the time

[Peter Manev]

On Tue, Oct 21, 2014 at 10:11 PM, Will Metcalf
<william.metcalf at gmail.com> wrote:
> Are you hitting stream cutoff? Probably would be useful for you to take
> sensitive info out of the config i.e. HOME_NET ip's and share your
> suricata.yaml.
>
> Regards,
>
> Will
>
> On Tue, Oct 21, 2014 at 3:09 PM, Jay M. <jskier at gmail.com> wrote:
>>
>> I'd like to add that, when I tried to troubleshoot more with the file
>> store I'm getting nothing but corrupt files and the same unknown hash
>> values (headers are usually intact though). Also, I'm using af_packet
>> listen mode.
>> --
>> Jay
>> jskier at gmail.com
>>
>>
>> On Tue, Oct 21, 2014 at 12:50 PM, Jay M. <jskier at gmail.com> wrote:
>> > Greetings,
>> >
>> > I'm new to the list, previously a snort user.
>> >
>> > Anyway, I'm testing suricata on a few boxes, and only need MD5 hashes
>> > logged on one of them (traffic between Cisco WSA proxy <> external
>> > net). I have hashing enabled, and the logs give a value for
>> > fileinfo.md5, however this value does not match the actual hash of the
>> > file itself unless the files are very, very small. I've tried png,
>> > jpg, zip, and pdf files as samples.
>> >
>> > I'm running suricata 2.1beta1 (the selks 64-bit Debian live cd) within
>> > VMware 10 which is fed an rspan over USB3 dongle (ax88179_178a).
>> >
>> > I did the following to see if this would help (it did not):
>> > sudo ethtool -K eth1 tso off
>> > sudo ethtool -K eth1 gso off
>> > sudo ethtool -K eth1 gro off
>> > sudo ethtool -K eth1 ufo off
>> > sudo ethtool -K eth1 tx off
>> > sudo ethtool -K eth1 rx off
>> >
>> > Any insight into what is causing the hashes to be inaccurate? So far
>> > I'm looking into possible causes between the proxy and external net
>> > that may manipulate the files (something like compression). Any other
>> > suggestions are appreciated!

You might be hitting stream.reassembly.depth counter limit (as
suggested by Will)? Make sure that limit is bigger than the size of
the files you are trying to get the md5/store.
The stream.checksum_validation should be disabled as well.
Also very important are those :

    http:
      enabled: yes
      # memcap: 64mb

and

request-body-limit, response-body-limit.
as explained here:
https://redmine.openinfosecfoundation.org/projects/suricata/wiki/File_Extraction

Thanks



-- 
Regards,
Peter Manev