[Oisf-users] MD5 hashing of files not correct most of the time

[Jay M.]

Excellent, thanks for the help.

Suricata seems to hash properly now, so long as the file state isn't
truncated, which happens from time to time. The hashing is a very nice
feature to have.
--
Jay
jskier at gmail.com


On Wed, Oct 22, 2014 at 1:07 AM, Peter Manev <petermanev at gmail.com> wrote:
> On Tue, Oct 21, 2014 at 10:11 PM, Will Metcalf
> <william.metcalf at gmail.com> wrote:
>> Are you hitting stream cutoff? Probably would be useful for you to take
>> sensitive info out of the config i.e. HOME_NET ip's and share your
>> suricata.yaml.
>>
>> Regards,
>>
>> Will
>>
>> On Tue, Oct 21, 2014 at 3:09 PM, Jay M. <jskier at gmail.com> wrote:
>>>
>>> I'd like to add that, when I tried to troubleshoot more with the file
>>> store I'm getting nothing but corrupt files and the same unknown hash
>>> values (headers are usually intact though). Also, I'm using af_packet
>>> listen mode.
>>> --
>>> Jay
>>> jskier at gmail.com
>>>
>>>
>>> On Tue, Oct 21, 2014 at 12:50 PM, Jay M. <jskier at gmail.com> wrote:
>>> > Greetings,
>>> >
>>> > I'm new to the list, previously a snort user.
>>> >
>>> > Anyway, I'm testing suricata on a few boxes, and only need MD5 hashes
>>> > logged on one of them (traffic between Cisco WSA proxy <> external
>>> > net). I have hashing enabled, and the logs give a value for
>>> > fileinfo.md5, however this value does not match the actual hash of the
>>> > file itself unless the files are very, very small. I've tried png,
>>> > jpg, zip, and pdf files as samples.
>>> >
>>> > I'm running suricata 2.1beta1 (the selks 64-bit Debian live cd) within
>>> > VMware 10 which is fed an rspan over USB3 dongle (ax88179_178a).
>>> >
>>> > I did the following to see if this would help (it did not):
>>> > sudo ethtool -K eth1 tso off
>>> > sudo ethtool -K eth1 gso off
>>> > sudo ethtool -K eth1 gro off
>>> > sudo ethtool -K eth1 ufo off
>>> > sudo ethtool -K eth1 tx off
>>> > sudo ethtool -K eth1 rx off
>>> >
>>> > Any insight into what is causing the hashes to be inaccurate? So far
>>> > I'm looking into possible causes between the proxy and external net
>>> > that may manipulate the files (something like compression). Any other
>>> > suggestions are appreciated!
>
> You might be hitting stream.reassembly.depth counter limit (as
> suggested by Will)? Make sure that limit is bigger than the size of
> the files you are trying to get the md5/store.
> The stream.checksum_validation should be disabled as well.
> Also very important are those :
>
>     http:
>       enabled: yes
>       # memcap: 64mb
>
> and
>
> request-body-limit, response-body-limit.
> as explained here:
> https://redmine.openinfosecfoundation.org/projects/suricata/wiki/File_Extraction
>
> Thanks
>
>
>
> --
> Regards,
> Peter Manev