[Oisf-users] MD5 hashing of files not correct most of the time

[Jay M.]

Awesome, I would be interested in the patch either way.
--
Jay
jskier at gmail.com


On Wed, Oct 22, 2014 at 9:59 AM, Tom DeCanio <decanio.tom at gmail.com> wrote:
> I'm close to submitting a PR for the MD5 calculation that can speed this up
> considerably.  It turns out that there is a lock in the libnss library that
> limits the performance of MD5 calculations expecially on small chunks of
> data. If you run into performance problems I can probably share the patch.
>
> On Wed, Oct 22, 2014 at 6:42 AM, Jay M. <jskier at gmail.com> wrote:
>>
>> Excellent, thanks for the help.
>>
>> Suricata seems to hash properly now, so long as the file state isn't
>> truncated, which happens from time to time. The hashing is a very nice
>> feature to have.
>> --
>> Jay
>> jskier at gmail.com
>>
>>
>> On Wed, Oct 22, 2014 at 1:07 AM, Peter Manev <petermanev at gmail.com> wrote:
>> > On Tue, Oct 21, 2014 at 10:11 PM, Will Metcalf
>> > <william.metcalf at gmail.com> wrote:
>> >> Are you hitting stream cutoff? Probably would be useful for you to take
>> >> sensitive info out of the config i.e. HOME_NET ip's and share your
>> >> suricata.yaml.
>> >>
>> >> Regards,
>> >>
>> >> Will
>> >>
>> >> On Tue, Oct 21, 2014 at 3:09 PM, Jay M. <jskier at gmail.com> wrote:
>> >>>
>> >>> I'd like to add that, when I tried to troubleshoot more with the file
>> >>> store I'm getting nothing but corrupt files and the same unknown hash
>> >>> values (headers are usually intact though). Also, I'm using af_packet
>> >>> listen mode.
>> >>> --
>> >>> Jay
>> >>> jskier at gmail.com
>> >>>
>> >>>
>> >>> On Tue, Oct 21, 2014 at 12:50 PM, Jay M. <jskier at gmail.com> wrote:
>> >>> > Greetings,
>> >>> >
>> >>> > I'm new to the list, previously a snort user.
>> >>> >
>> >>> > Anyway, I'm testing suricata on a few boxes, and only need MD5
>> >>> > hashes
>> >>> > logged on one of them (traffic between Cisco WSA proxy <> external
>> >>> > net). I have hashing enabled, and the logs give a value for
>> >>> > fileinfo.md5, however this value does not match the actual hash of
>> >>> > the
>> >>> > file itself unless the files are very, very small. I've tried png,
>> >>> > jpg, zip, and pdf files as samples.
>> >>> >
>> >>> > I'm running suricata 2.1beta1 (the selks 64-bit Debian live cd)
>> >>> > within
>> >>> > VMware 10 which is fed an rspan over USB3 dongle (ax88179_178a).
>> >>> >
>> >>> > I did the following to see if this would help (it did not):
>> >>> > sudo ethtool -K eth1 tso off
>> >>> > sudo ethtool -K eth1 gso off
>> >>> > sudo ethtool -K eth1 gro off
>> >>> > sudo ethtool -K eth1 ufo off
>> >>> > sudo ethtool -K eth1 tx off
>> >>> > sudo ethtool -K eth1 rx off
>> >>> >
>> >>> > Any insight into what is causing the hashes to be inaccurate? So far
>> >>> > I'm looking into possible causes between the proxy and external net
>> >>> > that may manipulate the files (something like compression). Any
>> >>> > other
>> >>> > suggestions are appreciated!
>> >
>> > You might be hitting stream.reassembly.depth counter limit (as
>> > suggested by Will)? Make sure that limit is bigger than the size of
>> > the files you are trying to get the md5/store.
>> > The stream.checksum_validation should be disabled as well.
>> > Also very important are those :
>> >
>> >     http:
>> >       enabled: yes
>> >       # memcap: 64mb
>> >
>> > and
>> >
>> > request-body-limit, response-body-limit.
>> > as explained here:
>> >
>> > https://redmine.openinfosecfoundation.org/projects/suricata/wiki/File_Extraction
>> >
>> > Thanks
>> >
>> >
>> >
>> > --
>> > Regards,
>> > Peter Manev
>> _______________________________________________
>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
>> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>> Training now available: http://suricata-ids.org/training/
>
>