[Oisf-users] MD5 hashing of files not correct most of the time

[Tom DeCanio]

I'm close to submitting a PR for the MD5 calculation that can speed this up
considerably.  It turns out that there is a lock in the libnss library that
limits the performance of MD5 calculations expecially on small chunks of
data. If you run into performance problems I can probably share the patch.

On Wed, Oct 22, 2014 at 6:42 AM, Jay M. <jskier at gmail.com> wrote:

> Excellent, thanks for the help.
>
> Suricata seems to hash properly now, so long as the file state isn't
> truncated, which happens from time to time. The hashing is a very nice
> feature to have.
> --
> Jay
> jskier at gmail.com
>
>
> On Wed, Oct 22, 2014 at 1:07 AM, Peter Manev <petermanev at gmail.com> wrote:
> > On Tue, Oct 21, 2014 at 10:11 PM, Will Metcalf
> > <william.metcalf at gmail.com> wrote:
> >> Are you hitting stream cutoff? Probably would be useful for you to take
> >> sensitive info out of the config i.e. HOME_NET ip's and share your
> >> suricata.yaml.
> >>
> >> Regards,
> >>
> >> Will
> >>
> >> On Tue, Oct 21, 2014 at 3:09 PM, Jay M. <jskier at gmail.com> wrote:
> >>>
> >>> I'd like to add that, when I tried to troubleshoot more with the file
> >>> store I'm getting nothing but corrupt files and the same unknown hash
> >>> values (headers are usually intact though). Also, I'm using af_packet
> >>> listen mode.
> >>> --
> >>> Jay
> >>> jskier at gmail.com
> >>>
> >>>
> >>> On Tue, Oct 21, 2014 at 12:50 PM, Jay M. <jskier at gmail.com> wrote:
> >>> > Greetings,
> >>> >
> >>> > I'm new to the list, previously a snort user.
> >>> >
> >>> > Anyway, I'm testing suricata on a few boxes, and only need MD5 hashes
> >>> > logged on one of them (traffic between Cisco WSA proxy <> external
> >>> > net). I have hashing enabled, and the logs give a value for
> >>> > fileinfo.md5, however this value does not match the actual hash of
> the
> >>> > file itself unless the files are very, very small. I've tried png,
> >>> > jpg, zip, and pdf files as samples.
> >>> >
> >>> > I'm running suricata 2.1beta1 (the selks 64-bit Debian live cd)
> within
> >>> > VMware 10 which is fed an rspan over USB3 dongle (ax88179_178a).
> >>> >
> >>> > I did the following to see if this would help (it did not):
> >>> > sudo ethtool -K eth1 tso off
> >>> > sudo ethtool -K eth1 gso off
> >>> > sudo ethtool -K eth1 gro off
> >>> > sudo ethtool -K eth1 ufo off
> >>> > sudo ethtool -K eth1 tx off
> >>> > sudo ethtool -K eth1 rx off
> >>> >
> >>> > Any insight into what is causing the hashes to be inaccurate? So far
> >>> > I'm looking into possible causes between the proxy and external net
> >>> > that may manipulate the files (something like compression). Any other
> >>> > suggestions are appreciated!
> >
> > You might be hitting stream.reassembly.depth counter limit (as
> > suggested by Will)? Make sure that limit is bigger than the size of
> > the files you are trying to get the md5/store.
> > The stream.checksum_validation should be disabled as well.
> > Also very important are those :
> >
> >     http:
> >       enabled: yes
> >       # memcap: 64mb
> >
> > and
> >
> > request-body-limit, response-body-limit.
> > as explained here:
> >
> https://redmine.openinfosecfoundation.org/projects/suricata/wiki/File_Extraction
> >
> > Thanks
> >
> >
> >
> > --
> > Regards,
> > Peter Manev
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> Training now available: http://suricata-ids.org/training/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20141022/c2d7ceba/attachment-0002.html>