[Oisf-users] Sha hashes not consistent in 3.2beta1, md5 OK

[Duarte Silva]

Hi Jeremy,

tried with the latest version from Git, and I'm not able to replicate the 
issue. My logs report the correct file hashes.

{"timestamp":"2016-10-11T17:30:15.949381+0000","flow_id":1222313897254293,"in_iface":"enp0s31f6","event_type":"fileinfo","src_ip":"141.211.32.32","src_port":80,"dest_ip":"192.168.0.1","dest_port":46650,"proto":"TCP","http":
{"hostname":"www.isr.umich.edu","url":"\/cps\/M-
ABLE\/materials\/EEWE\/Business%20Plan%20Template.pdf","http_user_agent":"curl\/7.50.3","http_content_type":"application\/pdf","http_method":"GET","protocol":"HTTP\/1.1","status":200,"length":748225},"app_proto":"http","fileinfo":
{"filename":"\/cps\/M-ABLE\/materials\/EEWE\/Business Plan 
Template.pdf","state":"CLOSED","md5":"0e26bfdecba382074c4b14d048ccd516","sha1":"081508453775965f197d711584b3343e680af436","sha256":"a5bed200ed4707c0499758f985176135209144e23bcbb8a0a2d21c9abcd3841d","stored":true,"file_id":5,"size":748225,"tx_id":0}}

I have also enabled file storing and this is the results:

$ md5sum files/file.5
0e26bfdecba382074c4b14d048ccd516  files/file.5

$ sha1sum files/file.5
081508453775965f197d711584b3343e680af436  files/file.5

$ sha256sum files/file.5
a5bed200ed4707c0499758f985176135209144e23bcbb8a0a2d21c9abcd3841d  files/file.5

$  cat files/file.5.meta 
TIME:              10/11/2016-19:30:14.563844
SRC IP:            141.211.32.32
DST IP:            192.168.0.1
PROTO:             6
SRC PORT:          80
DST PORT:          46650
APP PROTO:         http
HTTP URI:          /cps/M-ABLE/materials/EEWE/Business Plan Template.pdf?d
HTTP HOST:         www.isr.umich.edu
HTTP REFERER:      <unknown>
HTTP USER AGENT:   curl/7.50.3
FILENAME:          /cps/M-ABLE/materials/EEWE/Business Plan Template.pdf
MAGIC:             <unknown>
STATE:             CLOSED
MD5:               0e26bfdecba382074c4b14d048ccd516
SHA1:              081508453775965f197d711584b3343e680af436
SHA256:            
a5bed200ed4707c0499758f985176135209144e23bcbb8a0a2d21c9abcd3841d
SIZE:              748225

I used PCAP capture method but that wouldn't have much influence since MD5 is 
apparently reporting the hash correctly.

Cheers,
Duarte

On Tuesday 11 October 2016 10:30:35 Jeremy MJ wrote:
> Yes sir, can replicate this at two different locations. Testing for
> two sites (both pf_ring). Traffic is coming to / from proxy server
> (this network device is logging sha256, which is the correct value for
> this test). Eve logs here: http://pastebin.com/04NjQHeJ
> 
> Sample PDF take from random site:
> hxxp://www.isr.umich.edu/cps/M-ABLE/materials/EEWE/Business%20Plan%20Templat
> e.pdf Actual hash values of file:
> MD5: 0e26bfdecba382074c4b14d048ccd516
> SHA: 081508453775965f197d711584b3343e680af436
> SHA256: a5bed200ed4707c0499758f985176135209144e23bcbb8a0a2d21c9abcd3841d
> 
> Suricata IDS devices interpretation of hashes:
> MD5: 0e26bfdecba382074c4b14d048ccd516 (matches)
> SHA: e70f41a89c5389e97e489fbcb5818d6f17cb15ce (mismatch)
> SHA256: acdebd0906bbe479fd70be0cbe2c08067562d5590afff6aa88140f029465a67d
> (mismatch)
> 
> Let me know if you need anything else or have other questions,
> 
> --
> Jeremy MJ
> 
> On Sat, Oct 8, 2016 at 2:11 AM,  <duarte.silva at serializing.me> wrote:
> > Is there a way to replicate this behaviour? Can you isolate a use case
> > where this always happen?
> > 
> > 
> > 
> > 
> > 
> > De: Jeremy MJ
> > Enviado: 7 de outubro de 2016 23:30
> > Para: Duarte Silva
> > Cc: Open Information Security Foundation
> > Assunto: Re: [Oisf-users] Sha hashes not consistent in 3.2beta1, md5 OK
> > 
> > 
> > 
> > Good point. The logging side is reporting incorrect sha hashes
> > 
> > occasionally (sometimes it's correct).
> > 
> > 
> > 
> > Just did a test with sha1/256 rule and correct hash, no alert (md5
> > 
> > still correct, sha values are wrong). I'll try the incorrect hashes in
> > 
> > the rules and see what that does early next week.
> > 
> > 
> > 
> > --
> > 
> > Jeremy MJ
> > 
> > 
> > 
> > 
> > 
> > On Fri, Oct 7, 2016 at 2:27 PM, Duarte Silva
> > 
> > <duarte.silva at serializing.me> wrote:
> >> Hey Jeremy,
> >> 
> >> 
> >> 
> >> are you seeing the problems on the logging or on the rules matching?
> >> 
> >> 
> >> 
> >> Cheers,
> >> 
> >> Duarte
> >> 
> >> On Friday 07 October 2016 12:30:26 Jeremy MJ wrote:
> >>> Greetings,
> >>> 
> >>> 
> >>> 
> >>> I am testing sha1/256 hashing in Suricata 3.2beta1. I noticed that the
> >>> 
> >>> MD5 always matches the file stream, however on occasion the hash for
> >>> 
> >>> sha1/256 do not match the actual file stream (but the md5 does).
> >>> 
> >>> 
> >>> 
> >>> Typically this is on larger files. Is there a configuration setting I
> >>> 
> >>> should look at? Is anyone else observing this?
> >>> 
> >>> 
> >>> 
> >>> Regards,
> >>> 
> >>> 
> >>> 
> >>> --
> >>> 
> >>> Jeremy MJ
> >>> 
> >>> _______________________________________________
> >>> 
> >>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> >>> 
> >>> Site: http://suricata-ids.org | Support:
> >>> http://suricata-ids.org/support/
> >>> 
> >>> List:
> >>> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> >>> 
> >>> Suricata User Conference November 9-11 in Washington, DC:
> >>> http://suricon.net