[Oisf-users] Suricata filemagic issue leading to FN on 2009419 and probably others

[Victor Julien]

ET recently started using Suricata's filemagic keyword to determine
certain file types in HTTP. Martin and I identified a serious issue with
the concept. The problem is that for the file classification Suricata
relies on libmagic and it's file definitions. It turns out that there is
some variance between libmagic versions.

For example and Window exec we played with, on my system (Ubuntu 11.10,
libmagic1 5.04-5ubuntu3) returns:

"PE32 executable for MS Windows (GUI) Intel 80386 32-bit"

However, on Martin's SUSE install it returns:

"MS-DOS executable PE  for MS Windows (GUI) Intel 80386 32-bit"

This made SID 2000419 False Negative for Martin.

We have tried loading the more recent Ubuntu magic definitions in
Suricata on the SUSE system, but this failed to work as the format is
different. So distributing a set of magic definitions with ET is not
feasible.

One option would be to have several rules, one for each version of the
magic definition, but at this point I don't know how many variations
exist. This is probably a maintenance nightmare anyway.

Another option would be to make the match more generic, but this may
still FN with unknown variations and may FP if it's too broad.

So I think at this point it's best to revert the filemagic rules to
their originals.

In the future we may consider distributing libmagic with Suricata, like
we do with libhtp, so that we know for sure that everyone runs the same
version. This may not sit well with distributions shipping Suricata though.

Ideas / comments are welcome.

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------