[Oisf-users] Important note for anyone doing file extraction using the 'filemagic' keyword.

[Cooper F. Nelson]

If you are running alerts like this...

> #alert http any any -> any any (msg:"FILE magic -- windows"; flow:established,to_client; filemagic:"executable for MS Windows"; filestore; sid:18; rev:1;)

...it's going to kill your performance if you are using the default
libmagic database.  I was seeing over 40% CPU utilization for a single rule.

The issue is that suricata doesn't use the MPM to match the magic
number, rather it uses the libmagic matcher (and compiled rule database)
and then matches against what that returns.  The problem is that the
default database is over four megabytes in size and includes tens of
thousands of patterns.  So it's like running a second IDS on top of the
suricata.

However, it turns out there is a fairly simple fix for this.  Just
install the source for the same version of file/libmagic you are using
and you can build your own magic.mgc file.  Look in the "magic/Magdir"
directory for the source files.

Easiest thing to do is just concatenate just the files you want to a
single 'magic' file and then compile it.  Eg:

> cat linux msdos > /tmp/magic
> file -m /tmp/magic -C

Then reference that file in the suricata.yaml file instead of the system
one.  This reduces CPU overhead significantly.

If you want you can improve performance even more by extracting just the
magic numbers you want from the magic files themselves.  However, take
care you follow the syntax so they still compile correctly.

I'm thinking there is an opportunity here for someone to release a
curated magic numbers file for suricata that only includes the most
commonly observed file-types.

-- 
Cooper Nelson
Network Security Analyst
UCSD ITS Security Team
cnelson at ucsd.edu x41042

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20170119/d9b92db2/attachment.sig>