[Oisf-users] Important note for anyone doing file extraction using the 'filemagic' keyword.

[Peter Manev]

On Fri, Jan 20, 2017 at 12:35 AM, Cooper F. Nelson <cnelson at ucsd.edu> wrote:
> If you are running alerts like this...
>
>> #alert http any any -> any any (msg:"FILE magic -- windows"; flow:established,to_client; filemagic:"executable for MS Windows"; filestore; sid:18; rev:1;)
>
> ...it's going to kill your performance if you are using the default
> libmagic database.  I was seeing over 40% CPU utilization for a single rule.
>

Across all cores or on a single one?

> The issue is that suricata doesn't use the MPM to match the magic
> number, rather it uses the libmagic matcher (and compiled rule database)
> and then matches against what that returns.  The problem is that the
> default database is over four megabytes in size and includes tens of
> thousands of patterns.  So it's like running a second IDS on top of the
> suricata.
>
> However, it turns out there is a fairly simple fix for this.  Just
> install the source for the same version of file/libmagic you are using
> and you can build your own magic.mgc file.  Look in the "magic/Magdir"
> directory for the source files.
>
> Easiest thing to do is just concatenate just the files you want to a
> single 'magic' file and then compile it.  Eg:
>
>> cat linux msdos > /tmp/magic
>> file -m /tmp/magic -C
>

Just to expand on this a bit -

If you want to compile your own magic file you would need the magic
database sources.
As they do not come installed by default with the file utility - in
order to get those you can do (on Debian for example) -

apt-get source file

then in your current working dir you would end up with something like -
file-5.22+15
file_5.22+15-2+deb8u3.debian.tar.xz
file_5.22+15-2+deb8u3.dsc
file_5.22+15.orig.tar.xz

The magic sources would be under -
file-5.22+15/magic/Magdir/

for example all MS office file magic would be in (msooxml)-
file-5.22+15/magic/Magdir/msooxml

so if you would like to use a custom magic (as pointed by Cooper) for
only linux/ MS executables/ MS office and PDF :

cd file-5.22+15/magic/Magdir/ && cat linux msdos msooxml pdf >/tmp/magic
file -m /tmp/magic -C


> Then reference that file in the suricata.yaml file instead of the system
> one.  This reduces CPU overhead significantly.
>

Nice tip Cooper ! :)

> If you want you can improve performance even more by extracting just the
> magic numbers you want from the magic files themselves.  However, take
> care you follow the syntax so they still compile correctly.
>
> I'm thinking there is an opportunity here for someone to release a
> curated magic numbers file for suricata that only includes the most
> commonly observed file-types.
>



-- 
Regards,
Peter Manev