[Oisf-users] Massive kernel drops with HTTP traffic

[Michael Stone]

On Mon, Aug 20, 2018 at 02:53:55PM +0200, Konstantin Klinger wrote:
>@Mike: Do you have further experience in a workaround to not use libmagic?

Write rules for the traffic you're looking for. E.g., from 
https://redmine.openinfosecfoundation.org/projects/suricata/wiki/file_extraction:
you can look at file extensions:

alert http any any -> any any (msg:"FILE PDF file claimed"; fileext:"pdf"; filestore; sid:2; rev:1;)

you can also look for the magic signatures for particular file types:

alert http any any -> any any (msg:"FILE MS executable"; file_data; content:"MZ"; depth:2; filestore; sid:X; rev:1;)

That's essentially what the libmagic stuff does, but it's much more 
efficient if you're looking for only a few things to look specifically 
for those things rather than passing each file stream off to libmagic to 
check for thousands of different patterns (some of which are incredibly 
costly, like looking at the stream byte-by-byte to guess what kind of 
text file it might be). Suricata spends a lot of effort on using things 
like hyperscan to do really efficient pattern matching, then if you turn 
on magic support you basically toss all that and go back to doing 
character matches. You can look in the libmagic source to see what it's 
using for signatures. Depending on what you're trying to do, you may be 
able to accept more false positives (store possibly non-valid files) and 
use simpler signatures like the simple MZ test above without all the 
subsequent checks that filemagic does.

Adding the "noalert" keyword lets you just store the files without 
triggering alerts, which may be especially helpful if you're willing to 
accept FPs on file storage but then do more careful analysis in 
post-processing.

You can also try this suggestion for improving libmagic performance (I 
haven't):

https://lists.openinfosecfoundation.org/pipermail/oisf-users/2017-January/006725.html


FYI, another feature that sounds good but scales terribly is the 
tls.fingerprint keyword.