[Oisf-users] [Emerging-Sigs] Suricata filemagic issue leading to FN on 2009419 and probably others

Wed Apr 4 03:57:41 UTC 2012

How about a grouping or aliasing concept?

So we could essentially alias groups of names, which I don't think would be too difficult. Then use the alias. 

So for this particular one, we want any kind of win32 executable. PE, dos, whatever. So we group all of those together by full name string and have a generic name ("Any Windows Executable Type" for example) and distribute that with suricata. Then we can write rules to the aliased name, which will have all the known file magic db variations in there....

Now, I know that initially sounds as though it'll be admin overhead, but I don't think so> I suspect we'll have a small number of aliases. Off the top of my head, I imagine we'll have like:

Windows EXE
Mac EXE
Elf
Zip
Tar
Common Archive Format

Any PDF

Etc. I'm guessing we'd have less than 50 aliases. It'll be some work to get going, but as long as we keep an eye on the filemagic database changes long term maintenance ought to be trivial.

Thoughts?

Matt

On Mar 27, 2012, at 3:28 AM, Victor Julien wrote:

> On 03/24/2012 01:06 AM, Rodrigo Montoro(Sp0oKeR) wrote:
>> Why not a file into "etc/" and a configuration somewhere in suricata
>> config as snort has for unicode.map file ?
>> 
>> I think using the suggested magic file network admins will make sure
>> they cover in the correct way as the idea of unicode normalization for
>> different languages.
> 
> This could work, however there are several file formats to support.
> Martins libmagic used a format 5, mine a 7. So assuming there are at
> least 7 versions right now. Even if only 3 are more or less recent, it
> would mean ET and/or OISF would have to support 3 sets of files. Not
> impossible, but it adds a burden, especially for QA I think.
> 
> Cheers,
> Victor
> 
>> Regards,
>> 
>> On Fri, Mar 23, 2012 at 4:14 PM, Will Metcalf
>> <wmetcalf at emergingthreatspro.com> wrote:
>>> We are rolling these back to normal rules in the Suricata  rule sets.  This
>>> will happen in today's push.
>>> 
>>> Regards,
>>> 
>>> Will
>>> 
>>> On Fri, Mar 23, 2012 at 1:57 PM, Martin Holste <mcholste at gmail.com> wrote:
>>>> 
>>>> Given how many different possible versions of the library there may be
>>>> (FreeBSD, Solaris, etc.), my bet is that packaging the library with
>>>> Suricata will probably lead to the fewest installation problems.
>>>> 
>>>> On Fri, Mar 23, 2012 at 12:56 PM, Kyle Creyts <kyle.creyts at gmail.com>
>>>> wrote:
>>>>> Also, could just make it a requirement, unless you're distributing bins
>>>>> only.
>>>>> 
>>>>> On Mar 23, 2012 1:22 PM, "Victor Julien" <victor at inliniac.net> wrote:
>>>>>> 
>>>>>> ET recently started using Suricata's filemagic keyword to determine
>>>>>> certain file types in HTTP. Martin and I identified a serious issue
>>>>>> with
>>>>>> the concept. The problem is that for the file classification Suricata
>>>>>> relies on libmagic and it's file definitions. It turns out that there
>>>>>> is
>>>>>> some variance between libmagic versions.
>>>>>> 
>>>>>> For example and Window exec we played with, on my system (Ubuntu 11.10,
>>>>>> libmagic1 5.04-5ubuntu3) returns:
>>>>>> 
>>>>>> "PE32 executable for MS Windows (GUI) Intel 80386 32-bit"
>>>>>> 
>>>>>> However, on Martin's SUSE install it returns:
>>>>>> 
>>>>>> "MS-DOS executable PE  for MS Windows (GUI) Intel 80386 32-bit"
>>>>>> 
>>>>>> This made SID 2000419 False Negative for Martin.
>>>>>> 
>>>>>> We have tried loading the more recent Ubuntu magic definitions in
>>>>>> Suricata on the SUSE system, but this failed to work as the format is
>>>>>> different. So distributing a set of magic definitions with ET is not
>>>>>> feasible.
>>>>>> 
>>>>>> One option would be to have several rules, one for each version of the
>>>>>> magic definition, but at this point I don't know how many variations
>>>>>> exist. This is probably a maintenance nightmare anyway.
>>>>>> 
>>>>>> Another option would be to make the match more generic, but this may
>>>>>> still FN with unknown variations and may FP if it's too broad.
>>>>>> 
>>>>>> So I think at this point it's best to revert the filemagic rules to
>>>>>> their originals.
>>>>>> 
>>>>>> In the future we may consider distributing libmagic with Suricata, like
>>>>>> we do with libhtp, so that we know for sure that everyone runs the same
>>>>>> version. This may not sit well with distributions shipping Suricata
>>>>>> though.
>>>>>> 
>>>>>> Ideas / comments are welcome.
>>>>>> 
>>>>>> --
>>>>>> ---------------------------------------------
>>>>>> Victor Julien
>>>>>> http://www.inliniac.net/
>>>>>> PGP: http://www.inliniac.net/victorjulien.asc
>>>>>> ---------------------------------------------
>>>>>> 
>>>>>> _______________________________________________
>>>>>> Emerging-sigs mailing list
>>>>>> Emerging-sigs at lists.emergingthreats.net
>>>>>> http://lists.emergingthreats.net/mailman/listinfo/emerging-sigs
>>>>>> 
>>>>>> Support Emerging Threats! Subscribe to Emerging Threats Pro
>>>>>> http://www.emergingthreatspro.com
>>>>>> The ONLY place to get complete premium rulesets for Snort 2.4.0 through
>>>>>> Current!
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> Emerging-sigs mailing list
>>>>> Emerging-sigs at lists.emergingthreats.net
>>>>> http://lists.emergingthreats.net/mailman/listinfo/emerging-sigs
>>>>> 
>>>>> Support Emerging Threats! Subscribe to Emerging Threats Pro
>>>>> http://www.emergingthreatspro.com
>>>>> The ONLY place to get complete premium rulesets for Snort 2.4.0 through
>>>>> Current!
>>>> _______________________________________________
>>>> Emerging-sigs mailing list
>>>> Emerging-sigs at lists.emergingthreats.net
>>>> http://lists.emergingthreats.net/mailman/listinfo/emerging-sigs
>>>> 
>>>> Support Emerging Threats! Subscribe to Emerging Threats Pro
>>>> http://www.emergingthreatspro.com
>>>> The ONLY place to get complete premium rulesets for Snort 2.4.0 through
>>>> Current!
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> Emerging-sigs mailing list
>>> Emerging-sigs at lists.emergingthreats.net
>>> http://lists.emergingthreats.net/mailman/listinfo/emerging-sigs
>>> 
>>> Support Emerging Threats! Subscribe to Emerging Threats Pro
>>> http://www.emergingthreatspro.com
>>> The ONLY place to get complete premium rulesets for Snort 2.4.0 through
>>> Current!
>> 
>> 
>> 
> 
> 
> -- 
> ---------------------------------------------
> Victor Julien
> http://www.inliniac.net/
> PGP: http://www.inliniac.net/victorjulien.asc
> ---------------------------------------------
> 
> _______________________________________________
> Emerging-sigs mailing list
> Emerging-sigs at lists.emergingthreats.net
> http://lists.emergingthreats.net/mailman/listinfo/emerging-sigs
> 
> Support Emerging Threats! Subscribe to Emerging Threats Pro http://www.emergingthreatspro.com
> The ONLY place to get complete premium rulesets for Snort 2.4.0 through Current!

----------------------------------------------------
Matt Jonkman
Emerging Threats Pro
Open Information Security Foundation (OISF)
Phone 866-504-2523 x110
http://www.emergingthreatspro.com
http://www.openinfosecfoundation.org
----------------------------------------------------