[Oisf-users] Massive kernel drops with HTTP traffic

[Peter Manev]

> On 20 Aug 2018, at 07:39, Konstantin Klinger <konstantin.klinger at dcso.de> wrote:
> 
> 
> 
>> On 20.08.2018 15:29, Peter Manev wrote:
>> 
>> 
>>> On 20 Aug 2018, at 06:53, Konstantin Klinger <konstantin.klinger at dcso.de> wrote:
>>> 
>>> 
>>> 
>>>> On 18.08.2018 15:57, Peter Manev wrote:
>>>> 
>>>> 
>>>>> On 17 Aug 2018, at 07:35, Michael Stone <mstone at mathom.us> wrote:
>>>>> 
>>>>> On Fri, Aug 17, 2018 at 03:24:31PM +0200, you wrote:
>>>>>>> Do you have filemagic enabled?
>>>>>> 
>>>>>> Yes. We currently use filestore v1. And we use the filemagic value in
>>>>>> our rules for filestoring.
>>>>> 
>>>>> Unless you have customized the magic file it is very likely that you won't hit your performance target this way. I'd suggest rules specific to what you're trying to save rather than relying on libmagic (which is very inefficient).
>>>>> 
>>>> 
>>>> 
>>>> That could be easy to test and confirm if it is contributing or creating the mess- Konstantin is it possible to try it out and see ?
>>>> 
>>>> 
>>> 
>>> We made some test runs without filestore enabled and after that only
>>> without libmagic/filemagic (but filestore on) and that helped to
>>> decrease the number of packet drops (~30% -> ~5% and ~50% -> ~10%).
>>> Thank you. Our workaround will be not using filemagic rules anymore.
>>> 
>> 
>> If I remember correctly (please correct me if otherwise) you had a test run where you run Suri with no rules and the drops where still bad (30%+) - what is different from that test and the tests you mentioned above ? (Just having filestore switched to enabled in yaml?)
> 
> Yes, you are completely right. With filestore enabled in yaml and no
> rules loaded we had still 30%+ packet drops.

If not mistaken this is filestore v1 , correct ?
Is this the case with filestore v2 as well ?
Can you please post a bug report describing all the findings including Suricata version you are using (latest git if not mistaken?)


> 
>> 
>> 
>>> @Mike: Do you have further experience in a workaround to not use libmagic?
>>> @all: Is someone using libmagic/filemagic on high traffic sensors
>>> (>5Gb/sec) and has no performance issues? Is someone already using
>>> filestore v2 (we are still using v1) and has any experience with it's
>>> performance?
>>> 
>>> -- 
>>> Konstantin Klinger
>>> Security Content Engineer
>>> Threat Detection & Hunting (TDH)
>>> 
>>> +49 160 95476260
>>> konstantin.klinger at dcso.de
>>> 
>>> dcso.de
>>> blog.dcso.de
>>> 
>>> PGP: 180D C5B3 3C68 5C9A FB58 6F33 400E 5A35 3307 8D46
>>> 
>>> DCSO Deutsche Cyber-Sicherheitsorganisation GmbH • EUREF-Campus
>>> 22 • D-10829 Berlin
>>> Geschäftsführer: Dr.-Ing. Gunnar Siebert, Sitz der Gesellschaft: Berlin,
>>> Amtsgericht Charlottenburg HRB 172382
>>> _______________________________________________
>>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>>> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
>>> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>> 
>>> Conference: https://suricon.net
>>> Trainings: https://suricata-ids.org/training/
> 
> -- 
> Konstantin Klinger
> Security Content Engineer
> Threat Detection & Hunting (TDH)
> 
> +49 160 95476260
> konstantin.klinger at dcso.de
> 
> dcso.de
> blog.dcso.de
> 
> PGP: 180D C5B3 3C68 5C9A FB58 6F33 400E 5A35 3307 8D46
>  
> DCSO Deutsche Cyber-Sicherheitsorganisation GmbH • EUREF-Campus
> 22 • D-10829 Berlin
> Geschäftsführer: Dr.-Ing. Gunnar Siebert, Sitz der Gesellschaft: Berlin,
> Amtsgericht Charlottenburg HRB 172382