[Oisf-users] [Question] suricata test with pcap-file(After upgrading the suricata version(2.0.11 --> 3.2))

Fri Jan 27 17:07:58 UTC 2017

On Tue, Jan 24, 2017 at 11:49 PM, Andreas Herz <andi at geekosphere.org> wrote:
> On 24/01/17 at 15:01, 박경호 wrote:
>> Sorry to send an email which the huge file to be attached.
>> Instead of an email, i will share the google drive.
>> the link is https://drive.google.com/file/d/0B4Mdb8bpuRlnU282SzRmRGQ3VXc/view?usp=sharing.
>>
>
> Thanks for the pcap, we can reproduce the issue and will look into it!

After doing some investigation with a joint meerkat effort :)

- Andreas started digging ....

- Jason pointed out the timestamps are out of order (due many pcaps
merged into 1). Fixed by  -
reordercap testpcap.pcap testpcap-order.pcap

- I noticed that no matter how many read runs (-r) run we were always
hitting segment memcaps (default suricata.yaml settings) - once the
reassembly memcap was raised to 256mb we did not have the tcp segment
memcap hits any more and we were able to have consistent number of
logs/alerts through all the pcap runs (with --runmode=single) with the
provided pcap.

When using autofp runmode though i needed to up the reassembly memcap
again to 512mb this time to get to the same consistency. This left me
a bit puzzled and ...

- Victor came to the rescue with figuring it all out:

Many of the segments in the pcap have a size of 1460, but our code has pools for
1448, then 65535 (a catch all). So very many of the segments use those
64k memory blocks.
( https://redmine.openinfosecfoundation.org/projects/suricata/repository/revisions/master/entry/suricata.yaml.in#L1223
)

Add 1460 like so

      - size: 1448
        prealloc: 1024
      - size: 1460
        prealloc: 1024
      - size: 65535
        prealloc: 128

(when Suri is compiled with --enable-debug, you get relevant info on
shutdown if more was used than the prealloc ones )

That solution (better segments size allocation adjustment for this
particular case) made my  tcp segment memcaps hits
(stats.log/eve.json) go away in the test runs.

I think should also fix the issue you were having.

Thanks

>
>> -----Original Message-----
>> From: "Andreas Herz"<andi at geekosphere.org>
>> To: <oisf-users at lists.openinfosecfoundation.org>;
>> Cc:
>> Sent: 2017-01-24 (화) 05:11:57
>> Subject: Re: [Oisf-users] [Question] suricata test with pcap-file(After upgrading the suricata version(2.0.11 --> 3.2))
>>
>> You can't attach such huge files on mails to the mailinglist. Please try
>> to upload them somehwere you trust or try to reduce them to like 10 or
>> 20mb so you can send those to us directly via mail (not the
>> mailinglist!)
>>
>> On 20/01/17 at 13:11, 박경호 wrote:
>> >  대용량 첨부파일 1개(106MB)대용량 첨부 파일은 30일간 보관 / 100회까지 다운로드 가능  testpcap.pcap 106MB  다운로드 기간: 2017/01/20 ~ 2017/02/19I attached the pcap file to use  for testing.
>> > file size is 111MBytes.
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: "Andreas Herz"<andi at geekosphere.org>
>> > To: <oisf-users at lists.openinfosecfoundation.org>;
>> > Cc:
>> > Sent: 2017-01-20 (금) 06:13:22
>> > Subject: Re: [Oisf-users] [Question] suricata test with pcap-file(After upgrading the suricata version(2.0.11 --> 3.2))
>> >
>> > On 16/01/17 at 17:15, 박경호 wrote:
>> > > I did the test to use two smaller pcap files. one is 111MB and another is 66MB.
>> > > when i run the suricata twice with 111MB pcap file, the alert messages are different.
>> > > But when i run the suricata twice with 66MB pcap file, the alert messagte is same.
>> > > I merged the two pcap files(45MB, 66MB) to one pcap file(111MB) using wire-shark.
>> >
>> > Can you share thoe 11MB pcap here or with us from the OISF team?
>> >
>> > > Is the this issue  computing resources?(specially ram memory issue?)
>> >
>> > I wouldn't say for sure it's an memory issue.
>> >
>> > > Can you recommend me  how much memory i need in the following situation?
>> > > when i check some pcap files which the size is more than 1GB with suricata,  how much memory do i  need?
>> >
>> > 8GB are not that low IMHO.
>> >
>> > > And,
>> > > If I add the memory in my computer, which parts are changed in configuration file(suricata.yaml)?
>> >
>> > Without you changing it, nothing.
>> >
>> > >
>> > > -----Original Message-----
>> > > From: "박경호"<pgh5247 at naver.com>
>> > > To: "Andreas Herz"<andi at geekosphere.org>; <oisf-users at lists.openinfosecfoundation.org>;
>> > > Cc:
>> > > Sent: 2017-01-16 (월) 16:17:33
>> > > Subject: Re: [Oisf-users] [Question] suricata test with pcap-file(After upgrading the suricata version(2.0.11 --> 3.2))
>> > >
>> > >
>> > > -----Original Message-----
>> > > From: "Andreas Herz"<andi at geekosphere.org>
>> > > To: <oisf-users at lists.openinfosecfoundation.org>;
>> > > Cc:
>> > > Sent: 2017-01-14 (토) 06:19:16
>> > > Subject: Re: [Oisf-users] [Question] suricata test with pcap-file(After upgrading the suricata version(2.0.11 --> 3.2))
>> > >
>> > > On 12/01/17 at 10:48, 박경호 wrote:
>> > > > After upgrading the version from 2.0.11 to 3.2, I did the test again.
>> > > > Unfortunately, alert messages were different whenever the suricata was
>> > > > run with same a pcap-file.
>> > >
>> > > Can you be more verbose about that?
>> > > ==> i run the suricata like the following command : suricata -c suricata.yaml -r testpcap.pcap
>> > >       ( i never changed the configure file(.yaml)).
>> > >
>> > > > I didn't change the configure file(suricata.yaml) and pcap-file's size
>> > > > is 693MB.  (pc memory is 8GB, cpu is intel i5-4460, os is Ubuntu
>> > > > 16.06)
>> > >
>> > > Can you try to reproduce the issue with a smaller pcap file that you can
>> > > share with us?
>> > > ==> After i try to reproduce with a smaller pcap file, i will share the result and pcap file.
>> > >
>> > > > please explain to me about this situation.
>> > >
>> > > I still need more details about your suricata configuration, how do you
>> > > run suricata, what did you configure?
>> > >
>> > > An easy way to reproduce that for us will help to find a solution (after
>> > > we found what's the real issue you have).
>> > >
>> > > --
>> > > Andreas Herz
>> > > _______________________________________________
>> > > Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>> > > Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
>> > > List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>> > >
>> > >
>> >
>> > --
>> > Andreas Herz
>> > _______________________________________________
>> > Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>> > Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
>> > List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>
>> --
>> Andreas Herz
>> _______________________________________________
>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
>> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>
> --
> Andreas Herz
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users

-- 
Regards,
Peter Manev