<html><head><style>p{margin-top:0px;margin-bottom:0px;}</style></head><body><div style="font-size:10pt; font-family:Gulim;"><p>Thank you for your efforts.</p><p>i was also able to have consistent number of logs/alerts through all the pcap runs (with --runmode=single) with the provided pcap and other pcap files.</p><p>When i ran the suricata the multiple pcap files with 'autofp runmode', the resulsts were different through all the pcap runs(reassemble memcap was set '2gb')</p><p> </p><p style="padding: 0px 0px 0px 10pt; font-family: sans-serif; font-size: 10pt;"><span>-----Original Message-----</span><br><b>From:</b> "Peter Manev"<petermanev@gmail.com> <br><b>To:</b> "Andreas Herz"<andi@geekosphere.org>; <br><b>Cc:</b> "oisf-users@lists.openinfosecfoundation.org"<oisf-users@lists.openinfosecfoundation.org>; <br><b>Sent:</b> 2017-01-28 (토) 02:07:58<br><b>Subject:</b> Re: [Oisf-users] [Question] suricata test with pcap-file(After upgrading the suricata version(2.0.11 --> 3.2))<br> </p>On Tue, Jan 24, 2017 at 11:49 PM, Andreas Herz <andi@geekosphere.org> wrote:<br>> On 24/01/17 at 15:01, 박경호 wrote:<br>>> Sorry to send an email which the huge file to be attached.<br>>> Instead of an email, i will share the google drive.<br>>> the link is <a href="https://drive.google.com/file/d/0B4Mdb8bpuRlnU282SzRmRGQ3VXc/view?usp=sharing.">https://drive.google.com/file/d/0B4Mdb8bpuRlnU282SzRmRGQ3VXc/view?usp=sharing.</a><br>>><br>><br>> Thanks for the pcap, we can reproduce the issue and will look into it!<br><br>After doing some investigation with a joint meerkat effort :)<br><br>- Andreas started digging ....<br><br>- Jason pointed out the timestamps are out of order (due many pcaps<br>merged into 1). Fixed by -<br>reordercap testpcap.pcap testpcap-order.pcap<br><br>- I noticed that no matter how many read runs (-r) run we were always<br>hitting segment memcaps (default suricata.yaml settings) - once the<br>reassembly memcap was raised to 256mb we did not have the tcp segment<br>memcap hits any more and we were able to have consistent number of<br>logs/alerts through all the pcap runs (with --runmode=single) with the<br>provided pcap.<br><br>When using autofp runmode though i needed to up the reassembly memcap<br>again to 512mb this time to get to the same consistency. This left me<br>a bit puzzled and ...<br><br>- Victor came to the rescue with figuring it all out:<br><br><br>Many of the segments in the pcap have a size of 1460, but our code has pools for<br>1448, then 65535 (a catch all). So very many of the segments use those<br>64k memory blocks.<br>( https://redmine.openinfosecfoundation.org/projects/suricata/repository/revisions/master/entry/suricata.yaml.in#L1223<br>)<br><br>Add 1460 like so<br><br> - size: 1448<br> prealloc: 1024<br> - size: 1460<br> prealloc: 1024<br> - size: 65535<br> prealloc: 128<br><br>(when Suri is compiled with --enable-debug, you get relevant info on<br>shutdown if more was used than the prealloc ones )<br><br>That solution (better segments size allocation adjustment for this<br>particular case) made my tcp segment memcaps hits<br>(stats.log/eve.json) go away in the test runs.<br><br>I think should also fix the issue you were having.<br><br>Thanks<br><br>><br>>> -----Original Message-----<br>>> From: "Andreas Herz"<andi@geekosphere.org><br>>> To: <oisf-users@lists.openinfosecfoundation.org>;<br>>> Cc:<br>>> Sent: 2017-01-24 (화) 05:11:57<br>>> Subject: Re: [Oisf-users] [Question] suricata test with pcap-file(After upgrading the suricata version(2.0.11 --> 3.2))<br>>><br>>> You can't attach such huge files on mails to the mailinglist. Please try<br>>> to upload them somehwere you trust or try to reduce them to like 10 or<br>>> 20mb so you can send those to us directly via mail (not the<br>>> mailinglist!)<br>>><br>>> On 20/01/17 at 13:11, 박경호 wrote:<br>>> > 대용량 첨부파일 1개(106MB)대용량 첨부 파일은 30일간 보관 / 100회까지 다운로드 가능 testpcap.pcap 106MB 다운로드 기간: 2017/01/20 ~ 2017/02/19I attached the pcap file to use for testing.<br>>> > file size is 111MBytes.<br>>> ><br>>> ><br>>> ><br>>> > -----Original Message-----<br>>> > From: "Andreas Herz"<andi@geekosphere.org><br>>> > To: <oisf-users@lists.openinfosecfoundation.org>;<br>>> > Cc:<br>>> > Sent: 2017-01-20 (금) 06:13:22<br>>> > Subject: Re: [Oisf-users] [Question] suricata test with pcap-file(After upgrading the suricata version(2.0.11 --> 3.2))<br>>> ><br>>> > On 16/01/17 at 17:15, 박경호 wrote:<br>>> > > I did the test to use two smaller pcap files. one is 111MB and another is 66MB.<br>>> > > when i run the suricata twice with 111MB pcap file, the alert messages are different.<br>>> > > But when i run the suricata twice with 66MB pcap file, the alert messagte is same.<br>>> > > I merged the two pcap files(45MB, 66MB) to one pcap file(111MB) using wire-shark.<br>>> ><br>>> > Can you share thoe 11MB pcap here or with us from the OISF team?<br>>> ><br>>> > > Is the this issue computing resources?(specially ram memory issue?)<br>>> ><br>>> > I wouldn't say for sure it's an memory issue.<br>>> ><br>>> > > Can you recommend me how much memory i need in the following situation?<br>>> > > when i check some pcap files which the size is more than 1GB with suricata, how much memory do i need?<br>>> ><br>>> > 8GB are not that low IMHO.<br>>> ><br>>> > > And,<br>>> > > If I add the memory in my computer, which parts are changed in configuration file(suricata.yaml)?<br>>> ><br>>> > Without you changing it, nothing.<br>>> ><br>>> > ><br>>> > > -----Original Message-----<br>>> > > From: "박경호"<pgh5247@naver.com><br>>> > > To: "Andreas Herz"<andi@geekosphere.org>; <oisf-users@lists.openinfosecfoundation.org>;<br>>> > > Cc:<br>>> > > Sent: 2017-01-16 (월) 16:17:33<br>>> > > Subject: Re: [Oisf-users] [Question] suricata test with pcap-file(After upgrading the suricata version(2.0.11 --> 3.2))<br>>> > ><br>>> > ><br>>> > > -----Original Message-----<br>>> > > From: "Andreas Herz"<andi@geekosphere.org><br>>> > > To: <oisf-users@lists.openinfosecfoundation.org>;<br>>> > > Cc:<br>>> > > Sent: 2017-01-14 (토) 06:19:16<br>>> > > Subject: Re: [Oisf-users] [Question] suricata test with pcap-file(After upgrading the suricata version(2.0.11 --> 3.2))<br>>> > ><br>>> > > On 12/01/17 at 10:48, 박경호 wrote:<br>>> > > > After upgrading the version from 2.0.11 to 3.2, I did the test again.<br>>> > > > Unfortunately, alert messages were different whenever the suricata was<br>>> > > > run with same a pcap-file.<br>>> > ><br>>> > > Can you be more verbose about that?<br>>> > > ==> i run the suricata like the following command : suricata -c suricata.yaml -r testpcap.pcap<br>>> > > ( i never changed the configure file(.yaml)).<br>>> > ><br>>> > > > I didn't change the configure file(suricata.yaml) and pcap-file's size<br>>> > > > is 693MB. (pc memory is 8GB, cpu is intel i5-4460, os is Ubuntu<br>>> > > > 16.06)<br>>> > ><br>>> > > Can you try to reproduce the issue with a smaller pcap file that you can<br>>> > > share with us?<br>>> > > ==> After i try to reproduce with a smaller pcap file, i will share the result and pcap file.<br>>> > ><br>>> > > > please explain to me about this situation.<br>>> > ><br>>> > > I still need more details about your suricata configuration, how do you<br>>> > > run suricata, what did you configure?<br>>> > ><br>>> > > An easy way to reproduce that for us will help to find a solution (after<br>>> > > we found what's the real issue you have).<br>>> > ><br>>> > > --<br>>> > > Andreas Herz<br>>> > > _______________________________________________<br>>> > > Suricata IDS Users mailing list: oisf-users@openinfosecfoundation.org<br>>> > > Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/<br>>> > > List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users<br>>> > ><br>>> > ><br>>> ><br>>> > --<br>>> > Andreas Herz<br>>> > _______________________________________________<br>>> > Suricata IDS Users mailing list: oisf-users@openinfosecfoundation.org<br>>> > Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/<br>>> > List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users<br>>><br>>> --<br>>> Andreas Herz<br>>> _______________________________________________<br>>> Suricata IDS Users mailing list: oisf-users@openinfosecfoundation.org<br>>> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/<br>>> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users<br>><br>> --<br>> Andreas Herz<br>> _______________________________________________<br>> Suricata IDS Users mailing list: oisf-users@openinfosecfoundation.org<br>> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/<br>> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users<br><br><br><br>-- <br>Regards,<br>Peter Manev<br>_______________________________________________<br>Suricata IDS Users mailing list: oisf-users@openinfosecfoundation.org<br>Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/<br>List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users<br></div></body></html>
<table style="display:none"><tr><td><img src="https://mail.naver.com/readReceipt/notify/?img=1PYdKokcpXgwF6M%2FhAnXpAtqKxk4FrtwMqk4KztmMx3SF6tlM4KXKrJgMX%2B0Mogw74lR74lcWNFlbX30WLloWrdQareGbrMR%2BBF0bNFgWz0q%2BHK5WXI0W405p4eqp6F4WXk5pzJd16e574eZpm%3D%3D.gif" border="0"></td></tr></table>