[Oisf-users] Couple of questions regarding stats.log
Brandon Ganem
brandonganem+oisf at gmail.com
Sun Jun 10 15:07:09 UTC 2012
Correct, although it will still log one or two files per minute some times.
At start up it logs 7k-10k files a minute for about 3-5 minutes, gradually
reducing the number of files logged until it hits 1-2 a minute, sometimes
none for a large span of time.
My apologies if i'm not describing this well.
On Sun, Jun 10, 2012 at 4:31 AM, Peter Manev <petermanev at gmail.com> wrote:
> Hi Brandon,
>
> Ok. I am not sure what is the issue? -
> From what I understood - it logs md5s for a period of time and then
> stops , do i understand correct?
>
> thank you
>
> On 6/9/2012 6:05 PM, Brandon Ganem wrote:
> > Peter,
> >
> > #output module to log files tracked in a easily parsable json format
> > - file-log:
> > enabled: yes
> > filename: files-json.log
> > append: yes
> > #filetype: regular # 'regular', 'unix_stream' or 'unix_dgram'
> > force-magic: yes # force logging magic on all logged files
> > force-md5: yes # force logging of md5 checksums
> >
> > *do you use md5 sigs?*
> > No. I pretty much followed the setup from here:
> > https://redmine.openinfosecfoundation.org/projects/suricata/wiki/MD5
> > The very bottom heading. I do have file store enabled in case there is a
> > situation where I want to start plucking specific files off the wire.
> Could
> > that be causing the issue?
> >
> > On Sat, Jun 9, 2012 at 3:09 AM, Peter Manev <petermanev at gmail.com>
> wrote:
> >
> >> Hi,
> >>
> >> On Fri, Jun 8, 2012 at 9:56 PM, Brandon Ganem <
> brandonganem+oisf at gmail.com
> >>> wrote:
> >>> Changed, seems to have made a huge difference. Thank you!
> >>>
> >>> I'm not sure if this is related, but i've got suricata configured to
> md5
> >>> all files coming across the wire.
> >>>
> >> how do you have that configured?
> >>
> >>
> >>> At start-up it does ~ 7 to 10k a minute for just a few minutes then it
> >>> tappers off until it gets to almost zero files hashed every minute.
> Alerts
> >>> do not seem to be affected.
> >>>
> >> do you use md5 sigs?
> >>
> >>
> >>> Sorry for bombarding the list with questions and thank you for the
> help.
> >>>
> >>> On Fri, Jun 8, 2012 at 2:14 PM, Victor Julien <victor at inliniac.net
> >wrote:
> >>>
> >>>> This may be caused by another option that is only mentioned in the
> >>>> comment block above the stream settings in your yaml:
> >>>>
> >>>> # max-sessions: 262144 # 256k concurrent sessions
> >>>> # prealloc-sessions: 32768 # 32k sessions prealloc'd
> >>>>
> >>>> Max sessions puts a limit to the max number of concurrent tcp sessions
> >>>> tracked.
> >>>>
> >>>> Try setting it to something like:
> >>>>
> >>>> stream:
> >>>> max-sessions: 1000000
> >>>> prealloc-sessions: 500000
> >>>>
> >>>> Or something :)
> >>>>
> >>>> On 06/08/2012 07:24 PM, Brandon Ganem wrote:
> >>>>> It looks like *tcp.ssn_memcap_drop | Detect
> |
> >>>>> 6019 *is starting to add up now too.
> >>>>>
> >>>>> Thanks!
> >>>>>
> >>>>> On Fri, Jun 8, 2012 at 1:09 PM, Brandon Ganem
> >>>>> <brandonganem+oisf at gmail.com <mailto:brandonganem+oisf at gmail.com>>
> >>>> wrote:
> >>>>> /Up your memcap settings to 4GB each and see if the numbers
> >>>> improve.
> >>>>> Both memcap drop stats should be zero when everything's right. /
> >>>>> Done
> >>>>>
> >>>>> /This is odd. Your stream related memcap is 1GB, yet this shows
> >>>> 6GB in
> >>>>> use? Which again doesn't seem to match the memory usage you seem
> >>>> to be
> >>>>> seeing for the whole process. Smells like a bug to me... /
> >>>>> /
> >>>>> /
> >>>>> Let me know if you want me to compile in some debugging features.
> >>>> If
> >>>>> I can provide any additional information let me know.
> >>>>>
> >>>>> CPU / MEM: ~50-125% (similar to before) ~2-2.6GB(similar as
> well.)
> >>>>> Suricata has only been running for a few minutes, but here is a
> new
> >>>>> stats.log:
> >>>>>
> >>>>> tcp.sessions | Detect | 464890
> >>>>> *tcp.ssn_memcap_drop | Detect | 0 (maybe
> >>>>> better, it may have to run for a while to start adding up
> though?)*
> >>>>> tcp.pseudo | Detect | 10567
> >>>>> tcp.invalid_checksum | Detect | 0
> >>>>> tcp.no_flow | Detect | 0
> >>>>> tcp.reused_ssn | Detect | 0
> >>>>> tcp.memuse | Detect | 141604560
> >>>>> tcp.syn | Detect | 465555
> >>>>> tcp.synack | Detect | 233829
> >>>>> tcp.rst | Detect | 46181
> >>>>> *tcp.segment_memcap_drop | Detect | 1281114
> (I
> >>>>> don't think this is impoving)*
> >>>>> *tcp.stream_depth_reached | Detect | 70
> >>>>> (Looks like this is still going up*
> >>>>> tcp.reassembly_memuse | Detect |
> 6442450806
> >>>>> *(still 6GB not 4GB)*
> >>>>> *tcp.reassembly_gap | Detect | 44583
> >>>>> (Still going up)*
> >>>>> detect.alert | Detect | 25
> >>>>> flow_mgr.closed_pruned | FlowManagerThread | 150973
> >>>>> flow_mgr.new_pruned | FlowManagerThread | 207334
> >>>>> flow_mgr.est_pruned | FlowManagerThread | 0
> >>>>> flow.memuse | FlowManagerThread | 41834880
> >>>>> flow.spare | FlowManagerThread | 10742
> >>>>> flow.emerg_mode_entered | FlowManagerThread | 0
> >>>>> flow.emerg_mode_over | FlowManagerThread | 0
> >>>>> decoder.pkts | RxPFR1 | 17310168
> >>>>> decoder.bytes | RxPFR1 |
> 7387022602
> >>>>> decoder.ipv4 | RxPFR1 | 17309598
> >>>>> decoder.ipv6 | RxPFR1 | 0
> >>>>> decoder.ethernet | RxPFR1 | 17310168
> >>>>> decoder.raw | RxPFR1 | 0
> >>>>> decoder.sll | RxPFR1 | 0
> >>>>> decoder.tcp | RxPFR1 | 15519823
> >>>>> decoder.udp | RxPFR1 | 210
> >>>>> decoder.sctp | RxPFR1 | 0
> >>>>> decoder.icmpv4 | RxPFR1 | 1323
> >>>>> decoder.icmpv6 | RxPFR1 | 0
> >>>>> decoder.ppp | RxPFR1 | 0
> >>>>> decoder.pppoe | RxPFR1 | 0
> >>>>> decoder.gre | RxPFR1 | 0
> >>>>> decoder.vlan | RxPFR1 | 0
> >>>>> decoder.avg_pkt_size | RxPFR1 | 427
> >>>>> decoder.max_pkt_size | RxPFR1 | 1516
> >>>>> defrag.ipv4.fragments | RxPFR1 | 15
> >>>>> defrag.ipv4.reassembled | RxPFR1 | 5
> >>>>> defrag.ipv4.timeouts | RxPFR1 | 0
> >>>>> defrag.ipv6.fragments | RxPFR1 | 0
> >>>>> defrag.ipv6.reassembled | RxPFR1 | 0
> >>>>> defrag.ipv6.timeouts | RxPFR1 | 0
> >>>>>
> >>>>>
> >>>>> Here's what has been changed in the cfg:
> >>>>>
> >>>>> flow:
> >>>>> * memcap: 4gb*
> >>>>> hash-size: 65536
> >>>>> prealloc: 10000
> >>>>> emergency-recovery: 30
> >>>>> prune-flows: 5
> >>>>>
> >>>>> stream:
> >>>>> * memcap: 4gb*
> >>>>>
> >>>>> On Fri, Jun 8, 2012 at 12:31 PM, Victor Julien <
> >>>> victor at inliniac.net
> >>>>> <mailto:victor at inliniac.net>> wrote:
> >>>>>
> >>>>> On 06/08/2012 05:59 PM, Brandon Ganem wrote:
> >>>>> > tcp.reassembly_memuse | Detect |
> >>>> 6442450854
> >>>>> This is odd. Your stream related memcap is 1GB, yet this
> shows
> >>>>> 6GB in
> >>>>> use? Which again doesn't seem to match the memory usage you
> >>>> seem
> >>>>> to be
> >>>>> seeing for the whole process. Smells like a bug to me...
> >>>>>
> >>>>> --
> >>>>> ---------------------------------------------
> >>>>> Victor Julien
> >>>>> http://www.inliniac.net/
> >>>>> PGP: http://www.inliniac.net/victorjulien.asc
> >>>>> ---------------------------------------------
> >>>>>
> >>>>> _______________________________________________
> >>>>> Oisf-users mailing list
> >>>>> Oisf-users at openinfosecfoundation.org
> >>>>> <mailto:Oisf-users at openinfosecfoundation.org>
> >>>>>
> >>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> >>>>>
> >>>>>
> >>>>
> >>>> --
> >>>> ---------------------------------------------
> >>>> Victor Julien
> >>>> http://www.inliniac.net/
> >>>> PGP: http://www.inliniac.net/victorjulien.asc
> >>>> ---------------------------------------------
> >>>>
> >>>>
> >>> _______________________________________________
> >>> Oisf-users mailing list
> >>> Oisf-users at openinfosecfoundation.org
> >>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> >>>
> >>>
> >>
> >> --
> >> Regards,
> >> Peter Manev
> >>
> >>
>
>
> --
> Regards,
> Peter Manev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20120610/90fa3d5a/attachment-0002.html>
More information about the Oisf-users
mailing list