[Oisf-devel] tcp.ssn_memcap_drop
Martin Holste
mcholste at gmail.com
Wed Sep 21 16:22:14 UTC 2011
This one is on a bigger box (144 GB RAM, 16 CPU) with AF_PACKET:
Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffe4ff9710 (LWP 6178)]
0x00007ffff679d945 in raise () from /lib64/libc.so.6
(gdb) bt
#0 0x00007ffff679d945 in raise () from /lib64/libc.so.6
#1 0x00007ffff679ef21 in abort () from /lib64/libc.so.6
#2 0x00007ffff67da8ef in __libc_message () from /lib64/libc.so.6
#3 0x00007ffff67e0018 in malloc_printerr () from /lib64/libc.so.6
#4 0x00007ffff67e4f6c in free () from /lib64/libc.so.6
#5 0x00007ffff7bd8880 in htp_tx_destroy (tx=0x3233d7f0) at
htp_transaction.c:123
#6 0x00007ffff7bd5e12 in htp_conn_destroy (conn=0x2cc524b0) at
htp_connection.c:65
#7 0x00007ffff7bd1112 in htp_connp_destroy_all (connp=0x3957d280) at
htp_connection_parser.c:197
#8 0x000000000061e03a in HTPStateFree (state=<value optimized out>)
at app-layer-htp.c:210
#9 0x000000000061204b in AppLayerParserCleanupState
(f=0x7ffe6f639b10) at app-layer-parser.c:1240
#10 0x0000000000437315 in FlowL7DataPtrFree (f=0x180e) at flow.c:119
#11 0x000000000043737f in FlowClearMemory (f=0x7ffe6f639b10,
proto_map=<value optimized out>) at flow.c:1406
#12 0x0000000000437593 in FlowPrune (q=0x942190, ts=0x7fffe4ff8e60) at
flow.c:336
#13 0x000000000043794d in FlowPruneFlowQueue (ts=<value optimized
out>, q=<value optimized out>) at flow.c:355
#14 FlowManagerThread (ts=<value optimized out>, q=<value optimized
out>) at flow.c:1060
#15 0x00007ffff6f265f0 in start_thread () from /lib64/libpthread.so.0
#16 0x00007ffff683f87d in clone () from /lib64/libc.so.6
#17 0x0000000000000000 in ?? ()
On Wed, Sep 21, 2011 at 7:53 AM, Martin Holste <mcholste at gmail.com> wrote:
> Next seg fault, this one with an AFPACKET data source:
>
> Program received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 0x7ffff1c02700 (LWP 3947)]
> _int_malloc (av=0x7fffe0000020, bytes=24) at malloc.c:4338
> 4338 malloc.c: No such file or directory.
> in malloc.c
> (gdb) bt
> #0 _int_malloc (av=0x7fffe0000020, bytes=24) at malloc.c:4338
> #1 0x00007ffff69da454 in __libc_calloc (n=<value optimized out>,
> elem_size=<value optimized out>) at malloc.c:4065
> #2 0x00007ffff7bd1080 in htp_process_request_header_generic
> (connp=0x2c235e0) at htp_request_generic.c:30
> #3 0x00007ffff7bd5dbf in htp_connp_REQ_HEADERS (connp=0x2c235e0) at
> htp_request.c:421
> #4 0x00007ffff7bd5209 in htp_connp_req_data (connp=0x2c235e0,
> timestamp=24, data=0x7fffe0000088 "\340(\371\343\377\177", len=2) at
> htp_request.c:839
> #5 0x00000000006250c1 in HTPHandleRequestData (f=0x9721790,
> htp_state=0x2d38ac0, pstate=0xbb6e7d8,
> input=0x7ffff1c00c60 "GET <REDACTED> HTTP/1.1\r\n"...,
> input_len=2, output=0x7fffe0000028) at app-layer-htp.c:393
> #6 0x00000000006202ff in AppLayerDoParse (f=0x7fffe0000020,
> app_layer_state=0x18, parser_state=0x7fffe0000088, input=0x2 <Address
> 0x2 out of bounds>, input_len=1835559213, parser_idx=<value optimized
> out>,
> proto=1) at app-layer-parser.c:684
> #7 0x00000000006206ab in AppLayerParse (f=0x9721790, proto=<value
> optimized out>, flags=<value optimized out>, input=0x2 <Address 0x2
> out of bounds>, input_len=1835559213) at app-layer-parser.c:897
> #8 0x00000000005f0a4a in StreamTcpReassembleAppLayer (ra_ctx=<value
> optimized out>, ssn=<value optimized out>, stream=<value optimized
> out>, p=<value optimized out>) at stream-tcp-reassemble.c:3043
> #9 0x00000000005f0c2f in StreamTcpReassembleHandleSegmentUpdateACK
> (ra_ctx=<value optimized out>, ssn=<value optimized out>,
> stream=0x7fffcc2e8118, p=<value optimized out>) at
> stream-tcp-reassemble.c:3422
> #10 0x00000000005f2189 in StreamTcpReassembleHandleSegment
> (tv=0x7fffec03c7c0, ra_ctx=0x7fffdc00a450, ssn=0x7fffcc2e80c0,
> stream=0x7fffcc2e80c8, p=0xa0d59306d68692d, pq=0x7fffe0000028)
> at stream-tcp-reassemble.c:3496
> #11 0x00000000005e0fd6 in HandleEstablishedPacketToClient
> (tv=0x7fffec03c7c0, p=0x1c7d010, stt=0x4ddec30, ssn=0x7fffcc2e80c0,
> pq=<value optimized out>) at stream-tcp.c:1836
> #12 StreamTcpPacketStateEstablished (tv=0x7fffec03c7c0, p=0x1c7d010,
> stt=0x4ddec30, ssn=0x7fffcc2e80c0, pq=<value optimized out>) at
> stream-tcp.c:1990
> #13 0x00000000005e3a3d in StreamTcpPacket (tv=0x7fffec03c7c0,
> p=0x1c7d010, stt=0x4ddec30, pq=<value optimized out>) at
> stream-tcp.c:3570
> #14 0x00000000005e4f0c in StreamTcp (tv=0x7fffec03c7c0, p=0x1c7d010,
> data=0x4ddec30, pq=0x7fffec03c8d8, postpq=<value optimized out>) at
> stream-tcp.c:3769
> #15 0x00000000005c660f in TmThreadsSlotVarRun (tv=0x7fffec03c7c0,
> p=0x1c7d010, slot=<value optimized out>) at tm-threads.c:439
> #16 0x00000000005c6744 in TmThreadsSlotVar (td=0x7fffec03c7c0) at
> tm-threads.c:660
> #17 0x00007ffff71399ca in start_thread (arg=<value optimized out>) at
> pthread_create.c:300
> #18 0x00007ffff6a4270d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #19 0x0000000000000000 in ?? ()
>
>
> On Tue, Sep 20, 2011 at 2:11 PM, Martin Holste <mcholste at gmail.com> wrote:
>> Next seg fault:
>>
>> Program received signal SIGSEGV, Segmentation fault.
>> [Switching to Thread 0x7fffeaffd700 (LWP 29659)]
>> DeStateDetectContinueDetection (tv=<value optimized out>,
>> de_ctx=<value optimized out>, det_ctx=0x7ffff00387e0, f=<value
>> optimized out>, flags=8 '\b', alstate=0x7ffee5b70bc0, alproto=1,
>> alversion=2)
>> at detect-engine-state.c:731
>> 731 match = sigmatch_table[sm->type].
>> (gdb) bt
>> #0 DeStateDetectContinueDetection (tv=<value optimized out>,
>> de_ctx=<value optimized out>, det_ctx=0x7ffff00387e0, f=<value
>> optimized out>, flags=8 '\b', alstate=0x7ffee5b70bc0, alproto=1,
>> alversion=2)
>> at detect-engine-state.c:731
>> #1 0x000000000044abe3 in SigMatchSignatures (th_v=<value optimized
>> out>, de_ctx=<value optimized out>, det_ctx=0x7ffff00387e0,
>> p=0x1878e10) at detect.c:1447
>> #2 0x000000000044ad1c in Detect (tv=0x1, p=<value optimized out>,
>> data=0x40, pq=<value optimized out>, postpq=0x8) at detect.c:1842
>> #3 0x00000000005c6547 in TmThreadsSlotVarRun (tv=0x4f72bb0,
>> p=0x1878e10, slot=<value optimized out>) at tm-threads.c:441
>> #4 0x00000000005c6744 in TmThreadsSlotVar (td=0x4f72bb0) at tm-threads.c:660
>> #5 0x00007ffff71399ca in start_thread (arg=<value optimized out>) at
>> pthread_create.c:300
>> #6 0x00007ffff6a4270d in clone () at
>> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
>> #7 0x0000000000000000 in ?? ()
>>
>>
>> On Tue, Sep 20, 2011 at 1:06 PM, Martin Holste <mcholste at gmail.com> wrote:
>>> Yes, I was running on the git HEAD as of the 18th. I'll give your rev a shot.
>>>
>>> ...
>>>
>>> Much better performance! Seeing in the neighborhood of only about
>>> 25-33% drop rate on peak traffic (2-3k sessions/sec). Not perfect,
>>> but far better than before. Get that code merged!
>>>
>>> However, the segfault I've seen with all recent Suricatas is still present:
>>>
>>> Program received signal SIGSEGV, Segmentation fault.
>>> [Switching to Thread 0x7fffedc02700 (LWP 29336)]
>>> *__GI___libc_free (mem=0x4000000087394a0) at malloc.c:3709
>>> 3709 malloc.c: No such file or directory.
>>> in malloc.c
>>> (gdb) bt
>>> #0 *__GI___libc_free (mem=0x4000000087394a0) at malloc.c:3709
>>> #1 0x00007ffff7bd786e in htp_tx_destroy (tx=0x1aac82c0) at
>>> htp_transaction.c:115
>>> #2 0x00007ffff7bd4eb2 in htp_conn_destroy (conn=0x19894020) at
>>> htp_connection.c:65
>>> #3 0x00007ffff7bd00f2 in htp_connp_destroy_all (connp=0x19893ec0) at
>>> htp_connection_parser.c:197
>>> #4 0x000000000062647a in HTPStateFree (state=<value optimized out>)
>>> at app-layer-htp.c:210
>>> #5 0x0000000000619ffe in AppLayerParserCleanupState
>>> (f=0x7fff87bed5c0) at app-layer-parser.c:1240
>>> #6 0x0000000000437e45 in FlowL7DataPtrFree (f=0x4000000087394a0) at flow.c:119
>>> #7 0x0000000000437eb2 in FlowClearMemory (f=0x7fff87bed5c0,
>>> proto_map=0 '\000') at flow.c:1406
>>> #8 0x0000000000438093 in FlowPrune (q=0x94b190, ts=0x7fffedc01630) at
>>> flow.c:336
>>> #9 0x0000000000439d17 in FlowPruneFlowQueue (td=<value optimized
>>> out>) at flow.c:355
>>> #10 FlowManagerThread (td=<value optimized out>) at flow.c:1060
>>> #11 0x00007ffff71399ca in start_thread (arg=<value optimized out>) at
>>> pthread_create.c:300
>>> #12 0x00007ffff6a4270d in clone () at
>>> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
>>> #13 0x0000000000000000 in ?? ()
>>>
>>>
>>> On Tue, Sep 20, 2011 at 12:52 AM, Anoop Saldanha <poonaatsoc at gmail.com> wrote:
>>>> You are running the lastest master?
>>>>
>>>> Can you reset your git head to this commit and check how the engine
>>>> behaves with the same ruleset?
>>>>
>>>> cc4e89fbe1477d47e50fd720127e7c28d0d512ba
>>>>
>>>> On Mon, Sep 19, 2011 at 10:43 PM, Martin Holste <mcholste at gmail.com> wrote:
>>>>> Now I've reduced the ruleset to a single rule--my heartbeat sig, and
>>>>> it's still missing almost all the time. It appears that there's a
>>>>> major bottleneck in the flow distributer somewhere if the system can't
>>>>> grep for a single stream on just 600 Mb/sec. Anyone else running
>>>>> heartbeat sigs and seeing the same thing?
>>>>>
>>>>> On Mon, Sep 19, 2011 at 11:06 AM, Martin Holste <mcholste at gmail.com> wrote:
>>>>>> Ok, I'm giving that a shot, but so far that doesn't seem to have
>>>>>> improved things. Right now, it looks like the system is missing a ton
>>>>>> of heartbeats, so it's definitely not detecting everything even though
>>>>>> all the drop counters are zero. I'm running just 3k signatures on
>>>>>> about 600 Mb/sec of HTTP on 8 CPU/16 GB system.
>>>>>>
>>>>>> On Mon, Sep 19, 2011 at 10:48 AM, Victor Julien <victor at inliniac.net> wrote:
>>>>>>> On 09/19/2011 05:43 PM, Martin Holste wrote:
>>>>>>>> I've got memcap at 4GB and max_sessions is 256k by default. I'm
>>>>>>>
>>>>>>> You may want to try setting it a bit lower than the 4GB max, like 3.5GB
>>>>>>> or so. I think I've seen at least one occasion where it didn't behave
>>>>>>> properly with the max setting. Something we need to look into still.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Victor
>>>>>>>
>>>>>>>> having better luck now with more drastic emergency flow pruning:
>>>>>>>>
>>>>>>>> flow:
>>>>>>>> #memcap: 33554432
>>>>>>>> memcap: 4294967295
>>>>>>>> #hash_size: 65536
>>>>>>>> hash_size: 268435456
>>>>>>>> prealloc: 10000
>>>>>>>> emergency_recovery: 40 #30
>>>>>>>> prune_flows: 500 #5
>>>>>>>>
>>>>>>>> flow-timeouts:
>>>>>>>> default:
>>>>>>>> new: 1 # 30
>>>>>>>> established: 10 #300
>>>>>>>> closed: 0
>>>>>>>> emergency_new: 1 #10
>>>>>>>> emergency_established: 1 #100
>>>>>>>> emergency_closed: 0
>>>>>>>> tcp:
>>>>>>>> new: 1 #60
>>>>>>>> established: 10 #3600
>>>>>>>> closed: 120
>>>>>>>> emergency_new: 1 #10
>>>>>>>> emergency_established: 1 #300
>>>>>>>> emergency_closed: 20
>>>>>>>> udp:
>>>>>>>> new: 1 #30
>>>>>>>> established: 1 #300
>>>>>>>> emergency_new: 1 #10
>>>>>>>> emergency_established: 1 #100
>>>>>>>> icmp:
>>>>>>>> new: 1 #30
>>>>>>>> established: 1 #300
>>>>>>>> emergency_new: 1 #10
>>>>>>>> emergency_established: 1 #100
>>>>>>>>
>>>>>>>> I'm not yet sure how this will affect detection, but prior to this,
>>>>>>>> most new flows were being discarded. This policy should favor new
>>>>>>>> flows at the expense of old flows, which for malware detection should
>>>>>>>> be desired.
>>>>>>>>
>>>>>>>> On Mon, Sep 19, 2011 at 10:30 AM, Anoop Saldanha <poonaatsoc at gmail.com> wrote:
>>>>>>>>> stream:
>>>>>>>>> memcap: 33554432 # 32mb
>>>>>>>>>
>>>>>>>>> At the same time, you might also want to set max_sessions to something
>>>>>>>>> bigger. We default to 256k. You can try a bigger no and see how that
>>>>>>>>> works out
>>>>>>>>>
>>>>>>>>> On Mon, Sep 19, 2011 at 8:07 PM, Martin Holste <mcholste at gmail.com> wrote:
>>>>>>>>>> I'm seeing a ton of tcp.ssn_memcap_drop in my stats.log. Which memcap
>>>>>>>>>> do I need to tweak to decrease these drops? I've already set them all
>>>>>>>>>> to 4GB.
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Oisf-devel mailing list
>>>>>>>>>> Oisf-devel at openinfosecfoundation.org
>>>>>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Anoop Saldanha
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Oisf-devel mailing list
>>>>>>>> Oisf-devel at openinfosecfoundation.org
>>>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> ---------------------------------------------
>>>>>>> Victor Julien
>>>>>>> http://www.inliniac.net/
>>>>>>> PGP: http://www.inliniac.net/victorjulien.asc
>>>>>>> ---------------------------------------------
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Oisf-devel mailing list
>>>>>>> Oisf-devel at openinfosecfoundation.org
>>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> Oisf-devel mailing list
>>>>> Oisf-devel at openinfosecfoundation.org
>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Anoop Saldanha
>>>>
>>>
>>
>
More information about the Oisf-devel
mailing list