[Oisf-devel] suricata 1.3.4 coredump caused by segfault
Victor Julien
victor at inliniac.net
Tue Dec 11 11:34:01 UTC 2012
On 12/11/2012 06:35 AM, Anoop Saldanha wrote:
> The hsbd bug reported has been fixed and should be out soon.
This has been pushed to the git master. Please update to the latest
master-1.3.x branch and let us know how it goes!
Cheers,
Victor
> 64 bit and more ram along with it is preferable. The timeouts should
> help as well. Maybe you can profile your traffic's flow life over a
> period and set the values accordingly.
>
> On Tue, Dec 11, 2012 at 10:52 AM, xbadou xbadou <xbadou at gmail.com> wrote:
>>
>>
>> I can be sure that the problem is caused by insufficient memory now.
>>
>>
>>
>> Maybe I will need to buy more RAM and upgrade to 64bit linux os.
>>
>>
>>
>> I also tried to modify the settings in suricata.yaml.
>>
>> When I add ‘max_sessions’ and set it to a small value such as 10000, I found
>> the memory usage is not very large.
>>
>>
>>
>> I also noticed that there are some time-out settings in the ‘flow’ section,
>> I think the default value may be very large to me. I think these values may
>> also help to reduce the memory usage.
>>
>>
>>
>> Thank you all for helping me.
>>
>>
>>
>>
>>
>>
>> On Mon, Dec 10, 2012 at 2:55 PM, Anoop Saldanha <anoopsaldanha at gmail.com>
>> wrote:
>>>
>>> I see the bug for the hsbd crash. Low memory conditions probably has
>>> realloc failing and returning NULL, but we don't reset the length to
>>> 0.
>>>
>>> The second crash in libhtp probably falls in the low memory
>>> environment as well I guess.
>>>
>>> On Mon, Dec 10, 2012 at 11:46 AM, xbadou xbadou <xbadou at gmail.com> wrote:
>>>> Hi,
>>>>
>>>>
>>>> I’m sorry that the problem seems still exists.
>>>>
>>>>
>>>>
>>>> Sometimes suricata works well, but sometimes it segfault quickly.
>>>>
>>>> In one test, when memory use up to about 63%, it crashed.
>>>>
>>>>
>>>>
>>>> The following is the result calling the command every 10s after suricata
>>>> starts.
>>>>
>>>> Command: ps aux|grep /usr/bin/suricata|grep -v grep
>>>>
>>>>
>>>>
>>>> root 7775 91.1 2.5 84420 79976 ? Rs 13:17 0:06 ...
>>>>
>>>> root 7775 97.6 2.6 84552 80156 ? Rs 13:17 0:07 ...
>>>>
>>>> root 7775 96.3 5.2 166128 161792 ? Rs 13:17 0:16 ...
>>>>
>>>> root 7775 98.9 5.2 166128 161792 ? Rs 13:17 0:17 ...
>>>>
>>>> root 7775 97.7 6.6 208432 204092 ? Rs 13:17 0:26 ...
>>>>
>>>> root 7775 99.3 6.8 216204 211900 ? Rs 13:17 0:27 ...
>>>>
>>>> root 7775 90.1 25.9 916448 799844 ? Ssl 13:17 0:33 ...
>>>>
>>>> root 7775 89.2 26.1 930748 805284 ? Ssl 13:17 0:33 ...
>>>>
>>>> root 7775 81.6 27.5 1019852 848776 ? Ssl 13:17 0:38 ...
>>>>
>>>> root 7775 83.5 27.8 1038740 858592 ? Ssl 13:17 0:40 ...
>>>>
>>>> root 7775 80.0 29.5 1139304 910580 ? Ssl 13:17 0:45 ...
>>>>
>>>> root 7775 79.5 29.8 1156028 918168 ? Ssl 13:17 0:46 ...
>>>>
>>>> root 7775 74.2 31.3 1254744 966068 ? Ssl 13:17 0:49 ...
>>>>
>>>> root 7775 73.9 31.5 1268396 972692 ? Ssl 13:17 0:50 ...
>>>>
>>>> root 7775 71.2 33.0 1363408 1018400 ? Ssl 13:17 0:54 ...
>>>>
>>>> root 7775 71.0 33.2 1378684 1025360 ? Ssl 13:17 0:55 ...
>>>>
>>>> root 7775 67.6 34.0 1421352 1048824 ? Ssl 13:17 0:58 ...
>>>>
>>>> root 7775 67.2 34.1 1426892 1050720 ? Ssl 13:17 0:59 ...
>>>>
>>>> root 7775 63.4 35.0 1503220 1080580 ? Ssl 13:17 1:01 ...
>>>>
>>>> root 7775 63.1 35.2 1517584 1086492 ? Ssl 13:17 1:01 ...
>>>>
>>>> root 7775 60.0 36.5 1604040 1124600 ? Ssl 13:17 1:04 ...
>>>>
>>>> root 7775 59.9 36.6 1616372 1130352 ? Rsl 13:17 1:04 ...
>>>>
>>>> root 7775 57.2 37.8 1699944 1167504 ? Ssl 13:17 1:06 ...
>>>>
>>>> root 7775 57.6 38.1 1715524 1174520 ? Ssl 13:17 1:07 ...
>>>>
>>>> root 7775 56.4 39.5 1809116 1216956 ? Ssl 13:17 1:11 ...
>>>>
>>>> root 7775 56.4 39.7 1826124 1224640 ? Ssl 13:17 1:12 ...
>>>>
>>>> root 7775 55.0 40.9 1912060 1262440 ? Ssl 13:17 1:15 ...
>>>>
>>>> root 7775 54.9 41.1 1926992 1268452 ? Rsl 13:17 1:15 ...
>>>>
>>>> root 7775 53.4 42.3 2007988 1303280 ? Ssl 13:17 1:18 ...
>>>>
>>>> root 7775 53.6 42.5 2020344 1309912 ? Ssl 13:17 1:19 ...
>>>>
>>>> root 7775 52.2 43.4 2084536 1340084 ? Ssl 13:17 1:22 ...
>>>>
>>>> root 7775 52.2 43.6 2093808 1344668 ? Ssl 13:17 1:22 ...
>>>>
>>>> root 7775 51.7 44.2 2126592 1362472 ? Rsl 13:17 1:26 ...
>>>>
>>>> root 7775 52.0 44.3 2131712 1365428 ? Ssl 13:17 1:27 ...
>>>>
>>>> root 7775 52.3 44.7 2158824 1379444 ? Ssl 13:17 1:32 ...
>>>>
>>>> root 7775 52.6 44.8 2162580 1380940 ? Rsl 13:17 1:33 ...
>>>>
>>>> root 7775 52.2 45.2 2201176 1395092 ? Ssl 13:17 1:37 ...
>>>>
>>>> root 7775 52.2 45.3 2210392 1398228 ? Ssl 13:17 1:38 ...
>>>>
>>>> root 7775 52.1 45.9 2260568 1415472 ? Ssl 13:17 1:42 ...
>>>>
>>>> root 7775 53.0 46.8 2268760 1444184 ? Ssl 13:17 1:45 ...
>>>>
>>>> root 7775 53.9 47.3 2310744 1459568 ? Ssl 13:17 1:51 ...
>>>>
>>>> root 7775 53.9 47.5 2326104 1466228 ? Ssl 13:17 1:52 ...
>>>>
>>>> root 7775 54.0 48.6 2398808 1499000 ? Ssl 13:17 1:57 ...
>>>>
>>>> root 7775 54.2 48.8 2414168 1505540 ? Rsl 13:17 1:58 ...
>>>>
>>>> root 7775 53.9 50.1 2500184 1544792 ? Ssl 13:17 2:03 ...
>>>>
>>>> root 7775 54.6 51.1 2573912 1576212 ? Ssl 13:17 2:10 ...
>>>>
>>>> root 7775 54.7 51.2 2583128 1580168 ? Ssl 13:17 2:15 ...
>>>>
>>>> root 7775 53.8 51.6 2618968 1590756 ? Ssl 13:17 2:18 ...
>>>>
>>>> root 7775 53.1 52.4 2675288 1614728 ? Ssl 13:17 2:22 ...
>>>>
>>>> root 7775 52.4 53.4 2739800 1645236 ? Ssl 13:17 2:25 ...
>>>>
>>>> root 7775 51.8 54.6 2826840 1683572 ? Ssl 13:17 2:29 ...
>>>>
>>>> root 7775 51.4 56.0 2931288 1725660 ? Ssl 13:17 2:33 ...
>>>>
>>>> root 7775 51.0 56.7 2983512 1748040 ? Ssl 13:17 2:37 ...
>>>>
>>>> root 7775 51.0 57.4 3035736 1770964 ? Ssl 13:17 2:42 ...
>>>>
>>>> root 7775 51.1 58.4 3102296 1801280 ? Ssl 13:17 2:47 ...
>>>>
>>>> root 7775 51.3 59.4 3145724 1831824 ? Ssl 13:17 2:53 ...
>>>>
>>>> root 7775 51.4 59.6 3145724 1837372 ? Ssl 13:17 2:59 ...
>>>>
>>>> root 7775 51.9 59.8 3145724 1844756 ? Ssl 13:17 3:05 ...
>>>>
>>>> root 7775 51.8 60.1 3145724 1854628 ? Ssl 13:17 3:11 ...
>>>>
>>>> root 7775 52.6 60.6 3145724 1868488 ? Ssl 13:17 3:19 ...
>>>>
>>>> root 7775 53.1 61.0 3145724 1880768 ? Ssl 13:17 3:26 ...
>>>>
>>>> root 7775 53.5 61.3 3145724 1890252 ? Ssl 13:17 3:33 ...
>>>>
>>>> root 7775 53.3 61.6 3145724 1900852 ? Ssl 13:17 3:38 ...
>>>>
>>>> root 7775 54.2 62.2 3141052 1918648 ? Ssl 13:17 3:47 ...
>>>>
>>>> root 7775 54.3 62.3 3145396 1920420 ? Ssl 13:17 3:53 ...
>>>>
>>>> root 7775 55.0 62.3 3145724 1922368 ? Ssl 13:17 4:01 ...
>>>>
>>>> root 7775 55.7 62.4 3145724 1923980 ? Ssl 13:17 4:10 ...
>>>>
>>>> root 7775 56.1 62.5 3145192 1926960 ? Ssl 13:17 4:17 ...
>>>>
>>>> root 7775 56.8 62.5 3142088 1926220 ? Ssl 13:17 4:26 ...
>>>>
>>>> root 7775 56.7 62.6 3145724 1930768 ? Ssl 13:17 4:31 ...
>>>>
>>>> root 7775 56.7 62.7 3145724 1932628 ? Ssl 13:17 4:37 ...
>>>>
>>>> root 7775 56.5 62.8 3145724 1937868 ? Ssl 13:17 4:42 ...
>>>>
>>>> root 7775 56.4 63.4 3145724 1955808 ? Ssl 13:17 4:47 ...
>>>>
>>>> root 7775 55.6 63.5 3145724 1956508 ? Dsl 13:17 4:48 ...
>>>>
>>>> root 7775 54.8 63.5 3145724 1956508 ? Dsl 13:17 4:50 ...
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> In some other case, I try to read the source code where segfault
>>>> occurred.
>>>> For example:
>>>>
>>>>
>>>>
>>>> #0 0x0810b769 in SCACSearch (mpm_ctx=0xd743538,
>>>> mpm_thread_ctx=0x15908cf4,
>>>> pmq=0x15908d0c, buf=0x0, buflen=13915)
>>>>
>>>> at util-mpm-ac.c:1232
>>>>
>>>> #1 0x08090057 in HttpServerBodyPatternSearch (det_ctx=0x15908ca0,
>>>> body=0x0,
>>>> body_len=<value optimized out>, flags=8 '\b')
>>>>
>>>> at detect-engine-mpm.c:359
>>>>
>>>> #2 0x0809cc0a in DetectEngineRunHttpServerBodyMpm (de_ctx=0xa6f4ba8,
>>>> det_ctx=0x15908ca0, f=0x68e3c0d0, htp_state=0x9a174f18,
>>>>
>>>> flags=8 '\b') at detect-engine-hsbd.c:248
>>>>
>>>> #3 0x08077a9a in SigMatchSignatures (th_v=0xb5900878, de_ctx=0xa6f4ba8,
>>>> det_ctx=0x15908ca0, p=0x9db4f90) at detect.c:1264
>>>>
>>>> #4 0x08077b82 in Detect (tv=0xb5900878, p=0x9db4f90, data=0x15908ca0,
>>>> pq=0xb59009f0, postpq=0x0) at detect.c:1995
>>>>
>>>> #5 0x0814e244 in TmThreadsSlotVarRun (tv=0xb5900878, p=0x9db4f90,
>>>> slot=0xb59008f8) at tm-threads.c:508
>>>>
>>>> #6 0x0814e4fc in TmThreadsSlotVar (td=0xb5900878) at tm-threads.c:732
>>>>
>>>> #7 0xb78174c0 in start_thread () from /lib/i686/cmov/libpthread.so.0
>>>>
>>>> #8 0xb774e84e in clone () from /lib/i686/cmov/libc.so.6
>>>>
>>>>
>>>>
>>>> I find that the pointer of ‘buf’ is 0x0. Then I read the code in
>>>> detect-engine-hsbd.c:248.
>>>>
>>>>
>>>>
>>>> int DetectEngineRunHttpServerBodyMpm(DetectEngineCtx *de_ctx,
>>>>
>>>> DetectEngineThreadCtx *det_ctx,
>>>> Flow
>>>> *f,
>>>>
>>>> HtpState *htp_state, uint8_t flags)
>>>>
>>>> {
>>>>
>>>> int i;
>>>>
>>>> uint32_t cnt = 0;
>>>>
>>>>
>>>>
>>>> FLOWLOCK_WRLOCK(f);
>>>>
>>>> DetectEngineBufferHttpServerBodies(de_ctx, det_ctx, f, htp_state,
>>>> flags);
>>>>
>>>> FLOWLOCK_UNLOCK(f);
>>>>
>>>>
>>>>
>>>> if (det_ctx->hsbd != NULL && det_ctx->hsbd_buffers_list_len) {
>>>>
>>>> for (i = 0; i < det_ctx->hsbd_buffers_list_len; i++) {
>>>>
>>>> if (det_ctx->hsbd[i].buffer_len == 0)
>>>>
>>>> continue;
>>>>
>>>>
>>>>
>>>> cnt += HttpServerBodyPatternSearch(det_ctx,
>>>>
>>>> det_ctx->hsbd[i].buffer,
>>>>
>>>> det_ctx->hsbd[i].buffer_len,
>>>>
>>>> flags);
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> return cnt;
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> In the source, we just check det_ctx->hsbd[i].buffer_len here but
>>>> det_ctx->hsbd[i].buffer is NULL. That results the segfault. I think this
>>>> may
>>>> be a problem. But I am new to suricata. I don’t know why
>>>> det_ctx->hsbd[i].buffer is NULL here.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> In my test network, I think something different is that we have some
>>>> Network
>>>> Vulnerability System running. For example, nessus, IBM AppScan, and WVS.
>>>> Sometimes we get alert logs very frequently.
>>>>
>>>>
>>>>
>>>> Hope it helps.
>>>>
>>>>
>>>>
>>>> Finally, the two other core dump backtrace:
>>>>
>>>>
>>>>
>>>> #0 0x0809ceb6 in DetectEngineBufferHttpHeaders (det_ctx=0xb6e01e98,
>>>> f=<value optimized out>, htp_state=0xa08fc198, flags=4 '\004')
>>>>
>>>> at detect-engine-hhd.c:144
>>>>
>>>> #1 0x0809d817 in DetectEngineRunHttpHeaderMpm (det_ctx=0xb6e01e98,
>>>> f=0x55f14010, htp_state=0xa08fc198, flags=4 '\004')
>>>>
>>>> at detect-engine-hhd.c:200
>>>>
>>>> #2 0x08077938 in SigMatchSignatures (th_v=0x1097ee98, de_ctx=0xb2fdba8,
>>>> det_ctx=0xb6e01e98, p=0xa57f0c0) at detect.c:1280
>>>>
>>>> #3 0x08077b82 in Detect (tv=0x1097ee98, p=0xa57f0c0, data=0xb6e01e98,
>>>> pq=0xb6e00768, postpq=0x0) at detect.c:1995
>>>>
>>>> #4 0x0814e244 in TmThreadsSlotVarRun (tv=0x1097ee98, p=0xa57f0c0,
>>>> slot=0xb6e00670) at tm-threads.c:508
>>>>
>>>> #5 0x0814e4fc in TmThreadsSlotVar (td=0x1097ee98) at tm-threads.c:732
>>>>
>>>> #6 0xb775c4c0 in start_thread () from /lib/i686/cmov/libpthread.so.0
>>>>
>>>> #7 0xb769384e in clone () from /lib/i686/cmov/libc.so.6
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> #0 0xb7771cea in htp_normalize_parsed_uri () from
>>>> /usr/lib/libhtp-0.2.so.1
>>>>
>>>> #1 0xb777387e in htp_connp_REQ_LINE () from /usr/lib/libhtp-0.2.so.1
>>>>
>>>> #2 0xb7773106 in htp_connp_req_data () from /usr/lib/libhtp-0.2.so.1
>>>>
>>>> #3 0x08186332 in HTPHandleRequestData (f=0x3cdb6588,
>>>> htp_state=0xa0f5d020,
>>>>
>>>> pstate=0xa0f5cff8,
>>>>
>>>> input=0xb2ffc8e0 "GET /images HTTP/1.0\r\nAccept-Charset:
>>>> iso-8859-1,utf-8;q
>>>> =0.9,*;q=0.1\r\nAccept-Language: en\r\nConnection: Close\r\nUser-Agent:
>>>> Mozilla/ 4.0
>>>> (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)\r\nPragma:
>>>> no-ca"...,
>>>>
>>>> input_len=284, local_data=0x0, output=0xb2ffc800) at
>>>> app-layer-htp.c:667
>>>>
>>>> #4 0x0817fbb6 in AppLayerDoParse (local_data=0x0, f=0x3cdb6588,
>>>>
>>>> app_layer_state=0xa0f5d020, parser_state=0xa0f5cff8,
>>>>
>>>> input=0xb2ffc8e0 "GET /images HTTP/1.0\r\nAccept-Charset:
>>>> iso-8859-1,utf-8;q
>>>> =0.9,*;q=0.1\r\nAccept-Language: en\r\nConnection: Close\r\nUser-Agent:
>>>> Mozilla/ 4.0
>>>> (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)\r\nPragma:
>>>> no-ca"...,
>>>>
>>>> input_len=284, parser_idx=<value optimized out>, proto=1)
>>>>
>>>> at app-layer-parser.c:726
>>>>
>>>> #5 0x0817feb1 in AppLayerParse (local_data=0x0, f=0x3cdb6588,
>>>>
>>>> proto=<value optimized out>, flags=5 '\005',
>>>>
>>>> input=0xb2ffc8e0 "GET /images HTTP/1.0\r\nAccept-Charset:
>>>> iso-8859-1,utf-8;q
>>>> =0.9,*;q=0.1\r\nAccept-Language: en\r\nConnection: Close\r\nUser-Agent:
>>>> Mozilla/ 4.0
>>>> (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)\r\nPragma:
>>>> no-ca"...,
>>>>
>>>> input_len=284) at app-layer-parser.c:935
>>>>
>>>> #6 0x0816d607 in StreamTcpReassembleAppLayer (tv=0xb3818a20,
>>>>
>>>> ---Type <return> to continue, or q <return> to quit---
>>>>
>>>> ra_ctx=0xb6e00b98, ssn=0x9e61bf28, stream=0x9e61bf64, p=0xa520638)
>>>>
>>>> at stream-tcp-reassemble.c:2942
>>>>
>>>> #7 0x0816d823 in StreamTcpReassembleHandleSegmentUpdateACK
>>>> (tv=0xb3818a20,
>>>>
>>>> ra_ctx=0xb6e00b98, ssn=0x9e61bf28, stream=0x9e61bf64, p=0xa520638)
>>>>
>>>> at stream-tcp-reassemble.c:3310
>>>>
>>>> #8 0x0816f2ef in StreamTcpReassembleHandleSegment (tv=0xb3818a20,
>>>>
>>>> ra_ctx=0xb6e00b98, ssn=0x9e61bf28, stream=0x9e61bf2c, p=0xa520638,
>>>>
>>>> pq=0xb6e00470) at stream-tcp-reassemble.c:3384
>>>>
>>>> #9 0x0816701b in StreamTcpPacketStateEstablished (tv=0xb3818a20,
>>>> p=0xa520638,
>>>>
>>>> stt=0xb6e00468, ssn=0x9e61bf28, pq=0xb6e00470) at stream-tcp.c:1779
>>>>
>>>> #10 0x0816a1de in StreamTcpPacket (tv=0xb3818a20, p=0xa520638,
>>>> stt=0xb6e00468,
>>>>
>>>> pq=0xb3818ac0) at stream-tcp.c:3512
>>>>
>>>> #11 0x0816b40f in StreamTcp (tv=0xb3818a20, p=0xa520638,
>>>> data=0xb6e00468,
>>>>
>>>> pq=0xb3818ac0, postpq=0xb3818b14) at stream-tcp.c:3752
>>>>
>>>> #12 0x0814e244 in TmThreadsSlotVarRun (tv=0xb3818a20, p=0xa520638,
>>>>
>>>> slot=0xb3818aa0) at tm-threads.c:508
>>>>
>>>> #13 0x0814e4fc in TmThreadsSlotVar (td=0xb3818a20) at tm-threads.c:732
>>>>
>>>> #14 0xb76f14c0 in start_thread () from /lib/i686/cmov/libpthread.so.0
>>>>
>>>> #15 0xb762884e in clone () from /lib/i686/cmov/libc.so.6
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Dec 7, 2012 at 5:13 PM, Victor Julien <victor at inliniac.net>
>>>> wrote:
>>>>>
>>>>> On 12/05/2012 01:25 PM, xbadou xbadou wrote:
>>>>>> Hi，
>>>>>>
>>>>>>
>>>>>>
>>>>>> It seems that 1.3-dev works well for a whole day. After it starts,
>>>>>> the
>>>>>> memory usage slowly increase. When the usage is above 80%, it remains
>>>>>> unchanged for a long time.
>>>>>>
>>>>>>
>>>>>>
>>>>>> There isn’t any coredump file too. Everything is all right. But I
>>>>>> find
>>>>>> that there are some logs as follows.
>>>>>
>>>>> Great, thanks for reporting and testing! 1.3.5 was released yesterday
>>>>> containing the fixes.
>>>>>
>>>>> Cheers,
>>>>> Victor
>>>>>
>>>>> --
>>>>> ---------------------------------------------
>>>>> Victor Julien
>>>>> http://www.inliniac.net/
>>>>> PGP: http://www.inliniac.net/victorjulien.asc
>>>>> ---------------------------------------------
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Suricata IDS Devel mailing list: oisf-devel at openinfosecfoundation.org
>>>> Site: http://suricata-ids.org | Participate:
>>>> http://suricata-ids.org/participate/
>>>> List:
>>>> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>>> Redmine: https://redmine.openinfosecfoundation.org/
>>>
>>>
>>>
>>> --
>>> Anoop Saldanha
>>
>>
>
>
>
--
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------
More information about the Oisf-devel
mailing list