[Oisf-devel] suricata 1.3.4 coredump caused by segfault
Anoop Saldanha
anoopsaldanha at gmail.com
Tue Dec 11 05:35:55 UTC 2012
The hsbd bug reported has been fixed and should be out soon.
64 bit and more ram along with it is preferable. The timeouts should
help as well. Maybe you can profile your traffic's flow life over a
period and set the values accordingly.
On Tue, Dec 11, 2012 at 10:52 AM, xbadou xbadou <xbadou at gmail.com> wrote:
>
>
> I can be sure that the problem is caused by insufficient memory now.
>
>
>
> Maybe I will need to buy more RAM and upgrade to 64bit linux os.
>
>
>
> I also tried to modify the settings in suricata.yaml.
>
> When I add ‘max_sessions’ and set it to a small value such as 10000, I found
> the memory usage is not very large.
>
>
>
> I also noticed that there are some time-out settings in the ‘flow’ section,
> I think the default value may be very large to me. I think these values may
> also help to reduce the memory usage.
>
>
>
> Thank you all for helping me.
>
>
>
>
>
>
> On Mon, Dec 10, 2012 at 2:55 PM, Anoop Saldanha <anoopsaldanha at gmail.com>
> wrote:
>>
>> I see the bug for the hsbd crash. Low memory conditions probably has
>> realloc failing and returning NULL, but we don't reset the length to
>> 0.
>>
>> The second crash in libhtp probably falls in the low memory
>> environment as well I guess.
>>
>> On Mon, Dec 10, 2012 at 11:46 AM, xbadou xbadou <xbadou at gmail.com> wrote:
>> > Hi,
>> >
>> >
>> > I’m sorry that the problem seems still exists.
>> >
>> >
>> >
>> > Sometimes suricata works well, but sometimes it segfault quickly.
>> >
>> > In one test, when memory use up to about 63%, it crashed.
>> >
>> >
>> >
>> > The following is the result calling the command every 10s after suricata
>> > starts.
>> >
>> > Command: ps aux|grep /usr/bin/suricata|grep -v grep
>> >
>> >
>> >
>> > root 7775 91.1 2.5 84420 79976 ? Rs 13:17 0:06 ...
>> >
>> > root 7775 97.6 2.6 84552 80156 ? Rs 13:17 0:07 ...
>> >
>> > root 7775 96.3 5.2 166128 161792 ? Rs 13:17 0:16 ...
>> >
>> > root 7775 98.9 5.2 166128 161792 ? Rs 13:17 0:17 ...
>> >
>> > root 7775 97.7 6.6 208432 204092 ? Rs 13:17 0:26 ...
>> >
>> > root 7775 99.3 6.8 216204 211900 ? Rs 13:17 0:27 ...
>> >
>> > root 7775 90.1 25.9 916448 799844 ? Ssl 13:17 0:33 ...
>> >
>> > root 7775 89.2 26.1 930748 805284 ? Ssl 13:17 0:33 ...
>> >
>> > root 7775 81.6 27.5 1019852 848776 ? Ssl 13:17 0:38 ...
>> >
>> > root 7775 83.5 27.8 1038740 858592 ? Ssl 13:17 0:40 ...
>> >
>> > root 7775 80.0 29.5 1139304 910580 ? Ssl 13:17 0:45 ...
>> >
>> > root 7775 79.5 29.8 1156028 918168 ? Ssl 13:17 0:46 ...
>> >
>> > root 7775 74.2 31.3 1254744 966068 ? Ssl 13:17 0:49 ...
>> >
>> > root 7775 73.9 31.5 1268396 972692 ? Ssl 13:17 0:50 ...
>> >
>> > root 7775 71.2 33.0 1363408 1018400 ? Ssl 13:17 0:54 ...
>> >
>> > root 7775 71.0 33.2 1378684 1025360 ? Ssl 13:17 0:55 ...
>> >
>> > root 7775 67.6 34.0 1421352 1048824 ? Ssl 13:17 0:58 ...
>> >
>> > root 7775 67.2 34.1 1426892 1050720 ? Ssl 13:17 0:59 ...
>> >
>> > root 7775 63.4 35.0 1503220 1080580 ? Ssl 13:17 1:01 ...
>> >
>> > root 7775 63.1 35.2 1517584 1086492 ? Ssl 13:17 1:01 ...
>> >
>> > root 7775 60.0 36.5 1604040 1124600 ? Ssl 13:17 1:04 ...
>> >
>> > root 7775 59.9 36.6 1616372 1130352 ? Rsl 13:17 1:04 ...
>> >
>> > root 7775 57.2 37.8 1699944 1167504 ? Ssl 13:17 1:06 ...
>> >
>> > root 7775 57.6 38.1 1715524 1174520 ? Ssl 13:17 1:07 ...
>> >
>> > root 7775 56.4 39.5 1809116 1216956 ? Ssl 13:17 1:11 ...
>> >
>> > root 7775 56.4 39.7 1826124 1224640 ? Ssl 13:17 1:12 ...
>> >
>> > root 7775 55.0 40.9 1912060 1262440 ? Ssl 13:17 1:15 ...
>> >
>> > root 7775 54.9 41.1 1926992 1268452 ? Rsl 13:17 1:15 ...
>> >
>> > root 7775 53.4 42.3 2007988 1303280 ? Ssl 13:17 1:18 ...
>> >
>> > root 7775 53.6 42.5 2020344 1309912 ? Ssl 13:17 1:19 ...
>> >
>> > root 7775 52.2 43.4 2084536 1340084 ? Ssl 13:17 1:22 ...
>> >
>> > root 7775 52.2 43.6 2093808 1344668 ? Ssl 13:17 1:22 ...
>> >
>> > root 7775 51.7 44.2 2126592 1362472 ? Rsl 13:17 1:26 ...
>> >
>> > root 7775 52.0 44.3 2131712 1365428 ? Ssl 13:17 1:27 ...
>> >
>> > root 7775 52.3 44.7 2158824 1379444 ? Ssl 13:17 1:32 ...
>> >
>> > root 7775 52.6 44.8 2162580 1380940 ? Rsl 13:17 1:33 ...
>> >
>> > root 7775 52.2 45.2 2201176 1395092 ? Ssl 13:17 1:37 ...
>> >
>> > root 7775 52.2 45.3 2210392 1398228 ? Ssl 13:17 1:38 ...
>> >
>> > root 7775 52.1 45.9 2260568 1415472 ? Ssl 13:17 1:42 ...
>> >
>> > root 7775 53.0 46.8 2268760 1444184 ? Ssl 13:17 1:45 ...
>> >
>> > root 7775 53.9 47.3 2310744 1459568 ? Ssl 13:17 1:51 ...
>> >
>> > root 7775 53.9 47.5 2326104 1466228 ? Ssl 13:17 1:52 ...
>> >
>> > root 7775 54.0 48.6 2398808 1499000 ? Ssl 13:17 1:57 ...
>> >
>> > root 7775 54.2 48.8 2414168 1505540 ? Rsl 13:17 1:58 ...
>> >
>> > root 7775 53.9 50.1 2500184 1544792 ? Ssl 13:17 2:03 ...
>> >
>> > root 7775 54.6 51.1 2573912 1576212 ? Ssl 13:17 2:10 ...
>> >
>> > root 7775 54.7 51.2 2583128 1580168 ? Ssl 13:17 2:15 ...
>> >
>> > root 7775 53.8 51.6 2618968 1590756 ? Ssl 13:17 2:18 ...
>> >
>> > root 7775 53.1 52.4 2675288 1614728 ? Ssl 13:17 2:22 ...
>> >
>> > root 7775 52.4 53.4 2739800 1645236 ? Ssl 13:17 2:25 ...
>> >
>> > root 7775 51.8 54.6 2826840 1683572 ? Ssl 13:17 2:29 ...
>> >
>> > root 7775 51.4 56.0 2931288 1725660 ? Ssl 13:17 2:33 ...
>> >
>> > root 7775 51.0 56.7 2983512 1748040 ? Ssl 13:17 2:37 ...
>> >
>> > root 7775 51.0 57.4 3035736 1770964 ? Ssl 13:17 2:42 ...
>> >
>> > root 7775 51.1 58.4 3102296 1801280 ? Ssl 13:17 2:47 ...
>> >
>> > root 7775 51.3 59.4 3145724 1831824 ? Ssl 13:17 2:53 ...
>> >
>> > root 7775 51.4 59.6 3145724 1837372 ? Ssl 13:17 2:59 ...
>> >
>> > root 7775 51.9 59.8 3145724 1844756 ? Ssl 13:17 3:05 ...
>> >
>> > root 7775 51.8 60.1 3145724 1854628 ? Ssl 13:17 3:11 ...
>> >
>> > root 7775 52.6 60.6 3145724 1868488 ? Ssl 13:17 3:19 ...
>> >
>> > root 7775 53.1 61.0 3145724 1880768 ? Ssl 13:17 3:26 ...
>> >
>> > root 7775 53.5 61.3 3145724 1890252 ? Ssl 13:17 3:33 ...
>> >
>> > root 7775 53.3 61.6 3145724 1900852 ? Ssl 13:17 3:38 ...
>> >
>> > root 7775 54.2 62.2 3141052 1918648 ? Ssl 13:17 3:47 ...
>> >
>> > root 7775 54.3 62.3 3145396 1920420 ? Ssl 13:17 3:53 ...
>> >
>> > root 7775 55.0 62.3 3145724 1922368 ? Ssl 13:17 4:01 ...
>> >
>> > root 7775 55.7 62.4 3145724 1923980 ? Ssl 13:17 4:10 ...
>> >
>> > root 7775 56.1 62.5 3145192 1926960 ? Ssl 13:17 4:17 ...
>> >
>> > root 7775 56.8 62.5 3142088 1926220 ? Ssl 13:17 4:26 ...
>> >
>> > root 7775 56.7 62.6 3145724 1930768 ? Ssl 13:17 4:31 ...
>> >
>> > root 7775 56.7 62.7 3145724 1932628 ? Ssl 13:17 4:37 ...
>> >
>> > root 7775 56.5 62.8 3145724 1937868 ? Ssl 13:17 4:42 ...
>> >
>> > root 7775 56.4 63.4 3145724 1955808 ? Ssl 13:17 4:47 ...
>> >
>> > root 7775 55.6 63.5 3145724 1956508 ? Dsl 13:17 4:48 ...
>> >
>> > root 7775 54.8 63.5 3145724 1956508 ? Dsl 13:17 4:50 ...
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > In some other case, I try to read the source code where segfault
>> > occurred.
>> > For example:
>> >
>> >
>> >
>> > #0 0x0810b769 in SCACSearch (mpm_ctx=0xd743538,
>> > mpm_thread_ctx=0x15908cf4,
>> > pmq=0x15908d0c, buf=0x0, buflen=13915)
>> >
>> > at util-mpm-ac.c:1232
>> >
>> > #1 0x08090057 in HttpServerBodyPatternSearch (det_ctx=0x15908ca0,
>> > body=0x0,
>> > body_len=<value optimized out>, flags=8 '\b')
>> >
>> > at detect-engine-mpm.c:359
>> >
>> > #2 0x0809cc0a in DetectEngineRunHttpServerBodyMpm (de_ctx=0xa6f4ba8,
>> > det_ctx=0x15908ca0, f=0x68e3c0d0, htp_state=0x9a174f18,
>> >
>> > flags=8 '\b') at detect-engine-hsbd.c:248
>> >
>> > #3 0x08077a9a in SigMatchSignatures (th_v=0xb5900878, de_ctx=0xa6f4ba8,
>> > det_ctx=0x15908ca0, p=0x9db4f90) at detect.c:1264
>> >
>> > #4 0x08077b82 in Detect (tv=0xb5900878, p=0x9db4f90, data=0x15908ca0,
>> > pq=0xb59009f0, postpq=0x0) at detect.c:1995
>> >
>> > #5 0x0814e244 in TmThreadsSlotVarRun (tv=0xb5900878, p=0x9db4f90,
>> > slot=0xb59008f8) at tm-threads.c:508
>> >
>> > #6 0x0814e4fc in TmThreadsSlotVar (td=0xb5900878) at tm-threads.c:732
>> >
>> > #7 0xb78174c0 in start_thread () from /lib/i686/cmov/libpthread.so.0
>> >
>> > #8 0xb774e84e in clone () from /lib/i686/cmov/libc.so.6
>> >
>> >
>> >
>> > I find that the pointer of ‘buf’ is 0x0. Then I read the code in
>> > detect-engine-hsbd.c:248.
>> >
>> >
>> >
>> > int DetectEngineRunHttpServerBodyMpm(DetectEngineCtx *de_ctx,
>> >
>> > DetectEngineThreadCtx *det_ctx,
>> > Flow
>> > *f,
>> >
>> > HtpState *htp_state, uint8_t flags)
>> >
>> > {
>> >
>> > int i;
>> >
>> > uint32_t cnt = 0;
>> >
>> >
>> >
>> > FLOWLOCK_WRLOCK(f);
>> >
>> > DetectEngineBufferHttpServerBodies(de_ctx, det_ctx, f, htp_state,
>> > flags);
>> >
>> > FLOWLOCK_UNLOCK(f);
>> >
>> >
>> >
>> > if (det_ctx->hsbd != NULL && det_ctx->hsbd_buffers_list_len) {
>> >
>> > for (i = 0; i < det_ctx->hsbd_buffers_list_len; i++) {
>> >
>> > if (det_ctx->hsbd[i].buffer_len == 0)
>> >
>> > continue;
>> >
>> >
>> >
>> > cnt += HttpServerBodyPatternSearch(det_ctx,
>> >
>> > det_ctx->hsbd[i].buffer,
>> >
>> > det_ctx->hsbd[i].buffer_len,
>> >
>> > flags);
>> >
>> > }
>> >
>> > }
>> >
>> >
>> >
>> > return cnt;
>> >
>> > }
>> >
>> >
>> >
>> > In the source, we just check det_ctx->hsbd[i].buffer_len here but
>> > det_ctx->hsbd[i].buffer is NULL. That results the segfault. I think this
>> > may
>> > be a problem. But I am new to suricata. I don’t know why
>> > det_ctx->hsbd[i].buffer is NULL here.
>> >
>> >
>> >
>> >
>> >
>> > In my test network, I think something different is that we have some
>> > Network
>> > Vulnerability System running. For example, nessus, IBM AppScan, and WVS.
>> > Sometimes we get alert logs very frequently.
>> >
>> >
>> >
>> > Hope it helps.
>> >
>> >
>> >
>> > Finally, the two other core dump backtrace:
>> >
>> >
>> >
>> > #0 0x0809ceb6 in DetectEngineBufferHttpHeaders (det_ctx=0xb6e01e98,
>> > f=<value optimized out>, htp_state=0xa08fc198, flags=4 '\004')
>> >
>> > at detect-engine-hhd.c:144
>> >
>> > #1 0x0809d817 in DetectEngineRunHttpHeaderMpm (det_ctx=0xb6e01e98,
>> > f=0x55f14010, htp_state=0xa08fc198, flags=4 '\004')
>> >
>> > at detect-engine-hhd.c:200
>> >
>> > #2 0x08077938 in SigMatchSignatures (th_v=0x1097ee98, de_ctx=0xb2fdba8,
>> > det_ctx=0xb6e01e98, p=0xa57f0c0) at detect.c:1280
>> >
>> > #3 0x08077b82 in Detect (tv=0x1097ee98, p=0xa57f0c0, data=0xb6e01e98,
>> > pq=0xb6e00768, postpq=0x0) at detect.c:1995
>> >
>> > #4 0x0814e244 in TmThreadsSlotVarRun (tv=0x1097ee98, p=0xa57f0c0,
>> > slot=0xb6e00670) at tm-threads.c:508
>> >
>> > #5 0x0814e4fc in TmThreadsSlotVar (td=0x1097ee98) at tm-threads.c:732
>> >
>> > #6 0xb775c4c0 in start_thread () from /lib/i686/cmov/libpthread.so.0
>> >
>> > #7 0xb769384e in clone () from /lib/i686/cmov/libc.so.6
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > #0 0xb7771cea in htp_normalize_parsed_uri () from
>> > /usr/lib/libhtp-0.2.so.1
>> >
>> > #1 0xb777387e in htp_connp_REQ_LINE () from /usr/lib/libhtp-0.2.so.1
>> >
>> > #2 0xb7773106 in htp_connp_req_data () from /usr/lib/libhtp-0.2.so.1
>> >
>> > #3 0x08186332 in HTPHandleRequestData (f=0x3cdb6588,
>> > htp_state=0xa0f5d020,
>> >
>> > pstate=0xa0f5cff8,
>> >
>> > input=0xb2ffc8e0 "GET /images HTTP/1.0\r\nAccept-Charset:
>> > iso-8859-1,utf-8;q
>> > =0.9,*;q=0.1\r\nAccept-Language: en\r\nConnection: Close\r\nUser-Agent:
>> > Mozilla/ 4.0
>> > (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)\r\nPragma:
>> > no-ca"...,
>> >
>> > input_len=284, local_data=0x0, output=0xb2ffc800) at
>> > app-layer-htp.c:667
>> >
>> > #4 0x0817fbb6 in AppLayerDoParse (local_data=0x0, f=0x3cdb6588,
>> >
>> > app_layer_state=0xa0f5d020, parser_state=0xa0f5cff8,
>> >
>> > input=0xb2ffc8e0 "GET /images HTTP/1.0\r\nAccept-Charset:
>> > iso-8859-1,utf-8;q
>> > =0.9,*;q=0.1\r\nAccept-Language: en\r\nConnection: Close\r\nUser-Agent:
>> > Mozilla/ 4.0
>> > (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)\r\nPragma:
>> > no-ca"...,
>> >
>> > input_len=284, parser_idx=<value optimized out>, proto=1)
>> >
>> > at app-layer-parser.c:726
>> >
>> > #5 0x0817feb1 in AppLayerParse (local_data=0x0, f=0x3cdb6588,
>> >
>> > proto=<value optimized out>, flags=5 '\005',
>> >
>> > input=0xb2ffc8e0 "GET /images HTTP/1.0\r\nAccept-Charset:
>> > iso-8859-1,utf-8;q
>> > =0.9,*;q=0.1\r\nAccept-Language: en\r\nConnection: Close\r\nUser-Agent:
>> > Mozilla/ 4.0
>> > (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)\r\nPragma:
>> > no-ca"...,
>> >
>> > input_len=284) at app-layer-parser.c:935
>> >
>> > #6 0x0816d607 in StreamTcpReassembleAppLayer (tv=0xb3818a20,
>> >
>> > ---Type <return> to continue, or q <return> to quit---
>> >
>> > ra_ctx=0xb6e00b98, ssn=0x9e61bf28, stream=0x9e61bf64, p=0xa520638)
>> >
>> > at stream-tcp-reassemble.c:2942
>> >
>> > #7 0x0816d823 in StreamTcpReassembleHandleSegmentUpdateACK
>> > (tv=0xb3818a20,
>> >
>> > ra_ctx=0xb6e00b98, ssn=0x9e61bf28, stream=0x9e61bf64, p=0xa520638)
>> >
>> > at stream-tcp-reassemble.c:3310
>> >
>> > #8 0x0816f2ef in StreamTcpReassembleHandleSegment (tv=0xb3818a20,
>> >
>> > ra_ctx=0xb6e00b98, ssn=0x9e61bf28, stream=0x9e61bf2c, p=0xa520638,
>> >
>> > pq=0xb6e00470) at stream-tcp-reassemble.c:3384
>> >
>> > #9 0x0816701b in StreamTcpPacketStateEstablished (tv=0xb3818a20,
>> > p=0xa520638,
>> >
>> > stt=0xb6e00468, ssn=0x9e61bf28, pq=0xb6e00470) at stream-tcp.c:1779
>> >
>> > #10 0x0816a1de in StreamTcpPacket (tv=0xb3818a20, p=0xa520638,
>> > stt=0xb6e00468,
>> >
>> > pq=0xb3818ac0) at stream-tcp.c:3512
>> >
>> > #11 0x0816b40f in StreamTcp (tv=0xb3818a20, p=0xa520638,
>> > data=0xb6e00468,
>> >
>> > pq=0xb3818ac0, postpq=0xb3818b14) at stream-tcp.c:3752
>> >
>> > #12 0x0814e244 in TmThreadsSlotVarRun (tv=0xb3818a20, p=0xa520638,
>> >
>> > slot=0xb3818aa0) at tm-threads.c:508
>> >
>> > #13 0x0814e4fc in TmThreadsSlotVar (td=0xb3818a20) at tm-threads.c:732
>> >
>> > #14 0xb76f14c0 in start_thread () from /lib/i686/cmov/libpthread.so.0
>> >
>> > #15 0xb762884e in clone () from /lib/i686/cmov/libc.so.6
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Fri, Dec 7, 2012 at 5:13 PM, Victor Julien <victor at inliniac.net>
>> > wrote:
>> >>
>> >> On 12/05/2012 01:25 PM, xbadou xbadou wrote:
>> >> > Hi，
>> >> >
>> >> >
>> >> >
>> >> > It seems that 1.3-dev works well for a whole day. After it starts,
>> >> > the
>> >> > memory usage slowly increase. When the usage is above 80%, it remains
>> >> > unchanged for a long time.
>> >> >
>> >> >
>> >> >
>> >> > There isn’t any coredump file too. Everything is all right. But I
>> >> > find
>> >> > that there are some logs as follows.
>> >>
>> >> Great, thanks for reporting and testing! 1.3.5 was released yesterday
>> >> containing the fixes.
>> >>
>> >> Cheers,
>> >> Victor
>> >>
>> >> --
>> >> ---------------------------------------------
>> >> Victor Julien
>> >> http://www.inliniac.net/
>> >> PGP: http://www.inliniac.net/victorjulien.asc
>> >> ---------------------------------------------
>> >>
>> >
>> >
>> > _______________________________________________
>> > Suricata IDS Devel mailing list: oisf-devel at openinfosecfoundation.org
>> > Site: http://suricata-ids.org | Participate:
>> > http://suricata-ids.org/participate/
>> > List:
>> > https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>> > Redmine: https://redmine.openinfosecfoundation.org/
>>
>>
>>
>> --
>> Anoop Saldanha
>
>
--
Anoop Saldanha
More information about the Oisf-devel
mailing list