[Oisf-devel] suricata 1.3.4 coredump caused by segfault
xbadou xbadou
xbadou at gmail.com
Tue Dec 11 05:22:16 UTC 2012
I can be sure that the problem is caused by insufficient memory now.
Maybe I will need to buy more RAM and upgrade to 64bit linux os.
I also tried to modify the settings in suricata.yaml.
When I add ‘max_sessions’ and set it to a small value such as 10000, I
found the memory usage is not very large.
I also noticed that there are some time-out settings in the ‘flow’ section,
I think the default value may be very large to me. I think these values may
also help to reduce the memory usage.
Thank you all for helping me.
On Mon, Dec 10, 2012 at 2:55 PM, Anoop Saldanha <anoopsaldanha at gmail.com>wrote:
> I see the bug for the hsbd crash. Low memory conditions probably has
> realloc failing and returning NULL, but we don't reset the length to
> 0.
>
> The second crash in libhtp probably falls in the low memory
> environment as well I guess.
>
> On Mon, Dec 10, 2012 at 11:46 AM, xbadou xbadou <xbadou at gmail.com> wrote:
> > Hi,
> >
> >
> > I’m sorry that the problem seems still exists.
> >
> >
> >
> > Sometimes suricata works well, but sometimes it segfault quickly.
> >
> > In one test, when memory use up to about 63%, it crashed.
> >
> >
> >
> > The following is the result calling the command every 10s after suricata
> > starts.
> >
> > Command: ps aux|grep /usr/bin/suricata|grep -v grep
> >
> >
> >
> > root 7775 91.1 2.5 84420 79976 ? Rs 13:17 0:06 ...
> >
> > root 7775 97.6 2.6 84552 80156 ? Rs 13:17 0:07 ...
> >
> > root 7775 96.3 5.2 166128 161792 ? Rs 13:17 0:16 ...
> >
> > root 7775 98.9 5.2 166128 161792 ? Rs 13:17 0:17 ...
> >
> > root 7775 97.7 6.6 208432 204092 ? Rs 13:17 0:26 ...
> >
> > root 7775 99.3 6.8 216204 211900 ? Rs 13:17 0:27 ...
> >
> > root 7775 90.1 25.9 916448 799844 ? Ssl 13:17 0:33 ...
> >
> > root 7775 89.2 26.1 930748 805284 ? Ssl 13:17 0:33 ...
> >
> > root 7775 81.6 27.5 1019852 848776 ? Ssl 13:17 0:38 ...
> >
> > root 7775 83.5 27.8 1038740 858592 ? Ssl 13:17 0:40 ...
> >
> > root 7775 80.0 29.5 1139304 910580 ? Ssl 13:17 0:45 ...
> >
> > root 7775 79.5 29.8 1156028 918168 ? Ssl 13:17 0:46 ...
> >
> > root 7775 74.2 31.3 1254744 966068 ? Ssl 13:17 0:49 ...
> >
> > root 7775 73.9 31.5 1268396 972692 ? Ssl 13:17 0:50 ...
> >
> > root 7775 71.2 33.0 1363408 1018400 ? Ssl 13:17 0:54 ...
> >
> > root 7775 71.0 33.2 1378684 1025360 ? Ssl 13:17 0:55 ...
> >
> > root 7775 67.6 34.0 1421352 1048824 ? Ssl 13:17 0:58 ...
> >
> > root 7775 67.2 34.1 1426892 1050720 ? Ssl 13:17 0:59 ...
> >
> > root 7775 63.4 35.0 1503220 1080580 ? Ssl 13:17 1:01 ...
> >
> > root 7775 63.1 35.2 1517584 1086492 ? Ssl 13:17 1:01 ...
> >
> > root 7775 60.0 36.5 1604040 1124600 ? Ssl 13:17 1:04 ...
> >
> > root 7775 59.9 36.6 1616372 1130352 ? Rsl 13:17 1:04 ...
> >
> > root 7775 57.2 37.8 1699944 1167504 ? Ssl 13:17 1:06 ...
> >
> > root 7775 57.6 38.1 1715524 1174520 ? Ssl 13:17 1:07 ...
> >
> > root 7775 56.4 39.5 1809116 1216956 ? Ssl 13:17 1:11 ...
> >
> > root 7775 56.4 39.7 1826124 1224640 ? Ssl 13:17 1:12 ...
> >
> > root 7775 55.0 40.9 1912060 1262440 ? Ssl 13:17 1:15 ...
> >
> > root 7775 54.9 41.1 1926992 1268452 ? Rsl 13:17 1:15 ...
> >
> > root 7775 53.4 42.3 2007988 1303280 ? Ssl 13:17 1:18 ...
> >
> > root 7775 53.6 42.5 2020344 1309912 ? Ssl 13:17 1:19 ...
> >
> > root 7775 52.2 43.4 2084536 1340084 ? Ssl 13:17 1:22 ...
> >
> > root 7775 52.2 43.6 2093808 1344668 ? Ssl 13:17 1:22 ...
> >
> > root 7775 51.7 44.2 2126592 1362472 ? Rsl 13:17 1:26 ...
> >
> > root 7775 52.0 44.3 2131712 1365428 ? Ssl 13:17 1:27 ...
> >
> > root 7775 52.3 44.7 2158824 1379444 ? Ssl 13:17 1:32 ...
> >
> > root 7775 52.6 44.8 2162580 1380940 ? Rsl 13:17 1:33 ...
> >
> > root 7775 52.2 45.2 2201176 1395092 ? Ssl 13:17 1:37 ...
> >
> > root 7775 52.2 45.3 2210392 1398228 ? Ssl 13:17 1:38 ...
> >
> > root 7775 52.1 45.9 2260568 1415472 ? Ssl 13:17 1:42 ...
> >
> > root 7775 53.0 46.8 2268760 1444184 ? Ssl 13:17 1:45 ...
> >
> > root 7775 53.9 47.3 2310744 1459568 ? Ssl 13:17 1:51 ...
> >
> > root 7775 53.9 47.5 2326104 1466228 ? Ssl 13:17 1:52 ...
> >
> > root 7775 54.0 48.6 2398808 1499000 ? Ssl 13:17 1:57 ...
> >
> > root 7775 54.2 48.8 2414168 1505540 ? Rsl 13:17 1:58 ...
> >
> > root 7775 53.9 50.1 2500184 1544792 ? Ssl 13:17 2:03 ...
> >
> > root 7775 54.6 51.1 2573912 1576212 ? Ssl 13:17 2:10 ...
> >
> > root 7775 54.7 51.2 2583128 1580168 ? Ssl 13:17 2:15 ...
> >
> > root 7775 53.8 51.6 2618968 1590756 ? Ssl 13:17 2:18 ...
> >
> > root 7775 53.1 52.4 2675288 1614728 ? Ssl 13:17 2:22 ...
> >
> > root 7775 52.4 53.4 2739800 1645236 ? Ssl 13:17 2:25 ...
> >
> > root 7775 51.8 54.6 2826840 1683572 ? Ssl 13:17 2:29 ...
> >
> > root 7775 51.4 56.0 2931288 1725660 ? Ssl 13:17 2:33 ...
> >
> > root 7775 51.0 56.7 2983512 1748040 ? Ssl 13:17 2:37 ...
> >
> > root 7775 51.0 57.4 3035736 1770964 ? Ssl 13:17 2:42 ...
> >
> > root 7775 51.1 58.4 3102296 1801280 ? Ssl 13:17 2:47 ...
> >
> > root 7775 51.3 59.4 3145724 1831824 ? Ssl 13:17 2:53 ...
> >
> > root 7775 51.4 59.6 3145724 1837372 ? Ssl 13:17 2:59 ...
> >
> > root 7775 51.9 59.8 3145724 1844756 ? Ssl 13:17 3:05 ...
> >
> > root 7775 51.8 60.1 3145724 1854628 ? Ssl 13:17 3:11 ...
> >
> > root 7775 52.6 60.6 3145724 1868488 ? Ssl 13:17 3:19 ...
> >
> > root 7775 53.1 61.0 3145724 1880768 ? Ssl 13:17 3:26 ...
> >
> > root 7775 53.5 61.3 3145724 1890252 ? Ssl 13:17 3:33 ...
> >
> > root 7775 53.3 61.6 3145724 1900852 ? Ssl 13:17 3:38 ...
> >
> > root 7775 54.2 62.2 3141052 1918648 ? Ssl 13:17 3:47 ...
> >
> > root 7775 54.3 62.3 3145396 1920420 ? Ssl 13:17 3:53 ...
> >
> > root 7775 55.0 62.3 3145724 1922368 ? Ssl 13:17 4:01 ...
> >
> > root 7775 55.7 62.4 3145724 1923980 ? Ssl 13:17 4:10 ...
> >
> > root 7775 56.1 62.5 3145192 1926960 ? Ssl 13:17 4:17 ...
> >
> > root 7775 56.8 62.5 3142088 1926220 ? Ssl 13:17 4:26 ...
> >
> > root 7775 56.7 62.6 3145724 1930768 ? Ssl 13:17 4:31 ...
> >
> > root 7775 56.7 62.7 3145724 1932628 ? Ssl 13:17 4:37 ...
> >
> > root 7775 56.5 62.8 3145724 1937868 ? Ssl 13:17 4:42 ...
> >
> > root 7775 56.4 63.4 3145724 1955808 ? Ssl 13:17 4:47 ...
> >
> > root 7775 55.6 63.5 3145724 1956508 ? Dsl 13:17 4:48 ...
> >
> > root 7775 54.8 63.5 3145724 1956508 ? Dsl 13:17 4:50 ...
> >
> >
> >
> >
> >
> >
> >
> > In some other case, I try to read the source code where segfault
> occurred.
> > For example:
> >
> >
> >
> > #0 0x0810b769 in SCACSearch (mpm_ctx=0xd743538,
> mpm_thread_ctx=0x15908cf4,
> > pmq=0x15908d0c, buf=0x0, buflen=13915)
> >
> > at util-mpm-ac.c:1232
> >
> > #1 0x08090057 in HttpServerBodyPatternSearch (det_ctx=0x15908ca0,
> body=0x0,
> > body_len=<value optimized out>, flags=8 '\b')
> >
> > at detect-engine-mpm.c:359
> >
> > #2 0x0809cc0a in DetectEngineRunHttpServerBodyMpm (de_ctx=0xa6f4ba8,
> > det_ctx=0x15908ca0, f=0x68e3c0d0, htp_state=0x9a174f18,
> >
> > flags=8 '\b') at detect-engine-hsbd.c:248
> >
> > #3 0x08077a9a in SigMatchSignatures (th_v=0xb5900878, de_ctx=0xa6f4ba8,
> > det_ctx=0x15908ca0, p=0x9db4f90) at detect.c:1264
> >
> > #4 0x08077b82 in Detect (tv=0xb5900878, p=0x9db4f90, data=0x15908ca0,
> > pq=0xb59009f0, postpq=0x0) at detect.c:1995
> >
> > #5 0x0814e244 in TmThreadsSlotVarRun (tv=0xb5900878, p=0x9db4f90,
> > slot=0xb59008f8) at tm-threads.c:508
> >
> > #6 0x0814e4fc in TmThreadsSlotVar (td=0xb5900878) at tm-threads.c:732
> >
> > #7 0xb78174c0 in start_thread () from /lib/i686/cmov/libpthread.so.0
> >
> > #8 0xb774e84e in clone () from /lib/i686/cmov/libc.so.6
> >
> >
> >
> > I find that the pointer of ‘buf’ is 0x0. Then I read the code in
> > detect-engine-hsbd.c:248.
> >
> >
> >
> > int DetectEngineRunHttpServerBodyMpm(DetectEngineCtx *de_ctx,
> >
> > DetectEngineThreadCtx *det_ctx, Flow
> > *f,
> >
> > HtpState *htp_state, uint8_t flags)
> >
> > {
> >
> > int i;
> >
> > uint32_t cnt = 0;
> >
> >
> >
> > FLOWLOCK_WRLOCK(f);
> >
> > DetectEngineBufferHttpServerBodies(de_ctx, det_ctx, f, htp_state,
> > flags);
> >
> > FLOWLOCK_UNLOCK(f);
> >
> >
> >
> > if (det_ctx->hsbd != NULL && det_ctx->hsbd_buffers_list_len) {
> >
> > for (i = 0; i < det_ctx->hsbd_buffers_list_len; i++) {
> >
> > if (det_ctx->hsbd[i].buffer_len == 0)
> >
> > continue;
> >
> >
> >
> > cnt += HttpServerBodyPatternSearch(det_ctx,
> >
> > det_ctx->hsbd[i].buffer,
> >
> > det_ctx->hsbd[i].buffer_len,
> >
> > flags);
> >
> > }
> >
> > }
> >
> >
> >
> > return cnt;
> >
> > }
> >
> >
> >
> > In the source, we just check det_ctx->hsbd[i].buffer_len here but
> > det_ctx->hsbd[i].buffer is NULL. That results the segfault. I think this
> may
> > be a problem. But I am new to suricata. I don’t know why
> > det_ctx->hsbd[i].buffer is NULL here.
> >
> >
> >
> >
> >
> > In my test network, I think something different is that we have some
> Network
> > Vulnerability System running. For example, nessus, IBM AppScan, and WVS.
> > Sometimes we get alert logs very frequently.
> >
> >
> >
> > Hope it helps.
> >
> >
> >
> > Finally, the two other core dump backtrace:
> >
> >
> >
> > #0 0x0809ceb6 in DetectEngineBufferHttpHeaders (det_ctx=0xb6e01e98,
> > f=<value optimized out>, htp_state=0xa08fc198, flags=4 '\004')
> >
> > at detect-engine-hhd.c:144
> >
> > #1 0x0809d817 in DetectEngineRunHttpHeaderMpm (det_ctx=0xb6e01e98,
> > f=0x55f14010, htp_state=0xa08fc198, flags=4 '\004')
> >
> > at detect-engine-hhd.c:200
> >
> > #2 0x08077938 in SigMatchSignatures (th_v=0x1097ee98, de_ctx=0xb2fdba8,
> > det_ctx=0xb6e01e98, p=0xa57f0c0) at detect.c:1280
> >
> > #3 0x08077b82 in Detect (tv=0x1097ee98, p=0xa57f0c0, data=0xb6e01e98,
> > pq=0xb6e00768, postpq=0x0) at detect.c:1995
> >
> > #4 0x0814e244 in TmThreadsSlotVarRun (tv=0x1097ee98, p=0xa57f0c0,
> > slot=0xb6e00670) at tm-threads.c:508
> >
> > #5 0x0814e4fc in TmThreadsSlotVar (td=0x1097ee98) at tm-threads.c:732
> >
> > #6 0xb775c4c0 in start_thread () from /lib/i686/cmov/libpthread.so.0
> >
> > #7 0xb769384e in clone () from /lib/i686/cmov/libc.so.6
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > #0 0xb7771cea in htp_normalize_parsed_uri () from
> /usr/lib/libhtp-0.2.so.1
> >
> > #1 0xb777387e in htp_connp_REQ_LINE () from /usr/lib/libhtp-0.2.so.1
> >
> > #2 0xb7773106 in htp_connp_req_data () from /usr/lib/libhtp-0.2.so.1
> >
> > #3 0x08186332 in HTPHandleRequestData (f=0x3cdb6588,
> htp_state=0xa0f5d020,
> >
> > pstate=0xa0f5cff8,
> >
> > input=0xb2ffc8e0 "GET /images HTTP/1.0\r\nAccept-Charset:
> > iso-8859-1,utf-8;q
> > =0.9,*;q=0.1\r\nAccept-Language: en\r\nConnection: Close\r\nUser-Agent:
> > Mozilla/ 4.0
> > (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)\r\nPragma: no-ca"...,
> >
> > input_len=284, local_data=0x0, output=0xb2ffc800) at
> app-layer-htp.c:667
> >
> > #4 0x0817fbb6 in AppLayerDoParse (local_data=0x0, f=0x3cdb6588,
> >
> > app_layer_state=0xa0f5d020, parser_state=0xa0f5cff8,
> >
> > input=0xb2ffc8e0 "GET /images HTTP/1.0\r\nAccept-Charset:
> > iso-8859-1,utf-8;q
> > =0.9,*;q=0.1\r\nAccept-Language: en\r\nConnection: Close\r\nUser-Agent:
> > Mozilla/ 4.0
> > (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)\r\nPragma: no-ca"...,
> >
> > input_len=284, parser_idx=<value optimized out>, proto=1)
> >
> > at app-layer-parser.c:726
> >
> > #5 0x0817feb1 in AppLayerParse (local_data=0x0, f=0x3cdb6588,
> >
> > proto=<value optimized out>, flags=5 '\005',
> >
> > input=0xb2ffc8e0 "GET /images HTTP/1.0\r\nAccept-Charset:
> > iso-8859-1,utf-8;q
> > =0.9,*;q=0.1\r\nAccept-Language: en\r\nConnection: Close\r\nUser-Agent:
> > Mozilla/ 4.0
> > (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)\r\nPragma: no-ca"...,
> >
> > input_len=284) at app-layer-parser.c:935
> >
> > #6 0x0816d607 in StreamTcpReassembleAppLayer (tv=0xb3818a20,
> >
> > ---Type <return> to continue, or q <return> to quit---
> >
> > ra_ctx=0xb6e00b98, ssn=0x9e61bf28, stream=0x9e61bf64, p=0xa520638)
> >
> > at stream-tcp-reassemble.c:2942
> >
> > #7 0x0816d823 in StreamTcpReassembleHandleSegmentUpdateACK
> (tv=0xb3818a20,
> >
> > ra_ctx=0xb6e00b98, ssn=0x9e61bf28, stream=0x9e61bf64, p=0xa520638)
> >
> > at stream-tcp-reassemble.c:3310
> >
> > #8 0x0816f2ef in StreamTcpReassembleHandleSegment (tv=0xb3818a20,
> >
> > ra_ctx=0xb6e00b98, ssn=0x9e61bf28, stream=0x9e61bf2c, p=0xa520638,
> >
> > pq=0xb6e00470) at stream-tcp-reassemble.c:3384
> >
> > #9 0x0816701b in StreamTcpPacketStateEstablished (tv=0xb3818a20,
> > p=0xa520638,
> >
> > stt=0xb6e00468, ssn=0x9e61bf28, pq=0xb6e00470) at stream-tcp.c:1779
> >
> > #10 0x0816a1de in StreamTcpPacket (tv=0xb3818a20, p=0xa520638,
> > stt=0xb6e00468,
> >
> > pq=0xb3818ac0) at stream-tcp.c:3512
> >
> > #11 0x0816b40f in StreamTcp (tv=0xb3818a20, p=0xa520638, data=0xb6e00468,
> >
> > pq=0xb3818ac0, postpq=0xb3818b14) at stream-tcp.c:3752
> >
> > #12 0x0814e244 in TmThreadsSlotVarRun (tv=0xb3818a20, p=0xa520638,
> >
> > slot=0xb3818aa0) at tm-threads.c:508
> >
> > #13 0x0814e4fc in TmThreadsSlotVar (td=0xb3818a20) at tm-threads.c:732
> >
> > #14 0xb76f14c0 in start_thread () from /lib/i686/cmov/libpthread.so.0
> >
> > #15 0xb762884e in clone () from /lib/i686/cmov/libc.so.6
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Fri, Dec 7, 2012 at 5:13 PM, Victor Julien <victor at inliniac.net>
> wrote:
> >>
> >> On 12/05/2012 01:25 PM, xbadou xbadou wrote:
> >> > Hi，
> >> >
> >> >
> >> >
> >> > It seems that 1.3-dev works well for a whole day. After it starts, the
> >> > memory usage slowly increase. When the usage is above 80%, it remains
> >> > unchanged for a long time.
> >> >
> >> >
> >> >
> >> > There isn’t any coredump file too. Everything is all right. But I find
> >> > that there are some logs as follows.
> >>
> >> Great, thanks for reporting and testing! 1.3.5 was released yesterday
> >> containing the fixes.
> >>
> >> Cheers,
> >> Victor
> >>
> >> --
> >> ---------------------------------------------
> >> Victor Julien
> >> http://www.inliniac.net/
> >> PGP: http://www.inliniac.net/victorjulien.asc
> >> ---------------------------------------------
> >>
> >
> >
> > _______________________________________________
> > Suricata IDS Devel mailing list: oisf-devel at openinfosecfoundation.org
> > Site: http://suricata-ids.org | Participate:
> > http://suricata-ids.org/participate/
> > List:
> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
> > Redmine: https://redmine.openinfosecfoundation.org/
>
>
>
> --
> Anoop Saldanha
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-devel/attachments/20121211/7e0757c8/attachment-0002.html>
More information about the Oisf-devel
mailing list