[Oisf-devel] suricata 1.3.4 coredump caused by segfault

xbadou xbadou xbadou at gmail.com
Mon Dec 10 06:16:45 UTC 2012 

Hi,


I’m sorry that the problem seems still exists.



Sometimes suricata works well, but sometimes it segfault quickly.

In one test, when memory use up to about 63%, it crashed.



The following is the result calling the command every 10s after suricata
starts.

Command:  ps aux|grep /usr/bin/suricata|grep -v grep



root 7775 91.1 2.5 84420 79976 ? Rs 13:17 0:06 ...

root 7775 97.6 2.6 84552 80156 ? Rs 13:17 0:07 ...

root 7775 96.3 5.2 166128 161792 ? Rs 13:17 0:16 ...

root 7775 98.9 5.2 166128 161792 ? Rs 13:17 0:17 ...

root 7775 97.7 6.6 208432 204092 ? Rs 13:17 0:26 ...

root 7775 99.3 6.8 216204 211900 ? Rs 13:17 0:27 ...

root 7775 90.1 25.9 916448 799844 ? Ssl 13:17 0:33 ...

root 7775 89.2 26.1 930748 805284 ? Ssl 13:17 0:33 ...

root 7775 81.6 27.5 1019852 848776 ? Ssl 13:17 0:38 ...

root 7775 83.5 27.8 1038740 858592 ? Ssl 13:17 0:40 ...

root 7775 80.0 29.5 1139304 910580 ? Ssl 13:17 0:45 ...

root 7775 79.5 29.8 1156028 918168 ? Ssl 13:17 0:46 ...

root 7775 74.2 31.3 1254744 966068 ? Ssl 13:17 0:49 ...

root 7775 73.9 31.5 1268396 972692 ? Ssl 13:17 0:50 ...

root 7775 71.2 33.0 1363408 1018400 ? Ssl 13:17 0:54 ...

root 7775 71.0 33.2 1378684 1025360 ? Ssl 13:17 0:55 ...

root 7775 67.6 34.0 1421352 1048824 ? Ssl 13:17 0:58 ...

root 7775 67.2 34.1 1426892 1050720 ? Ssl 13:17 0:59 ...

root 7775 63.4 35.0 1503220 1080580 ? Ssl 13:17 1:01 ...

root 7775 63.1 35.2 1517584 1086492 ? Ssl 13:17 1:01 ...

root 7775 60.0 36.5 1604040 1124600 ? Ssl 13:17 1:04 ...

root 7775 59.9 36.6 1616372 1130352 ? Rsl 13:17 1:04 ...

root 7775 57.2 37.8 1699944 1167504 ? Ssl 13:17 1:06 ...

root 7775 57.6 38.1 1715524 1174520 ? Ssl 13:17 1:07 ...

root 7775 56.4 39.5 1809116 1216956 ? Ssl 13:17 1:11 ...

root 7775 56.4 39.7 1826124 1224640 ? Ssl 13:17 1:12 ...

root 7775 55.0 40.9 1912060 1262440 ? Ssl 13:17 1:15 ...

root 7775 54.9 41.1 1926992 1268452 ? Rsl 13:17 1:15 ...

root 7775 53.4 42.3 2007988 1303280 ? Ssl 13:17 1:18 ...

root 7775 53.6 42.5 2020344 1309912 ? Ssl 13:17 1:19 ...

root 7775 52.2 43.4 2084536 1340084 ? Ssl 13:17 1:22 ...

root 7775 52.2 43.6 2093808 1344668 ? Ssl 13:17 1:22 ...

root 7775 51.7 44.2 2126592 1362472 ? Rsl 13:17 1:26 ...

root 7775 52.0 44.3 2131712 1365428 ? Ssl 13:17 1:27 ...

root 7775 52.3 44.7 2158824 1379444 ? Ssl 13:17 1:32 ...

root 7775 52.6 44.8 2162580 1380940 ? Rsl 13:17 1:33 ...

root 7775 52.2 45.2 2201176 1395092 ? Ssl 13:17 1:37 ...

root 7775 52.2 45.3 2210392 1398228 ? Ssl 13:17 1:38 ...

root 7775 52.1 45.9 2260568 1415472 ? Ssl 13:17 1:42 ...

root 7775 53.0 46.8 2268760 1444184 ? Ssl 13:17 1:45 ...

root 7775 53.9 47.3 2310744 1459568 ? Ssl 13:17 1:51 ...

root 7775 53.9 47.5 2326104 1466228 ? Ssl 13:17 1:52 ...

root 7775 54.0 48.6 2398808 1499000 ? Ssl 13:17 1:57 ...

root 7775 54.2 48.8 2414168 1505540 ? Rsl 13:17 1:58 ...

root 7775 53.9 50.1 2500184 1544792 ? Ssl 13:17 2:03 ...

root 7775 54.6 51.1 2573912 1576212 ? Ssl 13:17 2:10 ...

root 7775 54.7 51.2 2583128 1580168 ? Ssl 13:17 2:15 ...

root 7775 53.8 51.6 2618968 1590756 ? Ssl 13:17 2:18 ...

root 7775 53.1 52.4 2675288 1614728 ? Ssl 13:17 2:22 ...

root 7775 52.4 53.4 2739800 1645236 ? Ssl 13:17 2:25 ...

root 7775 51.8 54.6 2826840 1683572 ? Ssl 13:17 2:29 ...

root 7775 51.4 56.0 2931288 1725660 ? Ssl 13:17 2:33 ...

root 7775 51.0 56.7 2983512 1748040 ? Ssl 13:17 2:37 ...

root 7775 51.0 57.4 3035736 1770964 ? Ssl 13:17 2:42 ...

root 7775 51.1 58.4 3102296 1801280 ? Ssl 13:17 2:47 ...

root 7775 51.3 59.4 3145724 1831824 ? Ssl 13:17 2:53 ...

root 7775 51.4 59.6 3145724 1837372 ? Ssl 13:17 2:59 ...

root 7775 51.9 59.8 3145724 1844756 ? Ssl 13:17 3:05 ...

root 7775 51.8 60.1 3145724 1854628 ? Ssl 13:17 3:11 ...

root 7775 52.6 60.6 3145724 1868488 ? Ssl 13:17 3:19 ...

root 7775 53.1 61.0 3145724 1880768 ? Ssl 13:17 3:26 ...

root 7775 53.5 61.3 3145724 1890252 ? Ssl 13:17 3:33 ...

root 7775 53.3 61.6 3145724 1900852 ? Ssl 13:17 3:38 ...

root 7775 54.2 62.2 3141052 1918648 ? Ssl 13:17 3:47 ...

root 7775 54.3 62.3 3145396 1920420 ? Ssl 13:17 3:53 ...

root 7775 55.0 62.3 3145724 1922368 ? Ssl 13:17 4:01 ...

root 7775 55.7 62.4 3145724 1923980 ? Ssl 13:17 4:10 ...

root 7775 56.1 62.5 3145192 1926960 ? Ssl 13:17 4:17 ...

root 7775 56.8 62.5 3142088 1926220 ? Ssl 13:17 4:26 ...

root 7775 56.7 62.6 3145724 1930768 ? Ssl 13:17 4:31 ...

root 7775 56.7 62.7 3145724 1932628 ? Ssl 13:17 4:37 ...

root 7775 56.5 62.8 3145724 1937868 ? Ssl 13:17 4:42 ...

root 7775 56.4 63.4 3145724 1955808 ? Ssl 13:17 4:47 ...

root 7775 55.6 63.5 3145724 1956508 ? Dsl 13:17 4:48 ...

root 7775 54.8 *63.5* 3145724 1956508 ? Dsl 13:17 4:50 ...







In some other case, I try to read the source code where segfault occurred.
For example:



#0  0x0810b769 in SCACSearch (mpm_ctx=0xd743538, mpm_thread_ctx=0x15908cf4,
pmq=0x15908d0c,* buf=0x0, buflen=13915*)

    at util-mpm-ac.c:1232

#1  0x08090057 in HttpServerBodyPatternSearch (det_ctx=0x15908ca0, *body=0x0,
body_len=<value optimized out>*, flags=8 '\b')

    at detect-engine-mpm.c:359

#2  0x0809cc0a in DetectEngineRunHttpServerBodyMpm (de_ctx=0xa6f4ba8,
det_ctx=0x15908ca0, f=0x68e3c0d0, htp_state=0x9a174f18,

    flags=8 '\b') at detect-engine-hsbd.c:248

#3  0x08077a9a in SigMatchSignatures (th_v=0xb5900878, de_ctx=0xa6f4ba8,
det_ctx=0x15908ca0, p=0x9db4f90) at detect.c:1264

#4  0x08077b82 in Detect (tv=0xb5900878, p=0x9db4f90, data=0x15908ca0,
pq=0xb59009f0, postpq=0x0) at detect.c:1995

#5  0x0814e244 in TmThreadsSlotVarRun (tv=0xb5900878, p=0x9db4f90,
slot=0xb59008f8) at tm-threads.c:508

#6  0x0814e4fc in TmThreadsSlotVar (td=0xb5900878) at tm-threads.c:732

#7  0xb78174c0 in start_thread () from /lib/i686/cmov/libpthread.so.0

#8  0xb774e84e in clone () from /lib/i686/cmov/libc.so.6



I find that the pointer of ‘buf’ is 0x0.  Then I read the code in
detect-engine-hsbd.c:248.



int DetectEngineRunHttpServerBodyMpm(DetectEngineCtx *de_ctx,

                                     DetectEngineThreadCtx *det_ctx, Flow
*f,

                                     HtpState *htp_state, uint8_t flags)

{

    int i;

    uint32_t cnt = 0;



    FLOWLOCK_WRLOCK(f);

    DetectEngineBufferHttpServerBodies(de_ctx, det_ctx, f, htp_state,
flags);

    FLOWLOCK_UNLOCK(f);



    if (det_ctx->hsbd != NULL && det_ctx->hsbd_buffers_list_len) {

        for (i = 0; i < det_ctx->hsbd_buffers_list_len; i++) {

            if (det_ctx->hsbd[i].buffer_len == 0)

                continue;



            cnt += HttpServerBodyPatternSearch(det_ctx,

                    det_ctx->hsbd[i].buffer,

                    det_ctx->hsbd[i].buffer_len,

                    flags);

        }

    }



    return cnt;

}



In the source, we just check det_ctx->hsbd[i].buffer_len here but
det_ctx->hsbd[i].buffer is NULL. That results the segfault. I think this
may be a problem. But I am new to suricata. I don’t know why
det_ctx->hsbd[i].buffer
is NULL here.





In my test network, I think something different is that we have some
Network Vulnerability System running. For example, nessus, IBM AppScan, and
WVS. Sometimes we get alert logs very frequently.



Hope it helps.



Finally, the two other core dump backtrace:



#0  0x0809ceb6 in DetectEngineBufferHttpHeaders (det_ctx=0xb6e01e98,
f=<value optimized out>, htp_state=0xa08fc198, flags=4 '\004')

    at detect-engine-hhd.c:144

#1  0x0809d817 in DetectEngineRunHttpHeaderMpm (det_ctx=0xb6e01e98,
f=0x55f14010, htp_state=0xa08fc198, flags=4 '\004')

    at detect-engine-hhd.c:200

#2  0x08077938 in SigMatchSignatures (th_v=0x1097ee98, de_ctx=0xb2fdba8,
det_ctx=0xb6e01e98, p=0xa57f0c0) at detect.c:1280

#3  0x08077b82 in Detect (tv=0x1097ee98, p=0xa57f0c0, data=0xb6e01e98,
pq=0xb6e00768, postpq=0x0) at detect.c:1995

#4  0x0814e244 in TmThreadsSlotVarRun (tv=0x1097ee98, p=0xa57f0c0,
slot=0xb6e00670) at tm-threads.c:508

#5  0x0814e4fc in TmThreadsSlotVar (td=0x1097ee98) at tm-threads.c:732

#6  0xb775c4c0 in start_thread () from /lib/i686/cmov/libpthread.so.0

#7  0xb769384e in clone () from /lib/i686/cmov/libc.so.6









#0  0xb7771cea in htp_normalize_parsed_uri () from /usr/lib/libhtp-0.2.so.1

#1  0xb777387e in htp_connp_REQ_LINE () from /usr/lib/libhtp-0.2.so.1

#2  0xb7773106 in htp_connp_req_data () from /usr/lib/libhtp-0.2.so.1

#3  0x08186332 in HTPHandleRequestData (f=0x3cdb6588, htp_state=0xa0f5d020,

    pstate=0xa0f5cff8,

    input=0xb2ffc8e0 "GET /images HTTP/1.0\r\nAccept-Charset:
iso-8859-1,utf-8;q
=0.9,*;q=0.1\r\nAccept-Language: en\r\nConnection: Close\r\nUser-Agent:
Mozilla/                                                       4.0
(compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)\r\nPragma: no-ca"...,

    input_len=284, local_data=0x0, output=0xb2ffc800) at app-layer-htp.c:667

#4  0x0817fbb6 in AppLayerDoParse (local_data=0x0, f=0x3cdb6588,

    app_layer_state=0xa0f5d020, parser_state=0xa0f5cff8,

    input=0xb2ffc8e0 "GET /images HTTP/1.0\r\nAccept-Charset:
iso-8859-1,utf-8;q
=0.9,*;q=0.1\r\nAccept-Language: en\r\nConnection: Close\r\nUser-Agent:
Mozilla/                                                       4.0
(compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)\r\nPragma: no-ca"...,

    input_len=284, parser_idx=<value optimized out>, proto=1)

    at app-layer-parser.c:726

#5  0x0817feb1 in AppLayerParse (local_data=0x0, f=0x3cdb6588,

    proto=<value optimized out>, flags=5 '\005',

    input=0xb2ffc8e0 "GET /images HTTP/1.0\r\nAccept-Charset:
iso-8859-1,utf-8;q
=0.9,*;q=0.1\r\nAccept-Language: en\r\nConnection: Close\r\nUser-Agent:
Mozilla/                                                       4.0
(compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)\r\nPragma: no-ca"...,

    input_len=284) at app-layer-parser.c:935

#6  0x0816d607 in StreamTcpReassembleAppLayer (tv=0xb3818a20,

---Type <return> to continue, or q <return> to quit---

    ra_ctx=0xb6e00b98, ssn=0x9e61bf28, stream=0x9e61bf64, p=0xa520638)

    at stream-tcp-reassemble.c:2942

#7  0x0816d823 in StreamTcpReassembleHandleSegmentUpdateACK (tv=0xb3818a20,

    ra_ctx=0xb6e00b98, ssn=0x9e61bf28, stream=0x9e61bf64, p=0xa520638)

    at stream-tcp-reassemble.c:3310

#8  0x0816f2ef in StreamTcpReassembleHandleSegment (tv=0xb3818a20,

    ra_ctx=0xb6e00b98, ssn=0x9e61bf28, stream=0x9e61bf2c, p=0xa520638,

    pq=0xb6e00470) at stream-tcp-reassemble.c:3384

#9  0x0816701b in StreamTcpPacketStateEstablished (tv=0xb3818a20,
p=0xa520638,

    stt=0xb6e00468, ssn=0x9e61bf28, pq=0xb6e00470) at stream-tcp.c:1779

#10 0x0816a1de in StreamTcpPacket (tv=0xb3818a20, p=0xa520638,
stt=0xb6e00468,

    pq=0xb3818ac0) at stream-tcp.c:3512

#11 0x0816b40f in StreamTcp (tv=0xb3818a20, p=0xa520638, data=0xb6e00468,

    pq=0xb3818ac0, postpq=0xb3818b14) at stream-tcp.c:3752

#12 0x0814e244 in TmThreadsSlotVarRun (tv=0xb3818a20, p=0xa520638,

    slot=0xb3818aa0) at tm-threads.c:508

#13 0x0814e4fc in TmThreadsSlotVar (td=0xb3818a20) at tm-threads.c:732

#14 0xb76f14c0 in start_thread () from /lib/i686/cmov/libpthread.so.0

#15 0xb762884e in clone () from /lib/i686/cmov/libc.so.6









On Fri, Dec 7, 2012 at 5:13 PM, Victor Julien <victor at inliniac.net> wrote:

> On 12/05/2012 01:25 PM, xbadou xbadou wrote:
> > Hi，
> >
> >
> >
> > It seems that 1.3-dev works well for a whole day. After it starts, the
> > memory usage slowly increase. When the usage is above 80%, it remains
> > unchanged for a long time.
> >
> >
> >
> > There isn’t any coredump file too. Everything is all right. But I find
> > that there are some logs as follows.
>
> Great, thanks for reporting and testing! 1.3.5 was released yesterday
> containing the fixes.
>
> Cheers,
> Victor
>
> --
> ---------------------------------------------
> Victor Julien
> http://www.inliniac.net/
> PGP: http://www.inliniac.net/victorjulien.asc
> ---------------------------------------------
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-devel/attachments/20121210/457d7f08/attachment-0002.html>

More information about the Oisf-devel mailing list