<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Arial;color:#333333;background:#fbfcfe">I can be sure that the
problem is caused by insufficient memory</span> now. </p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Arial;color:#333333;background:#fbfcfe"> </span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Arial;color:#333333;background:#fbfcfe">Maybe I will need to buy
more RAM and upgrade to 64bit linux os.</span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Arial;color:#333333;background:#fbfcfe"> </span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Arial;color:#333333;background:#fbfcfe">I also tried to modify the
settings in suricata.yaml.</span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Arial;color:#333333;background:#fbfcfe">When I add ‘max_sessions’
and set it to a small value such as 10000, I found the memory usage is not very
large.</span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Arial;color:#333333;background:#fbfcfe"> </span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:Arial;color:#333333;background:#fbfcfe">I also noticed that there are
some time-out settings in the ‘flow’ section, I think the default value may be
very large to me. I think these values may also help to reduce the memory
usage.</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal">Thank you all for helping me.</p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p><br><div class="gmail_quote">On Mon, Dec 10, 2012 at 2:55 PM, Anoop Saldanha <span dir="ltr"><<a href="mailto:anoopsaldanha@gmail.com" target="_blank">anoopsaldanha@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I see the bug for the hsbd crash. Low memory conditions probably has<br>
realloc failing and returning NULL, but we don't reset the length to<br>
0.<br>
<br>
The second crash in libhtp probably falls in the low memory<br>
environment as well I guess.<br>
<div class="HOEnZb"><div class="h5"><br>
On Mon, Dec 10, 2012 at 11:46 AM, xbadou xbadou <<a href="mailto:xbadou@gmail.com">xbadou@gmail.com</a>> wrote:<br>
> Hi,<br>
><br>
><br>
> I’m sorry that the problem seems still exists.<br>
><br>
><br>
><br>
> Sometimes suricata works well, but sometimes it segfault quickly.<br>
><br>
> In one test, when memory use up to about 63%, it crashed.<br>
><br>
><br>
><br>
> The following is the result calling the command every 10s after suricata<br>
> starts.<br>
><br>
> Command: ps aux|grep /usr/bin/suricata|grep -v grep<br>
><br>
><br>
><br>
> root 7775 91.1 2.5 84420 79976 ? Rs 13:17 0:06 ...<br>
><br>
> root 7775 97.6 2.6 84552 80156 ? Rs 13:17 0:07 ...<br>
><br>
> root 7775 96.3 5.2 166128 161792 ? Rs 13:17 0:16 ...<br>
><br>
> root 7775 98.9 5.2 166128 161792 ? Rs 13:17 0:17 ...<br>
><br>
> root 7775 97.7 6.6 208432 204092 ? Rs 13:17 0:26 ...<br>
><br>
> root 7775 99.3 6.8 216204 211900 ? Rs 13:17 0:27 ...<br>
><br>
> root 7775 90.1 25.9 916448 799844 ? Ssl 13:17 0:33 ...<br>
><br>
> root 7775 89.2 26.1 930748 805284 ? Ssl 13:17 0:33 ...<br>
><br>
> root 7775 81.6 27.5 1019852 848776 ? Ssl 13:17 0:38 ...<br>
><br>
> root 7775 83.5 27.8 1038740 858592 ? Ssl 13:17 0:40 ...<br>
><br>
> root 7775 80.0 29.5 1139304 910580 ? Ssl 13:17 0:45 ...<br>
><br>
> root 7775 79.5 29.8 1156028 918168 ? Ssl 13:17 0:46 ...<br>
><br>
> root 7775 74.2 31.3 1254744 966068 ? Ssl 13:17 0:49 ...<br>
><br>
> root 7775 73.9 31.5 1268396 972692 ? Ssl 13:17 0:50 ...<br>
><br>
> root 7775 71.2 33.0 1363408 1018400 ? Ssl 13:17 0:54 ...<br>
><br>
> root 7775 71.0 33.2 1378684 1025360 ? Ssl 13:17 0:55 ...<br>
><br>
> root 7775 67.6 34.0 1421352 1048824 ? Ssl 13:17 0:58 ...<br>
><br>
> root 7775 67.2 34.1 1426892 1050720 ? Ssl 13:17 0:59 ...<br>
><br>
> root 7775 63.4 35.0 1503220 1080580 ? Ssl 13:17 1:01 ...<br>
><br>
> root 7775 63.1 35.2 1517584 1086492 ? Ssl 13:17 1:01 ...<br>
><br>
> root 7775 60.0 36.5 1604040 1124600 ? Ssl 13:17 1:04 ...<br>
><br>
> root 7775 59.9 36.6 1616372 1130352 ? Rsl 13:17 1:04 ...<br>
><br>
> root 7775 57.2 37.8 1699944 1167504 ? Ssl 13:17 1:06 ...<br>
><br>
> root 7775 57.6 38.1 1715524 1174520 ? Ssl 13:17 1:07 ...<br>
><br>
> root 7775 56.4 39.5 1809116 1216956 ? Ssl 13:17 1:11 ...<br>
><br>
> root 7775 56.4 39.7 1826124 1224640 ? Ssl 13:17 1:12 ...<br>
><br>
> root 7775 55.0 40.9 1912060 1262440 ? Ssl 13:17 1:15 ...<br>
><br>
> root 7775 54.9 41.1 1926992 1268452 ? Rsl 13:17 1:15 ...<br>
><br>
> root 7775 53.4 42.3 2007988 1303280 ? Ssl 13:17 1:18 ...<br>
><br>
> root 7775 53.6 42.5 2020344 1309912 ? Ssl 13:17 1:19 ...<br>
><br>
> root 7775 52.2 43.4 2084536 1340084 ? Ssl 13:17 1:22 ...<br>
><br>
> root 7775 52.2 43.6 2093808 1344668 ? Ssl 13:17 1:22 ...<br>
><br>
> root 7775 51.7 44.2 2126592 1362472 ? Rsl 13:17 1:26 ...<br>
><br>
> root 7775 52.0 44.3 2131712 1365428 ? Ssl 13:17 1:27 ...<br>
><br>
> root 7775 52.3 44.7 2158824 1379444 ? Ssl 13:17 1:32 ...<br>
><br>
> root 7775 52.6 44.8 2162580 1380940 ? Rsl 13:17 1:33 ...<br>
><br>
> root 7775 52.2 45.2 2201176 1395092 ? Ssl 13:17 1:37 ...<br>
><br>
> root 7775 52.2 45.3 2210392 1398228 ? Ssl 13:17 1:38 ...<br>
><br>
> root 7775 52.1 45.9 2260568 1415472 ? Ssl 13:17 1:42 ...<br>
><br>
> root 7775 53.0 46.8 2268760 1444184 ? Ssl 13:17 1:45 ...<br>
><br>
> root 7775 53.9 47.3 2310744 1459568 ? Ssl 13:17 1:51 ...<br>
><br>
> root 7775 53.9 47.5 2326104 1466228 ? Ssl 13:17 1:52 ...<br>
><br>
> root 7775 54.0 48.6 2398808 1499000 ? Ssl 13:17 1:57 ...<br>
><br>
> root 7775 54.2 48.8 2414168 1505540 ? Rsl 13:17 1:58 ...<br>
><br>
> root 7775 53.9 50.1 2500184 1544792 ? Ssl 13:17 2:03 ...<br>
><br>
> root 7775 54.6 51.1 2573912 1576212 ? Ssl 13:17 2:10 ...<br>
><br>
> root 7775 54.7 51.2 2583128 1580168 ? Ssl 13:17 2:15 ...<br>
><br>
> root 7775 53.8 51.6 2618968 1590756 ? Ssl 13:17 2:18 ...<br>
><br>
> root 7775 53.1 52.4 2675288 1614728 ? Ssl 13:17 2:22 ...<br>
><br>
> root 7775 52.4 53.4 2739800 1645236 ? Ssl 13:17 2:25 ...<br>
><br>
> root 7775 51.8 54.6 2826840 1683572 ? Ssl 13:17 2:29 ...<br>
><br>
> root 7775 51.4 56.0 2931288 1725660 ? Ssl 13:17 2:33 ...<br>
><br>
> root 7775 51.0 56.7 2983512 1748040 ? Ssl 13:17 2:37 ...<br>
><br>
> root 7775 51.0 57.4 3035736 1770964 ? Ssl 13:17 2:42 ...<br>
><br>
> root 7775 51.1 58.4 3102296 1801280 ? Ssl 13:17 2:47 ...<br>
><br>
> root 7775 51.3 59.4 3145724 1831824 ? Ssl 13:17 2:53 ...<br>
><br>
> root 7775 51.4 59.6 3145724 1837372 ? Ssl 13:17 2:59 ...<br>
><br>
> root 7775 51.9 59.8 3145724 1844756 ? Ssl 13:17 3:05 ...<br>
><br>
> root 7775 51.8 60.1 3145724 1854628 ? Ssl 13:17 3:11 ...<br>
><br>
> root 7775 52.6 60.6 3145724 1868488 ? Ssl 13:17 3:19 ...<br>
><br>
> root 7775 53.1 61.0 3145724 1880768 ? Ssl 13:17 3:26 ...<br>
><br>
> root 7775 53.5 61.3 3145724 1890252 ? Ssl 13:17 3:33 ...<br>
><br>
> root 7775 53.3 61.6 3145724 1900852 ? Ssl 13:17 3:38 ...<br>
><br>
> root 7775 54.2 62.2 3141052 1918648 ? Ssl 13:17 3:47 ...<br>
><br>
> root 7775 54.3 62.3 3145396 1920420 ? Ssl 13:17 3:53 ...<br>
><br>
> root 7775 55.0 62.3 3145724 1922368 ? Ssl 13:17 4:01 ...<br>
><br>
> root 7775 55.7 62.4 3145724 1923980 ? Ssl 13:17 4:10 ...<br>
><br>
> root 7775 56.1 62.5 3145192 1926960 ? Ssl 13:17 4:17 ...<br>
><br>
> root 7775 56.8 62.5 3142088 1926220 ? Ssl 13:17 4:26 ...<br>
><br>
> root 7775 56.7 62.6 3145724 1930768 ? Ssl 13:17 4:31 ...<br>
><br>
> root 7775 56.7 62.7 3145724 1932628 ? Ssl 13:17 4:37 ...<br>
><br>
> root 7775 56.5 62.8 3145724 1937868 ? Ssl 13:17 4:42 ...<br>
><br>
> root 7775 56.4 63.4 3145724 1955808 ? Ssl 13:17 4:47 ...<br>
><br>
> root 7775 55.6 63.5 3145724 1956508 ? Dsl 13:17 4:48 ...<br>
><br>
> root 7775 54.8 63.5 3145724 1956508 ? Dsl 13:17 4:50 ...<br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
> In some other case, I try to read the source code where segfault occurred.<br>
> For example:<br>
><br>
><br>
><br>
> #0 0x0810b769 in SCACSearch (mpm_ctx=0xd743538, mpm_thread_ctx=0x15908cf4,<br>
> pmq=0x15908d0c, buf=0x0, buflen=13915)<br>
><br>
> at util-mpm-ac.c:1232<br>
><br>
> #1 0x08090057 in HttpServerBodyPatternSearch (det_ctx=0x15908ca0, body=0x0,<br>
> body_len=<value optimized out>, flags=8 '\b')<br>
><br>
> at detect-engine-mpm.c:359<br>
><br>
> #2 0x0809cc0a in DetectEngineRunHttpServerBodyMpm (de_ctx=0xa6f4ba8,<br>
> det_ctx=0x15908ca0, f=0x68e3c0d0, htp_state=0x9a174f18,<br>
><br>
> flags=8 '\b') at detect-engine-hsbd.c:248<br>
><br>
> #3 0x08077a9a in SigMatchSignatures (th_v=0xb5900878, de_ctx=0xa6f4ba8,<br>
> det_ctx=0x15908ca0, p=0x9db4f90) at detect.c:1264<br>
><br>
> #4 0x08077b82 in Detect (tv=0xb5900878, p=0x9db4f90, data=0x15908ca0,<br>
> pq=0xb59009f0, postpq=0x0) at detect.c:1995<br>
><br>
> #5 0x0814e244 in TmThreadsSlotVarRun (tv=0xb5900878, p=0x9db4f90,<br>
> slot=0xb59008f8) at tm-threads.c:508<br>
><br>
> #6 0x0814e4fc in TmThreadsSlotVar (td=0xb5900878) at tm-threads.c:732<br>
><br>
> #7 0xb78174c0 in start_thread () from /lib/i686/cmov/libpthread.so.0<br>
><br>
> #8 0xb774e84e in clone () from /lib/i686/cmov/libc.so.6<br>
><br>
><br>
><br>
> I find that the pointer of ‘buf’ is 0x0. Then I read the code in<br>
> detect-engine-hsbd.c:248.<br>
><br>
><br>
><br>
> int DetectEngineRunHttpServerBodyMpm(DetectEngineCtx *de_ctx,<br>
><br>
> DetectEngineThreadCtx *det_ctx, Flow<br>
> *f,<br>
><br>
> HtpState *htp_state, uint8_t flags)<br>
><br>
> {<br>
><br>
> int i;<br>
><br>
> uint32_t cnt = 0;<br>
><br>
><br>
><br>
> FLOWLOCK_WRLOCK(f);<br>
><br>
> DetectEngineBufferHttpServerBodies(de_ctx, det_ctx, f, htp_state,<br>
> flags);<br>
><br>
> FLOWLOCK_UNLOCK(f);<br>
><br>
><br>
><br>
> if (det_ctx->hsbd != NULL && det_ctx->hsbd_buffers_list_len) {<br>
><br>
> for (i = 0; i < det_ctx->hsbd_buffers_list_len; i++) {<br>
><br>
> if (det_ctx->hsbd[i].buffer_len == 0)<br>
><br>
> continue;<br>
><br>
><br>
><br>
> cnt += HttpServerBodyPatternSearch(det_ctx,<br>
><br>
> det_ctx->hsbd[i].buffer,<br>
><br>
> det_ctx->hsbd[i].buffer_len,<br>
><br>
> flags);<br>
><br>
> }<br>
><br>
> }<br>
><br>
><br>
><br>
> return cnt;<br>
><br>
> }<br>
><br>
><br>
><br>
> In the source, we just check det_ctx->hsbd[i].buffer_len here but<br>
> det_ctx->hsbd[i].buffer is NULL. That results the segfault. I think this may<br>
> be a problem. But I am new to suricata. I don’t know why<br>
> det_ctx->hsbd[i].buffer is NULL here.<br>
><br>
><br>
><br>
><br>
><br>
> In my test network, I think something different is that we have some Network<br>
> Vulnerability System running. For example, nessus, IBM AppScan, and WVS.<br>
> Sometimes we get alert logs very frequently.<br>
><br>
><br>
><br>
> Hope it helps.<br>
><br>
><br>
><br>
> Finally, the two other core dump backtrace:<br>
><br>
><br>
><br>
> #0 0x0809ceb6 in DetectEngineBufferHttpHeaders (det_ctx=0xb6e01e98,<br>
> f=<value optimized out>, htp_state=0xa08fc198, flags=4 '\004')<br>
><br>
> at detect-engine-hhd.c:144<br>
><br>
> #1 0x0809d817 in DetectEngineRunHttpHeaderMpm (det_ctx=0xb6e01e98,<br>
> f=0x55f14010, htp_state=0xa08fc198, flags=4 '\004')<br>
><br>
> at detect-engine-hhd.c:200<br>
><br>
> #2 0x08077938 in SigMatchSignatures (th_v=0x1097ee98, de_ctx=0xb2fdba8,<br>
> det_ctx=0xb6e01e98, p=0xa57f0c0) at detect.c:1280<br>
><br>
> #3 0x08077b82 in Detect (tv=0x1097ee98, p=0xa57f0c0, data=0xb6e01e98,<br>
> pq=0xb6e00768, postpq=0x0) at detect.c:1995<br>
><br>
> #4 0x0814e244 in TmThreadsSlotVarRun (tv=0x1097ee98, p=0xa57f0c0,<br>
> slot=0xb6e00670) at tm-threads.c:508<br>
><br>
> #5 0x0814e4fc in TmThreadsSlotVar (td=0x1097ee98) at tm-threads.c:732<br>
><br>
> #6 0xb775c4c0 in start_thread () from /lib/i686/cmov/libpthread.so.0<br>
><br>
> #7 0xb769384e in clone () from /lib/i686/cmov/libc.so.6<br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
> #0 0xb7771cea in htp_normalize_parsed_uri () from /usr/lib/libhtp-0.2.so.1<br>
><br>
> #1 0xb777387e in htp_connp_REQ_LINE () from /usr/lib/libhtp-0.2.so.1<br>
><br>
> #2 0xb7773106 in htp_connp_req_data () from /usr/lib/libhtp-0.2.so.1<br>
><br>
> #3 0x08186332 in HTPHandleRequestData (f=0x3cdb6588, htp_state=0xa0f5d020,<br>
><br>
> pstate=0xa0f5cff8,<br>
><br>
> input=0xb2ffc8e0 "GET /images HTTP/1.0\r\nAccept-Charset:<br>
> iso-8859-1,utf-8;q<br>
> =0.9,*;q=0.1\r\nAccept-Language: en\r\nConnection: Close\r\nUser-Agent:<br>
> Mozilla/ 4.0<br>
> (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)\r\nPragma: no-ca"...,<br>
><br>
> input_len=284, local_data=0x0, output=0xb2ffc800) at app-layer-htp.c:667<br>
><br>
> #4 0x0817fbb6 in AppLayerDoParse (local_data=0x0, f=0x3cdb6588,<br>
><br>
> app_layer_state=0xa0f5d020, parser_state=0xa0f5cff8,<br>
><br>
> input=0xb2ffc8e0 "GET /images HTTP/1.0\r\nAccept-Charset:<br>
> iso-8859-1,utf-8;q<br>
> =0.9,*;q=0.1\r\nAccept-Language: en\r\nConnection: Close\r\nUser-Agent:<br>
> Mozilla/ 4.0<br>
> (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)\r\nPragma: no-ca"...,<br>
><br>
> input_len=284, parser_idx=<value optimized out>, proto=1)<br>
><br>
> at app-layer-parser.c:726<br>
><br>
> #5 0x0817feb1 in AppLayerParse (local_data=0x0, f=0x3cdb6588,<br>
><br>
> proto=<value optimized out>, flags=5 '\005',<br>
><br>
> input=0xb2ffc8e0 "GET /images HTTP/1.0\r\nAccept-Charset:<br>
> iso-8859-1,utf-8;q<br>
> =0.9,*;q=0.1\r\nAccept-Language: en\r\nConnection: Close\r\nUser-Agent:<br>
> Mozilla/ 4.0<br>
> (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)\r\nPragma: no-ca"...,<br>
><br>
> input_len=284) at app-layer-parser.c:935<br>
><br>
> #6 0x0816d607 in StreamTcpReassembleAppLayer (tv=0xb3818a20,<br>
><br>
> ---Type <return> to continue, or q <return> to quit---<br>
><br>
> ra_ctx=0xb6e00b98, ssn=0x9e61bf28, stream=0x9e61bf64, p=0xa520638)<br>
><br>
> at stream-tcp-reassemble.c:2942<br>
><br>
> #7 0x0816d823 in StreamTcpReassembleHandleSegmentUpdateACK (tv=0xb3818a20,<br>
><br>
> ra_ctx=0xb6e00b98, ssn=0x9e61bf28, stream=0x9e61bf64, p=0xa520638)<br>
><br>
> at stream-tcp-reassemble.c:3310<br>
><br>
> #8 0x0816f2ef in StreamTcpReassembleHandleSegment (tv=0xb3818a20,<br>
><br>
> ra_ctx=0xb6e00b98, ssn=0x9e61bf28, stream=0x9e61bf2c, p=0xa520638,<br>
><br>
> pq=0xb6e00470) at stream-tcp-reassemble.c:3384<br>
><br>
> #9 0x0816701b in StreamTcpPacketStateEstablished (tv=0xb3818a20,<br>
> p=0xa520638,<br>
><br>
> stt=0xb6e00468, ssn=0x9e61bf28, pq=0xb6e00470) at stream-tcp.c:1779<br>
><br>
> #10 0x0816a1de in StreamTcpPacket (tv=0xb3818a20, p=0xa520638,<br>
> stt=0xb6e00468,<br>
><br>
> pq=0xb3818ac0) at stream-tcp.c:3512<br>
><br>
> #11 0x0816b40f in StreamTcp (tv=0xb3818a20, p=0xa520638, data=0xb6e00468,<br>
><br>
> pq=0xb3818ac0, postpq=0xb3818b14) at stream-tcp.c:3752<br>
><br>
> #12 0x0814e244 in TmThreadsSlotVarRun (tv=0xb3818a20, p=0xa520638,<br>
><br>
> slot=0xb3818aa0) at tm-threads.c:508<br>
><br>
> #13 0x0814e4fc in TmThreadsSlotVar (td=0xb3818a20) at tm-threads.c:732<br>
><br>
> #14 0xb76f14c0 in start_thread () from /lib/i686/cmov/libpthread.so.0<br>
><br>
> #15 0xb762884e in clone () from /lib/i686/cmov/libc.so.6<br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
> On Fri, Dec 7, 2012 at 5:13 PM, Victor Julien <<a href="mailto:victor@inliniac.net">victor@inliniac.net</a>> wrote:<br>
>><br>
>> On 12/05/2012 01:25 PM, xbadou xbadou wrote:<br>
>> > Hi，<br>
>> ><br>
>> ><br>
>> ><br>
>> > It seems that 1.3-dev works well for a whole day. After it starts, the<br>
>> > memory usage slowly increase. When the usage is above 80%, it remains<br>
>> > unchanged for a long time.<br>
>> ><br>
>> ><br>
>> ><br>
>> > There isn’t any coredump file too. Everything is all right. But I find<br>
>> > that there are some logs as follows.<br>
>><br>
>> Great, thanks for reporting and testing! 1.3.5 was released yesterday<br>
>> containing the fixes.<br>
>><br>
>> Cheers,<br>
>> Victor<br>
>><br>
>> --<br>
>> ---------------------------------------------<br>
>> Victor Julien<br>
>> <a href="http://www.inliniac.net/" target="_blank">http://www.inliniac.net/</a><br>
>> PGP: <a href="http://www.inliniac.net/victorjulien.asc" target="_blank">http://www.inliniac.net/victorjulien.asc</a><br>
>> ---------------------------------------------<br>
>><br>
><br>
><br>
</div></div><div class="HOEnZb"><div class="h5">> _______________________________________________<br>
> Suricata IDS Devel mailing list: <a href="mailto:oisf-devel@openinfosecfoundation.org">oisf-devel@openinfosecfoundation.org</a><br>
> Site: <a href="http://suricata-ids.org" target="_blank">http://suricata-ids.org</a> | Participate:<br>
> <a href="http://suricata-ids.org/participate/" target="_blank">http://suricata-ids.org/participate/</a><br>
> List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel" target="_blank">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel</a><br>
> Redmine: <a href="https://redmine.openinfosecfoundation.org/" target="_blank">https://redmine.openinfosecfoundation.org/</a><br>
<br>
<br>
<br>
--<br>
</div></div><span class="HOEnZb"><font color="#888888">Anoop Saldanha<br>
</font></span></blockquote></div><br>