Switch to a 64 bit kernel, and all the low-mem problems go away (there's no low-mem! Yay!! Assuming, of course, that you can.). Slabtop will tell you about low-mem usage, if you haven't looked at it.<br><br><div class="gmail_quote">
On Sat, Dec 10, 2011 at 9:03 AM, Delta Yeh <span dir="ltr"><<a href="mailto:delta.yeh@gmail.com">delta.yeh@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi all,<br>
I have 32 bit debian squeeze box with 4G mem, runing suricata+pfring,<br>
there is out of memory in the past two days.<br>
<br>
Kernel is 2.6.39, suricata is 1.1, pfring is 5.1 which support pfring<br>
bpf filter.<br>
According to the dmesg, the memory of suricata is low(about 40MB) ,<br>
but the kernel low memory is almost zero.<br>
The traffic is not high at all, only 75 http req/s . It seems kernel<br>
run out of memory for some reason.<br>
<br>
To reproduce it, I run a http stress test with load runner,<br>
Soon , the available kernel low mem became 220M, and it decreased continually.<br>
Then I kill suricata( 4XMB memory according to ps output) , the<br>
available low mem is back to 6XXM, which I think is<br>
reasonable.<br>
<br>
<br>
dmesg when OOM is :<br>
<br>
<br>
[108769.088032] php5-cli invoked oom-killer: gfp_mask=0xd0, order=1,<br>
oom_adj=0, oom_score_adj=0<br>
[108769.088093] php5-cli cpuset=/ mems_allowed=0<br>
[108769.088123] Pid: 32213, comm: php5-cli Not tainted 2.6.39.4 #1<br>
[108769.088157] Call Trace:<br>
[108769.088187] [<c1070cc6>] dump_header+0x69/0x17d<br>
[108769.088220] [<c11e14d6>] ? ___ratelimit+0xca/0xe0<br>
[108769.088251] [<c1070f94>] oom_kill_process+0x2e/0x201<br>
[108769.088284] [<c1071432>] out_of_memory+0x1ee/0x2b1<br>
[108769.088315] [<c10743ce>] __alloc_pages_nodemask+0x439/0x581<br>
[108769.088351] [<c102e103>] copy_process+0xe1/0xd9d<br>
[108769.088382] [<c11a2687>] ? security_file_alloc+0xf/0x11<br>
[108769.088416] [<c10a0177>] ? get_empty_filp+0x9b/0x13f<br>
[108769.088448] [<c102efab>] do_fork+0xd7/0x1eb<br>
[108769.088479] [<c1008120>] sys_clone+0x1f/0x24<br>
[108769.088509] [<c134f571>] ptregs_clone+0x15/0x24<br>
[108769.088540] [<c134f49c>] ? sysenter_do_call+0x12/0x2c<br>
[108769.088571] Mem-Info:<br>
[108769.088595] Node 0 DMA per-cpu:<br>
[108769.088623] CPU 0: hi: 0, btch: 1 usd: 0<br>
[108769.088654] CPU 1: hi: 0, btch: 1 usd: 0<br>
[108769.088684] Node 0 Normal per-cpu:<br>
[108769.088713] CPU 0: hi: 186, btch: 31 usd: 0<br>
[108769.088744] CPU 1: hi: 186, btch: 31 usd: 0<br>
[108769.088774] Node 0 HighMem per-cpu:<br>
[108769.088802] CPU 0: hi: 186, btch: 31 usd: 0<br>
[108769.088833] CPU 1: hi: 186, btch: 31 usd: 0<br>
[108769.088866] active_anon:91593 inactive_anon:4805 isolated_anon:0<br>
[108769.088867] active_file:24583 inactive_file:8171 isolated_file:0<br>
[108769.088868] unevictable:444 dirty:33 writeback:0 unstable:0<br>
[108769.088869] free:599897 slab_reclaimable:47561 slab_unreclaimable:159717<br>
[108769.088870] mapped:22437 shmem:17145 pagetables:660 bounce:0<br>
[108769.089032] Node 0 DMA free:3536kB min:64kB low:80kB high:96kB<br>
active_anon:0kB inactive_anon:0kB active_file:72kB inactive_file:0kB<br>
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15804kB<br>
mlocked:0kB dirty:8kB writeback:0kB mapped:0kB shmem:0kB<br>
slab_reclaimable:2876kB slab_unreclaimable:9168kB kernel_stack:256kB<br>
pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB<br>
pages_scanned:0 all_unreclaimable? yes<br>
[108769.089234] lowmem_reserve[]: 0 867 4007 4007<br>
[108769.089269] Node 0 Normal free:6696kB min:3732kB low:4664kB<br>
high:5596kB active_anon:0kB inactive_anon:0kB active_file:48kB<br>
inactive_file:0kB unevictable:0kB isolated(anon):0kB<br>
isolated(file):0kB present:887976kB mlocked:0kB dirty:12kB<br>
writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:187368kB<br>
slab_unreclaimable:629700kB kernel_stack:864kB pagetables:0kB<br>
unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:48<br>
all_unreclaimable? yes<br>
[108769.089474] lowmem_reserve[]: 0 0 25124 25124<br>
[108769.089508] Node 0 HighMem free:2389356kB min:512kB low:3892kB<br>
high:7272kB active_anon:366372kB inactive_anon:19220kB<br>
active_file:98212kB inactive_file:32700kB unevictable:1776kB<br>
isolated(anon):0kB isolated(file):0kB present:3215956kB mlocked:1776kB<br>
dirty:112kB writeback:0kB mapped:89748kB shmem:68580kB<br>
slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB<br>
pagetables:2640kB unstable:0kB bounce:0kB writeback_tmp:0kB<br>
pages_scanned:0 all_unreclaimable? no<br>
[108769.089719] lowmem_reserve[]: 0 0 0 0<br>
[108769.089752] Node 0 DMA: 76*4kB 16*8kB 14*16kB 8*32kB 9*64kB<br>
4*128kB 2*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 3536kB<br>
[108769.089829] Node 0 Normal: 1215*4kB 214*8kB 11*16kB 0*32kB 0*64kB<br>
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6748kB<br>
[108769.089907] Node 0 HighMem: 487*4kB 14588*8kB 7079*16kB 2792*32kB<br>
1022*64kB 234*128kB 62*256kB 30*512kB 16*1024kB 14*2048kB 463*4096kB =<br>
2389356kB<br>
[108769.089990] 50284 total pagecache pages<br>
[108769.090018] 0 pages in swap cache<br>
[108769.090045] Swap cache stats: add 0, delete 0, find 0/0<br>
[108769.090077] Free swap = 0kB<br>
[108769.090102] Total swap = 0kB<br>
[108769.103348] 1245168 pages RAM<br>
[108769.103376] 1017346 pages HighMem<br>
[108769.103403] 232402 pages reserved<br>
[108769.103429] 65775 pages shared<br>
[108769.103455] 386797 pages non-shared<br>
[108769.103484] [ pid ] uid tgid total_vm rss cpu oom_adj<br>
oom_score_adj name<br>
[108769.103545] [ 1357] 0 1357 1348 228 1 0<br>
0 sshd<br>
[108769.103600] [ 1477] 0 1477 394 57 1 0<br>
0 readproctitle<br>
[108769.103657] [ 1481] 0 1481 437 96 1 0<br>
0 svscan<br>
[108769.103713] [ 1483] 0 1483 397 57 0 0<br>
0 supervise<br>
[108769.103770] [ 1484] 0 1484 432 100 1 0<br>
0 multilog<br>
[108769.103826] [ 1634] 0 1634 445 444 1 -17<br>
-1000 watchdog<br>
[108769.103883] [ 1910] 107 1910 37680 1568 1 0<br>
0 postgres<br>
[108769.103939] [ 1913] 107 1913 37718 16333 1 0<br>
0 postgres<br>
[108769.104016] [ 1914] 107 1914 37718 296 1 0<br>
0 postgres<br>
[108769.104081] [ 1915] 107 1915 37756 351 1 0<br>
0 postgres<br>
[108769.104152] [ 1916] 107 1916 3704 275 1 0<br>
0 postgres<br>
[108769.104212] [ 2111] 0 2111 9419 409 0 0<br>
0 rsyslogd<br>
[108769.104450] [ 2128] 0 2128 691 284 1 0<br>
0 redis-server<br>
[108769.104508] [ 2130] 0 2130 691 281 1 0<br>
0 redis-server<br>
[108769.104565] [ 2137] 0 2137 17079 10799 1 0<br>
0 php5-cli<br>
[108769.104622] [ 2140] 0 2140 17078 10734 0 0<br>
0 php5-cli<br>
[108769.104678] [ 2160] 107 2160 37940 1018 0 0<br>
0 postgres<br>
[108769.104734] [ 2161] 107 2161 37940 1024 0 0<br>
0 postgres<br>
[108769.104790] [ 2176] 0 2176 18371 2347 0 0<br>
0 python<br>
[108769.104846] [ 2177] 107 2177 37924 7389 0 0<br>
0 postgres<br>
[108769.104902] [ 2179] 107 2179 37951 13986 0 0<br>
0 postgres<br>
[108769.104958] [ 2251] 0 2251 576 159 0 0<br>
0 getty<br>
[108769.105014] [ 2252] 0 2252 681 294 1 0<br>
0 rc.initial<br>
[108769.105071] [ 1214] 0 1214 20093 13917 1 0<br>
0 php5-cli<br>
[108769.105127] [ 1289] 0 1289 43124 25098 1 0<br>
0 suricata<br>
[108769.105296] [ 1316] 107 1316 37924 990 1 0<br>
0 postgres<br>
[108769.105519] [ 6938] 0 6938 397 59 1 0<br>
0 supervise<br>
[108769.106267] [ 6942] 0 6942 435 120 0 0<br>
0 uscheduled<br>
[108769.106324] [29249] 0 29249 2074 664 0 0<br>
0 sshd<br>
[108769.106379] [29292] 0 29292 1208 356 1 0<br>
0 sftp-server<br>
[108769.106436] [29339] 0 29339 2034 666 1 0<br>
0 sshd<br>
[108769.106492] [29342] 0 29342 698 371 0 0<br>
0 sh<br>
[108769.106602] [31015] 0 31015 1242 313 1 0<br>
0 sftp-server<br>
[108769.106660] [32210] 0 32210 663 248 1 0<br>
0 monitor<br>
[108769.106715] [32211] 0 32211 663 249 0 0<br>
0 safecheck<br>
[108769.106772] [32212] 0 32212 13763 9395 0 0<br>
0 php5-cli<br>
[108769.106828] [32213] 0 32213 13764 9406 0 0<br>
0 php5-cli<br>
[108769.106883] Out of memory (oom_kill_allocating_task): Kill process<br>
32213 (php5-cli) score 0 or sacrifice child<br>
_______________________________________________<br>
Oisf-users mailing list<br>
<a href="mailto:Oisf-users@openinfosecfoundation.org">Oisf-users@openinfosecfoundation.org</a><br>
<a href="http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" target="_blank">http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div>".... We are such stuff</div><div>As dreams are made on; and our little life</div><div><div>Is rounded with a sleep."</div></div><div>-- Shakespeare, The Tempest - Act 4</div>
<br>