[Oisf-devel] Suricata core dumps with luajit rule and large ruleset?

[Chris Wakelin]

I've been trying out my little Lua(jit) XORed-binary detector and
discovered that when I include a large ruleset such as
emerging-trojans.rules (6.5k) or emerging-malware.rules (4k), Suricata
segfaults:-

> Program terminated with signal 11, Segmentation fault.
> #0  0x00007f26a3eaf580 in ?? () from /usr/lib/x86_64-linux-gnu/libluajit-5.1.so.2
> #0  0x00007f26a3eaf580 in ?? () from /usr/lib/x86_64-linux-gnu/libluajit-5.1.so.2
> No symbol table info available.
> #1  0x00007f26a3ee2bb0 in lua_tolstring () from /usr/lib/x86_64-linux-gnu/libluajit-5.1.so.2
> No symbol table info available.
> #2  0x00000000004ba368 in DetectLuajitThreadInit (data=<optimised out>) at detect-luajit.c:395
>         _sc_log_err_msg = "[799] 19/9/2012 -- 22:58:45 - (detect-luajit.c:387) <Error> (DetectLuajitThreadInit) -- [ERRCODE: SC_ERR_LUAJIT_ERROR(218)] - ", '\000' <repeats 1921 times>
>         _sc_log_err_temp = <optimised out>
>         luajit = <optimised out>
>         __PRETTY_FUNCTION__ = "DetectLuajitThreadInit"
>         t = 0x7f269c503740
>         __FUNCTION__ = "DetectLuajitThreadInit"
>         status = <optimised out>
> #3  0x0000000000443f1f in DetectEngineThreadCtxInitKeywords (de_ctx=<optimised out>, det_ctx=<optimised out>) at detect-engine.c:711
>         item = 0x40ad750
> #4  0x00000000004455ce in DetectEngineThreadCtxInitKeywords (de_ctx=<optimised out>, det_ctx=<optimised out>) at detect-engine.c:698
> No locals.
> #5  DetectEngineThreadCtxInit (tv=0x5ddb280, initdata=0x3fc69d0, data=0x7f26a0cbd628) at detect-engine.c:803
>         de_ctx = 0x3fc69d0
>         det_ctx = 0x7f269c007730
>         __FUNCTION__ = "DetectEngineThreadCtxInit"
> #6  0x00000000005131c0 in TmThreadsSlotPktAcqLoop (td=0x5ddb280) at tm-threads.c:633
>         slot_data = 0x0
>         tv = 0x5ddb280
>         s = 0x5dc30e0
>         r = <optimised out>
>         slot = <optimised out>
>         __FUNCTION__ = "TmThreadsSlotPktAcqLoop"
> #7  0x00007f26a33fae9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #8  0x00007f26a2cad4bd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> No symbol table info available.
> #9  0x0000000000000000 in ?? ()
> No symbol table info available.

This is using subsets of ET ruleset, plus local my rules. With ~5.5k
total rules it seems fine; add one of the big ones and it breaks.

My luajit rule:

alert http $EXTERNAL_NET any -> $HOME_NET any (msg:"LUAJIT test - match
XORed binary"; flowbits:isset,ET.http.javaclient.vulnerable;
flowbits:isnotset,ET.http.binary; luajit:xor-n-plus4-or-6.lua;
sid:379000001; rev:2;)

Victor, I think I sent you the Lua script already? I'll send it to
anyone else interested if they ask ;-)

Suricata is latest git - 1.4dev (rev 9a4b612) - and was started with
"--runmode=single -r <1GB pcap>". I've not tried other runmodes (such as
live traffic) yet. I'm running Ubuntu 12.04 64-bit.

(BTW the rule uses a lot of ticks, but isn't the worst by a long way.
Surprisingly, making it return 0 instantly doesn't seem to speed it up
much!)

Best Wishes,
Chris

-- 
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
Christopher Wakelin,                           c.d.wakelin at reading.ac.uk
IT Services Centre, The University of Reading,  Tel: +44 (0)118 378 2908
Whiteknights, Reading, RG6 6AF, UK              Fax: +44 (0)118 975 3094