<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="color: rgb(0, 0, 0); font-family: Calibri,Helvetica,sans-serif; font-size: 12pt;">
<span>Hi All,</span></div>
<div style="color: rgb(0, 0, 0); font-family: Calibri,Helvetica,sans-serif; font-size: 12pt;">
<span><br>
</span></div>
<div style="color: rgb(0, 0, 0); font-family: Calibri,Helvetica,sans-serif; font-size: 12pt;">
<span>There's some additional information which may be helpful as I've attached.</span></div>
<div style="color: rgb(0, 0, 0); font-family: Calibri,Helvetica,sans-serif; font-size: 12pt;">
<span>IMAP related items in flow information is a bit abnormal, as state is "new" while reason is "shutdown":</span></div>
<div style="color: rgb(0, 0, 0); font-family: Calibri,Helvetica,sans-serif; font-size: 12pt;">
<span><span>{"CREATE_TIME":"2019-05-22T16:30:48.587578+0800","flow_id":309972231175741,"LOG_TYPE":"flow","src_mac":"04:7D:7B:B4:01:4A","SRC_IP":"10.21.37.60","SRC_PORT":10345,"dst_mac":"00:1B:0C:CE:5B:D0","DST_IP":"10.21.17.206",</span><span style="color: rgb(81, 167, 249);">"DST_PORT":143</span><span>,"PROTOCOL":"TCP","flow":{"pkts_toserver":4,"pkts_toclient":5,"bytes_toserver":285,"bytes_toclient":1955,"start":"2019-05-22T16:30:50.638525+0800","end":"2019-05-22T16:30:50.878531+0800","age":0,</span><span style="color: rgb(200, 38, 19);">"state":"new","reason":"shutdown",</span><span>"alerted":false},"tcp":{"tcp_flags":"00","tcp_flags_ts":"00","tcp_flags_tc":"00"}}<br>
</span><br>
</span></div>
<div style="color: rgb(0, 0, 0); font-family: Calibri,Helvetica,sans-serif; font-size: 12pt;">
<span><br>
</span></div>
<div style="color: rgb(0, 0, 0); font-family: Calibri,Helvetica,sans-serif; font-size: 12pt;">
<span>Normal item:</span></div>
<div style="color: rgb(0, 0, 0); font-family: Calibri,Helvetica,sans-serif; font-size: 12pt;">
<span>// SMTP</span></div>
<div style="color: rgb(0, 0, 0); font-family: Calibri,Helvetica,sans-serif; font-size: 12pt;">
<span>// state is closed while reason is "shutdown"</span></div>
<div style="color: rgb(0, 0, 0); font-family: Calibri,Helvetica,sans-serif; font-size: 12pt;">
<span><span>{"CREATE_TIME":"2019-05-22T16:30:48.587578+0800","flow_id":724273366290807,"LOG_TYPE":"flow","src_mac":"04:7D:7B:B4:01:4A","SRC_IP":"10.21.37.60","SRC_PORT":10367,"dst_mac":"00:1B:0C:CE:5B:D0","DST_IP":"10.21.17.206",</span><span style="color: rgb(81, 167, 249);">"DST_PORT":25</span><span>,"PROTOCOL":"TCP","APP_PROTOCOL":"smtp","app_proto_tc":"failed","flow":{"pkts_toserver":11,"pkts_toclient":12,"bytes_toserver":1208,"bytes_toclient":959,"start":"2019-05-22T16:30:47.184695+0800","end":"2019-05-22T16:30:47.244888+0800","age":0</span><span style="color: rgb(200, 38, 19);">,"state":"closed","reason":"shutdown"</span><span>,"alerted":false},"tcp":{"tcp_flags":"1b","tcp_flags_ts":"1b","tcp_flags_tc":"1b","syn":true,"fin":true,"psh":true,"ack":true,"state":"closed"}}<br>
</span><span></span><br>
</span></div>
<div>
<div id="appendonsend"></div>
<div style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font color="#000000" face="Calibri, sans-serif" style="font-size:11pt"><b>发件人:</b> Ma Allen <mazhh@outlook.com><br>
<b>发送时间:</b> 2019年5月28日 17:16<br>
<b>收件人:</b> oisf-devel-request@lists.openinfosecfoundation.org; oisf-users@lists.openinfosecfoundation.org<br>
<b>主题:</b> [4.1.0-beta1] I implement IMAP protocol detection & parser. When replaying pcap file with ONLY IMAP protocol, it works well. However, while replaying pcap file mixing with SMTP and IMAP protocols, SMTP parsing works well and IMAP parsing fails.</font>
<div> </div>
</div>
<div dir="ltr">
<div style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
<span>Hi All,</span></div>
<div style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
<span><br>
</span></div>
<div style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
<span>Phenomenon:</span></div>
<div style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
[4.1.0-beta1] I implement IMAP protocol detection & parser. When replaying pcap file with ONLY IMAP protocol, it works well. However, while replaying pcap file mixing with SMTP and IMAP protocols, SMTP parsing works well and IMAP parsing fails. There seems
something wrong with reading IMAP packets.</div>
<div style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<div style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
I tried to locate where IMAP packets are lost, and add the following debug code in 3 functions
<span style="font-family:Calibri,Helvetica,sans-serif; font-size-adjust:none; font-stretch:100%; margin-bottom:0px; margin-top:0px">
</span><span style="border-bottom-color:rgb(237,92,87); border-left-color:rgb(237,92,87); border-right-color:rgb(237,92,87); border-top-color:rgb(237,92,87); color:rgb(237,92,87); font-family:Calibri,Helvetica,sans-serif; font-size-adjust:none; font-stretch:100%; margin-bottom:0px; margin-top:0px">AppLayerParserParse,
<span style="border-bottom-color:rgb(200,38,19); border-left-color:rgb(200,38,19); border-right-color:rgb(200,38,19); border-top-color:rgb(200,38,19); color:rgb(200,38,19); font-family:Calibri,Helvetica,sans-serif; font-size-adjust:none; font-stretch:100%; margin-bottom:0px; margin-top:0px">
TCPProtoDetect </span>and <span style="border-bottom-color:rgb(200,38,19); border-left-color:rgb(200,38,19); border-right-color:rgb(200,38,19); border-top-color:rgb(200,38,19); color:rgb(200,38,19); font-family:Calibri,Helvetica,sans-serif; font-size-adjust:none; font-stretch:100%; margin-bottom:0px; margin-top:0px">
AppLayerHandleTCPData</span></span>:</div>
<div style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
<span> bool check_flag = false;<br>
</span>
<div> if (BasicSearchNocase(data, data_len, " FETCH ", strlen(" FETCH ")) != NULL) { // " FETCH " is the command I'd like to parse in IMAP</div>
<div> check_flag = true;</div>
<div> }</div>
and add breakpoint in the line, "<span style="display:inline!important; font-family:Calibri,Helvetica,sans-serif; font-size-adjust:none; font-stretch:100%">check_flag = true;". However, no breakpoints are effective. So I believe that IMAP packets are lost in
the lower function in the following calling stack.</span><br>
</div>
<div style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<div style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
<span>#0 </span><span style="color:rgb(237,92,87)">AppLayerParserParse </span><span>(tv=tv@entry=0x48ab230, alp_tctx=0x7fffd8011d30, f=f@entry=0x1485c50, alproto=7, flags=flags@entry=5 '\005',<br>
</span>
<div> input=input@entry=0x7fffd82a8520 "EHLO 10.21.37.60\r\n", input_len=input_len@entry=18) at app-layer-parser.c:1092</div>
<div>#1 0x0000000000418bbf in <span style="color:rgb(200,38,19)">TCPProtoDetect </span>
(tv=<optimized out>, ra_ctx=<optimized out>, app_tctx=app_tctx@entry=0x7fffd8011940, p=p@entry=0x7fffe0268ea0,</div>
<div> f=f@entry=0x1485c50, ssn=ssn@entry=0x7fffd80da6a0, stream=stream@entry=0x7fffd80da730, data=data@entry=0x7fffd82a8520 "EHLO 10.21.37.60\r\n",</div>
<div> data_len=data_len@entry=18, flags=flags@entry=5 '\005') at app-layer.c:431</div>
<div>#2 0x0000000000419166 in <span style="color:rgb(200,38,19)">AppLayerHandleTCPData
</span>(tv=tv@entry=0x48ab230, ra_ctx=ra_ctx@entry=0x7fffd8011910, p=p@entry=0x7fffe0268ea0, f=0x1485c50,</div>
<div> ssn=ssn@entry=0x7fffd80da6a0, stream=stream@entry=0x7fffd80da730, data=0x7fffd82a8520 "EHLO 10.21.37.60\r\n", data_len=data_len@entry=18,</div>
<div> flags=5 '\005') at app-layer.c:590</div>
<div>#3 0x000000000059d5a2 in ReassembleUpdateAppLayer (dir=<optimized out>, p=<optimized out>, stream=<optimized out>, ssn=<optimized out>,</div>
<div> ra_ctx=<optimized out>, tv=<optimized out>) at stream-tcp-reassemble.c:1063</div>
<div>#4 StreamTcpReassembleAppLayer (tv=0x48ab230, ra_ctx=0x7fffd8011910, ssn=0x7fffd80da6a0, stream=0x7fffd80da730, p=0x7fffe0268ea0,</div>
<div> dir=UPDATE_DIR_OPPOSING) at stream-tcp-reassemble.c:1136</div>
<div>#5 0x000000000059e101 in StreamTcpReassembleHandleSegmentUpdateACK (p=0x7fffe0268ea0, stream=0x7fffd80da730, ssn=0x7fffd80da6a0, ra_ctx=0x7fffd8011910,</div>
<div> tv=0x48ab230) at stream-tcp-reassemble.c:1685</div>
<div>#6 StreamTcpReassembleHandleSegment (tv=tv@entry=0x48ab230, ra_ctx=0x7fffd8011910, ssn=ssn@entry=0x7fffd80da6a0, stream=0x7fffd80da6b0,</div>
<div> p=p@entry=0x7fffe0268ea0, pq=pq@entry=0x7fffd80115f8) at stream-tcp-reassemble.c:1724</div>
<div>#7 0x0000000000594d97 in HandleEstablishedPacketToClient (stt=<optimized out>, pq=<optimized out>, p=<optimized out>, ssn=<optimized out>,</div>
<div> tv=<optimized out>) at stream-tcp.c:2362</div>
<div>#8 StreamTcpPacketStateEstablished (tv=tv@entry=0x48ab230, p=p@entry=0x7fffe0268ea0, stt=stt@entry=0x7fffd80115f0, ssn=ssn@entry=0x7fffd80da6a0,</div>
<div> pq=pq@entry=0x7fffd80115f8) at stream-tcp.c:2599</div>
<div>#9 0x00000000005992fb in StreamTcpPacket (tv=0x48ab230, p=0x7fffe0268ea0, stt=0x7fffd80115f0, pq=0x7fffd80008e0) at stream-tcp.c:4645</div>
<div>#10 0x000000000059a690 in StreamTcp (tv=tv@entry=0x48ab230, p=p@entry=0x7fffe0268ea0, data=<optimized out>, pq=pq@entry=0x7fffd80008e0,</div>
<div> postpq=postpq@entry=0x0) at stream-tcp.c:5020</div>
<div>#11 0x0000000000529901 in FlowWorker (tv=0x48ab230, p=0x7fffe0268ea0, data=0x7fffd80008c0, preq=0x1baf0e0, unused=<optimized out>) at flow-worker.c:216</div>
<div>#12 0x00000000005a5fa4 in TmThreadsSlotVarRun (tv=tv@entry=0x48ab230, p=p@entry=0x7fffe0268ea0, slot=slot@entry=0x1baf0a0) at tm-threads.c:143</div>
<div>#13 0x00000000005a8eba in TmThreadsSlotVar (td=0x48ab230) at tm-threads.c:598</div>
<div>#14 0x00007ffff6c33dd5 in start_thread () from /lib64/libpthread.so.0</div>
<div>#15 0x00007ffff60a5ead in clone () from /lib64/libc.so.6</div>
<div><br>
</div>
<div>I attached the pcap file leading to the problem. As it works well while replaying the pcap file containing ONLY IMAP protocol, I believe that SMTP packets before IMAP make IMAP protocol detection fail.</div>
<div><br>
</div>
<div>SMTP & IMAP Registration:</div>
<div>// SMTP -- Pattern Matcher</div>
<div><span>static int SMTPRegisterPatternsForProtocolDetection(void)<br>
</span>
<div>{</div>
<div>if (AppLayerProtoDetectPMRegisterPatternCI(IPPROTO_TCP,</div>
<div> ALPROTO_SMTP,</div>
<div> "EHLO",</div>
<div> 4,</div>
<div> 0,</div>
<div> STREAM_TOSERVER) < 0)</div>
<div>{</div>
<div> return -1;</div>
<div>}</div>
<div>if (AppLayerProtoDetectPMRegisterPatternCI(IPPROTO_TCP,</div>
<div> ALPROTO_SMTP,</div>
<div> "HELO",</div>
<div> 4,</div>
<div> 0,</div>
<div> STREAM_TOSERVER) < 0)</div>
<div>{</div>
<div> return -1;</div>
<div>}</div>
<div>if (AppLayerProtoDetectPMRegisterPatternCI(IPPROTO_TCP,</div>
<div> ALPROTO_SMTP,</div>
<div> "QUIT",</div>
<div> 4,</div>
<div> 0,</div>
<div> STREAM_TOSERVER) < 0)</div>
<div>{</div>
<div> return -1;</div>
<div>}</div>
<div></div>
<div>return 0;</div>
<div>}</div>
<span></span><br>
</div>
<span></span>// IMAP -- Probing Parser</div>
<div style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
<span>// client command: a0003 SELECT "INBOX"\r\n<br>
</span>
<div>// tag length:5 Maximum command(AUTHENTICATE) length: 12 Assumed mximum argument length: 120</div>
<div>#define IMAP_MIN_TO_SERVER_FRAME_LEN (5 + 1 + 12 + 1 + 120 + 2)</div>
<span></span><br>
</div>
<div style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
<span> AppLayerProtoDetectPPRegister(IPPROTO_TCP,<br>
</span>
<div> IMAP_DEFAULT_PORT, ALPROTO_IMAP, 0,</div>
<div> IMAP_MIN_TO_SERVER_FRAME_LEN, STREAM_TOSERVER,</div>
<div> ImapProbingParser, ImapProbingParser);</div>
<div> AppLayerProtoDetectPPRegister(IPPROTO_TCP,</div>
<div> IMAP_DEFAULT_PORT, ALPROTO_IMAP, 0,</div>
<div> IMAP_MIN_TO_SERVER_FRAME_LEN, STREAM_TOCLIENT,</div>
<span> ImapProbingParser, ImapProbingParser);</span></div>
<div style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<div style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
Besides, the pcap replaying also triggers 2 alerts:</div>
<div style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
<span>[mazh@localhost log]$ vim fast.log<br>
</span>
<div>[mazh@localhost log]$ cat fast.log</div>
<div>05/22/2019-16:30:48.587578 [**] [1:2000328:12] ET POLICY Outbound Multiple Non-SMTP Server Emails [**] [Classification: policy-violation] [Priority: 3] {TCP} 10.21.37.60:10373 -> 10.21.17.206:25</div>
<div>05/22/2019-16:30:48.587578 [**] [1:2002087:10] ET POLICY Inbound Frequent Emails - Possible Spambot Inbound [**] [Classification: policy-violation] [Priority: 3] {TCP} 10.21.37.60:10373 -> 10.21.17.206:25</div>
<div>[mazh@localhost log]$</div>
<span></span><br>
</div>
<div style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<div style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
Build-info:</div>
<div style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
<span>[mazh@localhost test_spiderFlow]$ bin/spiderflow --build-info<br>
</span>
<div>This is spiderflow version 4.1.2</div>
<div>Features: DEBUG PCAP_SET_BUFF AF_PACKET NETMAP HAVE_PACKET_FANOUT LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_LIBJANSSON TLS MAGIC</div>
<div>SIMD support: SSE_4_2 SSE_4_1 SSE_3</div>
<div>Atomic intrisics: 1 2 4 8 16 byte(s)</div>
<div>64-bits, Little-endian architecture</div>
<div>GCC version 4.8.5 20150623 (Red Hat 4.8.5-16), C version 199901</div>
<div>compiled with _FORTIFY_SOURCE=0</div>
<div>L1 cache line size (CLS)=64</div>
<div>thread local storage method: __thread</div>
<div>compiled with LibHTP v0.5.26, linked against LibHTP v0.5.26</div>
<div></div>
<div>Suricata Configuration:</div>
<div> AF_PACKET support: yes</div>
<div> eBPF support: no</div>
<div> XDP support:</div>
<div> PF_RING support: no</div>
<div> NFQueue support: no</div>
<div> NFLOG support: no</div>
<div> IPFW support: no</div>
<div> Netmap support: yes</div>
<div> DAG enabled: no</div>
<div> Napatech enabled: no</div>
<div></div>
<div> Unix socket enabled: yes</div>
<div> Detection enabled: yes</div>
<div></div>
<div> Libmagic support: yes</div>
<div> libnss support:</div>
<div> libnspr support:</div>
<div> libjansson support: yes</div>
<div> liblzma support: no</div>
<div> hiredis support: no</div>
<div> hiredis async with libevent: no</div>
<div> Prelude support: no</div>
<div> PCRE jit: yes</div>
<div> LUA support: no</div>
<div> libluajit: no</div>
<div> libgeoip: no</div>
<div> libmysql: no</div>
<div> Non-bundled htp: yes</div>
<div> Old barnyard2 support: no</div>
<div> Hyperscan support: no</div>
<div> Libnet support: yes</div>
<div> liblz4 support: no</div>
<div></div>
<div> Rust support (experimental): no</div>
<div> Rust strict mode: no</div>
<div> Rust debug mode: no</div>
<div></div>
<div> Suricatasc install: yes</div>
<div></div>
<div> Profiling enabled: no</div>
<div> Profiling locks enabled: no</div>
<div></div>
<div>Development settings:</div>
<div> Coccinelle / spatch: no</div>
<div> Unit tests enabled: no</div>
<div> Debug output enabled: yes</div>
<div> Debug validation enabled: no</div>
<div></div>
<div>Generic build parameters:</div>
<div> Installation prefix: /home/mazh/test_spiderFlow</div>
<div> Configuration directory: /home/mazh/test_spiderFlow/etc/suricata/</div>
<div> Log directory: /home/mazh/test_spiderFlow/log/suricata/</div>
<div></div>
<div> --prefix /home/mazh/test_spiderFlow</div>
<div> --sysconfdir /home/mazh/test_spiderFlow/etc</div>
<div> --localstatedir /home/mazh/test_spiderFlow</div>
<div></div>
<div> Host: x86_64-unknown-linux-gnu</div>
<div> Compiler: gcc (exec name) / gcc (real)</div>
<div> GCC Protect enabled: no</div>
<div> GCC march native enabled: yes</div>
<div> GCC Profile enabled: no</div>
<div> Position Independent Executable enabled: no</div>
<div> CFLAGS -g -O0 -march=native</div>
<div> PCAP_CFLAGS</div>
<div> SECCFLAGS</div>
<div>[mazh@localhost test_spiderFlow]$</div>
<span></span><br>
</div>
<div style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
Any constructive suggestions are appreciated.</div>
<div style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<div style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
BR,</div>
<div style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif; font-size:12pt">
Allen Ma</div>
</div>
</div>
</body>
</html>