[Oisf-users] 回复: [4.1.0-beta1] I implement IMAP protocol detection & parser. When replaying pcap file with ONLY IMAP protocol, it works well. However, while replaying pcap file mixing with SMTP and IMAP protocols, SMTP parsing works well and IMAP parsing fails.
Ma Allen
mazhh at outlook.com
Tue May 28 13:36:14 UTC 2019
Hi All,
There's some additional information which may be helpful as I've attached.
IMAP related items in flow information is a bit abnormal, as state is "new" while reason is "shutdown":
{"CREATE_TIME":"2019-05-22T16:30:48.587578+0800","flow_id":309972231175741,"LOG_TYPE":"flow","src_mac":"04:7D:7B:B4:01:4A","SRC_IP":"10.21.37.60","SRC_PORT":10345,"dst_mac":"00:1B:0C:CE:5B:D0","DST_IP":"10.21.17.206","DST_PORT":143,"PROTOCOL":"TCP","flow":{"pkts_toserver":4,"pkts_toclient":5,"bytes_toserver":285,"bytes_toclient":1955,"start":"2019-05-22T16:30:50.638525+0800","end":"2019-05-22T16:30:50.878531+0800","age":0,"state":"new","reason":"shutdown","alerted":false},"tcp":{"tcp_flags":"00","tcp_flags_ts":"00","tcp_flags_tc":"00"}}
Normal item:
// SMTP
// state is closed while reason is "shutdown"
{"CREATE_TIME":"2019-05-22T16:30:48.587578+0800","flow_id":724273366290807,"LOG_TYPE":"flow","src_mac":"04:7D:7B:B4:01:4A","SRC_IP":"10.21.37.60","SRC_PORT":10367,"dst_mac":"00:1B:0C:CE:5B:D0","DST_IP":"10.21.17.206","DST_PORT":25,"PROTOCOL":"TCP","APP_PROTOCOL":"smtp","app_proto_tc":"failed","flow":{"pkts_toserver":11,"pkts_toclient":12,"bytes_toserver":1208,"bytes_toclient":959,"start":"2019-05-22T16:30:47.184695+0800","end":"2019-05-22T16:30:47.244888+0800","age":0,"state":"closed","reason":"shutdown","alerted":false},"tcp":{"tcp_flags":"1b","tcp_flags_ts":"1b","tcp_flags_tc":"1b","syn":true,"fin":true,"psh":true,"ack":true,"state":"closed"}}
________________________________
发件人: Ma Allen <mazhh at outlook.com>
发送时间: 2019年5月28日 17:16
收件人: oisf-devel-request at lists.openinfosecfoundation.org; oisf-users at lists.openinfosecfoundation.org
主题: [4.1.0-beta1] I implement IMAP protocol detection & parser. When replaying pcap file with ONLY IMAP protocol, it works well. However, while replaying pcap file mixing with SMTP and IMAP protocols, SMTP parsing works well and IMAP parsing fails.
Hi All,
Phenomenon:
[4.1.0-beta1] I implement IMAP protocol detection & parser. When replaying pcap file with ONLY IMAP protocol, it works well. However, while replaying pcap file mixing with SMTP and IMAP protocols, SMTP parsing works well and IMAP parsing fails. There seems something wrong with reading IMAP packets.
I tried to locate where IMAP packets are lost, and add the following debug code in 3 functions AppLayerParserParse, TCPProtoDetect and AppLayerHandleTCPData:
bool check_flag = false;
if (BasicSearchNocase(data, data_len, " FETCH ", strlen(" FETCH ")) != NULL) { // " FETCH " is the command I'd like to parse in IMAP
check_flag = true;
}
and add breakpoint in the line, "check_flag = true;". However, no breakpoints are effective. So I believe that IMAP packets are lost in the lower function in the following calling stack.
#0 AppLayerParserParse (tv=tv at entry=0x48ab230, alp_tctx=0x7fffd8011d30, f=f at entry=0x1485c50, alproto=7, flags=flags at entry=5 '\005',
input=input at entry=0x7fffd82a8520 "EHLO 10.21.37.60\r\n", input_len=input_len at entry=18) at app-layer-parser.c:1092
#1 0x0000000000418bbf in TCPProtoDetect (tv=<optimized out>, ra_ctx=<optimized out>, app_tctx=app_tctx at entry=0x7fffd8011940, p=p at entry=0x7fffe0268ea0,
f=f at entry=0x1485c50, ssn=ssn at entry=0x7fffd80da6a0, stream=stream at entry=0x7fffd80da730, data=data at entry=0x7fffd82a8520 "EHLO 10.21.37.60\r\n",
data_len=data_len at entry=18, flags=flags at entry=5 '\005') at app-layer.c:431
#2 0x0000000000419166 in AppLayerHandleTCPData (tv=tv at entry=0x48ab230, ra_ctx=ra_ctx at entry=0x7fffd8011910, p=p at entry=0x7fffe0268ea0, f=0x1485c50,
ssn=ssn at entry=0x7fffd80da6a0, stream=stream at entry=0x7fffd80da730, data=0x7fffd82a8520 "EHLO 10.21.37.60\r\n", data_len=data_len at entry=18,
flags=5 '\005') at app-layer.c:590
#3 0x000000000059d5a2 in ReassembleUpdateAppLayer (dir=<optimized out>, p=<optimized out>, stream=<optimized out>, ssn=<optimized out>,
ra_ctx=<optimized out>, tv=<optimized out>) at stream-tcp-reassemble.c:1063
#4 StreamTcpReassembleAppLayer (tv=0x48ab230, ra_ctx=0x7fffd8011910, ssn=0x7fffd80da6a0, stream=0x7fffd80da730, p=0x7fffe0268ea0,
dir=UPDATE_DIR_OPPOSING) at stream-tcp-reassemble.c:1136
#5 0x000000000059e101 in StreamTcpReassembleHandleSegmentUpdateACK (p=0x7fffe0268ea0, stream=0x7fffd80da730, ssn=0x7fffd80da6a0, ra_ctx=0x7fffd8011910,
tv=0x48ab230) at stream-tcp-reassemble.c:1685
#6 StreamTcpReassembleHandleSegment (tv=tv at entry=0x48ab230, ra_ctx=0x7fffd8011910, ssn=ssn at entry=0x7fffd80da6a0, stream=0x7fffd80da6b0,
p=p at entry=0x7fffe0268ea0, pq=pq at entry=0x7fffd80115f8) at stream-tcp-reassemble.c:1724
#7 0x0000000000594d97 in HandleEstablishedPacketToClient (stt=<optimized out>, pq=<optimized out>, p=<optimized out>, ssn=<optimized out>,
tv=<optimized out>) at stream-tcp.c:2362
#8 StreamTcpPacketStateEstablished (tv=tv at entry=0x48ab230, p=p at entry=0x7fffe0268ea0, stt=stt at entry=0x7fffd80115f0, ssn=ssn at entry=0x7fffd80da6a0,
pq=pq at entry=0x7fffd80115f8) at stream-tcp.c:2599
#9 0x00000000005992fb in StreamTcpPacket (tv=0x48ab230, p=0x7fffe0268ea0, stt=0x7fffd80115f0, pq=0x7fffd80008e0) at stream-tcp.c:4645
#10 0x000000000059a690 in StreamTcp (tv=tv at entry=0x48ab230, p=p at entry=0x7fffe0268ea0, data=<optimized out>, pq=pq at entry=0x7fffd80008e0,
postpq=postpq at entry=0x0) at stream-tcp.c:5020
#11 0x0000000000529901 in FlowWorker (tv=0x48ab230, p=0x7fffe0268ea0, data=0x7fffd80008c0, preq=0x1baf0e0, unused=<optimized out>) at flow-worker.c:216
#12 0x00000000005a5fa4 in TmThreadsSlotVarRun (tv=tv at entry=0x48ab230, p=p at entry=0x7fffe0268ea0, slot=slot at entry=0x1baf0a0) at tm-threads.c:143
#13 0x00000000005a8eba in TmThreadsSlotVar (td=0x48ab230) at tm-threads.c:598
#14 0x00007ffff6c33dd5 in start_thread () from /lib64/libpthread.so.0
#15 0x00007ffff60a5ead in clone () from /lib64/libc.so.6
I attached the pcap file leading to the problem. As it works well while replaying the pcap file containing ONLY IMAP protocol, I believe that SMTP packets before IMAP make IMAP protocol detection fail.
SMTP & IMAP Registration:
// SMTP -- Pattern Matcher
static int SMTPRegisterPatternsForProtocolDetection(void)
{
if (AppLayerProtoDetectPMRegisterPatternCI(IPPROTO_TCP,
ALPROTO_SMTP,
"EHLO",
4,
0,
STREAM_TOSERVER) < 0)
{
return -1;
}
if (AppLayerProtoDetectPMRegisterPatternCI(IPPROTO_TCP,
ALPROTO_SMTP,
"HELO",
4,
0,
STREAM_TOSERVER) < 0)
{
return -1;
}
if (AppLayerProtoDetectPMRegisterPatternCI(IPPROTO_TCP,
ALPROTO_SMTP,
"QUIT",
4,
0,
STREAM_TOSERVER) < 0)
{
return -1;
}
return 0;
}
// IMAP -- Probing Parser
// client command: a0003 SELECT "INBOX"\r\n
// tag length:5 Maximum command(AUTHENTICATE) length: 12 Assumed mximum argument length: 120
#define IMAP_MIN_TO_SERVER_FRAME_LEN (5 + 1 + 12 + 1 + 120 + 2)
AppLayerProtoDetectPPRegister(IPPROTO_TCP,
IMAP_DEFAULT_PORT, ALPROTO_IMAP, 0,
IMAP_MIN_TO_SERVER_FRAME_LEN, STREAM_TOSERVER,
ImapProbingParser, ImapProbingParser);
AppLayerProtoDetectPPRegister(IPPROTO_TCP,
IMAP_DEFAULT_PORT, ALPROTO_IMAP, 0,
IMAP_MIN_TO_SERVER_FRAME_LEN, STREAM_TOCLIENT,
ImapProbingParser, ImapProbingParser);
Besides, the pcap replaying also triggers 2 alerts:
[mazh at localhost log]$ vim fast.log
[mazh at localhost log]$ cat fast.log
05/22/2019-16:30:48.587578 [**] [1:2000328:12] ET POLICY Outbound Multiple Non-SMTP Server Emails [**] [Classification: policy-violation] [Priority: 3] {TCP} 10.21.37.60:10373 -> 10.21.17.206:25
05/22/2019-16:30:48.587578 [**] [1:2002087:10] ET POLICY Inbound Frequent Emails - Possible Spambot Inbound [**] [Classification: policy-violation] [Priority: 3] {TCP} 10.21.37.60:10373 -> 10.21.17.206:25
[mazh at localhost log]$
Build-info:
[mazh at localhost test_spiderFlow]$ bin/spiderflow --build-info
This is spiderflow version 4.1.2
Features: DEBUG PCAP_SET_BUFF AF_PACKET NETMAP HAVE_PACKET_FANOUT LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_LIBJANSSON TLS MAGIC
SIMD support: SSE_4_2 SSE_4_1 SSE_3
Atomic intrisics: 1 2 4 8 16 byte(s)
64-bits, Little-endian architecture
GCC version 4.8.5 20150623 (Red Hat 4.8.5-16), C version 199901
compiled with _FORTIFY_SOURCE=0
L1 cache line size (CLS)=64
thread local storage method: __thread
compiled with LibHTP v0.5.26, linked against LibHTP v0.5.26
Suricata Configuration:
AF_PACKET support: yes
eBPF support: no
XDP support:
PF_RING support: no
NFQueue support: no
NFLOG support: no
IPFW support: no
Netmap support: yes
DAG enabled: no
Napatech enabled: no
Unix socket enabled: yes
Detection enabled: yes
Libmagic support: yes
libnss support:
libnspr support:
libjansson support: yes
liblzma support: no
hiredis support: no
hiredis async with libevent: no
Prelude support: no
PCRE jit: yes
LUA support: no
libluajit: no
libgeoip: no
libmysql: no
Non-bundled htp: yes
Old barnyard2 support: no
Hyperscan support: no
Libnet support: yes
liblz4 support: no
Rust support (experimental): no
Rust strict mode: no
Rust debug mode: no
Suricatasc install: yes
Profiling enabled: no
Profiling locks enabled: no
Development settings:
Coccinelle / spatch: no
Unit tests enabled: no
Debug output enabled: yes
Debug validation enabled: no
Generic build parameters:
Installation prefix: /home/mazh/test_spiderFlow
Configuration directory: /home/mazh/test_spiderFlow/etc/suricata/
Log directory: /home/mazh/test_spiderFlow/log/suricata/
--prefix /home/mazh/test_spiderFlow
--sysconfdir /home/mazh/test_spiderFlow/etc
--localstatedir /home/mazh/test_spiderFlow
Host: x86_64-unknown-linux-gnu
Compiler: gcc (exec name) / gcc (real)
GCC Protect enabled: no
GCC march native enabled: yes
GCC Profile enabled: no
Position Independent Executable enabled: no
CFLAGS -g -O0 -march=native
PCAP_CFLAGS
SECCFLAGS
[mazh at localhost test_spiderFlow]$
Any constructive suggestions are appreciated.
BR,
Allen Ma
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20190528/329b4996/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: eve-flow-flow-2019-05-28-16_40_01.dat
Type: application/octet-stream
Size: 10270 bytes
Desc: eve-flow-flow-2019-05-28-16_40_01.dat
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20190528/329b4996/attachment-0001.obj>
More information about the Oisf-users
mailing list