[Oisf-users] [4.1.0-beta1] I implement IMAP protocol detection & parser. When replaying pcap file with ONLY IMAP protocol, it works well. However, while replaying pcap file mixing with SMTP and IMAP protocols, SMTP parsing works well and IMAP parsing fails.

Ma Allen mazhh at outlook.com
Tue May 28 09:16:26 UTC 2019

Hi All,

[4.1.0-beta1] I implement IMAP protocol detection & parser. When replaying pcap file with ONLY IMAP protocol, it works well. However, while replaying pcap file mixing with SMTP and IMAP protocols, SMTP parsing works well and IMAP parsing fails. There seems something wrong with reading IMAP packets.

I tried to locate where IMAP packets are lost, and add the following debug code in 3 functions AppLayerParserParse, TCPProtoDetect and AppLayerHandleTCPData:
    bool check_flag = false;
    if (BasicSearchNocase(data, data_len, " FETCH ", strlen(" FETCH ")) != NULL) {​  // " FETCH " is the command I'd like to parse in IMAP
        check_flag = true;​
and add breakpoint in the line, "check_flag = true;". However, no breakpoints are effective. So I believe that IMAP packets are lost in the lower function in the following calling stack.

#0  AppLayerParserParse (tv=tv at entry=0x48ab230, alp_tctx=0x7fffd8011d30, f=f at entry=0x1485c50, alproto=7, flags=flags at entry=5 '\005',
    input=input at entry=0x7fffd82a8520 "EHLO\r\n", input_len=input_len at entry=18) at app-layer-parser.c:1092​
#1  0x0000000000418bbf in TCPProtoDetect (tv=<optimized out>, ra_ctx=<optimized out>, app_tctx=app_tctx at entry=0x7fffd8011940, p=p at entry=0x7fffe0268ea0,​
    f=f at entry=0x1485c50, ssn=ssn at entry=0x7fffd80da6a0, stream=stream at entry=0x7fffd80da730, data=data at entry=0x7fffd82a8520 "EHLO\r\n",​
    data_len=data_len at entry=18, flags=flags at entry=5 '\005') at app-layer.c:431​
#2  0x0000000000419166 in AppLayerHandleTCPData (tv=tv at entry=0x48ab230, ra_ctx=ra_ctx at entry=0x7fffd8011910, p=p at entry=0x7fffe0268ea0, f=0x1485c50,​
    ssn=ssn at entry=0x7fffd80da6a0, stream=stream at entry=0x7fffd80da730, data=0x7fffd82a8520 "EHLO\r\n", data_len=data_len at entry=18,​
    flags=5 '\005') at app-layer.c:590​
#3  0x000000000059d5a2 in ReassembleUpdateAppLayer (dir=<optimized out>, p=<optimized out>, stream=<optimized out>, ssn=<optimized out>,​
    ra_ctx=<optimized out>, tv=<optimized out>) at stream-tcp-reassemble.c:1063​
#4  StreamTcpReassembleAppLayer (tv=0x48ab230, ra_ctx=0x7fffd8011910, ssn=0x7fffd80da6a0, stream=0x7fffd80da730, p=0x7fffe0268ea0,​
    dir=UPDATE_DIR_OPPOSING) at stream-tcp-reassemble.c:1136​
#5  0x000000000059e101 in StreamTcpReassembleHandleSegmentUpdateACK (p=0x7fffe0268ea0, stream=0x7fffd80da730, ssn=0x7fffd80da6a0, ra_ctx=0x7fffd8011910,​
    tv=0x48ab230) at stream-tcp-reassemble.c:1685​
#6  StreamTcpReassembleHandleSegment (tv=tv at entry=0x48ab230, ra_ctx=0x7fffd8011910, ssn=ssn at entry=0x7fffd80da6a0, stream=0x7fffd80da6b0,​
    p=p at entry=0x7fffe0268ea0, pq=pq at entry=0x7fffd80115f8) at stream-tcp-reassemble.c:1724​
#7  0x0000000000594d97 in HandleEstablishedPacketToClient (stt=<optimized out>, pq=<optimized out>, p=<optimized out>, ssn=<optimized out>,​
    tv=<optimized out>) at stream-tcp.c:2362​
#8  StreamTcpPacketStateEstablished (tv=tv at entry=0x48ab230, p=p at entry=0x7fffe0268ea0, stt=stt at entry=0x7fffd80115f0, ssn=ssn at entry=0x7fffd80da6a0,​
    pq=pq at entry=0x7fffd80115f8) at stream-tcp.c:2599​
#9  0x00000000005992fb in StreamTcpPacket (tv=0x48ab230, p=0x7fffe0268ea0, stt=0x7fffd80115f0, pq=0x7fffd80008e0) at stream-tcp.c:4645​
#10 0x000000000059a690 in StreamTcp (tv=tv at entry=0x48ab230, p=p at entry=0x7fffe0268ea0, data=<optimized out>, pq=pq at entry=0x7fffd80008e0,​
    postpq=postpq at entry=0x0) at stream-tcp.c:5020​
#11 0x0000000000529901 in FlowWorker (tv=0x48ab230, p=0x7fffe0268ea0, data=0x7fffd80008c0, preq=0x1baf0e0, unused=<optimized out>) at flow-worker.c:216​
#12 0x00000000005a5fa4 in TmThreadsSlotVarRun (tv=tv at entry=0x48ab230, p=p at entry=0x7fffe0268ea0, slot=slot at entry=0x1baf0a0) at tm-threads.c:143​
#13 0x00000000005a8eba in TmThreadsSlotVar (td=0x48ab230) at tm-threads.c:598​
#14 0x00007ffff6c33dd5 in start_thread () from /lib64/libpthread.so.0​
#15 0x00007ffff60a5ead in clone () from /lib64/libc.so.6​

I attached the pcap file leading to the problem. As it works well while replaying the pcap file containing ONLY IMAP protocol, I believe that SMTP packets before IMAP make IMAP protocol detection fail.

SMTP & IMAP Registration:
// SMTP -- Pattern Matcher
static int SMTPRegisterPatternsForProtocolDetection(void)
if (AppLayerProtoDetectPMRegisterPatternCI(IPPROTO_TCP,​
 return -1;​
if (AppLayerProtoDetectPMRegisterPatternCI(IPPROTO_TCP,​
 return -1;​
if (AppLayerProtoDetectPMRegisterPatternCI(IPPROTO_TCP,​
 return -1;​
return 0;​

// IMAP -- Probing Parser
// client command: a0003 SELECT "INBOX"\r\n
// tag length:5   Maximum command(AUTHENTICATE) length: 12  Assumed mximum argument length: 120​
#define IMAP_MIN_TO_SERVER_FRAME_LEN (5 + 1 + 12 + 1 + 120 + 2)​

                    IMAP_DEFAULT_PORT, ALPROTO_IMAP, 0,​
                    ImapProbingParser, ImapProbingParser);​
                    IMAP_DEFAULT_PORT, ALPROTO_IMAP, 0,​
                    ImapProbingParser, ImapProbingParser);

Besides, the pcap replaying also triggers 2 alerts:
[mazh at localhost log]$ vim fast.log
[mazh at localhost log]$ cat fast.log​
05/22/2019-16:30:48.587578  [**] [1:2000328:12] ET POLICY Outbound Multiple Non-SMTP Server Emails [**] [Classification: policy-violation] [Priority: 3] {TCP} ->​
05/22/2019-16:30:48.587578  [**] [1:2002087:10] ET POLICY Inbound Frequent Emails - Possible Spambot Inbound [**] [Classification: policy-violation] [Priority: 3] {TCP} ->​
[mazh at localhost log]$​

[mazh at localhost test_spiderFlow]$ bin/spiderflow --build-info
This is spiderflow version 4.1.2​
SIMD support: SSE_4_2 SSE_4_1 SSE_3​
Atomic intrisics: 1 2 4 8 16 byte(s)​
64-bits, Little-endian architecture​
GCC version 4.8.5 20150623 (Red Hat 4.8.5-16), C version 199901​
compiled with _FORTIFY_SOURCE=0​
L1 cache line size (CLS)=64​
thread local storage method: __thread​
compiled with LibHTP v0.5.26, linked against LibHTP v0.5.26​
Suricata Configuration:​
  AF_PACKET support:                       yes​
  eBPF support:                            no​
  XDP support:​
  PF_RING support:                         no​
  NFQueue support:                         no​
  NFLOG support:                           no​
  IPFW support:                            no​
  Netmap support:                          yes​
  DAG enabled:                             no​
  Napatech enabled:                        no​
  Unix socket enabled:                     yes​
  Detection enabled:                       yes​
  Libmagic support:                        yes​
  libnss support:​
  libnspr support:​
  libjansson support:                      yes​
  liblzma support:                         no​
  hiredis support:                         no​
  hiredis async with libevent:             no​
  Prelude support:                         no​
  PCRE jit:                                yes​
  LUA support:                             no​
  libluajit:                               no​
  libgeoip:                                no​
  libmysql:                                no​
  Non-bundled htp:                         yes​
  Old barnyard2 support:                   no​
  Hyperscan support:                       no​
  Libnet support:                          yes​
  liblz4 support:                          no​
  Rust support (experimental):             no​
  Rust strict mode:                        no​
  Rust debug mode:                         no​
  Suricatasc install:                      yes​
  Profiling enabled:                       no​
  Profiling locks enabled:                 no​
Development settings:​
  Coccinelle / spatch:                     no​
  Unit tests enabled:                      no​
  Debug output enabled:                    yes​
  Debug validation enabled:                no​
Generic build parameters:​
  Installation prefix:                     /home/mazh/test_spiderFlow​
  Configuration directory:                 /home/mazh/test_spiderFlow/etc/suricata/​
  Log directory:                           /home/mazh/test_spiderFlow/log/suricata/​
  --prefix                                 /home/mazh/test_spiderFlow​
  --sysconfdir                             /home/mazh/test_spiderFlow/etc​
  --localstatedir                          /home/mazh/test_spiderFlow​
  Host:                                    x86_64-unknown-linux-gnu​
  Compiler:                                gcc (exec name) / gcc (real)​
  GCC Protect enabled:                     no​
  GCC march native enabled:                yes​
  GCC Profile enabled:                     no​
  Position Independent Executable enabled: no​
  CFLAGS                                   -g -O0 -march=native​
[mazh at localhost test_spiderFlow]$​

Any constructive suggestions are appreciated.

Allen Ma
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20190528/acb433d7/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: imap-10mail-foxmail.pcap
Type: application/octet-stream
Size: 74699 bytes
Desc: imap-10mail-foxmail.pcap
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20190528/acb433d7/attachment-0001.obj>

More information about the Oisf-users mailing list