[Oisf-devel] htp callback HTPCallbackRequest is not called when upload a "large" file( >5KB)

[Delta Yeh]

I did some tests with suricata git head.

The test setup is

client box ------suricata box (bridge)----webserver

I start suricata with  ./suricata -c /tmp/suricata.yaml -i eth0


I upload 2 file, file a is 45 bytes ,file b is 6.1KB
I add debug codes  in htp callback function
   1. HTPCallbackRequestBodyData
   2. HTPCallbackRequest
   3. HTPCallbackResponse
  in  file  app-layer-htp.c

According to the design, I should see the debug output of the above 3 function.

When upload file a, the output is as expected:

[25497] 12/7/2012 -- 15:47:16 - (app-layer-htp.c:1804) <Error>
(HTPCallbackRequestBodyData) -- [ERRCODE: SC_ERR_INVALID_VALUE(129)] -
===>HTPCallbackRequestBodyData(0<-376)
[25497] 12/7/2012 -- 15:47:16 - (app-layer-htp.c:1991) <Error>
(HTPCallbackRequest) -- [ERRCODE: SC_ERR_INVALID_VALUE(129)] -
===>HTPCallbackRequest
[25497] 12/7/2012 -- 15:47:16 - (app-layer-htp.c:2031) <Error>
(HTPCallbackResponse) -- [ERRCODE: SC_ERR_INVALID_VALUE(129)] -
<===HTPCallbackResponse


but for file b, I didn't see invocation of  HTPCallbackRequest  !!!!

[25497] 12/7/2012 -- 15:42:09 - (app-layer-htp.c:1804) <Error>
(HTPCallbackRequestBodyData) -- [ERRCODE: SC_ERR_INVALID_VALUE(129)] -
===>HTPCallbackRequestBodyData(0<-1460)
[25497] 12/7/2012 -- 15:42:09 - (app-layer-htp.c:1804) <Error>
(HTPCallbackRequestBodyData) -- [ERRCODE: SC_ERR_INVALID_VALUE(129)] -
===>HTPCallbackRequestBodyData(1460<-1460)
[25497] 12/7/2012 -- 15:42:09 - (app-layer-htp.c:2031) <Error>
(HTPCallbackResponse) -- [ERRCODE: SC_ERR_INVALID_VALUE(129)] -
<===HTPCallbackResponse


And the request body processed is less than the file size.
There should be 6K, but only 1460+1460 is processed.


The config of suricata is:

%YAML 1.1
---
pid-file: /var/run/suricata.pid
runmode: workers
autofp-scheduler: active-packets
default-packet-size: 1514
max-pending-packets: 500

default-log-dir: /tmp/logs


# Configure the type of alert (and other) logging you would like.
outputs:
  # a line based alerts log similar to Snort's fast.log
  - fast:
      enabled: yes
      filename: fast.log

  - http-log:
      enabled: no
      filename: http.log
  - stats:
      enabled: no
      filename: websensor.stat
      interval: 2592000

nfq:
#  mode: accept
#  repeat-mark: 1
#  repeat-mask: 1
#  route-queue: 2

defrag:
  max-frags: 65535
  prealloc: yes
  timeout: 3
detect-engine:
  - profile: custom
  - custom-values:
      toclient-src-groups: 2
      toclient-dst-groups: 2
      toclient-sp-groups: 2
      toclient-dp-groups: 2
      toserver-src-groups: 2
      toserver-dst-groups: 3
      toserver-sp-groups: 2
      toserver-dp-groups: 5
  - sgh-mpm-context: single
  - inspection-recursion-limit: 10

threading:
  set-cpu-affinity: no
  detect-thread-ratio: 1.5

mpm-algo: ac
pattern-matcher:
  - b2gc:
      search-algo: B2gSearchBNDMq
      hash-size: low
      bf-size: low
  - b2gm:
      search-algo: B2gSearchBNDMq
      hash-size: low
      bf-size: low
  - b2g:
      search-algo: B2gSearchBNDMq
      hash-size: low
      bf-size: low
  - b3g:
      search-algo: B3gSearchBNDMq
      hash-size: low
      bf-size: low
  - wumanber:
      hash-size: low
      bf-size: low

# Defrag settings:
defrag:
  max-frags: 65535
  prealloc: yes
  timeout: 20

flow:
  memcap: 32mb
  hash-size: 65536
  prealloc: 10000
  emergency-recovery: 30
  prune-flows: 5

flow-timeouts:

  default:
    new: 3
    established: 5
    closed: 0
    emergency-new: 1
    emergency-established: 1
    emergency-closed: 0
  tcp:
    new: 3
    established: 5
    closed: 0
    emergency-new: 1
    emergency-established: 1
    emergency-closed: 0
  udp:
    new: 1
    established: 1
    emergency-new: 1
    emergency-established: 1
  icmp:
    new: 1
    established: 1
    emergency-new: 1
    emergency-established: 1

stream:
  memcap: 32mb
  checksum-validation: no
  max-sessions: 2000000
  midstream: false
  inline: no                    # no inline mode
  reassembly:
    memcap: 64mb
    depth: 1mb                  # reassemble 1mb into a stream
    toserver-chunk-size: 2560
    toclient-chunk-size: 2560


host:
  hash-size: 4096
  prealloc: 1000
  memcap: 16777216
logging:
  default-log-level: error
pcap:
  - interface: eth0
    buffer-size: 32768
    checksum-checks: no


classification-file: /etc/suricata/conf/classification.config
reference-config-file: /etc/suricata/conf/reference.config

action-order:
  - pass
  - drop
  - reject
  - alert
pcre:
  match-limit: 3500
  match-limit-recursion: 1500

libhtp:

   default-config:
     personality: IDS
     request-body-limit: 0
     response-body-limit: 1024
coredump:
  max-dump: unlimited