[Oisf-devel] htp callback HTPCallbackRequest is not called when upload a "large" file( >5KB)

[Victor Julien]

What happens if you record the traffic and process the pcap with Suricata?

On 07/12/2012 10:00 AM, Delta Yeh wrote:
> I did some tests with suricata git head.
> 
> The test setup is
> 
> client box ------suricata box (bridge)----webserver
> 
> I start suricata with  ./suricata -c /tmp/suricata.yaml -i eth0
> 
> 
> I upload 2 file, file a is 45 bytes ,file b is 6.1KB
> I add debug codes  in htp callback function
>    1. HTPCallbackRequestBodyData
>    2. HTPCallbackRequest
>    3. HTPCallbackResponse
>   in  file  app-layer-htp.c
> 
> According to the design, I should see the debug output of the above 3 function.
> 
> When upload file a, the output is as expected:
> 
> [25497] 12/7/2012 -- 15:47:16 - (app-layer-htp.c:1804) <Error>
> (HTPCallbackRequestBodyData) -- [ERRCODE: SC_ERR_INVALID_VALUE(129)] -
> ===>HTPCallbackRequestBodyData(0<-376)
> [25497] 12/7/2012 -- 15:47:16 - (app-layer-htp.c:1991) <Error>
> (HTPCallbackRequest) -- [ERRCODE: SC_ERR_INVALID_VALUE(129)] -
> ===>HTPCallbackRequest
> [25497] 12/7/2012 -- 15:47:16 - (app-layer-htp.c:2031) <Error>
> (HTPCallbackResponse) -- [ERRCODE: SC_ERR_INVALID_VALUE(129)] -
> <===HTPCallbackResponse
> 
> 
> but for file b, I didn't see invocation of  HTPCallbackRequest  !!!!
> 
> [25497] 12/7/2012 -- 15:42:09 - (app-layer-htp.c:1804) <Error>
> (HTPCallbackRequestBodyData) -- [ERRCODE: SC_ERR_INVALID_VALUE(129)] -
> ===>HTPCallbackRequestBodyData(0<-1460)
> [25497] 12/7/2012 -- 15:42:09 - (app-layer-htp.c:1804) <Error>
> (HTPCallbackRequestBodyData) -- [ERRCODE: SC_ERR_INVALID_VALUE(129)] -
> ===>HTPCallbackRequestBodyData(1460<-1460)
> [25497] 12/7/2012 -- 15:42:09 - (app-layer-htp.c:2031) <Error>
> (HTPCallbackResponse) -- [ERRCODE: SC_ERR_INVALID_VALUE(129)] -
> <===HTPCallbackResponse
> 
> 
> And the request body processed is less than the file size.
> There should be 6K, but only 1460+1460 is processed.
> 
> 
> The config of suricata is:
> 
> %YAML 1.1
> ---
> pid-file: /var/run/suricata.pid
> runmode: workers
> autofp-scheduler: active-packets
> default-packet-size: 1514
> max-pending-packets: 500
> 
> default-log-dir: /tmp/logs
> 
> 
> # Configure the type of alert (and other) logging you would like.
> outputs:
>   # a line based alerts log similar to Snort's fast.log
>   - fast:
>       enabled: yes
>       filename: fast.log
> 
>   - http-log:
>       enabled: no
>       filename: http.log
>   - stats:
>       enabled: no
>       filename: websensor.stat
>       interval: 2592000
> 
> nfq:
> #  mode: accept
> #  repeat-mark: 1
> #  repeat-mask: 1
> #  route-queue: 2
> 
> defrag:
>   max-frags: 65535
>   prealloc: yes
>   timeout: 3
> detect-engine:
>   - profile: custom
>   - custom-values:
>       toclient-src-groups: 2
>       toclient-dst-groups: 2
>       toclient-sp-groups: 2
>       toclient-dp-groups: 2
>       toserver-src-groups: 2
>       toserver-dst-groups: 3
>       toserver-sp-groups: 2
>       toserver-dp-groups: 5
>   - sgh-mpm-context: single
>   - inspection-recursion-limit: 10
> 
> threading:
>   set-cpu-affinity: no
>   detect-thread-ratio: 1.5
> 
> mpm-algo: ac
> pattern-matcher:
>   - b2gc:
>       search-algo: B2gSearchBNDMq
>       hash-size: low
>       bf-size: low
>   - b2gm:
>       search-algo: B2gSearchBNDMq
>       hash-size: low
>       bf-size: low
>   - b2g:
>       search-algo: B2gSearchBNDMq
>       hash-size: low
>       bf-size: low
>   - b3g:
>       search-algo: B3gSearchBNDMq
>       hash-size: low
>       bf-size: low
>   - wumanber:
>       hash-size: low
>       bf-size: low
> 
> # Defrag settings:
> defrag:
>   max-frags: 65535
>   prealloc: yes
>   timeout: 20
> 
> flow:
>   memcap: 32mb
>   hash-size: 65536
>   prealloc: 10000
>   emergency-recovery: 30
>   prune-flows: 5
> 
> flow-timeouts:
> 
>   default:
>     new: 3
>     established: 5
>     closed: 0
>     emergency-new: 1
>     emergency-established: 1
>     emergency-closed: 0
>   tcp:
>     new: 3
>     established: 5
>     closed: 0
>     emergency-new: 1
>     emergency-established: 1
>     emergency-closed: 0
>   udp:
>     new: 1
>     established: 1
>     emergency-new: 1
>     emergency-established: 1
>   icmp:
>     new: 1
>     established: 1
>     emergency-new: 1
>     emergency-established: 1
> 
> stream:
>   memcap: 32mb
>   checksum-validation: no
>   max-sessions: 2000000
>   midstream: false
>   inline: no                    # no inline mode
>   reassembly:
>     memcap: 64mb
>     depth: 1mb                  # reassemble 1mb into a stream
>     toserver-chunk-size: 2560
>     toclient-chunk-size: 2560
> 
> 
> host:
>   hash-size: 4096
>   prealloc: 1000
>   memcap: 16777216
> logging:
>   default-log-level: error
> pcap:
>   - interface: eth0
>     buffer-size: 32768
>     checksum-checks: no
> 
> 
> classification-file: /etc/suricata/conf/classification.config
> reference-config-file: /etc/suricata/conf/reference.config
> 
> action-order:
>   - pass
>   - drop
>   - reject
>   - alert
> pcre:
>   match-limit: 3500
>   match-limit-recursion: 1500
> 
> libhtp:
> 
>    default-config:
>      personality: IDS
>      request-body-limit: 0
>      response-body-limit: 1024
> coredump:
>   max-dump: unlimited
> 


-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------