[Oisf-users] Issues with Application Layer Filtering

[Jason Batchelor]

Hello:

After having no luck with PF_RING (see email on File Extraction Woes) I
decided to recompile a new kernel (3.14.5) on my CentOS machine and give
AF_PACKET a shot. I am happy to report that the total loss is at a much
more manageable level.

Unfortunately, I seemed to have encountered a different issue concerning
filtering and alerting at the application layer. Suri seems to have a
difficult time working with HTTP rules I create as well as logging when run
on the wire.

For example, take the following rule...

alert tcp any any -> 216.34.181.45 any (msg:"This is a test TCP Rule";
flow:established,to_server; content:"slashdot.org"; http_header; sid:9998;
rev:1;)

When running Suri on the wire. This alert will simply not fire at all.
Moreover, I notice on several occasions the requests are not logged
properly to http.log. This of course, has an amplifying effect on file
extraction where samples are truncated at best, and at worst not ever
logged.

I decided to test things by taking the same data as a pcap, and running it
against Suri...

/usr/bin/suricata -c /etc/suricata/suricata.yaml --runmode single -r
slashdot.pcap

At this point, Suri does detect on the traffic in the PCAP.

So it would seem the issue is isolated to traffic on the wire. I decided to
check the following parameters to see if there were any issues...

Example thread (1 of 16).
capture.kernel_packets    | AFPacketp4p21             | 60645314
capture.kernel_drops      | AFPacketp4p21             | 71483
tcp.segment_memcap_drop   | AFPacketp4p21             | 0
tcp.reassembly_memuse     | AFPacketp4p21             | 1306325000
decoder.invalid           | AFPacketp4p21             | 0

Seems fine to me, and the rest of the threads are more/less the same.

Now if I take a non application layer rule, like a simple ICMP rule below...

alert icmp $MY_IP_ADDRESS any -> any any (msg:"This is a test ICMP Rule";
sid:9999; rev:1;)

I hit reliably both on the wire and on a pcap. This is why I feel there has
to be some issue with application layer processing. I am somewhat perplexed
by this and seek any guidance those with more experience may be able to
give. Thanks in advance!

Here are some additional config attributes that might be helpful.

== Suri Config==

runmode: workers

host-mode: sniffer-only

default-packet-size: 9000
## Stream Settings ##
stream:
  memcap: 1gb
  checksum-validation: no
  prealloc-sessions: 10000000
  inline: no
  reassembly:
    memcap: 2gb
    depth: 6mb                  # reassemble 1mb into a stream
    toserver-chunk-size: 2560
    toclient-chunk-size: 2560
    randomize-chunk-size: yes
    #randomize-chunk-range: 10
    #raw: yes
    chunk-prealloc: 100000
    segments:
      - size: 4
        prealloc: 100000
      - size: 16
        prealloc: 100000
      - size: 112
        prealloc: 500000
      - size: 248
        prealloc: 250000
      - size: 512
        prealloc: 100000
      - size: 768
        prealloc: 50000
      - size: 1448
        prealloc: 300000
      - size: 65535
        prealloc: 3000

### Flow Timeouts Snip ###
  tcp:
    new: 60
    established: 120
    closed: 30
    emergency-new: 10
    emergency-established: 10
    emergency-closed: 10

### Flow ###
flow:
  memcap: 1gb
  hash-size: 1048576
  prealloc: 1048576
  emergency-recovery: 30

### Detect Engine ###
detect-engine:
  - profile: custom
  - custom-values:
      toclient-src-groups: 200
      toclient-dst-groups: 200
      toclient-sp-groups: 200
      toclient-dp-groups: 300
      toserver-src-groups: 200
      toserver-dst-groups: 400
      toserver-sp-groups: 200
      toserver-dp-groups: 250
  - sgh-mpm-context: auto
  - inspection-recursion-limit: 3000

Using the following hardware:

== Hardware Specs ==
CPU: Intel Xeon CPU @ 2.40Ghz x 32
RAM: 48G
NIC:
  *-network:1
       description: Ethernet interface
       product: Ethernet 10G 2P X520 Adapter
       vendor: Intel Corporation
       physical id: 0.1
       bus info: pci at 0000:42:00.1
       logical name: p4p2
       version: 01
       serial: a0:36:9f:07:ec:02
       capacity: 1GB/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi msix pciexpress vpd bus_master cap_list rom
ethernet physical fibre 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=ixgbe
driverversion=3.19.1-k duplex=full firmware=0x8000030d latency=0 link=yes
multicast=yes port=fibre
       resources: irq:76 memory:d0f00000-d0f7ffff ioport:7ce0(size=32)
memory:d0ffc000-d0ffffff memory:d1100000-d117ffff memory:d1380000-d147ffff
memory:d1480000-d157ffff
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20140606/3aa28a28/attachment.html>