[Oisf-users] where are my missing packets ?

[Martin Holste]

Some key settings I'm using on 750 Mb/sec with similar hardware but more RAM:

# Set context to full (you might not have enough RAM for this)
detect-engine:
  - profile: high
  - custom-values:
      toclient_src_groups: 2
      toclient_dst_groups: 2
      toclient_sp_groups: 2
      toclient_dp_groups: 3
      toserver_src_groups: 2
      toserver_dst_groups: 4
      toserver_sp_groups: 2
      toserver_dp_groups: 25
  - sgh-mpm-context: full
  - inspection-recursion-limit: 3000

# Run ac (you might not have enough RAM for this)
mpm-algo: ac

# Larger than default memcap and prune_flows
flow:
  memcap: 3294967295
  hash_size: 108435456
  prealloc: 10000
  emergency_recovery: 40
  prune_flows: 500

# Use fewer detect threads
threading:
  detect_thread_ratio: .5

# Aggressive flow timeouts
flow-timeouts:
  default:
    new: 1 # 30
    established: 10 #300
    closed: 0
    emergency_new: 1 #10
    emergency_established: 1 #100
    emergency_closed: 0
  tcp:
    new: 1 #60
    established: 10 #3600
    closed: 0 #120
    emergency_new: 1 #10
    emergency_established: 5 #1 #300
    emergency_closed: 20
  udp:
    new: 1 #30
    established: 1 #300
    emergency_new: 1 #10
    emergency_established: 1 #100
  icmp:
    new: 1 #30
    established: 1 #300
    emergency_new: 1 #10
    emergency_established: 1 #100

# Larger stream buffer
stream:
  memcap: 3294967295
  checksum_validation: no      # reject wrong csums
  inline: no                    # no inline mode
  reassembly:
    memcap: 4294967295
    depth: 1048576              # reassemble 1mb into a stream
    toserver_chunk_size: 2560
    toclient_chunk_size: 2560



On Thu, Feb 23, 2012 at 7:18 AM, Victor Julien <victor at inliniac.net> wrote:
> On 02/23/2012 11:49 AM, Travel Factory S.r.l. wrote:
>> On Thu, 23 Feb 2012 11:18:33 +0100
>>  Victor Julien <victor at inliniac.net> wrote:
>>> On 02/23/2012 10:45 AM, Travel Factory S.r.l. wrote:
>>>>     depth: 50mb                  # reassemble 1mb into a stream
>>>
>>> Any particular reason for this setting? This means large transfers, like
>>> big downloads, will be tracked much longer than normal.
>>
>> No, actually I raised every parameter regarding memory. I should read
>> again the suricata.yaml parameters description.
>> Should I lower it ?
>
> Might help, ya.
>
>>
>> Anyhow, as expected, after 35:00,
>>
>> tcp.segment_memcap_drop   | Detect                    | 0
>> tcp.reassembly_memuse     | Detect                    | 38506791088.000000
>> tcp.segment_memcap_drop   | Detect                    | 0
>> tcp.reassembly_memuse     | Detect                    | 38596590000.000000
>> tcp.segment_memcap_drop   | Detect                    | 157
>> tcp.reassembly_memuse     | Detect                    | 38654700066.000000
>> tcp.segment_memcap_drop   | Detect                    | 6057
>> tcp.reassembly_memuse     | Detect                    | 38654705250.000000
>> tcp.segment_memcap_drop   | Detect                    | 13473
>
> You might want to lower the flow time outs for TCP in your yaml file.
>
>>
>>
>> The only rule file active has these 2 rules:
>>
>> alert tcp any any -> any any (msg:"FILE-IDENTIFY PDF file magic
>> detection"; flow:to_client,established; file_data; content:"%PDF-";
>> fast_pattern; nocase; flowbits:set,file.pdf; classtype:misc-activity;
>> sid:2049499999; rev:3;)
>>
>> alert ip [10.my.ip.address] any -> any any (msg:"FRANK traffic";
>> threshold: type limit, track by_src, seconds 60, count 1;
>> sid:2405998999; rev:277;)
>>
>>
>> The second rule is triggered and I see one message every 60 seconds, the
>> first rule is not triggered when I do traffic from my pc but I see it in
>> the log when traffic is made from other workstations... is the second
>> rule masking the first ??? Or am I still losing packets ???
>>
>
> Might be caused by bad checksums. Try disabling stream.checksum_validation.
>
> --
> ---------------------------------------------
> Victor Julien
> http://www.inliniac.net/
> PGP: http://www.inliniac.net/victorjulien.asc
> ---------------------------------------------
>
> _______________________________________________
> Oisf-users mailing list
> Oisf-users at openinfosecfoundation.org
> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users