[Oisf-users] Suricata at 10G, packet reassembly
Michał Purzyński
michalpurzynski1 at gmail.com
Wed Feb 1 22:50:32 UTC 2017
I'm not sure statistics you see in netstat is relevant to afpacket at all. I'm pretty sure it's not.
> On Feb 1, 2017, at 1:14 PM, Peter Manev <petermanev at gmail.com> wrote:
>
> On Wed, Feb 1, 2017 at 3:04 PM, Collyer, Jeffrey W. (jwc3f)
> <jwc3f at virginia.edu> wrote:
>> Sure,
>>
>> Some specs
>>
>> Machine
>> Dell R430
>> 2x14 core Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
>> 35M L3 cache
>> 128G Ram
>> Intel X520 ethernet card
>>
>> OS - Ubuntu 16.04.1 LTS
>> Kernel - 4.4.0-59-generic
>> Intel(R) 10GbE PCI Express Linux Network Driver - version 4.4.6
>>
>> Suricata seems to be using about 40.5G resident memory.
>>
>> Last stat update -
>>
>> The tcp.reassembly_memuse appears to be enormous ?16Pb?. Doesn’t seem
>> right.
>>
>
> Yes it does not look right....
>
> Can you please share your stream and reassembly section from your
> suricata.yaml as well ?
> ( https://redmine.openinfosecfoundation.org/projects/suricata/repository/revisions/master/entry/suricata.yaml.in#L1197
> )
>
>>
>> ------------------------------------------------------------------------------------
>> Date: 2/1/2017 -- 08:50:29 (uptime: 0d, 23h 31m 08s)
>> ------------------------------------------------------------------------------------
>> Counter | TM Name |
>> Value
>> ------------------------------------------------------------------------------------
>> capture.kernel_packets | Total |
>> 22604957367
>> capture.kernel_drops | Total |
>> 40166
>> decoder.pkts | Total |
>> 22605130865
>> decoder.bytes | Total |
>> 22092883178348
>> decoder.invalid | Total |
>> 4199422
>> decoder.ipv4 | Total |
>> 22606546275
>> decoder.ipv6 | Total |
>> 15489763
>> decoder.ethernet | Total |
>> 22605131052
>> decoder.tcp | Total |
>> 18070722539
>> decoder.udp | Total |
>> 4281815803
>> decoder.icmpv4 | Total |
>> 22078766
>> decoder.icmpv6 | Total |
>> 8290
>> decoder.ppp | Total |
>> 147901356
>> decoder.gre | Total |
>> 149364919
>> decoder.teredo | Total |
>> 15471659
>> decoder.avg_pkt_size | Total | 977
>> decoder.max_pkt_size | Total |
>> 16591
>> defrag.ipv4.fragments | Total | 3
>> defrag.ipv6.fragments | Total | 115
>> defrag.ipv6.reassembled | Total | 28
>> decoder.icmpv4.ipv4_trunc_pkt | Total | 6
>> decoder.icmpv4.ipv4_unknown_ver | Total | 451
>> decoder.tcp.hlen_too_small | Total |
>> 6276
>> decoder.tcp.invalid_optlen | Total | 230
>> decoder.tcp.opt_invalid_len | Total | 608
>> decoder.udp.pkt_too_small | Total | 16
>> decoder.udp.hlen_invalid | Total |
>> 4191833
>> decoder.gre.version0_recur | Total | 1
>> decoder.gre.version1_flags | Total | 1
>> tcp.sessions | Total |
>> 252444206
>> tcp.ssn_memcap_drop | Total |
>> 42676
>> tcp.pseudo | Total |
>> 10488865
>> tcp.invalid_checksum | Total |
>> 124112
>> tcp.syn | Total |
>> 301974943
>> tcp.synack | Total |
>> 117698991
>> tcp.rst | Total |
>> 108761603
>> tcp.segment_memcap_drop | Total |
>> 1292430868
>> tcp.stream_depth_reached | Total |
>> 12508
>> tcp.reassembly_gap | Total |
>> 227951862
>> detect.alert | Total |
>> 2333041
>> app_layer.flow.http | Total |
>> 4769888
>> app_layer.tx.http | Total |
>> 11055214
>> app_layer.flow.smtp | Total | 1
>> app_layer.tx.smtp | Total |
>> 577624
>> app_layer.flow.tls | Total |
>> 8893548
>> app_layer.flow.ssh | Total |
>> 518339
>> app_layer.flow.smb | Total |
>> 34653
>> app_layer.flow.dcerpc_tcp | Total | 24
>> app_layer.flow.dns_tcp | Total |
>> 11861
>> app_layer.tx.dns_tcp | Total |
>> 4980
>> app_layer.flow.failed_tcp | Total |
>> 6015530
>> app_layer.flow.dcerpc_udp | Total |
>> 18464
>> app_layer.flow.dns_udp | Total |
>> 55013837
>> app_layer.tx.dns_udp | Total |
>> 18797149
>> app_layer.flow.failed_udp | Total |
>> 46908581
>> flow_mgr.closed_pruned | Total |
>> 77180292
>> flow_mgr.new_pruned | Total |
>> 348444456
>> flow_mgr.est_pruned | Total |
>> 35974852
>> flow.spare | Total |
>> 12132
>> flow.emerg_mode_entered | Total |
>> 2417
>> flow.emerg_mode_over | Total |
>> 2417
>> flow.tcp_reuse | Total |
>> 336896
>> flow_mgr.flows_checked | Total |
>> 101564
>> flow_mgr.flows_notimeout | Total |
>> 99325
>> flow_mgr.flows_timeout | Total |
>> 2239
>> flow_mgr.flows_timeout_inuse | Total | 30
>> flow_mgr.flows_removed | Total |
>> 2209
>> flow_mgr.rows_checked | Total |
>> 65536
>> flow_mgr.rows_skipped | Total |
>> 52198
>> flow_mgr.rows_maxlen | Total | 21
>> tcp.memuse | Total |
>> 40454272
>> tcp.reassembly_memuse | Total |
>> 18446744073614294924
>> dns.memuse | Total |
>> 16901536
>> dns.memcap_global | Total |
>> 78771302
>> http.memuse | Total |
>> 15319405
>> flow.memuse | Total |
>> 129957280
>>
>>
>> Jeffrey Collyer
>> Information Security Engineer
>> University of Virginia
>> 434-297-6317
>>
>> On Feb 1, 2017, at 8:49 AM, Peter Manev <petermanev at gmail.com> wrote:
>>
>> On Wed, Feb 1, 2017 at 2:31 PM, Collyer, Jeffrey W. (jwc3f)
>> <jwc3f at virginia.edu> wrote:
>>
>> So I’ve followed the 10G tuning guide, to what looks like great success.
>> This is on a Intel X520 card with AF_PACKET and 1 RSS queue. The traffic
>> load fluctuates between 4 and a peak of about 8 Gbps.
>>
>>
>> Out of curiosity what specs do you have - for CPU/RAM/OS/kernel?
>>
>>
>> capture.kernel_packets | Total |
>> 22028847471
>> capture.kernel_drops | Total |
>> 40166
>> decoder.pkts | Total
>> | 22028920807
>>
>>
>> Can you please paste the last (full section ) log update?
>>
>>
>> In digging around further, netstat -s show about 3% of packet reassemblies
>> failing. Is this normal? This is my first foray into 10G capture and I
>> don’t know what is normal at what level of diagnostic yet, and was hoping
>> someone with more experience could tell me if this was a problem or not.
>>
>> I expanded the ipfrag_high_thresh kernel memory to try to allow more memory
>> for packet reassmembly in case that was a factor.
>>
>> # expand ip_frag threshod to help packet reassembly
>> net.ipv4.ipfrag_high_thresh = 8388608
>>
>> Ip:
>> 4509211 total packets received
>> 0 forwarded
>> 0 incoming packets discarded
>> 3351761 incoming packets delivered
>> 2369930 requests sent out
>> 121777 fragments dropped after timeout
>> 258550565 reassemblies required
>> 72996695 packets reassembled ok
>> 7823209 packet reassembles failed
>>
>> Thanks for any advice.
>> Jeff
>>
>>
>> Jeffrey Collyer
>> Information Security Engineer
>> University of Virginia
>> 434-297-6317
>>
>>
>> _______________________________________________
>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
>> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>
>>
>>
>>
>> --
>> Regards,
>> Peter Manev
>>
>>
>
>
>
> --
> Regards,
> Peter Manev
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
More information about the Oisf-users
mailing list