[Oisf-users] Suricata at 10G, packet reassembly
Peter Manev
petermanev at gmail.com
Wed Feb 1 21:14:30 UTC 2017
On Wed, Feb 1, 2017 at 3:04 PM, Collyer, Jeffrey W. (jwc3f)
<jwc3f at virginia.edu> wrote:
> Sure,
>
> Some specs
>
> Machine
> Dell R430
> 2x14 core Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
> 35M L3 cache
> 128G Ram
> Intel X520 ethernet card
>
> OS - Ubuntu 16.04.1 LTS
> Kernel - 4.4.0-59-generic
> Intel(R) 10GbE PCI Express Linux Network Driver - version 4.4.6
>
> Suricata seems to be using about 40.5G resident memory.
>
> Last stat update -
>
> The tcp.reassembly_memuse appears to be enormous ?16Pb?. Doesn’t seem
> right.
>
Yes it does not look right....
Can you please share your stream and reassembly section from your
suricata.yaml as well ?
( https://redmine.openinfosecfoundation.org/projects/suricata/repository/revisions/master/entry/suricata.yaml.in#L1197
)
>
> ------------------------------------------------------------------------------------
> Date: 2/1/2017 -- 08:50:29 (uptime: 0d, 23h 31m 08s)
> ------------------------------------------------------------------------------------
> Counter | TM Name |
> Value
> ------------------------------------------------------------------------------------
> capture.kernel_packets | Total |
> 22604957367
> capture.kernel_drops | Total |
> 40166
> decoder.pkts | Total |
> 22605130865
> decoder.bytes | Total |
> 22092883178348
> decoder.invalid | Total |
> 4199422
> decoder.ipv4 | Total |
> 22606546275
> decoder.ipv6 | Total |
> 15489763
> decoder.ethernet | Total |
> 22605131052
> decoder.tcp | Total |
> 18070722539
> decoder.udp | Total |
> 4281815803
> decoder.icmpv4 | Total |
> 22078766
> decoder.icmpv6 | Total |
> 8290
> decoder.ppp | Total |
> 147901356
> decoder.gre | Total |
> 149364919
> decoder.teredo | Total |
> 15471659
> decoder.avg_pkt_size | Total | 977
> decoder.max_pkt_size | Total |
> 16591
> defrag.ipv4.fragments | Total | 3
> defrag.ipv6.fragments | Total | 115
> defrag.ipv6.reassembled | Total | 28
> decoder.icmpv4.ipv4_trunc_pkt | Total | 6
> decoder.icmpv4.ipv4_unknown_ver | Total | 451
> decoder.tcp.hlen_too_small | Total |
> 6276
> decoder.tcp.invalid_optlen | Total | 230
> decoder.tcp.opt_invalid_len | Total | 608
> decoder.udp.pkt_too_small | Total | 16
> decoder.udp.hlen_invalid | Total |
> 4191833
> decoder.gre.version0_recur | Total | 1
> decoder.gre.version1_flags | Total | 1
> tcp.sessions | Total |
> 252444206
> tcp.ssn_memcap_drop | Total |
> 42676
> tcp.pseudo | Total |
> 10488865
> tcp.invalid_checksum | Total |
> 124112
> tcp.syn | Total |
> 301974943
> tcp.synack | Total |
> 117698991
> tcp.rst | Total |
> 108761603
> tcp.segment_memcap_drop | Total |
> 1292430868
> tcp.stream_depth_reached | Total |
> 12508
> tcp.reassembly_gap | Total |
> 227951862
> detect.alert | Total |
> 2333041
> app_layer.flow.http | Total |
> 4769888
> app_layer.tx.http | Total |
> 11055214
> app_layer.flow.smtp | Total | 1
> app_layer.tx.smtp | Total |
> 577624
> app_layer.flow.tls | Total |
> 8893548
> app_layer.flow.ssh | Total |
> 518339
> app_layer.flow.smb | Total |
> 34653
> app_layer.flow.dcerpc_tcp | Total | 24
> app_layer.flow.dns_tcp | Total |
> 11861
> app_layer.tx.dns_tcp | Total |
> 4980
> app_layer.flow.failed_tcp | Total |
> 6015530
> app_layer.flow.dcerpc_udp | Total |
> 18464
> app_layer.flow.dns_udp | Total |
> 55013837
> app_layer.tx.dns_udp | Total |
> 18797149
> app_layer.flow.failed_udp | Total |
> 46908581
> flow_mgr.closed_pruned | Total |
> 77180292
> flow_mgr.new_pruned | Total |
> 348444456
> flow_mgr.est_pruned | Total |
> 35974852
> flow.spare | Total |
> 12132
> flow.emerg_mode_entered | Total |
> 2417
> flow.emerg_mode_over | Total |
> 2417
> flow.tcp_reuse | Total |
> 336896
> flow_mgr.flows_checked | Total |
> 101564
> flow_mgr.flows_notimeout | Total |
> 99325
> flow_mgr.flows_timeout | Total |
> 2239
> flow_mgr.flows_timeout_inuse | Total | 30
> flow_mgr.flows_removed | Total |
> 2209
> flow_mgr.rows_checked | Total |
> 65536
> flow_mgr.rows_skipped | Total |
> 52198
> flow_mgr.rows_maxlen | Total | 21
> tcp.memuse | Total |
> 40454272
> tcp.reassembly_memuse | Total |
> 18446744073614294924
> dns.memuse | Total |
> 16901536
> dns.memcap_global | Total |
> 78771302
> http.memuse | Total |
> 15319405
> flow.memuse | Total |
> 129957280
>
>
> Jeffrey Collyer
> Information Security Engineer
> University of Virginia
> 434-297-6317
>
> On Feb 1, 2017, at 8:49 AM, Peter Manev <petermanev at gmail.com> wrote:
>
> On Wed, Feb 1, 2017 at 2:31 PM, Collyer, Jeffrey W. (jwc3f)
> <jwc3f at virginia.edu> wrote:
>
> So I’ve followed the 10G tuning guide, to what looks like great success.
> This is on a Intel X520 card with AF_PACKET and 1 RSS queue. The traffic
> load fluctuates between 4 and a peak of about 8 Gbps.
>
>
> Out of curiosity what specs do you have - for CPU/RAM/OS/kernel?
>
>
> capture.kernel_packets | Total |
> 22028847471
> capture.kernel_drops | Total |
> 40166
> decoder.pkts | Total
> | 22028920807
>
>
> Can you please paste the last (full section ) log update?
>
>
> In digging around further, netstat -s show about 3% of packet reassemblies
> failing. Is this normal? This is my first foray into 10G capture and I
> don’t know what is normal at what level of diagnostic yet, and was hoping
> someone with more experience could tell me if this was a problem or not.
>
> I expanded the ipfrag_high_thresh kernel memory to try to allow more memory
> for packet reassmembly in case that was a factor.
>
> # expand ip_frag threshod to help packet reassembly
> net.ipv4.ipfrag_high_thresh = 8388608
>
> Ip:
> 4509211 total packets received
> 0 forwarded
> 0 incoming packets discarded
> 3351761 incoming packets delivered
> 2369930 requests sent out
> 121777 fragments dropped after timeout
> 258550565 reassemblies required
> 72996695 packets reassembled ok
> 7823209 packet reassembles failed
>
> Thanks for any advice.
> Jeff
>
>
> Jeffrey Collyer
> Information Security Engineer
> University of Virginia
> 434-297-6317
>
>
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>
>
>
>
> --
> Regards,
> Peter Manev
>
>
--
Regards,
Peter Manev
More information about the Oisf-users
mailing list