<div dir="ltr"><p>Hello,</p><p>I am having some issues with file extraction in Suricata, and after attempting to do many optimizations and review of others experiences I am still finding myself out of luck. Below is some verbose output of my current configuration and some sample data after ~12 hours of running. I have also included a smaller time frame with a few subtle changes for consideration.</p>
<p>Under this configuration the rule I have, which is an IP based rule below...</p><p>alert http any any -> $MY_LOCAL_IP any (msg:"FILE PDF"; filemagic:"PDF document"; filestore; sid:1; rev:1;)</p><p>
Does not trigger at all when the reassembly mem cap is reached. Even when it does (when reassembly memcap is below the threshold), I get truncated PDFs. I have tried adjusting things like the reassembly memcap, however, when I do, I very quickly run into a large amount of packet loss because the number of free slots PF_RING can issue is not able to keep up (details below). Additionally, reassembly mem cap seems to slowly increase over time, eventually reaching its peak before the number of free ring slots can finally keep up (presumably due to segments being dropped).</p>
<p>I have struggled playing with time out values as well really to no avail (details below). </p><p>When I turn http logging on, I do see the website that I go to being properly logged fwiw.</p><p>I feel like I must be doing something wrong, or I am not seeing something obvious. After reviewing many blogs and howtos, it seems folks are able to do what I am trying to accomplish with the same (sometime more) data rates and much less hardware. </p>
<p>I have tried the following:<br>- Increased min_num_slots to 65534 for PF_RING<br>- Tinkered with TCP timeout settings<br>- Adjusted reassembly memcap</p><p>Kindly take a look at the details I have listed below and let me know if there is anything you can suggest. I am curious if I am just plain at the limit of my hardware and need to consider upgrading and/or getting PF_RING with DNA. Or, perhaps there are a few more items I should consider within the application itself. </p>
<p>One final thing to consider, would tcp sequence randomization significantly impact things? I would need to get in touch with the folks responsible to see if we have this on but thought I would ask here as well! </p><p>
Many thanks in advance for your time looking at this!</p><p>== Profile ==</p><p>CentOS 6.5 Linux <br>Kernel 2.6.32-431.11.2.el6.x86_64</p><p>Installed Suricata 2.0 and PF_RING 6.0.1 from source.</p><p>Machine sees ~400MB/s at peek load.</p>
<p>== Tuning ==</p><p>I've tuned the ixgbe NIC with the following settings...</p><p>ethtool -K p4p2 tso off<br>ethtool -K p4p2 gro off<br>ethtool -K p4p2 lro off<br>ethtool -K p4p2 gso off<br>ethtool -K p4p2 rx off<br>
ethtool -K p4p2 tx off<br>ethtool -K p4p2 sg off<br>ethtool -K p4p2 rxvlan off<br>ethtool -K p4p2 txvlan off<br>ethtool -N p4p2 rx-flow-hash udp4 sdfn<br>ethtool -N p4p2 rx-flow-hash udp6 sdfn<br>ethtool -n p4p2 rx-flow-hash udp6 <br>
ethtool -n p4p2 rx-flow-hash udp4<br>ethtool -C p4p2 rx-usecs 1000<br>ethtool -C p4p2 adaptive-rx off</p><p>It is also using the latest driver available. I have also tried to optimize things in the sysctl.conf</p><p># -- 10gbe tuning from Intel ixgb driver README -- #</p>
<p># turn off selective ACK and timestamps<br>net.ipv4.tcp_sack = 0<br>net.ipv4.tcp_timestamps = 0</p><p># memory allocation min/pressure/max.<br># read buffer, write buffer, and buffer space<br>net.ipv4.tcp_rmem = 10000000 10000000 10000000<br>
net.ipv4.tcp_wmem = 10000000 10000000 10000000<br>net.ipv4.tcp_mem = 10000000 10000000 10000000</p><p>net.core.rmem_max = 524287<br>net.core.wmem_max = 524287<br>net.core.rmem_default = 524287<br>net.core.wmem_default = 524287<br>
net.core.optmem_max = 524287<br>net.core.netdev_max_backlog = 300000</p><p>== Hardware Specs ==<br>CPU: Intel Xeon CPU @ 2.40Ghz x 32<br>RAM: 48G<br>NIC: <br> *-network:1<br> description: Ethernet interface<br> product: Ethernet 10G 2P X520 Adapter<br>
vendor: Intel Corporation<br> physical id: 0.1<br> bus info: pci@0000:42:00.1<br> logical name: p4p2<br> version: 01<br> serial: a0:36:9f:07:ec:02<br> capacity: 1GB/s<br> width: 64 bits<br>
clock: 33MHz<br> capabilities: pm msi msix pciexpress vpd bus_master cap_list rom ethernet physical fibre 1000bt-fd autonegotiation<br> configuration: autonegotiation=on broadcast=yes driver=ixgbe driverversion=3.21.2 duplex=full firmware=0x8000030d latency=0 link=yes multicast=yes port=fibre promiscuous=yes<br>
resources: irq:76 memory:d0f00000-d0f7ffff ioport:7ce0(size=32) memory:d0ffc000-d0ffffff memory:d1100000-d117ffff(prefetchable) memory:d1380000-d147ffff(prefetchable) memory:d1480000-d157ffff(prefetchable)</p><p>== Suricata Config ==<br>
Below are some details that may be relevant...</p><p>runmode: workers</p><p>host-mode: sniffer-only</p><p>default-packet-size: 9000</p><p>- file-store:<br> enabled: yes # set to yes to enable<br> log-dir: files # directory to store the files<br>
force-magic: yes # force logging magic on all stored files<br> force-md5: yes # force logging of md5 checksums<br> waldo: file.waldo # waldo file to store the file_id across runs</p><p>defrag:<br> memcap: 512mb <br>
hash-size: 65536<br> trackers: 65535 # number of defragmented flows to follow<br> max-frags: 65535 # number of fragments to keep (higher than trackers)<br> prealloc: yes<br> timeout: 30 </p><p>flow:<br> memcap: 1gb <br>
hash-size: 1048576 <br> prealloc: 1048576 <br> emergency-recovery: 30</p><p>flow-timeouts:<br> default:<br> new: 1<br> established: 5<br> closed: 0<br> emergency-new: 1<br> emergency-established: 1<br>
emergency-closed: 0<br> tcp:<br> new: 15<br> established: 100<br> closed: 5<br> emergency-new: 1<br> emergency-established: 1<br> emergency-closed: 0<br> udp:<br> new: 5<br> established: 10<br>
emergency-new: 1<br> emergency-established: 1<br> icmp:<br> new: 1<br> established: 5<br> emergency-new: 1<br> emergency-established: 1</p><p>stream:<br> memcap: 10gb <br> checksum-validation: no # reject wrong csums<br>
prealloc-sesions: 500000 <br> midstream: false <br> asyn-oneside: false <br> inline: no # auto will use inline mode in IPS mode, yes or no set it statically<br> reassembly:<br>
memcap: 1.5gb <br> depth: 5mb <br> toserver-chunk-size: 2560<br> toclient-chunk-size: 2560<br> randomize-chunk-size: yes</p><p>host:<br> hash-size: 4096<br> prealloc: 1000<br>
memcap: 16777216</p><p><br>pfring:<br> - interface: p4p2<br> threads: 16<br> cluster-id: 99<br> cluster-type: cluster_flow<br> checksum-checks: no<br> - interface: default</p><p>http:<br> enabled: yes<br> libhtp:<br>
default-config:<br> personality: IDS</p><p> # Can be specified in kb, mb, gb. Just a number indicates<br> # it's in bytes.<br> request-body-limit: 12mb<br> response-body-limit: 12mb<br>
<br>== ~12 hours (above config) =</p><p>top - 14:58:59 up 18:23, 3 users, load average: 6.44, 4.83, 4.32<br>Tasks: 664 total, 1 running, 663 sleeping, 0 stopped, 0 zombie<br>Cpu(s): 17.9%us, 0.1%sy, 0.0%ni, 80.3%id, 0.0%wa, 0.0%hi, 1.7%si, 0.0%st<br>
Mem: 49376004k total, 29289768k used, 20086236k free, 68340k buffers<br>Swap: 2621432k total, 0k used, 2621432k free, 820172k cached</p><p> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br>
17616 root 20 0 27.0g 26g 16g S 621.4 55.8 3532:51 Suricata-Main</p><p>watch 'cat /proc/net/pf_ring/*p4p2* | egrep "Num Free Slots|Tot Packets|Tot Pkt Lost"'<br>; First three threads...<br>Tot Packets : 627370957<br>
Tot Pkt Lost : 3582014<br>Num Free Slots : 118705<br>Tot Packets : 676753767<br>Tot Pkt Lost : 5292092<br>Num Free Slots : 118745<br>Tot Packets : 665348839<br>Tot Pkt Lost : 3841911<br>
Num Free Slots : 118677<br>...</p><p>watch -n 10 'cat stats.log | egrep "reassembly_memuse|segment_memcap_drop" | tail -n 32'<br>; First three threads...<br>tcp.segment_memcap_drop | RxPFRp4p21 | 25782329<br>
tcp.reassembly_memuse | RxPFRp4p21 | 1610612705<br>tcp.segment_memcap_drop | RxPFRp4p22 | 26161478<br>tcp.reassembly_memuse | RxPFRp4p22 | 1610612705<br>tcp.segment_memcap_drop | RxPFRp4p23 | 25813867<br>
tcp.reassembly_memuse | RxPFRp4p23 | 1610612705</p><p>grep 'reassembly_gap' stats.log | tail -n 10<br>tcp.reassembly_gap | RxPFRp4p27 | 777366<br>tcp.reassembly_gap | RxPFRp4p28 | 774896<br>
tcp.reassembly_gap | RxPFRp4p29 | 781761<br>tcp.reassembly_gap | RxPFRp4p210 | 776427<br>tcp.reassembly_gap | RxPFRp4p211 | 778734<br>tcp.reassembly_gap | RxPFRp4p212 | 773203<br>
tcp.reassembly_gap | RxPFRp4p213 | 781125<br>tcp.reassembly_gap | RxPFRp4p214 | 776043<br>tcp.reassembly_gap | RxPFRp4p215 | 781790<br>tcp.reassembly_gap | RxPFRp4p216 | 783368</p>
<p>== PF RING ==</p><p>PF_RING Version : 6.0.1 ($Revision: exported$)<br>Total rings : 16</p><p>Standard (non DNA) Options<br>Ring slots : 65534<br>Slot version : 15<br>Capture TX : No [RX only]<br>
IP Defragment : No<br>Socket Mode : Standard<br>Transparent mode : Yes [mode 0]<br>Total plugins : 0<br>Cluster Fragment Queue : 9175<br>Cluster Fragment Discard : 597999</p><p>
== ~30 min (with changes) ==</p><p>FWIW, when I increase reassembly memcap and time outs to the following...</p><p>flow-timeouts:<br> default:<br> new: 5<br> established: 50<br> closed: 0<br> emergency-new: 1<br>
emergency-established: 1<br> emergency-closed: 0<br> tcp:<br> new: 15<br> established: 100<br> closed: 10<br> emergency-new: 1<br> emergency-established: 1<br> emergency-closed: 0<br> udp:<br> new: 5<br>
established: 50<br> emergency-new: 1<br> emergency-established: 1<br> icmp:<br> new: 1<br> established: 5<br> emergency-new: 1<br> emergency-established: 1</p><p>reassembly:<br> memcap: 3gb <br>
depth: 5mb<br> <br>These are the results, note how there are no more free slots for PF_RING. I believe this results in increased packet loss... which is likely resulting in my truncated files that I receive when I pull a PDF.</p>
<p>watch 'cat /proc/net/pf_ring/*p4p2* | egrep "Num Free Slots|Tot Packets|Tot Pkt Lost"'<br>; First three threads...<br>Tot Packets : 80281541<br>Tot Pkt Lost : 44290194<br>Num Free Slots : 0<br>
Tot Packets : 81926241<br>Tot Pkt Lost : 17412402<br>Num Free Slots : 0<br>Tot Packets : 80108557<br>Tot Pkt Lost : 14667061<br>Num Free Slots : 0</p><p>watch -n 10 'cat stats.log | egrep "reassembly_memuse|segment_memcap_drop" | tail -n 32'<br>
; First three threads...<br>tcp.segment_memcap_drop | RxPFRp4p21 | 0<br>tcp.reassembly_memuse | RxPFRp4p21 | 1681598708<br>tcp.segment_memcap_drop | RxPFRp4p22 | 0<br>tcp.reassembly_memuse | RxPFRp4p22 | 1681626644<br>
tcp.segment_memcap_drop | RxPFRp4p23 | 0<br>tcp.reassembly_memuse | RxPFRp4p23 | 1681597556<br>tcp.segment_memcap_drop | RxPFRp4p24 | 0<br>*** Important to note here, the reassembly memuse seems to steadily increase overtime. After a few minutes of putting this in it has risen to 2022140776 across. Makes me think things are not offloading quickly... (timeout/depth issue?)</p>
<p>grep 'reassembly_gap' stats.log | tail -n 10<br>tcp.reassembly_gap | RxPFRp4p27 | 27603<br>tcp.reassembly_gap | RxPFRp4p28 | 26677<br>tcp.reassembly_gap | RxPFRp4p29 | 26869<br>
tcp.reassembly_gap | RxPFRp4p210 | 25031<br>tcp.reassembly_gap | RxPFRp4p211 | 23988<br>tcp.reassembly_gap | RxPFRp4p212 | 23809<br>tcp.reassembly_gap | RxPFRp4p213 | 26420<br>
tcp.reassembly_gap | RxPFRp4p214 | 25271<br>tcp.reassembly_gap | RxPFRp4p215 | 26285<br>tcp.reassembly_gap | RxPFRp4p216 | 26848<br></p></div>