<div dir="ltr">Thanks Peter. <div><br></div><div>I will take a peek at these parameters and report back the outcome. I only have that one rule I posted in my initial email running however, that just looks for me downloading a PDF.</div>
<div><br></div><div>-Jason</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, May 31, 2014 at 4:39 AM, Peter Manev <span dir="ltr"><<a href="mailto:petermanev@gmail.com" target="_blank">petermanev@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On Fri, May 30, 2014 at 9:53 PM, Jason Batchelor <<a href="mailto:jxbatchelor@gmail.com">jxbatchelor@gmail.com</a>> wrote:<br>
> Thanks Victor/Cooper:<br>
><br>
>> If you have 48G of mem, I think you can use a lot more here. Like 16G or<br>
> something.<br>
><br>
> I used a large number like this initially, however, things eventually got<br>
> chaotic and the application ran out of memory. This lead me to believe that<br>
> it was allowing the cap on a per PF_RING thread basis, rather than as an<br>
> aggregate (though I could be totally wrong :) )<br>
><br>
> Important to note that with 1.5gb cap I was at 55.8% mem utilization earlier<br>
> over a ~12 hr period.<br>
><br>
> > As stream depth is 5mb, setting 12mb here doesn't really affect anything<br>
> I think. The stream depth cuts stream tracking at 5mb regardless of the<br>
> setting here.<br>
><br>
> Noted. For some reason I thought it treated reassembly for that traffic<br>
> differently if that setting was invoked.<br>
><br>
>> Also, if you have multiple vlans on the network, it may be worth trying<br>
> to disable.<br>
><br>
> We do, and I did do an earlier test run setting this just to false. To no<br>
> avail however, as the issues continued.<br>
><br>
>> In 2.0.1 the stream engine should use less memory and clear memory<br>
> quicker. Could you try 2.0.1?<br>
><br>
> Just installed and am working with this now. Unfortunately,<br>
> tcp.reassembly_memuse seems to continue to slowly increase. Below is some<br>
> data taken from a recent test over the course of 35 minutes with the same<br>
> data rates (only with 2.0.1). Reassem memcap is set to 3gb with this run.<br>
> Stats.log output is every 10 seconds so I picked out every 3rd record on a<br>
> random thread.<br>
><br>
> grep RxPFRp4p210 stats.log | grep reassembly_memuse | awk '!(NR%3)'<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 154471336<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 243540732<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 301571602<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 314913802<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 285059970<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 272284065<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 254263605<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 253255657<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 272511595<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 296499090<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 311359535<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 283427074<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 255654055<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 270931311<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 299100487<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 291949643<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 293440732<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 290335664<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 269864084<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 267911989<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 276945873<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 283443241<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 345384994<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 347958227<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 330984680<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 305309240<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 274026280<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 328726738<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 369907025<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 352353094<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 333345579<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 306374595<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 277904734<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 298247168<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 301306632<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 266160428<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 307500851<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 307852499<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 290149127<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 306533111<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 291460952<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 288518673<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 324852777<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 302582777<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 293625604<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 307370770<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 311457210<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 311108835<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 320206099<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 307685170<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 282411341<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 288700701<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 306848861<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 281831829<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 292346555<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 314350370<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 286442727<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 283934879<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 270730695<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 269700704<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 327235776<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 326038648<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 359062072<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 359215420<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 371124232<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 379226360<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 411893212<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 429465136<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 452973020<br>
> tcp.reassembly_memuse | RxPFRp4p210 | 464045120<br>
><br>
> Since reassembly memcap is not reached (yet) and pf_ring can't keep up, our<br>
> number of free slots for pf_ring threads is continuously at 0, which has<br>
> amplifying effects concerning packet loss.<br>
><br>
> grep capture.kernel_ stats.log | tail -n 10<br>
> capture.kernel_packets | RxPFRp4p212 | 57951223<br>
> capture.kernel_drops | RxPFRp4p212 | 5797512<br>
> capture.kernel_packets | RxPFRp4p213 | 70292808<br>
> capture.kernel_drops | RxPFRp4p213 | 7525246<br>
> capture.kernel_packets | RxPFRp4p214 | 55188744<br>
> capture.kernel_drops | RxPFRp4p214 | 3877128<br>
> capture.kernel_packets | RxPFRp4p215 | 57682274<br>
> capture.kernel_drops | RxPFRp4p215 | 5057610<br>
> capture.kernel_packets | RxPFRp4p216 | 55113863<br>
> capture.kernel_drops | RxPFRp4p216 | 4016017<br>
><br>
>> Are you stuck with PF_RING for some reason?<br>
><br>
> I am for the reasons Victor cited :)<br>
><br>
> Thanks all for the input so far. I am really hoping to get this working! If<br>
> there is any more information I can give to help please let me know.<br>
><br>
> Thanks,<br>
> Jason<br>
><br>
><br>
> On Fri, May 30, 2014 at 12:17 PM, Victor Julien <<a href="mailto:lists@inliniac.net">lists@inliniac.net</a>> wrote:<br>
>><br>
>> On 05/30/2014 06:31 PM, Jason Batchelor wrote:<br>
>> > Hello,<br>
>> ><br>
>> > I am having some issues with file extraction in Suricata, and after<br>
>> > attempting to do many optimizations and review of others experiences I<br>
>> > am still finding myself out of luck. Below is some verbose output of my<br>
>> > current configuration and some sample data after ~12 hours of running. I<br>
>> > have also included a smaller time frame with a few subtle changes for<br>
>> > consideration.<br>
>> ><br>
>> > Under this configuration the rule I have, which is an IP based rule<br>
>> > below...<br>
>> ><br>
>> > alert http any any -> $MY_LOCAL_IP any (msg:"FILE PDF"; filemagic:"PDF<br>
>> > document"; filestore; sid:1; rev:1;)<br>
>> ><br>
>> > Does not trigger at all when the reassembly mem cap is reached. Even<br>
>> > when it does (when reassembly memcap is below the threshold), I get<br>
>> > truncated PDFs. I have tried adjusting things like the reassembly<br>
>> > memcap, however, when I do, I very quickly run into a large amount of<br>
>> > packet loss because the number of free slots PF_RING can issue is not<br>
>> > able to keep up (details below). Additionally, reassembly mem cap seems<br>
>> > to slowly increase over time, eventually reaching its peak before the<br>
>> > number of free ring slots can finally keep up (presumably due to<br>
>> > segments being dropped).<br>
>> ><br>
>> > I have struggled playing with time out values as well really to no avail<br>
>> > (details below).<br>
>> ><br>
>> > When I turn http logging on, I do see the website that I go to being<br>
>> > properly logged fwiw.<br>
>> ><br>
>> > I feel like I must be doing something wrong, or I am not seeing<br>
>> > something obvious. After reviewing many blogs and howtos, it seems folks<br>
>> > are able to do what I am trying to accomplish with the same (sometime<br>
>> > more) data rates and much less hardware.<br>
>> ><br>
>> > I have tried the following:<br>
>> > - Increased min_num_slots to 65534 for PF_RING<br>
>> > - Tinkered with TCP timeout settings<br>
>> > - Adjusted reassembly memcap<br>
>> ><br>
>> > Kindly take a look at the details I have listed below and let me know if<br>
>> > there is anything you can suggest. I am curious if I am just plain at<br>
>> > the limit of my hardware and need to consider upgrading and/or getting<br>
>> > PF_RING with DNA. Or, perhaps there are a few more items I should<br>
>> > consider within the application itself.<br>
>> ><br>
>> > One final thing to consider, would tcp sequence randomization<br>
>> > significantly impact things? I would need to get in touch with the folks<br>
>> > responsible to see if we have this on but thought I would ask here as<br>
>> > well!<br>
>> ><br>
>> > Many thanks in advance for your time looking at this!<br>
>> ><br>
>> > == Profile ==<br>
>> ><br>
>> > CentOS 6.5 Linux<br>
>> > Kernel 2.6.32-431.11.2.el6.x86_64<br>
>> ><br>
>> > Installed Suricata 2.0 and PF_RING 6.0.1 from source.<br>
>> ><br>
>> > Machine sees ~400MB/s at peek load.<br>
>> ><br>
>> > == Tuning ==<br>
>> ><br>
>> > I've tuned the ixgbe NIC with the following settings...<br>
>> ><br>
>> > ethtool -K p4p2 tso off<br>
>> > ethtool -K p4p2 gro off<br>
>> > ethtool -K p4p2 lro off<br>
>> > ethtool -K p4p2 gso off<br>
>> > ethtool -K p4p2 rx off<br>
>> > ethtool -K p4p2 tx off<br>
>> > ethtool -K p4p2 sg off<br>
>> > ethtool -K p4p2 rxvlan off<br>
>> > ethtool -K p4p2 txvlan off<br>
>> > ethtool -N p4p2 rx-flow-hash udp4 sdfn<br>
>> > ethtool -N p4p2 rx-flow-hash udp6 sdfn<br>
>> > ethtool -n p4p2 rx-flow-hash udp6<br>
>> > ethtool -n p4p2 rx-flow-hash udp4<br>
>> > ethtool -C p4p2 rx-usecs 1000<br>
>> > ethtool -C p4p2 adaptive-rx off<br>
>> ><br>
>> > It is also using the latest driver available. I have also tried to<br>
>> > optimize things in the sysctl.conf<br>
>> ><br>
>> > # -- 10gbe tuning from Intel ixgb driver README -- #<br>
>> ><br>
>> > # turn off selective ACK and timestamps<br>
>> > net.ipv4.tcp_sack = 0<br>
>> > net.ipv4.tcp_timestamps = 0<br>
>> ><br>
>> > # memory allocation min/pressure/max.<br>
>> > # read buffer, write buffer, and buffer space<br>
>> > net.ipv4.tcp_rmem = 10000000 10000000 10000000<br>
>> > net.ipv4.tcp_wmem = 10000000 10000000 10000000<br>
>> > net.ipv4.tcp_mem = 10000000 10000000 10000000<br>
>> ><br>
>> > net.core.rmem_max = 524287<br>
>> > net.core.wmem_max = 524287<br>
>> > net.core.rmem_default = 524287<br>
>> > net.core.wmem_default = 524287<br>
>> > net.core.optmem_max = 524287<br>
>> > net.core.netdev_max_backlog = 300000<br>
>> ><br>
>> > == Hardware Specs ==<br>
>> > CPU: Intel Xeon CPU @ 2.40Ghz x 32<br>
>> > RAM: 48G<br>
>> > NIC:<br>
>> > *-network:1<br>
>> > description: Ethernet interface<br>
>> > product: Ethernet 10G 2P X520 Adapter<br>
>> > vendor: Intel Corporation<br>
>> > physical id: 0.1<br>
>> > bus info: pci@0000:42:00.1<br>
>> > logical name: p4p2<br>
>> > version: 01<br>
>> > serial: a0:36:9f:07:ec:02<br>
>> > capacity: 1GB/s<br>
>> > width: 64 bits<br>
>> > clock: 33MHz<br>
>> > capabilities: pm msi msix pciexpress vpd bus_master cap_list rom<br>
>> > ethernet physical fibre 1000bt-fd autonegotiation<br>
>> > configuration: autonegotiation=on broadcast=yes driver=ixgbe<br>
>> > driverversion=3.21.2 duplex=full firmware=0x8000030d latency=0 link=yes<br>
>> > multicast=yes port=fibre promiscuous=yes<br>
>> > resources: irq:76 memory:d0f00000-d0f7ffff ioport:7ce0(size=32)<br>
>> > memory:d0ffc000-d0ffffff memory:d1100000-d117ffff(prefetchable)<br>
>> > memory:d1380000-d147ffff(prefetchable)<br>
>> > memory:d1480000-d157ffff(prefetchable)<br>
>> ><br>
>> > == Suricata Config ==<br>
>> > Below are some details that may be relevant...<br>
>> ><br>
>> > runmode: workers<br>
>> ><br>
>> > host-mode: sniffer-only<br>
>> ><br>
>> > default-packet-size: 9000<br>
>> ><br>
>> > - file-store:<br>
>> > enabled: yes # set to yes to enable<br>
>> > log-dir: files # directory to store the files<br>
>> > force-magic: yes # force logging magic on all stored files<br>
>> > force-md5: yes # force logging of md5 checksums<br>
>> > waldo: file.waldo # waldo file to store the file_id across runs<br>
>> ><br>
>> > defrag:<br>
>> > memcap: 512mb<br>
>> > hash-size: 65536<br>
>> > trackers: 65535 # number of defragmented flows to follow<br>
>> > max-frags: 65535 # number of fragments to keep (higher than trackers)<br>
>> > prealloc: yes<br>
>> > timeout: 30<br>
>> ><br>
>> > flow:<br>
>> > memcap: 1gb<br>
>> > hash-size: 1048576<br>
>> > prealloc: 1048576<br>
>> > emergency-recovery: 30<br>
>> ><br>
>> > flow-timeouts:<br>
>> > default:<br>
>> > new: 1<br>
>> > established: 5<br>
>> > closed: 0<br>
>> > emergency-new: 1<br>
>> > emergency-established: 1<br>
>> > emergency-closed: 0<br>
>> > tcp:<br>
>> > new: 15<br>
>> > established: 100<br>
>> > closed: 5<br>
>> > emergency-new: 1<br>
>> > emergency-established: 1<br>
>> > emergency-closed: 0<br>
>> > udp:<br>
>> > new: 5<br>
>> > established: 10<br>
>> > emergency-new: 1<br>
>> > emergency-established: 1<br>
>> > icmp:<br>
>> > new: 1<br>
>> > established: 5<br>
>> > emergency-new: 1<br>
>> > emergency-established: 1<br>
>> ><br>
>> > stream:<br>
>> > memcap: 10gb<br>
>><br>
>> This is excessive, although it won't hurt.<br>
>><br>
>> > checksum-validation: no # reject wrong csums<br>
>> > prealloc-sesions: 500000<br>
>> > midstream: false<br>
>> > asyn-oneside: false<br>
>> > inline: no # auto will use inline mode in IPS<br>
>> > mode, yes or no set it statically<br>
>> > reassembly:<br>
>> > memcap: 1.5gb<br>
>><br>
>> If you have 48G of mem, I think you can use a lot more here. Like 16G or<br>
>> something.<br>
>><br>
>> > depth: 5mb<br>
>> > toserver-chunk-size: 2560<br>
>> > toclient-chunk-size: 2560<br>
>> > randomize-chunk-size: yes<br>
>> ><br>
>> > host:<br>
>> > hash-size: 4096<br>
>> > prealloc: 1000<br>
>> > memcap: 16777216<br>
>> ><br>
>> ><br>
>> > pfring:<br>
>> > - interface: p4p2<br>
>> > threads: 16<br>
>> > cluster-id: 99<br>
>> > cluster-type: cluster_flow<br>
>> > checksum-checks: no<br>
>> > - interface: default<br>
>> ><br>
>> > http:<br>
>> > enabled: yes<br>
>> > libhtp:<br>
>> > default-config:<br>
>> > personality: IDS<br>
>> ><br>
>> > # Can be specified in kb, mb, gb. Just a number indicates<br>
>> > # it's in bytes.<br>
>> > request-body-limit: 12mb<br>
>> > response-body-limit: 12mb<br>
>><br>
>> As stream depth is 5mb, setting 12mb here doesn't really affect anything<br>
>> I think. The stream depth cuts stream tracking at 5mb regardless of the<br>
>> setting here.<br>
>><br>
>> ><br>
>> > == ~12 hours (above config) =<br>
>> ><br>
>> > top - 14:58:59 up 18:23, 3 users, load average: 6.44, 4.83, 4.32<br>
>> > Tasks: 664 total, 1 running, 663 sleeping, 0 stopped, 0 zombie<br>
>> > Cpu(s): 17.9%us, 0.1%sy, 0.0%ni, 80.3%id, 0.0%wa, 0.0%hi, 1.7%si,<br>
>> > 0.0%st<br>
>> > Mem: 49376004k total, 29289768k used, 20086236k free, 68340k buffers<br>
>> > Swap: 2621432k total, 0k used, 2621432k free, 820172k cached<br>
>> ><br>
>> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br>
>> > 17616 root 20 0 27.0g 26g 16g S 621.4 55.8 3532:51<br>
>> > Suricata-Main<br>
>> ><br>
>> > watch 'cat /proc/net/pf_ring/*p4p2* | egrep "Num Free Slots|Tot<br>
>> > Packets|Tot Pkt Lost"'<br>
>> > ; First three threads...<br>
>> > Tot Packets : 627370957<br>
>> > Tot Pkt Lost : 3582014<br>
>> > Num Free Slots : 118705<br>
>> > Tot Packets : 676753767<br>
>> > Tot Pkt Lost : 5292092<br>
>> > Num Free Slots : 118745<br>
>> > Tot Packets : 665348839<br>
>> > Tot Pkt Lost : 3841911<br>
>> > Num Free Slots : 118677<br>
>> > ...<br>
>> ><br>
>> > watch -n 10 'cat stats.log | egrep<br>
>> > "reassembly_memuse|segment_memcap_drop" | tail -n 32'<br>
>> > ; First three threads...<br>
>> > tcp.segment_memcap_drop | RxPFRp4p21 | 25782329<br>
>> > tcp.reassembly_memuse | RxPFRp4p21 | 1610612705<br>
>> > tcp.segment_memcap_drop | RxPFRp4p22 | 26161478<br>
>> > tcp.reassembly_memuse | RxPFRp4p22 | 1610612705<br>
>> > tcp.segment_memcap_drop | RxPFRp4p23 | 25813867<br>
>> > tcp.reassembly_memuse | RxPFRp4p23 | 1610612705<br>
>> ><br>
>> > grep 'reassembly_gap' stats.log | tail -n 10<br>
>> > tcp.reassembly_gap | RxPFRp4p27 | 777366<br>
>> > tcp.reassembly_gap | RxPFRp4p28 | 774896<br>
>> > tcp.reassembly_gap | RxPFRp4p29 | 781761<br>
>> > tcp.reassembly_gap | RxPFRp4p210 | 776427<br>
>> > tcp.reassembly_gap | RxPFRp4p211 | 778734<br>
>> > tcp.reassembly_gap | RxPFRp4p212 | 773203<br>
>> > tcp.reassembly_gap | RxPFRp4p213 | 781125<br>
>> > tcp.reassembly_gap | RxPFRp4p214 | 776043<br>
>> > tcp.reassembly_gap | RxPFRp4p215 | 781790<br>
>> > tcp.reassembly_gap | RxPFRp4p216 | 783368<br>
>> ><br>
>> > == PF RING ==<br>
>> ><br>
>> > PF_RING Version : 6.0.1 ($Revision: exported$)<br>
>> > Total rings : 16<br>
>> ><br>
>> > Standard (non DNA) Options<br>
>> > Ring slots : 65534<br>
>> > Slot version : 15<br>
>> > Capture TX : No [RX only]<br>
>> > IP Defragment : No<br>
>> > Socket Mode : Standard<br>
>> > Transparent mode : Yes [mode 0]<br>
>> > Total plugins : 0<br>
>> > Cluster Fragment Queue : 9175<br>
>> > Cluster Fragment Discard : 597999<br>
>> ><br>
>> > == ~30 min (with changes) ==<br>
>> ><br>
>> > FWIW, when I increase reassembly memcap and time outs to the<br>
>> > following...<br>
>> ><br>
>> > flow-timeouts:<br>
>> > default:<br>
>> > new: 5<br>
>> > established: 50<br>
>> > closed: 0<br>
>> > emergency-new: 1<br>
>> > emergency-established: 1<br>
>> > emergency-closed: 0<br>
>> > tcp:<br>
>> > new: 15<br>
>> > established: 100<br>
>> > closed: 10<br>
>> > emergency-new: 1<br>
>> > emergency-established: 1<br>
>> > emergency-closed: 0<br>
>> > udp:<br>
>> > new: 5<br>
>> > established: 50<br>
>> > emergency-new: 1<br>
>> > emergency-established: 1<br>
>> > icmp:<br>
>> > new: 1<br>
>> > established: 5<br>
>> > emergency-new: 1<br>
>> > emergency-established: 1<br>
>> ><br>
>> > reassembly:<br>
>> > memcap: 3gb<br>
>> > depth: 5mb<br>
>> ><br>
>> > These are the results, note how there are no more free slots for<br>
>> > PF_RING. I believe this results in increased packet loss... which is<br>
>> > likely resulting in my truncated files that I receive when I pull a PDF.<br>
>> ><br>
>> > watch 'cat /proc/net/pf_ring/*p4p2* | egrep "Num Free Slots|Tot<br>
>> > Packets|Tot Pkt Lost"'<br>
>> > ; First three threads...<br>
>> > Tot Packets : 80281541<br>
>> > Tot Pkt Lost : 44290194<br>
>> > Num Free Slots : 0<br>
>> > Tot Packets : 81926241<br>
>> > Tot Pkt Lost : 17412402<br>
>> > Num Free Slots : 0<br>
>> > Tot Packets : 80108557<br>
>> > Tot Pkt Lost : 14667061<br>
>> > Num Free Slots : 0<br>
>> ><br>
>> > watch -n 10 'cat stats.log | egrep<br>
>> > "reassembly_memuse|segment_memcap_drop" | tail -n 32'<br>
>> > ; First three threads...<br>
>> > tcp.segment_memcap_drop | RxPFRp4p21 | 0<br>
>> > tcp.reassembly_memuse | RxPFRp4p21 | 1681598708<br>
>> > tcp.segment_memcap_drop | RxPFRp4p22 | 0<br>
>> > tcp.reassembly_memuse | RxPFRp4p22 | 1681626644<br>
>> > tcp.segment_memcap_drop | RxPFRp4p23 | 0<br>
>> > tcp.reassembly_memuse | RxPFRp4p23 | 1681597556<br>
>> > tcp.segment_memcap_drop | RxPFRp4p24 | 0<br>
>> > *** Important to note here, the reassembly memuse seems to steadily<br>
>> > increase overtime. After a few minutes of putting this in it has risen<br>
>> > to 2022140776 across. Makes me think things are not offloading<br>
>> > quickly... (timeout/depth issue?)<br>
>> ><br>
>> > grep 'reassembly_gap' stats.log | tail -n 10<br>
>> > tcp.reassembly_gap | RxPFRp4p27 | 27603<br>
>> > tcp.reassembly_gap | RxPFRp4p28 | 26677<br>
>> > tcp.reassembly_gap | RxPFRp4p29 | 26869<br>
>> > tcp.reassembly_gap | RxPFRp4p210 | 25031<br>
>> > tcp.reassembly_gap | RxPFRp4p211 | 23988<br>
>> > tcp.reassembly_gap | RxPFRp4p212 | 23809<br>
>> > tcp.reassembly_gap | RxPFRp4p213 | 26420<br>
>> > tcp.reassembly_gap | RxPFRp4p214 | 25271<br>
>> > tcp.reassembly_gap | RxPFRp4p215 | 26285<br>
>> > tcp.reassembly_gap | RxPFRp4p216 | 26848<br>
>><br>
>> In 2.0.1 the stream engine should use less memory and clear memory<br>
>> quicker. Could you try 2.0.1?<br>
>><br>
>> Also, if you have multiple vlans on the network, it may be worth trying<br>
>> to disable:<br>
>><br>
>> vlan:<br>
>> use-for-tracking: true<br>
>><br>
>> I think you've probably checked all or most things on them, but perhaps<br>
>> these diagrams here can be of some help here:<br>
>><br>
>> <a href="https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Self_Help_Diagrams" target="_blank">https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Self_Help_Diagrams</a><br>
>><br>
>> --<br>
>> ---------------------------------------------<br>
>> Victor Julien<br>
>> <a href="http://www.inliniac.net/" target="_blank">http://www.inliniac.net/</a><br>
>> PGP: <a href="http://www.inliniac.net/victorjulien.asc" target="_blank">http://www.inliniac.net/victorjulien.asc</a><br>
>> ---------------------------------------------<br>
>><br>
>> _______________________________________________<br>
>> Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org">oisf-users@openinfosecfoundation.org</a><br>
>> Site: <a href="http://suricata-ids.org" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" target="_blank">http://suricata-ids.org/support/</a><br>
>> List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" target="_blank">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
>> OISF: <a href="http://www.openinfosecfoundation.org/" target="_blank">http://www.openinfosecfoundation.org/</a><br>
><br>
><br>
><br>
> _______________________________________________<br>
> Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org">oisf-users@openinfosecfoundation.org</a><br>
> Site: <a href="http://suricata-ids.org" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" target="_blank">http://suricata-ids.org/support/</a><br>
> List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" target="_blank">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
> OISF: <a href="http://www.openinfosecfoundation.org/" target="_blank">http://www.openinfosecfoundation.org/</a><br>
<br>
<br>
</div></div>How many rules do you load ? Based on that I would suggest to try<br>
<br>
detect-engine:<br>
- profile: high<br>
- custom-values:<br>
toclient-src-groups: 2<br>
toclient-dst-groups: 2<br>
toclient-sp-groups: 2<br>
toclient-dp-groups: 3<br>
toserver-src-groups: 2<br>
toserver-dst-groups: 4<br>
toserver-sp-groups: 2<br>
toserver-dp-groups: 25<br>
- sgh-mpm-context: full<br>
- inspection-recursion-limit: 3000<br>
<br>
What is important here is "- profile: high" and "-<br>
sgh-mpm-context: full". However you have to be careful with the other<br>
memcaps not to go in OOM based on the amount of RAM you have. (it will<br>
use about 10G ram on 9-10K rules, just for that)<br>
<br>
You could also look at the output of suricata.log after Suri has run<br>
for a period of time (kill -15 pidofsuricata) , that will produce some<br>
useful stats in terms of segment pools, then you can tune accordingly.<br>
<br>
thanks<br>
<span class="HOEnZb"><font color="#888888"><br>
<br>
--<br>
Regards,<br>
Peter Manev<br>
</font></span></blockquote></div><br></div>