[Oisf-users] Tuning Suricata Inline IPS performance

[Eric Leblond]

Hello,

Here's a list of things you can do to improve performance:
      * use multiple queues: you can then use the multithreading
        capability of your boxe and avoid some per-cpu lock.
      * Increase netfilter queue length: you will then be able to resist
        better to any burst effect
      * Uses NFQ in worker mode: this mode were one thread does all the
        work since capture to decision should be the most efficient for
        NFQ (currently not in official tree, see patch provided).

Let's you've got a multicore system with 4 CPUs.

On Netfilter side:
	iptables -A FORWARD -j NFQUEUE --queue-balance 0:3

This will balance the packet between queue 0 to 3 with per-connection
load-balancing. Please note that in this case your injection tool must
be multi connection aware.

Increase max_pending_packets in suricata.yaml to a decent value like
1000 (this will use "some" memory).

Start suricata with:
	suricata -c suricata.yaml -q 0 -q 1 -q 2 -q 3

If you are able to compile a suricata on your system, you can use
current git tree, apply the attached patch and run:
	suricata -c suricata.yaml -q 0 -q 1 -q 2 -q 3 --runmode workers

Next thing that can also be done is to work on cpu affinity to
synchronize CPU used for capture and treatment in Suricata.

Please let us know how things are improving with that.

BR,

On Mon, 2011-11-21 at 17:48 -0800, Hariharan Thantry wrote:
> Hi Victor,
> 
> 
> I think this is not necessarily because of Suricata itself, but by the
> use of iptables/NFQUEUE in a purely bridged environment. (The Suricata
> IPS does not have an IP address for the bridge).  I used the very
> simple NFQUEUE user space
> handler http://www.netfilter.org/projects/libnetfilter_queue/doxygen/nfqnl__test_8c_source.html, stopped Suricata, and kept the following iptables entry
> 
> 
> $ sudo iptables -A FORWARD -j NFQUEUE --queue-num 0
> 
> 
> and used the above program (which just puts the packet back out) on my
> bridge machine, and observed the same throughput speeds (~ 400 Kbps)
> using iperf. (Only a single connection activated)
> 
> 
> Interestingly, when I used ebtables, and its handler
> (ulog) http://ebtables.sourceforge.net/examples/basic.html#ex_ulog,
> with the ebtables FORWARD chain I observed near line rate speeds (>
> 9Gbps)
> 
> 
> $sudo ebtables -A FORWARD --ulog-nlgroup 1
> 
> 
> The major difference that I can see between the two handlers, is that
> in the case of NFQUEUE, the whole packet payload is actually copied
> into user space, while for the test_ulog it isn't. I tried with the
> NFQNL_COPY_META as well, and the speeds for that was ~ 2Mbps.
> 
> 
> I know this isn't an iptables/ebtables forum, but wondering if anyone
> can throw some light on this? I read this document
> here: http://ebtables.sourceforge.net/br_fw_ia/br_fw_ia.html, and this
> figure
> here http://ebtables.sourceforge.net/br_fw_ia/PacketFlow.png seems to
> suggest that the bridged packets do indeed go through the iptables
> filter table FORWARD chain...., so clearly, there is something that I
> don't have a handle on. My CPU utilization is pretty low ( ~ 8%), so
> that clearly isn't the issue here....
> 
> 
> Thanks,
> Hari
> 
> 
> 
> 
> 
> On Mon, Nov 21, 2011 at 10:37 AM, Victor Julien <victor at inliniac.net>
> wrote:
>         On 11/21/2011 09:00 AM, Hariharan Thantry wrote:
>         > When I turn on Suricata (latest 1.1 release version), with
>         the defaults,
>         > the speeds range between 350kbps-1Mbps (using emerging
>         threats ruleset).
>         
>         
>         Those numbers are way to low. I run a 8k ruleset in nfq mode
>         on an Atom
>         N270 and it easily keeps up with 12mbit (which is my internet
>         connection). So on that hardware you should see much better
>         speeds.
>         
>         Do you see one of the threads hit 100% all the time?
>         
>         How many rules are you using? And are you using the specific
>         Suricata ET
>         version?
>         
>         --
>         ---------------------------------------------
>         Victor Julien
>         http://www.inliniac.net/
>         PGP: http://www.inliniac.net/victorjulien.asc
>         ---------------------------------------------
>         
>         
>         _______________________________________________
>         Oisf-users mailing list
>         Oisf-users at openinfosecfoundation.org
>         http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>         
> 
> 
> _______________________________________________
> Oisf-users mailing list
> Oisf-users at openinfosecfoundation.org
> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users

-- 
Eric Leblond 
Blog: http://home.regit.org/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: nfq-worker-mode.diff
Type: text/x-patch
Size: 41229 bytes
Desc: not available
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20111122/ef1739e4/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20111122/ef1739e4/attachment.sig>