<div dir="ltr">As I indicated to Xavier, due to statefulness of a connection through the IPS, a single connection can go as fast as a single thread of Suri can go. The overall throughput of the box is likely quite higher for multiple connections accross the IPS.<div><br><div>Another factor is TCP round trip time and the reliability of the connection - distant connections tend to have lower values of the TCP sliding windows setting, so transit time through the IPS (NICs, nftables and Suricata) affects the max throughput for the single thread as well (due to packet ACK requirement before transmission of the next window's worth of packet(s) in each direction). </div><div><br></div><div>Cheers.</div><div><br></div><div>Dave</div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 8, 2015 at 3:19 AM, Xavier Romero <span dir="ltr"><<a href="mailto:XRomero@nexica.com" target="_blank">XRomero@nexica.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="CA" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Hello,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Another user explained me that there is a limitation at using single connection. I can confirm that each test achieves the same result, no matter
if I there is 1 test running, or 4.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">So, I guess that each connection is limited to… I don’t know what, since I can multiply total throughput by running several tests in parallel,
so it’s not a system limitation (ram, cpu, disk, etc)…<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Anyway I’ve checked that counters, they look good.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.0pt;font-family:Consolas;color:#1f497d">[root@suricata]# cat /proc/net/netfilter/nfnetlink_queue<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.0pt;font-family:Consolas;color:#1f497d"> 0 5405 0 2 65531 0 0 47862 1<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.0pt;font-family:Consolas;color:#1f497d"> 1 -4139 0 2 65531 0 0 332937 1<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Best regards,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Xavier Romero<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span lang="ES" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">De:</span></b><span lang="ES" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> <a href="mailto:oisf-users-bounces@lists.openinfosecfoundation.org" target="_blank">oisf-users-bounces@lists.openinfosecfoundation.org</a> [mailto:<a href="mailto:oisf-users-bounces@lists.openinfosecfoundation.org" target="_blank">oisf-users-bounces@lists.openinfosecfoundation.org</a>]
<b>En nombre de </b>Peter Fyon<br>
<b>Enviado el:</b> dijous, 8 d'octubre de 2015 4:44<br>
<b>Para:</b> <a href="mailto:oisf-users@lists.openinfosecfoundation.org" target="_blank">oisf-users@lists.openinfosecfoundation.org</a><br>
<b>Asunto:</b> Re: [Oisf-users] Inline NFQ<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p>Have you checked your nfnetlink_queue for dropped packets?<u></u><u></u></p>
<p>From <a href="https://home.regit.org/netfilter-en/using-nfqueue-and-libnetfilter_queue/" target="_blank">
https://home.regit.org/netfilter-en/using-nfqueue-and-libnetfilter_queue/</a> :<u></u><u></u></p>
<p><span style="font-size:14.0pt;font-family:"Arial","sans-serif";color:#333333">nfnetlink_queue entry in /proc</span><u></u><u></u></p>
<p style="margin:0cm;margin-bottom:.0001pt;line-height:16.7pt;outline:0px"><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black">nfnetlink_queue has a dedicated entry in /proc: </span><code><span style="font-size:10.5pt;color:black;border:none windowtext 1.0pt;padding:0cm;background:#f7f7f7">/proc/net/netfilter/nfnetlink_queue</span></code><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black"><u></u><u></u></span></p>
<div style="border:solid #dddddd 1.0pt;padding:12.0pt 12.0pt 12.0pt 12.0pt;background:#f7f7f7">
<pre style="margin-right:0cm;margin-bottom:18.0pt;margin-left:0cm;line-height:18.0pt;background:#f7f7f7;border:none;padding:0cm;outline:0px;border-radius:5px;overflow:auto"><span style="font-size:10.5pt;color:black">cat /proc/net/netfilter/nfnetlink_queue <u></u><u></u></span></pre>
<pre style="margin-right:0cm;margin-bottom:18.0pt;margin-left:0cm;line-height:18.0pt;background:#f7f7f7;border:none;padding:0cm"><span style="font-size:10.5pt;color:black"> 40 23948 0 2 65531 0 0 106 1<u></u><u></u></span></pre>
</div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black">The content is the following:</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:18.0pt;line-height:16.7pt">
<u></u><span style="font-size:10.0pt;font-family:Symbol;color:black"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black">queue number<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:18.0pt;line-height:16.7pt">
<u></u><span style="font-size:10.0pt;font-family:Symbol;color:black"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black">peer portid: good chance it is process ID of software listening to the queue<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:18.0pt;line-height:16.7pt">
<u></u><span style="font-size:10.0pt;font-family:Symbol;color:black"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black">queue total: current number of packets waiting in the queue<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:18.0pt;line-height:16.7pt">
<u></u><span style="font-size:10.0pt;font-family:Symbol;color:black"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black">copy mode: 0 and 1 only message only provide meta data. If 2 message provide a part of packet of size copy range.<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:18.0pt;line-height:16.7pt">
<u></u><span style="font-size:10.0pt;font-family:Symbol;color:black"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black">copy range: length of packet data to put in message<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:18.0pt;line-height:16.7pt">
<u></u><span style="font-size:10.0pt;font-family:Symbol;color:black"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black">queue dropped: number of packets dropped because queue was full<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:18.0pt;line-height:16.7pt">
<u></u><span style="font-size:10.0pt;font-family:Symbol;color:black"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black">user dropped: number of packets dropped because netlink message could not be sent to userspace. If this counter is not zero, try to increase netlink buffer
size. On the application side, you will see gap in packet id if netlink message are lost.<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:18.0pt;line-height:16.7pt">
<u></u><span style="font-size:10.0pt;font-family:Symbol;color:black"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black">id sequence: packet id of last packet<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:18.0pt;line-height:16.7pt">
<u></u><span style="font-size:10.0pt;font-family:Symbol;color:black"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black">1<u></u><u></u></span></p>
<div>
<p class="MsoNormal">I've never tried to run suricata in IPS mode using netfilter queues, but I did run snort for a while like that. I recall that the maximum queue length (on a ubuntu machine, at least) was 5000 packets. You could up this with a sysctl somehow,
but I don't remember the setting offhand.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Peter<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">On Oct 7, 2015 9:44 AM, "Xavier Romero" <<a href="mailto:XRomero@nexica.com" target="_blank">XRomero@nexica.com</a>> wrote:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-GB">Hello,</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">I’m successfully running Suricata (detection mode) for a long time in a dedicated physical machine, processing about 2 Gbps with no problem.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">Now I need to set up another Suricata box for inline mode as a virtual machine (just for 50Mbps), it’s a small VM (CentOS 7, 2 CPUs & 2 GB RAM) but it should
be enough thought. I set up iptables that way:</span><u></u><u></u></p>
<p class="MsoNormal"><i><span lang="EN-GB" style="font-family:Consolas"> </span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span lang="EN-GB" style="font-family:Consolas">iptables -I FORWARD -j NFQUEUE --queue-bypass --queue-balance 0:1</span></i><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">My problem is, when I start Suricata on inline mode, the network throughput drops dramatically… I run internet speed test</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><i><span lang="EN-GB" style="font-family:Consolas">[15:24:30][admin@test ~]$ ./speedtest-cli</span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span lang="EN-GB" style="font-family:Consolas">Retrieving
<a href="http://speedtest.net" target="_blank">speedtest.net</a> configuration...</span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span lang="EN-GB" style="font-family:Consolas">Retrieving
<a href="http://speedtest.net" target="_blank">speedtest.net</a> server list...</span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span lang="EN-GB" style="font-family:Consolas">Selecting best server based on latency...</span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span lang="EN-GB" style="font-family:Consolas">Hosted by masmovil (Madrid) [0.00 km]: 13.574 ms</span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span lang="EN-GB" style="font-family:Consolas">Testing download speed........................................</span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span lang="EN-GB" style="font-family:Consolas;color:red">Download: 602.14 Mbit/s</span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span lang="EN-GB" style="font-family:Consolas">Testing upload speed..................................................</span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span lang="EN-GB" style="font-family:Consolas;color:red">Upload: 136.50 Mbit/s</span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span lang="EN-GB" style="font-family:Consolas"> </span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span lang="EN-GB" style="font-family:Consolas">[15:25:02][root@suricata ~]$
<span style="color:red">systemctl start suricata</span></span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span lang="EN-GB" style="font-family:Consolas;color:red"> </span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span lang="EN-GB" style="font-family:Consolas">[15:25:24][admin@test ~]$ ./speedtest-cli</span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span lang="EN-GB" style="font-family:Consolas">Retrieving
<a href="http://speedtest.net" target="_blank">speedtest.net</a> configuration...</span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span lang="EN-GB" style="font-family:Consolas">Retrieving
<a href="http://speedtest.net" target="_blank">speedtest.net</a> server list...</span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span lang="EN-GB" style="font-family:Consolas">Selecting best server based on latency...</span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span lang="EN-GB" style="font-family:Consolas">Hosted by masmovil (Madrid) [0.00 km]: 10.948 ms</span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span lang="EN-GB" style="font-family:Consolas">Testing download speed........................................</span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span lang="EN-GB" style="font-family:Consolas;color:red">Download: 14.18 Mbit/s</span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span lang="EN-GB" style="font-family:Consolas">Testing upload speed..................................................</span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span lang="EN-GB" style="font-family:Consolas;color:red">Upload: 3.08 Mbit/s</span></i><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">I’ve tried with 1 and 2 queues (-q 0 –q 1), and in both autofp and workers mode, no matter… always same results. Suricata threads does not consume much CPU, so
it does not look like I need more cores.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">Neither dmesg, journctl, /var/log/messages nor suricata logs are complaing about anything.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">I’ve no idea where to look or what to try. Any suggestion will be wellcome.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">Best regards,</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">Xavier Romero</span><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><br>
_______________________________________________<br>
Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org" target="_blank">
oisf-users@openinfosecfoundation.org</a><br>
Site: <a href="http://suricata-ids.org" target="_blank">http://suricata-ids.org</a> | Support:
<a href="http://suricata-ids.org/support/" target="_blank">http://suricata-ids.org/support/</a><br>
List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" target="_blank">
https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
Suricata User Conference November 4 & 5 in Barcelona: <a href="http://oisfevents.net" target="_blank">
http://oisfevents.net</a><u></u><u></u></p>
</div>
</div>
</div>
</div>
<br>_______________________________________________<br>
Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org">oisf-users@openinfosecfoundation.org</a><br>
Site: <a href="http://suricata-ids.org" rel="noreferrer" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" rel="noreferrer" target="_blank">http://suricata-ids.org/support/</a><br>
List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" rel="noreferrer" target="_blank">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
Suricata User Conference November 4 & 5 in Barcelona: <a href="http://oisfevents.net" rel="noreferrer" target="_blank">http://oisfevents.net</a><br></blockquote></div><br></div>