Since you have the pf_ring ongoing .... you could try workers mode and 16 threads - my suggestion.<br>Have a look at the CPU load/drops...stats.log see how it behaves.<br><br>If you need something don't hesitate to ask for help....<br>
<br><div class="gmail_quote">On Tue, Mar 5, 2013 at 3:39 PM, Benson Mathews <span dir="ltr"><<a href="mailto:benson.mathews@gmail.com" target="_blank">benson.mathews@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Yes right now I'm learning and evaluating Suricata running on a box with two 1G feeds coming in. If all goes well I'd like use it to for monitoring our 10G link which gets an avg of 5-6Gbps.<div class="HOEnZb"><div class="h5">
<br><br><div class="gmail_quote">
On Sun, Mar 3, 2013 at 5:51 AM, Peter Manev <span dir="ltr"><<a href="mailto:petermanev@gmail.com" target="_blank">petermanev@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br><br><div class="gmail_quote"><div>On Fri, Mar 1, 2013 at 11:19 PM, Benson Mathews <span dir="ltr"><<a href="mailto:benson.mathews@gmail.com" target="_blank">benson.mathews@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Alright, So I guess the second interface line was a miss config. I had copy that part from the previous Suricata 1.2 setup that was build by someone else who no longer works in my group.<br><br>And i'm trying to test the new version for monitoring upto 10G traffic. I not really familiar the different runmodes and its benefits. Whats the default mode, would you says its efficient to run it in worker? Would that help reduce the CPU usage?<br>
<br>My current stats:<br>CPU (avg): 40%<br>PF-Ring stats for Suricata: <br>Tot Packets : 11294652219<br>Tot Pkt Lost : 1588411<br></blockquote></div><div>These stats - on what kind of traffic do you have that - 2Gbps ? <br>
</div><div><div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>Thanks,<br>Benson<div><div><br><br><div class="gmail_quote">On Thu, Feb 28, 2013 at 5:09 AM, Peter Manev <span dir="ltr"><<a href="mailto:petermanev@gmail.com" target="_blank">petermanev@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br><br><div class="gmail_quote"><div>On Thu, Feb 28, 2013 at 10:40 AM, Duarte Silva <span dir="ltr"><<a href="mailto:duarte.silva@serializing.me" target="_blank">duarte.silva@serializing.me</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div><div><div>Hi Peter,<br><br></div>I would say it only makes sense that if he is running Suricata in workers mode.<br></div></div></div></blockquote></div><div>I thought that was the case... <br></div><div>
<div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div><div><br></div>Regards,<br></div>Duarte Silva<br></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Thu, Feb 28, 2013 at 7:50 AM, Peter Manev <span dir="ltr"><<a href="mailto:petermanev@gmail.com" target="_blank">petermanev@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br><br><div class="gmail_quote"><div>On Thu, Feb 28, 2013 at 1:31 AM, Benson Mathews <span dir="ltr"><<a href="mailto:benson.mathews@gmail.com" target="_blank">benson.mathews@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Yes I did verify if the process was running through ps and top.and I also tried commenting the echo altogether and had no luck. <br><br>This time i edited the config file to enable logging to the file and removed the sleep and commented the echo statement again. And extra logging in the suricata.log file indicated that the path to my threshold.config file was incorrect. So I updated that part and now it seems to start the process correctly.<br>
<br>My avg CPU (16 processors, E5520 @ 2.27GHz and 48G RAM) has jumped from 12% to 50%. I have around 8k emerging threat rules enabled and monitoring a 2Gbps feed. I have it setup with PF_RING 5.3 <br><br>pfring:<br> - interface: eth2,eth3<br>
threads: 1<br></blockquote></div><div><br>I would suggest "threads: 16" , since you have 16 cores.<br><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
interface: eth2,eth3<br></blockquote><div>why do you have 2 interfaces twice ?<br></div><div><div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> cluster-id: 99 <br>
cluster-type: cluster_round_robin<br>
<br><br>top - 19:23:58 up 1 day, 2:05, 1 user, load average: 2.85, 3.46, 2.96<br>Tasks: 330 total, 21 running, 309 sleeping, 0 stopped, 0 zombie<br>
Cpu0 : 53.0%us, 19.5%sy, 1.3%ni, 24.5%id, 0.0%wa, 0.0%hi, 1.7%si, 0.0%st<br>Cpu1 : 41.6%us, 24.0%sy, 2.2%ni, 30.6%id, 0.0%wa, 0.0%hi, 1.6%si, 0.0%st<br>Cpu2 : 44.1%us, 18.0%sy, 1.3%ni, 34.3%id, 0.0%wa, 0.0%hi, 2.3%si, 0.0%st<br>
Cpu3 : 36.5%us, 22.8%sy, 1.6%ni, 36.2%id, 0.0%wa, 0.0%hi, 2.9%si, 0.0%st<br>Cpu4 : 54.3%us, 11.8%sy, 2.8%ni, 29.1%id, 0.0%wa, 0.0%hi, 2.1%si, 0.0%st<br>Cpu5 : 48.2%us, 11.7%sy, 0.7%ni, 37.1%id, 0.0%wa, 0.0%hi, 2.3%si, 0.0%st<br>
Cpu6 : 51.5%us, 15.6%sy, 1.4%ni, 30.2%id, 0.0%wa, 0.0%hi, 1.4%si, 0.0%st<br>Cpu7 : 57.5%us, 11.0%sy, 1.0%ni, 27.6%id, 0.0%wa, 0.0%hi, 2.9%si, 0.0%st<br>Cpu8 : 45.8%us, 32.3%sy, 0.3%ni, 15.3%id, 0.0%wa, 0.0%hi, 6.2%si, 0.0%st<br>
Cpu9 : 42.0%us, 22.5%sy, 1.7%ni, 31.7%id, 0.0%wa, 0.0%hi, 2.0%si, 0.0%st<br>Cpu10 : 42.7%us, 22.2%sy, 2.6%ni, 30.1%id, 0.0%wa, 0.0%hi, 2.3%si, 0.0%st<br>Cpu11 : 35.0%us, 22.0%sy, 2.5%ni, 36.2%id, 0.0%wa, 0.0%hi, 4.3%si, 0.0%st<br>
Cpu12 : 48.7%us, 12.6%sy, 1.3%ni, 32.5%id, 0.0%wa, 0.0%hi, 5.0%si, 0.0%st<br>Cpu13 : 50.6%us, 9.0%sy, 1.0%ni, 34.8%id, 0.0%wa, 0.0%hi, 4.5%si, 0.0%st<br>Cpu14 : 48.2%us, 15.4%sy, 1.0%ni, 30.9%id, 0.0%wa, 0.0%hi, 4.5%si, 0.0%st<br>
Cpu15 : 42.3%us, 16.7%sy, 2.5%ni, 34.4%id, 0.0%wa, 0.0%hi, 4.1%si, 0.0%st<br><br><br><br>Thank you very much for helping me with this! Atleast I have it started now, need to work on tuning it.<span><font color="#888888"><br>
<br>-Benson</font></span><div><div><br><br><br>
<div class="gmail_quote">On Wed, Feb 27, 2013 at 2:22 AM, Duarte Silva <span dir="ltr"><<a href="mailto:duarte.silva@serializing.me" target="_blank">duarte.silva@serializing.me</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<p dir="ltr">Hi,</p>
<p dir="ltr">did you try to check if Suricata is running using ps? It might be that you are echoing a empty PID to the file after the sleep. I would remove lines all together as Suricata creates the file anyway. </p>
<p dir="ltr">Another thing is, since you are running using daemon mode you sould enable the suricata.log. In the configuration file, search for console, you will see some logging options, enable the one for file logging.</p>
<p dir="ltr">Cheers, <br>
Duarte Silva</p><div><div>
<div class="gmail_quote">On 26 Feb 2013 22:44, "Benson Mathews" <<a href="mailto:benson.mathews@gmail.com" target="_blank">benson.mathews@gmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
just tried running the suricata bin file directly with the same options.... same result.<br><br><div class="gmail_quote">On Tue, Feb 26, 2013 at 5:36 PM, Benson Mathews <span dir="ltr"><<a href="mailto:benson.mathews@gmail.com" target="_blank">benson.mathews@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Thank you for the quick response Duarte!<br><br>I tried comment the line that wrote the PID to the PIDFILE in my init.d script (also tried using a sleep 2 without commenting). This is time there is no error on the start.log but when i check the service status it says PID file /var/run/suricata.pid exists, but process not running!<br>
<br>init.d script:<br>NAME=suricata<br>DAEMON=/usr/local/suricata/current/bin/$NAME<br>SURCONF=/etc/suricata/suricata.yaml<br>PIDFILE=/var/run/suricata.pid<br>IDMODE=pfring <br><br>...<br>...<br><br>SURICATA_OPTIONS=" -c $SURCONF --pidfile $PIDFILE --pfring -D"<br>
<br>case "$1" in<br> start)<br> if [ -f $PIDFILE ]; then<br> PID1=`cat $PIDFILE`<br> if kill -0 "$PID1" 2>/dev/null; then<br> echo "$NAME is already running with PID $PID1"<br>
exit 0<br> fi<br> fi<br> echo -n "Starting suricata in $IDMODE mode..."<br> $DAEMON $SURICATA_OPTIONS > /var/log/suricata/suricata-start.log 2>&1 &<br> PID1=$!<br>
<br> sleep 2 ### JUST ADDED<br> echo "$PID1" > $PIDFILE<br> echo " done."<br> ;;<br>-------<br><br><br>cat /var/log/suricata/suricata-start.log<br>26/2/2013 -- 17:28:22 - <Info> - This is Suricata version 1.4 RELEASE<br>
26/2/2013 -- 17:28:22 - <Info> - CPUs/cores online: 16<br>26/2/2013 -- 17:28:22 - <Info> - Failure when trying to get MTU via ioctl: 19<br>26/2/2013 -- 17:28:22 - <Error> - [ERRCODE: SC_ERR_MISSING_CONFIG_PARAM(118)] - NO logging compatible with daemon mode selected, suricata won't be able to log. Please update 'logging.outputs' in the YAML.<br>
26/2/2013 -- 17:28:22 - <Info> - allocated 3670016 bytes of memory for the defrag hash... 65536 buckets of size 56<br>26/2/2013 -- 17:28:22 - <Info> - preallocated 65535 defrag trackers of size 144<br>26/2/2013 -- 17:28:22 - <Info> - defrag memory usage: <a href="tel:13107056" value="+13107056" target="_blank">13107056</a> bytes, maximum: 33554432<br>
26/2/2013 -- 17:28:22 - <Info> - AutoFP mode using default "Active Packets" flow load balancer<br><br><br>If there any file that would give more details about why the process is failing to start?<br><br>Thanks,<br>
Benson<div><div><br><br><br><div class="gmail_quote">On Tue, Feb 26, 2013 at 4:46 PM, Duarte Silva <span dir="ltr"><<a href="mailto:duarte.silva@serializing.me" target="_blank">duarte.silva@serializing.me</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">Hi,</p>
<p dir="ltr">that happened to me whe I started Suricata with the init.d script. That's because the init.d script forks Suricata to the background and then creates a pid file before Suricata. If you remove the line that echos the Suricata process identifier to the pid file, it should work fine.</p>
<p dir="ltr">Best regards, <br>
Duarte Silva</p>
<div class="gmail_quote"><div><div>On 26 Feb 2013 21:32, "Benson Mathews" <<a href="mailto:benson.mathews@gmail.com" target="_blank">benson.mathews@gmail.com</a>> wrote:<br type="attribution"></div>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div>
Hi,<br><br>I just installed Suricata 1.4 on my server and I'm attempting to run it with PF_RINGS, but I get the following error while I start suricata.<br>cat /var/log/suricata/suricata-start.log<br>26/2/2013 -- 00:03:18 - <Info> - This is Suricata version 1.4 RELEASE<br>
26/2/2013 -- 00:03:18 - <Info> - CPUs/cores online: 16<br>26/2/2013 -- 00:03:18 - <Info> - Failure when trying to get MTU via ioctl: 19<br>26/2/2013 -- 00:03:18 - <Error> - [ERRCODE: SC_ERR_MISSING_CONFIG_PARAM(118)] - NO logging compatible with daemon mode selected, suricata won't be able to log. Please update 'logging.outputs' in the YAML.<br>
26/2/2013 -- 00:03:18 - <Info> - allocated 3670016 bytes of memory for the defrag hash... 65536 buckets of size 56<br>26/2/2013 -- 00:03:18 - <Info> - preallocated 65535 defrag trackers of size 144<br>26/2/2013 -- 00:03:18 - <Info> - defrag memory usage: <a href="tel:13107056" value="+13107056" target="_blank">13107056</a> bytes, maximum: 33554432<br>
26/2/2013 -- 00:03:18 - <Info> - AutoFP mode using default "Active Packets" flow load balancer<br>26/2/2013 -- 00:03:18 - <Error> - [ERRCODE: SC_ERR_INITIALIZATION(45)] - pid file '/var/run/suricata.pid' exists. Is Suricata already running? Aborting!<br>
<br>I tried deleting the pid file and restarting it but get the same error. I'm new to this, any help would be much appreciated!<br><br>Thanks,<br>Benson<br><br>
<br></div></div>_______________________________________________<br>
Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org" target="_blank">oisf-users@openinfosecfoundation.org</a><br>
Site: <a href="http://suricata-ids.org" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" target="_blank">http://suricata-ids.org/support/</a><br>
List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" target="_blank">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
OISF: <a href="http://www.openinfosecfoundation.org/" target="_blank">http://www.openinfosecfoundation.org/</a><br></blockquote></div>
</blockquote></div><br>
</div></div></blockquote></div><br>
</blockquote></div>
</div></div></blockquote></div><br>
</div></div><br>_______________________________________________<br>
Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org" target="_blank">oisf-users@openinfosecfoundation.org</a><br>
Site: <a href="http://suricata-ids.org" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" target="_blank">http://suricata-ids.org/support/</a><br>
List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" target="_blank">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
OISF: <a href="http://www.openinfosecfoundation.org/" target="_blank">http://www.openinfosecfoundation.org/</a><br></blockquote></div></div></div><span><font color="#888888"><br><br clear="all"><br>-- <br>
<div>Regards,</div>
<div>Peter Manev</div>
</font></span></blockquote></div><br></div>
</div></div></blockquote></div></div></div><span><font color="#888888"><br><br clear="all"><br>-- <br><div>Regards,</div>
<div>Peter Manev</div>
</font></span></blockquote></div><br>
</div></div></blockquote></div></div></div><span><font color="#888888"><br><br clear="all"><br>-- <br><div>Regards,</div>
<div>Peter Manev</div>
</font></span></blockquote></div><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div>Regards,</div>
<div>Peter Manev</div>