[Oisf-users] Improved PF_RING support, please test!
Matthew Jonkman
jonkman at emergingthreatspro.com
Thu Mar 10 15:54:53 UTC 2011
Those are incredible numbers! Thanks Josh for sharing your testing.
So if the sweet spot is around 96 threads, where can you push that as far as max throughput?
Matt
On Mar 10, 2011, at 10:44 AM, Joshua White wrote:
> We've now tested with 128 threads, CPU usage maxes around 99% with a 2Gbps
> stream. ~11GB of RAM used, 10GB of Virtual Memory reserved per instance.
>
> The sweet spot, at least for us, seems to be around 96 threads which loads the
> CPUs at about 50%. Not bad for a solid fully saturated 2Gbps stream.
>
> Josh
>
>
> On Wednesday, March 09, 2011 12:04:00 pm Victor Julien wrote:
>> Weird, I just tested it and it works. I did:
>>
>> dkms remove -m pf_ring -v 4 --all
>> edit /usr/src/pf_ring-4/linux/pf_ring.h to set CLUSTER_LEN to 32
>> dkms add -m pf_ring -v 4
>> dkms build -m pf_ring -v 4
>> dkms install -m pf_ring -v 4
>>
>> then I updated my suricata.yaml to set threads to 32
>>
>> started pf_ring, didn't work.
>>
>> rmmod pf_ring
>> modprobe pf_ring
>>
>> then it worked.... 32 pf_ring recv threads (note that the last digit of
>> the thread name is cut off, I'll fix that soon)
>>
>> Cheers,
>> Victor
>>
>> On 03/09/2011 05:58 PM, josh at securemind.org wrote:
>>> Yes, this is with equally increased CLUSTER_LEN 16 and 24 ...
>>>
>>> Other suggestions,
>>>
>>> Josh
>>>
>>> -------- Original Message --------
>>> Subject: Re: [Oisf-users] Improved PF_RING support, please test!
>>> From: Victor Julien <victor at inliniac.net
>>> <mailto:victor at inliniac.net>> Date: Wed, March 09, 2011 11:45 am
>>> To: josh at securemind.org <mailto:josh at securemind.org>
>>> Cc: oisf-users at openinfosecfoundation.org
>>> <mailto:oisf-users at openinfosecfoundation.org>
>>>
>>> Is this with the increased cluster len?
>>>
>>> From earlier conversation:
>>>
>>> "
>>> This value is hardset in kernel/linux/pf_ring.h
>>> #define CLUSTER_LEN 8
>>> "
>>>
>>> Cheers,
>>> Victor
>>>
>>>
>>> On 03/09/2011 05:40 PM, josh at securemind.org
>>>
>>> <mailto:josh at securemind.org> wrote:
>>>> Ok, so the patch never put the thread option into the
>>>> suricata.yaml. I reapplied the patch and all was good.
>>>>
>>>> - Ran with 8 threads, everything worked great, CPU utilization
>>>> seems much better 2Gbps of traffic and load was less then 7%
>>>> across all CPU's - Ran with 16 threads, failed
>>>> - Ran with 24 threads, failed
>>>>
>>>> --
>>>> [722] 9/3/2011 -- 11:35:09 - (log-droplog.c:182) <Info>
>>>> (LogDropLogInitCtx) -- Drop log output initialized, filename:
>>>> drop.log [722] 9/3/2011 -- 11:35:09 - (source-pfring.c:167) <Info>
>>>> (PfringLoadConfig) -- Going to use 16 PF_RING receive threads
>>>> [725] 9/3/2011 -- 11:35:09 - (source-pfring.c:309) <Info>
>>>> (ReceivePfringThreadInit) -- (RecvPfring1) Using PF_RING v.4.6.0,
>>>> interface eth0, cluster-id 99
>>>> [727] 9/3/2011 -- 11:35:09 - (source-pfring.c:309) <Info>
>>>> (ReceivePfringThreadInit) -- (RecvPfring3) Using PF_RING v.4.6.0,
>>>> interface eth0, cluster-id 99
>>>> [728] 9/3/2011 -- 11:35:09 - (source-pfring.c:309) <Info>
>>>> (ReceivePfringThreadInit) -- (RecvPfring4) Using PF_RING v.4.6.0,
>>>> interface eth0, cluster-id 99
>>>> [730] 9/3/2011 -- 11:35:09 - (source-pfring.c:309) <Info>
>>>> (ReceivePfringThreadInit) -- (RecvPfring6) Using PF_RING v.4.6.0,
>>>> interface eth0, cluster-id 99
>>>> [731] 9/3/2011 -- 11:35:10 - (source-pfring.c:309) <Info>
>>>> (ReceivePfringThreadInit) -- (RecvPfring7) Using PF_RING v.4.6.0,
>>>> interface eth0, cluster-id 99
>>>> [726] 9/3/2011 -- 11:35:10 - (source-pfring.c:309) <Info>
>>>> (ReceivePfringThreadInit) -- (RecvPfring2) Using PF_RING v.4.6.0,
>>>> interface eth0, cluster-id 99
>>>> [729] 9/3/2011 -- 11:35:10 - (source-pfring.c:309) <Info>
>>>> (ReceivePfringThreadInit) -- (RecvPfring5) Using PF_RING v.4.6.0,
>>>> interface eth0, cluster-id 99
>>>> [722] 9/3/2011 -- 11:35:10 - (stream-tcp.c:344) <Info>
>>>> (StreamTcpInitConfig) -- stream "max_sessions": 262144
>>>> [722] 9/3/2011 -- 11:35:10 - (stream-tcp.c:356) <Info>
>>>> (StreamTcpInitConfig) -- stream "prealloc_sessions": 32768
>>>> [722] 9/3/2011 -- 11:35:10 - (stream-tcp.c:366) <Info>
>>>> (StreamTcpInitConfig) -- stream "memcap": 33554432
>>>> [722] 9/3/2011 -- 11:35:10 - (stream-tcp.c:373) <Info>
>>>> (StreamTcpInitConfig) -- stream "midstream" session pickups:
>>>> disabled [722] 9/3/2011 -- 11:35:10 - (stream-tcp.c:381) <Info>
>>>> (StreamTcpInitConfig) -- stream "async_oneside": disabled
>>>> [722] 9/3/2011 -- 11:35:10 - (stream-tcp.c:390) <Info>
>>>> (StreamTcpInitConfig) -- stream.reassembly
>>>
>>> <http://stream.reassembly> <http://stream.reassembly>
>>> <http://stream.reassembly>>;
>>>
>>>> "memcap": 67108864
>>>> [722] 9/3/2011 -- 11:35:10 - (stream-tcp.c:410) <Info>
>>>> (StreamTcpInitConfig) -- stream.reassembly
>>>
>>> <http://stream.reassembly> <http://stream.reassembly>
>>> <http://stream.reassembly>>;
>>>
>>>> "depth": 1048576
>>>> [722] 9/3/2011 -- 11:35:10 - (stream-tcp.c:421) <Info>
>>>> (StreamTcpInitConfig) -- stream."inline": disabled
>>>> [734] 9/3/2011 -- 11:35:10 - (source-pfring.c:309) <Info>
>>>> (ReceivePfringThreadInit) -- (RecvPfring1) Using PF_RING v.4.6.0,
>>>> interface eth0, cluster-id 99
>>>> [732] 9/3/2011 -- 11:35:10 - (source-pfring.c:303) <Error>
>>>> (ReceivePfringThreadInit) -- [ERRCODE:
>>>> SC_ERR_PF_RING_SET_CLUSTER_FAILED(37)] - pfring_set_cluster
>>>> returned -1 for cluster-id: 99
>>>> [733] 9/3/2011 -- 11:35:10 - (source-pfring.c:303) <Error>
>>>> (ReceivePfringThreadInit) -- [ERRCODE:
>>>> SC_ERR_PF_RING_SET_CLUSTER_FAILED(37)] - pfring_set_cluster
>>>> returned -1 for cluster-id: 99
>>>> [722] 9/3/2011 -- 11:35:10 - (tm-threads.c:1475) <Error>
>>>> (TmThreadWaitOnThreadInit) -- [ERRCODE: SC_ERR_THREAD_INIT(49)] -
>>>
>>> thread
>>>
>>>> "RecvPfring8" closed on initialization.
>>>> [722] 9/3/2011 -- 11:35:10 - (suricata.c:1245) <Error> (main) --
>>>> [ERRCODE: SC_ERR_INITIALIZATION(45)] - Engine initialization
>>>> failed, aborting...
>>>>
>>>> --
>>>> Josh
>>>>
>>>> -------- Original Message --------
>>>> Subject: Re: [Oisf-users] Improved PF_RING support, please test!
>>>> From: Victor Julien <victor at inliniac.net
>>>
>>> <mailto:victor at inliniac.net> <mailto:victor at inliniac.net>>
>>>
>>>> Date: Wed, March 09, 2011 10:47 am
>>>> To: oisf-users at openinfosecfoundation.org
>>>
>>> <mailto:oisf-users at openinfosecfoundation.org>
>>>
>>>> <mailto:oisf-users at openinfosecfoundation.org>
>>>>
>>>> Josh, are you setting the pfring.threads <http://pfring.threads>
>>>
>>> <http://pfring.threads> <http://pfring.threads>>;
>>>
>>>> option in your suricata.yaml?
>>>> It appears you have it either not set or set to 1. Set it like
>>>> this:
>>>>
>>>> # PF_RING configuration. for use with native PF_RING support
>>>> # for more info see http://www.ntop.org/PF_RING.html
>>>> pfring:
>>>> # Number of receive threads (>1 will enable experimental flow
>>>> pinned # runmode)
>>>> threads: 8
>>>>
>>>> # Default interface we will listen on.
>>>> interface: eth0
>>>>
>>>> # Default clusterid. PF_RING will load balance packets based on
>>>> flow. # All threads/processes that will participate need to have
>>>> the same # clusterid.
>>>> cluster-id: 99
>>>>
>>>> # Default PF_RING cluster type. PF_RING can load balance per flow
>>>> or per hash.
>>>> # This is only supported in versions of PF_RING > 4.1.1.
>>>> cluster-type: cluster_round_robin
>>>>
>>>> If I set it to 8 pf_ring reports 8 rings...
>>>>
>>>> Cheers,
>>>> Victor
>>>>
>>>>
>>>> On 03/09/2011 12:06 AM, jwhite at everisinc.com
>>>
>>> <mailto:jwhite at everisinc.com>
>>>
>>>> <mailto:jwhite at everisinc.com> wrote:
>>>>> I've set it to 24, recompiled and it seems to be running ok, (as
>>>>> in
>>>>
>>>> it's
>>>>
>>>>> handling 2Gbps worth of packets) but I'm unable to get the number
>>>>> of rings to report as anything but 1. Have I missed somthing or
>>>>> is
>>>
>>> trully
>>>
>>>>> not working correctly. I've run it based on on flow and as
>>>>> packet... still no go.
>>>>>
>>>>> EV-SVR-006:~$ cat /proc/net/pf_ring/info
>>>>> PF_RING Version : 4.6.0 ($Revision: exported$)
>>>>> Ring slots : 32768
>>>>> Slot version : 12
>>>>> Capture TX : No [RX only]
>>>>> IP Defragment : No
>>>>> Transparent mode : Yes (mode 0)
>>>>> Total rings : 1
>>>>> Total plugins : 0
>>>>>
>>>>> - josh
>>>>>
>>>>> ---
>>>>>
>>>>> Date: 3/8/2011 -- 16:52:58 (uptime: 0d, 00h 03m 21s)
>>>>> -----------------------------------------------------------------
>>>>> -- Counter | TM Name | Value
>>>>> -----------------------------------------------------------------
>>>>> -- decoder.pkts <http://decoder.pkts> <http://decoder.pkts>
>>>
>>> <http://decoder.pkts>>; <http://decoder.pkts >; |
>>>
>>>>> Decode1 | 16416387
>>>>> decoder.bytes <http://decoder.bytes> <http://decoder.bytes>
>>>
>>> <http://decoder.bytes>>; <http://decoder.bytes >; |
>>>
>>>>> Decode1 | 988040217
>>>>> decoder.ipv4 | Decode1 | 16461043
>>>>> decoder.ipv6 | Decode1 | 700
>>>>> decoder.ethernet <http://decoder.ethernet>
>>>
>>> <http://decoder.ethernet> <http://decoder.ethernet>>;
>>> <http://decoder.ethernet
>>>
>>>>> ; |
>>>>>
>>>>> Decode1 | 16406387
>>>>> decoder.raw | Decode1 | 0
>>>>> decoder.sll <http://decoder.sll> <http://decoder.sll>
>>>
>>> <http://decoder.sll>>; <http://decoder.sll >; |
>>>
>>>>> Decode1 | 0
>>>>> decoder.tcp <http://decoder.tcp> <http://decoder.tcp>
>>>
>>> <http://decoder.tcp>>; <http://decoder.tcp >; |
>>>
>>>>> Decode1 | 16406097
>>>>> decoder.udp | Decode1 | 1130
>>>>> decoder.icmpv4 | Decode1 | 3
>>>>> decoder.icmpv6 | Decode1 | 0
>>>>> decoder.ppp | Decode1 | 0
>>>>> decoder.pppoe | Decode1 | 0
>>>>> decoder.gre <http://decoder.gre> <http://decoder.gre>
>>>
>>> <http://decoder.gre>>; <http://decoder.gre >; |
>>>
>>>>> Decode1 | 0
>>>>> decoder.vlan | Decode1 | 0
>>>>> decoder.avg_pkt_size | Decode1 | 60.012753
>>>>> decoder.max_pkt_size <http://decoder.max_pkt_size>
>>>
>>> <http://decoder.max_pkt_size> <http://decoder.max_pkt_size>>;
>>>
>>>> <http://decoder.max_pkt_size >; |
>>>>
>>>>> Decode1 | 382
>>>>> defrag.ipv4.fragments <http://defrag.ipv4.fragments>
>>>
>>> <http://defrag.ipv4.fragments> <http://defrag.ipv4.fragments>>;
>>>
>>>> <http://defrag.ipv4.fragments >; |
>>>>
>>>>> Decode1 | 0
>>>>> defrag.ipv4.reassembled <http://defrag.ipv4.reassembled>
>>>
>>> <http://defrag.ipv4.reassembled> <http://defrag.ipv4.reassembled>>;
>>>
>>>> <http://defrag.ipv4.reassembled >; |
>>>>
>>>>> Decode1 | 0
>>>>> defrag.ipv4.timeouts | Decode1 | 0
>>>>> defrag.ipv6.fragments <http://defrag.ipv6.fragments>
>>>
>>> <http://defrag.ipv6.fragments> <http://defrag.ipv6.fragments>>;
>>>
>>>> <http://defrag.ipv6.fragments >; |
>>>>
>>>>> Decode1 | 0
>>>>> defrag.ipv6.reassembled <http://defrag.ipv6.reassembled>
>>>
>>> <http://defrag.ipv6.reassembled> <http://defrag.ipv6.reassembled>>;
>>>
>>>> <http://defrag.ipv6.reassembled >; |
>>>>
>>>>> Decode1 | 0
>>>>> defrag.ipv6.timeouts | Decode1 | 0
>>>>> tcp.sessions <http://tcp.sessions> <http://tcp.sessions>
>>>
>>> <http://tcp.sessions>>; <http://tcp.sessions >; |
>>>
>>>>> Stream1 | 844100
>>>>> tcp.ssn_memcap_drop | Stream1 | 0
>>>>> tcp.pseudo <http://tcp.pseudo> <http://tcp.pseudo>
>>>
>>> <http://tcp.pseudo>>; <http://tcp.pseudo >; |
>>>
>>>>> Stream1 | 0
>>>>> tcp.segment_memcap_drop <http://tcp.segment_memcap_drop>
>>>
>>> <http://tcp.segment_memcap_drop> <http://tcp.segment_memcap_drop>>;
>>>
>>>> <http://tcp.segment_memcap_drop >; |
>>>>
>>>>> Stream1 | 0
>>>>> tcp.stream_depth_reached <http://tcp.stream_depth_reached>
>>>
>>> <http://tcp.stream_depth_reached> <http://tcp.stream_depth_reached>>;
>>>
>>>> <http://tcp.stream_depth_reached >; |
>>>>
>>>>> Stream1 | 0
>>>>> detect.alert <http://detect.alert> <http://detect.alert>
>>>
>>> <http://detect.alert>>; <http://detect.alert >; |
>>>
>>>>> Detect | 24768
>>>>>
>>>>> - Josh
>>>>>
>>>>> On Tuesday, March 08, 2011 04:25:04 am Victor Julien wrote:
>>>>> Thanks Luca, I guess we'll find out soon enough if ppl run into
>>>>> perf issues... I think we have results for 16 and 24 soon.
>>>>>
>>>>> Cheers,
>>>>> Victor
>>>>>
>>>>> On 03/08/2011 08:42 AM, Luca Deri wrote:
>>>>>> Victor
>>>>>> this is just a define you can increase. However note that
>>>>>> clusters
>>>>
>>>> were
>>>>
>>>>>> designed to handle a few apps, thus if you increase the value to
>>>>>> a
>>>>
>>>> much
>>>>
>>>>>> higher value, we better review the code and see if it is still
>>>>
>>>> efficient
>>>>
>>>>>> enough for your purposes
>>>>>>
>>>>>> Regards Luca
>>>>>>
>>>>>> On Mar 7, 2011, at 9:49 PM, Victor Julien wrote:
>>>>>>
>>>>>> Thanks for figuring that out Will!
>>>>>>
>>>>>> Luca, can we have a higher (or no) limit? I keep hearing stories
>>>>
>>>> of ppl
>>>>
>>>>>> with 24 cores :)
>>>>>>
>>>>>> Cheers,
>>>>>> Victor
>>>>>>
>>>>>> On 03/07/2011 07:28 PM, Will Metcalf wrote:
>>>>>>>>>> It seems that the maximum is 8 threads here. Not sure if
>>>>>>>>>> that
>>>>
>>>> can be
>>>>
>>>>>>>>>> set higher if the number of slots in pfring are increased.
>>>
>>> Anyone
>>>
>>>>>>>>>> try that?
>>>>>>>>>
>>>>>>>>> This value is hardset in kernel/linux/pf_ring.h
>>>>>>>>>
>>>>>>>>> #define CLUSTER_LEN 8
>>>>>>>>>
>>>>>>>>> Modify accordingly, maybe Luca would be kind enough to
>>>
>>> increase the
>>>
>>>>>>>>> default value?
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Will
>>>>>>>>>
>>>>>>>>>> On Mon, Mar 7, 2011 at 11:47 AM, Victor Julien
>>>>>>
>>>>>> <victor at inliniac.net <mailto:victor at inliniac.net>
>>>
>>> <mailto:victor at inliniac.net>
>>>
>>>> <mailto:victor at inliniac.net>>
>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> On 03/07/2011 06:15 PM, Chris Wakelin wrote:
>>>>>>>>>>> On 28/02/11 20:23, Victor Julien wrote:
>>>>>>>>>>>> Hey guys,
>>>>>>>>>>>>
>>>>>>>>>>>> I know a couple of you are running PF_RING in a high speed
>>>>>>>>>>>> environment. The attached patch means to improve it's
>>>>
>>>> performance.
>>>>
>>>>>>>>>>>> It adds a new option called "pfring.threads" that controls
>>>>>>>>>>>> the number of reader threads the pfring code uses. I've
>>>>>>>>>>>> tested (lightly) with 1, 4 and 8 which all worked fine.
>>>>>>>>>>>> There are
>>>
>>> some
>>>
>>>>>>>>>>>> more improvements, including the removal of one memcpy per
>>>>>>>>>>>> packet...
>>>>>>>>>>>
>>>>>>>>>>> OK, giving it a go with 4 threads ...
>>>>>>>>>>
>>>>>>>>>> It seems that the maximum is 8 threads here. Not sure if
>>>>>>>>>> that
>>>>
>>>> can be
>>>>
>>>>>>>>>> set higher if the number of slots in pfring are increased.
>>>
>>> Anyone
>>>
>>>>>>>>>> try that?
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Victor
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Oisf-users mailing list
>>>>>>>>>> Oisf-users at openinfosecfoundation.org
>>>
>>> <mailto:Oisf-users at openinfosecfoundation.org>
>>>
>>>> <mailto:Oisf-users at openinfosecfoundation.org>
>>>>
>>>>>> <mailto:Oisf-users at openinfosecfoundation.org>
>>>
>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>
>>>>>> ---
>>>>>> Keep looking, don't settle - Steve Jobs
>>>>
>>>> _______________________________________________
>>>> Oisf-users mailing list
>>>> Oisf-users at openinfosecfoundation.org
>>>
>>> <mailto:Oisf-users at openinfosecfoundation.org>
>>>
>>>> <mailto:Oisf-users at openinfosecfoundation.org>
>>>>
>>>>> <mailto:Oisf-users at openinfosecfoundation.org>
>>>>
>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>>
>>>>> _______________________________________________
>>>>> Oisf-users mailing list
>>>>> Oisf-users at openinfosecfoundation.org
>>>
>>> <mailto:Oisf-users at openinfosecfoundation.org>
>>>
>>>> <mailto:Oisf-users at openinfosecfoundation.org>
>>>>
>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-user
>>>>> s
>>>>
>>>> --
>>>> ---------------------------------------------
>>>> Victor Julien
>>>> http://www.inliniac.net/
>>>> PGP: http://www.inliniac.net/victorjulien.asc
>>>> ---------------------------------------------
>>>>
>>>> _______________________________________________
>>>> Oisf-users mailing list
>>>> Oisf-users at openinfosecfoundation.org
>>>
>>> <mailto:Oisf-users at openinfosecfoundation.org>
>>>
>>>> <mailto:Oisf-users at openinfosecfoundation.org>
>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> _______________________________________________
> Oisf-users mailing list
> Oisf-users at openinfosecfoundation.org
> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
----------------------------------------------------
Matthew Jonkman
Emergingthreats.net
Emerging Threats Pro
Open Information Security Foundation (OISF)
Phone 765-807-8630 x110
Fax 312-264-0205
http://www.emergingthreatspro.com
http://www.openinfosecfoundation.org
----------------------------------------------------
PGP: http://www.jonkmans.com/mattjonkman.asc
More information about the Oisf-users
mailing list