<div dir="ltr">Hi Michał<div><br></div><div>Thanks a lot for your detailed reply and drawing in ASCII ! </div><div><br></div><div>I am testing with various options and I will let you know with the outcome.</div><div><br></div><div>Regards</div><div><br></div><div>Kashif</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Nov 1, 2019 at 9:38 PM Michał Purzyński <<a href="mailto:michalpurzynski1@gmail.com">michalpurzynski1@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>I think we are all over-complicating things here and no one shares a complete solution ;)</div><div><br></div><div><a href="https://www.arista.com/en/um-eos/eos-section-20-4-tap-aggregation-traffic-steering" target="_blank">https://www.arista.com/en/um-eos/eos-section-20-4-tap-aggregation-traffic-steering</a></div><div><br></div><div>If you have Arista in front of your cluster, you connect RX and TX from all your taps there, and then configure it creating bonding / aggregate interface on Arista (not on Linux) and connect those "bonded" ports to a number of network cards from the same or different sensors.</div><div>Traffic will be load balanced per 2 or 3-tuple. You can then start Suricata with AF_Packet on all Ethernet interfaces</div><div><br></div><div>OMG watch me drawing in ASCII ;)</div><div><br></div><div>---> (tap 1 RX) - |et1| | AR | <br></div><div>---> (tap 1 TX) - |et2| | IS | | et41 | -> | Po10 | -> | sensor's first card |<br></div><div>---> (tap 2 RX) - |et3| | TA | | et42 | -> | Po10 | -> | sensor's second card |<br></div><div>---> (tap 2 TX) - |et4| | |<br></div><div><br></div><div>Arista's configuration (and literally any packet broker will do, this one is cheap, Gigamon is expensive but makes a coffee while deduplicating packets and God knows what else. I deduplicated with a couple access-list, took me like one day)<br></div><div><br></div><div># conf t (hi, Cisco!!)<br></div><div><br></div><div>tap aggregation<br> mode exclusive</div><div><br></div><div>load-balance policies<br> load-balance fm6000 profile NSMConSymm<br> port-channel hash-seed 39<br> no fields mac<br> fields ip protocol dst-ip src-ip<br> distribution symmetric-hash mac-ip</div><div><br></div><div>reboot here</div><div><br></div><div>Create "output" aggregated links where you will connect your sensors, I have two (and I'm lying but that's for brevity)<br></div><div><br></div><div>interface Port-Channel10<br> description Bro production<br> l2 mtu 9000<br> switchport mode tool<br> switchport tool group set to_bro<br>!<br>interface Port-Channel20<br> description Suricata production<br> ip access-group drop_noise_before_suricata out<br> l2 mtu 9000<br> switchport mode tool<br> switchport tool group set to_suricata<br>!</div><div><br></div><div>Then for each port where you have a tap connected</div><div><br></div><div>interface Ethernet1<br> speed forced 10000full<br> l2 mtu 9000<br> ingress load-balance profile NSMConSymm<br> ip access-group drop_noise_from_taps in<br> service-policy type tapagg input from_taps<br> switchport mode tap</div><div><br></div><div>Repeat as necessary</div><div><br></div><div>Each sensor has two network cards here to deal with the "2x 10Gbit > 1x 10Gbit" problem and also for NUMA (optional)</div><div><br></div><div>First interface on the sensor, will get all packets that are part of the flow between 1.2.3.4 <-> 23.24.25.26 for example<br></div><div><br></div><div>interface Ethernet41<br> description nsm6:1a<br> channel-group 20 mode on<br> switchport mode tool<br>!</div><div><br></div><div>Second interface on the sensor, will get other 3-tuple hashed flows, etc<br></div><div><br></div><div>interface Ethernet42<br> description nsm6:2a<br> channel-group 20 mode on<br> switchport mode tool<br>!</div><div><br></div><div>Going back to Arista's configuration, now let's glue it all together</div><div><br></div><div>Class-map filters what's send to sensors, here we just send everything, while maintaining a configuration that lets me steer part of the traffic somewhere else, should I need to, like during DDoS.<br></div><div><br></div><div>class-map type tapagg match-any match_from_taps<br> 10 match ip access-group match_any<br>!<br>class-map type tapagg match-any match_noise<br> 10 match ip access-group send_noise_to_null<br>!</div><div><br></div><div>Take traffic "labeled" as from_taps, try to match it against class match_noise, if it does, send to ports tagged with "to_null. Take remaining traffic, try to match against class match_from_taps, everything that matches is send to ports tagged to_bro and to_suricata<br></div><div><br></div><div>policy-map type tapagg from_taps<br> 10 class match_noise<br> set aggregation-group to_null<br> !<br> 100 class match_from_taps<br> set aggregation-group group to_bro group to_suricata<br>!</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Nov 1, 2019 at 2:02 PM Nelson, Cooper <<a href="mailto:cnelson@ucsd.edu" target="_blank">cnelson@ucsd.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">What packet capture method are you using?
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">-Coop<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<div>
<div style="border-color:rgb(225,225,225) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:Calibri,sans-serif">From:</span></b><span style="font-size:11pt;font-family:Calibri,sans-serif"> Amar <<a href="mailto:amar@countersnipe.com" target="_blank">amar@countersnipe.com</a>>
<br>
<b>Sent:</b> Friday, November 1, 2019 1:58 PM<br>
<b>To:</b> Nelson, Cooper <<a href="mailto:cnelson@ucsd.edu" target="_blank">cnelson@ucsd.edu</a>><br>
<b>Cc:</b> mohammad kashif <<a href="mailto:kashif.alig@gmail.com" target="_blank">kashif.alig@gmail.com</a>>; Oisf-Users <<a href="mailto:oisf-users@lists.openinfosecfoundation.org" target="_blank">oisf-users@lists.openinfosecfoundation.org</a>><br>
<b>Subject:</b> Re: [Oisf-users] Suricata seperate Rx/Tx connection<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div id="gmail-m_6121730784821639393gmail-m_8961186362586601588edo-message">
<p class="MsoNormal">CounterSnipe default setup bonds all interfaces into a single bond#(0) and starts Suri with -i bond0 and it works fine. <u></u><u></u></p>
</div>
<div id="gmail-m_6121730784821639393gmail-m_8961186362586601588edo-original">
<div>
<p class="MsoNormal"><br>
<br>
<u></u><u></u></p>
<blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0in 0in 0in 3pt;margin-left:0in;margin-top:6pt;margin-right:0in">
<div>
<p class="MsoNormal" style="margin-bottom:12pt">On Nov 1, 2019 at 10:49 PM, <<a href="mailto:cnelson@ucsd.edu" target="_blank">Cooper Nelson</a>> wrote:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">That would work with pcap, not sure how AF_PACKET handles bonded interfaces.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">We use an Arista with two 10Gbit interfaces and pevma’s config.
</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">-Coop</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<div>
<div style="border-color:rgb(225,225,225) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:Calibri,sans-serif">From:</span></b><span style="font-size:11pt;font-family:Calibri,sans-serif"> Amar <<a href="mailto:amar@countersnipe.com" target="_blank">amar@countersnipe.com</a>>
<br>
<b>Sent:</b> Friday, November 1, 2019 8:19 AM<br>
<b>To:</b> mohammad kashif <<a href="mailto:kashif.alig@gmail.com" target="_blank">kashif.alig@gmail.com</a>><br>
<b>Cc:</b> Nelson, Cooper <<a href="mailto:cnelson@ucsd.edu" target="_blank">cnelson@ucsd.edu</a>>; Oisf-Users <<a href="mailto:oisf-users@lists.openinfosecfoundation.org" target="_blank">oisf-users@lists.openinfosecfoundation.org</a>><br>
<b>Subject:</b> Re: [Oisf-users] Suricata seperate Rx/Tx connection</span><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div id="gmail-m_6121730784821639393gmail-m_8961186362586601588edo-message">
<p class="MsoNormal">Could bonding be the solution here. Bond eth1 and 2 and simply monitor the bond. <u></u><u></u></p>
</div>
<div id="gmail-m_6121730784821639393gmail-m_8961186362586601588edo-message">
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div id="gmail-m_6121730784821639393gmail-m_8961186362586601588edo-original">
<div>
<p class="MsoNormal">On Nov 1, 2019 at 4:08 PM, <<a href="mailto:kashif.alig@gmail.com" target="_blank">mohammad kashif</a>> wrote:<br>
<br>
<br>
<u></u><u></u></p>
<blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0in 0in 0in 3pt;margin:6pt 0in 5pt">
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
_______________________________________________<br>
Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org" target="_blank">oisf-users@openinfosecfoundation.org</a><br>
Site: <a href="http://suricata-ids.org" rel="noreferrer" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" rel="noreferrer" target="_blank">http://suricata-ids.org/support/</a><br>
List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" rel="noreferrer" target="_blank">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
<br>
Conference: <a href="https://suricon.net" rel="noreferrer" target="_blank">https://suricon.net</a><br>
Trainings: <a href="https://suricata-ids.org/training/" rel="noreferrer" target="_blank">https://suricata-ids.org/training/</a></blockquote></div>
_______________________________________________<br>
Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org" target="_blank">oisf-users@openinfosecfoundation.org</a><br>
Site: <a href="http://suricata-ids.org" rel="noreferrer" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" rel="noreferrer" target="_blank">http://suricata-ids.org/support/</a><br>
List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" rel="noreferrer" target="_blank">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
<br>
Conference: <a href="https://suricon.net" rel="noreferrer" target="_blank">https://suricon.net</a><br>
Trainings: <a href="https://suricata-ids.org/training/" rel="noreferrer" target="_blank">https://suricata-ids.org/training/</a></blockquote></div>