[Oisf-users] Suricata under libvirt
Chris Boley
ilgtech75 at gmail.com
Wed Jun 15 19:22:37 UTC 2016
I'm not sure how to get it in there but would be eager to contribute.
On Wednesday, June 15, 2016, Andreas Herz <andi at geekosphere.org> wrote:
> On 31/05/16 at 09:02, Chris Boley wrote:
> > Andreas, this is lengthy sorry, I tried to include useful detail but,
> > unfortunately I ramble on a lot:
>
> Thanks for the detailed information. What could be interesting, that you
> put that in our redmine wiki so it won't get lost.
>
> Looks like a very special usecase/setup, thus not that many responses,
> but still worth it to be documented.
>
> >
> > I had this idea because I wanted to be able to have two or 3 working
> > Suricata test OS's for experimenting with things in the YAML and
> > other configs. This allows me to set up multiple VM's with varying
> configs
> > and find out what works and what doesn't.
> > That's why I did it.
> >
> > My basic setup.
> > I had 4 ethernet adapters installed in a dell 2970 with latest bios, 2 -
> 6
> > Core opterons running at 2.6 ghz, and 32 gigs of RAM.
> >
> > Why 4 nics? 1. host interface 2. VM Mgmt. Interface 3. Bridge-side A 4.
> > Bridge-Side B.
> >
> >
> > If your physical host interfaces are like em1 em2 or whatever in Ubuntu
> > edit /etc/default/grub
> >
> > find these two lines:
> > GRUB_CMDLINE_LINUX_DEFAULT="splash quiet"
> > GRUB_CMDLINE_LINUX=""
> >
> >
> > modify to look like
> >
> > GRUB_CMDLINE_LINUX_DEFAULT="splash quiet biosdevname=0"
> > GRUB_CMDLINE_LINUX="biosdevname=0"
> >
> > sudo update-grub
> >
> > Modify your operational host interface in /etc/network/interfaces file to
> > reflect eth0 instead of whatever it was named before.
> > If you don't, you'll find yourself without a working interface.
> >
> > then reboot
> >
> >
> ---------------------------------------------------------------------------------------------
> >
> > **macvtap with passthrough must have a dedicated physical interface to
> tie
> > itself to.
> >
> >
> > You've got to create a specialized network configuration within the
> libvirt
> > setup.
> > ** Caveats** You need to have hardware and BIOS platform that supports
> > SR-IOV.
> > How do you figure that out? Do the homework. Or simply experiment. I just
> > tried building the VM initially via virt-manager.
> > It let me do it. Meaning, the toolkit detected the right resources to
> > accomplish building the parameters I gave it.
> >
> > When I looked up SR-IOV capabilities online through DELL, I didn't see
> > where it was supported but I tried it anyway and it worked.
> > If it doesn't libvirt will just bark at you and tell you that you can't
> do
> > that. Here's a good link to explain SR-IOV:
> >
> >
> > http://blog.scottlowe.org/2009/12/02/what-is-sr-iov/
> >
> > ---A note about command line versus CLI installs
> > For those of you who are adamant about staying on ssh cli access only
> with
> > the host;
> > this script below will startup a virtual machine net install for ubuntu
> > 14.04 64 bit in your terminal with full hardware acceleration.
> > The machine will initially have a NAT based virtual network interface, 16
> > gig of ram and 8 vcpu's.
> > You'll be dealing with a qcow2 based harddisk file which doesn't offer
> the
> > best performance.
> > but I like it because of being able to snapshot. Look up libvirt
> snapshot +
> > qcow2 and you'll find some good articles on snappshotting.
> > I'm not going to write a book here. :)
> >
> > I toyed with setting up the interfaces right from this install script. In
> > the end and 4 hours later, it ended up being
> > more trouble than it was worth. I used virt-manager from a xwindows GUI
> in
> > lieu of all the headache.
> >
> > Plus, there are adjustments you can make in virt-manager that I've never
> > figured out how to do in virsh.
> > My most sincere recommendation is to install a really lightweight window
> > manager.
> > I tend to install Lubuntu minimal desktop during my server install and
> add
> > in some basics like leafpad but most importantly virt-manager.
> > Use whatever desktop you want. Maybe blackbox or fluxbox would be a great
> > choice as they use almost no resources.
> > You'll find manipulating VM's with Virt manager is much easier.
> > Moreover, you can tailor your cpu's settings more easily which is kind of
> > important for Suricata.
> >
> > Alternatively, you can stick to the cli and modify the xml with 'virsh
> > edit'. Beware that it uses 'vi' and is kind of a PITA.
> >
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> > sudo apt-get -y install qemu-kvm libvirt-bin bridge-utils qemu-system
> > & virt-manager if you're going to install a GUI....
> >
> > qemu-img create -f qcow2 -o preallocation=full
> > /home/ipsadmin/vmimgs/virtdrive.qcow2 64G
> >
> > virt-install --connect=qemu:///system \
> > --name=IPSTEST \
> > --ram 16384 \
> > --disk
> >
> path=/home/ipsadmin/vmimgs/virtdrive.qcow2,format=qcow2,bus=virtio,cache=none,size=64
> > \
> > --vcpus=8 \
> > --os-type linux \
> > --os-variant ubuntutrusty \
> > --network bridge=virbr0 \
> > --check-cpu \
> > --hvm \
> > --location '
> > http://archive.ubuntu.com/ubuntu/dists/trusty/main/installer-amd64/' \
> > --graphics none \
> > --console pty,target_type=serial \
> > --extra-args="console=ttyS0,115200n8 serial" \
> >
> >
> > ***add ' --debug ' flag into that script if you want to see verbose
> output
> > of what's going on.
> >
> > ---- for the second time you need to connect to the VM.-----
> >
> > The first time, based on the above settings, it will connect
> automatically.
> >
> > virsh start IPSTEST
> >
> > virsh console IPSTEST
> >
> >
> >
> -----------------------------------------------------------------------------------------------------------------------------------
> > ----for editing the machine xml if you want to stick to the CLI. ** This
> is
> > an example** modify to fit your needs.
> > Find the adapter and paste in the place of the original adapter segment,
> > something that looks like this below.
> > Obviously you need 3 adapters for the VM 2 for bridge and one for mgmt
> > iface. Be mindful of slot numbers and alias naming nomenclature.
> > Logically in the xml you'll find that most things like slot number
> > increment. Just make sure you don't conflict with something else.
> > You'll break the VM in that case. Make a backup before you edit. Eth1 in
> > the case below will take the place of what used to be
> >
> > your virtual adapter that was natted from inside to outside the host. Now
> > you've tied it to a physical interface. It will pull DHCP
> >
> > addresses from the physical network.
> >
> >
> > <interface type='direct'>
> > <mac address='52:54:00:67:7e:5d'/>
> > <source dev='eth1' mode='passthrough'/>
> > <target dev='macvtap0'/>
> > <model type='virtio'/>
> > <alias name='net0'/>
> > <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > function='0x0'/>
> > </interface>
> > <interface type='direct'>
> > <mac address='52:54:00:67:7e:5d'/>
> > <source dev='eth2' mode='passthrough'/>
> > <target dev='macvtap0'/>
> > <model type='virtio'/>
> > <alias name='net1'/>
> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> > function='0x0'/>
> > </interface>
> > <interface type='direct'>
> > <mac address='52:54:00:67:7e:5d'/>
> > <source dev='eth2' mode='passthrough'/>
> > <target dev='macvtap0'/>
> > <model type='virtio'/>
> > <alias name='net2'/>
> > <address type='pci' domain='0x0000' bus='0x00' slot='0x04'
> > function='0x0'/>
> > </interface>
> >
> ------------------------------------------------------------------------------------------------------------------------------------------
> >
> > My full xml off my primary test machine looks like this:
> >
> > idsadmin at SRVCHSURICATA1:~$ virsh dumpxml CSNIPESNSR01
> > <domain type='kvm' id='2'>
> > <name>CSNIPESNSR01</name>
> > <uuid>ec0d0563-aa0a-ef29-5f24-ba5ba1416b50</uuid>
> > <memory unit='KiB'>16777216</memory>
> > <currentMemory unit='KiB'>16777216</currentMemory>
> > <vcpu placement='static'>8</vcpu>
> > <resource>
> > <partition>/machine</partition>
> > </resource>
> > <os>
> > <type arch='i686' machine='pc-i440fx-trusty'>hvm</type>
> > <boot dev='hd'/>
> > </os>
> > <features>
> > <acpi/>
> > <apic/>
> > <pae/>
> > </features>
> > <cpu mode='custom' match='exact'>
> > <model fallback='allow'>Opteron_G3</model>
> > <vendor>AMD</vendor>
> > <feature policy='require' name='skinit'/>
> > <feature policy='require' name='vme'/>
> > <feature policy='require' name='mmxext'/>
> > <feature policy='require' name='fxsr_opt'/>
> > <feature policy='require' name='cr8legacy'/>
> > <feature policy='require' name='ht'/>
> > <feature policy='require' name='3dnowprefetch'/>
> > <feature policy='require' name='3dnowext'/>
> > <feature policy='require' name='wdt'/>
> > <feature policy='require' name='extapic'/>
> > <feature policy='require' name='pdpe1gb'/>
> > <feature policy='require' name='osvw'/>
> > <feature policy='require' name='ibs'/>
> > <feature policy='require' name='cmp_legacy'/>
> > <feature policy='require' name='3dnow'/>
> > </cpu>
> > <clock offset='utc'/>
> > <on_poweroff>destroy</on_poweroff>
> > <on_reboot>restart</on_reboot>
> > <on_crash>restart</on_crash>
> > <devices>
> > <emulator>/usr/bin/kvm-spice</emulator>
> > <disk type='file' device='disk'>
> > <driver name='qemu' type='qcow2'/>
> > <source file='/home/idsadmin/CSNIPE/snipedrive.qcow2'/>
> > <target dev='vda' bus='virtio'/>
> > <serial>1</serial>
> > <alias name='virtio-disk0'/>
> > <address type='pci' domain='0x0000' bus='0x00' slot='0x05'
> > function='0x0'/>
> > </disk>
> > <disk type='file' device='cdrom'>
> > <driver name='qemu' type='raw'/>
> > <target dev='hdc' bus='ide'/>
> > <readonly/>
> > <alias name='ide0-1-0'/>
> > <address type='drive' controller='0' bus='1' target='0' unit='0'/>
> > </disk>
> > <controller type='usb' index='0'>
> > <alias name='usb0'/>
> > <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> > function='0x2'/>
> > </controller>
> > <controller type='pci' index='0' model='pci-root'>
> > <alias name='pci.0'/>
> > </controller>
> > <controller type='ide' index='0'>
> > <alias name='ide0'/>
> > <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> > function='0x1'/>
> > </controller>
> > <interface type='direct'>
> > <mac address='52:54:00:b1:16:c2'/>
> > <source dev='eth3' mode='passthrough'/>
> > <target dev='macvtap0'/>
> > <model type='virtio'/>
> > <alias name='net0'/>
> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> > function='0x0'/>
> > </interface>
> > <interface type='direct'>
> > <mac address='52:54:00:3e:db:eb'/>
> > <source dev='eth0' mode='passthrough'/>
> > <target dev='macvtap1'/>
> > <model type='virtio'/>
> > <alias name='net1'/>
> > <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
> > function='0x0'/>
> > </interface>
> > <interface type='direct'>
> > <mac address='52:54:00:1c:71:70'/>
> > <source dev='eth1' mode='passthrough'/>
> > <target dev='macvtap2'/>
> > <model type='virtio'/>
> > <alias name='net2'/>
> > <address type='pci' domain='0x0000' bus='0x00' slot='0x08'
> > function='0x0'/>
> > </interface>
> > <serial type='pty'>
> > <source path='/dev/pts/2'/>
> > <target port='0'/>
> > <alias name='serial0'/>
> > </serial>
> > <console type='pty' tty='/dev/pts/2'>
> > <source path='/dev/pts/2'/>
> > <target type='serial' port='0'/>
> > <alias name='serial0'/>
> > </console>
> > <input type='mouse' bus='ps2'/>
> > <input type='keyboard' bus='ps2'/>
> > <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'>
> > <listen type='address' address='127.0.0.1'/>
> > </graphics>
> > <sound model='ich6'>
> > <alias name='sound0'/>
> > <address type='pci' domain='0x0000' bus='0x00' slot='0x04'
> > function='0x0'/>
> > </sound>
> > <video>
> > <model type='cirrus' vram='9216' heads='1'/>
> > <alias name='video0'/>
> > <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > function='0x0'/>
> > </video>
> > <memballoon model='virtio'>
> > <alias name='balloon0'/>
> > <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
> > function='0x0'/>
> > </memballoon>
> > </devices>
> > <seclabel type='dynamic' model='apparmor' relabel='yes'>
> > <label>libvirt-ec0d0563-aa0a-ef29-5f24-ba5ba1416b50</label>
> > <imagelabel>libvirt-ec0d0563-aa0a-ef29-5f24-ba5ba1416b50</imagelabel>
> > </seclabel>
> > </domain>
> >
> ----------------------------------------------------------------------------------------------------------------------------
> >
> > I set up my interfaces on the host like what's below here. I used my eth0
> > and eth1 on the VM Bridge. Eth2 is my host interface.
> > Eth3 is the 3rd interface on the VM dedicated to mgmt interface on the
> VM.
> > This way you can SSH into the VM and manage it.
> >
> > The ' interfacetune' file listed there in the interfaces script is a page
> > torn directly out of Peter Manev's Github page on how to
> > tune interfaces best for Suricata use. (thanks again Peter. Your shared
> > info is really awesome!) See what's in the script below.
> > Physical Host interfaces file:
> >
> > # The loopback network interface
> > auto lo
> > iface lo inet loopback
> >
> > auto eth2
> > iface eth2 inet static
> > address 192.168.5.6
> > netmask 255.255.255.0
> > gateway 192.168.5.1
> > dns-nameservers 192.168.5.1
> >
> > auto eth3
> > iface eth3 inet manual
> > pre-up modprobe 8021q
> > post-up ifconfig $IFACE up
> > pre-down ifconfig $IFACE down
> >
> > auto eth0
> > iface eth0 inet manual
> > post-up ifconfig $IFACE up
> > post-up ifconfig eth0 mtu 1520
> > post-up /etc/network/if-up.d/interfacetune
> > post-up ethtool -s eth0 autoneg off speed 1000 duplex full
> > pre-down ifconfig $IFACE down
> >
> > auto eth1
> > iface eth1 inet manual
> > post-up ifconfig $IFACE up
> > post-up ifconfig eth1 mtu 1520
> > post-up /etc/network/if-up.d/interfacetune
> > post-up ethtool -s eth1 autoneg off speed 1000 duplex full
> > pre-down ifconfig $IFACE down
> >
> >
> >
> ---------------------------------------------------------------------------------------------------------------------
> >
> >
> > idsadmin at SRVCHSURICATA1:~$ sudo cat /etc/network/if-up.d/interfacetune
> > /sbin/ethtool -G $IFACE rx 4096 >/dev/null 2>&1 ;
> > for i in rx tx sg tso ufo gso gro lro rxvlan txvlan; do /sbin/ethtool -K
> > $IFACE $i off >/dev/null 2>&1; done;
> >
> > /sbin/ethtool -N $IFACE rx-flow-hash udp4 sdfn >/dev/null 2>&1;
> > /sbin/ethtool -N $IFACE rx-flow-hash udp6 sdfn >/dev/null 2>&1;
> > /sbin/ethtool -C $IFACE rx-usecs 1 rx-frames 0 >/dev/null 2>&1;
> > /sbin/ethtool -C $IFACE adaptive-rx off >/dev/null 2>&1;
> >
> > exit 0
> >
> ----------------------------------------------------------------------------------------------------------------------------------------------------
> > HERES THE VM INTERFACE FILE:
> > idsadmin at SRVCHIPSSNSR01:~$ sudo cat /etc/network/interfaces
> > [sudo] password for idsadmin:
> > # This file describes the network interfaces available on your system
> > # and how to activate them. For more information, see interfaces(5).
> >
> > # The loopback network interface
> > auto lo
> > iface lo inet loopback
> >
> > # The primary network interface
> > auto eth0
> > iface eth0 inet static
> > address 192.168.5.5
> > netmask 255.255.255.0
> > gateway 192.168.5.1
> > dns-nameservers 192.168.5.1
> >
> > auto eth1
> > iface eth1 inet manual
> > pre-up modprobe 8021q
> > post-up ifconfig $IFACE up
> > post-up /etc/network/if-up.d/interfacetune
> > pre-down ifconfig $IFACE down
> >
> > auto eth2
> > iface eth2 inet manual
> > post-up ifconfig $IFACE up
> > post-up /etc/network/if-up.d/interfacetune
> > pre-down ifconfig $IFACE down
> >
> >
> > auto br0
> > iface br0 inet static
> > address 0.0.0.0
> > netmask 255.255.255.255
> > bridge_ports eth1 eth2
> > bridge_stp off
> > post-up ifconfig eth1 mtu 1520
> > post-up ifconfig eth2 mtu 1520
> > post-up ethtool -s eth2 autoneg off speed 1000 duplex full
> > post-up ethtool -s eth1 autoneg off speed 1000 duplex full
> > post-up /etc/network/if-up.d/interfacetune
> > post-down brctl delbr br0
> >
> -------------------------------------------------------------------------------------------------------------------------------
> > ** Notes-- I was scanning a 1 gigabit trunk interface between a CISCO
> 3750G
> > interface and a cisco 2911 ISR router with dot1q sub-interfaces.
> > If anyone cares. CISCO devices get pissed when you inject a linux bridge
> in
> > between it's CDP neighbor.
> > On your cisco parent interface configs, a ' no cdp enable ' is prudent
> for
> > the interfaces that are directly facing to the Linux bridge.
> > No need to set up individual subinterfaces on the bridge. The 'pre-up
> > modprobe 8021q' is all that's necessary to teach the bridge how
> > to pass vlan tags correctly.
> > And also hard code all speed and duplex settings or you're asking for a
> lot
> > of frustration and duplexing errors.
> >
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> > Last but not least, I also had HSRP (hot standby router protocol) which
> > uses multicast keepalives running between the router and the L3 switch.
> > I used these IPTABLES rules to shove the traffice toward NFQUEUE on the
> VM:
> >
> > Compile suricata on the guest... Do it with NFQUEUE flags.. That's a
> whole
> > different procedure in and of itself.
> > If anyone decides to do this I'd love to see if it works in the VM with
> the
> > directly copied attributes
> > of the CPU within the VM and compile with hyperscan on an INTEL cpu set.
> > I used Gen 3 OPTERONS in my rig so I couldn't utilize hyperscan.
> >
> ------------------------------------------------------------------------------------------------------------------------------------
> > sudo iptables -I FORWARD -m physdev --physdev-in eth1 -j NFQUEUE
> > --queue-balance 0:7
> > sudo iptables -I FORWARD -m physdev --physdev-in eth2 -j NFQUEUE
> > --queue-balance 0:7
> >
> > I started suricata like this:
> > sudo suricata -q 0 -q 1 -q 2 -q 3 -q 4 -q 5 -q 6 -q 7 -c
> > /homeidsadmin/suricata-3.0.1/suricata.yaml
> >
> > It works really well as far as I can tell. I'd post performance data
> too..
> > but I can only generate traffic with a few end user nodes as I'm not
> > running this in production. I'd like to see if anybody could post results
> > of high utilization environments with a similiar setup.
> >
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> >
> >
> >
> > On Sat, May 28, 2016 at 6:40 PM, Andreas Herz <andi at geekosphere.org
> <javascript:;>> wrote:
> >
> > > On 19/05/16 at 19:12, Chris Boley wrote:
> > > > I have been playing with using suricata ' inline ' using KVM/QEMU
> > > > <http://libvirt.org/drvqemu.html> by way of the libvirt toolkit.
> > > > I realize that the setups will vary wildly based on the hardware
> platform
> > > > capabilities. I'm wondering if anyone else here on the list could
> share
> > > > with me any experiences they've had on the networking I/O side of
> things
> > > > like tuning specifically for where it concerns suricata. For
> example, how
> > > > you have set up network configs on both the host systems and guest
> OS's
> > > to
> > > > get the best performance?
> > > > I've already got a config that's working, I'm just not sure it's the
> best
> > > > way to go about it.
> > >
> > > Can you share your config and experience?
> > >
> > > > If anybody can let me know I'd be really interested in getting that
> > > input.
> > > > Hopefully this is an appropriate topic for the list.
> > >
> > > Sure it is!
> > >
> > > > Thanks in advance,
> > > > Chris
> > >
> > > > _______________________________________________
> > > > Suricata IDS Users mailing list:
> oisf-users at openinfosecfoundation.org <javascript:;>
> > > > Site: http://suricata-ids.org | Support:
> > > http://suricata-ids.org/support/
> > > > List:
> > > https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> > > > Suricata User Conference November 9-11 in Washington, DC:
> > > http://oisfevents.net
> > >
> > >
> > > --
> > > Andreas Herz
> > > _______________________________________________
> > > Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> <javascript:;>
> > > Site: http://suricata-ids.org | Support:
> http://suricata-ids.org/support/
> > > List:
> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> > > Suricata User Conference November 9-11 in Washington, DC:
> > > http://oisfevents.net
>
> --
> Andreas Herz
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20160615/d6b569f0/attachment-0002.html>
More information about the Oisf-users
mailing list