[Oisf-users] Suricata under libvirt
Andreas Herz
andi at geekosphere.org
Wed Jun 15 19:21:33 UTC 2016
On 31/05/16 at 09:02, Chris Boley wrote:
> Andreas, this is lengthy sorry, I tried to include useful detail but,
> unfortunately I ramble on a lot:
Thanks for the detailed information. What could be interesting, that you
put that in our redmine wiki so it won't get lost.
Looks like a very special usecase/setup, thus not that many responses,
but still worth it to be documented.
>
> I had this idea because I wanted to be able to have two or 3 working
> Suricata test OS's for experimenting with things in the YAML and
> other configs. This allows me to set up multiple VM's with varying configs
> and find out what works and what doesn't.
> That's why I did it.
>
> My basic setup.
> I had 4 ethernet adapters installed in a dell 2970 with latest bios, 2 - 6
> Core opterons running at 2.6 ghz, and 32 gigs of RAM.
>
> Why 4 nics? 1. host interface 2. VM Mgmt. Interface 3. Bridge-side A 4.
> Bridge-Side B.
>
>
> If your physical host interfaces are like em1 em2 or whatever in Ubuntu
> edit /etc/default/grub
>
> find these two lines:
> GRUB_CMDLINE_LINUX_DEFAULT="splash quiet"
> GRUB_CMDLINE_LINUX=""
>
>
> modify to look like
>
> GRUB_CMDLINE_LINUX_DEFAULT="splash quiet biosdevname=0"
> GRUB_CMDLINE_LINUX="biosdevname=0"
>
> sudo update-grub
>
> Modify your operational host interface in /etc/network/interfaces file to
> reflect eth0 instead of whatever it was named before.
> If you don't, you'll find yourself without a working interface.
>
> then reboot
>
> ---------------------------------------------------------------------------------------------
>
> **macvtap with passthrough must have a dedicated physical interface to tie
> itself to.
>
>
> You've got to create a specialized network configuration within the libvirt
> setup.
> ** Caveats** You need to have hardware and BIOS platform that supports
> SR-IOV.
> How do you figure that out? Do the homework. Or simply experiment. I just
> tried building the VM initially via virt-manager.
> It let me do it. Meaning, the toolkit detected the right resources to
> accomplish building the parameters I gave it.
>
> When I looked up SR-IOV capabilities online through DELL, I didn't see
> where it was supported but I tried it anyway and it worked.
> If it doesn't libvirt will just bark at you and tell you that you can't do
> that. Here's a good link to explain SR-IOV:
>
>
> http://blog.scottlowe.org/2009/12/02/what-is-sr-iov/
>
> ---A note about command line versus CLI installs
> For those of you who are adamant about staying on ssh cli access only with
> the host;
> this script below will startup a virtual machine net install for ubuntu
> 14.04 64 bit in your terminal with full hardware acceleration.
> The machine will initially have a NAT based virtual network interface, 16
> gig of ram and 8 vcpu's.
> You'll be dealing with a qcow2 based harddisk file which doesn't offer the
> best performance.
> but I like it because of being able to snapshot. Look up libvirt snapshot +
> qcow2 and you'll find some good articles on snappshotting.
> I'm not going to write a book here. :)
>
> I toyed with setting up the interfaces right from this install script. In
> the end and 4 hours later, it ended up being
> more trouble than it was worth. I used virt-manager from a xwindows GUI in
> lieu of all the headache.
>
> Plus, there are adjustments you can make in virt-manager that I've never
> figured out how to do in virsh.
> My most sincere recommendation is to install a really lightweight window
> manager.
> I tend to install Lubuntu minimal desktop during my server install and add
> in some basics like leafpad but most importantly virt-manager.
> Use whatever desktop you want. Maybe blackbox or fluxbox would be a great
> choice as they use almost no resources.
> You'll find manipulating VM's with Virt manager is much easier.
> Moreover, you can tailor your cpu's settings more easily which is kind of
> important for Suricata.
>
> Alternatively, you can stick to the cli and modify the xml with 'virsh
> edit'. Beware that it uses 'vi' and is kind of a PITA.
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> sudo apt-get -y install qemu-kvm libvirt-bin bridge-utils qemu-system
> & virt-manager if you're going to install a GUI....
>
> qemu-img create -f qcow2 -o preallocation=full
> /home/ipsadmin/vmimgs/virtdrive.qcow2 64G
>
> virt-install --connect=qemu:///system \
> --name=IPSTEST \
> --ram 16384 \
> --disk
> path=/home/ipsadmin/vmimgs/virtdrive.qcow2,format=qcow2,bus=virtio,cache=none,size=64
> \
> --vcpus=8 \
> --os-type linux \
> --os-variant ubuntutrusty \
> --network bridge=virbr0 \
> --check-cpu \
> --hvm \
> --location '
> http://archive.ubuntu.com/ubuntu/dists/trusty/main/installer-amd64/' \
> --graphics none \
> --console pty,target_type=serial \
> --extra-args="console=ttyS0,115200n8 serial" \
>
>
> ***add ' --debug ' flag into that script if you want to see verbose output
> of what's going on.
>
> ---- for the second time you need to connect to the VM.-----
>
> The first time, based on the above settings, it will connect automatically.
>
> virsh start IPSTEST
>
> virsh console IPSTEST
>
>
> -----------------------------------------------------------------------------------------------------------------------------------
> ----for editing the machine xml if you want to stick to the CLI. ** This is
> an example** modify to fit your needs.
> Find the adapter and paste in the place of the original adapter segment,
> something that looks like this below.
> Obviously you need 3 adapters for the VM 2 for bridge and one for mgmt
> iface. Be mindful of slot numbers and alias naming nomenclature.
> Logically in the xml you'll find that most things like slot number
> increment. Just make sure you don't conflict with something else.
> You'll break the VM in that case. Make a backup before you edit. Eth1 in
> the case below will take the place of what used to be
>
> your virtual adapter that was natted from inside to outside the host. Now
> you've tied it to a physical interface. It will pull DHCP
>
> addresses from the physical network.
>
>
> <interface type='direct'>
> <mac address='52:54:00:67:7e:5d'/>
> <source dev='eth1' mode='passthrough'/>
> <target dev='macvtap0'/>
> <model type='virtio'/>
> <alias name='net0'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> function='0x0'/>
> </interface>
> <interface type='direct'>
> <mac address='52:54:00:67:7e:5d'/>
> <source dev='eth2' mode='passthrough'/>
> <target dev='macvtap0'/>
> <model type='virtio'/>
> <alias name='net1'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> function='0x0'/>
> </interface>
> <interface type='direct'>
> <mac address='52:54:00:67:7e:5d'/>
> <source dev='eth2' mode='passthrough'/>
> <target dev='macvtap0'/>
> <model type='virtio'/>
> <alias name='net2'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x04'
> function='0x0'/>
> </interface>
> ------------------------------------------------------------------------------------------------------------------------------------------
>
> My full xml off my primary test machine looks like this:
>
> idsadmin at SRVCHSURICATA1:~$ virsh dumpxml CSNIPESNSR01
> <domain type='kvm' id='2'>
> <name>CSNIPESNSR01</name>
> <uuid>ec0d0563-aa0a-ef29-5f24-ba5ba1416b50</uuid>
> <memory unit='KiB'>16777216</memory>
> <currentMemory unit='KiB'>16777216</currentMemory>
> <vcpu placement='static'>8</vcpu>
> <resource>
> <partition>/machine</partition>
> </resource>
> <os>
> <type arch='i686' machine='pc-i440fx-trusty'>hvm</type>
> <boot dev='hd'/>
> </os>
> <features>
> <acpi/>
> <apic/>
> <pae/>
> </features>
> <cpu mode='custom' match='exact'>
> <model fallback='allow'>Opteron_G3</model>
> <vendor>AMD</vendor>
> <feature policy='require' name='skinit'/>
> <feature policy='require' name='vme'/>
> <feature policy='require' name='mmxext'/>
> <feature policy='require' name='fxsr_opt'/>
> <feature policy='require' name='cr8legacy'/>
> <feature policy='require' name='ht'/>
> <feature policy='require' name='3dnowprefetch'/>
> <feature policy='require' name='3dnowext'/>
> <feature policy='require' name='wdt'/>
> <feature policy='require' name='extapic'/>
> <feature policy='require' name='pdpe1gb'/>
> <feature policy='require' name='osvw'/>
> <feature policy='require' name='ibs'/>
> <feature policy='require' name='cmp_legacy'/>
> <feature policy='require' name='3dnow'/>
> </cpu>
> <clock offset='utc'/>
> <on_poweroff>destroy</on_poweroff>
> <on_reboot>restart</on_reboot>
> <on_crash>restart</on_crash>
> <devices>
> <emulator>/usr/bin/kvm-spice</emulator>
> <disk type='file' device='disk'>
> <driver name='qemu' type='qcow2'/>
> <source file='/home/idsadmin/CSNIPE/snipedrive.qcow2'/>
> <target dev='vda' bus='virtio'/>
> <serial>1</serial>
> <alias name='virtio-disk0'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x05'
> function='0x0'/>
> </disk>
> <disk type='file' device='cdrom'>
> <driver name='qemu' type='raw'/>
> <target dev='hdc' bus='ide'/>
> <readonly/>
> <alias name='ide0-1-0'/>
> <address type='drive' controller='0' bus='1' target='0' unit='0'/>
> </disk>
> <controller type='usb' index='0'>
> <alias name='usb0'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> function='0x2'/>
> </controller>
> <controller type='pci' index='0' model='pci-root'>
> <alias name='pci.0'/>
> </controller>
> <controller type='ide' index='0'>
> <alias name='ide0'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> function='0x1'/>
> </controller>
> <interface type='direct'>
> <mac address='52:54:00:b1:16:c2'/>
> <source dev='eth3' mode='passthrough'/>
> <target dev='macvtap0'/>
> <model type='virtio'/>
> <alias name='net0'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> function='0x0'/>
> </interface>
> <interface type='direct'>
> <mac address='52:54:00:3e:db:eb'/>
> <source dev='eth0' mode='passthrough'/>
> <target dev='macvtap1'/>
> <model type='virtio'/>
> <alias name='net1'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
> function='0x0'/>
> </interface>
> <interface type='direct'>
> <mac address='52:54:00:1c:71:70'/>
> <source dev='eth1' mode='passthrough'/>
> <target dev='macvtap2'/>
> <model type='virtio'/>
> <alias name='net2'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x08'
> function='0x0'/>
> </interface>
> <serial type='pty'>
> <source path='/dev/pts/2'/>
> <target port='0'/>
> <alias name='serial0'/>
> </serial>
> <console type='pty' tty='/dev/pts/2'>
> <source path='/dev/pts/2'/>
> <target type='serial' port='0'/>
> <alias name='serial0'/>
> </console>
> <input type='mouse' bus='ps2'/>
> <input type='keyboard' bus='ps2'/>
> <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'>
> <listen type='address' address='127.0.0.1'/>
> </graphics>
> <sound model='ich6'>
> <alias name='sound0'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x04'
> function='0x0'/>
> </sound>
> <video>
> <model type='cirrus' vram='9216' heads='1'/>
> <alias name='video0'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> function='0x0'/>
> </video>
> <memballoon model='virtio'>
> <alias name='balloon0'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
> function='0x0'/>
> </memballoon>
> </devices>
> <seclabel type='dynamic' model='apparmor' relabel='yes'>
> <label>libvirt-ec0d0563-aa0a-ef29-5f24-ba5ba1416b50</label>
> <imagelabel>libvirt-ec0d0563-aa0a-ef29-5f24-ba5ba1416b50</imagelabel>
> </seclabel>
> </domain>
> ----------------------------------------------------------------------------------------------------------------------------
>
> I set up my interfaces on the host like what's below here. I used my eth0
> and eth1 on the VM Bridge. Eth2 is my host interface.
> Eth3 is the 3rd interface on the VM dedicated to mgmt interface on the VM.
> This way you can SSH into the VM and manage it.
>
> The ' interfacetune' file listed there in the interfaces script is a page
> torn directly out of Peter Manev's Github page on how to
> tune interfaces best for Suricata use. (thanks again Peter. Your shared
> info is really awesome!) See what's in the script below.
> Physical Host interfaces file:
>
> # The loopback network interface
> auto lo
> iface lo inet loopback
>
> auto eth2
> iface eth2 inet static
> address 192.168.5.6
> netmask 255.255.255.0
> gateway 192.168.5.1
> dns-nameservers 192.168.5.1
>
> auto eth3
> iface eth3 inet manual
> pre-up modprobe 8021q
> post-up ifconfig $IFACE up
> pre-down ifconfig $IFACE down
>
> auto eth0
> iface eth0 inet manual
> post-up ifconfig $IFACE up
> post-up ifconfig eth0 mtu 1520
> post-up /etc/network/if-up.d/interfacetune
> post-up ethtool -s eth0 autoneg off speed 1000 duplex full
> pre-down ifconfig $IFACE down
>
> auto eth1
> iface eth1 inet manual
> post-up ifconfig $IFACE up
> post-up ifconfig eth1 mtu 1520
> post-up /etc/network/if-up.d/interfacetune
> post-up ethtool -s eth1 autoneg off speed 1000 duplex full
> pre-down ifconfig $IFACE down
>
>
> ---------------------------------------------------------------------------------------------------------------------
>
>
> idsadmin at SRVCHSURICATA1:~$ sudo cat /etc/network/if-up.d/interfacetune
> /sbin/ethtool -G $IFACE rx 4096 >/dev/null 2>&1 ;
> for i in rx tx sg tso ufo gso gro lro rxvlan txvlan; do /sbin/ethtool -K
> $IFACE $i off >/dev/null 2>&1; done;
>
> /sbin/ethtool -N $IFACE rx-flow-hash udp4 sdfn >/dev/null 2>&1;
> /sbin/ethtool -N $IFACE rx-flow-hash udp6 sdfn >/dev/null 2>&1;
> /sbin/ethtool -C $IFACE rx-usecs 1 rx-frames 0 >/dev/null 2>&1;
> /sbin/ethtool -C $IFACE adaptive-rx off >/dev/null 2>&1;
>
> exit 0
> ----------------------------------------------------------------------------------------------------------------------------------------------------
> HERES THE VM INTERFACE FILE:
> idsadmin at SRVCHIPSSNSR01:~$ sudo cat /etc/network/interfaces
> [sudo] password for idsadmin:
> # This file describes the network interfaces available on your system
> # and how to activate them. For more information, see interfaces(5).
>
> # The loopback network interface
> auto lo
> iface lo inet loopback
>
> # The primary network interface
> auto eth0
> iface eth0 inet static
> address 192.168.5.5
> netmask 255.255.255.0
> gateway 192.168.5.1
> dns-nameservers 192.168.5.1
>
> auto eth1
> iface eth1 inet manual
> pre-up modprobe 8021q
> post-up ifconfig $IFACE up
> post-up /etc/network/if-up.d/interfacetune
> pre-down ifconfig $IFACE down
>
> auto eth2
> iface eth2 inet manual
> post-up ifconfig $IFACE up
> post-up /etc/network/if-up.d/interfacetune
> pre-down ifconfig $IFACE down
>
>
> auto br0
> iface br0 inet static
> address 0.0.0.0
> netmask 255.255.255.255
> bridge_ports eth1 eth2
> bridge_stp off
> post-up ifconfig eth1 mtu 1520
> post-up ifconfig eth2 mtu 1520
> post-up ethtool -s eth2 autoneg off speed 1000 duplex full
> post-up ethtool -s eth1 autoneg off speed 1000 duplex full
> post-up /etc/network/if-up.d/interfacetune
> post-down brctl delbr br0
> -------------------------------------------------------------------------------------------------------------------------------
> ** Notes-- I was scanning a 1 gigabit trunk interface between a CISCO 3750G
> interface and a cisco 2911 ISR router with dot1q sub-interfaces.
> If anyone cares. CISCO devices get pissed when you inject a linux bridge in
> between it's CDP neighbor.
> On your cisco parent interface configs, a ' no cdp enable ' is prudent for
> the interfaces that are directly facing to the Linux bridge.
> No need to set up individual subinterfaces on the bridge. The 'pre-up
> modprobe 8021q' is all that's necessary to teach the bridge how
> to pass vlan tags correctly.
> And also hard code all speed and duplex settings or you're asking for a lot
> of frustration and duplexing errors.
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> Last but not least, I also had HSRP (hot standby router protocol) which
> uses multicast keepalives running between the router and the L3 switch.
> I used these IPTABLES rules to shove the traffice toward NFQUEUE on the VM:
>
> Compile suricata on the guest... Do it with NFQUEUE flags.. That's a whole
> different procedure in and of itself.
> If anyone decides to do this I'd love to see if it works in the VM with the
> directly copied attributes
> of the CPU within the VM and compile with hyperscan on an INTEL cpu set.
> I used Gen 3 OPTERONS in my rig so I couldn't utilize hyperscan.
> ------------------------------------------------------------------------------------------------------------------------------------
> sudo iptables -I FORWARD -m physdev --physdev-in eth1 -j NFQUEUE
> --queue-balance 0:7
> sudo iptables -I FORWARD -m physdev --physdev-in eth2 -j NFQUEUE
> --queue-balance 0:7
>
> I started suricata like this:
> sudo suricata -q 0 -q 1 -q 2 -q 3 -q 4 -q 5 -q 6 -q 7 -c
> /homeidsadmin/suricata-3.0.1/suricata.yaml
>
> It works really well as far as I can tell. I'd post performance data too..
> but I can only generate traffic with a few end user nodes as I'm not
> running this in production. I'd like to see if anybody could post results
> of high utilization environments with a similiar setup.
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>
>
> On Sat, May 28, 2016 at 6:40 PM, Andreas Herz <andi at geekosphere.org> wrote:
>
> > On 19/05/16 at 19:12, Chris Boley wrote:
> > > I have been playing with using suricata ' inline ' using KVM/QEMU
> > > <http://libvirt.org/drvqemu.html> by way of the libvirt toolkit.
> > > I realize that the setups will vary wildly based on the hardware platform
> > > capabilities. I'm wondering if anyone else here on the list could share
> > > with me any experiences they've had on the networking I/O side of things
> > > like tuning specifically for where it concerns suricata. For example, how
> > > you have set up network configs on both the host systems and guest OS's
> > to
> > > get the best performance?
> > > I've already got a config that's working, I'm just not sure it's the best
> > > way to go about it.
> >
> > Can you share your config and experience?
> >
> > > If anybody can let me know I'd be really interested in getting that
> > input.
> > > Hopefully this is an appropriate topic for the list.
> >
> > Sure it is!
> >
> > > Thanks in advance,
> > > Chris
> >
> > > _______________________________________________
> > > Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> > > Site: http://suricata-ids.org | Support:
> > http://suricata-ids.org/support/
> > > List:
> > https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> > > Suricata User Conference November 9-11 in Washington, DC:
> > http://oisfevents.net
> >
> >
> > --
> > Andreas Herz
> > _______________________________________________
> > Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> > Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> > List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> > Suricata User Conference November 9-11 in Washington, DC:
> > http://oisfevents.net
--
Andreas Herz
More information about the Oisf-users
mailing list