quickly install things with bash for loops on many boxes

for i in {6..8}; do ssh root@192.168.1.$i apt-get install -y nova-api-metadata=1:2013.1-0ubuntu2; done;

The above for loop will ssh to compute nodes 6,7 and 8 and install the nova-api-metadata package listed.


Quantum OpenVSwitch plugin and GRE tunnels

First off let me just say that I knew nothing about OpenVSwitch prior to using OpenStack. OpenVSwitch is essentially an open source piece of software that gives one the ability to create switches in software… and is OpenFlow compatible. So naturally it is a great candidate for the OpenStack project. In my last couple of blog posts, I didn’t know that much about how OpenVswitch uses GRE tunnels to achieve tenant segregation. For example I had no idea how virtual machines that existed in one vlan could communicate with a default gateway for their subnet that WASN’T in their vlan. I FINALLY understand how. It’s actually very clever, because the people who wrote the code for this plugin really leverage the functionality of OpenVSwitch by using OpenFlow rules manipulating vlan tags on frames when they come into a virtual bridge. Let me show you what I mean.

Essentially what you have on each compute node is two switches that exist in software and are created by the OpenVSwitch software. These exist on each of the compute nodes and on the network node. You have what’s called the integration bridge or br-int and the tunnel bridge or Br-Tun. The GRE tunnels terminate on the tunnel bridge. The VMs of course connect to the Br-int bridge. Both the br-tun and br-int switches are connected together with what’s called a patch port. (Figure 1.1)


Figure 1.1

So why is OpenVSwich so neat? I guess an example would probably be best to describe it.

As soon as the virtual machine is spun up and attached to a network, The virtual machine will attach to the br-int bridge on the compute node. It will also be put into a vlan on that br-int. HOWEVER the vlan tag is only used to separate traffic locally on the compute node. For example if you have a network that is created in one tenant, and the SAME network is created in another tenant. And you have a virtual machine on one network, and another virtual machine on the other network. AND they exist on the same compute node they WILL be in separate vlans. Ya, it’s a little crazy.

The thing that I couldn’t figure out in my last post is how a virtual machine gets to it’s default gateway when the gateway and the virtual machines aren’t in the same vlans. And the answer is a combination of GRE tunnels AND  OpenFlow rules on the br-tun bridge. Here’s a really high level view of how it works.

Virtual machine needs to go to it’s gateway

Assume the virtual machine already has an ARP entry for the gateway.

So the virtual machine starts sending out traffic tagged with the vlan that it is in on through br-int on the compute node.

Refer to the flow of traffic below. This flow is from the virtual machine to the default gateway for the virtual network that was created previously.


NOW when the traffic gets to br-tun, this is where the magic/sdn part happens. There is a flow rule on br-tun that basically says, anything from a particular vlan, slap a GRE key tag on the traffic that leaves the br-tun bridge. Figure 1.2 shows the packet capture with the GRE key appended and Figure 1.3 shows the actual rule in place on br-tun that makes it happen.


Figure 1.2


Figure 1.3 – Flow rules on br-tun

^If you want to see the above flows go to a compute node and issue the following command: ovs-ofctl dump-flows br-tun

The bottom two rules are what I am talking about. It’s very clever. And very powerful.

Now when the traffic ends up getting to br-tun on the network node. There is ANOTHER flow rule that does the opposite on br-tun on the network node. It says anything coming into the br-tun bridge with a GRE tag/key of X CHANGE the vlan tag to what is locally configured on the br-int. For example if the gateway exists on Vlan 5 but the traffic from the vm was tagged with vlan 2 when it left the vm, a flow rule will change that tag when it reaches the br-tun bridge on the network node. Then it will be forwarded like a normal switch out the patch port towards br-int with the new vlan tag.

So what I learned is OpenFlow rules have the ability to change VLAN tags, which is sort of an interesting concept. In my next post I’m going to talk about how OpenStack leverages network namespaces to make overlapping ip addresses between tenants possible.

Quantum GRE tunnels OVS

Refer to my latest post here: https://visualne.wordpress.com/ It will finally explain how this works at a high level.


Forget what I said in the last post about dnsmasq interfaces and gateways and vms being in the right vlan. I have the dnsmasq interface as well as the gateway address for the subnet in a completely different vlan then any interface dealing with the vm on the compute node. Refer to the output below.


^The .2 address serves up ip addresses on the subnet.^ The .1 address is the gateway for the subnet.

Screen shot 2013-02-09 at 3.16.05 PM

Notice how in the above screenshot, both of those interface are in vlan 5.

The screenshot below is all ports attached to the OVS on the compute node.  NONE of them are in vlan 5, yet if I ping from the vm on the compute node. And that VM is in vlan 4. I can still ping, and that interface is in vlan 5.


Quantum OVS, gre tunnel, vlan…….weirdness

Alright, this is a follow up to this post here. https://visualne.wordpress.com/2013/01/25/tapdhcp-interface-not-comming-up-after-restart-vms-wont-get-private-ip-addresses/

I used this guide here: http://docs.openstack.org/folsom/basic-install/content/basic-install_intro.html to install folsom.  All the quantum network node services are running on the controller.

I’m sure some of you out there are seeing your vms not getting private ip addresses, which I talked about in my previous post. You still need to bring up the interface that the dhcp server serves up ip addresses on. So look at that last post.

Now lets assume you have brought up the dhcp interface and your vms still don’t get ip addresses. This might be because the interface associated with your vm on your ovs agent isn’t in the same vlan as the interface that is serving ip addresses. Refer to the output below.


The above screenshot is all the ports on the ovs agent on the compute node where my virtual machine spun up. What I put a rectangle around is the the interface that is associated with my vm. You see how it says “tag 9”, that’s the vlan tag. If you don’t know anything about vlans just know that broadcast messages don’t leave vlans, unless there’s some kind of helper address or something, but don’t worry about that. Just know that broadcasts don’t leave vlans.

A dhcp discover message is a broadcast message. It’s only going to go to ports in the SAME vlan. This includes the port that the dhcp server listens on, on the network node. If that dhcp interface isn’t in the same vlan on ovs, you will never get an ip address. In order to fix it you have to put the vm interface in the same vlan as the dhcp interface. Refer to the output below.


The screenshot directly above is the interface that hands out ip addresses. You see how that’s tagged with ‘5’. So you have the vm in vlan 9 and the dhcp port for that subnet  not in the same vlan. That VM will never get an ip address if they aren’t in the same vlan. In order to fix it you have to put the VM in the same vlan as the dhcp interface.

In my case I have to go to the compute node where this vm exists and issue this command.

“ovs-vsctl set Port qvocb9c5dd9-f8 tag=5”

vnc into your vm and do an /etc/init.d/networking restart, and you should get an ip.

In order for me to figure out what interface was associated with the vm, I just went through and started capturing traffic on individual ports on the ovs agent. When I saw discovers, I knew that was my vm. Refer to the output below.


Tap/DHCP interface not coming up after restart. VMs won’t get private IP addresses

I’m running all my quantum processes on the controller. Something odd, I don’t know if this is a bug or what. When my controller restarted. The gateway interfaces for tenant subnets do not come up, also the interface that the dnsmasq process serves up ip addresses on, doesn’t come up either.

So in my case I have created a subnet in a tenant. And I set the gateway to When you do this it will create two tap interfaces. One will be the gateway and one will be an interface that the dnsmasq process serves up ip addresses on. When I restarted the controller. Those interfaces do NOT come back up. I had to bring them up manually with the command you will see below.


If you want to see what interfaces are up. Do an “ifconfig” at the command line.

If you want to see what interfaces exist, but aren’t up, do an “ifconfig -a” Again you might see the gateway and dhcp interface that you created…..but they aren’t up. You gotta bring them up yourself….