Virtual Networking

The ability to manage virtual machines is something which is receiving a lot of focus right now. Xen, KVM, QEMU and others provide the infrastructure required to run a virtual machine, and each can provide guests with a virtual network interface. This proposal addresses the problem of how guests are networked together.

We aim:

User Visible Concepts

It's important to consider the manner in which we expose the functionality of virtual networking. What concepts will be exposing through the UI? Are those concepts well defined and consistent? Are those concepts more complex than neccessary? Or are the too simple to be able to support the functionality we want?

Real world, or "physical", concepts[1]:

With virtual networking, we will be exposing the following "virtual" concepts:

(Note, unprivileged users may create any of the above)

Finally, where the physical world meets the virtual world:

There are a few problems with all of the above:

  1. The distinction between a bridge and a router requires a lot of technical knowledge to fully understand. However, the model of e.g. a LinkSys router is familiar to a lot of people - a box which allows you to network your machines together and connect that network to (and firewall off) the Internet.
  2. This "shared physical interface" notion is very "makey upey". We could perhaps talk about the idea in terms of connecting a physical interface to a virtual bridge, but it exposes the bridge vs. router distinction more than we'd like.
  3. Guests are connected to a specific physical interface, whereas perhaps users wish guests to be connected to "the network" - i.e. if NetworkManager switched from wireless to wired while remaining on the same subnet, perhaps we'd like to automatically switch the bridge to the new network. In reality, though, bridged networking is only really sane for machines on a fairly static network connection.

[1] - Yes, these definitions aren't entirely accurate, but they describe the kind of understanding a moderately technical user might have of the concepts.

Example Networks

Below are some example networks users may configure and an explanation of how that network would be implemented in practice.

  1. A privileged user creates two (Xen) guests, each with a Virtual Network Interface. Without any special networking configuration, these two guests are connected to a default Virtual Network which contains a combined Virtual Bridge/Router/Firewall.

      +-----------+                   D           +-----------+
      |   Guest   |           N   D   H           |   Guest   |
      |     A     |           A   N   C           |     B     |
      |   +---+   |           T   S   P           |   +---+   |
      |   |NIC|   |           ^   ^   ^           |   |NIC|   |
      +---+-+-+---+           +---+---+           +---+-+-+---+
            ^                     |                     ^
            |   +--------+    +---+---+    +--------+   |
            +-->+ vif1.0 +----+ vnbr0 +----+ vif2.0 +<--+
                +--------+    +-------+    +--------+
    

    Notes:

  2. A privileged user does exactly the same thing as (1), but with QEMU guests.

                                      D
                              N   D   H
                              A   N   C
                              T   S   P
                              ^   ^   ^
                              +---+---+
                                  |
                              +---+---+
      +-----------+           | vnbr0 |           +-----------+
      |   Guest   |           +---+---+           |   Guest   |
      |     A     |               |               |     B     |
      |   +---+   |           +---+---+           |   +---+   |
      |   |NIC|   |           | vtap0 |           |   |NIC|   |
      +---+-+-+---+           +---+---+           +---+-+-+---+
            ^       +-------+     |     +-------+       ^
            |       |       | +---+---+ |       |       |
            +------>+ VLAN0 +-+  VDE  +-+ VLAN0 +<------+
                    |       | +-------+ |       |
                    +-------+           +-------+
    

    Notes:

  3. An unprivileged user does exactly the same thing as (2).

      +-----------+                               +-----------+
      |   Guest   |          +----+----+          |   Guest   |
      |     A     |          |userspace|          |     B     |
      |   +---+   |          | network |          |   +---+   |
      |   |NIC|   |          |  stack  |          |   |NIC|   |
      +---+-+-+---+          +----+----+          +---+-+-+---+
            ^       +-------+     |     +-------+       ^
            |       |       | +---+---+ |       |       |
            +------>+ VLAN0 +-+  VDE  +-+ VLAN0 +<------+
                    |       | +-------+ |       |
                    +-------+           +-------+
    

    Notes:

  4. Same as (2), except the user also creates two Xen guests.

      
      +-----------+                   D           +-----------+ 
      |   Guest   |           N   D   H           |   Guest   | 
      |     A     |           A   N   C           |     B     | 
      |   +---+   |           T   S   P           |   +---+   | 
      |   |NIC|   |           ^   ^   ^           |   |NIC|   | 
      +---+-+-+---+           +---+---+           +---+-+-+---+ 
            ^                     |                     ^       
            |   +--------+    +---+---+    +--------+   |       
            +-->+ vif1.0 +----+ vnbr0 +----+ vif2.0 +<--+       
                +--------+    +---+---+    +--------+           
                                  |                             
                              +---+---+                         
                              | vtap0 |                         
                              +---+---+                         
                                  |                             
                  +-------+    +--+--+   +-------+               
            +---->+ VLAN0 +----+ VDE +---+ VLAN0 +<-----+
            |     +-------+    +-----+   +-------+      |
            V                                           V 
      +---+-+-+---+                               +---+-+-+---+
      |   |NIC|   |                               |   |NIC|   |
      |   +---+   |                               |   +---+   |
      |   Guest   |                               |   Guest   |
      |     C     |                               |     D     |
      +-----------+                               +-----------+
    

    Notes:

  5. Same as (3) except Guests A and C are connected to a Shared Physical Interface.

     +-----------+                 |          D       +-----------+
     |   Guest   |          ^      |  N   D   H       |   Guest   |
     |     A     |          |      |  A   N   C       |     B     |
     |   +---+   |      +---+---+  |  T   S   P       |   +---+   |
     |   |NIC|   |      | eth0  |  |  ^   ^   ^       |   |NIC|   |
     +---+-+-+---+      +---+---+  |  +---+---+       +---+-+-+---+
           ^                |      |      |                 ^
           | +--------+ +---+---+  |  +---+---+ +--------+  |
           +>+ vif1.0 +-+ ebr0  +  |  + vnbr0 +-+ vif2.0 +<-+
             +--------+ +---+---+  |  +---+---+ +--------+  
                            |      |      |
                        +---+---+  |  +---+---+
                        | vtap1 |  |  | vtap0 |
                        +---+---+  |  +---+---+
                            |      |      |
              +-------+  +--+--+   |   +--+--+  +-------+
           +->+ VLAN0 +--+ VDE +   |   + VDE +--+ VLAN0 +<-+
           |  +-------+  +-----+   |   +-----+  +-------+  |
           V                       |                       V 
     +---+-+-+---+                 |                 +---+-+-+---+
     |   |NIC|   |                 |                 |   |NIC|   |
     |   +---+   |                 |                 |   +---+   |
     |   Guest   |                 |                 |   Guest   |
     |     C     |                 |                 |     D     |
     +-----------+                 |                 +-----------+
    

    Notes:

  6. Same as 2) except the QEMU guests are on a Virtual Network on another physical machine which is, in turn, connected to the Virtual Network on the first physical machine

      +-----------+                   D           +-----------+ 
      |   Guest   |           N   D   H           |   Guest   | 
      |     A     |           A   N   C           |     B     | 
      |   +---+   |           T   S   P           |   +---+   | 
      |   |NIC|   |           ^   ^   ^           |   |NIC|   | 
      +---+-+-+---+           +---+---+           +---+-+-+---+ 
            ^                     |                     ^       
            |   +--------+    +---+---+    +--------+   |       
            +-->+ vif1.0 +----+ vnbr0 +----+ vif2.0 +<--+       
                +--------+    +---+---+    +--------+           
                                  |                             
                              +---+---+
                              | vtap0 |
                              +---+---+
                                  |
                               +--+--+
                               | VDE |
                               +--+--+
                                  |
       First Physical Machine     V
       -------------------------------------------------------------
       Second Physical Machine    ^
                                  |
                  +-------+    +--+--+   +-------+
            +---->+ VLAN0 +----+ VDE +---+ VLAN0 +<-----+
            |     +-------+    +-----+   +-------+      |
            V                                           V 
      +---+-+-+---+                               +---+-+-+---+
      |   |NIC|   |                               |   |NIC|   |
      |   +---+   |                               |   +---+   |
      |   Guest   |                               |   Guest   |
      |     C     |                               |     D     |
      +-----------+                               +-----------+
    

    Notes:

One interesting thing to note from all of those examples is that although QEMU's networking options are very interesting, it doesn't actually make sense for a network to be implemented inside a guest. The network needs to be external to any guests, and so we use VDE to offer similar networking options to the ones QEMU provides. All QEMU needs to be able to do is to connect to VDE.

User Interface

This isn't meant a UI specification, but just some notes on how this stuff might be exposed in virt-manager.

Implementation

Parity with the current state of networking with Xen will be achieved by:

Virtual Networks will be implemented in libvirt. First, there will be an XML description of Virtual Networks e.g.:

  <network id="0">
    <name>Foo</name>
    <uuid>596a5d2171f48fb2e068e2386a5c413e</uuid>
    <listen address="172.31.0.5" port="1234" />
    <connections>
      <connection address="172.31.0.6" port="4321" />
    </conections>
    <dhcp enabled="true">
      <ip address="10.0.0.1" 
          netmask="255.255.255.0" 
          start="10.0.0.128"
          end="10.0.0.254" />
    </dhcp>
    <forwarding enabled="true">
      <incoming default="deny">
        <allow port="123" domain="foobar" destport="321" />
      </incoming>
      <outgoing default="allow">
        <deny port="25" />
      </outgoing>
    </forwarding>
  <network>

In a manner similar to libvirt's QEMU support, there will be a daemon to manage Virtual Networks. The daemon will have access to a store of network definitions. The daemon will be responsible for managing the bridge devices, vde_switch/dhcp/dnses processes and the iptables rules needed for SNAT/DNAT etc.

virsh command line interface would look like:

  $> virsh network-create foo.xml
  $> virsh network-dumpxml > foo.xml
  $> virsh network-define foo.xml
  $> virsh network-list
  $> virsh network-start Foo
  $> virsh network-stop Foo
  $> virsh network-restart Foo

The libvirt API for virtual networks would be modelled on the API for virtual machines:

/*
 * Virtual Networks API
 */

/**
 * virNetwork:
 *
 * a virNetwork is a private structure representing a virtual network.
 */
typedef struct _virNetwork virNetwork;

/**
 * virNetworkPtr:
 *
 * a virNetworkPtr is pointer to a virNetwork private structure, this is the
 * type used to reference a virtual network in the API.
 */
typedef virNetwork *virNetworkPtr;

/**
 * virNetworkCreateFlags:
 *
 * Flags OR'ed together to provide specific behaviour when creating a
 * Network.
 */
typedef enum {
    VIR_NETWORK_NONE = 0
} virNetworkCreateFlags;

/*
 * List active networks
 */
int			virConnectNumOfNetworks	(virConnectPtr conn);
int			virConnectListNetworks	(virConnectPtr conn,
						 int *ids,
						 int maxids);

/*
 * List inactive networks
 */
int			virConnectNumOfDefinedNetworks	(virConnectPtr conn);
int			virConnectListDefinedNetworks	(virConnectPtr conn,
							 const char **names,
							 int maxnames);

/*
 * Lookup network by name, id or uuid
 */
virNetworkPtr		virNetworkLookupByName	(virConnectPtr conn,
						 const char *name);
virNetworkPtr 		virNetworkLookupByID	(virConnectPtr conn,
						 int id);
virNetworkPtr 		virNetworkLookupByUUID	(virConnectPtr conn,
						 const unsigned char *uuid);
virNetworkPtr		virNetworkLookupByUUIDString	(virConnectPtr conn,
							 const char *uuid);

/*
 * Create active transient network
 */
virNetworkPtr		virNetworkCreateXML	(virConnectPtr conn,
						 const char *xmlDesc,
						 unsigned int flags);
/*
 * Define inactive persistent network
 */
virNetworkPtr		virNetworkDefineXML	(virConnectPtr conn,
						 const char *xmlDesc);

/*
 * Delete persistent network
 */
int			virNetworkUndefine	(virNetworkPtr network);

/*
 * Activate persistent network
 */
int			virNetworkCreate	(virNetworkPtr network);

/*
 * Network destroy/free
 */
int			virNetworkDestroy	(virNetworkPtr network);
int			virNetworkFree		(virNetworkPtr network);

/*
 * Network informations
 */
const char*		virNetworkGetName	(virNetworkPtr network);
unsigned int		virNetworkGetID		(virNetworkPtr network);
int			virNetworkGetUUID	(virNetworkPtr network,
						 unsigned char *uuid);
int			virNetworkGetUUIDString	(virNetworkPtr network,
						 char *buf);
char *			virNetworkGetXMLDesc	(virNetworkPtr network,
						 int flags);

Discussion points on the XML format and API:

Mark McLoughlin. Jan 15, 2007.