Overview of XenServer Distributed Virtual Switch/Controller and Troubleshooting Network Issues

(1)

Overview of XenServer Distributed Virtual

Switch/Controller and

Troubleshooting Network Issues

Blaine A. Anaya

XenServer Escalation Engineer

05/24/2011

(2)

Overview

Agenda

XenServer Networking Architecture / vSwitch Architecture

Troubleshooting the Network

(3)

Overview

Agenda

XenServer Networking Architecture / vSwitch Architecture Troubleshooting the Network

Performance Testing

(4)

Networking Terminology

(5)

XenServer Networking Terminology

PIF- Physical Interface Object – directly correlates to a physical interface

VIF- Virtual Interface Object- directly correlates to a virtual interface in a VM Bridge- Represents a network and is where PIFs and VIFs are plugged in

Dom0- Short form of Domain 0 the control domain in XenServer that manages network and storage connections for virtual machines

Bond- is the association of two network interface cards to make them appear as one

Trunk – a switch port designated to carry traffic for more than one VLAN

(6)

XenServer Networking

DomU DomU DomU DomU

Xen Hypervisor Xen Hypervisor

Dom0 Dom0

Toolstack

Toolstack App App App App App App App App

Native Driver /

PIF Native Driver /

PIF

netfront/

VIF netfront/

VIF Guest OS

Guest OS Guest OS Guest OS

Bridge Bridge

Host Machine (Hardware) Host Machine (Hardware) Host Machine (Hardware) Host Machine (Hardware)

netback/0 netback/0

netback/1 netback/1 netback/2

netback/2

netback/3

(7)

XenServer Networking Configurations- Linux Stack

Linux NIC Drivers Linux NIC

Drivers Linux

Config Files Linux Config

Files XenServer Pool

DB

XenServer Pool DB

Network Card XAPIXAPI

Command Line

XenCenter

xsconsole

(8)

XenServer Network Terminology

Internal Switches

PIF (eth0)

VIF VIF VIF

Virtual Machine

Network 0 (xenbr0)

Private (xapi1)

Network Card

(9)

XenServer Network Terminology

Internal Switches

PIF (eth1) PIF (eth0)

VIF VIF VIF

Virtual Machine

Network 1 (xenbr1) Network 0

(xenbr0)

Network Card Network

Card

(10)

XenServer Network Terminology

PIF (bond0) PIF

VIF VIF

Virtual Machine

Card

VIF

Bond 0+1 (xapi2)

PIF (eth0)

PIF (eth1)

(11)

Bonding Type (Balance SLB)

Virtual Machine

Card

Virtual Machine

Bond

0:00 SEC 0:10 SEC 0:20 SEC 0:30 SEC

Stacked Switches

Virtual Machine

(12)

Distributed vSwitch

(13)

Open Virtual Switch for XenServer

VM

Hypervisor Hypervisor

VM VM VM VM

Visibility· Resource control · Isolation · Security

VM VM

• Open Source Virtual Switch maintained at www.openvswitch.org

• Rich layer 2 feature set (in contrast to others on the market)

• Ships with XenServer 5.6 FP1 as a post-install configuration option

(14)

Distributed Virtual Switch Controller

Hypervisor Hypervisor Hypervisor

Hypervisor

VM VM VM VM VM VM VM VM VM VM VM

DVS Controller is a XenServer

Virtual Appliance that controls

multiple Open vSwitches

(15)

Distributed Virtual Switch

Hypervisor Hypervisor Hypervisor

Hypervisor

Built-in policy-based ACLs move with VMs

DVS

VM

VM VM VM VM VM VM VM VM VM VM

Virtual Interface (VIF) {MAC, IP} ACLs

permit tcp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 192.168.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 172.16.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 192.168.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 172.16.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq 123

Virtual Interface (VIF) {MAC, IP} ACLs

permit tcp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 192.168.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 172.16.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 192.168.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 172.16.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq 123

(16)

Enabling the vSwitch

Distributed Virtual Switch

[root@vswitch1-baa-r222 ~]# xe-switch-network-backend openvswitch Cleaning up old ifcfg files

Remove... ifcfg-bond0 Remove... ifcfg-bond1 Remove... ifcfg-eth0 Remove... ifcfg-eth1 Remove... ifcfg-eth2 Remove... ifcfg-eth3 Remove... ifcfg-eth4 Remove... ifcfg-eth5 Remove... ifcfg-xapi2 Remove... ifcfg-xapi4 Remove... ifcfg-xenbr0 Remove... ifcfg-xenbr3

Enabling openvswitch daemon

Configure system for openvswitch networking You *MUST* now reboot your system

#xe-switch-network-backend openvswitch (Command must be ran on each individual host)

(17)

vSwitch Architecture – Process Level View

Distributed Virtual Switch DVS Controller

OVS

Flow Table

Flow Table Cache

vSwitch Network A

Flow Table

Flow Table Cache

vSwitch Network B ovsdb-server vswitchd

OpenFlow JSON-RPC

PIF PIF

VIF VIF VIF

VIF

(18)

XenServer Networking Configurations- vSwitch

Linux NIC Drivers Linux NIC

Drivers vSwitch

Config vSwitch

Config XenServer Pool

DB

XenServer Pool DB

Network Card XAPIXAPI

Command Line

XenCenter

xsconsole

(19)

DVSC Web Interface

(20)

Overview

Agenda

XenServer Networking Architecture / vSwitch Architecture

Troubleshooting the Network

(21)

Troubleshooting The Network

Symptoms Issue

• Intermittent Packet Loss/ Dropped Connections ^• Physical Connection/Switch Configuration, Bonding

• Physical Connection/Switch Configuration, Change in Hardware, Configuration Conflict.

• Network Appears Disconnected

• Bond Fails To Pass Traffic

When One Leg is Disconnected

• Physical Connection/Switch

Configuration, Bond Mode

(22)

• Using Command Line Interface (CLI)

• Off-line using a system status report

• BareGrep Pro

• Xenoscope

Troubleshooting The Network

(23)

Troubleshooting The Network

1.Check switch port configuration – Physical – Layers1-3 (Cables,NICs,Switch/Router connections) 2.Verify enabled network backend (Linux Bridge/vSwitch)

4.Use “brctl show” to see bridge/bond association.

3.Use ifconfig –a to see bonds, physical interface statistics, bridges.

5.Verify bonding configuration

6.Use ethtool for NIC settings, driver and firmware versions.

7.Use xe network-list, xe pif-list, to check XAPI configuration.

(24)

Troubleshooting the Network

(25)

Common Configuration Items to Check

Troubleshooting The Network

/etc/xensource/network.conf /etc/sysconfig/network-scripts /proc/net/bonding/bond0

/etc/sysconfig/iptables

(26)

Linux Bridge/vSwitch Enabled

Troubleshooting The Network

# brctl show

# Shows the bridges and the interfaces plugged into them

[root@vswitch1-baa-r222 ~]# brctl show

bridge name bridge id STP enabled interfaces xapi2 0000.001517868b8f no bond1

eth5 eth4 xapi4 0000.001d09699d86 no bond0 eth1 eth2 vif5.0 vif6.0 xenbr0 0000.001d09699d84 no eth0 xenbr3 0000.001517868b8c no eth3

(27)

Linux Bridge Enabled

Troubleshooting The Network

#brctl showmacs <brname>

#Shows a list of learned MAC addresses for this bridge.

[root@localhost ~]# brctl showmacs xenbr0

port no mac addr is local? ageing timer 1 00:00:0c:07:ac:3c no 1.83

1 00:0c:29:3a:12:79 no 120.59

1 00:0c:29:fa:8e:e8 no 26.52

(28)

Linux Bridge/vSwitch Enabled

Troubleshooting The Network

# netstat -np

# Provides information on connections and processes.

[root@vswitch1-baa-r222 ~]# netstat -np Active Internet connections (w/o servers)

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.0.1:37259 127.0.0.1:443 ESTABLISHED 2645/stunnel

tcp 0 0 127.0.0.1:36806 127.0.0.1:80 ESTABLISHED 6280/stunnel tcp 0 52 10.12.45.209:22 10.54.75.163:63296 ESTABLISHED 31145/5

tcp 0 0 127.0.0.1:443 127.0.0.1:37259 ESTABLISHED 6280/stunnel tcp 0 0 10.12.45.209:443 10.12.45.114:39105 ESTABLISHED 6280/stunnel

tcp 0 0 10.12.45.209:34969 10.12.45.194:6633 ESTABLISHED 5304/ovs-vswitchd

(29)

Linux Bridge/vSwitch Enabled

Troubleshooting The Network

# netstat -s

# Provides summary statistics for each protocol.

[root@vswitch1-baa-r222 ~]# netstat -s Ip:

17340461 total packets received 9190 with invalid addresses

0 forwarded

0 incoming packets discarded

12463755 incoming packets delivered 14230986 requests sent out

8 dropped because of missing route

Tcp:

69504 active connections openings 126760 passive connection openings 0 failed connection attempts

229 connection resets received 17 connections established

12462000 segments received 13220998 segments send out 3144 segments retransmited 0 bad segments received.

416 resets sent

(30)

Linux Bridge/vSwitch Enabled

Troubleshooting The Network

#ethtool –k <interface>

#Provides information on current offload settings

[root@vswitch1-baa-r222 ~]# ethtool -k eth0 Offload parameters for eth0:

rx-checksumming: on tx-checksumming: on scatter-gather: on

tcp-segmentation-offload: on udp-fragmentation-offload: off generic-segmentation-offload: on generic-receive-offload: off

large-receive-offload: off

(31)

Linux Bridge/vSwitch Enabled

Troubleshooting The Network

#ethtool –i <interface>

#Provides information on driver/firmware versions for network cards [root@vswitch1-baa-r222 ~]# ethtool -i eth0

driver: bnx2

version: 2.0.8e

firmware-version: bc 2.9.1

bus-info: 0000:04:00.0

(32)

vSwitch Enabled

Troubleshooting The Network

#ovs-appctl bond/list

#Shows Bridge, Bond, Slave Association

[root@vswitch1-baa-r222 ~]# ovs-appctl bond/list bridge bond slaves

Xapi2 bond1 eth4, eth5 Xapi4 bond0 eth2, eth1

Disclaimer: Using OVS command line options for configuration purposes is not supported. The vSwitch should only be configured using XenCenter, xe CLI,

xsconsole, and the Distributed vSwitch Controller.The commands shared here

are for data collection and diagnostic purposes only.

(33)

vSwitch Enabled

Troubleshooting The Network

#ovs-appctl bond/show bond0

#Shows bond members, up/down delay, and next rebalance time.

[root@vswitch1-baa-r222 ~]# ovs-appctl bond/show bond0 updelay: 31000 ms

downdelay: 200 ms

next rebalance: 4314 ms slave eth2: enabled

active slave

hash 123: 1 kB load

86:43:b2:1a:f2:d0

slave eth1: enabled

(34)

vSwitch Enabled

Troubleshooting The Network

#ovs-appctl fdb/show <bridge_name>

#Shows MAC Table/VLAN information for the bridge

[root@vswitch1-baa-r222 ~]# ovs-appctl fdb/show xapi4 port VLAN MAC Age

3 0 00:1d:09:2c:c4:c9 58

3 0 0a:34:ee:08:53:06 47

3 0 6a:e8:14:89:5c:af 42

3 0 ba:89:bf:f5:b8:ab 35

3 0 00:16:c8:d8:f1:11 27

(35)

vSwitch Enabled

Troubleshooting The Network

#ovs-ofctl dump-flows <bridge_name>

#Shows FlowTable – (ACLs applied from controller)

[root@vswitch1-baa-r222 ~]# ovs-ofctl dump-flows xapi4 | grep drop

May 02 15:49:07|00001|ofctl|INFO|connecting to unix:/var/run/openvswitch/xapi4.mgmt

cookie=0x0, duration_sec=171s, duration_nsec=25000000ns, table_id=1, priority=32763, n_packets=0, n_bytes=0, tcp,dl_dst=86:43:b2:1a:f2:d0,nw_dst=10.12.45.151,tp_src=80,actions=drop

cookie=0x0, duration_sec=171s, duration_nsec=25000000ns, table_id=1, priority=65529, n_packets=15, n_bytes=930, tcp,in_port=4,dl_src=86:43:b2:1a:f2:d0,nw_src=10.12.45.78,tp_dst=80^,actions=drop

(36)

vSwitch Enabled

Troubleshooting The Network

#ovs-dpctl dump-flows <bridge_name>

#Shows FlowCache – (ACLs applied from controller)

[root@vswitch1-baa-r222 ~]# ovs-dpctl dump-flows xapi4 | grep mac86:43

tunnel00000000:in_port0004:vlan65535:pcp0 mac86:43:b2:1a:f2:d0->00:00:0c:07:ac:3c type0800 proto6 tos0 ip10.12.45.78-

>69.147.112.160 port4284->80, packets:1, bytes:62, used:2.160s, actions:drop

(37)

vSwitch Enabled

Troubleshooting The Network

#ovs-appctl vlog/list

#Show current logging levels

[root@vswitch1-baa-r222 ~]# ovs-appctl vlog/list console syslog file

--- --- --- bridge EMER ERR INFO vswitchd EMER ERR INFO xenserver EMER ERR INFO ofproto EMER ERR INFO sflow EMER ERR INFO jsonrpc EMER ERR INFO fail_open EMER ERR INFO netflow EMER ERR INFO ovsdb_error EMER ERR INFO

(38)

vSwitch Enabled

Troubleshooting The Network

#vlog/set module[:facility[:level]]

#Modify vswitch logging level

Sets the logging level for module in facility to level:

•Module may be any valid module name (as displayed by the --list action on ovs-appctl(8)), or the special name ANY to set the logging levels for all modules.

• Facility may be syslog, console, or file to set the levels for logging to the system log, the console, or a file respectively, or ANY to set the logging levels for both facilities. If it is omitted, facility defaults to ANY.

Note: The log level for the file facility has no effect unless ovs-vswitchd was invoked with the --log-file option.

•Level must be one of emer, err, warn, info, or dbg, designating the minimum severity of a message for it to be

logged. If it is omitted, level defaults to dbg.

(39)

Status Report and BareGrepPro

Troubleshooting The Network – Off-Line

(40)

Status Report and Xenoscope

Troubleshooting The Network – Off-Line

(41)

Status Report and Xenoscope

Troubleshooting The Network – Off-Line

(42)

Status Report and Xenoscope

Troubleshooting The Network – Off-Line

(43)

• CTX127885 - Introduction to XenServer Networking

• CTX123489 - XenServer VLAN Networking

• CTX124421 - Understanding Network Interface Card Bonds in XenServer

• CTX127970 - Distributing Guest Traffic Over Physical CPUs in XenServer

• CTX127065- XenServer Virtual Machine Performance Utility

• CTX123477 - How to Move a XenServer Pool to a Different IP Subnet

• CTX125358 - How to Identify the Network Adapters on XenServer

• CTX101810 - Communication Ports Used By Citrix Technologies

Useful Networking CTX Articles

(44)

Q & A

(45)

Before you leave…

• Recommended related breakout sessions:

• Session: YN203: Managing VM networking across the datacenter with XenServer distributed virtual switching

Date: Wednesday May 25

^th

Time: 4:30-5:15

Room: Moscone 2003-2005

• Session surveys are available online at www.citrixsummit.com starting Thursday, May 26

• Provide your feedback and pick up a complimentary gift at the registration desk

• Download presentations starting Friday, June 3, from your My Organizer

Tool located in your My Synergy Microsite event account

(46)