Category Archives: VLANs

Configuring LAG Groups between Dell 62xx Series Switches and ESXi 4.1

Okay, so we’ve already configured the basics on both our switches, and ESXi servers, now it’s time to configure the LAG groups, and vSwitches for each of our necessary purposes.

We’re going to configure one LAG group for each of the following:

  • Production network traffic for the VMs
  • iSCSI Traffic
  • Management and vMotion
  • We’re only going to be using one NIC for Fault Tolerance, so we’re not going to configure a LAG group for that.
Let’s start by first identifying which ports we’ll use on each switch, and for which purpose we’ll use each group. When we started we said we’ll by using vlan 2 for Management, vlan 3 for vMotion, vlan4 for Fault Tolerance, vlan 5 for iSCSI, and vlans 6 & 7 for various production VMs (also vlan 2 if you are going to virtualize the vCenter server, which we are).
So we’ll need a total for 3 LAG groups, two of which will be trunking more than one vlan. Let’s start by configuring the first LAG group. This one is going to be for the Management and vMotion purposes, we’ll need 1 port on each switch in the stack, so let’s use port 10 on both the first and second switch in the stack, start by doing the following:


  1. Open your connection to your switch stack
  2. switchstack> enable
  3. switchstack# config
  4. switchstack(config)# interface range ethernet 1/g10,2/g10
  5. switchstack(config-if)# channel-group 10 mode on
  6. switchstack(config-if)#exit
  7. switchstack(config)# interface port-channel 10
  8. switchstack(config-if-ch10)# spanning-tree portfast
  9. switchstack(config-if-ch10)# hashing-mode 6
  10. switchstack(config-if-ch10)# switchport mode trunk
  11. switchstack(config-if-ch10)# switchport trunk allowed vlan add 2-3
  12. switchstack(config-if-ch10)# exit
What we just did was build a new Link Aggregation Group, Added port 10 on both of the switches in the stack to the LAG group, enabled the port to transition to forwarding state right away, be enabling portfast, set the LAG group load balancing method to IP-Source-Destination (hashing-mode 6), and converted the LAG group to a trunk, and added vlans 2 & 3 to the trunk as tagged vlans on that trunk.
We’ll be doing the same thing for our next LAG, only we’re going to add some commands because this LAG will be handling iSCSI traffic. We’re going to use ports 11 on each switch for this next LAG group, start by entering the following:

 UPDATE: if you are configuring iSCSI for an Equal Logic Array, please see this post instead of configuring LAGs for you iSCSI traffic.

  1. switchstack(config)# interface range ethernet 1/g11,2/g11
  2. switchstack(config-if)# channel-group 11 mode on
  3. switchstack(config-if)#exit
  4. switchstack(config)# interface port-channel 10
  5. switchstack(config-if-ch11)# spanning-tree portfast
  6. switchstack(config-if-ch11)# hashing-mode 6
  7. switchstack(config-if-ch11)# switchport mode access
  8. switchstack(config-if-ch11)# switchport access vlan 5
  9. switchstack(config-if-ch11)# mtu 9216
  10. switchstack(config-if-ch11)# exit
What we’ve done here is pretty much what we did for the first lag, but we made this LAG an access port for only one vlan, instead of a trunk port for more than one. We also adjusted the mtu to support jumbo frames for the iSCSI traffic because that’s what this vlan is used for.
Our Final LAG group is going to contain three ports two on 1 switch, and just one port on the other, let’s start by:
  1. switchstack(config)# interface range ethernet 1/g12-1/g13,2/g12
  2. switchstack(config-if)# channel-group 12 mode on
  3. switchstack(config-if)#exit
  4. switchstack(config)# interface port-channel 12
  5. switchstack(config-if-ch12)# spanning-tree portfast
  6. switchstack(config-if-ch12)# hashing-mode 6
  7. switchstack(config-if-ch12)# switchport mode trunk
  8. switchstack(config-if-ch12)# switchport trunk allowed vlan add 2,6-7
  9. switchstack(config-if-ch12)# exit

Don’t forget to “copy run start” on you switch, you don’t wan’t to lose all that work you’ve just done! Okay, our first few LAGs are configured, time to setup our first ESXi server’s network configuration:

Now comes time to configure the networking on the first ESXi server. The first thing we’re going to do is setup the vSwitch that corresponds to the LAG group for the Management and vMotion vlans. Follow these steps:

  1. Log into your ESXi server using the vSphere Client.
  2. Click on the Configuration tab at the top.
  3. Click on “Networking” under the hardware section, in the left pane.
  4. We’re going to be adding a new vSwitch, so click on “Add Networking…” in the top right hand corner of the screen.
  5. Select the Option for “VMkernel”, because this vSwitch will be supporting non- Virtual Machine tasks, click Next.
  6. Select “Create New Virtual Switch” and then check two vmnics (make sure these two are plugged into port 10 on each switch) and then press “Next”.
  7. Give this network the label of “MGMT_Network” or whatever you’ve named vlan 2 on the switches, for VLAN ID, enter the value of “2”, Check the box labeled “use this port group for management traffic”, click “Next”.
  8. Assign an IP address and subnet mask that are within the subnet of vlan 2. Click Next.
  9. Click “Finish”.
  10. Find the newly created vSwitch and click on “Properties”.
  11. Click “Add” to add a new port group.
  12. Select “VMkernel” again, and then click “Next”.
  13. Give this port group a name of “vMotion”, and a VLAN ID of “3”, Check the box labeled “use this port group for VMotion”, click “Next”.
  14. Click Finish.
  15. Select the “vSwitch”, which should be the first item in the list when the Port Group window closes, click “Edit…”.
  16. Click on the “NIC Teaming” tab.
  17. Change the “Load Balancing:” setting to “Route based on IP hash”.
  18. Leave the defaults of “Link status only” and “Yes” for the middle two settings, and then change the setting “Failback:” to “No”.
  19. Verify that both vmnics are listed under the “Active Adapters” section.
  20. Close all of the windows.
What we’ve just done is this: We’ve created a vSwitch, added two NICs to it, both of which are plugged into the LAG on the switches, and we configured ip hashing as the method of balancing (which is the ONLY method you can use with a LAG group), and we disabled link failover on this vSwitch. We also created two Port Groups, assigned each a VLAN ID, and an IP address/subnet mask that match our existing vlans configured on the switches. We identified that these networks should be used for either management or vMotion, and gave them descriptive names that match the vlans on the switches.
We’ll repeat this process to creating new vSwitches 3 more times, here are the break downs:
  • iSCSI port group, two vmnics: both plugged into the ports that make up LAG 11 on the switches, assigned vlan 5, assigned the name “iSCSI” or whatever you named the vlan on the switch, assigned a IP address in that subnet, NIC teaming configuration exactly the same as the first vSwitch we configured.
  • Fault Tolerance port group, one vmnic: plugged into one of the switch ports configured as an access port on vlan 4, VLAN ID of 4, a name that matches the vlan name on the switches, check the box for “Fault Tolerance Logging”, and an ip address in the corresponding subnet, leave all of the NIC Teaming settings in their default states.
  • and finally a vSwitch that contains a port group for each of your production VM networks, Assign VLAN IDs to each, and plug them into the switch ports that make up your final LAG groups. Make sure the NIC Teaming settings match the example LAG group above. Don’t forgot to create a Port Group for MGMT traffic otherwise your vCenter server wont be able to communicate to the ESXi servers later.
That’s it, after it’s all configured on the ESXi side, it may take a reboot of the ESXi host when configuring and changing the Management port groups, it’s not supposed to require that, but sometimes it does, so if you reconfigure the management networks, and then lose the ability to ping or connect to it, reboot the system before you start other troubleshooting. Also you’re going to want to make sure all of your LAG groups came up properly on the switches you can use the following commands to test:
  • Show interfaces port-channel – this will display the status of all interfaces in all LAG groups
  • show interfaces switchport port-channel XX – This will display a list of all tagged or untagged vlans on this particular LAG group or Ethernet port
That’s it, we’re now ready to finish up our ESXi configurations, Install a VM to run vCenter, and configure our iSCSI storage.

Configuring a Dell 6248 Switch Stack for use with a EqualLogic PS4000E Storage Array

I’m going to be doing some write ups over the next few days pertaining to getting a small VMWare vSphere 4.1 installation set up. We’ll be using a pair of Dell 6248 Switches, configured in a stack, and a Dell EqualLogic PS4000E iSCSI Storage Array as our back end. In preparation for that I’m going to be going over our switch and network configuration in this post so that it’s clear as to how the network is configured.

We’ll have vlans for each of the following purposes:

  • Native vlan 1: we’ll use this as our isolated, un-trunked vlan for this switch, the vlan where unconfigured ports are placed by default. (vlan 1)
  • Management: things like DRACs, iLos, UPS management NICs, SAN  Management NICs, etc (vlan 2)
  • vMotion: Moving Virtual machines from one host to another host (vlan 3)
  • HA: VMWare Fault Tolerance (vlan 4)
  • iSCSI traffic (vlan 5)
  • and finally all vlans needed for the production virtual servers (vlans 6 & 7 )
As a perquisite, we’re going to be doing some basic setup of the switch stack, if you’re not setup the switches in a stack yet, please see this post.
Log into the switch and enter the following commands:
  1. switchstack> enable
  2. switchstack# config
  3. switchstack(config)# vlan database
  4. switchstack(config-vlan)# vlan 2-7
  5. switchstack(config-vlan)# exit
  6. switchstack(config)# interface vlan 2
  7. switchstack(config-if-vlan2)# name MGMT_VLAN
  8. switchstack(config-if-vlan2)# exit
  9. repeat steps 6-8 for each vlan, giving each a descriptive name
  10. switchstack(config)# spanning-tree mode rstp (assuming you are using rstp with your other switches in your network)
Now let’s configure some access ports for the MGMT Vlan devices to plug into, we’ll use the last 4 ports on each switch.
  1. switchstack(config)# interface range ethernet 1/g44-1/g48,2/g44-2/g48
  2. switchstack(config-if)# switchport mode access
  3. switchstack(config-if)# switchport access vlan 2
  4. switchstack(config-if)# spanning-tree portfast
  5. switchstack(config-if)# exit
We used spanning-tree portfast because we know these ports will be plugged into end devices, and we want them to come up instantly if the switch is rebooted, or a cable is unplugged and then plugged back in, we don’t want to wait for spanning tree to check for switching loops.

We’ll also need to define a few access ports for vlan 5, where we’ll be plugging in our pS4000E, follow the exact same steps we used above to configure vlan 2, but substitute vlan 5 for vlan 2, make sure you plug the ports 0 and 1 on the EqualLogic Controller Modules into the vlan 5 ports of your switch, and port 2 on your controller modules into the switch ports for vlan 2 (port 2 on the SAN controller module is strictly for management, and therefore should not be on the vlans used for iSCSI traffic). We’ll also need to enable jumbo frames on the on switch ports that will be moving iSCSI traffic, and disable unicast storm control. To do this enter the following commands:

  1. switchstack(config)# interface ethernet 1/g20
  2. switchstack(config-if-1/g20)# mtu 9216
  3. switchstack(config-if-1/g20)# no storm-control unicast
  4. switchstack(config-if-1/g20)# exit
  5. repeat steps 1 – 3 for each port that that connects to a storage array port (only 0 and 1, 2 is for management only)
Note: typically the mtu would be set to 9000, but when you run the “iSCSI enable” option on these switches it’s set to 9216, which is what I’ve chosen to implement here. I’ll update this post in the future if this turns out to be a problem with either the ESXi hosts or the EqualLogic SAN.

Also, I normally would not disable unicast storm-control, but when you enable a iSCSI optimization of the Dell Switches, they do this automatically when a EqualLogic SAN is detected on a port, If anyone has the explanation of why this happens please feel free to share it.

Finally we’ll also need to enable flow control at the switch level, to do this enter the following command:

  1. switchstack(config)# flowcontrol
We’re also going to place this switch into the MGMT_VLAN so that it’s management interface is on the same vlan as everything else we’re going to manage. Enter the following commands:
  1. switchstack(config)# IP Address vlan 2
  2. switchstack(config)# ip address x.x.x.x y.y.y.y
  3. switchstack(config)# ip default-gateway z.z.z.z
Where x.x.x.x is the IP address of your switch on the new vlan, y.y.y.y is your subnet mask, and z.z.z.z is your gateway on the mgmt_vlan.

That’s all of the configuration we’ll need at this point, we’ll now setup the EqualLogic San here, and later we’ll configure the switches for Link Aggregation Groups to handle the connections to our ESXi servers.

Configuring Dell 54xx Series and 62xx Series Switches – Cheat Sheet

In preparation for a few writes up that I’ll be posting up here soon I wanted to go through a some of the more common things you’d probably want to do with the Dell 54xx and 62xx series of switches. Why Dell switches? The foremost answer is cost, for around $1000.00 you can get yourself a nice 48 port managed gigabit switch with a great selection of functionality. Is it better then the equivalent Cisco? Not nessesarily, but it’s also a half of the cost.

One of the reasons I like these so much, is the fact that if you’re putting together a large order with Dell, you can get them to toss these in for hundreds off of the list price, and it’s a single point of support if you need to call dell for any SAN or Networking support.

So, What are some things you’d like to do with these? The more common things that I find myself configuring are:

  • VLANs
  • LAG Groups (Cisco calls these Ether-Channels)
  • Configuring Spanning Tree and Rapid Spanning Tree Protocols
  • Configuring PortFast

Assuming that you’ve already ran through the startup wizard, given the switch a username, password, IP address, Mask, and Gateway…

Configuring VLANs

  1. Telnet into the switch (or use the console)
  2. Type ” config ” to enter config mode
  3. The first order of business when configuring VLANs is to define which VLANs should be active on this switch. Type ” vlan database ” to enter the vlan configuration mode.
  4. Type ” vlan ## ” where ## is any number between 2 and 4094. Repeat this step for each VLAN that you’d like to define on this switch.

Once you’ve created the VLANs you can assign ports to each VLAN in one of two modes, Access or Trunk.

Access ports strip off the VLAN header from the TCP/IP packet and pass that traffic out that switch port without VLAN tags on it. Traffic coming into this switch port without a VLAN tag will have the VLAN tag added to it with the ID that the port is assigned to. The exception to this is VLAN 1 which is always untagged. Access ports are used to connect to Client computers, Printer, “dumb” switches that don’t understand VLANs or anything else that would not know what to do with a VLAN tag. Access ports can only be assigned to one VLAN at a time.

Trunk ports on the other hand can be assigned multiple VLANs at once. The reason for this is to allow you to pass traffic for each VLAN over single link to another switch. Trunk links don’t strip off the VLAN tag from the TCP/IP packets, and instead forward the packet in tact. This allows to receiver of the packet to identify what packets belong on what VLANs, any untagged packets are put onto VLAN 1.

To configure an Access Port:

  1. From the config# prompt type: ” interface ethernet g# ”  where the # the port number of the switch (to configure a range of ports at once type ” interface range ethernet g#-@ ”  where the # is the first port in the range, and @ is the last port in the range)
  2. From the config-if# prompt type: ” switchport mode access ” (this is the default, but also the way you switch back from being a trunk port)
  3. Fromt the config-if# prompt type: ” switchport access vlan # ”  Where # is the VLAN you want to assign to this port. To change back to the default setting type: ” switchport access vlan 1

To configure a Trunk Port:

  1. From the config# prompt type: ” interface ethernet g# ”  where the # the port number of the switch (to configure a range of ports at once type ” interface range ethernet g#-@ ”  where the # is the first port in the range, and @ is the last port in the range)
  2. From the config-if# prompt type: ” switchport mode trunk
  3. From the config-if# prompt type: ” switchport trunk allowed vlan add # ” where # is the VLAN number you want to add to the trunk. Repeat this step for each VLAN you want to add to the trunk. to remove a VLAN from a trunk type: ” switchport trunk allowed vlan remove #
  4. Make sure you configure both sides of a trunk link with the same VLANs or the traffic wont get passed between the switches.

To configure VLANs on LAG Groups (Ether channels) substitute ” interface port-channel # ” for ” interface ethernet g# ” in the above commands.

It may become necessary to change which VLAN is the un-tagged VLAN for a trunk link. If you need to do this you can use this command: ” switchport trunk native vlan XX ” where XX is the VLAN that should have its traffic passed WITHOUT VLAN tags (typically VLAN 1). You’ll have to perform this step on both ports that makeup the trunk link.

Configuring LAG Groups

LAG groups are a collection of links between switches, or server NIC teams where more than 1 ethernet cable can be combined into one logical connection. This allows for 2 gigabit ethernet cabled to make a 2 gigabit logical connection, or 4 cables to make a 4 gigabit logical connection. This is a great way to make a switching backbone, or to create multiple links between switches that can be used for load balancing instead of just redundancy.

  1. Telnet into the switch (or use the console)
  2. Type ” config ” to enter config mode
  3. Type ” interface ethernet g# ” or ” interface range ethernet g#-# ” to select one or a range of interfaces to add to the LAG group.
  4. Type ” channel-group # mode auto ” where # is the number of the LAG group you are adding the switch ports to, normally 1-8.

Verifying a LAG Group

  1. First make sure that the switch on the other end of the LAG group is also configured.
  2. To verify that the LAG is active and working properly type ” exit ” until you’re back at the config# prompt. Type ” show interfaces port-channel
  3. the output will tell you what ports are members of the LAG, and which ones are active.

Configuring Spanning Tree Protocol

Spanning Tree prevent loops in the switching fabric. Which is to say it keeps you from screwing things up (well not you, you never make mistakes, but it keeps co-workers from breaking things, yeah we’ll go with that). If you were to plug a switch port back into the same switch, or if you were to connect two switches together with more than 1 cable (and not create a LAG group) you’d quickly figure out that your switching fabric has become completely saturated with broadcasts, as you switches pass broadcasts back and forth between each other.

Spanning Tree is a global feature that’s turned on at the switch level and not the port level. You turn on Spanning Tree by performing the following:

  1. Telnet into the switch (or use the console)
  2. Type ” config ” to enter config mode
  3. Type ” spanning-tree
  4. That’s it. That’s all there is to it, but spanning tree takes quite awhile for the database to converge, so in more modern switches you can also configure Rapid Spanning Tree protocol which is done by typing in ” spanning-tree mode rstp
  5. If you have an environment that contains switches or devices that don’t understand Rapid Spanning Tree, but do understand Spanning Tree you can run both side by side by entering ” spanning-tree mode mstp ” the Spanning Tree type used will depend on the type being used by the neighboring device.

Configuring PortFast

PortFast is a way of speeding up Spanning Tree on interface ports that are KNOWN to not be connected to anything other than a end device, such as a Printer, Server, Workstation, etc. Spanning Tree takes some time to figure out where all of the loops are in the switching fabric, this isn’t a problem if the switching network has been up and running for awhile, or if a single switch is rebooted, but let’s say your entire switching network lost power all at once and were rebooted, the Spanning Tree protocol could take up to 2 minutes or more to fully converge and during this time no one would be able to send traffic over the switches. PortFast is a way to tell the switches that we KNOW nothing on this port will cause a loop, it’s okay to allow it to pass traffic without Spanning tree being fully converged.

To enable PortFast on an interface:

  1. Telnet into the switch (or use the console)
  2. Type ” config ” to enter config mode
  3. Type ” interface ethernet g# " to enter the config-if# prompt
  4. Type “ spanning-tree portfast ” To disable portfast type ” no spanning-tree portfast