In a past post I went into how to configure iSCSI over a LAG to give you some path redundancy over a single VMK IP. You can read about that here. For multiple reasons this is not the best way to configure Multipathing, so here is a write up on the proper way to setup the Multipathing Plugin on a VMWare ESXi 5 server (I’ve also included steps to undo what may have been setup in the past).
- Download and install winSCP from here.
- Download the EqualLogic Multipathing Agent for VMWare.
- Download, Install, and Configure the VMWare Management Agent (vMA), read about how to do that here.
- Optionally, Install VMware Update Manager, which can be used to install the MEM in the event that the
setup.pl --install script does not work.
If you’ve already had iSCSI configured on this host, it’s time to make note of a few things and then clean up before we get the EqualLogic MEM installed.
- Make note of all IPs that are being used by a host for iSCSI
- Make note of which NICs are being used by the vSwitch setup for iSCSI
- Delete the VM Kernel ports that are attached to the iSCSI vSwitch
- Delete the iSCSI vSwitch
Disable iSCSI on the Host
- Connect to the vMA using putty, and then attach to your host using the following command:
vifptarget -s <host's FQDN>
- For ESXi 4.x enter the following command:
- for ESXi 5 enter the following command:
esxcli iscsi software set -e false
- Reboot the Host
Enable iSCSI on the Host
- For ESXi 4.x enter the following command:
- for ESXi 5.0 enter the following command:
esxcli iscsi software set -e true</li>
Remove the old VMK bindings from the iSCSI HBA
For each of the VM Kernel ports that you made note of before, run the following command where <vmk_interface> is your vmk port such as vmk1, vmk2, and <vmhba_device> is your vmhba adapter for iSCSI such as vmhba38:
- For ESXi 4.x:
esxcli swiscsi nic remove –n <vmk_interface> –d <vmhba_device>
- for ESXi 5:
esxcli iscsi networkportal remove -n <vmk_interface> -A <vmhba_device>
Installing the EqualLogic Multipathing Agent
Now that our host is fresh and so clean clean, well in terms of iSCSI anyway, it’s time to start configuring the Multipathing Extension Module.
Move the Setup Script and Bundle to the vMA
- Connect to your vMA using winSCP, it should drop you into the home directory for the user ‘vi-admin’
- Find and locate the files that were extracted from the zip file you downloaded from Equal Logic, you are looking for “setup.pl” and “dell-eql-mem-esx5-X-X.X.XXXXXX.zip” the version of the .zip file will depend on whether or not you’re installing it on ESXi 4.x or ESXi 5, just make sure you copy the right file name.
- Once you’ve moved both files to the vMA, right click on the “setup.pl” file from within winSCP, select “properties”. Under the “Permissions” section of the setup.pl change the “Octal” value to “0777”, this will allow you to execute the script.
- Close WinSCP.
Configuring the MEM
- Connect to your vMA using ssh.
- You should automatically be logged into the home directory of the ‘vi-admin’ user, verify this by running a ls, and making sure you see the two files you uploaded.
- enter the following command to get started:
./setup.pl --configure --server=<esxi server's FQDN>
- Follow the bouncing ball once the script gets started, it’s going to ask you for a username and password for the host, it’s also going to ask you to name the new virtual switch, it’s going to ask you what nics to use, list each one with a space in between them, it will also ask you for an IP for each VMK port it creates, and it will ask for the IP of the Group IP you want to connect to, and a few other questions as well such as subnet mask and mtu size, whether or not to use chap, use the information you collected above and the configuration of the Array to answer the questions, and when the script completes you should see the new vSwitch and VMK ports in your configuration.
Installing the Bundle
- While still logged into your vMA run the following command: ./setup.pl –install –server=<esxi server’s FQDN>
- If you receive an error about being unable to install it, try disabling Admission Control on your HA cluster and re-running the command.
If for some reason you are unable to get the setup.pl –install command to work properly you can use the vmware Update Manager to install the Bundle.
- Install and configure vUM, according to VMware instructions.
- Import the MEM offline bundle into the vUM package repository by selecting the “Import Patches” option and browsing to the dell-eql-mem-esxn-version.zip.
- Create a baseline containing the MEM bundle. Be sure to choose a “Host Extension” type for the baseline.
- Optionally add the new baseline to a baseline group.
- Attach the baseline or baseline group to one or more hosts.
- Scan and remediate to install the MEM on the desired hosts. Update Manager will put the hosts in maintenance mode and reboot if necessary as part of the installation process.
- If you get the error:
fault.com.vmware.vcIntegrity.NoEntities.summary disable Addmission control and then try to remediate again.
Verifying that everything is working properly
- Once both the the –configure and the –install commands have been run you can run the follow command to make sure everything is working properly:
./setup.pl --query --server=<esxi server's FQDN>
It’s a little bit more work than the LAG setup, but this is the proper way to get a full and complete Equal Logic Multipathing setup installed and working.
When we setup our switches, we changed the mtu on our vlan for iSCSI traffic. Now we need to edit the mtu on our iSCSI port groups, and vSwitch to also allow jumbo frames.
The first thing we need to do is take stock of what virtual swtich and port group we’re using for iSCSI traffic on each ESXi host. Follow these steps:
- Log into your host or vCenter server and then navigate over to your host’s “Configuration” tab.
- Click “Networking” on the left.
- Verify the Port Group name, Virtual Switch name, vmk number, IP address, and
which vmnics are being used. See Figure 1.
- If you’ve not already installed either vCLI or vMA see the posts on how to install and configure them Here and Here.
- Open either vCLI or ssh into your vMA VM.
- enter the following command “
esxcfg-vswitch -m 9000 <vSwitch's name> --server <Hosts's FQDN>“
- When prompted for a username and password enter the name and password of an account with Administrator permissions on that host.
- Verify that this change has taken effect by running the following command: “
esxcfg-vswitch -l --server <Hosts's FQDN>“. The MTU for your vSwitch should now be displayed as 9000.
We can’t modify the mtu on our port group, so we’ll need to migrated any VMs on iSCSI storage on this Host off of this Host and then remove our iSCSI port group. Once you’ve migrated any running VMs follow these steps:
- Open the Preferences of your vSwitch that we just modified.
- Select the port group in questions, and then click “Remove”.
- Now enter the following command in either vCLI or the vMA, “
esxcfg-vswitch -A“. This command re-created our iSCSI port group, attached it to our vSwitch, but did not add a vmknic to the port group.
"iSCSI" <vSwitch's name> --server <Host's FQDN>
- Now enter the following command to re-create the vmknic, “
esxcfg-vmknic -a -i <ip address> -n <netmask> -m 9000 "vmk#" -p "iSCSI" --server <Host's FQDN>“.
- We can now verify that our port group has the correct mtu by running the following commands: “
esxcfg-vmknic -l --server <Host's FQDN>” and “
esxcfg-vmknic -l --server <Host's FQDN>“. Check the MTU settings on both your Port group and Nics, they should now both be 9000.
We now need to rescan our iSCSI software adapter, and refresh our storage view to make sure our iSCSI Datastores are re-connected. Follow these steps:
- Click on “Storage Adapters” under the “Configuration” tab of your Host.
- Scroll down to your iSCSI Software Adapter, and then click “Rescan All…” in the top right, Verify that the iSCSI LUN(s) have re-appeared.
- Now click on “Storage” under the “Configuration” tab of your Host.
- Click “Rescan All…” in the top right of the screen. Verify that your iSCSI Datastores have re-appeared.
Finally let’s verify that our iSCSI network is fully supporting our jumbo frame size. Follow these steps:
- Log into the console of your ESXi Host.
- Press F2 to customize your host.
- When prompted, log into your Host. Scroll down to “Troubleshooting Options”. Press “enter”.
- Press enter on “Enable Local Tech Support” to enable it.
- Now press “Alt and F1” to enter the console, and then log in again.
- Enter the following command: “
vmkping -s 9000 <IP Address of your SAN's iSCSI interface>“. The ping should work and confirm that the mtu is 9000. If this does not succeed double check the mtu settings on your switches and SAN.
- Press ” Alt F2″ to exit the local console.
- Press enter on “Disable Local Tech Support” to disable the local console on your host.
- Exit your hosts’s console.
That’s it, your host is now configured to use jumbo frames, and now you can repeat these steps on the remaining Hosts.
In another post, we’ve already got the basic setup of the SAN completed, now we just need to finish a few things and then provision some storage.
First let’s get the firmware updated. If’ you’ve not already configured an account with EqualLogic, do so now by going to http://support.equallogic.com and signing up.
Once you’ve downloaded the firmware we’ll update it by following these steps:
- Login to the management group ip of your device, expand “Members” in the left hand pane.
- Highlight the unit, and then click on the “Maintenance” tab.
- Click “Update firmware….”, Enter the admin password, and then click “OK”.
- Navigate and point to the .tgz file that you’ve downloaded from EqualLogic, and then press “OK”
- In the “Action” column click the link to upgrade and follow the steps to upgrade and reboot.
We’ll now configure some email alerting, Log back into your management group IP and perform the following:
- Click the “Notifications” tab.
- Check the box labeled “Send e-mail to addresses”.
- In the “E-mail recipients” window, click “Add” and enter the email address you’d like to receive alert emails.
- in the “SMTP Servers” window click “Add”, and enter the IP address of your SMTP server.
- Check the box labeled “Send e-mail alerts to customer support( E-Mail Home)”.
- Enter a reply email address so that customer support can return an email to you in the event that the SAN reports a hardware failure.
- Enter the email address you want the SAN to use when it sends out emails in the box labeled “Sender e-mail address”.
That’s it for notifications, now let’s configure our first volume that we’ll make available to our ESXi hosts. Follow these steps:
- First expand “Storage Pools” in the left pane, and then click on the “Default” Storage Pool.
- Click on “Create Volume”, Give the volume a name and description and then click “Next >”.
- Give the volume a size, it’s important to remember that ESXi has a limit of 2TB -512B for a LUN size, so for simplicty, don’t make the volume larger than 2047GB. Uncheck “thin provisioned volume” unless you want it to be thinly provisioned. If you are planning on using snapshots, leave this at 100%, otherwise if you are going to be backing up the SAN without using snapshots, change this to 0% to conserve storage space. Click “Next >”.
- Click “No access” for now, we’ll add access later. “Access Type” should be set to “read-write” and the box for “allow simultaneous connections from initiators with different IQN names” should be checked. Click “Next >”.
- Click “Finish”.
- Highlight the newly created volume, and then click the “Access” tab.
- Click “Add”, Check the box labeled “Limit access by IP address”, Enter the IP address of the first ESXi server (use the IP address for the nic team on the LAG we created for iSCSI in this post). Click “OK”.
- Repeat steps 6 & 7 for each of your ESXi hosts.
That’s it. We’ve got our SAN configured, at least enough to get vCenter installed and running properly.