Virtualization

multi-clone.py: Multi-threaded cloning of a template to multiple VMs

multi-clone.py: Multi-threaded cloning of a template to multiple VMs

In december, VMware released pyVmomi, a Python SDK for the vSphere API. In the past I have created a script to clone virtual machines using pySphere. This script has helped me (and others, if the mails I received were any indication), but i haven’t updated or improved it in a while.

So I decided this was a good moment to revisit the script and recreate it completely using pyVmomi. And not just recreate it, but improve a lot on it as well. The result is an all new multi-clone.py script which allows for the following list of capabilities:

  • Deploy a specified amount of virtual machines
  • Deploy in a specified folder
  • Deploy in a specified resource pool
  • Specify if the cloned virtual machines need to be powered on
  • Print out information of the main network interface (mac and ip, either IPv4 or IPv6)
  • Run a post-processing script with 3 parameters (virtual machine name, mac and ip)
  • Print logging to a log file or stdout
  • Do this in a threaded way

Threads

The previous version only allowed a sequential way of working, which meant the clone task had to complete, network information gathered and post-processing finished before the next deployment could start. Combined with pySphere, which could be rather slow in gathering information, made the old script slow and sluggish.

Using pyVmomi improved the speed of the script from the start, it’s a lot quicker in gathering information and executing commands. By adding the possibility to use threads, the speed is improved even more, altho this will depend on the IOPS of your datastore.

If the you specify the amount of threads, both the creation of the virtual machines and gathering the mac and ip information are threaded separately. To explain how this works, an example works best:

Imagine you want to create 4 virtual machines from a template and you specify the amount of threads to 2. At start, two clone tasks will start at about the same time. When a clone task is done, another is started. While this third clone task is started in a new thread (to replace the thread that finished), a first information gathering (mac,ip) and post-processing thread is started for the clone task that has finished. This all runs at the same time, so at this point two clone tasks are running, each in a thread, and one information gathering and post-processing thread is running. Once the second clone task is finished, two clone tasks threads are running and two information gathering and post-processing threads are running. If the two information gathering and post-processing threads aren’t finished when the two final clone tasks finish, two new information gathering and post-processing threads are put in the queue, but not started as only two threads can be running in that pool.

Deciding how many threads is the best for your case, will require a bit of experimenting and greatly depends on the IOPS of your datastore. The clone tasks takes the most time because it needs to copy the virtual disks to the newly created virtual machine.

Remember that vCenter, by default, will place any clone tasks in queue if more than 8 are started. So setting the amount of threads above 8 won’t really help, altho you are always able to do so.

Usage

Here is the output of the -h option, which provides an overview of the possible arguments and what they do:

Issues and feature requests

Feel free to use the Github issue tracker of the repository to post issues and feature requests.

Documentation

You can find all the documentation on GitHub.

multi-clone.py minimal user permissions

multi-clone.py minimal user permissions

Half a year ago, I posted about a Python script I created using PySphere, called multi-clone.py. I used this script to quickly deploy multiple vm’s from the same template and do some post-processing. This allowed me to easily setup a lab environment to test any sort of cluster tool, configuration management tool, …

This tool has been picked up by some other people. I’m happy to see my work is useful to others. This also meant that I got the occasional question about it.

Last week someone came to me with an issue, he got strange error messages. At first I thought it might have been a version mismatch as the original script was developed using PySphere 0.1.7, and he was using PySphere 0.1.8. After a quick update on my end and testing it with PySphere 0.1.8, everything worked fine. I had the same vSphere version, the same PySphere version and I did the same command as he did. Sadly, I couldn’t reproduce the error.

At this point, all I could think of was a permissions error. So we tested if the user could create a template with the exact same information, using the web-client. It seemed he couldn’t.

All this got me thinking about the minimal security permissions a user needed to run my script in a vSphere environment. So I tested a few permission setups and came up with a minimal permissions list. I added this to the Github repository readme file, but decided to post it here as well.

All permissions are only necessary on their appropriate item. For instance: datastore permissions are only necessary for the datastores on which the template and VMs will be located (or cluster if a Storage DRS cluster), so you can limit access to only a certain set of datastores.

Minimal permissions necessary to run multi-clone.py and all it’s features

  • Datastore
    • Allocate space
  • Network
    • Assign Network
  • Resource
    • Apply recommendation
    • Assign virtual machine to resource pool
  • Scheduled task
    • Create tasks
    • Run task
  • Virtual Machine
    • Configuration
      • Add new disk
    • Interaction
      • Power on
    • Inventory
      • Create from existing
    • Provisioning
      • Clone virtual machine (*)
      • Deploy from template

(*) This is in case you want to use the script to clone an actual VM instead of a VM template

vCenter 5.5 Server Appliance quirks

vCenter 5.5 Server Appliance quirks

Last week I upgraded my whole vSphere 5.1 environment to 5.5 and migrated to the vCenter 5.5 Server Appliance (VSA). Overall, I’m happy with this migration as the appliance gives me everything i need and the new web client works amazingly well, both with Mac OS X and Windows.

But there are a few quirks and small issues with it. Nothing to serious, and as i understand it, the VMware engineers are looking into it, but for those who are experiencing these issues, I wanted to provide a bit of explanation on how to fix them.

Quick stats on hostname is not up to date

20131003_vcenter_quick.stats.error

The first issue I noticed, was a message that kept appearing in the web client when I was looking at the summary of my hosts. At first I thought that there was a DNS or connection issue, but i was capable of managing my hosts, so that was all good.

Starting to investigate the issue on internet, I noticed a few people reporting this issue, and apparently VMware already posted a KB article (KB 2061008) on it.

Let’s go to the simple steps on how to fix this on the VSA:

  1. Make sure SSH is enabled in your VSA admin panel:
    20131003_vsa_ssh
  2. SSH to the VSA with user root and use the root password from the admin panel
  3. Copy the /etc/vmware-vpx/vpxd.cfg file to a save location, you will keep this as a backup
  4. Open the /etc/vmware-vpx/vpxd.cfg file with an editor
  5. Locate the </vpxd> tag
  6. Add the following text above that tag:
  7. It should more or less look like this:
    20131003_vpxd_config
  8. Save the file
  9. Restart your VSA, the easiest way is just to reboot it using the admin panel, or using the reboot command.

If you ever update the VSA, check the release notes, if this bug is fixed, you might want to remove these config values again.

Unable to connect to HTML5 VM Console

After a reboot of my VSA, I was unable to open the HTML5 VM Console from the web client. I got “Could not connect to x.x.x.x:7331”, the service seemed down. VMware is aware of this issue and a KB article (KB 2060604) is available.

The cause of this issue is a missing environment variable (VMWARE_JAVA_HOME). To make VSA aware of this variable, you can follow these steps:

  1. Make sure SSH is enabled in your VSA admin panel (see screenshot in step 1 of the issue above)
  2. SSH to the VSA with user root and the root password from the admin panel
  3. Open the /usr/lib/vmware-vsphere-client/server/wrapper/conf/wrapper.conf file with an editor
  4. Locate the Environment Variables part
  5. Add the following text to the list of environment variables:
  6. It should look more or less like this:
    20131003_vsa_wrapper
  7. Save the file
  8. Restart the vSphere Web client using:

That should fix the issue and the HTML5 VM Console should work fine!

Migrate vCenter 5.1 Standard to vCenter 5.5 Server Appliance with Distributed vSwitch and Update Manager

Migrate vCenter 5.1 Standard to vCenter 5.5 Server Appliance with Distributed vSwitch and Update Manager

At VMworld San Fransisco, VMware announced vSphere 5.5 and they officially released it a couple of days ago. With this new version of vSphere, the vCenter Server Appliance has been updated as well.

With this new version, the maximums have been increased. The vCenter Server Appliance was only usable in small environments with a maximum of 5 hosts and a 50 VM’s with the internal database. If you had more hosts and/or VMs, you had to connect your vCenter to an Oracle database. (Thanks Bert for noting this)

As of version 5.5, these limitations have been changed to a 100 hosts and 3000 VMs. With this change, vCenter Server Appliance becomes a viable alternative to a full fledged install on a Windows Server.

Until now I have always used vCenter as a full fledged install on Windows Server, with an SQL Server in my home lab. I used this setup to get a feel for running, maintaining and upgrading vCenter and all it’s components, while using multiple windows servers in a test domain. But with this new release, I’ve decided to migrate to the appliance and do a semi-fresh install.

I say semi-fresh, as I will migrate a few settings to this new vCenter server. Most settings will be handled manually or through the hosts, but the Distributed vSwitch are a bit more complicated. So I wanted to write down the steps I used to migrate from my standard setup to the appliance.

1. Export DvSwitch

You can export your DvSwitch using the web client with a few easy steps.

Export

Go to the Distributed vSwitch you want to migrate and right click it, go to All vCenter actions and select Export Configuration. Make sure you export all port groups and save the file to a convenient location.

2. Create a cluster in the new vCenter Server Appliance

Make sure the cluster has the same settings as the one in the old vCenter server. Focus on the EVC settings, the rest can be as you choose, but this is rather important if you are migrating live hosts and VMs.

3. Disable High Availability on the cluster

As you need to move hosts away from the cluster, you will have to disable the High Availability on it.

4. Disconnect the hosts from the old vCenter server and connect them to the new vCenter Server Appliance

At this point, you need to disconnect the hosts from the old vCenter server and connect them to the new vCenter Server Appliance. This might take a while, so be patient and watch the progress.

Your hosts might show a warning indicating an issue, but this can be safely ignored as it will be solved after the import of the Distributed vSwitch

5. Import the Distributed vSwitch into the new vCenter Appliance Server

Go to the network tab and right click on the cluster, go to All vCenter actions and select Import Distributed Switch.

Import

 

Make sure you select the ‘Preserve original distributed switch and port group identifiers’.

Preserve all settings

Give it a bit of time and your hosts will recognise the switch, and everything will be synced and connected again.

6. Update manager

There is one small issue with the great vCenter Server Appliance, it does lack an Update Manager in it’s regular setup. Luckily, you can connect a standard Update Manager install to the vCenter Server Appliance. I would suggest you just follow the standard guide. This one is still for vSphere 5.1, but the 5.5 version hasn’t changed much, so it should be pretty straightforward.

*update* Added extra information on the limitation of vCenter Server Appliance 5.1 (Oracle DB possibility)

PySphere script to clone a template into multiple VMs with post processing

PySphere script to clone a template into multiple VMs with post processing

*There is a better/newer version of this script, for more details, check this post: https://dellaert.org/2014/02/03/multi-clone-py-multi-threaded-cloning-of-a-template-to-multiple-vms/ – Or via github: https://github.com/pdellaert/vSphere-Python *

For an internal test setup i needed to be able to deploy multiple VMs from a template without to much hassle. So i started thinking that the best way to approach this, would be using a script. At first i thought i had to do this using PowerCLI as this is the preferred VMware way of scripting.

Luckily i came across the wonderful site of PySphere, which is a Python library that provides tools to access and administer a VMware vSphere setup. As Python wasn’t my strong suite, i was in a bit of a dilemma, i had almost no experience with either of those languages, so which to go for. Altho PowerCLI/Powershell has a lot more possibilities, as it is maintained and developed by VMware itself, Python had the great advantage i could do it all in a more familiar environment (Linux). It’s also closer to the languages i know than Powershell is. So i decided to just go for Python and see if it got me where i wanted to go.

This script deploys multiple VMs from a single template, you can specify how many and what basename the VMs should have. Each VM gets a name starting with the basename and a number which increments with each new VM. You are able to specify at what number it should start. You can also specify in which resource pool the VMs should be placed.

And as a final feature, you can specify a script which should be called after the VM has successfully booted and the guest OS has initiated it’s network interface. This script will be called with two arguments: the VM name and it’s IP. You can even specify it may only return a valid IPv6 address (i needed this to deploy VMs in an IPv6-only test environment).

The output if run with the help argument (-h):

To run the script, the command should look something like this:

And finally, the script itself, you can also download it on Github:

The script is probably a work in progress, as there are a lot of possibilities for improvement, if you have any requests feel free to contact me!

This is the first Python script i’ve ever written, so forgive me if i made some basic mistakes against best practices. Feel free to submit a patch.