Development

multi-clone.py improvements & random-vmotion.py

multi-clone.py improvements & random-vmotion.py

July 10, 2015 in Development, Virtualization

For a project i needed a couple of tools, so i improved my multi-clone.py script and created a new script: random-vmotion.py

multi-clone.py: setting advanced configuration parameters

The multi-clone.py script is improved so you are able to set advanced configuration parameters of the created VMs, this can be used if you have other tools that need some metadata or configuration.

To accomplish this, you have to use the CSV feature. The fields in the CSV have been changed slightly in comparison with the previous version:

Fields indicated with <> are mandatory, fields indicated with [] can be left empty.

For a full documentation, you can visit the GitHub repository.

random-vmotion.py

For some testing i needed a script which would be able to vMotion a set of VMs randomly to a set of Hosts. This script does exactly that.

The script accepts a file which contains the names of the VMs to move, one per line:

It also accepts a file which contains the names of the Hosts to randomly move to, one per line:

In the default way of working, it will continue to vMotion the VMs to a random host over and over again. For example: if you use the files above with the command:

It will start moving the first VM to one of the two hosts, wait 30 seconds and do the same for the second VM, and so on. Once it moved the fourth VM, it will start again with the first one.

There are more options which provide extra functionality:

  • Option to move each VM only once
  • Option to work in threads
  • Option to decrease the interval

The full documentation can be found on the GitHub page and an overview of the options can be found below:

 

multi-clone.py: CSV import and MAC customization

multi-clone.py: CSV import and MAC customization

March 2, 2015 in Development, Virtualization

After a very busy year, I finally came around to improving my multi-clone.py pyVmomi script. I have added a minor improvement, being able to disable SSL certificate warnings, and two major improvements: CSV import and MAC customization

CSV Import

There is a new command line flag, -C, which allows you to specify a CSV file. This CSV file provides multi-clone.py with the necessary information to clone the template into a VM for each line. The fields in the CSV should be:

  • VM Name (mandatory)
  • Resource Pool (can be empty)
  • Folder (can be empty)
  • MAC address (can be empty)
  • Post-processing script (can be empty)

A line should look like this:

If one of the fields is empty, but the command line flag is provided, the value from the command line is used. For instance, if you started the script with the following command line:

and used the following CSV:

What would happen is:

  • VM01 is created in the Resource Pool ‘Testing’, in the folder ‘Tools’ and the post-processing script ‘/tmp/tools.sh’ will run
  • VM02 is created in the Resource Pool ‘Development’, in the folder ‘Linux’ and the post-processing script ‘/tmp/script.sh’ will run

So, the command line values are only used if no values in the CSV are provided.

Mac customization

In the CSV, you can provide a MAC address for each VM. This MAC address will be assigned to the first NIC the script finds on the new VM before powering it on (if that is not disabled).

Two warnings:

  • It is your responsibility to provide MAC addresses that are not in use on your network
  • The order in which the NICs are presented to the script when there are more than one, is not always the same. In all my tests it seems it will provide the first one that has been added as the first, but i can not guarantee that.

Overview

As always, you can find the script on my GitHub page: https://github.com/pdellaert/vSphere-Python/blob/master/multi-clone.py

For more documentation: https://github.com/pdellaert/vSphere-Python/blob/master/docs/multi-clone.md

And just for good measure, the full description of all flags as presented by the -h flag:

 

multi-clone.py: Multi-threaded cloning of a template to multiple VMs

multi-clone.py: Multi-threaded cloning of a template to multiple VMs

February 3, 2014 in Development, Virtualization

In december, VMware released pyVmomi, a Python SDK for the vSphere API. In the past I have created a script to clone virtual machines using pySphere. This script has helped me (and others, if the mails I received were any indication), but i haven’t updated or improved it in a while.

So I decided this was a good moment to revisit the script and recreate it completely using pyVmomi. And not just recreate it, but improve a lot on it as well. The result is an all new multi-clone.py script which allows for the following list of capabilities:

  • Deploy a specified amount of virtual machines
  • Deploy in a specified folder
  • Deploy in a specified resource pool
  • Specify if the cloned virtual machines need to be powered on
  • Print out information of the main network interface (mac and ip, either IPv4 or IPv6)
  • Run a post-processing script with 3 parameters (virtual machine name, mac and ip)
  • Print logging to a log file or stdout
  • Do this in a threaded way

Threads

The previous version only allowed a sequential way of working, which meant the clone task had to complete, network information gathered and post-processing finished before the next deployment could start. Combined with pySphere, which could be rather slow in gathering information, made the old script slow and sluggish.

Using pyVmomi improved the speed of the script from the start, it’s a lot quicker in gathering information and executing commands. By adding the possibility to use threads, the speed is improved even more, altho this will depend on the IOPS of your datastore.

If the you specify the amount of threads, both the creation of the virtual machines and gathering the mac and ip information are threaded separately. To explain how this works, an example works best:

Imagine you want to create 4 virtual machines from a template and you specify the amount of threads to 2. At start, two clone tasks will start at about the same time. When a clone task is done, another is started. While this third clone task is started in a new thread (to replace the thread that finished), a first information gathering (mac,ip) and post-processing thread is started for the clone task that has finished. This all runs at the same time, so at this point two clone tasks are running, each in a thread, and one information gathering and post-processing thread is running. Once the second clone task is finished, two clone tasks threads are running and two information gathering and post-processing threads are running. If the two information gathering and post-processing threads aren’t finished when the two final clone tasks finish, two new information gathering and post-processing threads are put in the queue, but not started as only two threads can be running in that pool.

Deciding how many threads is the best for your case, will require a bit of experimenting and greatly depends on the IOPS of your datastore. The clone tasks takes the most time because it needs to copy the virtual disks to the newly created virtual machine.

Remember that vCenter, by default, will place any clone tasks in queue if more than 8 are started. So setting the amount of threads above 8 won’t really help, altho you are always able to do so.

Usage

Here is the output of the -h option, which provides an overview of the possible arguments and what they do:

Issues and feature requests

Feel free to use the Github issue tracker of the repository to post issues and feature requests.

Documentation

You can find all the documentation on GitHub.

Ansible roles to deploy Shorewall, Shorewall6 and DHCP server

Ansible roles to deploy Shorewall, Shorewall6 and DHCP server

December 22, 2013 in Development, Linux

Lately I’ve been looking into Ansible, a configuration management tool like Puppet or Chef. There is no specific reason why I’m using Ansible instead of the other possibilities, except that it seemed to be easier. One of the big advantages is that you do not need to install software on each managed host if you don’t want to. The whole system is Push, altho a Pull system is available.

Ansible uses modules, roles and playbooks. Modules are written in python and are used as building blocks of roles. They represent actions which can be taken on a host, like installing a package, making sure a file is present, … There is a whole list of standard modules on the Ansible documentation. Of course you can write your own modules to do more complex stuff, I recently wrote a module which checks out a WordPress application from a git repository and sets it up in a Plesk created vhost. I did this before I really got a feel for playbooks and roles in Ansible, because i’m sure that almost all steps in that module could be done using the default modules.

Roles use modules to setup certain server roles or configurations on hosts. For instance, you can create a role to install Apache or Nginx. Roles are a list of module actions which are executed in order. If one step fails for a host, the next step won’t be run for that host, but it does continue for other hosts. Roles should be limited to setting up a single server role. If you want to setup multiple items, like a LAMP stack, you should use playbooks.

Playbooks are a set of roles that needs to be present on one or more hosts. You could create a playbook which installs Apache, MySQL and PHP. This will probably call a couple of roles like one to install and setup Apache, another to install and setup MySQL and another to make sure PHP is installed and configured properly. This playbook, and others, are available as examples provided by Ansible Works.

Recently Ansible Works created a role exchange website called Ansible Galaxy. This gave me the idea to get a bit more serious and create a couple of roles which I would find interesting.

dhcp_server

Dhcp_server is a role which installs and configures a DHCP server on a host. It is capable of setting up the full capabilities of the standard DHCPv4 server in Ubuntu, Debian, CentOS, RHEL and Fedora.

For more details on the dhcp_server role, you can either visit the Ansible Works role page, or go directly to the GitHub page.

shorewall_simple & shorewall6_simple

Shorewall_simple is a role which installs Shorewall on a host and configure it. It is intended for simple setups, it can cope with multiple interfaces, but it is not capable in being used in a router or NAT gateway. For that a shorewall_full role is in development.

It is a role which is perfect to protect a single host with a firewall, controlling both incoming and outgoing packets. It provides the full use of policies and rules for a single host, even with multiple interfaces. So it is capable of setting up the standard rules like ACCEPT, REJECT and DROP, but also stuff like REDIRECT and DNAT (altho you wouldn’t really need that last one on a single host).

Shorewall6_simple is very similar to shorewall_simple, except it’s intended to install and configure Shorewall6, the IPv6 variant of Shorewall.

If you want to use this role, be aware that you are configuring a firewall on a remote server. If something goes wrong, you might have locked yourself out of your server and you might need to get direct access to the console of the server (which might be in a remote, difficult to reach, datacenter).

For more details on the shorewall_simple role, you can either visit the Ansible Works role page, or go directly to the GitHub page.

For more details on the shorewall6_simple role, you can either visit the Ansible Works role page, or go directly to the GitHub page.

shorewall_full & shorewall6_full

Shorewall_full and shorewall6_full are not released yet and are in full development. The aim of these roles is to provide a fully configurable Shorewall or Shorewall6 setup, which can be used in routers, nat gateways, VPN servers, …

multi-clone.py minimal user permissions

multi-clone.py minimal user permissions

November 14, 2013 in Development, Linux, Virtualization

Half a year ago, I posted about a Python script I created using PySphere, called multi-clone.py. I used this script to quickly deploy multiple vm’s from the same template and do some post-processing. This allowed me to easily setup a lab environment to test any sort of cluster tool, configuration management tool, …

This tool has been picked up by some other people. I’m happy to see my work is useful to others. This also meant that I got the occasional question about it.

Last week someone came to me with an issue, he got strange error messages. At first I thought it might have been a version mismatch as the original script was developed using PySphere 0.1.7, and he was using PySphere 0.1.8. After a quick update on my end and testing it with PySphere 0.1.8, everything worked fine. I had the same vSphere version, the same PySphere version and I did the same command as he did. Sadly, I couldn’t reproduce the error.

At this point, all I could think of was a permissions error. So we tested if the user could create a template with the exact same information, using the web-client. It seemed he couldn’t.

All this got me thinking about the minimal security permissions a user needed to run my script in a vSphere environment. So I tested a few permission setups and came up with a minimal permissions list. I added this to the Github repository readme file, but decided to post it here as well.

All permissions are only necessary on their appropriate item. For instance: datastore permissions are only necessary for the datastores on which the template and VMs will be located (or cluster if a Storage DRS cluster), so you can limit access to only a certain set of datastores.

Minimal permissions necessary to run multi-clone.py and all it’s features

  • Datastore
    • Allocate space
  • Network
    • Assign Network
  • Resource
    • Apply recommendation
    • Assign virtual machine to resource pool
  • Scheduled task
    • Create tasks
    • Run task
  • Virtual Machine
    • Configuration
      • Add new disk
    • Interaction
      • Power on
    • Inventory
      • Create from existing
    • Provisioning
      • Clone virtual machine (*)
      • Deploy from template

(*) This is in case you want to use the script to clone an actual VM instead of a VM template

Twitter

  • RT : There's a lot of talk about logs lately, but do you know why we call it a "log"? It's actually pretty interesting! The ter… about 3 years ago

Pictures

vExpert-2019 vExpert-Count
VCP-DCV
VCA-NVVCA-DCV VCA-CloudVCA-WM