VMware vCenter error “Unable to access file since it is locked” during disk consolidation solved

My setup consists of a vCenter 6 appliance and VMware’s Data Protection (VDP) 6 appliance. Every day there is a scheduled job that backup differentially all the VMs in our organization. Unfortunately this does not work always smoothly, as I would like to. One day I came across the “Unable to access file since it is locked” error. The first encounter was a failed log of VDP appliance on my e-mail inbox, but I am used to logs like this since as I said VDPA does not work always smoothly. So I logged into vCenter web client, connected to Data Protection appliance and re-runned the failed job for the outdated VMs. Unfortunately the job failed again!

Then I noticed an error message on my vCenter client saying “virtual machine disks consolidation needed”.

2

ok, I said. It seemed that the backup job did not delete the backup snapshot, let’s wipe it out manually. I got into the snapshot manager window and pressed the “Delete All” button and after a few minutes a new error message appeared.

1

Pressing the “Consolidate” button did not solved the problem neither. It seemed that the problem is bigger than I first thought. The  events on the problematic VM matched with the whole picture of the problem.

5

My internet searches were not fruitful. I guessed that the VDP appliance was root of the problem, so I shut it down and tried to delete all the snapshots of the problematic VM. Voila, I was near the solution, since the virtual disks consolidated into one disk and the VM properties disk file linked to the right vmdk file.

3

Browsing the datastore of the VM I noticed that an old differential disk file was not deleted (it is the vmdk file that it’s filename ends with a number). This was the causing trouble virtual disk.

4

I wiped it out by right clicking the vmdk file on the datastore browsing window. I restarted the VM to check that everything is ok.

Then I powered on the Data Protection Appliance but it failed to boot up correctly with the following error:

7

This was the root of the problem, VDP appliance had mounted the problematic virtual disk, that’s why this file was locked on the VM. By editing the VDP appliance properties, I deleted the mounted virtual disk that belonged to my problematic VM and left only the four default virtual disks of the appliance.

6

The appliance powered on smoothly and I checked that the services were up and running via VDPA’s console.

Job automation is good when it does not fail! If it fails it causes only troubles. Good luck!

VMware’s vSphere Client copy files between hosts tip

Usually I store windows or vmware iso files in host’s local datastores instead of SAN datastores. So, sometimes I need to access the iso file from another host located on another cluster. There is a quick and dirty way to copy an iso file from one host’s local storage to another host’s local storage or SAN datastore. Go to VMware’s vSphere Client and browse the source storage by right clicking the datastore, dive into the iso folder, select the file and copy it by right clicking it.

1

Browse the source local datastore and copy the file

Now goto the destination datastore, browse it and create a new folder.

2

Create a destination folder

Dive into and paste the iso file.

3

Paste the iso file

The copy takes place in a flash, it’s a local copy between the hosts.

Clone a VMware’s VM without vCenter in ESXi 5.x by commands (the official way)

Almost a year ago, I wrote an article about cloning VMWare VMs via vSphere client GUI and actually this is one of the most viewed posts in my blog. This method had the disadvantage of keeping the vm filenames the same. Let’s see an official way of cloning VMWare vms using vmkfstools command. Of course this method works with the free edition of ESXi 5.5 as well. I will show the procedure by cloning a snapshot of a vm since this is a little more tricky than cloning just a single vm.

First let’s make a snapshot of my Xp3 vm. Right click the vm and make a snapshot, let’s call it “mySnapshot”. Of course you don’t need to do this if you just want to clone a single vm. After that, shutdown the vm otherwise the files will be locked.

vm snapshot

vm snapshot

Now enable the ssh service on your ESXi by going to the configuration tab and the security policy option. Just start it for the moment, you don’t need to set it to start automatically. Use a ssh client like putty and connect to the hypervisor. Browse the folder under /vmfs/volumes path. It should look like this

datastores

datastores

so type

cd /vmfs/volumes
ls -la

in my case I have two datastores and my Xp3 vm is in datastore2, so let’s get into it.

Xp3 vm and snapshot

Xp3 vm and snapshot

cd datastore2
ls -la

As you can see now, there are more than one vmdk disk files, the flat vmdk, the delta vmdk and the vmdk descriptor file (the Xp3-000001.vmdk file in our example), along with the memory snapshot and various control and log files. What we should use here as a source file is the descriptor file that points to our snapshot. If you didn’t have a snapshot, the source file would be the vmdk descriptor file of your vm, Xp3.vmdk.

Now, what I need to do is clone my Xp3 snapshot to a new Xp4 vm. First create the destination folder in datastore2 folder and type in the following commands

mkdir Xp4

and clone the disk file

vmkfstools -i /vmfs/volumes/datastore2/Xp3/Xp3-000001.vmdk /vmfs/volumes/datastore2/Xp4/Xp4.vmdk -d thin

if it was a single vm without a snapshot, you should run

vmkfstools -i /vmfs/volumes/datastore2/Xp3/Xp3.vmdk /vmfs/volumes/datastore2/Xp4/Xp4.vmdk -d thin

if you list the files in the destination folder you will see the new flat file and the descriptor file.

cloned disk files

cloned disk files

The actual syntax of vmkfstools command is

vmkfstools -i source_path destination_path -d disk_format -a adapter_type

type man vmkfstools for more on this command

Finally create a new vm and force the use of an existing disk file.

create new vm

create new vm

use existing virtual disk

use existing virtual disk

select the cloned vmdk

select the cloned vmdk

You are ready to power it on.

power on

power on

if you list the folder of the cloned vm you will see something like that

cloned vm files

cloned vm files

Don’t forget to use sysprep command in windows to generalize your new virtual machine. More on this at the end of my previous article.

Reference: VMWare’s website article

Installing VMware vCenter Appliance on vSphere

VMware‘s vCenter server provides a centralized platform for managing your virtual infrastructure. The vCenter Server Appliance (vCSA) is a preconfigured Linux-based virtual machine that is optimized for running vCenter Server and associated services. Note that this version does not support Microsoft SQL server and IBM DB2. vCSA has some limitations, so it is not prefered in production environments. There is an interesting article summarizing these VMware vCenter Server Appliance 5.1 (vCSA) Limitations.

Goto myVMware Downloads page and press the button “View Download” at VMware vCenter Server record.

myVMware downloads page

myVMware downloads page

Then press the “Download Manager” button at the VMware-vCenter-Server-Appliance line with File type OVA. An OVA (open virtual appliance or application) is merely a single file distribution of the OVF and VMDK files stored in the TAR format.

An OVF refers to the Open Virtualization Format, which is a “packaging standard designed to address the portability and deployment of virtualization appliances”. The OVF file is a descriptor file of the packaging and VMDK files are containers for virtual hard disk drives of the appliance. For example, our OVA file includes the following files:

The ova file

The ova file

You can see the manifest file and the certificate files as well.

Appliance Download

Appliance Download

And download the appliance file. Then open the vCenter client and connect to your hypervisor. Select “Deploy OVF Template” from the main menu, to start the import wizard. Browse the downloaded ova file and press Next.

Source File

Source File

OVF Template Details

OVF Template Details

Enter the name for the Appliance, for example “vCenter Appliance”.

Name and Location

Name and Location

Select  Thick Provision Eager Zeroed disk format since this is the fastest way to write to a virtual disk (without zeroing blocks first or extending the disk file). I copy the policies from VMware’s website:

Thick Provision Lazy Zeroed

Creates a virtual disk in a default thick format. Space required for the virtual disk is allocated when the virtual disk is created. Data remaining on the physical device is not erased during creation, but is zeroed out on demand at a later time on first write from the virtual machine.

Using the default flat virtual disk format does not zero out or eliminate the possibility of recovering deleted files or restoring old data that might be present on this allocated space. You cannot convert a flat disk to a thin disk.

Thick Provision Eager Zeroed

A type of thick virtual disk that supports clustering features such as Fault Tolerance. Space required for the virtual disk is allocated at creation time. In contrast to the flat format, the data remaining on the physical device is zeroed out when the virtual disk is created. It might take much longer to create disks in this format than to create other types of disks.

Thin Provision

Use this format to save storage space. For the thin disk, you provision as much datastore space as the disk would require based on the value that you enter for the disk size. However, the thin disk starts small and at first, uses only as much datastore space as the disk needs for its initial operations.

Disk Format

Disk Format

Select the Network Map.

Network Mapping

Network Mapping

Check the Power on after deployment checkbox and press Finish.

Ready to Complete

Ready to Complete

It will take some time to deploy the vCenter Appliance.

Deploying

Deploying

Deploying

Deploying

Finally the vCenter Appliance will appear on vSphere client. It will power on and get an ip from your DHCP server.

vCenter Appliance Console

vCenter Appliance Console

Open a browser and type in the url of the appliance at SSL port 5480. for example https://10.1.1.10:5480

vCenter Appliance Administrative Console

vCenter Appliance Administrative Console

The default login credentials are username root and password vmware.

13

Finally configure the appliance pressing the setup wizard button!

Adding static routes to VMware’s vCenter Virtual Appliance

Sometimes it is useful to statically assign different gateway addresses for different network segments when configuring multiple Service Console network interfaces on VMware vCenter appliance.

A. To apply a static route on an ESXi host, effective immediately, but not persistent across a reboot:

Connect to host with a ssh client (I prefer putty) and execute the following command:

# route add -net <TargetNetwork> netmask <NetmaskAddress> gw <GatewayAddress> dev <interface>

e.g. route add -net 10.1.2.0 netmask 255.255.255.0 gw 10.1.1.1 dev eth0

then run

# route

to verify that the route was added successfully. In our example, the entry 10.1.2.0 10.1.1.1 255.255.255.0 UG 0 0 0 eth0 should appear.

B. To apply a persistent static route on the appliance:

Connect to host with a ssh client and open the static-routes file with a text editor:

# vi /etc/sysconfig/network/ifroute-eth0

(supposing the interface is eth0)

For every route add a line with the following format:

<TargetNetwork> <GatewayAddress> <Netmask> <Interface>

Then run the following command:

# service network restart

and finally verify the route with:

# route