Azure scalesets - a useful technique?

This post is describing how to take an existing VMSS running instance and then using that as a baseline to create a new base image for all other instances going forward. Now ordinarily you would think why on earth are you doing this - surely you already have a defined base image and all instances are a copy of that?

Well in this case the instances had been edited directly and quite a lot of work had gone into that and we didn't want to lose the work that had gone on (i know this situation should never have happened, but it did - we'll say no more than that).

So how can we deal with that - VMSS instances are not 'normal' machines - i.e. they don't show up in virtual machines for a start - how can we even access details of the hard drive to be able to progress this?

Well lets start off with how we find the base os disk that it's using:

This az command shows details of just one of the running instances (make sure to set correct subscription first)

az account set -s yoursubname
az vmss show -g yourrgname --name yourvmssname--instance yourinstanceid

The instance id can be found via other command line means or it can be seen in the instances screen of the VMSS itself - in the picture below you can see each instance is suffixed with the instance id

So in my case i chose instance id 14 as this was the one the developers told me was the one that was in the best state to take as the base image.

Within the output from that command will be a section like this

      "image": null,
      "managedDisk": {
        "id": "/subscriptions/xxxxxxxxx/resourceGroups/xxxxxxx/providers/Microsoft.Compute/disks/xxxxxxx_disk1_d7d1b3b73b754a6684f540ee1ebe4a4c",
        "resourceGroup": "xxxxxxx",
        "storageAccountType": "Standard_LRS"
      },


The key part being the bit in orange - this is where the actual disk is held. Once we have this bit of information we need to snapshot that disk.

az snapshot create -g yourrgname -n yoursnapshotname--source "/subscriptions/xxxxxxxxx/resourceGroups/xxxxxxx/providers/Microsoft.Compute/disks/xxxxxxx_disk1_d7d1b3b73b754a6684f540ee1ebe4a4c"

So i now have a snapshot of that running vmss instance - now i need to create a disk based on that snapshot.

The main inputs you need are the snaphot location (which is in the output from the previous command) and the size of the disk - again in the info from the last command.

az disk create --resource-group yourrgname --name giveyourdiskaname --sku Standard_LRS --size-gb 127 --source "the path returned from the last command for the snapshot location"

Now we have a complete copy of that os disk and we can move on to the next steps.

Now for me this gets a little more complicated as we have domain joined windows machines - so i have to copy the disk into an isolated area so it doesn't try and talk to the domain and mess everything up with the running image as at the moment they have the exact same SID. If you have a standalone machine (or a linux one then you can maybe simplify the next section).

So to copy the disk to an isolated area i now switch to powershell (this is likely also possible in az command line but i found a simple example i could cut and paste from so that's what i used......)

The short script below just copies the disk to another subscription - honestly this feels overly long and I'm sure there is a shorter version than this.

#Provide the subscription Id of the subscription where managed disk exists
$sourceSubscriptionId="xxxxxx"

#Provide the name of your resource group where managed disk exists
$sourceResourceGroupName='yourrgname'

#Provide the name of the managed disk
$managedDiskName='richvmssdisk'

#Set the context to the subscription Id where Managed Disk exists
Select-AzSubscription -SubscriptionId $sourceSubscriptionId

#Get the source managed disk
$managedDisk= Get-AzDisk -ResourceGroupName $sourceResourceGroupName -DiskName $managedDiskName

#Provide the subscription Id of the subscription where managed disk will be copied to
#If managed disk is copied to the same subscription then you can skip this step
$targetSubscriptionId="yyyyyyyy"

#Name of the resource group where disk will be copied to
$targetResourceGroupName='newrgname'

#Set the context to the subscription Id where managed disk will be copied to
#If snapshot is copied to the same subscription then you can skip this step
Select-AzSubscription -SubscriptionId $targetSubscriptionId

$diskConfig = New-AzDiskConfig -SourceResourceId $managedDisk.Id -Location $managedDisk.Location -CreateOption Copy 

#Create a new managed disk in the target subscription and resource group
New-AzDisk -Disk $diskConfig -DiskName $managedDiskName -ResourceGroupName $targetResourceGroupName

The end result of that is just copying the disk to another subscription.....

Now we have the disk where i want it i now need to reattach it to a VM to create a 'temporary' machine.

Again many ways to do this - this time I'm using the portal - so navigate to the disk you just copied and then click the create VM button

Fill in all the usual stuff and wait couple of minutes for that machine to start up.

At this point we have an exact copy of the source VMSS instance.

Now i log on to that machine using the local administrator account, and again in my specific case i do an additional step which is to take the machine out of the domain using the standard process. A quick reboot to complete that step and we're ready to start turning this VM into an image.

Again i log in with the local admin account and run the sysprep tool to 'generalize' the machine - i.e. put it in a state where it can be used as the base of many new machines (so remove the machine specifics and have it in a 'new' machine state)

To trigger sysprep i run this

%WINDIR%\system32\sysprep\sysprep.exe /generalize /shutdown /oobe

That takes a few minutes and the machine is then shut down.

Now we create the image - to do this i use the portal (again an be done with other tooling but this was the first way i found to do it....)

Navigate to the VM page and then click capture

On the next screen just give it a name and click create

Now we have an image created of our machine - we can now set that to be the base image for our scaleset and complete the circle....

Just one quick step though (because of my domain joining 'issue') i just need to copy that image to the original subscription so it's visible to the scaleset. Again sure there is more than one way to do this but the simplest way i found is to navigate to the resource group containing the image and do a resource move from there - everything else i found seemed to be tens of lines of code for what should be a simple operation

So fro the resource group screen containing the image click on the following option

That will then be checked (but the image will have no dependencies and can easily be moved) - and it can go back to the original resource group where everything started out.

The final step is to now set this 'new' image to be the base image for the scaleset - i switch this time back to az command line as it's simple here and run the following

az vmss update --resource-group yourrgname --name yourscalesetname --set virtualMachineProfile.storageProfile.imageReference.id=/subscriptions/xxxxxxx/resourceGroups/yourrgname/providers/Microsoft.Compute/images/yourimagename

Now all of the instances in the scaleset are replaced with the 'new' image and any new instances added will also use that one.

And there you have it a 'simple' way to take a VMSS instance as a baseline and turn that back into a usable image for all the other instances.

Hopefully that's useful - even if it only highlights multiple different ways of interacting with the azure platform and using functionality that is maybe not an everyday thing to want to do.