The perfect nested virtualization-based demo LAB: adding Template VMs (and vSAN)

Recently, I have installed several template virtual machines in my lab. My primary objective is to automate large-scale migrations to the AWS Cloud using the AWS CloudEndure Migration Factory Solution. The secondary objective is to test VMware HCX based migrations to Cloud. This article gives you an idea of how to prepare vSAN based nested LAB to achieve similar goals.

List of template VMs I deployed so far.

It was an incredible experience to deploy various operating systems released in the last 2 decades. Amazon Linux 2 and VMware Photon OS were new to me. Surprised by how similar all Linux installations are (Oracle Enterprise Linux, Fedora, CentOS). SUSE installation wizard was nice. I used FreeBSD a lot in the past, port collection, compile, the same story in 2022. Windows, server, multiple versions. The oldest Windows 2008 R2 server and VMware tools were tricky, the newest Windows 11 actually requires you to add TPM, so I kept it on Workstation level (no nested virtualization).

I will check if TPM is possible in nested-virtualization (via virtual TPM) some point later. To have Windows 11 running in nested vSphere. But this is not important at this point for me. VDI pools are OK on Windows 10 for testing VMware Horizon in Hybrid deployments (with cloud) or understanding how Horizon Universal adds value.

I am going to use Terraform to automate LAB migration deployment using recently (manually) deployed templates. I will add NSX-T and HCX to this lab and connect with Cloud. I start with AWS. As mentioned in the subject, my primary objective is to gain knowledge with mass migration tools such as AWS CloudEndure Migration Factory.

Coordinate and automate large scale migrations to the AWS Cloud using the AWS CloudEndure Migration Factory Solution – AWS CloudEndure Migration Factory Solution (amazon.com)

This is my ESXi host. Since the host Workstation has 128GB, each nested server runs with 32GB RAM. 24 core in the host, 8 assigned to each guest. I think, there will be enough capacity for NSX-T and HCX testing (assuming management overhead and template VMs, basic production-like workloads for migration testing).

vSAN works perfectly in a nested-virtualized environment. I used both virtual-NVMe (cache tier) and virtual-SCSI (capacity tier) disks (they are both on physical NVMe). Each vSphere host is using a different physical NVMe disk, to ensure vSAN performance. Actually, vSAN is as fast as the local disk while running OS installation times (no IOPS or sophisticated performance testing was done).

There is no distributed switch or NSX-T deployment yet. Just some good old standard switch. Works very nice in nested VMware Workstation 16 environment.
I will introduce more complexity by adding gateways and VLANs, however standard Host-only networks with DHCP and defined subnets work very well for vMotion and vSAN traffic.

That’s it for today. I encourage everyone to build nested virtualized LAB. The best is this: fast, silent (workstation computer) and there is one button to hibernate the entire datacenter when I go to sleep. Actually, I hibernate the Windows 10 running the Workstation 16 with the vCenter Server and vSphere nodes. Including everything that runs inside. Even vSAN. I hope this continues to work as the “datacenter” gets more complex (ie. adding NSX-T).

I am also planning to add a VMware Kubernetes engine (Tanzu) to see what the management of cloud-native workloads looks like in vSphere and how migration tools can pick up workloads from there.

I will not forget about VMware Site Recovery Manager and Horizon View. I am also planning to test vRealize automation. Large enterprises often use such VMware products (not only vSphere) and we need to have answers of how they work in a hybrid cloud deployment, how they integrate with cloud-native workloads.

Stay tuned, the best is yet to come!

Related posts

Fixing Proxmox Boot Hangs When Passing Through 2× RTX 3090 GPUs: Step-by-Step Troubleshooting Guide

Running multiple NVIDIA GPUs for AI workloads in Proxmox VE can cause early boot hangs if the host OS tries to load conflicting drivers. In this guide I document how my Proxmox host with 2× RTX 3090 was stuck at systemd-modules-load, how I debugged it, which files to inspect (/etc/default/grub, /etc/modprobe.d/, /etc/modules-load.d/), and the final stable configuration for rock-solid GPU passthrough to an Ubuntu VM.

Building the Perfect Edge AI Supercomputer – Adding an Edge Virtualization Layer with Proxmox and GPU Passthrough

I built on my edge AI hardware by adding Proxmox VE as the virtualization layer. After prepping BIOS, using Rufus with the nomodeset trick, and installing Proxmox, I enabled IOMMU, configured VFIO, and passed through 2× RTX 3090 GPUs to a single Ubuntu VM. This setup lets me run private AI workloads at near bare-metal speed, while keeping Windows and native Ubuntu for special use cases.

Budget AI Supercomputers: Dell Server vs. Threadripper Build vs. Next-Gen AI Desktop

Exploring three budget AI supercomputer paths: a Dell R740xd for enterprise labs with big storage but limited GPU flexibility, a TRX50 + Threadripper 7970X workstation offering fast DDR5, Gen5 NVMe, and dual RTX GPU power, and the futuristic GB10 AI desktop with unified CPU/GPU memory. Dell is lab-friendly, GB10 is AI-only, but the TRX50 build strikes the best balance today.

Building the Perfect Edge AI Supercomputer – Cost Effective Hardware

Keeping up with today’s technology is both exciting and demanding. My passion for home labs started many years ago, and while my family often jokes about the time and money I spend on self-education, they understand the value of staying ahead in such a fast-moving field. What started as curiosity has grown into a journey of building cost-effective supercomputers for edge AI and virtualization.

Fix VMware Workstation Performance Issues on Windows 11: Disable Hyper-V and VBS

This blog explains why VMware Workstation runs slower on Windows 11 compared to Windows 10, focusing on changes like Hyper-V, VBS, and HVCI being enabled by default on modern CPUs. It explores why sharing hypervisors with native hardware causes performance issues, and why disabling Hyper-V restores full VMware performance. Step-by-step PowerShell scripts are provided to toggle Hyper-V on or off safely.

Terraform deployment for FortiGate Next-Generation Firewall in Microsoft Azure

This blog explores deploying FortiGate VM in Azure, tackling challenges like license restrictions, Terraform API changes, and Marketplace agreements. It offers insights, troubleshooting tips, and lessons learned for successful single VM deployment in Azure. Using an evaluation license combined with B-series Azure VMs running FortiGate is primarily intended for experimentation and is not recommended for production environments.

Privacy Overview

This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.