How to install ESXi nested inside ProxMox VE

Nautilus inception

So, one of the tasks I had to complete this past week was: Install an ESXi hypervisor in a virtual machine, on top of a Proxmox VE 4 infrastructure. This post documents the process (since I wasn’t able to find a complete guide, I wrote one : ).

Why ESXi inside of Proxmox?

Test? Laboratory? Proof of Concept? Well the reasons behind this setup may vary, but if you’re going to do this, avoid it in production. From my tests the performance is comparable, but still you will get a lot of headaches and lose many of the benefits a vSphere environment has to offer. But first things first:


The first thing to look at is requirements, you will need:

  • A Proxmox node capable of Hardware Assisted Virtualization. (Intel: VT-x ; AMD: AMD-V)
  • QEMU >= 2.3.0
  • Kernel >= 3.19.0-21
  • At least 4GB of RAM for Proof of Concept; 16GB of RAM to add vCenter Server Appliance and another ESXi host.
  • Patience and this guide : )

During this process I used an Intel CPU, Proxmox VE 4.2, pve-qemu-kvm 2.4_14, kernel 4.4.6-1-pve and ESXi 6.0.

Although you shouldn’t encounter any problem with a slightly different environment or a diffrent version of ESXi, it is always best to get the latest software.

Enable nested KVM on host

The first thing you need to do is SSH into the node you plan to install ESXi on. Once in determine the processor:

model name      : Intel(R) Xeon(R) CPU E3-1245 v5 @ 3.50GHz

In this case I had an Intel CPU, if you have AMD it is not a problem, follow the appropriate tab:

Create this file (it shouldn’t exist): /etc/modprobe.d/kvm-intel.conf

Using your favourite editor add the following content to the file:

options kvm ignore_msrs=y
options kvm-intel nested=Y ept=Y

Then reload the modules:

# modprobe -r kvm-intel kvm; modprobe kvm kvm-intel

If this doesn’t work for you, you will have to restart the node.

Create this file (it shouldn’t exist): /etc/modprobe.d/kvm-amd.conf

Using your favourite editor add the following content to the file:

options kvm ignore_msrs=y
options kvm-amd nested=Y ept=Y

Then reload the modules:

# modprobe -r kvm-amd kvm; modprobe kvm kvm-amd

If this doesn’t work for you, you will have to restart the node.

Creating the Virtual Machine

This step is quite straightforward: you just need to create the new virtual machine, but you need to pay attention to three options:

  • OS Type must be “Other OS Types“.
  • CPU Type must be “host“.
  • Network Type must be “VMWare vmxnet3“.

Now, take note of the machine id that you used during the installation. SSH in the node hosting the newly created virtual machine and edit the file named: /etc/pve/qemu-server/YOURVMID.conf (of course replace YOURVMID with the ID of the virtual machine you created). And add at the end of the file:

args: -machine vmport=off
Each and every time you will modify things through the Proxmox interface, you will lose all the custom changes to this file. Unfortunately I have yet to come up with a simple way to avoid this without using glued solutions or configuration management.

ESXi Nested - ESXi InstalledIn this way you should be able to complete the ESXi installation without problems. However, when you will try starting your first virtual machine, you will notice that you can’t start virtual machines inside of the ESXi host.

Enabling Nested Virtualization inside ESXi

You will notice that you won’t be able to start virtual machines inside of your new ESXi, that is because you would have to add: vmx.allowNested = “TRUE” to each and every virtual machine inside of that host. That doesn’t sit well with me. The first thing you need to do is to enable SSH on the ESXi host.

Enabling SSH on ESXi

Enabling SSH isn’t a big deal. First access its console, then press F2 and enter your password. You will be facing the menu, from there select Troubleshooting Options and press enter. Then just enable SSH.

Now you have to SSH into the ESXi host and edit (nano and vi are available) the file /etc/vmware/config and append the following vmx.allowNested = “TRUE”. Reboot the host and voilà! You should be able to spawn machines correctly now.


It might not be the best for performance, but this setup is pretty good for proof of concept and laboratories if you already have an existing environment. I was even able to install vCenter Server Appliance though with a bit of pain.ESXi Nested - vCenterThanks to Jitze Couperus for the amazing image on the top.

Thanks to: Matt’s blog and The Perils and Triumphs of Being a Geek blog, from which I gained many information that are part of this guide.

Image courtesy of mark | marksei

The following two tabs change content below.
The IT guy with a slight look of boredom in his eyes. Freelancer. Current interests: Kubernetes, Tensorflow, shiny new things.

You may also like...

Secured By miniOrange