For as long as two virtual machines have shared the same physical host, people have worried about security. And you can rarely talk about virtual machine security for long without the subject of encryption coming up.
It's no surprise, then, that encryption of virtual machines is a thing, and it's been a thing for quite some time. But here's another thing: encryption within a virtual machine has never really taken off, and that's because of the negative performance impact it inevitably has.
That's why there's been a good deal of interest in VMware's vSphere 6.5, which was released in mid-November. This release of vSphere offers virtual machine encryption within the hypervisor. As Mike Foley, a VMware Senior Technical Marketing Manager, explains, "As I/O comes out of the virtual disk controller in the VM, it is immediately encrypted by a module in the kernel before being sent to the kernel storage layer."
"Both VM Home files (VMX, snapshot, etc.) and VMDK files are encrypted," he says. vSphere 6.5 also enables vMotion encryption — automatically for encrypted VMs, and optionally for unencrypted VMs.
A major benefit of encryption in the hypervisor is that it makes the encryption VM and guest OS agnostic. The encryption keys are also not stored in VM memory; rather, key management is based on KMIP1.1, and vSphere vCenter has a KMIP client.
Encryption in vSphere 6.5 vs. VM-Based Encryption Performance Comparison
But what about performance? Specifically, how does hypervisor-based encryption affect performance compared to VM-based encryption?
The good news is that in November VMWare published a document somewhat unimaginatively entitled VMware vSphere Virtual Machine Encryption Performance, which attempts to address precisely this question.
Now of course, all the usual caveats apply: companies that produce these sorts of documents tend to choose tests that show their technology in the most favorable light, so these documents have to be read with that in mind. Nevertheless, it's worth taking a look at the findings.
So here's the summary. "For most regular types of storage, like enterprise-class SSD or VMware vSAN, the impact on I/O performance is very minimal," says the report. So far, so good.
But unfortunately the results are not so good when it comes to the use of state-of-the-art NVMe storage devices. "VM encryption can lead to bottlenecks in I/O throughput and latency for ultra-high-performance devices (like a high-end NVMe drive) that can support hundreds of thousands of IOPS," according to the report.
So let's dive a little deeper. How minimal is a minimal impact?
VMware starts with an Intel S3700 SSD, which is a mid-level device capable of doing about 75,000 random IOPS — similar to an enterprise storage array. Using 512KB sequential reads and writes, the study compares I/O throughput, I/O latency, and CPU cost per I/O for a regular and an encrypted VM.
And it turns out that there's almost no extra overhead when it comes to throughput or latency, although the CPU cycles per I/O go through the roof (>400%) due to the encryption of large quantities of data. When there are plenty of CPU resources available, this does not have an adverse effect on application performance according to VMware, but if there is a scarcity then this can add significant overhead to other applications.
"Therefore, the ESXi server requires a sufficient amount of CPU resources when VM encryption is enabled," VMware says, rather stating the obvious.
The story is pretty much the same with 4KB random reads and writes, except that in this case the CPU load is only roughly double with encryption enabled.
The Performance Hit on Low-Latency Storage Devices
Now let's move on to looking at the fast stuff: storage provided by an NVMe device that has ultra-low latency performance and high throughput: 750,000 read IOPS per device.
What we see now is a significant impact on I/O throughput and latency when VM encryption is used. The bandwidth with encryption is about 30-50% of standard performance, with a similar increase in I/O latency.
Here's how VMware explains this: "Because the device performance is high, the per-I/O latency we add in the IOFilter path for encryption (in the case of writes) and decryption (in the case of reads), which is in the order of a few microseconds, quickly adds up and shows as a bottleneck."
So overall, what we can conclude is that if you want virtual machine encryption using vSphere 6.5 and you use storage devices and subsystems in the latency range of a few hundred microseconds or slower, then the increased CPU costs does not translate into a noticeable increase in latency or a decrease in throughput. All you need to worry about is whether you have sufficiently grunty CPUs to handle the actual encryption and decryption operations.
But if you are using ultra-low latency devices like NVMe drives, the impact of the higher CPU costs of encryption and decryption directly translates into reduced throughput and increased I/O latency.
So ultimately the message is that in some circumstances using vSphere 6.5's built-in encryption is a no-brainer. But if you're using ultra-high performance storage, then unfortunately it isn't.
Paul Rubens is a technology journalist and contributor to ServerWatch, EnterpriseNetworkingPlanet and EnterpriseMobileToday. He has also covered technology for international newspapers and magazines including The Economist and The Financial Times since 1991.