My name is Mike Foley. I'm an industry recognized subject matter expert on virtualization security (or so I'm told). I get a lot of questions about VM Escape on VMware. Seriously, all.the.freaking.time. So much that the meme above gets posted on my Twitter feed every time I get a question.
I know, I know.... Your security folks are in a panic right now. They are bringing up the Cloudburst demo from 2009 at Blackhat. While that was technically "cool" it was 2009!! And it was on an old version of VMware Workstation and NOT VMware's bare metal hypervisor, ESXi. I've talked to the author, Kostya, and thanked him for my job security! But even he has said that Cloudburst didn't work on ESXi.
VM Escape is not the threat your security guy thinks it is. It's really, really hard to do. Here's some bullet points to help you talk him or her off the ledge so I don't have to...
Did we forget the discussion of "Risk Management"? Is there a risk? Of course. There is also a risk of getting in a car crash as you drive to work. But you still do, right? Because the risk is very, very small. And I would argue that the risk of a true "take over the hypervisor" level VM Escape is FAR smaller than your getting in a crash on the drive to work.
Finally, if VM Escape was a real "thing" don't you think you'd be hearing more about all the cool exploits it supposedly has allowed to happen? Put things in perspective folks. Let's put a stop to the fear mongering and focus on the real and immediate threat.
The far more effective compromise of Admin Escape
That's when your administrator with all of the privs (because nobody actually implements least privilege!) gets compromised and the bad guys walk in through the front door. That's a proven method of attack. It works all the time. It's cheaper and far more effective. All this talk about VM Escape is a distraction from the real and proven challenges that face IT Security.
Every major breach in the past 5 years has NOT been via a VM Escape but via poor IT and security practices. Protecting a database with the password "password" or storing sensitive data, unencrypted, on Amazon S3 or having an IT process in place that DISCOURAGES running newer software that mitigates many issues that you are concerned about.
The problem is people. Not technology. Technology is relatively easy to fix. A broken process is MUCH harder to address.
Links
There's a ton of info out there on this subject. Here's a few of the links I suggest.
I talked about security in a virtualized world with my friends and colleagues at VMworld 2015
Finally, here are some links about me and where I'm coming from.
Thanks for reading,
mike
@mikefoley on Twitter
If you're not interested in a mix of personal and work stuff, I retweet a "curated/only the VMware stuff" via my other Twitter account @vspheresecurity
I know, I know.... Your security folks are in a panic right now. They are bringing up the Cloudburst demo from 2009 at Blackhat. While that was technically "cool" it was 2009!! And it was on an old version of VMware Workstation and NOT VMware's bare metal hypervisor, ESXi. I've talked to the author, Kostya, and thanked him for my job security! But even he has said that Cloudburst didn't work on ESXi.
VM Escape is not the threat your security guy thinks it is. It's really, really hard to do. Here's some bullet points to help you talk him or her off the ledge so I don't have to...
- You're forgetting about defense in depth. You need to root the VM to get the chance to even try a VM Escape! The question to ask your security people is "How did that exploit actually make it on to the VM in the first place?"
- Provided your existing defenses allowed that to happen, the only thing a VM gets to see is a subset of x86 instructions and interrupts. That malware better be good because one wrong move and that VM is crashing. And you'd submit that crash to VMware. And we'd see other crashes from other customers. And we'd know something funky was up. We haven't seen those types of crashes. (see: Canary in a Coal Mine)
- The process that runs a VM runs in real hardware ring 3. Should that uber malware (that broke through your lines of defense and rooted your VM) actually make it down to the VMX process it would then need to root that VMX process! Very, very few people have those kinds of coding chops AND internal knowledge of the ESXi hypervisor. We're talking single digits of people.
- You don't get to a shell nor do you get to a well-defined API. Just x86 instructions and interrupts. (see "sandbox" below)
- ESXi is not Linux and nor is it Windows. Successful attacks against those platforms don't necessarily mean that ESXi is vulnerable. In most cases, it's not. Just because ESXi has a Bash like shell doesn't mean the kernel is the same.
- VMware takes this all very seriously and they are constantly evolving the architecture (see "sandboxing" below)
- Isolation of I/O, Memory and devices are done at the HARDWARE level. ALL hypervisors are dependent on Intel VT and AMD-V virtualization enhancements to provide isolation.
- In vSphere 6.5, VMware introduced VMX Sandboxing. The VMX process *running in HW Ring 3* (!!!), is now sandboxed. What does "sandboxing" actually mean? There's a configuration file on ESXi (don't edit it) that dictates what the VMX process is allowed and not allowed to access. For example, the VMX process is allowed to make specific system calls but not some. It's allowed to read from certain files but doesn't have access to those files it doesn't need. e.g. /etc/passwd It doesn't have acces to SSH binaries nor is it allowed to run scripts. Is this the ultimate solution for a "VM Escape? No. It's a new layer of defense in depth, making it exponentially harder to weaponize the hypervisor.
Did we forget the discussion of "Risk Management"? Is there a risk? Of course. There is also a risk of getting in a car crash as you drive to work. But you still do, right? Because the risk is very, very small. And I would argue that the risk of a true "take over the hypervisor" level VM Escape is FAR smaller than your getting in a crash on the drive to work.
Finally, if VM Escape was a real "thing" don't you think you'd be hearing more about all the cool exploits it supposedly has allowed to happen? Put things in perspective folks. Let's put a stop to the fear mongering and focus on the real and immediate threat.
The far more effective compromise of Admin Escape
That's when your administrator with all of the privs (because nobody actually implements least privilege!) gets compromised and the bad guys walk in through the front door. That's a proven method of attack. It works all the time. It's cheaper and far more effective. All this talk about VM Escape is a distraction from the real and proven challenges that face IT Security.
Every major breach in the past 5 years has NOT been via a VM Escape but via poor IT and security practices. Protecting a database with the password "password" or storing sensitive data, unencrypted, on Amazon S3 or having an IT process in place that DISCOURAGES running newer software that mitigates many issues that you are concerned about.
The problem is people. Not technology. Technology is relatively easy to fix. A broken process is MUCH harder to address.
Links
There's a ton of info out there on this subject. Here's a few of the links I suggest.
- My ESXi hypervisor security whitepaper is here:
- I have plenty of interviews on virtualization security here:
- I did a vBrownbag session on VM Escape here
- Common Criteria addressed VM Isolation here. This is a PDF. See pages 70-71
- Edward Haletky, an industry recognized expert on virtualization security, has written a great blog on the subject
I talked about security in a virtualized world with my friends and colleagues at VMworld 2015
- Hemma Prafullchandra from Hytrust
- Edward Haletky of The Virtualization Practice
- Simon Crosby of Bromium
Finally, here are some links about me and where I'm coming from.
- I have a patent on virtualization security here
- My personal website is Yelof.com
- My corporate blog is here:
Thanks for reading,
mike
@mikefoley on Twitter
If you're not interested in a mix of personal and work stuff, I retweet a "curated/only the VMware stuff" via my other Twitter account @vspheresecurity