All your eggs in one basket, privileged user access, no segregation of duties, little or no monitoring and no preventive controls

The rest of the story can be found here…

The MOB as it is known is used mainly by developers or for debugging. It has direct access to the VMkernel so as well as allowing view access it allows changes to be made. It can be access over port 443 (HTTPS) and requires authentication using the ROOT account password. The MOB if left accessible is a dangerous interface in the hands of an attacker. Therefore it is recommended you disable it or restrict access to it through access control methods.

Remediation

1. Logon to the ESX/ESXi console using SSH or directly.
2. Change directories to /etc/vmware/hostd
3. Edit the following file proxy.xml but before doing so make a copy
4. Find the line "<pipeName>/var/run/vmware/proxy-mob</pipeName>"
5. Comment out or remove the entire section between "<e id=" and "</e>"
6. Then make sure that all "<e id=" are in sequence by renumbering them accordingly
7. Save the file

IMPORTANT: If you mess the file up then make sure you have a backup as you will effect the authentication process and access to the Host.

“Air travel is statistically the safest transport type. Of course, there are many factors that make for safe airline travel. Arguably, sound aircraft maintenance is the hallmark of safe carriers. Each airliner has thousands of components. While operating, feedback is continuously sent back to the flight and maintenance crews. Standard practice in a well run airline”

How does this relate to virtualization, and virtualization security specifically?

“Well, the aircraft is the platform, our virtualization system. Everyone on-board it is safe, they have had a health check and we know as best we can that they carry no viruses. They have also been through security at the airport so we are firewalled from ‘the bad guys’. The problem is you are expecting me – as well as flying the plane to run all the maintenance. I have to keep the plane running to schedule – when we land, or even while flying, you expect me to take the plane apart and check all the components are in good order AND I have to do this manually! I can do it, if you can give me more people then I can keep the airliner as save as the people in it. What I would really like is an automated tool to do all the checks – non intrusively”

Interestingly we looked up some statistics, the odds of being on an airline flight which results in at least one fatality on the top 25 airlines is 1 in 5.4 million; whereas on an airline flight with the bottom 25 safety record the odds are 1 in 159,119 – 500 times more likely!

There are some more analogies for cloud computing, puns really – we will spare you those today!

VMinformer’s consultants have spent the last 5 years researching and conducting virtualization audits. In 100% of cases organizations fail our audits because basic security controls have been ignored – time, and time again, the misconfiguration failures boiled down to human performance and behavior.

Of course, system failure due to human oversight is nothing new, so why should we be worried this time? There are several factors at play. Firstly, most people take too much comfort from the fact that the individual elements of their virtual computing infrastructure is protected from malware and firewalled – which indicates that there is a lack of understanding, or ignorance, of their virtualization manufacturer’s security recommendations.

The best firewalls, AV, IDS (no disrespect to those vendors – their products do an important job) provide little protection to virtual and cloud infrastructure.

Next in our experience, systems are unpatched and security updates (which there are many, and diligently communicated by the virtualization manufacturers) are ignored. Is it because the virtualization administrators are too busy? Maybe they don’t have visibility over their virtual computing estate, or lack in-depth expertise across so many of the IT disciplines they now control (network, storage, security). Generally it is a combination of all these factors. Now is the time, if you are responsible for a virtualization platform, to take a step back and do some risk analysis.

“What is the consequence of a loss of, or within, our virtual computing infrastructure and what is the likelihood of that happening?”

Unfortunately, there are more factors at play.

• Where’s the segregation of duties in your virtual computing environment?
• Who’s responsible for looking at the what, where, when and how?
• Who has oversight?
• What audit or assurance are you aware of?

You only have to look at what happened in the financial markets with its over confidence, self regulation and ultimate collapse to know that your overworked system administration team has no chance.

This platform is just too critical to fail….

but fail it will if these factors are not addressed. But this time it could be on a scale that we’ve never experienced before in corporate computing. You can envisage not just the inconvenience of losing a few systems, but the loss of an entire IT infrastructure. Why and how will it happen? The usual reason – humans ignoring the warning signs, and simply not taking precautionary preventitive action.

A classic example is the release of SQL Slammer in 2003 – it effected 75,000 systems in 10 minutes causingan estimated billion dollars worth of damage. How could this have happened? Microsoft had issued a patch 6 months before. A patch that was almost universally ignored!

VMware, Citrix and Microsoft all provide patches and make recommendations on setup and configuration but do people follow them? Generally, they don’t. Worse still people assume that the configuration that they deployed in development three years ago is suitable for production or even DMZ use today! Of course the usual reasons are at the forefront: people either don’t have the time to inquire; or are scared that if they make a change it may break a critical business process.

Human behavior, whether it be oversight, ignorance, a laissez faire attitude, lack of visibility, or lack of responsibility, will be the fundamental cause and systemic failure of security in virtualization and cloud computing infrastructures. Those organizations that choose to be complacent and ignore the warnings will be part of the most catastrophic system failure we have ever seen.

We don’t know how, or on what scale, but the warning signs are clear…it is inevitable.

It’s your choice what are you going to do…?

Traditional physical security controls such as Firewalls, IPS/IDS, Anti Virus are unfortunately not suited to virtualization at all. Although it is better to have something than nothing if such traditional measures are implemented they will ultimately end up costing organizations more, and undermine the initial cost benefits of virtualization.

The added push towards “Cloud” computing only compounds the situation, so should we bury how heads in the sand, be alarmed or run for the hills and give up? Or are there alternative approaches that can be adopted?

If you’ve got this far you won’t be surprised to hear me say a resounding “YES!” to the question “are there alternative approaches!”

Traditional security solutions have had there day, it’s time to wake up and smell the coffee!

The dynamic nature that is virtualization and for that matter cloud computing means that it is trivial for an administrator inadvertently or otherwise to bypass security controls such as firewall’s simply by adding an additional network card to a virtual machine. This behavior can often go unnoticed due to lack of visibility or controls.

Virtualization and cloud computing have many benefits, the most obvious being cost and ability to scale in a way that has not been possible before. This flexibility however brings with it complexities that unless understood will allow human traits such as complacency, ignorance and it won’t happen to me attitude to reduce the overall effectiveness, security and cost savings of virtualization.

There are no perimeters, endpoints such as the server, desktop and application are becoming blurred. In the future there will be no operating system as we understand it today and therefore we must adapt, those who don’t will be left behind. If you are unable or sceptical about doing anything make sure you do one thing, monitor and provide yourself with some visibility.

As Yoda said, “Do or do not. There is no try”

Those organizations that do approach the security concerns they have often end up deploying traditional security measures like Firewalls, Anti-Virus and IDS. These solutions are typically not built for virtualization frameworks and will start to erode any cost benefits that virtualization has brought to the business. The fact that these solutions have not necessarily been built for virtualization and have simply been virtualized from their physical form means they will impact the underlying virtual infrastructure. What I mean by this is that they will start to compete for the same resources that your critical business applications need and therefore will starve them of the computing power they need.

Virtualization is a dynamic framework and as machines and applications move around this infrastructure it is important that any security measures are maintained. Systems designed to protect these valuable resources often do not adequately maintain the security state of these machines. The end result is that when machines or applications move they may not be protected at all.

Virtualization security solutions are only part of the answer another element to this story is “Human Traits”. History has shown that despite best intentions, processes, responsibility of vendors in supplying patches we are still hearing stories today of systems being compromised and often in the simplest of ways. Despite have token based access control systems people are still using password management systems with weak passwords.

Why? Because it’s easy and convenient, “why bother with all the extra cost I’ll never get hacked”!

People are complacent, even the experts! This isn’t good enough, we should be better at this and some people are but there are a lot of organizations who aren’t.

When was the last time you looked at who was accessing your virtual infrastructure? or what is going on and by whom? Do you keep a good audit trail? What visibility do you have?

Worried? Not bothered? Think this is a load of FUD propaganda! Well maybe it is, but then again perhaps not. When SQL Slammer hit on Jan 25th 2003 it caught everyone by surprise, it spread with extreme ease and speed globally, it effected some 75,000 victims in 10 minutes! A patch had been released some 6 months previously by Microsoft.

History has an uncanny way of repeating itself don’t become complacent, take the time and effort to monitor what is going on in your virtual world.

VMware have provided a patch for this details of which can be found here:

[Security-announce] VMSA-2011-0002 Cisco Nexus 1000V VEM updates address denial of service in VMware ESX/ESXi

As well as carrying out threat intelligence and risk modeling organizations must when auditing and monitoring virtualization infrastructure look at relevance and impact to the business operations of the organization before applying any kind of policy. This must be a regular and on going process as the dynamic nature of virtualization means organizations can no longer afford to be complacent when aiming to provide and build a solid virtualization auditing and monitoring program.

You can’t protect what you can’t see!

If you do anything at all when it comes to virtualization security make sure you do one thing right, provide your business with visibility!

With this in mind I set out to write a short document that would hopefully impart to the reader practical advice on how to secure their virtual environment. You can check out the document at the following link (A Practical Guide to Securing Your Virtual Environment), if you like it or even if you don’t please let me know by leaving your comments below…(spiv)

Thought I would share this presentation with everyone from a recent seminar that I gave at the back end of last year entitled “Security in a Virtual World”.

You can check out the presentation at this link: http://bit.ly/8Vh4MM