For quite a while now we’ve been using VMware Virtual Infrastructure with the bells and whistles of Vmotion and HA on a Fibre Channel SAN. In my eyes it’s all been very good in that it gives us a resilient cluster to run our VM’s and therefore services on. Unfortunately, it’s not necessarily cheap, not least of all because of ongoing licensing costs, hardware maintenance cost and the inevitable upgrades. In an ideally world we all want the shiny shiny new solution, but if your on a budget of zero, then I’m going tell of how you can use older servers to create a SAN and Virtualisation solution. It is based on how I’ve used a solution myself on a number of older servers for running development and testing machines on.
The Virtualisation Solution
Well firstly It’s quite a while since I mentioned about a limited version of VMware ESXi being freely available. This is still available even with the latest vSphere 4.0 Hypervisor software – which I’m happily running on a small number of servers. It’s a little bit more fussy that ESXi 3.5 was, for instance you must now have 2Gb+ of RAM, and it no longer supports 10/100 Nics. Realistically, you would want something fairly modern anyway. Ideally you want a new server with Quad Core’s and HyperThreading, at at least 16Gb RAM, and ideally something supported by VMware. If not then check out the WhiteBox HCL
If your going to re-use an existing server, then you need to ideally be using something with either Dual or Quad Core processors, rather than just HyperThreading alone, 2GB of RAM is the absolute minimum for ESXi 4.0 to run, but you want nearer 8Gb to actually get any good use of it. You will also need at least one copy of ESXi (either 3.5 or 4.0) with the Free license.
If your only going to have just one server then make sure it has a RAID subsystem that is supported by VMware and you then simply store the Virtual Machines on it. You will have to accept the risk that you have a lot of eggs in this single server. Make sure you do reliable backups of each servers OS (as you would normally for any physical server)
If your going to have more than one server, then consider having two servers to run VMware and some kind of SAN or NAS, as you don’t want to have to rely on the local storage of each of those Virtualisation servers, but instead using the shared storage, will allow you to manually share the VM’s across the servers. Remember you can put the VMware ESXi Hypervisor onto a USB stick and boot the server from that, it if supports it.
So what about a SAN or NAS Solution?
Well you want something that works over a LAN so iSCSI or NFS are your options. In an ideal world you want something that is fully supported by VMware, but again it’s not always possible to afford this. You can look at higher of the SOHO grade NAS devices that can do RAID and either iSCSI or NFS.
However, if you have a server that has a fair amount of storage or that you can load up with storage cheaply, think of re-using the storage from your Virtualisation servers above, then you can use that as a Shared SAN/NAS solution. I highly recommend OpenFiler for the job of a SAN/NAS server box. It’s actually all a Linux distribution (based on rPath Linux), but please don’t let that put you off ! It has a very good web interface, and it’s very rare that you need to get into the Linux command line. RAID generally isn’t an issue with OpenFiler, either use the built-in RAID on the server (if it’s supported by Linux) or if you don’t have RAID then you can use Software RAID under Linux, this can be configured in the OpenFiler installer and in the web interface.
See here for how to set it up using iSCSI to present storage to VMware ESXi, you can also use NFS instead of iSCSI to present storage to VMware. It’s swings and roundabouts as to which is best;
An iSCSI setup presents the disks (aka a LUN) to VMware it allows you to format using VMFS (VMware’s File System) and with VMware ESXi / vSphere you can then use features like thin provisioning. However, there’s no way of seeing the files outside of VMware ESXi, which can make backup and recovery more challenging.
If you use NFS then you can see the files outside of VMware. I’m guessing here, but with an LVM Snapshots, you could probably use that and create some scripts within VMware to mount a read-only Snapshot and take backups of your VM’s file from undernath. So they won’t be crash consistent, but you’d have something to work from….
Some considerations
(which are probably relevant for any SAN and Virtualisation deployment)
- VMware support’s VLAN Trunks/tagging of traffic into VLAN’s, make use of it, and use your additional server NIC’s either to increase bandwidth or to improve resilience, rather than dedicating a separate NIC to each VLAN.
- Use a separate private VLAN for iSCSI/NFS traffic to your SAN/NAS
- Ensure that all connections are at 1Gb/s +
- Ideally use a pair of network switches for resilience (or separate cards on a chassis based switch)
- Try to use dual power supplies, ensure they are on separate power strips, UPS’s
- Backups – ensure you backup the machines fully. You could also routinely power down the VM’s and Export them as an OVF Template
Finally…
Well hopefully, if you used this as a guide you might have enough info to setup a small scale solution of some description, even just for development and testing as I’ve done or even a solution to start you on the road to virtualisation. Even if you use it for what Sales people call ‘the low hanging fruit’!
If you want comparison of some systems that I’ve used for various solutions using VMWare ESXi 3.5 and 4.0 then here goes..
Permanent development and testing solution
2 xDell PowerEdge 1850, P4 Xeon 2.8Ghz, 6Gb RAM (originally had 1-2Gb when I first started running ESXi on them), 2 x 72Gb Disks (no RAID – “eek!”). One server running vSphere 4.0, other running ESXi 3.5. Each will run about 8 VM’s.
1 x Dell PowerEdge 750, P4 2.4Ghz HT, 1.5GB RAM, 2 x 160Gb (Software RAID), Running OpenFiler (32-bit)
Single Server Solution
1 x HP ML110, P4 3.0Ghz HT, 5GB RAM, 2x 500Gb S-ATA HDD, Adaptec RAID controller. Will also run about 6 VM’s.