Another Crash

The same system impacted in April from a lost SAN went down again yesterday. The VM file vanished after the operators performed a routine system restart. We believe something else occurred and the operators did more than a routine reboot. As frustrating as the news was, the server admins rebuilt the OS and I restored Oracle - the system was up and open for business in 4 hours when the call was made to recreate the environment. The db was restored to the point of failure too. I am in the process of restoring the final integration components today.

I used the same notes I put down in the page: Database Fever

Virtualised Environments and Shared Storage

In my experience, the uncontrollable database disasters outside of the DBA's direct control have been to do with SAN failures. Mixed with virtual machines and less than adequate forward thinking can result in a system setup that fails on a long enough timeline.

SANs are feasible storage solutions that offer convenient storage with decent disk redundancy, however, what I've found is that some administrators can setup one production server on one VM and then setup either the DR or "offsite" backup server on another VM without realising they've setup both the production and DR VMs to function on the same SAN. In the event that the SAN fails, both VMs are lost and the whole disaster recovery is a failure and the organisation is then dependent on either a week old tape backup or, by chance, a deliberately copied database backup - either from a recent database refresh or for testing. The same can be seen with some Oracle RAC installations - both nodes are virtualised yet they've been implemented on the very same disk array outside of the ASM storage. So if the SAN dies - both nodes are going to fail thus defeating one of the main incentives of using RAC.

I write specifically about SANs but it is possible the same arrangement exists on any disk array used to house a multitude of virtual machines. If it's believed that SANs and disk arrays are safe from failure, one may have not accounted for potential administration error where server engineers can potentially destroy partitions or neglect periodic disk checks leading to a slow collapse of a disk system.

What this means is that VMs and SAN structures tend to be obscured behind layers of virtualised machines and disk partitions and I find that it is in the organisationsbest interest to question and document the underlying foundation of the disk structure. If the engineers have setup the system in this fashion, I make alternative arrangements for the backups to be synchronised to disk that is not part of the production disk.