SAN or Storage Array Offline? What to Do First

A storage array failure is different from a single server failure, because everything sitting on that storage goes down together: virtual machines, databases, file shares, sometimes the backup repository as well. The pressure to get it back quickly is enormous, and that pressure is exactly what leads to the steps that make recovery harder.

Record the array's own status messages, which disks it reports as failed or foreign, the order the disks sit in, and any recent changes such as a firmware update or a reconfiguration. Photograph the management console and the physical disk bays. This costs a few minutes and is often the difference between a straightforward reconstruction and guesswork later, because the array's own view of its layout is the thing at risk.

The instinct when an array reports a failed disk is to replace it and let the array rebuild. When the array is merely degraded and the rest of the disks are healthy, that is usually right. When the array is already offline, has lost more disks than its redundancy allows, or is reporting members as foreign, a rebuild is a different proposition: it writes across the remaining disks, and if the layout information is already wrong, it can write the wrong data over data you needed.

This applies to vendor support too. Vendors are generally excellent at restoring service, which is not the same goal as recovering data. If the data on the array matters more than the hours it takes to assess, say so explicitly before anyone initialises, reinitialises or rebuilds anything.

Pulling disks to inspect them and returning them to different slots can leave the array unable to identify its own members. If disks must come out, label each one with its slot number first. The same applies to moving disks to another chassis: without the original order, reconstructing the layout becomes significantly harder.

A LUN that has disappeared, or a volume that suddenly shows as unformatted, does not usually mean the blocks underneath are gone. It more often means the metadata describing where the volume starts and how it is laid out has been damaged or overwritten. Do not create a new LUN or volume in its place, and do not accept a prompt to initialise, since that writes new structures over the ones needed to find the old data.

Before planning recovery, establish honestly what else depended on this storage. If the backup repository lived on the same array, or the virtual machines running your backup software were hosted on it, the recovery plan is different from one where backups are intact on separate hardware. It is better to find this out deliberately in the first hour than to discover it after a restore attempt fails.

If the failure followed a firmware update or a configuration change, note the exact version and what changed. Some storage layout problems after an update are recoverable by understanding what the update altered, rather than by treating the data as damaged. Rolling the change back without understanding it can compound the problem.

Ask a specialist before a rebuild when the array is offline rather than degraded, when more disks have dropped than the redundancy allows, when a LUN or volume has vanished, or when the backups lived on the same storage. Recovery is done from images of every member disk, with the array reconstructed virtually, so the original disks are read rather than written to. Outcomes depend on the state of the disks and what has already been done to them, and we say what looks realistic after an assessment.