Virtual machine backup in VMware vSphere 5.1 environment is normally a pretty specialist field and one that, let's face it, is hardly likely to get the pulse racing.
Which makes it all the more amazing that this little backup backwater exploded into acrimony after a Symantec employee appeared to hurl a calculated insult in the direction of Veeam, a provider of – you guessed it – backup software for vSphere 5.1.
It started when VMware released a knowledgebase article relating to vSphere 5.1 titled, "Third-party backup software using VDDK 5.1 may encounter backup/restore failures."
It describes "an issue with the VMware Virtual Disk Development Kit (VDDK) that may cause backup and restore operations to hang or fail. Third-party backup vendors that are using the VMware VDDK may encounter backup or restore issues when backing up VMware vSphere environments."
Symantec, which doesn't currently offer a vSphere 5.1 hypervisor solution, blogged about this, saying "any 3rd party vendor depending on the current API cannot perform consistent backups and cannot ensure a reliable recovery point," and a tweet from the official Backup Exec Twitter account said that, "Any vendor claiming support for vStorage 5.1 API CANNOT perform consistent backups."
This left Veeam fuming because its backup solution uses the vStorage 5.1 API, and yet Doug Hazelman, Veeam's chief evangelist, says the company's product does perform consistent backups. That's because Veeam was aware of similar bugs in the past and architected its product to expect and work around them.
"Symantec was using this issue as an excuse for not supporting 5.1, and was using scare tactics against other vendors that do," says Hazelman. "They said any vendor claiming support is lying. We were getting calls from customers within 30 minutes asking us what was going on because Symantec had been spreading fear, uncertainty and doubt." (Note: Symantec has apparently updated the wording of its blog post to say that "any 3rd party vendor depending on the current API may be impacted by these issues.")
In simplified terms, the bug can cause VDDK calls to hang, and Veeam's architecture uses a watchdog process so that if a hang does happen, the process is simply killed without affecting the job. "With an alternative architecture not featuring such isolation, VDDK code hang will cause the whole backup and restore "job" to hang just as the VMware KB article explains," says Hazelman. "Think of it like a maze. If you hit a dead end, some people give up and stop, but with our architecture we try another way till we get there."
Veeam was apparently not prepared to take Symantec's perceived slight lying down, preferring to throw a bit of FUD right back in Symantec's direction. "If they can't figure out support for 5.1 due to this bug, then the issue is the other bugs in the VDDK storage API, which Veeam works around but which Symantec apparently does not," says Hazelman. His conclusion is that Symantec customers need to go back and verify the recoverability all of their VMware VM backups, as per Symantec's own tweet.
Now backup is a fairly serious business, and you may have your own opinion on the advisability of relying on a Veeam-style workaround for buggy software. Symantec's blog calls this "a more cavalier 'ship-and-fix' data protection approach than Symantec is willing to accept."
But what about the bugs in previous versions of the API that Hazelman mentioned? Was Symantec aware of any, and if so did any of its backup solutions call for a similarly cavalier approach in order to work? Or might they really be as unreliable as Hazelman suggests they could be? Sadly Symantec has not responded to my email — make of that what you will.
The serious side of this is the fear, uncertainty and doubt that an incident like this generates — no matter who (if anyone) is at fault and who (if anyone) is to blame. The inner workings of server virtualization technology is still quite mysterious, and the last thing anyone needs are lingering doubts about the integrity of their virtual machine backups.
Paul Rubens is a technology journalist and contributor to ServerWatch, EnterpriseNetworkingPlanet and EnterpriseMobileToday. He has also covered technology for international newspapers and magazines including The Economist and The Financial Times since 1991.