Pixar’s Billion-Dollar Delete Button Nearly Lost Toy Story 2 Animation (Source: Pixabay.com)
We’ve all been there. An email, spreadsheet, project proposal, image or essay you’ve been working on for hours is lost following a sudden power loss, accidental file deletion or a saving error. It’s exasperating as you think about all the time and effort that has gone down the drain thus forcing you to start over. Now if you think that is a disaster, imagine losing an entire film.
It may sound like the stuff of fiction (no pun intended) because one would assume given the infrastructure and financial resources the world’s biggest film production and distribution studios have behind them, there would be adequate systems, processes, and safeguards that make it difficult for this to happen. Yet, permanently deleting an entire film is a predicament that leading animation film studio Pixar Animation Studios (now a Walt Disney subsidiary) almost found itself in. So how did it happen?
Command Executed in Error
In 1998, just months before Toy Story 2 was scheduled for release, one of Pixar’s employees erroneously executed a command that started to wipe out everything from the file system. It was meant to be a routine Unix file system cleanup but the haste with which they entered the instructions led to a flurry of unintended consequences.
Instead of running the command in the specific directory they wanted to clear of unwanted files, they had done so at the Toy Story 2 server’s root. The system was therefore tracking and deleting every single Toy Story 2 asset on the server.
The first signs were picked up during a meeting of three project executives as they looked at the server directories containing Toy Story 2. They saw Woody’s hat slowly disappear, followed by his boots before Woody himself faded from view. A similar fate awaited Rex, Hamm, Mr. Potato Head and Buzz Lightyear.
They soon noticed that each time they refreshed the directory, there were fewer and fewer files. Puzzled by what was going on, the crew decided to shut down all machines and proceed to lunch. On their return, they realized that only 10 percent of the animation’s files remained.
Backup Tapes Anticlimax
Staff were now starting to appreciate just what was unfolding before their very eyes. But the crew remained calm as they expected anything that could have led to the loss of files in the production environment could easily be remedied by restoring data from tape backups.
Indeed, the team retrieved the backup tapes, restored the project’s digital assets, ran a couple of validation tests, got the pictures back and no longer experienced errors. Initially, all seemed well and the crew was confident that they could continue working on the film where they’d left off before. But the tranquility didn’t last long.
It turns out Pixar didn’t regularly test their backups and therefore failed to realize that a critical system error had prevented their backup protocols from accurately and completely archiving the film’s assets. After several days working on the restored files, the crew began to comprehend the magnitude of the nightmarish scenario before them. The restored files were corrupt and incomplete.
A Stroke of Fortune Saves the Day
It was time to make some difficult decisions. In a pivotal meeting held by Pixar and project executives, a verdict had to be made on whether to scrap, restart, reconstruct or delay the film. In this somber atmosphere, a wonderful stroke of good luck emerged. Here’s how data backup saved the Toy Story franchise.
Apparently, one of the staff that contributed to the project had saved a copy of the animation on her computer when she worked from home as she took care of her newborn baby. If this backup existed, it was arguably the last shot at making sure Toy Story 2 could be completed on time or, at worst, only marginally delayed.
The team embarked on the delicate process of retrieving the data from the employee’s computer and establishing whether it could indeed be the savior they hoped for. The first step was comparing the employee’s copy with a much older but good quality tape backup from two months back. There were plenty of inconsistencies between the two that made it extremely difficult to determine which provided the best place to start.
For this reason, the team had to embark on the painstaking process of assembling a new copy by manually interrogating and choosing the best asset one file at a time. This process meant going through about 100,000 files. Roughly 70,000 files could be easily compared and verified. For the remaining 30,000 or so, the crew had to put in 8-hour shifts over a weekend to finalize the exercise. As a result of this backbreaking team effort, Toy Story 2 was back on schedule.
A Multimillion-Dollar Lesson on Data Protection
Toy Story 2 cost nearly $100 million to make and eventually raked in more than half a billion dollars. This multimillion-dollar investment and the resultant revenue was on the verge of vanishing into thin air.
It’s a dramatic example of just how fundamental data protection is to everyday organizational operations. The loss or corruption of critical data can have profound repercussions. In the absence of a robust data protection regimen, recovery is an arduous and stressful journey whose success is never guaranteed.
Toy Story 2 may never have made it to the big screen (at least in its original form). Yet, all it would have taken to quickly mitigate the disaster would have been to devote a couple of minutes each day to testing tape backups for integrity and availability. Fortunately, information technology has come a long way since the 1990s. Backup testing can be automated thus allowing the checking of backup quality in near real-time.
Another lesson from Pixar’s averted catastrophe is the need to manage system permissions. At the time, staff working on Toy Story 2 had fairly liberal access to the file system. A data security disaster was simply waiting to happen.
It would have been far less likely to occur if users were restricted to changing and manipulating only the files and folders they required to do their job. It may have come across as micro-managing the team but would have made it much harder for someone to unintentionally sabotage the project.