As many of you are well aware, the wiki is currently down. For those of you that haven't been around during my IRC explanations, I'll briefly explain the situation in a more public area:
<span style="color: #bbbbbb;">
In an effort to clean up our MySQL database and rid ourselves of antiquated tables from 2004 - I began what would soon be known as Suck Day For The DJB late on Tuesday night, soda in hand...sunflower seeds close by.
I began - as any good planner when attempted a mass deletion of about 70 database tables - with a database backup. With a gleam in my eye and 2 sunflower seeds in my mouth, I set off to make ANOTHER MySQL database to copy all the tables to in case something went awry. I soon found out that I was not able to add any additional databases to the DJB site due to the limits of our hosting plan. I swigged some soda and moved on to Plan B...generating my own backup file.
Navigating deftly around phpMyAdmin, I managed to generate a hefty backup file for the database I was working on. Life was good. I ate 12 sunflower seeds (spitting out the shells, of course).
With the backup safely on my desktop, my mouse danced across the screen as I checked boxes next to tables - sentencing each to their doom. My hands became slick with sweat from the shear effort of sealing the fate of so many tables. I took a breath. I took a drink. I clicked "Drop."
Knowing that my clicking skills were top notch - and if any foul up were to happen, I always had the backup - I didn't read over the list of tables (there were a lot) and promptly confirmed their death.
Work complete. I sighed in relief. I ate 34 more sunflower seeds. I drank from my seemingly endless glass of ice chilled soda.
My ears perked up at the sound of...what was that? A private message in IRC.
"Yay. I'm popular!" I thought to myself as I Apple+Tabbed to my IRC window (yes, I use a Mac).
A message from Shadow: <span style="color: cyan;">"the wiki is f'd."</span> (paraphrased)
My hands flew over my keyboard and mouse as I navigated to the database. I looked at the tables. Was something missing? I checked my own personal installs of MediaWiki (the software we use for DBWiki) and counted the tables and compared. The numbers didn't jive.
My eyes sprang open, sunflower seeds rocketed from my mouth and hit my wife, soda gushed from my nose wetting my pants.
Then I gathered my senses. "No worries," I thought, "I have a BACKUP!"
As if possessed by 12 coder ninja monkeys, I knew that those tables would be easier to recover with a full restore. I purged the tables and got to work on the restore!
I looked closer at the backup file. It was truncated partway through. Despite the successfull downloading of the backup file...phpMyAdmin had given me a partial one that cut out partway through an insert statement.
Not good. Not good at all.
So that's that.
This is where we are now. Jac is trying to get a backup from our server admins (who do a backup on a nightly basis). If that all works out - and there is a 99% chance that it will - we will lose anywhere from 5 to 24 hours worth of data. If a backup cannot be restored by them, we may be reverting to an older copy of the database which will set us back further than would be desirable.
This predicament grew from an effort to make our site better but has turned into something frustrating/worrisome for everyone. For that I am sorry, as I was the one that clicked the delete button.
Has everything been lost? Definitely not. But some data will be unrecoverable. Stay tuned and cross your fingers that we have only lost a day's worth of data.
You need to be logged in to post comments