Sometime this morning before 9AM CST the database on the server started having issues with the table that houses all member photos, videos, and sounds. After several hours of attempting to rebuild the table to no avail, it was determined that we would have to go back to a 2 week old backup.
Everything is up and running again. Folks who posted photos, audio files, or video files after 9/1/03 will need to re-post them. All other data was unaffected.
We will be running daily preventative maintenance checks on the database for the next several weeks to ensure this doesn't happen again.
If you have any questions regarding this incident, please contact us via the support page.
Isn't two weeks a long time between backups? Or is this a normal amount of time?
Hi Bob,
Here's a little background to the framework of the site:
SuperFord.org consists of nearly two dozen database tables that are used to keep track of all of the data for the site (user accounts, photo albums, files posted by members, forum posts, club registries, traffic monitoring, album sections, etc). In most cases, each of these individual tables are very small and are backed up every other day as well as once per month.
The strategy with a backup system is to also keep a history of backups. For example, you don't want to just keep backing up a table each day and writing over previous backups. The reason for keeping a history of backups is in case something happens on Thursday during the week that causes problems on Friday. Going back to the backup from the night before wouldn't be a good solution because the data was already compromised then. For these situations, you would want to go back to a backup earlier in the week.
Since a site like this is visited by 10's of thousands of people per month, we also keep a backup for the entire month. This covers all of our bases in the event that something creeps up that was caused much earlier in the week or earlier in the month.
Now, with regards to the 2-week old backup of the media files. This is a special case where it is difficult to make daily/frequent backups. When database tables are backed up, the safest way to do this is locking out the table which means nobody can access the data. The media table is currently about 12 Gigabytes in size and grows at an exponential rate. To keep daily backups of this would require a lot more hard drive space than is currently available on the server. It would also require locking members out of viewing/posting media files each day for at least 1 hour while the backup runs.
So, the reason why we don't keep daily backups of the table that houses all posted media from members is because it is currently impossible with this server. In the nearly 2 years that we've been running this database structure, this is the first corruption problem we've had which would suggest that situations like these are highly unlikely to occur. With the new database maintenance procedures we've implemented as of yesterday, we also feel that something like this will never happen again unless a hardware error creeps up that is impossible to avoid (i.e. crashed hard drive, etc). The SuperFord.org site runs on a RAID hard drive setup which means in the event that a hard drive does crash, it can rebuild itself automatically w/o risk of data loss when a new drive is installed.
In situations like this, preventative maintenance is what keeps the site alive and running w/o downtime.
Thanks for explaining it Eric. Was there another problem? I uploaded images this morning, and they are gone now. Did you need to go back to a different backup again?
Thanks for explaining it Eric. Was there another problem? I uploaded images this morning, and they are gone now. Did you need to go back to a different backup again?
You're quick -- we reverted back to last night's backup this morning due to an error we found while optimizing the database. It was one of those instances where we should have complied with the "if it ain't broke, don't fix it" motto. We will save further database optimization procedures until the server is upgraded when extended downtime can be planned.
Did this have adverse affects on the movies we have uploaded? For somereason I can't view any of mine anymore. They are there, but it always says the format is wrong now or something...
Did this have adverse affects on the movies we have uploaded? For somereason I can't view any of mine anymore. They are there, but it always says the format is wrong now or something...
Posted:9/19/2003 16:46 
