UPDATE: RWG IS BACK!!!!!
Okay, the re-configuration went well and all appears to be working as it should be, so the doors are back open again. Fingers crossed it's for good this time!
By the way, I'm afraid we've lost some more posts and PMs due to the database corruption as I've had to roll back to a backup from last Thursday morning. Apologies for the inconvenience, but with luck this will be the last time.
Thanks for your patience, folks!
RWG IS DOWN AGAIN!!!!!!!!
Okay, this is the current situation as of 20.13pm Sat 10th Sept 2011 (UK time)
Board had been running okay. I did a database backup on Thursday, which was fine.
Ran another backup this afternoon and it brought the system to its knees - used up all RAM and CPU cycles. It seems that the problems with the old server went back further than I had thought and there was a case of creeping db corruption with the backup I rolled us back to last week.
insert rude words here
okay, so here's what's happening. Every attempt I've made to backup today's database has crapped out the server. The database is in a sufficiently bad state that none of the diagnostic tools - command-line or otherwise - that I have available to me online will access it at all. In addition, whilst, oddly, the board itself can load, the admin area is completely hosed due to a corrupt table.
So. Here's what I'm doing right now. I've got a 36-hour old copy of the database. I've currently got it loaded into mysql on the machine I'm typing this on and I'm going through it with a fine toothed comb (at least, as far as you can with 1gb of sql data. I've so far found damage to two crucial database tables and have fixed it. I'm looking for more...
Tomorrow, I'm not going to mess around. I'm going to re-image the server (mainly because I took the default configuration last time for the sake of speed of setup - it was the wrong choice, so I'm going to go for something a bit more appropriate and a lot more familiar this time). I will then upload the (hopefully) fixed database, forum software and other stuff. I'll then pray REAL hard.
Hopefully we'll be back up sometime tomorrow.
I gotta say, I've been doing this kinda thing for a fairly long time and this is by FAR the most difficult set of issues I have ever had to deal with. Wish me luck - I'm going to need it!
Thanks for your patience, folks and apologies for the downtime!
Greg_r
Admin: replica-watches-guide.com
UPDATE: 21.54pm 10 sept 2011
Okay, we're making progress. I have a copy of the database from Thursday sitting in a mysql installation on my server at home. After some serious hacking of the db in a text editor, phpmyadmin and mysqlcheck are finally reporting that the database is 100% clean. I want to believe them, I really do....
I also have a backup of the actual board software and associated garbage sitting on my desktop. I'm currently waiting for a re-install of the server software to complete (using a more familiar linux distro and control panel than last time). Once that's done, I'll set up the DNS, point the domain name at a parking page and quit for the night.
Tomorrow I'll import the database into the new config and run some more checks. If everything passes, I'll upload the main board files and test some more.
Watch this space for updates.
------------------------------------------------------------------------------------------------------------------
OLDER EVENTS:
Starting a new thread for this so that the info doesn't get lost in the noise in the other thread. Please keep this thread on topic to make it easy for folks to find out what is happening. I will edit this post with status updates as appropriate.
----
Okay. Here's the deal.
RWG's server is toast. Turns out that there was a problem with the disk controller (which was my original suspicion, but they assured me otherwise at the time - oh well, these things happen).
The hosts are replacing the hardware. However, the problem means that I now have to re-install everything from a backup. If I'm going to do that - given that we're in need of an upgrade anyhow - I might as well go ahead and do the planned server upgrade now (3 months ahead of original intentions), so we're moving onto a shiny new box that's going to be a lot more powerful than the old one. That's the good news.
The bad news is that due to the disk fault, the site has gradually become more and more corrupted since the big crash on Monday. This means that to be sure the site continues to be stable and reliable, I have no choice but to roll back to last Sunday again - I have daily backups since then, but it seems likely that they will be partially corrupt and I therefore can't risk using them. This means that all posts and PMs for the last three and a bit days are going to be lost.
I'm really sorry to do this to you guys, but if the site is going to be stable for the forseeable future, we can't risk corruption in the database. Better that we take a hit of a few days posts now than to lose a lot more data down the road.
Timescale: It's going to take at least 24 hours to provision the new server and could be longer. Last time I did this with this particular host, they provisioned the server in about 18 hours, but they quote anything up to a week. After that, I need to re-install the board software, import the old database and repoint the domain names to the new hardware. Realistically this is going to take several days.
I will post more info here when I have it.
Edit: By the way, the current error page will disappear within the next 24 hours as I'm having to move the domain name over. It will point at some random parking page for a short while until I get it directed to the right place. Do not panic - this is a normal procedure
UPDATE: Thurs evening
Okay, I'm going to quit for today. The server has been provisioned (in double-quick time, yet) and whilst an error page is all we have, it is at least running on the new server
I'll start fighting with mysql in the morning. LOTS of data to upload, which is going to take several hours.
UPDATE 2 - 9:30am Friday
Okay - this is going faster than expected. Server is up and running with RWGshop and our portal site. As I type this I'm waiting for a sql dump the board database to upload, which is going to take at least a couple of hours. At that point I can start re-configuring the board to work at its new location, assuming that the database imports properly (fingers crossed!). I'm not promising that we'll be up today, but if everything goes well, it's just about possible. I'll post again when there's more progress.
Okay, the re-configuration went well and all appears to be working as it should be, so the doors are back open again. Fingers crossed it's for good this time!
By the way, I'm afraid we've lost some more posts and PMs due to the database corruption as I've had to roll back to a backup from last Thursday morning. Apologies for the inconvenience, but with luck this will be the last time.
Thanks for your patience, folks!
RWG IS DOWN AGAIN!!!!!!!!
Okay, this is the current situation as of 20.13pm Sat 10th Sept 2011 (UK time)
Board had been running okay. I did a database backup on Thursday, which was fine.
Ran another backup this afternoon and it brought the system to its knees - used up all RAM and CPU cycles. It seems that the problems with the old server went back further than I had thought and there was a case of creeping db corruption with the backup I rolled us back to last week.
insert rude words here
okay, so here's what's happening. Every attempt I've made to backup today's database has crapped out the server. The database is in a sufficiently bad state that none of the diagnostic tools - command-line or otherwise - that I have available to me online will access it at all. In addition, whilst, oddly, the board itself can load, the admin area is completely hosed due to a corrupt table.
So. Here's what I'm doing right now. I've got a 36-hour old copy of the database. I've currently got it loaded into mysql on the machine I'm typing this on and I'm going through it with a fine toothed comb (at least, as far as you can with 1gb of sql data. I've so far found damage to two crucial database tables and have fixed it. I'm looking for more...
Tomorrow, I'm not going to mess around. I'm going to re-image the server (mainly because I took the default configuration last time for the sake of speed of setup - it was the wrong choice, so I'm going to go for something a bit more appropriate and a lot more familiar this time). I will then upload the (hopefully) fixed database, forum software and other stuff. I'll then pray REAL hard.
Hopefully we'll be back up sometime tomorrow.
I gotta say, I've been doing this kinda thing for a fairly long time and this is by FAR the most difficult set of issues I have ever had to deal with. Wish me luck - I'm going to need it!
Thanks for your patience, folks and apologies for the downtime!
Greg_r
Admin: replica-watches-guide.com
UPDATE: 21.54pm 10 sept 2011
Okay, we're making progress. I have a copy of the database from Thursday sitting in a mysql installation on my server at home. After some serious hacking of the db in a text editor, phpmyadmin and mysqlcheck are finally reporting that the database is 100% clean. I want to believe them, I really do....
I also have a backup of the actual board software and associated garbage sitting on my desktop. I'm currently waiting for a re-install of the server software to complete (using a more familiar linux distro and control panel than last time). Once that's done, I'll set up the DNS, point the domain name at a parking page and quit for the night.
Tomorrow I'll import the database into the new config and run some more checks. If everything passes, I'll upload the main board files and test some more.
Watch this space for updates.
------------------------------------------------------------------------------------------------------------------
OLDER EVENTS:
Starting a new thread for this so that the info doesn't get lost in the noise in the other thread. Please keep this thread on topic to make it easy for folks to find out what is happening. I will edit this post with status updates as appropriate.
----
Okay. Here's the deal.
RWG's server is toast. Turns out that there was a problem with the disk controller (which was my original suspicion, but they assured me otherwise at the time - oh well, these things happen).
The hosts are replacing the hardware. However, the problem means that I now have to re-install everything from a backup. If I'm going to do that - given that we're in need of an upgrade anyhow - I might as well go ahead and do the planned server upgrade now (3 months ahead of original intentions), so we're moving onto a shiny new box that's going to be a lot more powerful than the old one. That's the good news.
The bad news is that due to the disk fault, the site has gradually become more and more corrupted since the big crash on Monday. This means that to be sure the site continues to be stable and reliable, I have no choice but to roll back to last Sunday again - I have daily backups since then, but it seems likely that they will be partially corrupt and I therefore can't risk using them. This means that all posts and PMs for the last three and a bit days are going to be lost.
I'm really sorry to do this to you guys, but if the site is going to be stable for the forseeable future, we can't risk corruption in the database. Better that we take a hit of a few days posts now than to lose a lot more data down the road.
Timescale: It's going to take at least 24 hours to provision the new server and could be longer. Last time I did this with this particular host, they provisioned the server in about 18 hours, but they quote anything up to a week. After that, I need to re-install the board software, import the old database and repoint the domain names to the new hardware. Realistically this is going to take several days.
I will post more info here when I have it.
Edit: By the way, the current error page will disappear within the next 24 hours as I'm having to move the domain name over. It will point at some random parking page for a short while until I get it directed to the right place. Do not panic - this is a normal procedure
UPDATE: Thurs evening
Okay, I'm going to quit for today. The server has been provisioned (in double-quick time, yet) and whilst an error page is all we have, it is at least running on the new server
I'll start fighting with mysql in the morning. LOTS of data to upload, which is going to take several hours.
UPDATE 2 - 9:30am Friday
Okay - this is going faster than expected. Server is up and running with RWGshop and our portal site. As I type this I'm waiting for a sql dump the board database to upload, which is going to take at least a couple of hours. At that point I can start re-configuring the board to work at its new location, assuming that the database imports properly (fingers crossed!). I'm not promising that we'll be up today, but if everything goes well, it's just about possible. I'll post again when there's more progress.
