Server migration is complete!
We've now tested our main server out for a full month -- it has been running very well.
I was on the verge of moving the server over last Sunday but had second thoughts with the storm (and our datacenter's mixed record with losing power). Sure enough, I got an e-mail Monday morning saying that they had lost power and they were operating off their battery backup and their generator (which was actually great news, since it was their generator / transformer system that had failed last time).
Now I think we are truly and finally very ready to move back to our main server (which will also help us out somewhat financially, since the server we are on now is expensive). I'll likely do this over the weekend.
Among the backup measures that will be in place after the move:
--Live, real time replication of the database to an offsite server (a instant carbon copy of the database) through a secure, encrypted SSH tunnel. Also, a daily synchronization of the server code, including all uploaded member files (such as avatars). The offsite server is also backed up daily by a commercial provider. This will enable a very quick switchover in case the main server goes offline due to a failure at the datacenter.
--Primary database storage is on a solid state drive, which is significantly better at withstanding physical trauma than a mechanical drive (the primary advantage, however, is random access speed -- it is about 5-10x faster, which will help reduce loading time)
--Twice daily database backup to a (brand new) mechanical hard drive (providing a historical record of several days worth of data, with at least a dozen or so once a week backups)
--Encrypted backup to a USB drive dangling from the server itself
--A regular backup of the server configuration itself, which will make migration to a new server easier in the event this is necessary
You should notice a small speedup once we make the switch, but the difference in capacity will be much more important once we start allowing new members to join the site again.