EMERGENCY! CRASHED SERVER - BACKUP CORRUPT!

[Pugglewuggle]

Hi, Windows Server tanked last night and our mail server went down. I was lucky enough to get AMS back up to run a full Export (backup) and save the file.

I then reinstalled the server with Windows Server 2008 and tried to Import the file - I get this error:

"Subset: All. Unable to read section from backup file."

Please see the attached screenshot.

I need this fixed immediately! I'm up for a Skype call if this can help get this fixed. Please let me know. Rob and Chris: you will need to message me on here - my email is down so I can't receive messages that way.

Error

ams.jpg (45.76 KiB) Viewed 17534 times

[Code Crafters]

This error usually means that one of the backup parts to the file is damaged. Try to read each subset one at a time by ticking only one box at a time. This should allow you to get any undamaged parts of the backup file. Its possible the whole file was corrupted if done during the crash. You should, however, be able to just copy the "C:\Program Files\Code-Crafters\Ability Mail Server 2" folder from your old mail server installation to copy the whole mail server accross assuming you haven't already overwritten that with a new Windows install or the hard drive being inaccessible after the crash somehow.

It's generally good practice to backup your mail server configuration to another hard drive or onto CD periodically. we make a CD once a week along with our code changes in case any computers / hard drives fail. After the initial release of AMS3 next year we are planning to add an event scheduler to auto-backup using the export tool as often as you choose.

[Pugglewuggle]

Hi Chris,

I managed to get the server back up through a 29 hour process of rebuilding partitions and recovering corrupted files. I basically ended up copying the old installation from Program Files as well as the account files from the RAID array (once I got both drives rebuild and cleaned ).

If you take a look at the screenshot you'll see that the boxes are all greyed out. I cannot tick them. The only options available are to select the file and then click Read. I suppose that the file must be read before I can click Import? It won't get past the read stage (this takes about 2 seconds to error out once clicked).

The reason it crashed is because the server OS had a weird problem with the hot swap hard drive we use for periodic backup. This then cause the system storage controller to corrupt the other drives.

I am a bit concerned about the backup being corrupt though... I ran the export AFTER the drives were recovered and the system was in an operational state... there should be no reason the backup is corrupt.

Do you have any ideas?

[Code Crafters]

Yes, you're right. With the import of the backup you press the read button first to read the file. This then enables any boxes that were in the file so that you can then choose from these which parts to import. Obviously, if the read failed then the whole file and headers is corrupt and not recoverable.

I'm glad you got sorted out though and hope you didn't lose any data or at least not much. We recommend you make a backup once a week or if you want once a day. You can either do this manually via the export tool, or you can make a .bat file to copy the whole "Ability Mail Server 2" folder from program files using basic copy and rename commands. If you don't backup onto CD then we recommend you backup onto a different hard drive to avoid a corrupt hard drive losing the backup too. You can also use windows scheduler to trigger this backup .bat file once a day at something like 3AM when traffic on the mail server should be minimum. Obviously, we are adding our own event scheduler to AMS3 after the initial release next year which will be capable of event scheduling daily backups using the export tool.

[Pugglewuggle]

Very cool, that will be a nice feature. So you don't have any ideas as to why the backup was corrupt after exporting it from a fully functional AMS installation?

Also, regarding making backups - is AMS capable of being backed up WHILE the service is initialized, or must the server be taken offline for a backup? The reason I ask is because from my experience it's possible consistency problems could occur if writes are being made as the backup is being processed. I've seen this happen before and I don't want to compromise our backups.

[m1byo]

I make a backup copy of my server every night using robocopy. (I mirror it to a folder on another server over a VPN)

To prevent any problems, I exclude:
files: sync
folders: temp

This works, and I have found that it has sometimes been invaluble to resucue the server or part of when there is a problem.

[Pugglewuggle]

Lol that's exactly what I do too (except I've been taking the server offline to make the backups)!

I love robocopy. It's even got a nice GUI if you're lazy now. http://technet.microsoft.com/en-us/magazine/cc160891.aspx

I've been considering using DFS with RDC in Windows Server to mirror the AMS files to another server and keep them ALWAYS in sync so that if the main server goes down I'm not stuck with a "snapshot" per se (where some newer emails get lost), but a fully up-to-date copy of the files so all I have to do is install AMS on another machine.

With regards to the exclusions, what exactly do they do? Is temp just like scratch space? What is sync?

[m1byo]

1. DFS.
I would not recomend considering DFS, I have tried DFS and I have had some problems.
1. Certain Users stop beng able to be logged on this can be fixed at the next setting reload interval.
2. The service can crash because of files being synced, see the post, viewtopic.php?f=6&t=346 (this was caused by DFS)

2. Temp/Sync
Unfortunatley Chris/Rob would be the best people to answer what they are, however I think the temp folder is just a scratch pad.

I hope this helps

Ian

[Pugglewuggle]

Very interesting on the DFS... Thanks for the info!

I'll be waiting on a response from Rob/Chris now.

Cheers!

[Code Crafters]

We personally do daily backups using a .bat file with basic rename / delete / copy folder commands and Windows scheduler to trigger the copy of the "C:\Program Files\Code-Craftes\Ability Mail Server 2" folder. Obviously you should backup onto another hard drive in case the original is damaged. The built in export tool can be done while the mail server is online and busy as it accesses the files using the software sync access you mentioned. An external backup will generally be fine but should ideally be done at say 3AM when the server isn't in heavy use or locked files may not be copied correctly if being written to at the time, for example.

The temp folder is where antivirus scan parts of the mail etc are put and then removed after finished with. The sync folder is for internal use of the mail server to synchronise between multiple threads (connections) accessing the same files at the same time to avoid corruption or file access problems.

We can't really comment on the successfulness of using a file synchronisation software with Ability Mail Server other than to say its not designed for such things and using one is more likely to cause file corruptions than if the files were not externally tampered with. It is generally much safer to do a snapshot backup and risk losing at most 1 days emails.

[Pugglewuggle]

Again, by snapshot, you mean running an AMS export? And it IS okay to run this when the system is initialized? And it DOES take into account files that are currently being written and backs them up accordingly once written?

Cheers!

[Code Crafters]

Yes, a copy of the files either by the export tool or using a copy of the program files folder at a non-busy time to automate the process until we build in an event scheduler to auto-trigger the proper export tool. The export tool should not fail to copy any files if the hard drive is working correctly.

Also, be careful that if you are running Windows Vista that you must run the AMS application / service as an administrator account to make the settings appear in the program files folder instead of the user data area.

[Pugglewuggle]

Gotta love Vista. Not.

Windows Server 2008, however is a whole different story. I love it. Lots.