Posted on: July 3rd, 2011 by Famous Phil

Over the past couple of months, I’ve found that the script that I posted at http://famousphil.com/blog/2011/01/a-decent-backup-strategy-for-exchange-2010-sp1/ hasn’t been working perfectly for my needs.  This relatively short blog is kind of a back track blog to explain the issues and provide some fixes for them.

The original script I posted does not verify that the backups actually completed.  The emails I got were simply gibberish.  I was willing to accept that for a while until during monthly maintenance when I manually verify backups, I was finding that backups didn’t always complete.  I’ve tracked this down in the error logs and found that the memory in the server isn’t enough for the backups at times which has them fail at certain times.  Due to the lack of user base on my server (4 light users), I can’t justify adding more ram (I currently have 4GB) because I’d have to upgrade the entire server.  So instead, I did some RAM optimization and re-wrote the backup script to email me the actual backup names that completed successfully.

First for the tips.  I recently learned that Active Directory can be modified from the backend, so using this, I modified the Information Store service (store.exe) in Exchange to only use at most 512MB of ram.  I used the information at http://terrytlslau.blogspot.com/2011/03/limiting-exchange-server-2007-and-2010.html for doing this, I briefly repeat the procedure here in the even that this link is no longer reachable.

1. At Domain Controller, login as a Domain Administrator.
2. Click "Start", enter "adsiedit.msc" into the search box, hit enter.
3. Right-click "ADSI Edit", select "Connect to".
4. Enable the Naming Context view, click ok to connect
4. Under the "Naming Context" menu, select "Configuration".
6. Expand to "Configuration > Services > Microsoft Exchange >  > Administrative Groups > Exchnage Administrative Group > Servers >  > InformationStore".
7. Right-click "InformationStore", select "Properties".
8. Select "msExchESEParamCacheSizeMax".
9. This value is set in pages, in Exchange 2010 the size is 32KB/page; Exchange 2007 is 8KB/page.  Simply figure out the number of pages for the amount of ram you want to limit store.exe to using.

For instance, if you want to limit the Database Cache to 4 GB of an Exchange 2010 server, set msExchESEParamCacheSizeMax to 131072 (4 GB = 4.194.304 KB / 32 KB). If you want to limit the Database Cache to 2 GB of an Exchange 2007 server, set msExchESEParamCacheSizeMax to 262144 (2 GB = 2.097.152 KB / 8KB).
10. Ok everything and restart the Information Store service (possibly the server)

After limiting the Exchange Information Store service, I simply restarted the Information Store service and that seems to have fixed the gouging memory issue.

As a second optimization procedure, I started tackling the IIS Worker Processes.  Exchange has several application pools that it uses, you can think of an application pool in IIS as a separate instance of Tomcat or Apache for each website.  Application Pools isolate websites to that they can’t affect each other.  On the downside, application pools also hog a great amount of memory and for the features of Exchange that you may not use often (e.g. powershell, calendar, exchange control panel), it takes some time for these features to load initially (for me, its about 30 seconds).  My solution was to limit Exchange to two application pools.  For anything service related, I used the Exchange Service Pool (e.g. EWS, Powershell, Autodiscover), and anything client site based (e.g. OWA, Calendar, ECP, ActiveSync) in the OWA pool.  I still do not know if any update to Exchange may reverse this or break this, but I do keep it in mind during updates.  The result of doing this is that not only is memory consumption reduced significantly, but Outlook Web Access, Exchange’s Calendar display (for the public), and Exchange Control Panel all load much faster now since the overhead in IIS is already loaded.  Of course, I wouldn’t recommend doing this unless you can’t easily upgrade the amount of memory in your server.

Finally, I will leave you with an improved export script that replaces the script in my previous blog at http://famousphil.com/blog/2011/01/a-decent-backup-strategy-for-exchange-2010-sp1/.  This script verifies that all of the users were actually uploaded and emails the complete report to you (instead of some garble).  I’ve found it to be very helpful in determining if a mailbox was failing to export to a PST without having to login to the file server and check.  As always, use this script at your own risk, I am willing to provide limited support as time permits.

# Exchange 2010 SP1 Mailbox Export Script
# Originally from Steve Goodman.
# Modified by Philip Matuskiewicz for Matthouse.us / famousphil.com 1/2/11 FIXED *7/2/11*

#define information here
$server = "host.example.com" #server hostname
$users = @("Joe", "Mary", "Phil") #users to archive
$destination = "localhostpstbackups" #network share to backup to
$emailfrom = "server@yourdomain.com"
$emailto = "you@yourdomain.com"
#define some internal variables
$output = ""
$error = 0
$date = Get-Date

#check for errors
if (!(Get-ExchangeServer $server -ErrorAction SilentlyContinue)){
    $output += "Exchange Server $server not found`n";
	$error = 1
}
if (!(Get-MailboxDatabase -Server $server -ErrorAction SilentlyContinue)){
    $output += "Exchange Server $server does not have mailbox databases";
	$error = 1
}

#create a batch job if the above tests succeeded
if ($error -ne 1){

	$jobname = "Export_$($date.Year)-$($date.Month)-$($date.Day)_$($date.Hour)-$($date.Minute)-$($date.Second)"
	$output += "Job title is: '$($jobname)' `n"
	Write-Output "Job title is: '$($jobname)' "

	foreach ($mailbox in $users){
		#remove existing PST file
		if (Get-Item "$($destination)$($mailbox).PST" -ErrorAction SilentlyContinue){
			Remove-Item "$($destination)$($mailbox).PST" -Confirm:$false
			$output += "Existing PST was deleted (Normal): '$($mailbox)' `n"
			Write-Output "Existing PST was deleted (Normal): '$($mailbox)' "
		} # end if

		#request a backup of the mailbox, Exclude the recoverable items / deleted items
		$mailboxjobname = "$($mailbox)-$($jobname)"
		New-MailboxExportRequest -BatchName $jobname -Mailbox $($mailbox) -FilePath "$($destination)$($mailbox).PST" -ExcludeDumpster -Name $mailboxjobname
		$output += "Mailbox Queued: '$($mailbox)' `n"
		Write-Output "Mailbox Queued: '$($mailbox)' "

	} #end foreach
} #end $error -ne 1

#wait for the jobs to complete
$time = 0;
while ((Get-MailboxExportRequest -BatchName $jobname | Where {$_.Status -eq "Queued" -or $_.Status -eq "InProgress"})){
	Write-Output "Waiting on backup, it has been $($time) seconds"
	$output += "Waiting on backup, it has been $($time) seconds `n"
	sleep 600 #10 minutes
	$time = $time + 720;
} #end while

#check for any jobs that didn't complete
$incomplete = Get-MailboxExportRequest -BatchName $jobname | Where {$_.Status -ne "Completed"} | Get-MailboxExportRequestStatistics | Format-List
$complete = Get-MailboxExportRequest -BatchName $jobname | Where {$_.Status -eq "Completed"} | Get-MailboxExportRequestStatistics | Format-List

if($incomplete){
	Write-Output "ERROR: Something didn't complete, output is '$($incomplete)'"
	$output += "ERROR: Something didn't complete, output is '$($incomplete)' `n"
}

if($complete){
	Write-Output "Completed Successfully, output is '$($complete)'"
	$output += "Completed Successfully, output is '$($complete)' `n"
}

# Remove Requests and clean up
Write-Output "Cleaning up requests that were part of the job '$($jobname)'"
$output += "Cleaning up requests that were part of the job '$($jobname)' `n"
Get-MailboxExportRequest -BatchName $jobname | Remove-MailboxExportRequest -Confirm:$false

#verify that all the PST files were created...
foreach ($mailbox in $users){
		#remove existing PST file
		if (Get-Item "$($destination)$($mailbox).PST" -ErrorAction SilentlyContinue){
			$output += "PST FOUND!!!: '$($mailbox)' `n"
			Write-Output "PST FOUND!!!: '$($mailbox)' "
		}else{
			$output += "ERROR: PST NOT FOUND: '$($mailbox)' `n"
			Write-Output "ERROR: PST NOT FOUND: '$($mailbox)' "
		}
}

$SmtpClient = new-object system.net.mail.smtpClient("double.matthouse.org")
$msg = new-object Net.Mail.MailMessage
$msg.From = "$($emailfrom)"
$msg.To.Add("$($emailto)")
$msg.Subject = "EXCHANGE EMAIL BACKUP DETAILS"
$msg.Body = $output
$SmtpClient.Send($msg)

Write-Output "Script complete!"

Tags: backup, exchange 2010, powershell, pst, script
Posted in Hosting / Server Administration, Technology
|| 2 Comments »

Posted on: January 11th, 2011 by Famous Phil

There is a new blog with additional information on this topic at http://famousphil.com/blog/2011/07/revisiting-exchange-2010-sp1-pst-backups-improved-script/

About a week ago, I finally got around to reviewing and upgrading my existing backup routine for my exchange server.  My previous backup scheme involved pushing a full server backup image to a remote server on a weekly basis.  If the server failed during a backup, I would have no viable way of recovering from a complete disaster.  Obviously, this required some changes.

With the release of SP1 for Exchange 2010, a few new power shell commandlets came out that provide functionality to back up Exchange accounts directly on the server (no needing second computer with outlook and exchange management tools anymore!).  This meant that I decided to utilize some backup scripts that backup each mailbox nightly.  I also modified the weekly system backup.

There are scripts attached to this post.  With any kind of solution that I provide, I always provide it on an as is basis with no warranty provided that it will work for your situation, although I try my best to cover as many scenarios as I can.

So what was decided upon? Read the rest of this entry »

Tags: backup, exchange 2010, Firewall, Linux, powershell, scp, server
Posted in Hosting / Server Administration
|| 3 Comments »

Posted on: July 23rd, 2010 by Famous Phil

I am delighted today to bring you a guest posting from Alexis Bonari!

Anyone who does any sort of work on their computer can tell you a hard drive crash is the stuff nightmares are made of.  While it’s easy to pass judgment on such individuals for failing to use an external hard drive, doing so is admittedly time-consuming and, in some cases, expensive.

The solution: online backup sites. For a small fee, these off-site servers back up all data stored on the computer in case of a hard drive failure.  Here are the top three such services and what they have to offer:

1. Carbonite
(http://www.carbonite.com/en/default.aspx)
For only $54.95/ year, Carbonite offers unlimited backup on their server.  No matter what your computer’s storage limit, the Carbonite system can handle it.  For security purposes, files are encrypted before being sent to the Carbonite server for storage. For ease of use, the files are automatically backed up each time the computer is connected to the Internet. Restoring the files is as simple as logging into the Carbonite website and clicking the “restore” button listed on your account.

2. MozyHome Free
(http://mozy.com/home/free)
Unlike Carbonite, Mozy doesn’t charge a fee for the first 2 GB backed up.  The system for retrieving files and backing them up is essentially the same the one used by Carbonite.  If you want to store more than 2 GB, Mozy charges $4.95/month.  This gives you unlimited data storage for only slightly more per year than Carbonite.

3. SugarSync
(https://www.sugarsync.com/)
Many experts believe that online syncing represents the future of online backup technology.  SugarSync.com is the current leader in online syncing technology. Instead of simply backing up a set of files on one computer, SugerSync notifies other computers and devices you’ve listed of any changes made. You can work from nearly anywhere in the world and have your files backed up in real-time by the Sugarsync server.  Devices supported include Mac’s, PC’s, Ipads, smart phones, and many others.

Those who sign up get a 30 day free trial.  After that, the price goes up to $10/month for 60 GB of storage.  While the limited storage and the higher price might be prohibitive for some customers, many are happy to pay extra for the ability to sync documents over multiple devices.

Bio: Alexis Bonari is a freelance writer and blog junkie. She spends much of her days blogging about Education and CollegeScholarships. In her spare time, she enjoys square-foot gardening, swimming, and avoiding her laptop.

Tags: alexis bonari, backup, disaster, guest post
Posted in Technology
|| 18 Comments »

Posted on: March 11th, 2010 by Famous Phil

Today, I will pursue a topic that all of us wish wasn’t necessary to discuss, but it is a fact of life. Backups are required in life and this isn’t a new concept, but poor backups can be devastating when you find out that it doesn’t work (and that is usually when you need it the most). I’ve spent a lot of time making changes to my backup systems over the years, but they are never perfect and I constantly find new flaws that I constantly work to fix.

A few weeks ago, I found a major flaw in my backup system that I never really thought about in the past.  My backups would run every night by essentially pushing the latest backup to another server that I run.  This server had an account on it where the backup would neatly upload then not be touched again until the next backup cycle ran.  To access the backup, all I had to do is login as root on my production system (the system that runs famousphil), then request the backup.  Passwords are entirely controlled through keys, so I didn’t have to do anything special or know anything to gain access to my backup server.  Unfortunately, if all I needed was my root password / key to do that, if a hacker got into my system, they essentially could completely destroy my backup system as well.

With the ever growing threat of hackers attempting to compromise my network of servers, I’ve began giving more thought to hack attempts and prevention along with hard disk and other physical server failures (which also happen).  Because of these thoughts, I’ve implemented a new backup routine that fixes most of my issues with a hacker gaining access to my network.  If I discover a hacker has penetrated one of my servers within 12 hours, I will always be able to restore a backup that is no more than 3 days old now.  The same goes for a hard disk failure / natural disaster. I do not feel it is in the best interest of Matthouse to disclose the new backup system and exact scheduled times, but I thought that it would be appropriate to share a little insight into that thought that I’m sure many other system administrators might fall into at one point or another.

Another threat that I found with my backup system was that part of my backups were incomplete.  For any admin that thinks backing up /var/lib/mysql is appropriate to backup MySQL, think again.  About 2 months ago, I thought this was appropriate, so I began backing up MySQL this way (I never really thought to check it initially since the server was brand new).  I normally check the redundancy of my backups about once every 2-3 months.  During my redundancy check, I always check everything and I discovered that I could not restore the MySQL database server from that directory easily.  Since I have the original server still running, I came up with a new backup procedure that works much better and can actually be restored.

For all of you who need to backup an entire MySQL server, here are the 2 commands that will help:

• Backup: mysqldump -u root -pPASSWORD –all-databases > backup.sql
• Restore: mysql -u root -pPASSWORD < backup.sql

Those are the 2 major points I wanted to make in this post.  In conclusion, I recommend that you make a backup schedule for your servers and verify that loop holes don’t exist in it.  I’m constantly looking for loop holes and I patch them as I find them.  Also, verify your backups!  I would have never realized that MySQL wasn’t being backed up unless it was for verifying that my backups are in tact and can be restored.  I verify my backups every other month and I recommend that every system admin does so.

For anyone at home, HARD DRIVES DO FAIL.  I strongly recommend keeping any data that you could not afford losing on a flash drive or external hard drive where if your computer did fail, you would still have your data.  Unfortunately, data is unrecoverable at cheap rates, making a backup is a cheap insurance policy to avoid complete disaster!

Tags: backup, hacker, hard drive failure, mysql
Posted in Hosting / Server Administration
|| 2 Comments »

Posted on: July 10th, 2009 by Famous Phil

As you probably know by now, I have a Microsoft Exchange 2007 server.  With any kind of server, a backup and disaster recovery plan is a vital part of any kind of professional hosting.  Hosting email can be quite tricky as it is, but taking network backups was a new road that I’ve never covered in Windows.

First, I have a fairly simple environment set up for my backup.  I have a Linux backup server running samba sitting in Seattle (mthsweb2).  This server has the Windows IP white listed so that only my Windows server can connect to it to map a network share.  For those of you who don’t know what samba is, samba is a daemon in Linux that will allow Windows to naively connect to a Linux server for file sharing.  Samba is a very simple solution for mapping a network drive to Linux in Windows without needing any specialized software such as sftpdrive (not called something else).

Read the rest of this entry »

Tags: backup, batch, Linux, mapped network drive, network, ntbackup, samba, script, server 2003, windows
Posted in Hosting / Server Administration
|| 3 Comments »