About a week ago, I finally got around to reviewing and upgrading my existing backup routine for my exchange server.  My previous backup scheme involved pushing a full server backup image to a remote server on a weekly basis.  If the server failed during a backup, I would have no viable way of recovering from a complete disaster.  Obviously, this required some changes.

With the release of SP1 for Exchange 2010, a few new power shell commandlets came out that provide functionality to back up Exchange accounts directly on the server (no needing second computer with outlook and exchange management tools anymore!).  This meant that I decided to utilize some backup scripts that backup each mailbox nightly.  I also modified the weekly system backup.

There are scripts attached to this post.  With any kind of solution that I provide, I always provide it on an as is basis with no warranty provided that it will work for your situation, although I try my best to cover as many scenarios as I can.

So what was decided upon?

Full Image Backup: On the local Exchange front, I opted to make the monthly full system image backup run the first Sunday of every month.  It is configured to back up to a network share that is actually on the same server.  When backing up to a local drive, Windows Backup typically appends to the backup and I prefer to have the previous backup removed so that disk space usage is minimized.  The full backup takes about 25 minutes to complete and is stored locally.

rmdir /S /Q localhostbackupwinbackup
mkdir localhostbackupwinbackup
C:WindowsSystem32wbadmin.exe START BACKUP -backupTarget:localhostbackupwinbackup -vssFull -allcritical -quiet

Then, on the eighth day of each month, my Linux-based backup server connects via SCP to pull the monthly image that was created.  Technically, it would take a dual hard drive failure (both the system drive, and backup drive) to completely render the system un-restorable with this scheme.  It is very rare to see a dual drive failure, but if it were to occur, I still would have the nightly backups to fall back on and it might take about 5 to 10 hours more to complete a restoration.

SCP as I’ve mentioned in the past is a way to copy files between two Linux based servers securely.  It operates using the SFTP protocol which runs on top of SSH.  Remember that SSH is a way to bring up a command prompt that runs on the remote system.  Since Windows doesn’t natively support any decent secure file transfer techniques that I’m aware of, I installed copssh (http://www.itefix.no/i2/copssh), which is a Cygwin based solution for Windows that supports SSH and SFTP (and ultimately the SCP command).  With Cygwin working on the Exchange server, I added a user to Cygwin with bash shell access and I firewalled port 22 to where only the remote backup server could connect (for security reasons mostly).  I will leave the actual SCP script writing up to the reader, although here is a helpful page (located here).  I will also hint that the reader will need to learn about public keys to allow for a passwordless remote ssh login so that sftp doesn’t prompt for a password when ran as a cron job.

Nightly Backup: When Exchange 2010 SP1 was released, several commandlets were created in powershell which allow for the export of PST files without needing Outlook installed on the same computer (which should never be installed on an exchange server directly).

I’ve found that in my initial testing of the PST export, there is a fairly large performance hit on the server, so I recommend doing it at the least used hour available.  I’ve also found that it takes a fair amount of time (10 minutes or so) and cpu power to export a large mailbox (> 1GB).  I also found that when restoring (importing) a PST, it can only be imported to another folder of the mailbox, it cannot overwrite existing mail in the box (I guess this is a good thing and a bad thing).

Anyways, I managed to find how to do a mass export using powershell at Steve Goodman’s Tech Blog.  I modified the script because his script included mailboxes that I didn’t care about (such as alias boxes, the discovery box, etc).  My Script takes in a constant array of usernames (mailbox alias names) and only backs up those boxes sequentially (posted with this article).  This happens nightly at 12:45am since that is the least active time for my system.  I chose to export to the same local drive as my other backups so that they could be pulled in by my backup server (via SFTP).  This maintains security of all the email data I handle.

The first task when implementing this backup plan is to set appropriate permissions to make the backup work.  The folder or server share (in my case) where the export will be saved needs to have the modify permission set for the Exchange Trusted Subsystem.  Remembering my firewall woes post, make sure that file and printer sharing rules are present in the firewall (even if you have custom rules and disable the default rules), otherwise you will get an illegal exception error without any real clue for what the problem is.  Then, within exchange’s powershell interface, the user who will be running the export commandlet will need to have permissions set.  This can be done by the command:

New-ManagementRoleAssignment –Role “Mailbox Import Export” –User “ADAdmin”

Next, it is probably helpful to get a listing of all the mailboxes so that we know the aliases of each mailbox that needs to be backed up.  I will note that I’ve also noticed that lower/upper case doesn’t matter on Windows Powershell like it would on a Linux based shell.  The command to get the listing is:

get-mailbox

Using the output of the Alias column, you should be able to fill in the script that I’ve modified (from Steve Goodman) and have it run.  At the end of a successful run, I have powershell send me an email telling me if the job was completed successfully or not.  I’ve noticed that my spam filters caught that email, so I added an exception for my Exchange server.  The end of this post has the actual script, and the source can be found here.   The email portion of my script initially threw errors to me, eventually I used telnet to test if port 25 was open and I discovered that it wasn’t.  It turns out that my exchange server was not listening on port 25 of the localhost address.  If you run into this situation, I’d recommend checking that or the firewall before jumping to conclusions.  Telnet is a very powerful tool!

Last but not least, I pull the nightly backups from Exchange every day at 8am, well after the nightly backup is finished.  The script that pulls these backups is very similar to the monthly script but it copies the existing daily backup to a weekly retention folder every Sunday before it begins grabbing the latest nightly backup.  I’ve included that script below for your reference.

Conclusion: Backups are a necessary part of any viable disaster recovery plan.  I’ve outlined one above that I use on my own systems.  I’m sure it isn’t perfect, but it’s well thought out and I have used it to recover systems in full before.  As with any backup plan, I usually always test it out immediately after its implementation to verify that it will work in the event of a disaster.  As a system admin, I never want to be left holding onto a faulty backup that I believe worked and I’m sure that you can also appreciate this thinking.

#Crontab Entry
#5 8 * * * /back/exchangedaily.sh  | mail -s "daily exchange offsite backup finished" phil@matthouse.us
#!/bin/bash
cd /backup/exchange
if [ $(date +%A) = "Sunday" ]
then
        rm -Rf weekly
        mv daily weekly
        mkdir daily
        cd daily
else
        rm -Rf daily
        mkdir daily
        cd daily
fi
scp -r admin@matthouse.org:/cygdrive/e/bck/pstbackup ./

This script has been updated:  http://famousphil.com/blog/2011/07/revisiting-exchange-2010-sp1-pst-backups-improved-script/

# Exchange 2010 SP1 Mailbox Export Script
# Steve Goodman. Use at your own risk! - modified by Philip Matuskiewicz for Matthouse 1/2/11

#create a .cmd file with the following line to invoke this:
#C:WindowsSystem32WindowsPowerShellv1.0powershell.exe -command ". 'c:Program FilesMicrosoftExchange ServerV14binRemoteExchange.ps1'; Connect-ExchangeServer -auto; .MassExport.ps1"

#what to back up from?
$Server = "exchangeserver.matthouse.org"

#who to backup, use the alias from commandlet Get-Mailbox -Server $Server (replace $server appropriately)
$USRALIAS = @("FamousPhil", "MyUserOne", "Staff ", "AnotherUSER")

# Share to export mailboxes to. Needs R/W by Exchange Trusted Subsystem
$ExportShare = "localhostbackuppstbackup"

#error checking

    if (!(Get-ExchangeServer $Server -ErrorAction SilentlyContinue))
    {
        throw "Exchange Server $Server not found";
    }
    if (!(Get-MailboxDatabase -Server $Server -ErrorAction SilentlyContinue))
    {
        throw "Exchange Server $Server does not have mailbox databases";
    }
# Make batch name
$date=Get-Date
$BatchName = "Export_$($date.Year)-$($date.Month)-$($date.Day)_$($date.Hour)-$($date.Minute)-$($date.Second)"

        Write-Output "Queuing Pre-defined mailboxes as batch '$($BatchName)' sequentially"

foreach ($element in $USRALIAS)
{

        if (Get-Item "$($ExportShare)$($element).PST" -ErrorAction SilentlyContinue)
        {
                Remove-Item "$($ExportShare)$($element).PST" -Confirm:$false
        }
        New-MailboxExportRequest -BatchName $BatchName -Mailbox $($element) -FilePath "$($ExportShare)$($element).PST"

        while ((Get-MailboxExportRequest -BatchName $BatchName | Where {$_.Status -eq "Queued" -or $_.Status -eq "InProgress"}))
        {
                Write-Output "Waiting on backup of $($element), refreshing every 60 seconds"
                sleep 60
        }
}

###REPLACE SMTP SERVER, FROM, and TO addresses and potentially subjects below.

        $SmtpClient = new-object system.net.mail.smtpClient("exchangeserver.com")
        $msg = new-object Net.Mail.MailMessage
        $msg.From = "phil@matthouse.us"
        $msg.To.Add("phil@matthouse.us")

    $Incomplete = Get-MailboxExportRequest -BatchName $BatchName | Where {$_.Status -ne "Completed"} | Get-MailboxExportRequestStatistics | Format-List
    if ($Incomplete)
    {
        Write-Output "Process didn't complete, Emailing report to admin"
        $msg.Subject = "Double.matthouse.org Backup Incomplete"
        $msg.Body = $Incomplete
        $SmtpClient.Send($msg)
    }

    $Completed = Get-MailboxExportRequest -BatchName $BatchName | Where {$_.Status -eq "Completed"} | Get-MailboxExportRequestStatistics | Format-List
    if ($Completed)
    {

        Write-Output "Backup done, sending the report to admin"
        $msg.Subject = "Double.matthouse.org Backup Complete"
        $msg.Body = $Completed
        $SmtpClient.Send($msg)
    }

# Remove Requests and clean up
Write-Output "Removing requests created as part of batch '$($BatchName)'"
Get-MailboxExportRequest -BatchName $BatchName | Remove-MailboxExportRequest -Confirm:$false

Write-Output "Process complete!"

Let me start off with wishing you a happy thanksgiving holiday.  I was hoping to get this blog done sooner but I have been facing a shortage of time with my regular responsibilities as a student.  Anyways, enough about that, this blog post is going to explain how I can break through a restrictive firewall and have a fully usable IP address from the inside of that firewall.  This task took me about a month to successfully complete with about 30 consecutive hours so I felt the urge to blog it for the future me, you, the reader can benefit too I suppose.  This is a huge blog so it will take a while to read.

The motivation:

I’m a huge fan of landline communication because it has a more stable and clear signal.  Unfortunately, my University removed the phone lines from all the rooms and wants $300 for a telephone jack activation and $28/month for local area calling (meaning the 716 area code only).  Furthermore, my parents have a calling plan that allows for extreme local calling that doesn’t include the University phone prefix.  Before, they didn’t mind paying extra to call me, but now my parents would have to pay for my University phone and the privilege to call me.  This just doesn’t make sense.

With this in mind, I started searching for a VoIP provider (Voice over Internet Protocol) that uses SIP (Session Initiation Protocol).  For those of you who have never heard of VoIP, it is a simple way to connect a special device to the internet and get a regular telephone jack.  This jack can then be connected to an old fashioned telephone (or cordless phone in my case) that can act similarly to a landline.  The quality for VoIP is very comparable to a landline when connected to a land based internet connection (meaning it sucks at home), but UB has a land based connection which means it will be fine.

I tested the device at UB over the summer on a network link in a room where I’m completing my master’s project and the device worked fine (I have some control over that firewall).  Unfortunately, in my on campus apartment, that connection can’t be altered in my favor.  I’ve found that the provider I selected requires unsolicited incoming connections to properly work, something that I know UB would never allow for.  Instead of going through Bureaucratic Bulls**t where I would surely loose, I took the alternate route, unblock the connection myself (and use it for my own computers too).

In the end, I have public IP addresses that are totally un-firewalled within my apartment where I can do trace routes and other network diagnostic tasks.  I also appear like I’m in Michigan (since that is where the datacenter is).

If you want to know more about how this is done, please continue by reading more.  DISCLAIMER: I WILL NOT SUPPORT ANYTHING WITHIN THIS BLOG THAT IS OF YOUR OWN FAULT… The information is provided on an AS-IS basis.  I’m sure there are minor mistakes here and there since this blog is almost 12 pages long in Microsoft Word!

One last note: DO NOT ATTEMPT DOING THIS if you are a novice user or someone who doesn’t know what they’re doing!  Firewalls are put up for a reason, breaking through them without having a really good reason to do so could harm you and the network you’re on in the long run, especially if you’re at work!  I take no responsibility for anything that happens to you.

Let us start looking at the network I’m dealing with:

——INTERNET —— University’s NETWORK that is FIREWALLED ——– ME

Basically, UB has a huge firewall called Resnet that blocks all incoming connections that weren’t initiated by me initially (like a request for a website).  They also block many network diagnostic tools like ping, traceroute, etc.  They do this to prevent students from running insecure game servers, web servers, IRC servers, and email servers within the network.  Since servers are open to the world and the world is full of hackers, it isn’t a good idea to have an insecure computer on your network.  These insecure machines on a network like my colleges can be difficult to find and if a hacker begins using these computers to do malicious tasks, they can do it much faster than a home computer since the network is hundreds of times faster than a typical home network computer.  Unfortunately, blocking these bad activities can also block legitimate traffic such as VoIP devices that need access both ways.  Strangely, they also block all outgoing DNS requests blocking OpenDNS which is a great DNS provider that prevents malicious websites from loading before they actually are requested.

Thankfully, this is where VPN can come in and save the day.  VPN is virtual private networking and it works by creating an encrypted tunnel to a server on the outside world and then sends all the traffic to that outside server to be eventually routed to its final destination.  Doing this allows you to show up in an entirely different physical location and it also secures all your traffic.  Getting back to my scenario, I’ve decided to run my VPN tunnels on port 80 which is the port used by HTTP or common website browsing.  I did this because I know UB can’t block this port without blocking most websites on the world.  I know they’d have a lot of angry students and faculty if they did that.

So with this in mind, it was time to begin configuration of OpenVPN on the remote end.  I chose to go with a provider that I trust not to overload virtual private servers (VPS).  The VPS had to run xen which is a virtualization that allows for customizing firewall rules and network links at the kernel level (lowest level of an operating system).  I also knew the VPS wouldn’t need much memory or cpu availability so I opted for 128MB of ram and 400Mhz of CPU units.  Being a VPN, I knew I’d need a lot of network capacity so I requested 100mbps port on the VPS host machine so I would be guaranteed at least 20Mbps for VPS traffic (although I typically peak around 100Mbps).  Finally, I needed several public IP addresses on the VPS so that I could assign them to clients at my university (like my VoIP adapter).  The final bit was an operating system, I already know CentOS really well so I opted for that OS.

The provider needed to be close to me to improve latency and speed that are a huge issue for OpenVPN.  If you’d like to know the provider I choose, it was corenetworks.net.  They are located in Michigan on Great Lakes Comnet.  They have direct connectivity to Level 3 in Chicago.  My University directly connects to Level 3 in Buffalo which has a direct link to Chicago.  So basically I have to go through 4 hops from my University to the datacenter, this takes a total of 30ms which isn’t optimal, but providers in Chicago are touch and go and I wanted someone reliable that had a reasonable price and reliability (I’m trying to keep the cost under $28/month).  Keep in mind that for my configuration, there is a minimum 30ms lag for any connection I need to make to the outside world.  Compared to Verizon wireless at home, this is still a 60ms advantage so I’m quite happy with this.

Now that I have a decent server for OpenVPN, it was time to configure the server to accept connections.  With any new server, I do the usual initial hardening of the server so I changed the root password, enabled an ssh passwordless login using RSA keys.  I also modified the resolv.conf file to have opendns as nameservers.  I also modified the hostname for my server’s hostname, there are 2 files to do this with, 1 /etc/hosts and 2 /etc/sysconfig/network.

The next step is to get opendns installed.  For CentOS, I opted to use yum which is a package manager, unfortunately though it requires an extension called RPMForge to get OpenVPN.  The commands I used to get RPMForge functioning are (keep in mind that new versions come out constantly so you may need to update the link for wget which downloads the package for you):

• cd /root
• wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.1-1.el5.rf.i386.rpm
• rpm -Uhv rpmforge-release-0.5.1-1.el5.rf.i386.rpm

The next step was to install the required packages and update the server using Yum.  Below are the commands that I used for this, there are additional packages listed because they help later when configuring OpenVPN and other programs.

• yum update
• yum install -y sendmail sendmail-cf openssh-server syslog nail bridge-utils openssh-clients gcc gcc-c++ openssl-devel zlib-devel perl-HTML-Parser perl-DBI perl-Net-DNS perl-Digest-SHA1 vim-common vim-enhanced vim-minimal nano zip unzip patch rpm-build perl-libwww-perl ntp rpm-build autoconf.noarch pam-devel openvpn lzo lzo-devel

At this point, it made sense to disable the firewall and automatically start some services.  NTP is a time synchronization daemon that will ensure that my server will always have the correct time to satisfy the encryption standards for OpenVPN.  I will leave it up to you to learn what the other commands do.

• service iptables stop
• service ntpd start
• service syslog start
• chkconfig ntpd on
• chkconfig iptables on
• chkconfig crond on?
• chkconfig syslog on

Next, I wanted to get sendmail working.  This is a mail server daemon that allows for the server to mail me when it has problems with itself.  It’s really helpful for my server monitoring software, nagios that isn’t included in this blog.

• echo “YOUREMAILADDRESS” > /root/.forward

Using a text editor like vim, add your fully qualified domain name (something.domain.com) to /etc/mail/local-host-names to allow sendmail to use the FQDN (Fully Qualified Domain Name) for sending mail.  Also edit /etc/mail/sendmail.mc and change the line like LOCAL_DOMAIN(`localhost’) to have localhost become your FQDN specified in local-host-names.  Finally:

• cd /etc/mail
• m4 sendmail.mc>sendmail.cf
• service sendmail restart
• chkconfig sendmail on

At this point sendmail will be able to send mail to you, you can always use the linux mail command to try this out.

Moving onto OpenVPN, we should now begin configuring the server.  Here is some simple copy paste stuff that I did.

• cp -r /usr/share/doc/openvpn-2.0.9/easy-rsa/ /etc/openvpn/
• cp /usr/share/doc/openvpn-2.0.9/sample-config-files/server.conf /etc/openvpn
• cp /usr/share/doc/openvpn-2.0.9/easy-rsa/openssl.cnf /etc/openvpn
• cd /etc/openvpn/easy-rsa
• chmod +x clean-all
• chmod 744 build*

Lets vim /etc/openvpn/easy-rsa/vars and modify the last 5 or so lines.

• export KEY_COUNTRY=US
• export KEY_PROVINCE=NA
• export KEY_CITY=TOWN
• export KEY_ORG=”ORGANIZATIONNAME”
• export KEY_EMAIL=”YOUREMAILADDRESS”

Now we need to generate some certificates.  You want to copy paste each line individually since some commands will require you to read and give information.  The file above that we edited should pre-fill most of the information except for a common name.

• . ./vars
• mkdir /etc/openvpn/keys
• ./clean-all
• cd /etc/openvpn/easy-rsa
• ./build-ca
• ./build-key-server vpnserver
• ./build-dh

Now we need to create a client key so that the client can connect to our server.  You may change mths to whatever you want.  The nail command is an easy way to attach files and send emails from the command line, its similar to mail.

• cd /etc/openvpn/easy-rsa
• ./build-key mths
  • You’re going to have to fill in information
• cd keys
• zip keys.zip ca.crt ca.key mths.crt mths.csr mths.key
• nail -s “Keys” -a keys.zip YOUREMAILADDRESS
• .

The next step is to configure OpenVPN, this is somewhat easy.  Simply edit the /etc/openvpn/server.conf file and remove all the existing lines.  Now we need to fill it in with new information, keep in mind the uppercase statements should be replaced.  I also have port 443 used here since HTTPS is also difficult to block.

local SERVERIPADDRESS
port 443
proto udp
dev tap0
server 192.168.4.0 255.255.255.0
push “dhcp-option DNS 208.67.222.222″
push “dhcp-option DNS 208.67.220.220″
push “redirect-gateway”
push “route 192.168.5.0 255.255.255.0 192.168.4.254″
push “route 128.205.0.0 255.255.0.0 192.168.4.254″
client-config-dir /etc/openvpn/ccd
tls-server
ca /etc/openvpn/easy-rsa/keys/ca.crt
cert /etc/openvpn/easy-rsa/keys/vpnserver.crt
key /etc/openvpn/easy-rsa/keys/vpnserver.key
dh /etc/openvpn/easy-rsa/keys/dh1024.pem
max-clients 8
keepalive 10 120
comp-lzo
user nobody
group users
persist-key
persist-tun
verb 3
client-to-client
duplicate-cn
log /var/log/openvpn.log
up /etc/openvpn/firewall.sh

Pretty much everything above is straight forward, but I’m going to explain a few of the lines and what they’re actually doing since they might vary for you.

“server 192.168.4.0 255.255.255.0” means assign IP addresses to clients dynamically for this range of IP addresses.  My entire network runs on the 192.168.4.0 subnet but I will explain how its translated into a public IP later on.

Now for routes:
push “route 192.168.5.0 255.255.255.0 192.168.4.254″
push “route 128.205.0.0 255.255.0.0 192.168.4.254″

I might as well release something that you can find out regardless… this is UB’s public IP address range.  Basically when I connect to my VPN network, I want access to my private UB network (runs on 192.168.5.0) as well as access to the entire UB network from the inside (since UB likes to block most external traffic making some of my work impossible to do remotely, oh by the way, their VPN method sucks over a cellular link but that’s a topic for later as well).  These routes allow my VPN to use the gateway 192.168.4.254 to gain access to my private network.  This is accomplished by using a DD-WRT router called python.matthouse.us that knows about both networks.  If this is convoluted, don’t worry, I will be posting a diagram at the end.

I have 8 public IP Addresses for this VPN server, so I limit the clients to 8.  The rest of the stuff is really not required to know but it guarantees solid connections as well as allowing multiple connections using the same key (MTHS).  The last line that starts with up means upon the OpenVPN server starting, run this script.

So now with all that explained, it’s time to get to OpenVPN methods of connectivity to the real world.  This is the part that took me over 20 hours to figure out.  OpenVPN works in one of 2 layers in the OSI model for networking.  Layer 2 or Layer 3 and each layer has its advantages and disadvantages.  Layer 2 of the OSI model works at the MAC address layer.  The MAC layer is physical connectivity and is useful for low level protocols like DHCP packets that can’t be routed by the Internet.  This is very useful for gammers.  Layer 3 operates at the IP packet level which is what level routers work on.  Most applications work at this level except for those that assign IP addresses like DHCP.

For a long time, I was trying to get something called server-bridge working with a network bridge on the VPN server because I wanted layer 2 connectivity.  Unfortunately, without supporting a custom kernel on the VPS, I was unable to get my VPS to route ARP (address resolution protocol) packets over the bridge correctly.  Through weeks of packet sniffing, I could not get an ARP packet over the bridge because of networking issues with Linux (I really hate open source software, Windows supported this within a few clicks, but Windows is too inefficient and vulnerable to be a router).  I also looked at the possibility of ebtables, but I didn’t need MAC routing bad enough to want to use it.  Since I hate having to support custom modifications to an operating system, I opted to work at Layer 3 of the OSI model and turn my VPN server into a router.  This greatly simplified the setup procedure from creating a bridge to making a few firewall rules in IPTables (which is what we shut off earlier).

So this is a good time to explain a few physical aspects of a server in a datacenter.  Typically your server is going to be connected to the Internet using a single network card, using the TAP0 network adapter, we now have 2 adapters on the VPN server which will need to be bridged via Layer 3 to enable VPN traffic to reach the internet.  Due to problems with routing table loops, it would be nearly impossible to connect the 2 network segments with IPTables, this is why we use Bridges.  So this is why I was forced to use Network Address Translation (NAT).  Using NAT, I can easily connect a 192.168.x.x network to a public network.  With NAT, the clients appear as if they are from a public IP Address and the router does all the work of figuring out where the information should go within the internal network.  NAT is also capable of forwarding ports from a public IP to an IP within a private network.

So this is when we need to make the VPS server accept the additional public IP Addresses via additional Ethernet adapters.

To do this cd /etc/sysconfig/network-scripts/ then inside of that folder, copy the configuration of ifcfg-eth0 to additional files.  For 4 IP addresses, it might look like:

• cd /etc/sysconfig/network-scripts/
• cp ifcfg-eth0 ifcfg-eth0:0
• cp ifcfg-eth0 ifcfg-eth0:1
• cp ifcfg-eth0 ifcfg-eth0:2
• cp ifcfg-eth0 ifcfg-eth0:3

Now we need to edit each file and change the device appropriately and also change the IP address appropriately.  I will leave this up to you.

Finally we need to make IPTables route traffic appropriately using NAT.  Here is a script that will help you accomplish this task using a text editor like vim.  The script uses x.x.x.x, y.y.y.y, z.z.z.z, w.w.w.w because those are the 4 individual public IP addresses that your server will have for VPN clients.  We simply route each one to the appropriate internal IP address and back out.

Vim /etc/openvpn/firewall.sh

——————————————————————–

#!/bin/bash
# My system IP/set ip address of server
SERVER_IP=”SERVER IP ADDRESS IN HERE”
# Flushing all rules
iptables -F
iptables -X
# Setting default filter policy
iptables -P INPUT DROP
iptables -P OUTPUT ACCEPT
iptables -P FORWARD ACCEPT
service iptables restart
iptables -F
# Allow incoming ssh only to the main server IP Address and OPENVPN traffic to port 443
iptables -A INPUT -p tcp -s 0/0 -d $SERVER_IP –sport 513:65535 –dport 22 -m state –state NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT -p udp –dport 443 -j ACCEPT
#enable NAT to and from the main server IP
#I exclude 202, 203, 204, 205 because these need custom routing for 4 ips
#basically custom routing for incoming unsolicited traffic like DMZ from your router
for i in {2..254}
do
if [ $i -lt 202 ]; then
iptables -t nat -A POSTROUTING -s 192.168.4.$i -j SNAT –to-source $SERVER_IP
fi
if [ $i -gt 205 ]; then
iptables -t nat -A POSTROUTING -s 192.168.4.$i -j SNAT –to-source $SERVER_IP
fi
done
iptables -t nat -A POSTROUTING -s 192.168.4.202 -j SNAT –to-source x.x.x.x
iptables -t nat -A POSTROUTING -s 192.168.4.203 -j SNAT –to-source y.y.y.y
iptables -t nat -A POSTROUTING -s 192.168.4.204 -j SNAT –to-source z.z.z.z
iptables -t nat -A POSTROUTING -s 192.168.4.205 -j SNAT –to-source w.w.w.w
iptables -A FORWARD -i tap0 -o eth0 -j ACCEPT
iptables -t nat -I PREROUTING -d x.x.x.x  -j DNAT –to-dest 192.168.4.202
iptables -t nat -I PREROUTING -d y.y.y.y -j DNAT –to-dest 192.168.4.203
iptables -t nat -I PREROUTING -d z.z.z.z -j DNAT –to-dest 192.168.4.204
iptables -t nat -I PREROUTING -d w.w.w.w  -j DNAT –to-dest 192.168.4.205
iptables -A INPUT -m conntrack –ctstate ESTABLISHED,RELATED -j ACCEPT
#allow ping
iptables -A INPUT -p icmp -m icmp –icmp-type 8 -j ACCEPT
iptables  -A INPUT -j DROP

————————– END Script

I’m still working out tiny bugs here and there with this script but it does the job fine.  Now we should change the permissions on the firewall script as well as get openvpn running.

• chmod 755 /etc/openvpn/firewall.sh
• service openvpn start
  • Look for errors before continuing
• chkconfig openvpn on

Now we need to enable IP Forwarding on the server so that IPTables can do its job.

• vim /etc/sysctl.conf
  • change ipforward = 0 to ipforward = 1
  • get out of the text editor
• sysctl –p
  • refreshes the kernel options

Since I want my OpenVPN server to see my internal network at my University, I also need to add a route.  Edit /etc/rc.d/rc.local and add the line: “route add -net 192.168.5.0 gw 192.168.4.254 netmask 255.255.255.0” without quotes.  This now tells the server to use the gateway 192.168.4.254 for all inquiries to 192.168.5.0.

At this point, the server is all set, so reboot it and make sure that everything comes up.  Since OpenVPN starts automatically, it will configure IP Tables automatically for our firewall to allow for SSH traffic and OpenVPN traffic to only the server IP.  We don’t do anything to block IPs that are dedicated to clients in the network.  Also note that clients that are on ips like 192.168.4.22 will appear to come from the main Server IP Address and they will be able to surf normally since there is a special NAT rule for it.  They just won’t get incoming traffic.

Now we switch to the client side.  This part of my network is a little more convoluted but much more simple to set up.  First I need an old cheap computer that will act as a bridging router for VPN traffic to my internal network.   The VPN client bridging server will obtain the IP address 192.168.4.253.  My VPN bridging client server has to run Server 2003 R2 from Windows because I didn’t feel like finding a Linux operating system that bridges correctly.  My idea is to bridge the VPN connection to the local network.  The specs might include a single core Pentium 4 processor at 1.8Ghz, 256MB of ram, and 2 network cards.  Since this server will not have a fully open connection (it will be able to connect to the internet only if it requests traffic from a website), I don’t mind having Windows on it.  Furthermore it won’t request anything but Microsoft Updates on a manual basis so I am also not going to install Antivirus since it most likely won’t get a Virus if I keep it up to date manually on a monthly schedule.  Finally, it will act as a DHCP server to the local network since OpenVPN will only assign 1 IP per connection.

The first thing we need to do is configure the server, the 192.168.4.253 interface to the local VPN connected network needs to have a static IP set now (255.255.255.0 Subnet Mask and nothing else), to do this, you will need to connect it to a router in your network.  Keep in mind that this router can’t touch your original network that contains 192.168.5.0 addresses.  We need to have the other interface DHCP enabled to grab an address dynamically from the 192.168.5.0 internal network connected to my University through a router.  Now we need to install all the critical updates to the Windows Server.  After several update cycles and reboots, we need to install the DHCP role.  When doing this, allow it to assign addresses from 192.168.4.220 to 192.168.4.250 (30 IP Addresses).  OpenVPN will mostly likely never assign addresses this high so you should be safe.  DHCP will automatically assign these addresses to the internal 192.168.4.253 interface because it is the only static interface on the system.  Now we need to install OpenVPN.  This is simple too.  Once done, goto the configuration directory and put all the extracted files from keys.zip (that you emailed yourself earlier via nail) into this folder.  Also make a new client.ovpn file that contains the following configuration in that directory.

client
route-method exe
route-delay 2
dev tap0
proto udp
remote YOURSERVERIPADDRESS 443
resolv-retry infinite
nobind
persist-key
persist-tun
ca ca.crt
cert mths.crt
key mths.key
ns-cert-type server
comp-lzo
verb 3
ping 15
ping-restart 200
ping-timer-rem

At this point, you should start OpenVPN and connect on the windows machine.  It should connect and give you the green.  To test that it works, goto http://canyouseeme.org and make sure that your IP is your VPN server’s IP.

Now we need to finally bridge the VPN adapter to the internal network adapter.  This is as simple as going to the network connections folder under the control panel.  Ctrl and click both connections (VPN and Local 192.168.4.253) and with both selected right click on the VPN connection and click bridge connections.  A few moments later they’re bridged.  Now we need to assign a static IP to the bridge again (Isn’t that fun).  Right click the bridge and goto properties.  Set the static IP to 192.168.4.253 with subnet mask 255.255.255.0, gateway as 192.168.4.1 and DNS to opendns or your choice.    Once done, reconnect the openvpn connection, It should work fine.

Now we need one more component to the network, this is the 192.168.4.254 router that should be physically connected to the port of the VPN server (as mentioned earlier).  I used a DD-WRT Linksys router and gave it the host name python.matthouse.us as well as a wireless id of python.matthouse.us.  Its IP is set to 192.168.4.254 and its subnet mask is 255.255.255.0.  I also disabled DHCP on it for internal clients.  Finally to connect the 192.168.5.0 network to our 192.168.4.0 network, we need to plug a cable in from 192.168.5.x to its internet jack.  Now everything should be all set for your internal network.  You can simply plug in a computer and it should get an IP from the DHCP server that will be routed to the OpenVPN server.  Finally for the clients that need public IP traffic routed in, simply give them a static IP or reservation in DHCP that assigns them to 192.168.4.202 or 203, etc.  Now they have a completely un-firewalled DMZ connection.

Obviously, you might want to have the local bridging server do all the connection stuff automatically on startup, for this, we need to create a bat file and put it in the startup folder under the windows start menu.  We also need to configure autologin for Windows Server 2003.

First the inefficient bat file on a 64 bit install of server 03 r2 (but it still works):

c:
cd..
cd..
cd..
cd c:windowssystem32
rundll32.exe user32.dll,LockWorkStation
cd ..
cd ..
cd “C:Program Files (x86)OpenVPNconfig”
“C:Program Files (x86)OpenVPNbinopenvpn-gui-1.0.3.exe” –connect client.ovpn

Now for the autologin, goto Microsoft: http://support.microsoft.com/kb/324737

My login script will lock the computer upon login and execute openvpn to connect automatically.  This will be annoying but a good way to make sure that someone can’t gain access to your server simply by rebooting it.  Also note that the password will be in the registry via plain text.  I know that you can get around this by using something like xynt service, but I opted not to since I don’t anticipate anyone trying to break into my system locally (I am the only one who can gain access to my internal network legitimately and physically).

With that, both ends are now automatic and all set to go.  The final step was to plug in my VoIP device and have DHCP reserve an IP address in the public range for it.  My VoIP device now works perfectly.  The latency is 42ms to the VoIP server since I choose to have that in Chicago with a reputable company.

The final step would be to post the final convoluted network diagram.  This is available at http://famousphil.com/wp-content/uploads/2010/11/netpost.jpg.

Now I have a few final afterthoughts.  I thought it would be great to have an unfirewalled connection to the internet, but unfortunately, now my machines need to run their own firewalls via Symantec.  I also am surprised but Symantec pops up frequently with all sorts of notices of attacks and blocks.  I also used option 254 (or 253 I can’t remember) in DHCP to assign static routes.  Basically these send all my University bound traffic, Hulu and Youtube bound traffic through the University’s connection instead of the VPN connection.  This allows for connections to UB from the inside and saves bandwidth for video watching.  Traffic isn’t cheap and VPN traffic isn’t as fast as using the local University connection via their gateway.

Another thought is that you may need to monitor your own network traffic.  This can be achieved through vnstat on the OpenVPN server.

• cd /root
• wget http://humdi.net/vnstat/vnstat-1.10.tar.gz
• tar xzf vnstat-1.10.tar.gz
• cd vnstat-1.10
• make
• make install
• vnstat –testkernel
• vnstat -u -i eth0
• cd examples/init.d/centos
• cp vnstat /etc/init.d/
• chkconfig vnstat on
• service vnstat start

Now when you want bandwidth statistics, simply type vnstat on the OpenVPN server.

Hopefully this is helpful to anyone else who wants to go through all that I did to get a VoIP adapter to work.  As you can guess, I must really hate Cell Phones, and the truth is that I really do.  I like my cell phone (HTC Eris with Android) for reading email on the go, but I rarely use it as a phone (I avoid that actually).

One final tweak that I made to my network was adding QOS (Quality of Service) control to my network.  My 192.168.5.0 network gives priority to my VPN Client Server, which gives priority to the DD-WRT router which in turn gives priority to my VoIP device.  Therefore, if I decide to do a huge file transfer, my VoIP traffic (phone call) won’t be interrupted and I won’t notice choppiness.  I’ve also noticed that UB gives priority to encrypted traffic on their network, so my VoIP traffic even has an advantage on UB’s network.

Over the past weekend, I finally decided to take the plunge as a system administrator and upgrade my Exchange 2010 server to Service Pack 1.  Overall, SP1 introduces many welcomed improvements, most noteably:

• Outlook Web Access themes are now available, meaning users no longer can complain to me about the crappy looking default theme!
• Outlook Web Access calendars can now be published to the internet either via ICAL format or a live webpage.
• Outlook Web Access now allows for the user to reposition the message body and message list making it easier to read email on a netbook
• The Exchange Management console now allows for administrators to export exchange accounts to PST files which can easily be used to backup individual mailboxes (no more full server backups or expensive backup software)

Overall, I was impressed with the new Exchange 2010 feel and look, I also noticed a significant increase in speed.  Now comes the technical bit that might save some administrator’s arse.

I should note at this point that my Exchange server contains the Hub Transport, Mailbox, and Client Access roles, the server itself is running Server 2008 R2 Datacenter.  I only host 8 Exchange Mailboxes, so having all the roles on a single server for me really doesn’t degrade performance, and it makes sense financially.  The upgrade instructions below for the most part will work on a single server, but the error that I explain will also work on clustered Exchange systems.

First, to upgrade the Exchange server, I strongly recommend first grabbing all of the available Windows Updates via the built in Windows Update application.  Do a reboot to verify that everything updated successfully.  I would strongly recommend doing a full backup at this point.  For me, I started a full backup the night before the upgrade process.

To upgrade Exchange, first, grab the upgrade executable and run that file (its available on the Microsoft Website here).  The file that you download simply extracts the service pack files to the location of your choice.  You will also want to download 5 other updates that are listed here.  Additionally, you will want the Office 2010 filter pack located here.  Make sure that you install the Office 2010 Filter Pack and other updates before you begin the Exchange 2010 upgrade.  I simply told each update to restart later to avoid several reboots which take a long time to complete.

I’d also recommend installing 2 server roles with the prerequisites, these are the IIS 6 Management Console and the IIS 7 Compression addon.  You can add both of these through the Computer Management Console by right clicking on the IIS Server Role and adding these to it.  After this, it is time to reboot the server.

The Exchange upgrade can now be started.  I found that the pre-requisites took about 5 minutes to check and then I was able to quickly click on the upgrade button.  The upgrade took about an hour to get to the MailBox Role upgrade step (basically one of the final steps during the upgrade.  Unfortunately, it produced an error that goes something like:

Mailbox Role Failed

Error:
The following error was generated when “$error.Clear();
buildToBuildUpgrade-ExsetdataAtom -AtomName SystemAttendant -DomainController $RoleDomainController

” was run: “An error occurred with error code ’3221685221′ and message ‘Overlapped I/O operation is in progress.’.”.

There was no way to go back so I had to basically cancel the installation and I was left with an unusable upgrade installation.  Fortunately, the Exchange upgrade is able to pick off close to where it left off, so don’t panic if you get to this step.

The problem isn’t with hardware (I/O is In/Out) like I initially thought, but rather, it is due to a poor implementation of the Microsoft Firewall.  Basically, my Exchange server is configured to allow all traffic to go to and from itself, and then I open up specific ports to the world.  All of my rules are custom rules that are specifically configured to block everything but what is absolutely required.

It turns out that Microsoft Exchange looks for the pre-made rules in the firewall and makes sure that those are enabled.  So instead of checking to see if physical access is granted, it instead looks for a name of a rule (which could have a completely different definition when it comes to actual allowing and blocking of services).  The solution was to export my current firewall policy and make a new policy based on the default firewall settings (so I basically reset the firewall).  I then turned off the firewall profiles (not the service) and restarted the Exchange upgrade and it started a few steps behind where it left off.  Overall, it took about 3 hours to get the upgrade to fully run along with a few reboots.

This isn’t the first time that I’ve seen incompetence in Microsoft software.  If your server is running as a file server, if you don’t have the default file sharing rules enabled (oh, and did I mention they’re very insecure too?), when you goto share a folder, you will get some strange error that has nothing to do with the firewall.  Microsoft really should look at actual permissions, not just the name of a rule.  Oh well, hopefully they will do this by the time Windows 8 is released!

So anyways, the firewall gets me again.  Hopefully the next time I run into a problem, I will look at the firewall first before attempting any solutions.

These programs are IIS 7  (web server, Internet Information Services) and MSSQL Server 2008 R2 (Microsoft Standard Query Language Server).  In the past when I ran my own computers on an un-firewalled internet connection (public wi-fi, home DSL), with home security software installed (like Norton 360), I have also noticed these types of log messages and popup warnings.

Most people (I’d estimate at least 50%) probably have some sort of broadband, un-firewalled, connection setup in their home that is directly connected to their computer.  Most people probably subscribe to some security solution like Norton 360, and they probably run some sort of firewall.  Normally, these firewalls catch all the bad stuff that can harm your computer, but stuff still could potentially come through.

Although there will always be loop holes for these security vulnerabilities, there is another means of protection that most people would NEVER think of!  If you’re thinking Wireless router from the local Wally World (Wal-Mart), you read my thoughts. Yes, Wireless routers don’t just share an internet connection wirelessly like most people think.

So what else does a wireless router do?  A wireless router is simply a ROUTER with a wireless ability built into it.  Routers are complex pieces of engineering that connect many computers together.  Without getting into too much detail, routers connect two separate networks together to bring multiple endpoints together.  The internet has many subnets that are connected to each other through routers. Think of the telephone system when I mention this, more specifically area codes and dialing prefixes.  The area code for Matthouse is 716, the prefix is 584.  So 1-716-584-xxxx gets routed to a particular telephone.   In my example, when you dial the full number, 1 means connect to the main US router which knows all the US phone area codes.  Next, 716 means connect to the router which  handles the Western New York area prefixes, then finally that router sends the call to the router than handles the 584 prefix.  That router then is practically directly connected to the xxxx number which will ring a phone and help establish your connection.  The internet is connected in a similar fashion.

Traffic is sent in internet packets that run on a certain port number.  For simplicity, a port is required to connect to a computer.  Computers listen on ports for connections and there are 65535 possible ports.  You might think of a port like a way to get to your house from the road at your address.  Each drive way is a unique path in and it accepts only a certain type of car.  Hackers tend to send a car into that drive way that acts and looks like the car it accepts, but once it’s in, it can cause havoc in your home (computer).

So what am I getting to? Routers connect different networks, so they inherently have to forward all the traffic from one network to another, including all the ports.  Since ports are easy ways to get into your computer (provided your computer is actively listening / accepting on that port), hackers tend to go for these ports.  Some ports on web servers (like port 80) are absolutely necessary to leave open, but other ports like 5109 (which happens to be the AOL Instant Messenger port) probably isn’t needed on that web server.   For a home computer, blocking all the ports inbound to the computer is probably smart, while allowing all the outgoing ports from the computer to the internet.

NOTE: I probably should add that with outbound connections through firewalls, if you request something from an external source (say a webserver) while having all incoming connections blocked, you will still get the response from that external source.  Firewalls are smart about allowing replies back through while blocking all new connections that are probably hacker initiated.

BOTTOM LINE: All computers have different needs. A wireless router when added to your network will block all incoming ports by default and allow all outgoing connections on all ports.  Therefore, by adding one of these cheap boxes, you’re not only gaining a wireless network access point, but you’re also protecting yourself from the nasty dangers of hackers that probe computers for open listening ports.  Since many ISPs provide un-firewalled public IP addresses to residential customers, those customers would be wise to install one of these routers.  Who knows, it might save their computer from a severe attack from a hacker some day!  I’ve also found that when I run a firewall in terms of a router, I don’t need as much protection from Norton 360 on my computer, so I basically have a faster computer (it isn’t working on blocking bad stuff anymore).

Hopefully this helps you!

Over the past couple of months, I’ve been wanting to put together a Windows based web server for myself.  Its purpose was originally to keep my Outlook 2007 application on Exchange open (to update my website calendar / update my webmail RSS feed folder).  I’m beginning to run other stuff on the server (e.g. my instant messenger) so that I can have a better online presence (lets face reality, email isn’t as live as some of my friends would like).  Since I chose to run Windows Server 2003, I decided that it might be worth my time to enable Internet Information Services and work on developing some ASP based web applications just for the fun of it.  Of course, when the time is right, I will open up a few more accounts on the server through Matthouse for others who share my desire to learn ASP for fun.  For those of you who are wondering, I’m waiting for a good billing system to come along, I have one, I’m simply waiting on the developer to make a few changes.

Anyways, today I was playing around with the server and I enabled FTP publishing so that I could easily upload a few files to the server through my local computer.  Unfortunate for me, going to the windows components page under the control panel’s add/remove programs section to add FTP Publishing wasn’t enough.  Sure enough, the server did install FTP and made it available in the Internet Information Services Manager section of the computer management console.

After I deleted the default ftp site and made a new site with isolated user home directories, ftp didn’t work correctly.  Note that this server does not connect to nor run Windows Active Directory, so all the accounts are local.    Anyways, after some research, I found that under the ftp root, I needed to add a folder called “LocalUser” and then I had to add the username of the account under that directory.  To ensure that other users could not read files / data the directory, I changed permissions on the user folder to allow only that user and administrators access. At this point FTP began to sort of work!

I say sort of work because FTP only worked in active mode, meaning the Windows Firewall was getting in the way.  After spending a few minutes looking for a solution, I figured that I would try adding the internet information services executable to the exceptions list.  Sure enough, this fixed the passive mode problem.  The steps I took to fix the problem were: 1. open up the Windows Firewall control panel console under the Windows control panel.  I went to the exceptions tab and added a problem.  I browsed for additional programs and added “C:\WINDOWS\system32\inetsrv\inetinfo.exe” to the exceptions list.  After this was added, passive mode worked as expected.

Although Server 2003 is outdated by now, it is still a very stable system to do any experimental work on, which is why I choose to run it over a better and never version of Windows.   It is also much cheaper to run than a newer version of Windows like the current Exchange Server which is running Windows Server 2008 R2 Datacenter.