Unfortunately, computers are not able to think and identify patterns without a lot of help and special cases already entered into databases for them to compare against.  Thankfully, emailing lists have certain unique identifiers that allow this application to effectively identify and remove users from mailing lists, although I will not disclose exactly how it is done since that research was done on company time and therefore remains property of Conceptual.ly.  But I did want to take the time to publish a cheat sheet for those of you who need to communicate with IMAP servers via telnet.

CONNECTING: The general command to telnet to an insecure IMAP server is “telnet imap.server.com 143”, remembering that IMAP runs on port 143.  If you’d like to connect through IMAPS over port 993, your computer will need OpenSSL and the command to do so is “openssl s_client -connect imap.server.com:993”.  OpenSSL will take care of all the connection securing / establishing for you, the commands following the initial connection are exactly the same.

IDENTIFIERS: IMAP commands start with an identifier, I’ve seen “. “ as the identifier (a dot followed by a space) and I’ve also seen “a1 “ (a1 followed by a space) as the identifier.  I’ve also seen incrementing the 1 to 2, 3, etc work, and reading the RFC was not specific to what had to be used.  I typically use “. “ as my identifier and fall back to using a1 for all of the commands I enter, it has worked for every IMAP implementation I’ve dealt with thus far (including Gmail, Dovecot, AOL, and Exchange).

LOGIN: To login to the IMAP server, you need to enter the identifier followed by “login username password” replacing username and password with the appropriate login information.  Note that your username may be your email address.

LIST: This command is typically followed by a login and lists all of the available mailboxes in the IMAP account.  The syntax for this command is list “” “*”  which simply gets all of the mailboxes and subfolders.  The usual response is * LIST (\HasNoChildren) “.” “INBOX” which can easily be parsed, the \HasNoChildren can change if there are children.  Usually with children, the output will make the actual folder (the last segment of the output) contain the parent and a . so that you can simply use this to formulate a connection to a specific folder to retrieve messages, etc.  This isn’t always the case, so it is helpful to use IMAP over telnet to verify the response of the command for the specific IMAP server that you are connecting to.

STATUS: This command will return information about the folder you inquire about, I’ve only seen it return the total messages and unread messages in a folder in my experiences.  The syntax of this command is “STATUS FOLDER (flags which are optional)”.

SELECT: In order to search or retrieve items from a folder, you must first select that folder.  The syntax of this command is “SELECT FOLDER”.

SEARCH: This command has some ambiguity that caused me some difficulty that I will get to shortly.  The syntax for search is “SEARCH (PARAMS)”.  The parameters are identified in the RFC3501, but there are some specifics I’d like to talk about.  First, I’ve found that SENTSINCE does not need to be in (), instead you would do “SEARCH SENTSINCE 01-01-11 (PARAMS)”.  Next, if you want to search for more than 2 items using the OR syntax, you would need to use nested ORs, this might look like the following: “(OR (OR HEADER From yahoo.com BODY hello) BODY phil)” to search for a message that might be from yahoo.com (partial match) with the body containing hello, or just having “phil” in the body.  This works similarly for the AND parameter.  When search returns, it returns a space separated array of numbers which are identifiers for the FETCH command.  Remember that to search, you have to SELECT the folder to search first.

FETCH: This command will retrieve messages from a folder, you must have selected that folder first before you can retrieve a message.  Typically, messages are identified by a number which the FETCH command uses, so a SEARCH is generally executed before fetching messages.  The syntax is “IDENTIFIER FETCH (PARAMS)”.  IDENTIFIER is the number that search returned, and there are numerous PARAMS that you can do.  I’ve seen (RFC822) work at grabbing the entire raw message source (both headers and body), but in AOL’s implementation it does not, so I’ve reverted back to getting the header and body of each message through 2 fetches, the first parameter is (RFC822.HEADER) and the second is (RFC822.TEXT).  I do not believe that IMAP servers return a decoded version of the message in all cases, so I also used an email parsing engine in Python to decode the message source that these commands returned.

LOGOUT: The syntax for this command is “LOGOUT”.  It simply disconnects you properly from the IMAP server.

For the needs of my program, these were all the commands I needed, so I’d strongly suggest that you look at RFC 3501 at http://tools.ietf.org/html/rfc3501 for more information about the IMAP protocol and its usage.

I would also like to note that GMail has added commands to their implementation that better follow the speed and reliability of their web interface, those commands can be found at http://code.google.com/apis/gmail/imap/ (this was kind of hidden from my initial searches for information about GMail’s IMAP implementation, so I wanted to make this more visible).  If you use standard IMAP commands on Gmail, you can expect it to be much slower than other IMAP servers and much slower than their web interface.

Finally, AOL’s IMAP implementation doesn’t allow searching certain header fields and the body of the message, so I’d recommend testing your commands ahead of time to ensure you are getting responses that you expect.  As a tradeoff, AOL’s IMAP implementation is very fast at returning messages via the FETCH command, so where the SEARCH lacks ability on AOL’s IMAP server, the speed in downloading messages far makes up for this lack of functionality.  It is also possible that AOL’s IMAP implementation has certain extensions that I didn’t notice, so you may want to look into that possibility also if you’re dealing with their IMAP servers.

Hopefully this blog post can help someone save time so that they don’t have to do as much searching as I had to for connecting to IMAP servers.

Now onto the rest of my blog (and the main reason why I’m even blogging)

I really think its time I take the time to update other parts of my site.  My blog gets a lot of my attention because it is fun to write when I get a chance to write about stuff I care about!

Lets start with why am I taking this sudden initiative to update the rest of my site.   A few days ago, I was talking to John who has a low page rank (PR) on his blogging site at johnciacia.com.  John wants to increase his PR.  I currently have a much higher rank than he does but I’d like a higher rank too.  Higher PR means more popularity and higher search rankings for common key words.

Why would I want a higher PR?  Higher PR means my popularity will go up on search engines and I will get more traffic to my site.  I really want to get 10,000 people reading my blog within a few years.  I feel this is a very obtainable goal because I tend to write 2 to 3 very technical articles per month that aren’t easily obtainable elsewhere on the web.  I constantly find the Microsoft knowledgebase is insufficient at helping me fix my own problems, so when I come up with a solution elsewhere, I tend to post it either here or on Adminreference.com.  I have looked back on AR quite often to find my own solutions to recurring problems due to corruption, etc.

Now time to move onto my photo gallery

I finally got around to putting a few photos up from the same old photos forever.  I have yet to filter out the spam comments, but they will go eventually.  Within the next week or two, I plan on posting a few of my nature photo galleries that I’ve kept on my computer now for quite some time.  These include a fox that comes to our house quite often and a rare shot of baby skunks.

Anyways, I would like to point your attention to my latest gallery: http://famousphil.com/gallery/?level=album&id=4 (You can get to this from the main page by clicking the photos navigation link).  This is of a practice barn burning for the Humphrey Volunteer Fire Department (no website available).  Our Fire Department is very small and right down the road from us.  We had a barn on our property that was falling down and we felt it was time to burn it before someone got hurt in it from beams falling, etc.  Anyways, I got a few really cool shots and thought I’d share them   Look forward to more nature photos in a few days.

Over the past week, I got a new server to host Microsoft Exchange which is a powerful email server from Microsoft.  Before you go all crazy on Microsoft (I know I typically do), Exchange is one of the few excellent products they make.  I am actually very hard pressed to find anything that compares to it that is open source and can easily run on Linux which 99% of  my hosting business up until now has ran off from.  Man, I never thought that I would say that

So the first logical question is, why move your email to exchange?  As you know, I’ve had 1and1 mailxchange now for quite some time.  I really wanted a solution that would sync my calendar, contacts, tasks, files, and email to every device I use on a daily basis.  Mailxchange was that solution but there are many problems. First the web client is very slow,  sure its flashy, but it takes 5 minutes to load on my connection (that is fairly fast).  I don’t have the time to wait on this client to load.  The next problem is it needs custom software to connect to Outlook and Mobile Devices, I’m not into installing “connectors” to software when it has functionality built in with other products.  Perhaps one of my biggest problems is the level of support I’ve gotten from 1and1.  My mail has gone down on a few occasions and I’ve been unable to easily send a support ticket in asking what is wrong.  I’m not even sure if 1and1 backs up my email and I have no method of backup, so I’m kind of stuck if they go down or don’t back up the server.  Its kind of scary actually since I save all of my email.

So about 2 weeks ago, I started talking to a few friends.  I know that I get a free msdn copy of Microsoft Exchange 2003 and Server 2003 from my University.  I figured if I could find a few friends who were interested in small mailboxes on exchange, I could cover the cost for the hardware to host my copies of this software.  I figured that I could host 4 people and handle a server that costs $25 a month from 3dgwebhosting which I’ve had in the past and they run excellent hosting on Windows server 2003.  They cover the license cost, so I’d only be covering exchange.  The downfall was I would only have 10GB to work with which isn’t a lot for email and backups.  Because of this, I looked for alternate hosting.  I decided that if I could find xen hosting, xen would support Windows.

About this time when I was looking, I knew that http://fsckvps.com who is a child company of vaserv in England hosted xen vps machines.  I went to that site to look up their support email and found out about the horrible hypervm owner hanging and they were down.  Anxious to get this hosting off the ground, I began looking at alternate places for hosting.  Shortly after, I found good reviews on other blogs of a new hosting company called Elite Data Hosting.  I contacted them about a 10mbps plan to host exchange on and they got an account for me on a xen vps using my server key.  I’m basically paying $15 a month for ~325MB of ram and 30GB of hard disk space.  The server is a high end server and I have had no complaints.  They even took the time to install Windows for me from my disk!

Elite Data Hosting is good news for me because I now can have my 2 guaranteed friends and myself have a guaranteed 5GB of space for files / mailboxes a piece.  It will also be very easy to automate backups of these mailboxes.  We all split the $5 a month cost for the server so I’m basically paying what I would be paying 1and1 but I control my backups and have a better piece of mind.

So now I started the daunting task of setting up the Exchange server. Normally with Microsoft products, it takes about 5 seconds and about 10 clicks of the next button to install software and another 3 minutes to say configure this software to do this.  By that point, everything normally works flawlessly (except for the occasional crashes of Microsoft Windows).  On linux, there is always a lot of configuration, but linux always works without the crashes and instability.  Perhaps this is the way to tell what is good and bad???

To get back to Exchange, I must say, this is the hardest piece of software I have ever had to install on both Linux and Windows.  Part of the reason is the way Exchange relies on existing Server 2003 infustructure to improve itself.  I’m not so sure if I’d rely on a Windows Server operating system, but I really have no choice with Exchange.  Exchange requires Active Directory among other server features to run correctly and the prerequisite list is a nightmare to get through in less than 5 hours if you ask me.  I started with a clean server a week from last Tuesday and didn’t get Exchange running until about Monday and I had 8 hours a day into it at the very least.  I will take part of the blame for not knowing what I was doing past Active Directory configuration, but Exchange was no day at the beach to figure out.  I also had a lot of errors that I spent hours reading about to find simple fixes.  Finally after all of the struggle, I got exchange fully working to the point where I wanted it about 2 days ago. During my struggle, I posted a lot about my solutions on Admin Reference which is my site where I post solutions to all of my problems.  I picture it as another *free* experts exchange but more tutorial based than question based.  Maybe some day it will do a little of both   That is my goal anyways!

One side note that I should add is, when I first loaded Outlook Web Access, I got a crappy looking interface.  I found out quickly that Exchange only supports Internet Explorer in its premium interface (the one that looks nice and loads quick).  Sadly, this is the only reason why I have opened Internet Explorer, and I have found that Firefox can open an IE tab, so I’ve began using that.  I will also likely find a solution when I migrate completely to Linux (my next upcoming project).

So now that Exchange works, what was so difficult? Most of my difficulty was from I never managed an exchange server in the past, and I couldn’t find any decent documentation on how to do it. That is why I posted a lot to Admin Reference unlike I normally would.  My biggest issue was the domain errors which were caused by firewalls and figuring out how to get Outlook Web Access and Outlook Mobile Access working with SSL encryption.  I also was not prepared to spend money on an SSL certificate (required by exchange) and provide antivirus / spam scanning to the server.  I was under the impression that spam/virus protection was built in, but it isn’t, and the freeware gfi version is no longer free.  I figured out how to migrate linux spamassassin to the server and that is adequate for spam protection

One last question that I should cover is why didn’t I go with Exchange 2007?  I will admit that Exchange 2007 is very nice software, but there are a few problems:

• My first issue would be, Exchange 2007 is really bloated.  If you compare the 2003 to 2007 installation disks, the 2003 install disk is about 300MB, the 2007 version is closer to 1.7GB.  That is a huge difference, one that I’m not willing to upgrade for.
• My next issue is due to the bloat, I would need a much powerful server.  I could upgrade to the 600MB RAM server plan with a 50GB hard disk or so for 30 dollars a month, but then I would have to start hosting more mailboxes than I’d want to to cover the costs, and I’m not really into that idea.  I might upgrade for 2003 if people are interested and it won’t take too many server resources or hurt my rigged spam fighting solution, but that is a decision that I’d rather not make now since it works perfectly as is!
• My final issue is, newer software normally sucks.  I always wait for at least Service Pack 1 (2 if possible) until I start using a product mainstream.  Exchange 2003 is at SP2 while Exchange 2007 is at SP1.  With other Microsoft software, I’ve found that when I compare a fresh install of Server 2003 to Server 2008: Microsoft Server 2003 with a GUI (Graphical User Interface or your windows desktop) uses 400MB on a new install, while the Microsoft Server 2008 Core Edition (no desktop, strictly command line to reduce bloat) uses 800MB on a new install with nothing configured.  This is a huge jump and I have a feeling that Exchange 2003-2007 will be very similar (the requirements for 2003 is 256MB of ram, 2007: 2GB of ram).  BIG DIFFERENCE, huh!

All in all, I figure I am paying about $200 bucks total for my new email solution, but my friends really do help cut the cost down to where I can happily afford it.  I still have 1 slot open but have a feeling that will be closed before long.  For a private email server, I consider it an excellent learning experience, and a good way to get some good content on Admin Reference! Hopefully you got some helpful tips out of this.

One final note:  I’d like to put a plug out there to any other system admins.  If you are like me, you are always running into new problems that don’t have easy solutions.  Why not take a few minutes when you find the answer and post it to Admin Reference?  Maybe someday you will look back on it (I know I have) and say thats how to fix it!  Someday when it gets a little more material, I plan on integrating the forum into a wiki that is easily searachable for solutions to problems.

So here is the comment to a blog “Google features and what I utilize from Google“,

User: Afforsenerede
email: aninuebit@gmail.com
ip: 209.239.112.48
submitted on 2009/04/19 at 11:20am:

“Yooo..

I kno it has nothing to do with what you wrote, but have you ever heard of http://www.bluestickers.info/ringtones.php . They seems to promise free ringtones

PS. Dont be an ass, this is NOT spam ”

Why did I post this although it only deserved a deletion?  It is quite simple, I get enough spam email a day to know that when I post my email on a public site that is quite popular, I will get more spam to that address. My goal is to get this spammer a lot more spam to hopefully overwhelm him to where he has to increase his spam filter and possibly start flagging legitimate email as spam.

To get around spam, I implement a system to where my emails are filtered at the address level then forwarded on through more filters on other servers before being forwarded to a central email address that I have set up.  My central email address is not filtered because I rely on email to be filtered before it gets to my central email account.  This way if I notice a high volume of spam, I can simply increase the filtering on public addresses and start deleting legitimate email without too much worry.  People who know me probably know one of my lightly filtered addresses and can send the same email to multiple addresses to ensure delivery.

If the above confuses you, I simply have a bunch of public email addresses that I filter then forward to an unfiltered address that I look at email from.  Spam doesn’t bother me much anymore, especially since I use a text email client that doesn’t show pictures and makes deleting a message as easy as the “d” key (this is called pine).  Deleting 100 spam messages takes me about 1 minute now compared to about 5 minutes with a web based client like gmail or yahoo mail or hotmail.

Pine is a great program and all of my hosting clients may use pine on my server through ssh, I have compiled pine with password file support (.pinepass).  Pine is a bit tricky to set up, but I posted a little bit about the process on adminreference.

Finally, I would like to thank this spammer that goes by the name: Afforsenerede for getting me on this topic.  I will have a reply out to you soon   I also intentionally removed the link to that info site just to be a bastard .