Posted on: November 13th, 2011 by Famous Phil

It has been some time since my last post, and I think this is going to be the norm for at least the next year, sadly.  Today’s topic is based on information that I needed about IMAP (Internet Message Access Protocol) last week. Basically, I’m developing an application for conceptual.ly that allows for normal users to enter their login information for their email and then the application will unsubscribe them from internet mailing lists that people often dislike, yet get plentiful email from.  Since the solution is completely artificial (and requires no human intervention), it exceeds what others have done in the past.

Unfortunately, computers are not able to think and identify patterns without a lot of help and special cases already entered into databases for them to compare against.  Thankfully, emailing lists have certain unique identifiers that allow this application to effectively identify and remove users from mailing lists, although I will not disclose exactly how it is done since that research was done on company time and therefore remains property of Conceptual.ly.  But I did want to take the time to publish a cheat sheet for those of you who need to communicate with IMAP servers via telnet.

CONNECTING: The general command to telnet to an insecure IMAP server is “telnet imap.server.com 143”, remembering that IMAP runs on port 143.  If you’d like to connect through IMAPS over port 993, your computer will need OpenSSL and the command to do so is “openssl s_client -connect imap.server.com:993”.  OpenSSL will take care of all the connection securing / establishing for you, the commands following the initial connection are exactly the same.

IDENTIFIERS: IMAP commands start with an identifier, I’ve seen “. “ as the identifier (a dot followed by a space) and I’ve also seen “a1 “ (a1 followed by a space) as the identifier.  I’ve also seen incrementing the 1 to 2, 3, etc work, and reading the RFC was not specific to what had to be used.  I typically use “. “ as my identifier and fall back to using a1 for all of the commands I enter, it has worked for every IMAP implementation I’ve dealt with thus far (including Gmail, Dovecot, AOL, and Exchange).

LOGIN: To login to the IMAP server, you need to enter the identifier followed by “login username password” replacing username and password with the appropriate login information.  Note that your username may be your email address.

LIST: This command is typically followed by a login and lists all of the available mailboxes in the IMAP account.  The syntax for this command is list “” “*”  which simply gets all of the mailboxes and subfolders.  The usual response is * LIST (\HasNoChildren) “.” “INBOX” which can easily be parsed, the \HasNoChildren can change if there are children.  Usually with children, the output will make the actual folder (the last segment of the output) contain the parent and a . so that you can simply use this to formulate a connection to a specific folder to retrieve messages, etc.  This isn’t always the case, so it is helpful to use IMAP over telnet to verify the response of the command for the specific IMAP server that you are connecting to.

STATUS: This command will return information about the folder you inquire about, I’ve only seen it return the total messages and unread messages in a folder in my experiences.  The syntax of this command is “STATUS FOLDER (flags which are optional)”.

SELECT: In order to search or retrieve items from a folder, you must first select that folder.  The syntax of this command is “SELECT FOLDER”.

SEARCH: This command has some ambiguity that caused me some difficulty that I will get to shortly.  The syntax for search is “SEARCH (PARAMS)”.  The parameters are identified in the RFC3501, but there are some specifics I’d like to talk about.  First, I’ve found that SENTSINCE does not need to be in (), instead you would do “SEARCH SENTSINCE 01-01-11 (PARAMS)”.  Next, if you want to search for more than 2 items using the OR syntax, you would need to use nested ORs, this might look like the following: “(OR (OR HEADER From yahoo.com BODY hello) BODY phil)” to search for a message that might be from yahoo.com (partial match) with the body containing hello, or just having “phil” in the body.  This works similarly for the AND parameter.  When search returns, it returns a space separated array of numbers which are identifiers for the FETCH command.  Remember that to search, you have to SELECT the folder to search first.

FETCH: This command will retrieve messages from a folder, you must have selected that folder first before you can retrieve a message.  Typically, messages are identified by a number which the FETCH command uses, so a SEARCH is generally executed before fetching messages.  The syntax is “IDENTIFIER FETCH (PARAMS)”.  IDENTIFIER is the number that search returned, and there are numerous PARAMS that you can do.  I’ve seen (RFC822) work at grabbing the entire raw message source (both headers and body), but in AOL’s implementation it does not, so I’ve reverted back to getting the header and body of each message through 2 fetches, the first parameter is (RFC822.HEADER) and the second is (RFC822.TEXT).  I do not believe that IMAP servers return a decoded version of the message in all cases, so I also used an email parsing engine in Python to decode the message source that these commands returned.

LOGOUT: The syntax for this command is “LOGOUT”.  It simply disconnects you properly from the IMAP server.

For the needs of my program, these were all the commands I needed, so I’d strongly suggest that you look at RFC 3501 at http://tools.ietf.org/html/rfc3501 for more information about the IMAP protocol and its usage.

I would also like to note that GMail has added commands to their implementation that better follow the speed and reliability of their web interface, those commands can be found at http://code.google.com/apis/gmail/imap/ (this was kind of hidden from my initial searches for information about GMail’s IMAP implementation, so I wanted to make this more visible).  If you use standard IMAP commands on Gmail, you can expect it to be much slower than other IMAP servers and much slower than their web interface.

Finally, AOL’s IMAP implementation doesn’t allow searching certain header fields and the body of the message, so I’d recommend testing your commands ahead of time to ensure you are getting responses that you expect.  As a tradeoff, AOL’s IMAP implementation is very fast at returning messages via the FETCH command, so where the SEARCH lacks ability on AOL’s IMAP server, the speed in downloading messages far makes up for this lack of functionality.  It is also possible that AOL’s IMAP implementation has certain extensions that I didn’t notice, so you may want to look into that possibility also if you’re dealing with their IMAP servers.

Hopefully this blog post can help someone save time so that they don’t have to do as much searching as I had to for connecting to IMAP servers.

Tags: email, IMAP, IMAPS, OpenSSL, Protocol, telnet
Posted in Programming, Technology
|| No Comments »

Posted on: July 7th, 2009 by Famous Phil

First, I think I should put in a plug for exchange hosting. I got exchange 2007 up after deciding to get a bigger server and host a few more people.  Exchange 2007 is a bit more bloated but has nicer spam fighting features and has a much nicer interface in browsers like firefox.  I’m still looking for a few people to help me pay for the server.  I put all the details on codecall in this thread: http://forum.codecall.net/hosting-registrars/18780-exchange-2007-email-hosting-offer-personal-server-looking-4-more-people.html.  If you’re interested, feel free to drop me an email

Now onto the rest of my blog (and the main reason why I’m even blogging)

I really think its time I take the time to update other parts of my site.  My blog gets a lot of my attention because it is fun to write when I get a chance to write about stuff I care about!

Lets start with why am I taking this sudden initiative to update the rest of my site.   A few days ago, I was talking to John who has a low page rank (PR) on his blogging site at johnciacia.com.  John wants to increase his PR.  I currently have a much higher rank than he does but I’d like a higher rank too.  Higher PR means more popularity and higher search rankings for common key words.

Why would I want a higher PR?  Higher PR means my popularity will go up on search engines and I will get more traffic to my site.  I really want to get 10,000 people reading my blog within a few years.  I feel this is a very obtainable goal because I tend to write 2 to 3 very technical articles per month that aren’t easily obtainable elsewhere on the web.  I constantly find the Microsoft knowledgebase is insufficient at helping me fix my own problems, so when I come up with a solution elsewhere, I tend to post it either here or on Adminreference.com.  I have looked back on AR quite often to find my own solutions to recurring problems due to corruption, etc.

Now time to move onto my photo gallery

I finally got around to putting a few photos up from the same old photos forever.  I have yet to filter out the spam comments, but they will go eventually.  Within the next week or two, I plan on posting a few of my nature photo galleries that I’ve kept on my computer now for quite some time.  These include a fox that comes to our house quite often and a rare shot of baby skunks.

Anyways, I would like to point your attention to my latest gallery: http://famousphil.com/gallery/?level=album&id=4 (You can get to this from the main page by clicking the photos navigation link).  This is of a practice barn burning for the Humphrey Volunteer Fire Department (no website available).  Our Fire Department is very small and right down the road from us.  We had a barn on our property that was falling down and we felt it was time to burn it before someone got hurt in it from beams falling, etc.  Anyways, I got a few really cool shots and thought I’d share them   Look forward to more nature photos in a few days.

Tags: barn, burn, burning, department, email, Exchange 2007, fire, Google, humphrey, page rank, pr, search, seo
Posted in Hosting / Server Administration, Personal, Technology
|| 7 Comments »

Posted on: June 20th, 2009 by Famous Phil

First I guess I should apologize for not posting anything in the past week here!  I really hate it when I have to ignore my own blog for more important admin business elsewhere.  The good thing is, I always manage to learn a lot of new stuff that I can easily share

Over the past week, I got a new server to host Microsoft Exchange which is a powerful email server from Microsoft.  Before you go all crazy on Microsoft (I know I typically do), Exchange is one of the few excellent products they make.  I am actually very hard pressed to find anything that compares to it that is open source and can easily run on Linux which 99% of  my hosting business up until now has ran off from.  Man, I never thought that I would say that

Read the rest of this entry »

Tags: 1and1, 3dgwebhosting, active directory, Admin Reference, administration, Elite data hosting, email, exchange, exchange 2003, Exchange 2007, fsckvps, mailxchange, Microsoft, Microsoft Exchange, mobile, outlook, outlook web access, owa, server 2003, vaserv, vps, windows
Posted in Hosting / Server Administration, Mobile Technology, My Site
|| 3 Comments »

Posted on: April 20th, 2009 by Famous Phil

So today I was reading through my comments to approve and disapprove suggestions / comments on blogs.  I usually get about 30 to 40 spam comments a week that I have to manually delete.  This time I got a funny spam comment that I thought deserves its own blog (no I didn’t approve it, rather, I promoted it!).  Just to be an “ass”, I thought I would also post this user’s IP address and email publically.  You may feel free to spam him

So here is the comment to a blog “Google features and what I utilize from Google“,

User: Afforsenerede
email: aninuebit@gmail.com
ip: 209.239.112.48
submitted on 2009/04/19 at 11:20am:

“Yooo..

I kno it has nothing to do with what you wrote, but have you ever heard of http://www.bluestickers.info/ringtones.php . They seems to promise free ringtones

PS. Dont be an ass, this is NOT spam ”

Why did I post this although it only deserved a deletion?  It is quite simple, I get enough spam email a day to know that when I post my email on a public site that is quite popular, I will get more spam to that address. My goal is to get this spammer a lot more spam to hopefully overwhelm him to where he has to increase his spam filter and possibly start flagging legitimate email as spam.

To get around spam, I implement a system to where my emails are filtered at the address level then forwarded on through more filters on other servers before being forwarded to a central email address that I have set up.  My central email address is not filtered because I rely on email to be filtered before it gets to my central email account.  This way if I notice a high volume of spam, I can simply increase the filtering on public addresses and start deleting legitimate email without too much worry.  People who know me probably know one of my lightly filtered addresses and can send the same email to multiple addresses to ensure delivery.

If the above confuses you, I simply have a bunch of public email addresses that I filter then forward to an unfiltered address that I look at email from.  Spam doesn’t bother me much anymore, especially since I use a text email client that doesn’t show pictures and makes deleting a message as easy as the “d” key (this is called pine).  Deleting 100 spam messages takes me about 1 minute now compared to about 5 minutes with a web based client like gmail or yahoo mail or hotmail.

Pine is a great program and all of my hosting clients may use pine on my server through ssh, I have compiled pine with password file support (.pinepass).  Pine is a bit tricky to set up, but I posted a little bit about the process on adminreference.

Finally, I would like to thank this spammer that goes by the name: Afforsenerede for getting me on this topic.  I will have a reply out to you soon   I also intentionally removed the link to that info site just to be a bastard .

Tags: email, pine, smart, spam
Posted in Hosting / Server Administration, Technology
|| 1 Comment »