Blog entries tagged with "email"

Three months ago I switched away from an iPhone to an Android based device. About a week after it arriving by courier the iPhone 4 was released in Australia and I began writing a post about my first week and a bit with the Samsung Galaxy S (SGS).

I didn’t get very far as the week turned into a month, then into two and now into three. Part of my excuse is that the official release of Froyo (Android 2.2) for the phone was imminent. That is still to happen, but of course it will be released for Australia tomorrow now that I have written this post.

So what is the phone like?

To be honest it is nothing special, because it does exactly what I expect from a device of its kind. It does phone calls, text messages and most importantly it has a web browser. Of course it has other features, which have been occasionally useful. These include a camera, playing videos, viewing photos, listening to music and downloadable apps. But these are all expected.

One thing that did surprise me was the Gmail client. At first I still had my email hosted at home so I tried out a few IMAP clients. None of them worked as well as I wanted them to.

However, once I moved my mail up into Gmail I was able to use the Gmail app, which, unlike the IMAP apps I tried, just worked. Actually, no. It didn’t just work, it worked extremely well. It works so well that I am happy to state that anyone reading email on an Android device that isn’t Gmail is doing it wrong.

I would also take that statement the other way, that anyone using IMAP to access Gmail is also doing it wrong. The Gmail model of messages is different enough that IMAP has to make compromises to work at all. I assume there are some people that can make Gmail over IMAP work reliably, but the majority of people don’t and that just seems to cause problems.

But back to the phone…

During the three months I have played around with other aspects of the phone (including SL4A, the Scripting Layer for Android, which allowed me to control the phone using perl) but none have progressed from playing to regular use.

What I have realised from this is that I am too lazy to care about shiny things, I need them to be practical. The SGS has proved to be that, practical.

After seeing that external mail was being delivered correctly to my Google Apps domain, I changed my rss2email config to deliver to Google Apps instead of directly to the local mailbox.

When filtering with procmail you can filter on anything you want, but in most cases a few regexes will get you what you need. In contrast the filtering options in Gmail are extremely limited. There are a couple of headers you can match against, and then just a simple string match.

I was able to tweak rss2email to add the URL of the RSS feed as the List-Id header. I can then setup a filter for each RSS feed. Again, like forwarding addresses, I monitor a lot of RSS feeds. Over 120 which means I will also need over 120 filters just for RSS feeds.

What would work is for the RSS feeds to be categorised before I send out the emails, I then only need a filter per category. An additional benefit is that when another RSS feed is added, a new filter is not required.

As I am considering further modifying rss2email (or replacing it completely), what else could I do?

Something that I don’t like about Gmail (and certain other mail clients) is that the message lists display the time the message was received by Google, not the time it was sent or the time in the message headers. This means that the RSS messages are clumped together because the script only runs once every few hours.

This cannot be changed as long as the messages are delivered via SMTP. But, thanks to a small project at work, I know that if I were to write the messages in directly via IMAP, the dates will be what I want.

If I were writing the messages in via IMAP, filters will not be run, but writing the messages directly to the appropriate label means that the filters are not even needed. I would have no idea how to modify rss2email to use IMAP, so I would be writing my own solution from scratch.

This method of direct injection via IMAP is also how my identi.ca/Twitter/Facebook updates should be delivered. In this case I indend to write something that uses the appropriate API, not the RSS feed as is the case for identi.ca and Twitter. I already use the API for Facebook, but only to produce an RSS feed that is then picked up by rss2email. It is a bit convoluted, bit it has worked.

Yes, I know I should get around to finishing my post about how I am finding Android, but other things are happening that are all interrelated to some degree.

Some time ago I started to setup Google Apps for my domain. Yesterday, after recovering my password, I completed that setup and got to the point where new mail would be delivered to Google. I also have RSS items delivered there as well, but more on that later.

One of the things that made me hesitant about switching to Google Apps was in regard to forwarding/nicknames/aliases. Whenever something asks me for an email address I will create a specific forwarding address. I do not handle this through an email catchall as it is possible to delete a forwarding address if it is no longer needed, or starts to get too much spam.

I currently have 150 forwarding addresses. Not too long ago I had over 200, but I did clean some of them out.

Google calls them nicknames and they appear to have a limit of 30 per account. Now I could create a chain of multiple accounts in Google Apps, each with 30 nicknames that forward on to a single account, but that just seems ugly. I am also considering paying for a Premium account (for greater piece of mine and the ability to turn of ads), but that is US$50 per account per year. Maintaining nicknames in Google is not worth US$250+ per year.

So I needed a way to keep the MX records pointing at my hosting where the 150 forwarders all point to a single mailbox. I then need that single mailbox to forward the messages on to Google. I had seen this done using the test-google-a.com temporary user address.

Two problems with that. First Google says that that address will dissappear from the Google Apps domain after a period of time, and second that had already happened to my domain.

After talking this over with Rob, he suggested that I setup a subdomain on my hosting and add it as a domain alias within Google Apps. The MX records for this subdomain point at google, and the single mailbox on my domain forwards to an address on the subdomain. Since Google knows about the domain it will accept mail for it and the messages end up in my account. It is working fine so far.

I have not attempted to migrate my existing messages yet, I will work out how to structure the labels and what filters I want to setup.

Now filters, that will be the topic of another post.

Despite what I have said in the past, my mind is gradually changing in regard to moving my email up into Gmail instead of hosting it myself on my home linux box.

What has changed?

Since my previous comments I have played around a bit more with Gmail (in no small part due to work) and I am starting to think that the conversation and label model might work better for me.

Currently I file both messages I receive and messages I send (for some topics) in the same folder. The benefit is that I can then see all messages in the thread in one place. While a downside is that there is no longer a copy in my sent folder. Conversations give me the benefit, without that downside.

Another advantage is that messages can be labelled and still be in my inbox. So I can choose to still have some mailing lists appear in my inbox, but once I have read it I will archive instead of delete. The message will then only appear under the label (and all mail.)

Currently I need to either rememer to check the folder for that list, or once I have read the message I need to move it to the correct folder. I have found both approaches to be problematic.

But Gmail is not without it’s disadvantages.

The primary copy of my mail will now be in the cloud (again shown to have it’s problems) so I need to make sure that I have an automated backup (offlineimap run from cron should suit) and I will need an Internet connection to access my mail (also needed everywhere except home)

The bulk of my investigations have been into how well the labels and conversations work with IMAP. I have found that if a single message of a conversation has a label, the web interface makes it appear as if the conversation has the label. But over IMAP only the message with the label is in the folder. So to get the most advantage of labels I would need to use the web interface.

My next concern is how I manage forwards/nicknames/aliases as I currently have a single mailbox, but over 200 forwards. Almost everytime something needs an email address I will create a new forward. This makes it very clear who leaked/sold my email address to a spammer. If I were to use Google Apps, it appears that there is a limit of 30 nicknames per mailbox. Not early enough. Although it won’t be as convenient, I could continue to manage the forwards my domain host.

I think that I am at the stage where I will give it a try by redirecting my mail and I won’t move all of my existing mail over until I am satisfied. This should also give me an idea of what mail functionality I actually need on my iPhone.

It is now the end of a five day (one more than most others as we get Easter Tuesday) long weekend. Apart from relaxing and watching a lot of tv, I did manage to do some cleaning up.

In the two and a half years since I shifted my email to my Ubuntu box I have been meaning to post about it (one of the unfinished posts). Over the weekend I came across some notes that I had made when setting up everthing under Ubuntu (from packages with minimal configuration), one page of which is a diagram.

This seemed like a good starting point so here it is:

After all this time the setup has not changed, apart from the addition of another client in the form of my iPhone.

So if someone were to send me an email message:

• It is delivered to a mailbox on my hosting provider (possibly going through a forward)
• Fetchmail runs as a daemon and polls the mailbox every five minutes and delivers to the local postfix
• Postfix determines the message is for a local mailbox and hands it to procmail
• Procmail writes it into a Maildir mailbox, possibly running it through SpamAssassin first

My other source of messages is from scripts (such as rss2email) that run on the same box, these are also delivered to postfix and then to procmail.

If I want to read the messages all of the access is through the Dovecot IMAP server, only configured for SSL, which works nicely for Thunderbird, my iPhone and SquirrelMail (which I rarely use).

When sending a message, Thunderbird on my desktop is configured to send it through the previously mentioned Postfix. If it is addressed to an external address the message is handed off to my ISP’s SMTP server for delivery (I would prefer to deliver it directly, but their network configuration prevents that). A copy is placed in my sent folder over IMAP.

The fact that none of this has changed in over two years shows how stable this has been for me, although there have actually been some internal changes:

• Since OSDC last December I have been filtering all of the messages from rss2email into a separate folder. This enables me to process news less often than other messages.
• I no longer have a copy of all the SPAM messages I have received in the last four years. This was 2.6GB of messages that I never did anything with except a couple of times to check what addresses I was receiving the SPAM through. Now once I have run sa-learn on the messages that I have manually filtered and reviewed how I was getting the SPAM, I will simply delete them.

Now, after finally posting about it, I have another post to write: about whether or not to ditch this setup and outsource it to the cloud. Actually that is two posts as I have another (long) post in mind about how much to trust the cloud.

Just before I went to OSDC I moved the contents of my Inbox to a new folder so during the conference I only had to worry about anything new that had come in. My first attempt at applying the concept I have seen referred to as ‘process to empty’.

This worked well and I ended up using it as a place to store anything that came in during the conference that I wanted to deal with when I returned home. Which I did.

However, once I had dealt with all of the recent items I accidentally deleted the folder. This meant that a couple of emails that had been hanging around in my Inbox for a long time were gone. And I still needed them.

Six months ago I had burnt a backup of my photos and documents that I was storing off-site (aka at work) so today it was a simple matter of grabbing the appropriate disc, extracting the archive of my home directory, and picking out my Inbox from that. Then when I got home it was a matter of dumping the files in the appropriate directory (under Maildir) and looking at the messages in Thunderbird.

Now, the data that I lost wasn’t particularly important, but I do need it in order to follow some things up so I was thankful.

Something I need to improve is the interval. The off-site backup is 6 months old. While my other backup is a nightly rsync that gives up to 24 hours. I have been meaning to use an external hard drive which could give a number of week intervals.

A month ago I installed a Thunderbird extension to give me a simple feature that has made the processing of my Inbox easier.

Ever since I started using SpamAssassin on my incoming email I have had two SPAM folders one that procmail automatically moves messages identified as SPAM into, and another that I manually move SPAM messages that were not identified as such.

Every few months I then run the contents of the manual folder through sa-learn to train the bayesian classifier.

So why is this significant?

Ever since I started to use rss2email for reading RSS feeds I have been running through the messages in my Inbox at least once a day in order to process it to (nearly) empty in a (sort of) GTD manner. As the majority of the messages are from RSS feeds the my most common action is delete, via the toolbar button (I click on the article link to bring it up in a new tab in Firefox and then return to my Inbox).

My next most common action is to move the message to the Spam/Manual folder. Even though it isn’t very complicated, every so often I drag the message to the wrong folder in the list down the side of the window. This was an annoyance and I wished that there could be a button to move the message.

A month ago I found Buttons!, an extension for Thunderbird that allows you to add a number of additional buttons to the toolbar. One of these is Archive! which moves the current message to a predefined folder.

So I now have two buttons; the standard Delete and right next to it a button that moves the message to Spam/Manual. This covers almost all of the actions I perform to process my Inbox.

In hindsight this doesn’t seem like much. But an optimisation of a daily process can be worth a lot more than you think.

Three months ago I talked about the first step I was making to move from a default permit strategy to a default deny strategy for dealing with spam. Instead of having messages to any address at my domain ending up in my inbox I setup forwarders so only messages to a hundred or so addresses (based on non-spam emails that I had kept) ended up in my inbox with the remainder ended up in my (previously unused) gmail account.

So what have I found over the past three months?

• between fifty to five hundred messages were delivered to my gmail account every day
• of these at most ten per day were not identified as spam
• I identified two addresses that I then added to my list of valid addresses
• only eight of the addresses in my list have received spam
• of these there is one that receives the majority of the spam, almost two thousand in the last two weeks

This evening I changed the catch-all so the default action for my domain is now to reject messages and removed the forwarder rule for the spammed address so messages to that address will also be rejected. My next action is to go through my list of addresses and change the ones that I don’t think I need anymore to forward to my gmail account. A few months after that I will remove them if they are actually no longer used.

Back when I first setup this domain I configured a catch-all email address so that I could simply make up a unique email address on the fly when one was required for website registrations and similar uses. This has become unworkable as a few months ago someone started to use my domain in the from addresses which meant that all the bounce messages (not picked up by SpamAssassin) came to me, in the hundreds.

At one point (before I did some analysis and found out that most of the messages used different addresses) I considered changig the mail server config so these addresses were dropped. This accounts to a default permit policy which is the first of six dumb security ideas.

So what have I done? First I extracted all of the addresses that were used to send me the emails that I have retained, about 70 all up but a number of those are questionable, and created forwarding rules so that mesages send to these addresses will be forwarded on. Eventually I will remove the catch-all which will mean all of the unwanted emails will be dropped but in the interim I changed the catch-all to be my gmail account (that I never really used) so I can be sure I didn’t miss any addresses.

This does mean that I will need to create a new forwarding rule whenever I make up a new address but in the greater scheme I don’t do that very often and that small inconvenice will be overweighed by the huge reduction in the amount of spam I need to deal with.