Blog entries tagged with "rss feeds"

For a long time I didn’t like most YouTube videos, often I would be directed (from a blog, twitter, etc post) to a YouTube video. Sometimes they would be good, but what I remember most was being annoyed by a video that took minutes to show or say something that could be explained in a single paragraph or image.

But the quality improved and I started to notice that there were a few creators whose videos I enjoyed. When I ended up at one of theirs I would find myself also watching other videos that they had made.

My first attempt at getting notifications about new videos was to subscribe to the channels to get the email notifications. This didn’t work so well as I was either not notified about a new video, or would be notified days after it was released. It also didn’t help that there were some creators where I was only interested in some of their videos, not all of them.

I remembered that there was some form of RSS feed and the first information I found was about an RSS feed of the videos in the channels that you are subscribed to. But this had since been discontinued and also would have given more videos than I wanted.

I then looked at what RSS feeds were still available and after some digging I found that you could target a user, a channel or a specific playlist:

• https://www.youtube.com/feeds/videos.xml?user=
• https://www.youtube.com/feeds/videos.xml?channel_id=
• https://www.youtube.com/feeds/videos.xml?playlist_id=

This is exactly what I was looking for as I could add these to my rss2email config with new gmail filters. I do also subscribe to the user/channel via YouTube as a small way of indicating my support.

So what are the videos I have found myself watching? These ones:

• SmarterEveryDay
• minutephysics
• Veritasium
• This Old House
• CinemaSins
• Practical Engineering
• colinfurze
• frank howarth
• Vintage Space
• Tom Scott
• The Slow Mo Guys
• Periodic Videos
• The 8-Bit Guy
• standupmaths
• Mighty Car Mods
• CaptainDisillusion
• Dirt Every Day
• Roadkill
• Tested

Like the sites I follow via RSS feeds I expect this list to grow over time. And looking back at that post I should post an updated list of feeds and also grab anything I want from Delicious as I haven’t used it in a long time and I don’t know what will happen to it…

After seeing that external mail was being delivered correctly to my Google Apps domain, I changed my rss2email config to deliver to Google Apps instead of directly to the local mailbox.

When filtering with procmail you can filter on anything you want, but in most cases a few regexes will get you what you need. In contrast the filtering options in Gmail are extremely limited. There are a couple of headers you can match against, and then just a simple string match.

I was able to tweak rss2email to add the URL of the RSS feed as the List-Id header. I can then setup a filter for each RSS feed. Again, like forwarding addresses, I monitor a lot of RSS feeds. Over 120 which means I will also need over 120 filters just for RSS feeds.

What would work is for the RSS feeds to be categorised before I send out the emails, I then only need a filter per category. An additional benefit is that when another RSS feed is added, a new filter is not required.

As I am considering further modifying rss2email (or replacing it completely), what else could I do?

Something that I don’t like about Gmail (and certain other mail clients) is that the message lists display the time the message was received by Google, not the time it was sent or the time in the message headers. This means that the RSS messages are clumped together because the script only runs once every few hours.

This cannot be changed as long as the messages are delivered via SMTP. But, thanks to a small project at work, I know that if I were to write the messages in directly via IMAP, the dates will be what I want.

If I were writing the messages in via IMAP, filters will not be run, but writing the messages directly to the appropriate label means that the filters are not even needed. I would have no idea how to modify rss2email to use IMAP, so I would be writing my own solution from scratch.

This method of direct injection via IMAP is also how my identi.ca/Twitter/Facebook updates should be delivered. In this case I indend to write something that uses the appropriate API, not the RSS feed as is the case for identi.ca and Twitter. I already use the API for Facebook, but only to produce an RSS feed that is then picked up by rss2email. It is a bit convoluted, bit it has worked.

It has been two and a half years since I started using rss2email and it still works for me.

However, over the last few months there are items from some feeds that I usually delete without reading beyond the title. Due to the large number of feeds (over 100), I had been putting off any effort to sort through the list and remove feeds that I no longer want to read.

This afternoon I decided it was time, so I reviewed each feed to decide whether or not I wanted to continue reading it. I eliminated quite a few feeds (down to under 70) and now have a better picture of the topics that interest me:

• team and people management
• presenting and communication
• time management and organisation
• software development techniques
• web design and development
• perl news
• people from the local tech community
• local tech events
• tech news
• photography
• security (technical and people)
• censorship and rights
• trivia and other cool stuff
• other people that I know

I was initially going to list all of the feeds here, but that was going to be too much work (ability to export OPML from rss2email would have been handy), so as I sorted through them I bookmarked them (with some exceptions) on Delicious with the tag feed-subscription. Go have a look if you want.

For over two years I have used rss2email to deliver new items from quite a large number of RSS feeds (which is a post of its own) into my Inbox. This works extreemly well, except for one problem: pizzaburger.

I didn’t know what that is, but that this is the title that around half of the items from the FAIL Blog come through with. The other half have a title similar to fail-owned-desert-foundation-fail, but that isn’t the real title either. Two recent posts with these strange titles are actually titled Pen Trick Fail and Foundation Fail.

At first I thought there might be something screwed up with the RSS feed. However a quick look showed that the title tags are as they should be. After looking a bit deeper I saw where the strange titles were coming from, the media:title tag inside a media:content tag.

After checking that I was using the latest version of rss2email my thought was that rss2email is using media:title instead of title. It looked like it, but I decided to find out. One problem, rss2email is in python, which (despite meaning to for the last few years) I have yet to take a decent look at.

Hey, debugging someone elses code isn’t that bad a place to start…

Half an hour later I have determined that the problem was not in rss2email. It appeared to be in feedparser, a module for parsing all types of feeds, so after checking that I had the latest version I wrote a simple script to confirm.

Instead of digging into the feedparser code I first turned to Google to find a blog post about the same problem which in turn led to a bug report.

I now thought the problem was on its way to being solved, until I saw that the bug had been resolved just over a year ago. So why didn’t the latest version have the fix?

I then spotted the nightly build page that references version 4.2. That’s greater than 4.1 which is the latest stable release! From there it was a simple matter to download the latest nightly build and use my script to verify that 4.2 contained the fix I needed.

Even though I had resolved a problem that had annoyed me for months I wasn’t quite happy. Why had the feedparser maintainer sat on 4.2 for so long?

And how did I find python?

Once I started to get my head around the different syntax there wasn’t anything fundamentally different to other languages. As expected I had the most difficulty with understanding how someone else had structured their code.

Two weeks ago I declared that I had found the perfect (for me at least) RSS aggregator. I was wrong.

After a few days of using reddit I found that I was reading the same headlines over and over as there was no indication of which ones I had seen before. My first thought was to go back and have another look at the aggregators that provide this functionality but then I thought about my inbox.

I currently run an IMAP server on my home linux server to access my mail which means that I can look at it from my desktop, my laptop, my work pc or even via a web interface and as soon as a message is read it is marked read everywhere. Since I check my mail on a regular basis anyway I realised that an RSS to email solution might be the answer and a quick google brought up rss2email. Fifteen minutes later I had it setup to check my list of feeds every four hours and email me any changes.

Every new item now appears as a separate message in my inbox with the summary from the feed (often the entire article which is the way I prefer it) as the message body. Now I can see which ones I hadn’t seen before (they are unread) and I can simply not delete the message if I want to follow up on it in some way later.

I have had this running for a few days now and as it is going well I have removed reddit from the folder of sites I regularly check. (I have a bookmarks folder in Firefox on my home desktop that I can simply middle-click on to load each site into a separate tab)

For quite some time my periodic investigations of RSS aggregators has not borne fruit. This changed tonight when I discovered that reddit (which I had first heard about a few weeks ago but hadn’t looked into) has a feeds aspect which is so far (I have added about a dozen feeds) looks like it is what I want.

So what do I want out of an RSS aggregator? I’m still not 100% sure about this but a very short list is:

• web based so I can access it from any computer. ie my laptop, my home desktop or even my work desktop.
• always on so it is regularly checking for updates. I have been monitoring some RSS feeds for our wiki at work through Thunderbird and I don’t like having to wait for it to check for updates.

Periodically I have checked out a few aggregators, both public services and projects that I install on my home server, but they never felt right. For some reason I just like the reddit interface…

Hi, my name is Stephen and I have a problem with time management.

In all seriousness this is true both with the time I spend at work as well as the time I spend at home. While it is going to be frustrating for the next six weeks as work as I will be 50% acting manager of the development team it is much more frustrating in relation to my personal projects.

This weekend just past is a perfect example as all I really did was go see a movie with a friend and drop my wheel off at the bike shop. That accounts for around three hours out of twenty-eight (fourty eight minus ten hours a day for sleep) leaving twenty-five. Including the three or so hours that I spent on saturday playing with Transformers and another two hours sorting out computers from my computer collection takes it down to twenty. What did I do in all that time? I know I didn’t work on any of the projects that I’ve been thinking about such as finding a nice RSS aggregator that suits my needs.

It is interesting that the site 43 Folders and the book that that site was inspired by keep popping up. Maybe they are onto something that works…

(It is ironic that I am sitting here at 1:30AM typing this when I should be asleep as I need to get up in six hours in order to go to work.)

Since I only just got around to installing Thunderbird I only just realised that it, like Firefox, has an RSS Reader feature. It looks more like a full aggregator as you can say that all the entries from multiple feeds will a grouped together in a single folder and it also shows the page that the item refers to as the message body. Unfortunately it is still tied to a copy of Thunderbird installed on a single computer which doesn’t suit my requirements.

Although there should be something about sharing/ syncing Firefox or Thurderbird profiles…

After being reminded today, technically yesterday, about the Live Bookmarks feature in Firefox I decided that it was about time that I switched Shaun, my windows box at home, over to the Firefox and Thunderbird combination instead of the Mozilla Suite.

In all its a pretty cool feature but I’m not sure if I’ll make much use of it, especially since my browsing is divided between my work box, laptop and home box. What I have actually been using is a prototype feed aggregator that I wrote a long while ago for work that runs from my work sandbox. It is just a single page that displays the contents of a collection of feeds that I have gathered over time. I might get something like AmphetaDesk set up on Gromit, my core linux box, to access both at home and remotely.

The offshoot of this that has occupied my time for the past hour and a half is adding in the link tag to the template of this site so that the people who do use Firefox get an icon saying there is an RSS feed available. Who knows I may even get someone to read it one of these days :) I then fixed up the navigation in the blog archive section by adding the ability to move back and forward between posts…