The Joy of Splicing

The other day, Fred Wilson made some interesting points about services in the age of Web 2.0, when customers of a service like TypePad might have their personal output being published on many services (TypePad, Flickr, del.icio.us, etc).

As a personal example of that, even within the scope of our own tools & services, Mena has 5 or 6 feeds for her various weblogs, photos, moblogs, etc. I can read all of those feeds in one place by subscribing to all of the feeds, of course, but sometimes I like to see a combined view of just her activity in the past couple of days: all of the posts aggregated from her various feeds, combined into reverse chronological order.

Of course, services like FeedBurner have offered feed splicing for quite some time (giving users the ability to splice together a couple of feeds containing their content around the web). But I always enjoy building my own tools, just for fun.

So I recently updated my XML::Feed module on CPAN to support feed format conversion and splicing, and built a simple feed splicer that takes multiple feeds, splices them together, and produces an Atom feed that contains all of her activity. It's not specific to Mena, of course--it could be used for any combination of feeds.

I'll probably be writing more about this in the future, but for fun, here's a one-liner feed splicer (requires XML::Feed 0.07 or higher):

    $ perl -MXML::Feed -e '$b=XML::Feed->new("Atom"); map
    $b->splice(XML::Feed->parse(URI->new($_))->convert('Atom')),
    @ARGV; print$b->as_xml' [list of feed URIs]

Or, if you don't like reading condensed Perl code:

• Source code for simple-splice.pl

Comments

[Timothy Appnel]

Very interesting and great work as usual.

I notice that your one-liner is generating the spliced feeds in Atom. Does that feed splicer guarantee valid Atom output even if one or more of the sources are RSS?

[Benjamin Trott]

Well, "guarantee" is a strong word. :)

But yes, it will auto-convert all of the RSS feeds to Atom--that's what the convert('Atom') code does. It's basically a no-op for feeds that are already Atom, and it'll convert RSS feeds to Atom.

[Timothy Appnel]

So in other words, garbage in garbage out. If you try to splice in an odd, funky or unneighborly RSS feed that is missing data Atom requires, the splice will output an invalid feed.

[Benjamin Trott]

In the current implementation, yes--neither XML::Atom nor XML::Feed do validation on created feeds.

XML::Feed does, however, try to do some intelligent detection of where your full post content is in RSS feeds, by looking at a combination of description, content:encoded, and xhtml:body. No guarantees there, either, of course, but then again, that confusion is part of the reason why Atom exists. :)

[Anil Dash]

I can't believe you said "funky", Tim. :)

[Timothy Appnel]

Sorry Anil I forgot funky syndication is a taboo subject around your parts.

Thanks for the explanation Ben.

[Josh Friedland]

Is there a way that a typepad user could implement this (keeping in mind that said user doesn't even know what perl is)?

If it's way too complicated, I'll just recommend that you offer feed splicing as a new feature on typepad.