Blog posts tagged "fof"

Same Web, Different Front Ends

February 23rd, 2005

Ted says RSS aggregators are the killer app, killer in the sense that they are

the app that does so much that it consumes all available CPU, memory, network, and disk.

He goes on to say

It’s a reflection of the way that my relationship to the web has changed. I hardly use a standalone browser anymore — mostly for searching or printing. I don’t have time to go

Funny, I’m following this trend to opposite end of the curve. I’ve been noticing lately that I spend less and less time using anything other then my standalone browser, Firefox. Once I’ve got a decent coding environment built in, pretty much the only external apps will become ssh and chat. Functionally Firefox is very close to becoming my operating system. (though in practice I still prefer running it on platforms other then Windows)

While the news has been full of the problems with network based and centralized services (see Preshrunk or Schneier), their nativeness suggests that perhaps some of the problems Ted is seeing are an impedance mismatch.

And a quick, semi-related tip, if you’re setting up several people on FoF, consider setting up a shared cache directory to get some economies of scale.

Tagged: Uncategorized , , , ,

Ads in RSS

December 7th, 2004

Kottke did some footwork and got some great response from major aggregator developers regarding ads in RSS. I thought Brent’s comments that it isn’t about ad blocking as much as importance and attention managment are particularily right on the money. But in particular I thought I’d call out this comment:

The only ads in feeds I have recieved so far have been google adsense ads – and since I use Feed On Feeds and a userContent.css file that blocks those ads, I don’t see them anyway.

Tagged: Uncategorized , ,

RSS/RDF catch up

August 12th, 2004

Life has a been a little crazy (diagnosis: monomaniacal) of late. If I haven’t gotten back to you since around June 25th it isn’t personal, if I have, you’ve been lucky.

Among the many items piling up in my inbox, have been RSS/Magpie/RDF links. Here is a dump.

rss2jabber is a wicked cool script for sending out RSS updates over Jabber. Haven’t played with it extensively, but the possibles uses boggle the mind.

RSS-WML is a RSS feed viewer (aggregator) for WAP devices. Requires PHP, and MagpieRSS.

Jo wins the “making RDF accessible” award of late with Class::RDF, a CDBI based triplestore, and RDF::Simple. I think there must be something about England that encourages Perl coding, be it climate, or culture.

The “much anticipated” XML::RSS 1.05 should hit CPAN in the next few hours. There is also a chance it will be going to better caretaker soon. I just don’t have what it takes to drag this beast into the modern era. (but I’ll happily chatter on about what is wrong with it, if anyone cares)

In the Magpie coming soon (I promise, really) department, Steve’s character encoding patches, a patch from Glen Davies adding support for fetching RSS via authenticated proxies , improved documentation courtesy of firetrap, and an update of the recycle code to work with the latest and greatest FoF from Ernie Oporto.

Tagged: Uncategorized , , , , , ,

Recycle’d RSS Items

June 15th, 2004

A couple of weeks ago at CATS I promised the Riseup folks that there was an easy way to aggregate and selectively republish RSS items, aka “reBlogging”. Well a cursory scan of the web didn’t turn up anything suitable, so I sat down and wrote a quick patch to FoF (I think it was during the Certificate Anti-Authority discussion).

Made a couple of tweaks to it last night to make it less of a diff from Feed on Feed 0.1.3, and now I’m releasing it onto an unsuspecting public, as a patch file, plus a couple of auxiliary files. Works just like FoF, except there is now a checkbox labeled ‘Recycle’. Any items you choose to recycle will be available in your re-published RSS feed (at<recycle-install>/rss.php).

There is a lot of room for improvement including better RSS feeds (they are pretty simple right now, and more suited for a headlines box, then full re-syndication), and support for conditional GETs.

You can see it the FoF interface and the RSS feed on my test server (though I just blew away the database so you’ll have to add some items)

My favorite feature is I’m using “submit in place” Javascript, meaning the page doesn’t have to refresh when you recycle an item. See the behindthescenes.js file, lifted largely from orkut.

Tagged: Uncategorized , , ,

Magpie, PHP 4.3.2 Memory Error, an Update

June 25th, 2003

Steve tracked down the root of the problem that is causing MagpieRSS to run out of memory after upgrading to 4.3.2. It lies in my (apparently dodgy) patch to Snoopy to add gzip support.

Temporary Solution

To turn off gzip encoding (and thereby avoid this bug) add:

define('MAGPIEUSEGZIP', false);

to the top of your script.


After much hair pulling trying to get gzip encoding working in PHP, I found this magic recipe in comments attached to gzencode:

$results = substr($results, 10); $results = gzinflate($results);

Seems you had to strip off the gzip header to make it work. This was confirmed when I found Simon’s httpClient.

Apparently this is changed in 4.3.2, and stripping the first 10 characters leads eventually (through a yet untraced path) to an out of memory error.

Next Steps

Does anyone have any info about this? Any suggestions on a good way to support 4.3.2 and pre-4.3.2 PHPs? Gzip encoding really isn’t an optional feature in a modern aggregator, so turning it off is a short term solution.

Update 6/25 2:16

Okay, so I’ve got PHP 4.2.3 locally now (via Marc Liyanage’s excellent package), and I’m seeing similar behaviour. I don’t actually get an out of memory error, but I can see memory usage spike (with iPulse) and I get a warning about a gzinflate buffer error. Unforunately removing “$results = substr($results, 10);” does not fix the problem for me. So currently the only solution is to turn gzip encoding off :(

Curiouser and Curiouser

So removing the substr() call changed the error message form “gzinflate(): buffer error”, to “gzinflate(): data error”. However only for certain websites! Files served with gzip encoding from some sites (like mine) work, but from others, it doesn’t. Ack.

Tagged: Uncategorized , , ,

The State of RSS, Magpie, and FoF

May 28th, 2003

I downloaded Feed On Feeds 0.1 (changes), and am very happy with it. Steve is right, the new features I asked for do clutter up his interface a bit, but they work great! I imported an OPML file of the 85 feeds I most often read and it gave me the chance to kick not only FoF‘s tires, but push Magpie a little more then I usually do, and check on the general state of RSS in the wild.

And the news is good!

FeedOnFeeds: The Good News

FoF smoothly imported my OPML subscription list of my 85 most read sites with only 5 failures, one of which was clearly marked as an HTTP timeout.(a feed, whoops!) I was thrilled. I’ve been living in Straw’s parse-at-all-costs bunker, while I assumed the horrors of invalid, unparseable RSS ravaged the outside world. Imagine my surprise to peak my head up expecting Armageddon and to be greeted with flowers, sunshine, and a 4% failure rate. It gets better.

RSS: The Good News

FeedOnFeeds generates a link to Sam and Mark’s RSS Validator(which, with its high visibility and plain language feedback has been hugely beneficial in cleaning up RSS), and I was able to quickly check why the feeds were failing. The validator threw “Sorry an error occurred” when parsing 1 of the 4, marked the 2 down for style points but passed it, and said 2 of my rejected 4 were perfectly fine and why was I wasting its time. A little puzzled, I went back to FoF and added each of these feeds one at a time, and now they all succeeded. I think it must have been timeout problems, as Magpie sets very low timeouts by default.

Of the 85 feeds I read, all of them were parseable with an XML parser, none of them were invalid. I know we’ve got to comprise our ideals of perfection, deal with the fact that content producers can’t be trusted to produce valid XML, I accept that, and I’m willing to accept a 0% failure rate if you are :)

Magpie: The Pretty Good News

As much as I’m loath to confess it (and I’ll probably have a pack of agile developers howling for my blood) I’ve never really tested Magpie extensively. It seemed to work, in theory it should work, and I assumed that users would let me know if it didn’t.(bless ESR’s little heart for such a well made rationalization) So while I wasn’t surprised to see that Magpie dealt handily with a diverse array of RSS files, I was very happy, and quietly relieved.. With one exception. Kevin Burton’s Peerfear feed.

I knew it was out there, lurking, waiting to pounce. I first heard dark rumors of this feed last Fall; whispers in the street, seen it in the haunted, hunted looks in the faces of the parsers and aggregators around me, and finally one day, XML::RSS turned up on my door step, confused, and clearly in pain. You see it isn’t just an 80k behemoth of a feed, Kevin, quite legally, uses the XHTML namespace as the default namespace for his feed, instead of the RSS namespace we all expect. Having patched XML::RSS I knew someday I would be faced with the problem in Magpie, but up until this point I had cleverly managed to avoid ever facing the issue. (Damn you Steve for seducing me into this with your nifty web aggregator.) I’ve known for a long time Magpie’s innocent approach to namespace could never last, but I think I’ll let it enjoy its youth for a little while longer.


  • You could argue I stacked the deck a bit on the RSS failure rate, as yesterday I helped fix a broken that would have caused me problems today. But I like to think of that as a happy accident.
  • Does your spell checker ask “FoF: Did you mean FOAF?”, mine does (admittedly not be default)

Tagged: Uncategorized , , ,