Laughing Meme » gzip

Magpie, PHP 4.3.2 Memory Error, an Update

Kellan — Wed, 25 Jun 2003 20:49:00 +0000

Steve tracked down the root of the problem that is causing MagpieRSS to run out of memory after upgrading to 4.3.2. It lies in my (apparently dodgy) patch to Snoopy to add gzip support.

Temporary Solution

To turn off gzip encoding (and thereby avoid this bug) add:

define('MAGPIEUSEGZIP', false);

to the top of your script.

Explanation

After much hair pulling trying to get gzip encoding working in PHP, I found this magic recipe in comments attached to gzencode:

$results = substr($results, 10); $results = gzinflate($results);

Seems you had to strip off the gzip header to make it work. This was confirmed when I found Simon’s httpClient.

Apparently this is changed in 4.3.2, and stripping the first 10 characters leads eventually (through a yet untraced path) to an out of memory error.

Next Steps

Does anyone have any info about this? Any suggestions on a good way to support 4.3.2 and pre-4.3.2 PHPs? Gzip encoding really isn’t an optional feature in a modern aggregator, so turning it off is a short term solution.

Update 6/25 2:16

Okay, so I’ve got PHP 4.2.3 locally now (via Marc Liyanage’s excellent package), and I’m seeing similar behaviour. I don’t actually get an out of memory error, but I can see memory usage spike (with iPulse) and I get a warning about a gzinflate buffer error. Unforunately removing “$results = substr($results, 10);” does not fix the problem for me. So currently the only solution is to turn gzip encoding off

Curiouser and Curiouser

So removing the substr() call changed the error message form “gzinflate(): buffer error”, to “gzinflate(): data error”. However only for certain websites! Files served with gzip encoding from some sites (like mine) work, but from others, it doesn’t. Ack.

MagpieRSS 0.5.1

Kellan — Fri, 13 Jun 2003 06:21:00 +0000

MagpieRSS 0.5.1 is out with a fix for the gzip silent failure bug that showed up when attempting to fetch an RSS file with gzip encoding when PHP had been built without zlib support. Also include random other small clean up.

Btw, that also means Magpie now runs on OS X’s stock Apache/PHP.

Magpie GZIP Bug

Kellan — Fri, 30 May 2003 20:27:00 +0000

MagpieRSS 0.5 can fail without errors when fetching a RSS feed using GZIP encoding if your PHP doesn’t support gzinflate() (i.e. it hasn’t been built with --with-zlib)

As of 0.5 sends an “Accept-encoding: gzip” header by default when fetching RSS files. You turn this feature off by adding define(MAGPIEUSEGZIP, 0) to the top of your script.

A patched version of the locally maintained extlib/Snoopy.class.inc that checks function_exists(gzinflate) is available in CVS. This fix seems to work, I haven’t had a chance to test it widely. It is a busy week, but I’ll try to have a final fix, and new version out in the next couple of days.

(sidenote: It seems silly to fork Snoopy, and give it a new name over one minor change like adding gzip support however development on Snoopy has stopped and its confusing this maintaing a local version because I don’t want to imply there is a bug with the real Snoopy, just my version. What do you think?)

MagpieRSS 0.5: Now with GZIP support

Kellan — Tue, 13 May 2003 03:21:00 +0000

MagpieRSS 0.5 just hit Sourceforge. This is not the promised 0.5 with its mythical new parser, this is a much more modest release that adds support for HTTP gzip encoding to get Magpie off Bill and Ted‘s bad aggregator list.(flattering to make the short list really)

Also if you’re interested, a patch to Snoopy that gives it gzip encoding support.

Gzip Encoding

Kellan — Tue, 29 Apr 2003 05:24:00 +0000

Magpie has been “outed” for not supporting GZIP encoding. I’m thinking of switching from Snoopy to Simon’s HttpClient in order to get gzip, and a few other features. Has anyone ever played with it? Snoopy has been solid, and error free, and while development stopped on it a while ago, this is sometimes a good thing. HttpClient on the other hand looks to be undergoing very active development, and I like some of the design decisions I’m seeing.

One problem, is it just me, or is there no way to send arbitrary headers?(like If-None-Match, to pick a purely hypothetical example) That would seem very strange, but quick glance threw the code seems to confirm this.