Conditional GET with LWP & Perl

I was arguing recently that implementing a conditional GET with LWP is trivial and there was no reason why someone wouldn’t support it. I assumed there must be a dozen examples of how to do this. Afterall O’reilly has “open sourced” their original LWP book, there is an LWP cookbook, and reams of POD.

No Such Luck

Well a quick search didn’t turn up anything. A more concerted one might have but it was easier to write this example then keep searching. If you’re looking for more general info on Conditional GETs try Charles Miller’s HTTP Conditional Get for RSS Hackers. If you’re looking for an implementation in PHP, you might look in rss_fetch of my RSS parser/aggregator Magpie. ### Conditional GET

The basic idea is, when you request a file you remember the ETag and Last-Modified HTTP headers, passing them along with your next request as If-None-Match and If-Last-Modified. If the file has changed then you’ll get the content as normal, if the file hasn’t changed you’ll get a ‘304 Not Modified’ header. This is something of a toy example, but I try to be as correct as possible with it. Noteable in its absence is doing anything with the file you’ve fetched. (for example parsing and storing an RSS feed) Also I use a simple file to store ETag and Last-Modified, you might want to use a different backend.
See the Code

Example Code

<pre class="code">

use LWP::UserAgent;
use HTTP::Request;

my $url = "http://localhost/rss/laughingmeme.rdf";
my $cache_file = 'cache';
my %headers;

if ( -e $cache_file ) {
    open (CACHE, "< $cache_file") or die "Couldn't open: $!";
    %headers = (
        If_None_Match => <CACHE>,
        If_Last_Modified => <CACHE>
    );
    close CACHE;
}

my $ua = new LWP::UserAgent();
$ua->agent("Conditionally Enabled v0.1");

my $req = HTTP::Request->new( GET => $url );
$req->header(%headers);

my $res = $ua->request($req);
if ($res->is_success) {
    print "new!\n";
    # save ETag & Last-Modified
    open (CACHE, "> $cache_file") or die "Couldn't open: $!";
    print CACHE $res->header('ETag'), "\n";
    print CACHE $res->header('Last-Modified'), "\n";
    close CACHE;
}
elsif ( $res->code() eq '304' ) {
    print "not modified, go to cache\n";
    # do logic for RSS not modified
}
else {
    print "fooey! somthing went wrong\n";
}