Conditional GET with LWP & Perl

March 1st, 2003

I was arguing recently that implementing a conditional GET with LWP is trivial and there was no reason why someone wouldn’t support it. I assumed there must be a dozen examples of how to do this. Afterall O’reilly has “open sourced” their original LWP book, there is an LWP cookbook, and reams of POD.

No Such Luck

Well a quick search didn’t turn up anything. A more concerted one might have but it was easier to write this example then keep searching. If you’re looking for more general info on Conditional GETs try Charles Miller’s HTTP Conditional Get for RSS Hackers. If you’re looking for an implementation in PHP, you might look in rss_fetch of my RSS parser/aggregator Magpie.

Conditional GET

The basic idea is, when you request a file you remember the ETag and Last-Modified HTTP headers, passing them along with your next request as If-None-Match and If-Last-Modified. If the file has changed then you’ll get the content as normal, if the file hasn’t changed you’ll get a ‘304 Not Modified’ header.

This is something of a toy example, but I try to be as correct as possible with it. Noteable in its absence is doing anything with the file you’ve fetched. (for example parsing and storing an RSS feed) Also I use a simple file to store ETag and Last-Modified, you might want to use a different backend.
See the Code

Example Code

use LWP::UserAgent;
use HTTP::Request;

my $url = "http://localhost/rss/laughingmeme.rdf";
my $cache_file = 'cache';
my %headers;

if ( -e $cache_file ) {
    open (CACHE, "< $cache_file") or die "Couldn't open: $!";
    %headers = (
        If_None_Match => <CACHE>,
        If_Last_Modified => <CACHE>
    close CACHE;

my $ua = new LWP::UserAgent();
$ua->agent("Conditionally Enabled v0.1");

my $req = HTTP::Request->new( GET => $url );

my $res = $ua->request($req);
if ($res->is_success) {
    print "new!\n";
    # save ETag & Last-Modified
    open (CACHE, "> $cache_file") or die "Couldn't open: $!";
    print CACHE $res->header('ETag'), "\n";
    print CACHE $res->header('Last-Modified'), "\n";
    close CACHE;
elsif ( $res->code() eq '304' ) {
    print "not modified, go to cache\n";
    # do logic for RSS not modified
else {
    print "fooey! somthing went wrong\n";

Tagged: Uncategorized , , , , ,

5 responses to “Conditional GET with LWP & Perl”

  1. Dave says:

    This conditional get sample was perlfect for what i need to do


  2. Teenburg says:

    Thnx for link in open book “Web Client Programming with Perl”

  3. Jason says:

    The code has a small bug.

    Line 12, the hash key should be IfModifiedSince not IfLastModified.

    Thanks for the example!

  4. Al says:

    Awesome. You made my day.

  5. Is this different than LWP::Simple::mirror? Or has that been added since you originally wrote this?