There is something fundamentally brash, brazen even, to releasing a class named DateTime. As a calendar geek I imagine upon seeing “new DateTime()” I feel something akin to what an old thespian feels when they see a company putting on a production of the Scottish play — it’s a decidedly mixed emotion. But I’m going to bump my way through learning how to use this new DateTime lib, bringing all my preconceptions about how it should work. The odds of this being interesting to you is probably nil unless you’re in one or two very small cliques, feel free to read on, or browse away.

I’m primarily working in PHP4 right now, so my first step was to grab a copy of MAMP 1.5b getting me a nice PHP5.2 sandbox to play with.

The new objects are documented here, apparently there are functional equivalents for each of the object methods, and they use the PECL timezomedb.

Hey! timezonedb! First fence cleared! A timezone database compiled into a native format based on Olson is the one true solution, and I can update it independently, the most recent release being based on 2007b. Sweet.

Constructor takes an initialization string that it passes to strtotime(), and an optional DateTimeZone obj. Defaults to “now”

$date = new DateTime();
echo $date . "\n";
> Object of class DateTime could not be converted to string 

Oops, no __toString() method defined. You’ll need to use the format() instance method. If you end up using the DateTime objects, you’ll be seeing a lot of format(), more on that in a bit.

format() uses the date() formatting strings (not the strftime format strings). Also takes a number of useful constants, most usefully your pal and mine RFC3339 (aka W3CDTF aka Dublin Core/Atom date format).

echo $date->format(DATE_RFC3339) . "\n";
> 2007-02-22T15:23:47-05:00

Note: thats a constant, if you pass in the string ‘DATE_RFC3339′, and you’ll get odd looking results.

Here we can see the default constructor sets both the time and a timezone — correctly, for the moment, identifying my timezone as America/New_York. That’s somewhat contentious behaviour, some people will tell you that dates with unspecified timezones should either be in UTC or be “floating”, divorced from any timezone. Why? At least in part because across platforms and boxes timezone guessing is going to be non-deterministic — the script that worked when you ran it locally on your Mac laptop in New York, might fail on your ISP’s servers. You get a hint of this reading over the timezone guessing rules on date_default_timezone_get. There is also the fact that I’m currently moving at about 400mph and will be in a different timezone real soon now. However you can set the default to something reasonable in a script, or in the php.ini. (consider this my recommendation)

date_default_timezone_set('UTC');
$date = new DateTime();
echo $date->format(DATE_RFC3339);
> 2007-02-22T20:44:49+00:00

Yay, that worked. Okay, but lets display that datetime in the local timezone. (after all the point of this entire exercise will be the ability to work painlessly in multiple timezones).

$date->setTimezone('America/New_York');
> DateTime::setTimezone() expects parameter 1 to be DateTimeZone

Siiiigh. Not smart enough to cast strings into TimeZone objects (holds true for the constructor as well, so no new DateTime('now', 'UTC')). Now its time to learn how to use DateTimeZone.

Working with DateTimeZone, All Hail Olson

I mentioned briefly earlier that PHP is now shipping with an extension timezonedb, which is a compiled version of the Olson database. The Olson database is a massive, largely volunteer effort to catalog the various timezones both in use, and those that have been in the past. Time is a political issue, particularly day light savings, and as such the rules governing it are arbitrary, whimsical, and subject to frequent change. (p.s. gotten a panicked memo yet about new daylight savings compliance for March 11th? No? Where did you say you worked?)

Note: Olson also uses a longer form of the zone names then we usually see in the U.S., this is to combat ambiguity. See Appendix H for a list of timezone names, including some handy shortcuts.

$tz = new DateTimeZone('America/New_York');
$date->setTimezone($tz);
echo $date->format(DATE_RFC3339) . "\n";
> 2007-02-22T16:02:55-05:00

This is starting to get long winded, but, hey, PHP5 supports object dereferencing on returns. Maybe this will work.

echo $date->setTimezone($tz)->format(DATE_RFC3339) . "\n";
>  Call to a member function format() on a non-object

Nope. Oh well.

Date vs Datetime?

Say I’ve got a nice platonic date, say November 11th. There is no time element associated with this, so timezones are kind of irrelevant. I mean Nov. 11th starts at different times through out the world, but Nov. 11th is universal. (as long as you’re using the same version of Gregorian as most of the rest of us) Ideally this date would float above timezone issues, but that isn’t how PHP does it, 2007-11-11 is treated internally as midnight on the 11th, which is certainly simpler, but disappointing. You can prove this like so:

$date = new DateTime('2007-11-11');
$date->setTimeZone($tz);
echo $date->format(DATE_RFC3339) . "\n";
> 2007-11-10T19:00:00-05:00

The other useful DateTimeZone method is getOffest()

echo $tz->getOffset($date); 
> -18000

Daylight Saving, March 11th, and Why Programmers Are a Grouchy Lot

Note: getOffset, which returns a timezone’s offset in seconds from UTC, takes a DateTime obj because offsets can be date sensitive due to daylight savings. Really without daylight saving this stuff would all be pretty straightforward. Let’s test to make sure the offsets are correct at the boundary.

echo $tz_nyc->getOffset(new DateTime('2007-03-11 1:00')) . "\n";
echo $tz_nyc->getOffset(new DateTime('2007-03-11 2:00')) . "\n";
> -18000
> -14400

(-18000/(60*60) == -5 hours) 
(-14400/(60*60) == -4 hours) 

Yay! They got the memo about U.S. Energy Policy Act of 2005.

The Basics: Accessors and Mutators

So what are some other basic desires?

Get epoch seconds! Except for their kind of limited range epoch seconds are great, and have helped a generation of programmers put off worrying about timezones as long as possible. They’re also the backbone of PHP’s traditional date/time methods.

Alas, there isn’t an accessor method for getting epoch seconds, you’ll have to use format().

In fact DateTime doesn’t expose any of the accessors you’d expect, so you’ll be using format a lot if you want to access pieces of your date. (for you know, display purposes, or manipulation, or building queries, or pretty much doing anything you’d want to do with a date)

examples of the format() as all purpose accessor pattern:

epoch:  $date->format('U'); // 1173596400
year:   $date->format('Y'); // 2007
month:  $date->format('n'); // 3
day:    $date->format('j'); // 11
dow:    $date->format('l'); // Sunday

… etc …

So now you have accessors for the full range of date() formatting strings. You just have to jump through a hope.

Pretty much the only accessor is getTimezone()

echo $date->getTimeZone();   // hope springs eternal!
> Object of class DateTimeZone could not be converted to string
echo $date->getTimeZone()->getName() . "\n";
> America/New_York

Mutators

Speaking of accessors, DateTime is a little sparse on mutators as well: setTime(), setDate(), and the mysteriously named setISODate().

$date->setDate('2007', '1', '1')->format(DATE_RFC3339);  // who am I kidding?
> Call to a member function format() on a non-object 
$date-> setDate('2007', '1', '1');
echo $date->format(DATE_RFC3339) . "\n";
> 2007-01-01T02:00:00-05:00

Now what if I want to set just the day?

Maybe

$date-> setDate(null, null, '11');
echo $date->format(DATE_RFC3339) . "\n";
> -001-12-11T02:00:00-05:00

Nope.

Instead you’ll need to pull out the year and month (using our format() accessors) and pass those back in just to set the day.

$date-> setDate('2007', '1', '1');   // jan 1.  
$date->setDate($date->format('Y'), $date->format('n'), 11);
echo $date->format(DATE_RFC3339) . "\n"; 

Clunky.

setTime() works the same, but for time.

e.g. Setting just the minutes, 33 minutes past, but keep hours, and seconds constant:

$date->setTime($date->format('G'), 33, $date->format('s'));
echo $date->format(DATE_RFC3339) . "\n"; 
> 2007-01-11T02:33:00-05:00

So what is an ISODate? I’m unclear, and so is PHP’s documentation. The docs show the call signature taking a $year, $week, and optional $day, while the description talks about $year, $month, $day. Looking at the code looks like $week is the proper call, $month is cut and paste error from setDate(). So I guess this is a method for setting day by the “week of the year” a concept more popular in Europe then in the US. Not sure what ISO has to do with it. So what is our current week of the year?

echo $date->format('N') . "\n";  // 'N' is new in 5.1.0
> 4 

Jan 11th was in the 4th week of the 2007? Go figure.

$date->setISODate(2007, 4, 5);  // fifth day of the 4th week?
echo $date->format(DATE_RFC3339) . "\n";
> 2007-01-26T02:33:00-05:00

Um. You know what? You’re on your own with setISODate, sorry.

Date Math: Adding and Subtracting Deltas aka $date->modify($str)

PHP5 for better or worse has very limited operator overloading, so no $dt1 + $dt2 * $dt3 / $dt4. Instead the primary method for doing math is modify()

Thankfully PHP’s strtotime() method is a gem, and one of the things it handles is relative dates. strtotime() + relative dates is the secret to doing math with PHP5’s DateTime.

Lets get a basic date to start with:

$date = new DateTime('today');
echo $date->format(DATE_RFC3339) . "\n";
> 2007-02-22T00:00:00+00:00

Note: modify() is destructive. It changes the original datetime object (as the name suggests). You’ll need to jump through some hopes to keep a copy of your original date. More later.

Add/subtract N days:

foreach (range(1,10) as $n) {
   $date->modify("+1 days");
   echo $date->format("Y-m-d") . "\n";
}

> 2007-02-23
> 2007-02-24
> 2007-02-25
> 2007-02-26
> 2007-02-27
> 2007-02-28
> 2007-03-01
> 2007-03-02
> 2007-03-03
> 2007-03-04

$date->modify("-10 days");
echo $date->format("Y-m-d") . "\n";

> 2007-02-22

$date->modify("-1 month");
echo $date->format("Y-m-d") . "\n";
> 2007-01-22
// or alternately
$date->modify("1 month ago");
echo $date->format("Y-m-d") . "\n";
> 2006-12-22

Cloning DateTime objects to work around modify

Of course you usually want to keep the original when doing date math, so modify()’s lack of idempotentce is annoying. Lets say I’m building a SQL query to select events happening in the next 7 days.

In an ideal world the code would like this:

$start = new DateTime('today');
$end = $start + 7;
echo "select between " . $start->format('Y-m-d') . " and " . $end->format('Y-m-d') . "\n";

The above of course is just a pipe dream. But wouldn’t it be nice?

I’d settle for:

$end = $start->calc("+7 days");

Or even:

$end = $start->clone->modify('+7 days');

None of the above examples remotely work.

Instead use:

$start = new DateTime('today');
$end = clone $start;
$end->modify('7 days 3 minutes 42 seconds ago');

Now format our SQL query for our example:

echo "select between " . $start->format(DATE_RFC3339) . " and " . $end->format(DATE_RFC3339) . "\n";
> select between 2007-02-22T00:00:00+00:00 and 2007-02-14T23:56:18+00:00

Awkward, but it gets the job done.

At least the relative date format is super flexible and expressive. As far as I know the closest thing to documentation is from the GNU tar manual on date input formats. (just like CVS) Btw. if you ever want nightmares, take a look at the scan method in PHP’s parse_date.c and be thankful that isn’t your job to maintain

Date Math: Comparison and Differences

Beyond adding deltas (“+7 days”), the other common date math is comparing two datetimes, to find out which is more recent, and getting the difference between them. DateTime supports no methods for comparing two datetimes. The simplest solution for doing comparison is to compare epoch seconds.

Note: This method only works for dates that can be represented by epoch seconds. PHP uses a signed int for epoch seconds, so the range is limited by the size of the max int on your platform. Generally you get approximately 138 years, 1901 to 2038. There are other schemes besides epoch seconds for mapping dates to an easily comparable number; MJDs, and Tai time being two. See also Rheingold & Dershowitz 1997

$d1 = new DateTime("today");
$d2 = new DateTime("tomorrow");
if ($d1->format('U') < $d2->format('U')) {
   echo "true\n";
} 
> true

If you’re going to be comparing a large number of dates you might consider a memoization technique like the Schwartzian transform.

We can get the difference in seconds using the same hack of casting to epochs.

echo $d2->format('U') - $d1->format('U') . "\n";
> 86400

Ideally we’d then divide the difference seconds to get the difference in hours, days, weeks, or months. However the following naive solution won’t work.

$diff / (60*60*24);  // calculate difference in days, **BADLY**

Why not? Because days don’t always have 24 hours. Sometimes they have 23 hours, sometimes they have 25. Daylight saving strikes again. (If you want to be even more pedantic, minutes are also not 60 seconds long, sometimes they’re 61 seconds long if we have a leap second)

Basically you need to break yourself of thinking of datetime units as being fungible. You can’t simply calculate minutes from seconds, or days from hours. Just like you can’t divide days by 30 to get an accurate number of months. There are solutions, but they’re a bit beyond this blog post.

new DateTime from Epoch Seconds

So, non-fungible, remember that.

But sometimes you’ve cast DateTimes down to epochs to do math. And then you’ll want to cast back to a DateTime.

Alas DateTime doesn’t have a constructor that takes an epoch, and passing a epoch to the default constructor will throw an exception, rather you want:

$from_epoch = new DateTime(date('c', '-568080000'));
echo $from_epoch->format('Y-m-d') . "\n";
> 1952-01-01   // expected result

Conclusions

DateTime/DateTimeZone get timezones right, and for solving that hard problem they deserve all possibles accolades.

The rest of the API however is kind of simplistic, awkward to work with, and verbose.

Single most useful change: have DateTime methods actually return the object making it possible to use a slightly more abbreviated calls.

I had thought about writing up a few more recipes, like nth dow of the month, and such. But we were coming in for descent, and it was time to be done. Might happen in the future.

Also if anyone has any power to enhance the DateTime object, I hope some of the above can act as a road map for a more expressive and powerful core library. Or ping me, happy to chat.

For a while there was largely nothing, then there was Allconsuming which was wonderful, but slowly died, and went dark before being re-incarnated in the mold of a 43x tool. And I have this memory of there being a nifty little $14/mo tool, back in the days when I didn’t pay for websites, but I wasn’t able to find it.

Last Fall, I started sketching down notes towards building my own, and in the intervening year its become an interestingly crowded space. (who knew so many other people felt the pull) Even in the 6 weeks since I first started jotting down sites for this blog post, the space has evolved with LibraryThing coming out solidly on top as the most active: most actively developed, most actively used, and most actively engaged developer.

That said, in a cursory search (mostly of my del.icio.us links) I turned up 5 other very similar services

• Allconsuming: the Next Generation
• Bibliophil
• Books We Like – “activist e-commerce and collective intelligence”
• Listal – where “all” is defined as DVDs, Books, Music, and Games
• Reader2 – repurposing of the MyProgs codebase

Also the Bookshelf example app from 24L, and the intersting related services What Should I Read Next?, and Library Elf

None of them are quite there yet, and I want more, more, more!

LibraryThing

• So like I said, LT is rocking out in terms of development and growth. When I first found it, it lacked import, ratings, and feeds. Wow.

• Additionally the search is amazingly comprehensive, fast, and accurate. Search is where LibraryThing originally shined, and it blows every other service out of the water. LT was the only service to successfully come up with “Someone Comes to Town, Someone Leaves Town” when I searched for “someone comes”. Which is good, because there is no way I’m typing in full titles. (just re-tested, and Allconsuming which wasn’t finding anything last time I checked, also came up with the correct results)

• Surprisingly zippy.

• One-click export. Sweet.

LibraryThings: The Downside

• No per-tag feeds, apparently on the todo list, but I need per feed tags.

• No feed of reviews, also on the todo list, but I’m simply not willing to create content without an RSS feed. (which is why I’ve stopped writing $0.02 reviews on Netflix)

• Feeds lack anything beyond basic data. No structured meta-data (authors, isbn/asin, cover art, etc), so very limited usefulness.

• One dimensional tags. (no tag combos)

• Ugly, and awkward. Totally in the eye of the beholder, but I find the interface consistently confusing and awkward to use. I only have a handful of books in my catalog, but already its feeling unmanageable. Not a designer, so I can’t do much more then complain.

• One comment/review per book, meaning you can’t use it to blog your ongoing experience with a book. One of my use cases for a book service is that I can use it to power a book blog.

• Uses frames!?!?? In this day and age? Meaning it can’t be the that “definitive URI for books” that I keep looking for.

Listal

• More then just books if you’re in to that thing. I’m not really, and find it kind of frustrating that I sometimes end up searching DVDs.

• Slick, clean, attractive.

• Allows browsing by author! (key missing feature for LibraryThing)

• Per tag feeds, multiple reviews/comments per book.

Listal: Cons

• Feels sluggish, aggravated by a click heavy interface.

• Limited import.

• Not run by a book nut, and in general the site is voiceless failing to expose either the developer, or the user community. Feels stagnant.

• Search failed the “Someone Comes” test.

Bookshelf

I thought I was going to like Bookshelf. I love the concept of the Ning “Pivot”, if not the implementation. Unfortunately Bookshelf is very slow, and buggy. I dived into the code (I can do that with Ning, whoohoo!!!), and unfortunately rather then the domain specific langauge for building social webapps I expected to find, I just found a messy of PHP code.

What I Still Want/Need/Dream About

I wrote down a list of things I wanted a while back, and LT is converging on it very quickly, and for that I’m inclined to overlook the interface.

Syndication

I need more feeds. Per tag feeds allow for intelligent interaction with the site. Beyond topical tags, functional tags (e.g. *to:read) allow me to syndicate the information out in useful ways

And richer feeds please. E.g. if you can’t syndicate the cover art give me the necessary metadata so I can pull it from AWS.

More data and Webservices

And note, I’m happy with my primary webservice being a feed, but I need richer way to interact with the data before I can make it work for me.

I’d like to be able to hook this up to my local library (auto-request books flagged *to:read), Upcoming, and Amazon.

Plus I should be able to power my involvement in What to Read Next/Books We Like/innovative new service from within LT (or at least with my LT data)

More Pivots

I’d like author, and ratings smooshed down into the tag namespace. So I can browse by author, and by rating. Additionally I’d like to be able to browse tag combos, author plus tag combos, and author plus rating combos.

Insta-Community

I love the idea of integrating a forum into the site. And really integrating it.

1. Allow people to start new threads from a book page
2. Tag those threads with the book
3. Display only properly tagged threads on the book page
4. Aggregate all threads in a more standard forum view.

(We’ve discussed doing something similar for Social Source Commons, though haven’t yet)

Book Scanning

Ala Monster’s Delicious

What would be involved I wonder if writing a simple app/Firefox plugin to scan barcodes using something like an iSight or a CueCat, and have it post the data to a URL?

Blue Sky: Peerflix for Books?

Like Bookcrossing for books you want to get back someday.

update: Also Stuffopolis, and Douban

Anil posits that Flickr’s users, creators of value and “interestingness” are getting short changed, or at least in the future our understanding of Flickr’s value proposition will lead us to conclude their users are being short changed. It’s part of an ongoing struggle to define our norms around participation, community, hosted tools, and ownership. (On a side note, syndication can mix into this explosively, as with this thread last Summer on Meetup and EVDB)

Actually Anil’s point was more interesting and more subtle, and worth reading, but as the signal bounced around the echo chamber, it degraded into “Hey, I make Flickr interesting, pay me!”.

I mean as software tends towards commodification (as t approaches 0), clearly Flickr derives its value from its participants, yes?

No. Quite the opposite.

I could replicate Flickr’s software (call it Flickah, a Boston Flickr derivative), give it away free, and still people would pay to be part of Flickr. And in fact if I ever managed to grow the community to a fraction of Flickr’s size I’d be in trouble. Flickr isn’t a photo hosting site, it’s a salon, and unsurprisingly value accumulates most quickly to the salon owner. Value arises from the centralization.

Community Service Models?

So assuming software, what alternatives models exist for a community to host a service they find useful? How do communities gain and support the values of centralization without handing over control? A Flickr, an Upcoming, or an Audioscrobbler provide value in direct proportion to the size of the community, while the centralization of a Google Maps (or a Geocoder) makes an expensive resource affordable. It’s a question I’ve been wrestling with for a while (community+service). And a question I asked at techdinner recently to surprising results.

I expected to hear about grid computing, alternate economic models, p2p, etc. Instead it was suggested that maintaining such a resource, or at least some subset of such community resources is the role of the Academy in the 21st century. (less surprising given the presence of Berkman-ites in the crowd)

Perhaps not a Google Maps, or Flickr but maybe Harvard should be hosting the definitive URI for books? I was intrigued. (not to mention a little appalled given my stint doing tech for Higher Ed.)

Last thought, in the multitude responses to Anil, it was pointed out that interestingness can be gamed, as can most deployed reputation systems. Yet eBay works? How? By making buy in into the system cost real cash, something Flickr print is poised to do. As a print service not terribly exciting, but what a great way to quantify interestingness.

One line that jumped out at me at me was

I do appear to shell out at least $50 per month in internet services beyond my bandwidth bill.

That got me thinking. A few years ago this would have been an unprecedentedly large amount. The idea that we were all going to get rich selling online services was so firmly rejected that it became a commonly accepted truism that “people won’t pay for things online”, and yet, quietly, almost under the radar this seems to be changing.

Looking at my personal expenses online they can be broken down into: paid content, online tools, online services, personal hosting, and net facilitated donations.

Paid content

I maintain a Safari account (which after several years of comp I started paying for last year), I’m a Zmag sustainer, and in the past I’ve subscribed to several premium info sources. 90-95% of my daily information consumption is network mediated — blogs, online newspapers, email newsletters, and radio streams. The bulk of the rest of it comes from magazines. (which I either subscribe to, or pick up on the newsstand depending on whether I want to financially support the publisher).

Monthly cost: ~$15

Online tools

I use a large number of online tools in my daily life, from the ubiquitous Google, to the essential Bloglines. I currently experimenting with using Gmail, having tried nearly all of the webmail products at one point or another over the years. Currently I’m a light weight user of Use Tasks, for online managed tasks, and used to be a regular user of the Anyday.com hosted pim service (I also was an Anyday developer)

For a while I was using All Consuming to facilitate my book reading habit, and in its heyday I was a heavy user of BackFlip, a dotcom era del.icio.us. (I feel like I’m forgetting a handful of key tools here, I’ll have to back fill them later)

Beyond my text editor, most of my work (and most of my day) happens within the confines of a Firefox window. Currently the only tool I’m paying for is Use Tasks.

Monthly cost: ~$4

Online services

Hard to split online services from online tools really, but I guess I’m thinking of net facilitated services. Netflix is a good example, as is the iTunes Music Store. Pobox mail forwarding is a slightly murkier one. Automated clipping services like PubSub are largely indistinguishable from tools. You could argue that webmail, or a provider like Fastmail should actually be in this category.

A chunk of the my monthly online spending goes to this category, mostly Netflix, with handful of change going to various more obscure services.

Monthly cost: ~$27

Personal Hosting

The cost of maintaining an online presence. My primary web and email hosting are covered as a side benefit of some of the tech activism I do, but I do pay for a hosted dev box (a VLS really), and have been contemplating setting up a new solution for email.

Monthly cost: $10

Donations

I make both regular and irregular donations to a number of online services. Some of the donations are towards tech activist projects like Riseup, or Activista. Some are for funding drives for web commons projects (e.g. Wikipedia, or Mirror Project), or blogger bailouts (e.g. the Daring Fireball pledge drive)

Of all the discussed categories, this is my largest expense.

Monthly cost: ~$35

Running the Numbers

Some obvious places to be spending more

It is odd to want to spend more, and frankly, I don’t. In particular the problem in spending more on services means you need to either figure out how to have services swell, and shrink with changing income, or you get stuck having raised the minimum you can live on, which is a dangerous relationship to get into with capitalism.

Still, I really should be spending another $5-$10/monthly on personal hosting in order to get a more functional email setup, as a person who lives largely online spending so little to maintain the presence seems….off.

I really should be paying more for online tools. Another $5 monthly for Bloglines at least in a no brainer. Some amount of money to a meaningful and useful social software/collaboration tool is burning a hole in my pocket. And a good hosted pim/tasks/reminders service still needs to get (re-)built.

Noticeably absent from the list are any self supporting community or discussion spaces. I’m not sure I believe in virtual communities, which doesn’t seem to stop me from being involved in a ridiculously large number of them. Most exist in a nebulous hybrid mailing list/forum/irc space, none of which ever seems to get paid for, and none of which ever seem to improve and become more useful. I know several people, like my brother, who participate in incredibly specialized and erudite online communities. A way of striking a balance between getting the right people into the community, and supporting growth needs to found.

Lastly independent media is still doing a lousy job of developing models to deliver information both locally and globally using the internet, and doing an even worse job of figuring out how to use the net to engage meaningful participation and support. Right now most of my net mediated media consumption is filtered corporate media, the filters are important, interesting, and useful, to a self sustaining alternative it ain’t.

I’ve totally failed to answer Les’ question, but I thought it might be worth re-examining our received wisdom about people’s online spending habits, or at least mine.

Doing some rough calculations based on current income levels, amount of value I’m deriving or wish to be deriving, and looking at the above list, I’d say there is about $150-$250/monthly that I should be spending in one of these 5 categories that I’m not for lack of a meaningful place to spend it.

That said, as we move towards more and more hosted/online services and tools, its going to become increasingly important to develop new models of engagement, and transparency. What do the ideals of free software mean in the context of a hosted service? How do I fork if I don’t like where things are going? What is the role of collective ownership in these projects, or is it assumed I’m simply a consumer? Thats a whole other essay, and one I’m too tired to start right now.

And TeleDyn has an excellent essay Living with Webservices, which seems to be closely related in ways my brain is totally refusing to articulate right now. Good night.

This week as I learned both the name of this pattern, and the SQL to implement it efficiently in MySQL I thought I’d share some notes on what we’ve come up with.

Class Table Inheritance

We’re using CTI for a number of reasons, some excellent, and some because that is how our brains work. One key benefit is we get the ability to operate on heterogenous collections of object. (i.e. display a sorted list of contact objects irrespective of whether they are individuals, organizations, etc.) The fact that something as fundamental as polymorphism can be difficult to accomplish using other O/R techniques hilights the general difficulty and impedance mismatch of object to relational mapping.

There are, however a few challenges involved.

The Model

Challenge 1:

Solution 1:

SELECT 
  c.id as id, c.contactType,
  i.firstName, i.lastName,
  o.name as orgName
FROM
  Contact as c
LEFT JOIN Individual as i ON i.id = c.id 
LEFT JOIN Organization as o ON o.id = c.id

(And it’s pretty easy to coax the necessary SQL out of our modified version of DB_DataObject. I’ll demonstrate that in a future entry)

Challenge 2:

One solution would be select all known records, and do sorting and slicing at the application level. (This is so wrong it makes my head hurt) Another solution would be store a field like “sortName” on Contact, which you could calculate and save on inserts and updates. This is on the right track.

Solution 2:

SELECT 
  c.id as id, c.contactType,
  i.firstName, i.lastName,
  o.name as orgName,
CASE 
 WHEN c.contactType = 'Individual' THEN CONCAT(i.lastName, i.firstName)
 WHEN c.contactType = 'Organization' THEN o.name
END as sortName
FROM
  Contact as c
LEFT JOIN Individual as i ON i.id = c.id 
LEFT JOIN Organization as o ON o.id = c.id
ORDER by sortName

This simply adds a switch statement to your field list conditionally setting the value of sortName.

There is nothing quite so satisfying as finding the right tool for the right job. I’m currently totally in love with the CASE statement, and think everyone should know about it. (hence this bit of evangelism)

Tip: don’t forget that ‘,’ after orgName, I keep forgetting it and wondering why my SQL isn’t working.

So concludes today’s edition of “Enterprise Development with MySQL” (“Kellan Learns SQL” didn’t sound as impressive). In the near future I hope to get a chance to write up some of the PHP libs we’ve been building/modifying to support object rich web development techniques, completing the picture. (OLamp anyone?)

Mixing Metaphors

The Case of the Missing Timezone

A classic mistake when one sets out to write calendar software, particularly web calendar software, is to ignore timezones. “Now” is treated as the current time on the web server, new events are added to the database without reference to the timezone of their creator. And amazingly, by and large this will work, for a while. Often your first set of users will be in your timezone, or thereabouts, perhaps you aren’t displaying hours, so it’s just a few hours early in the morning, and late at night when people notice your calendar is displaying the wrong date.

Problems start to crop up as your audience becomes more international, or you start trying to add more time sensitive services. So you try to retro-fit timezones, add them on. And now you’re living in a world of pain.

A Not So Hypothetical Example

Say you’re running a web calendar off a server in your closest in Massachusetts, in a timezone affectionately if rather unprecisely known as Eastern time. You have a user in Sydney who has been happily adding events, timezone free. Now if you wanted to display the correct date for a given event to your users in Sydney you need to calculate the offset between the two locations. Simple enough? Is MA observing day light savings? Is Sydney? How about when the event was added? Any one of those questions can be annoying to answer, getting all 3 right, and collated… well trust me, it gets ugly quick.

Yet Another Problem

What happens when you move that server out of your closest to a colo off the coast of England? All of a sudden you’ve switched from adding events in “Eastern” time to some British local time. How do you know how to calculate the offsets now? One solution I’ve seen (when I was working on a website for a certain very large PDA manufacturer who shall rename nameless) was to add a switch into the code:

 if (date < kDayWeMovedToColo) { ... } else { ... }

This problem can bite you even if you were clever enough to get your users setup with timezones from day one.

A Simple Solution

After all that hand wringing, and doom saying, you’ll be happy to know that the solution is simple. Store all your dates in UTC (sometimes called GMT, sometimes called Zulu time); a zero offset time that doesn’t observe day light savings. As added bonus a number of languages even support converting to GMT time, though many of them are broken. (PHP’s are, rumor has it so are C#’s, Java got it wrong for its first 3-4 versions)

Some Code!

As Perl hackers we’ve got an advantage, we’ve got the best date/time library in any language I’ve ever seen, DateTime.pm, which just happens to have the most complete TimeZone implementation you’ve ever dreamed of. (if you’re a calendar geek, and dream about this stuff)

And we’ve got Class::DBI, which as much as I’ve said some nasty things about it, has a few nice features in the db-to-object mapping sphere.

I use a simple adapter class to handle my conversions to and from GMT, but a more proficient CDBI wizard could probably replace my class with a couple of code refs.

In your Class::DBI definitions you would add some code like:

MyEvent->has_a(
   start_date => 'DateAdapater',
   inflate => sub { DateAdapter->inflate(shift) },
   deflate => 'deflate'
);

Where DateApapater looks like:

package DateAdapater;
use base qw(DateTime);
# this is a DateTime::Format::DBI
# which I haven't figured out how to combine with cdbi
#
use DateTime::Format::MySQL; 

sub inflate {
  my ($class, $value) = @_;
   my $dt = $class->parse($value);
   # starts in 'floating' tz
   $dt->set_time_zone('UTC');
   return bless $dt, $class;
}

sub deflate {
    my $dt = shift;
    # convert to UTC
    $dt->set_time_zone('UTC');
    return DateTime::Format::MySQL->format_datetime($dt);
}

sub parse {
    my ($class, $value) = @_;
    my $dt = eval { return DateTime::Format::MySQL->parse_datetime($value); };
    if ($dt) { return $dt; }
    else {
       return DateTime->new(year => 1970);
    }
}

A Few Things to Note

MySQL’s datetime fields don’t maintain a concept of timezone, so DT creates them with a floating timezone. Therefore casting them to UTC doesn’t change their settings for year, month, hour, etc. DateTime objects coming in will presumably be set in the user’s timezone, therefore casting them to UTC will change their year, month, hour, etc fields, but the object will still refer to the exact same instant in time.

Lastly, you know you’ve been hacking Unix dates too long when you’re idea of an error message is to return some day in 1970.

(Also it’s strikingly odd how much longer it takes to write a blog entry about some code, then to just write the code. Noticed it for both this, and the timezone selector. Need a code to blog adapter.)

Jon is right, the state of calendaring standards is a fiasco. There are a couple of reasons for that. I think the two key factors have been:

• the CALSCH working group, like many people, tried to solve the scheduling problem, with business use as the target use case, versus just building some basic calendar building blocks. Despite this focus iCalendar still works pretty well as a basic building block. However the 7 years since it was released have fundamentally changed the landscape (like the standardization on XML), changes CALSCH hasn’t been able to keep up with or respond to.
• Microsoft. Aren’t they always the problem? A huge amount of time, energy, effort and tears has been spent trying to build a calendar tools that work with Outlook, and Exchange. Its a nightmare, even when MS supports standards, they do it brokenly, and the Outlook/Exchange combo has been specifically engineered to look out alternatives. Anyone who navigates the treacherous waters of MAPI, and Outlook’s various modes feel entitled to be paid for their efforts, and rightly so.

Apple, a Hope Which Fell Short

Apple really did a great job of reviving interesting in the iCalendar spec, and calendaring, and calendaring standards in general. They combined a pretty interface with a modern implementation of the standard, and a good enough, worse is better network protocol that worked over HTTP, and integrated with the Web, and web services.

However, they pulled up short. Where they could have pushed forward and set a quick and dirty standard for calendar sharing over HTTP that bypassed the byzantine, perpetually under construction CAP, they failed to implement a real, two way, pub-sub protocol. Presumably to drive people to use their .Mac premium service, and iSync application. Disappointing really, a very understandable, uninspired, profit driven decision. I’m sure Microsoft would be proud.

Ray’s Questions

Ray asked a few questions in post, probably rhetorical, and designed to draw attention to the daunting absence of good work being done in this field. However I’ll take a stab at answersing them. (oh, looks like Libby answered them as well)

• Has a method to embed iCal into XML ever been approved or agreed upon? If so, let me refer to it as xCal.

  Yes, xCal existed in draft form. Calsch decided to let it die, according to Doug Royer, “because there is no reason to have 2 standards for the same thing.” I’m inclined to also point to an certain hostility towards things web, and XML that I’ve seen over the years on the Calsch mailing list. There was even a parser for it, for a while.

• Has a method to embed xCal events/etc ever been suggested as a viable item type for RSS?

  The mod_event module was proposed in May, 2001, and is a simple, straightforward syntax for marking up an RSS feed to indicate event information. Many peoples’ reaction to seeing mod_event is to start trying to lard in the iCal syntax. I think this misses the point of an RSS event feed, a need which, in conjunction with other RSS modules, the mod_event serves very well.

• Has anyone built websites that publish venues’ event calendars in such a format for subscription/aggregation?

  Protest.net does, and has for quite a while. I know a handful of other projects that are in prototype or various stages of completion.

• Has anyone built an Outlook or Notes adapter that publishes personal or team calendars to such a feed, OR
• Has anyone built an RSS aggregator that can aggregate multiple calendar RSS feeds into a desktop or web calendar UI?

  Not that I know of. But remind me to get back to the question of web calendar UIs.

  Magpie does include basic support for mod_event, and I’ve built a couple of scripts using XML::RSS and DateTime::Format::W3CDTF.

• Has anyone built an RSS aggregator that can aggregate multiple calendar RSS feeds into your Outlook or Notes personal calendar?

  See my earlier point about Microsoft being one of the two key factors behind the calendar fiasco.

There Are Other Hammers

That said, I don’t think RSS is a particularly natural fit as transport for all calendaring and scheduling information. As Jon said, “Stop punishing RSS for its success”. RSS, and an RSS event feed works wonderfully well for public event calendars, and as a notification feed. It does not, and will not, map well to representing someone’s personal calendar in all its, repeating, complex, conditional, glory.

A Brief Note on Web UIs for Calendars

Next Steps?

Event Auto-discovery

Zend has an article Writing Libraries in PHP (you’ll have to trust me on the title, as their CMS seems to be broken). It is a good article as far as it goes, but I think the title over sells the article. Which is a shame, because of all of PHP’s many faults and quirks, perhaps it’s most telling (and most crippling) is a cultural one, PHP programmers write applications, not libraries. I don’t consider myself a PHP guru, being more comfortable in Perl and Java, but I do consider myself a good software developer, and so I’ll try to capture a few of the lesson I learned when developing MagpieRSS, my PHP library for parsing RSS.

First, lets identify the problem.

PHP, its a cultural thang

1. Do one thing, do it well

   You aren’t building a CMS, you aren’t building an interface, you’re trying to support other people in those tasks.

2. Don’t echo or print content, return it!

   This is one of the key problems you see in much PHP code. If you echo out the results of some function directly to the web page, when I’m trying to use your code to write the output to a file, or run it in a testing environment I’m going to be frustrated. What if I’m trying to build an internationalized app, and you’re echo’ing out English? Don’t assume you know in what context your code will be run, return objects, or strings. Don’t print.

3. Return data, strings, or objects, not HTML.

   The corollary to the don’t print rule, and abused nearly as often, if not more so, is don’t return formatted HTML. (unless you are writing an HTML widget, in which case that is all you should be doing) If you return the results of your function as a formatted table, it might be pretty and easy to use, but I can’t pass those results to another function to sort them, or integrate it with my pure CSS layout. It is really common to see code like echo('<font color="red">$errormsg</font>');. Don’t do it.

4. Allow intelligent error handling

   One of the most common reasons a library will print content directly to the web page is on encountering an error. This makes makes it very difficult for the application using your library to figure out what happened and respond accordingly. Don’t assume your code is the most important part of someone’s application, maybe they don’t care if you failed? Or maybe they care a whole bunch, and want to totally change what they were doing?

5. Use an error() function

   I’m still struggling to come up with the best way to do error handling as a library in PHP. Once PHP5 arrives and we have real exceptions, much of this will be irrelevant. In the meantime.

   What I do with Magpie is provide an error() method for each part of the application.

   error() takes a message, and an optional error level, appends phperrormsg if trackerrors is on, prepends a string identifying the library that is throwing the error, sets up a package/class variable with the resulting error, and, if debugging is on, triggers an error message.

   Why is this good? Code using your library has easy way to check for error conditions (if ($lib->error) { ... do error handling .. } ), error messages are very complete, and consistently formatted, when someone is developing with your app they can easily find out what went wrong based on their php.ini settings, and lastly if someone does need to hack or override your chosen behaviour, they only have to do it once.

6. Allow simple configuration

   If you don’t provide a way for people to change the behaviour of your library, then you force them to hack on it. If someone has had to go into your code and hack on it, then they’ll be resistant to upgrading as new versions become available, and any changes they make that you want to roll back into the core will be more difficult to apply.

   Configuration can be parameters passed to a class’s constructor, set/get methods called later, or constants defined at a runtime. (more on this later)

7. Choose intelligent defaults

   This is true with any application, or library in any environment. It is particular true with PHP where a great number of your users aren’t programmers by profession or choice, but just trying to get something working to support their real work.

8. Break your library into multiple files

   One way to simplify your code, and encourage encapsulation and reuse is to split your library into logically sub pieces, and move these pieces into their own files.

   Something I didn’t do with Magpie, but wish I had was store all the files in a lib/ directory. Having all your files in a single directory makes it much easier for people to install your code. (When it came time to bundle an external library, a modified version of Snoopy, I had learned, and put it in extlib/ )

   This tip is really an excuse for the next tip.

9. Don’t assume everyone’s PHP looks like yours.

   PHP has a lot of configuration options, runs on dozens of different platforms, and is used in all sorts of different ways. Keep that in mind when writing a library.

10. If you have multiple files, allow your user to define a base dir. (aka don’t make assumptions #1)

    This is trick from Smarty that was pointed out to me.

    If you have a core library file (e.g. class.inc) that will be including support files don’t make assumptions about the PHP include path on the various machines where your library will be installed.

    For example assume there is a constant MAGPIEDIR defined then you can include the support libraries with:

    requireonce( MAGPIEDIR . 'rssparse.inc' );
    requireonce( MAGPIEDIR . 'rsscache.inc' );
    

    This allows code that uses your library to inform your code about the local environment, rather then forcing your client code into contortion to match your expectation of how a PHP install should work.

    MAGPIEDIR (or YOURLIBRARYDIR) might be set up with code like:

    if (!defined('DIRSEP')) {
        define('DIRSEP', DIRECTORY_SEPARATOR);
    }
    
    
    if (!defined('MAGPIEDIR')) {
        define('MAGPIEDIR', dirname(FILE) . DIR_SEP);
    }
    

    Which fill set the MAGPIE_DIR to the current directory (useful as ‘.’ isn’t always in the include path) unless you override it with a statement like (for example):

    define('MAGPIE_DIR', '../../magpiefiles');
    

        </p>
        <p>
            (More on using constants for configuration later.) 
        </p>
    </li>
    <li>
        If you're using a semi-obscure PHP extension test that it has been compiled in.  (aka don't make assumptions #2)
        <p>
            This <a href="http://laughingmeme.org/archives/000811.html">bit me hard</a> when developing MagpieRSS.  I add supported HTTP gzip encoding, and suddenly for a small number of users Magpie starting failing.  This was a surprisingly difficult bug to track down, I recomend avoiding it all together.  This is what PHP's <code>function_exists()</code> function is for.
        </p>
        <p>
        In code that uses gzinflate I might add a conditional like
        <pre class="code">
    

    if ( functionexists(‘gzinflate’) ) { …. } Or at the beginning of the Magpie RSS parser I check to make sure PHP has been built with XML support with

    if (!functionexists('xmlparsercreate')) {
     ... trigger error ...
    }
    

11. Don't pollute the global namespace

    All functions share a namespace in PHP. What that means is, if I have a parsefile() function in my library, and you have a parsefile() function in your library PHP has no way of telling them apart, and we've got a serious problem. Classes help with this.

    Another option is to prefix the functions in your library with a common string. Steve does this with Feed on Feeds, prefixing all his functions with fof, e.g. parsefile() becomes fofparsefile(). Cuts down on conflicts, and increases readability.

12. If you use database tables allow a table prefix. (aka don't pollute the other global namespace)

    Most libraries aren't going to work directly with a database, that is the province of applications usually, but if you are consider allowing the user to configure a table prefix, much like the function prefix from the previous tip. Many users are on low end hosting platforms with only a single MySQL database, this creates another global namespace. If my library has a user table, and your library has a user table, and we have different schemas (almost a guarantee) then we've got a problem.

13. Provide a well designed, object oriented interface to your library, that follows the above rules.

    This is outside the scope of these tips, but see the following corollary.

14. Provide a functional, PHP-like interface that builds on your OO interface.

    PHP is not a language for building airy, abstract object hierarchies. It is a quick and dirty language for throwing together webpages with a minimum of fuss. While its important to provide the more elegant interface for your advanced users, and to encourage proper design, the majority of your users are going to want something simpler.

    With Magpie I provide an object oriented RSS parsing class. I also provide a simple, rssfetch() one function front-end that is designed to be used directly from within a PHP page.

    My design consideration for rssfetch were fetching remote files is time consuming, and parsing an XML file can be resource intensive. In most languages/environments you would setup a cron script to run in the background handling these tasks, and spitting out HTML fragments. This is not very PHP-like, and often beyond the technical ability of many PHP users. (or beyond what is offered in their hosting environment). So rssfetch transparently uses PHP's serialize and unserialize to cache the results of the time/resource intensive calls, and serve subsequent calls quickly.

    This makes it very easy for the end users to do the right thing, while writing simple, idiomatic PHP. I think this is very important for writing useful PHP libraries, and is one of the challenges that comes with the territory.

15. Configuring the functional interface using defines.

    (aside: by functional we means as opposed to object-oriented, not as opposed to non-functional or dysfunctional)

    Choose intelligent defaults, and provide a simple default means our functional interface should "just work" for many people. But when it doesn't, it should be equally easy to configure without cluttering up the API.

    What I've done with Magpie, and its worked well, is extend the technique used in "setting a base dir".

    Conditionalize your library behaviour on a set of constants (e.g. MAGPIEUSEGZIP, and MAGPIECACHEON). Document these constants. People can now change the behaviour of your library with some simple statements at the top of their PHP, like:

    define('MAGPIECACHEON', false);  // turn off cacheing
    

    Then the first thing your function should do is call an init() function which sets up those intelligent default we talked about:

    function init () {
        if ( defined('MAGPIEINITALIZED') ) {
            return;
        }
    
    
    if ( !defined('MAGPIE_CACHE_ON') ) {
        define('MAGPIE_CACHE_ON', true);
    }
    ... other constants ....
    define('MAGPIE_INITALIZED', true);
    
    
    }
    

16. Provide examples

    Always a good idea, but particularly important when distributing PHP libraries. Don't assume people will read the documentation, or if they do your programmer-esque understanding of your tool will be meaningful to them. What will make sense is example.

17. Provide examples, carefully

    Be careful what your examples look like because whatever you do in your example is what 90% of your users will do in their scripts.

    Test your examples or your support queue will fill up with people who cut and pasted your code and it isn't working for them.

    Make your examples as attractive as you can while still being simple, otherwise you'll be forced to look at your ugly HTML all over the web.

    Show examples that show proper use of your library, including best practices like error handling. If your examples show how to use a feature it will be used, if they don't, them most people won't use them.

18. Document

    Document document document. Provide inline document. Provide a README. Provide a FAQ. Provide a website. Provide hints of where to go looking for more info. Hints like "this class is used by rssfetch() you can stop reading now if you just want to use the simple interface" can also be useful. (of course don't sound too smug about it, as chances are people are there trying to find one of your bugs)

    Running code is good documentation.

    If you're going to provide code snippets make them longer then seems necessary as people will often have different instincts about what should come before and after the line in question. This is again the joy and trouble with providing a PHP library.

    I personally like "cookbooks", a hybrid of a FAQ and running code samples.

19. Use it Yourself, Use Consistent Names, Plan to Expand

    And just to re-iterate a few of Moran's suggestions

    Until you really use your library you'll never know how well you've succeeded. When you write a library you shouldn't conceive of yourself as the only user, but certainly one of the users. Also developing good relationship with people who use your library will cause it to improve dramatically.

    If you name half your methods getRSSFile() and the other half getcache_dir() your users (and yourself in a few months) will be confused, and your code will look messy.

    Moran says, "Never return a single value when you can return an array. Never return an array when you can return an object."

    I think that is overstating the case, often you'll want to return a single value for simplicity sake, but take into account when you might not want to. Often returning an object will make the most sense, as long as your clearly document how to use the object.

PHP needs libraries

Resisting Temptation, Educating, and Refactoring

Thanks to Steve, Martin, Scott, and Evan for their feedback and suggestions.

SocioTechnical

Something Different

1. No mailing list
2. Incredible speed of convergence

The other factor which has been incredible (and a bit alarming) is how fast it pulled together. Arguably this is because the work had already been done. Everybody knew what the new format was supposed to look like, the project was just to clarify a few details, and rename it. Actually it turned out that there was a fair amount of debate on the wiki about how to do things, even among the small group who showed up, and yet still the convergence has happened blindingly quickly. (Some have suggested there is a story there still to be told, but I wouldn’t know.) Progress is good, concrete results are good, the maxim of open source development is “release early, release often”, so what is the problem? Well combine a fevered working pace, with a difficult to penetrate discussion space and the barrier to entry gets pretty high.

Haunting Echoes

Well Formed, and Whither?

Otherwise all that has been accomplished is changing some names, and perhaps changing who has the power, but an opportunity to create something more interesting and more open will have been missed, and the chance to tap the insight of the larger community will be lossed. In political organizing there is a word for what we have on our hands, we call it a Tyranny of Structurelessness.

Its an idea that occurred to me when I first started playing around with delivering events via RSS, and realized that RSS wasn’t an XML file for headlines, but a webservice pipeline I could shove all types of data through.
Sourceforge figured out the same thing, and Technorati has a nice little service (don’t know how “successful” in the business since its been) selling a personalized RSS feed that can watch Google queries, or keep track of who is linking to you.

Tim mentions a few of the same types of feeds I was thinking of, misses a couple, and mentions one I never would have thought of, traffic reports. (being a non-driver and all) Unfortunately/fortunately most of my entrepreneurial flare was burned out of me during the brief years I was running my own little dotcom, and so like a handful of other business models, it sits collecting dust.

Weather

(long rambling thoughts on RSS weather service [see: insomnia-induced]. includes running code!)

A Quick Spec

Current State of Things

Notes On My Approach

For national weather, given a city and state, scrape the NWS website. This gives you anywhere for 3-6 days of forecast. Formats vary based on locality (there is a certain amount of free form data entry going on at the sources) but thats okay, I don’t need to understand the data, just classify it, and re-display it.

For international weather, given a city and country, scrape Wunderground. (Wunderground’s international info is nice and predictable, while their national coverage tends to be chatty, and harder to interpret)

Interpretation

RSS Design Descisions

Code?

Weather::RSS::Source is an interface that determines if your asking for US weather, or weather from somewhere else and then hands off the heavy lifting (filling out forms, handling redirects, HTML scraping) to Weather::RSS::Source:WeatherGov and Weather::RSS::Source::Wunderground respectively, whom returned detailed current info, the extended forecast, and any weather alerts, or hazard warnings they happened to find. (this piece should probably be re-packaged and released to CPAN as I like it better then any of the existing Weather:: packages, maybe as Weather::Source? Weather::Forecast? Weather::RainGods?)

Weather::RSS on the other hand is merely responsible for taking the resultant weather source object, and spitting out an RSS feed.

Also include is a little cgi that steals the logic from Aaron’s Acme::Test::Weather and attempts to guess at a visitors location, and provide the proper RSS feed.

• Download Weather-RSS-0.01.tar.gz.
• Example output: New York, Paris, Santa Cruz, Montevideo.

Next Steps

Improvements could come from being more intelligent about when to produce new RSS items, and folding/condensing the forecasts as mentioned above. Also including doppler, doing a better job of tracking down extreme weather info, and maybe cute little weather icons like everyone seems to have.

Also the current version (if its still the current version) will probably brake sometime next Winter, as Winter weather reports just look different.

Lastly more and better datasources. If you know of a better source of info, let me know. Thanks.

Providing a Service

Right now its just some Perl modules, and few scripts, you’re welcome to download it. Like I said I kind of burned out my entrepreneurial side hasn’t really re-grown. But IF you really did want to pay me to provide a RSS weather service, something like $8-$12/year probably I might be interested. I wonder what the legality of re-selling other peoples data is? There are a number of desktop clients which do it, but they’re selling clients, not the data. (I guess I’m just selling a webservice client at that) In the process of jack hammering my thoughts out of Perl and into English this morning (Trader Joe’s Moka Java) I think I was hit for the first time what confuses and baffles people about open source. At commercial development prices this is a $1500-$2000 dollar investment before infrastructure, as open source its something to do until the rain stops. (hey, or you could hire me, and I’ll thrown in setting up a customized RSS feed of just about anything as a bonus)

In the meantime, expect at least a minimal project page, ala cvs2rss in the near future. (and try to resist making it into a wildly successful commericial venture, that would be depressing)

ps. this post almost was titled, “It was a Dark and Stormy Night” (after all it was!), this is why people are bad at meta-data.

A Thought

So I compiled an overview of the problems, the challeges, and work that has gone before on building an RSS threading standard.

A Quick Review of the State of the Archiver

MHonarc is something of a standard, but I can’t standard the archives it generates, it should be possible to generate something attractive and usable, but I’m still waiting. Current otherings run from the pedestrian Mail Archive to byzantine Sympa style)

Mailman’s archives ( pipermail) are pleasingly straightforward and clean, however their threading algorithm is a little weak, the archives are fragile with slight changes to the mbox changing URLs, and rebuilding the archives for very active, old lists can be incredibly slow. This and pipermail has no clue what to do with attachments.

Zest is an intriguing alternative I’ve mentioned before, however people I’ve shown it to find it confusing, and while I think they could learn, I hesitate to recommend it as a drop in replacement for MHonarc/Pipermail.

The Problem with MMRSS

Unfortunately mmrss doesn’t solve our problems. It scrapes the existing Mailman archives, and creates very simple RSS 0.91. Because RSS 0.91 isn’t very flexible, there is no way to include most of the interesting information, including the basic meta-data like creator (From:), and date sent (Date:) as well as more email specific info (like attachments, user agent, and message id), and the messages threading information (In-Reply-To:)

This means that while MMRSS could be useful for watching a list (ie. getting notified when it updates) it does not provide a meaningful alternative to reading the list.(this could partially be due to the script started life to generate RSS feeds for FoRK, an email based proto-blog with an interesting role in the history of RSS/Email convergence)

What would we want?

On the <channel> definition we’ll want the usual suspect, including:

• the name of the mailing list in the <title>
• the URL of either the lists webpage if it has one (as all Mailman and Sympa lists do), or the link to the web accessible archives (if for some reason you are using something else, and haven’t setup a webpage)
• the date of the most recent message to the list in <dc:date>
• and as much other meta-data as possible including description, and language.

• the list, and list admin addresses, perhaps in <dc:creator> and <dc:publisher>? or would that be an inappropriate re-use?

On a particular <item> we’ll want:

• the URL to the web archived email
• the subject of the email in the <title>
• the contents of the From: field in <dc:creator> (or one of the sha1/foaf based email obscuring technologies being discussed on rss-dev)
• the date the email was sent (or received by the mailing list software) in <dc:date>
• the full content of the message in the <description> (if you’re serious about providing an alternative interface to the list)

Ideally we would also have:

• information about the messages membership in a thread (see below)
• links to web archives of any attachments that might have been included in the email
• a link (or mail address) to reply to this message particular message

There might also be reasons to include:

• Message-ID (perhaps for constructing replies)
• User-Agent
• Spam Status (or similar Spam flagging header)
• CC information

Examining the Prior Art in Threading

Not surprisingly, displaying email as RSS, displaying mailing lists as RDF, and building interchange formats for threaded discussion in RSS have all been discussed before. No one change up with exactly the same feature set (surprise!) because everyone had slightly different conceptions of the problem.

The first important insight when considering an implementation (and examining prior art) is to realize that much of what is important to representing a mailing list is important to representing any form of threaded discussion, like the comments from a message board or blog, or the posts from a newsgroup.

There is the proposed RSS threading module, that adds one tag: <child>. Which is nice and simple, but also kind of awkward as email tends to be linked into threads by referring to a parent.(In-Reply-To:)

ThreadML was an interesting initiative of Steve Yost of Quicktopic, partially created in a response to this article on Joho. It was a standard composed of RSS 1.0, modcontent, modthreading to represent parent-to-child, and modannotation to represent child-to-parent relations. There was a very active quicktopic discussing ThreadML for a while, but it seems to have gone quiet. An example Quicktopic RSS feed to see how it might have looked.

Discussion of creating an RSS feed for the W3C’s mailing lists, prompted the proposal of a modemail which might be useful, but doesn’t seem to focus on the commonality between different mediums enough for me.

The PHP mailing lists are available as RSS feeds, unfortunately here again, they aren’t very useful RSS feeds, with the From information stuck into a <mailto:> tag in the <description> tag. Odd.

Yahoogroups (formerly eGroups) provides RSS feeds for the lists (e.g. RSS-DEV[18] they host, see the original announcement on FoRK)

Mail-archive.com makes a simple RSS 0.9 feed available for each list, for example mod_perl’s RSS

Lastly, the Thread Description Language is an interesting attempt to build a rich RDF syntax for talking about all sorts of different threads. Some of its concepts like agreesWith and disagreesWith would be very cool to add to a RSS/RDF feed based on Zest and its inline mark up.

Conclusion, and Concerns

All of that is very interesting, but I don’t feel like any of the above directly maps to the features I mentioned above, it might be possible to assemble something out of the pieces, but a few items (like the url/mailto to respond to a post) is totally missing from any of this.

It might be worth looking at an email<->NTTP gateway to see what tricks they play, and what they consider necessary.

One problem with marking up email in XML is that it makes it very very easy for spammers to identify email addresses. There is a thread on RSS-DEV about ways to combat this. It seems like it should be possible with the collusion of the list archiver to send out “privatized” emails, where contact information is replaced with some sort of smart URI to confuse harvesters without interrupting communication. Yahoogroups kind of does this.

It would cut down on the complexity immensely to just skip the whole threading thing, but I think threading is an important feature for facilitating a culture of discussion, often discourages in spaces that use “linear threading” (like the traditional display of an email client’s inbox) Might be suitable for a version 1.0

Yet another approach would be to get away from the concept of an individual post, and syndicate conversations (threads). This would map very well to Zest’s concept of threads, but would work pretty well for normal archives and threads as well. The one trick would be figuring out distribute threads in such as way as to be useful with the most recent post readily accessible.

Related Posts:

• Revolutionizing the Mail Archive

This is my brain dump so far. Lots of questions, a few answers. The next step will be firing up the text editor and looking under the hood.

Steal from the Best

• Moveable Type – in my few glaces at the code its seemed intelligent, and well structured. Runs under mod_perl and CGI, handles some complex UI demands like pages that use redirects to update progress bars, and re-entrant forms.
• Bricolage – based on Mason, I’ve been hearing good things about this CMS, particular about its clever Burners abstraction, hope to find other good things under the hood. Don’t really know much about Mason, not sure if that is going to be barrier
• RT – popular, mod_perl and Mason, ticket tracking software. Also supports email, and command line interfaces.
• Scoop – haven’t looked at Scoop in a while, but I remember it seemed well done (having just come from looking at Slashcode 1.0), I wonder if it will stand the test of time. Not expecting to find anything all that relevant, but maybe. Will probably do a quick skim of the CMS parade: Slash, Everything2, and maybe LiveJournal. Definitely want to look at LiveJournal’s embedding instructions.
• ?? Do you any Perl web applications you would consider examples of best practices??

Perl Frameworks

• OpenInteract seems popular, and uses TT2, but leaves me cold, partially because I’m just not crazy about persistence layers. But I should probably spend some time looking at it. ?? Any expirences using it ??
• Mason natively uses an embedded, page based execution model, which seems PHP-like to me, but Design Issues with Mason: What are Components For? seems to be simple guidelines for avoiding the siren calls. There is an Oreilly book coming out on it, which will be available online, but the author says, don’t expect it until mid-October.
• Wombat is billed as an implementation of the Servlet API in Perl, but I can find exactly zero reference to anyone using it, never a good sign.
• AxKit is a web framework built along the lines of Cocoon. I bet there are seriously cool ideas to steal in there, but I’ve been burned pretty badly playing with XSLT and don’t want to go back into that water right now.
• OpenFrame also seems interesting. It has the request and response objects ala servlets, and acme is being paid to work on it, and it has a “Slot” concept which seems similiar to Brico’s Burner concept, and supports TT2. However it bills itself as an “open source application framework for distributed media applications.” Which isn’t really what we are building.
• A few others are listed at Application Servers and Toolkits based on modperl. Also, Web Frameworks and their Template Engines is interesting.
• Last time I was looking at this stuff there was JellyBean, which no longer seems maintained, and Iadio, which no longer exists.
• I found a simple home-rolled framework that feels simliar to the one I did for Rockwood. It was mentioned in this interesting thread.
• The book, Perl for the Web from New Riders, is freely available online and seems interesting if out of date.

• We’ll definitely be using Template Toolkit. Read Perrin’s classic article on Template engines. (the sequel to his classic article on building the eToys website.)
• Apache::Session is a given. Not finding a good webpage on it, but it has great POD.
• Cache::Cache is a given. The Perl Cache project is one of those tools, like TT2, that makes you never want to leave Perl.
• DBI is an of course, and I think we’ll be avoiding the DBIx namespace in favour of having a peer layer with hand written SQL.
• Many people are reccomending CGI::Application as an implementation of the central dispatchers. An article on using CGI::Application, and on using CGI::Application with Template Toolkit. I’ve looked at it before, and it was simpler to roll my own, but I’m going to check it out again. It runs cleanly under Apache::Registry, but isn’t really a full blown modperl app.
• Data::FormValidator is recommended, and includes a tutorial, but some people prefer HTML::FormValidator, but I don’t really know what the difference is yet. Hmmm, they look like they might actually be the same module. ?? Any idea ??
• ?? Did I leave out your favorite module ??

UPDATE: After looking at the available options, and at Struts 1.1, I would go with Java if I didn’t know I would have an insurrection on my hands.

I was reminded of it recently when researching fastmail.fm, whom I think has done a great job implementing this exact same business plan, but for email. Keep prices low, allow people to upgrade for a small, and well disclosed fee, grow slowly and stably on the back of income so you’re around for the long haul, and provide what people are asking for. I was also reminded of it by my partner just recently started working as an office manager at a university and said, “I finally understand why Anyday was cool.” 7 months after Palm shut it down unfortunately.

Anyday was a web-based PIM, inspired by the Palm pilot, and built largely by ex-Lotus people (and a few of us with different calendar backgrounds). The primary goal was to get a functional online PIM that could sync with your desktop PIM, and Palm pilot, for the purposes of remote access, backup, or act simply as a stand alone product. Once people were online, there was the opportunity to use that connectivity to do things impossible with the traditional applications, scheduling, free/busy lookup, delegation, etc.

A while back I noticed that the Horde had almost all the components neccessary to satisfy that primary goal. And I was tempted to see if it could be used to build an Anyday clone, just to see how fast it could be done, and how irrelevant all of us overpaid Java programmers were. Once Anyday was gone the idea evolved.

Horde provides: an addressbook with an LDAP backend, webmail with an IMAP backend, a web calendar, and hooks for integrating them. So it would be real simple to throw up something like Anyday, minus groups, scheduling, and sync.

Sync

They(we) bought into a solution (Extended Connect), that said “All things must go through me.” Obviously Extended liked this because it made then more valuable.

But Sync’ing with the Palm could be really simple. I think a little desktop client, that can talk XML-RPC to the server, used pilot-link to the Palm, and parsed SyncML, would do the trick nicely. And if it was properly open and interesting, would take off as its own open source project.

(update: not sure now iSync, and OS X fit on the landscape since this was written, might function as an open source alternative to iSync)

Self-Sustaining

Here is where I think the most intersting parts come in. I’m inspired by Yahoo’s “sort-of-mirco-payments” at mail.yahoo.com, where you get 6M free, and then you get another chunk for $2/month

The basic service would be free. And then you pay for extras.

Extras would include

• Direct access to the LDAP server behind the addressbook for sync’ing to Outlook, Evolution and such.
• Direct access to the IMAP server again for access from Outlook, Evolution and kin.
• More disk space, not sure if this is just for email, or total. (update: I notice that diskspace isn’t fastmail.fm’s primary concern, but bandwidth, that makes sense, the collorary to the trusim “storage is cheap” is “bandwidth isn’t”)

CAP access when CAP becomes more exciting. Reefknot is almost ready! (update: Well Reefknot seems to have slowed down, but iCal.app, Entourage, Evolution, and Outlook are all becoming serious iCalendar players)

You would want an account page, that summarizes all expenses, and makes people feel very aware of how much they are spending. I think part of what scares people about mirco-payents is feeling out of control.

Another trick would be to only bill when the amount owed gets large enough to make it worth it. And to be clear about this. (sort of the reverse of what Amazon does for affiliate accounts) (update: And to send a nice email about it the way Safari does)

I think the billing system would be the most complicated piece of development. Probably 2 weeks work. (+2-3 weeks to debug (update: I’ve sinced learn I badly underestimate the time things take these days)

Further Development

I think Groups is an important concept to really make a service like this useful. Both formal groups, and ad-hoc ones. And of course I would like to expand the ideas of group scheduling that I was playing with before But I don’t think they’re critical for initial acceptance.

update: Since I originally came up with this idea, iCal has been annouced which makes some noise about the potentials for sharing calendar information, and syncing. It will be interesting to see (in a couple of weeks now) what they’ve come up with.

A long road ahead

Heading out Rt. 2 we nearly turned back in despair several times, only sustained by the how beautiful and green that part of Mass can be. Even after finally seeing the “Entering North Adams” sign, we hadn’t arrived, it was merely a lure to reel in the reluctant. We would pass through 2 more towns, and cross a small mountain range, before we descended (beautiful view) into the Hoosac river valley, and the old industrial town of N. Adams.

Doubts

Franz West

Walking into the exhibit room a cheerful note proclaims, “All Chairs May Be Sat In.” A 30ft tall sculpture of a Pepto Bismol pink intestine, “Drama (Model)” quickly draws the eye. As does the 90ft long white table, “Kantine”“, that attempts to play with Kant’s 2 ideals of pleasure. We’re told at the MAK this was a performance piece, where you ate heavy German peasant food while staring at the high art intestine so that the lump in your stomach and your growing panic of not understanding why Austria’s most influential artist has sculpted a giant intestine come to symbolize Kant’s interested and disinterested pleasures. I don’t personally remember Swedish meatballs coming up in our discussions of the Enlightenment but it was an intriguing piece.

Erwin Wurm

A self portrait of the artist in a hot pink button up and tight black pants, shot from a decidedly unflattering angle, looks down upon a car which sags with heavy curves of flesh. The hood is chubby; the bumper and skirt, and mirrors droop with cellulite; the door handle, and key hole are deep dimples. The whole car is the unreal pink of a “Flesh” crayon, and fat. Unlike West’s more conceptual intestine, endless hours have clearly been put into to making the hard plastic shell of this car look like you would squish into its fleshy expanses if pushed up against it. I feel as if we are supposed to be repulsed by its obesity, but to someone like myself who has never embraced the sleek metallic aesthetic of modern cars it is also sexy.

As we stand in awe, staring at this contrivance, we are excitedly informed by a fellow museum goer that underneath all that pink excessive is “a Ford Escort! Can you believe that? A Ford Escort!”. I feel deeply alienated from my fellow citizens.

(Update: Actually the photo behind the car is not the artist, a man, but the curator. Suddenly my whole understanding of the piece bucks, and swims before my eyes, as I reconsider how I feel about it…)

Lois Weinberger

Probably my favorite artist from the show, and probably the understated. His little alcove revolved around a collection of plastic bags, filled with dirt and plants under a grow light, “Portable Garden”. The plants were weeds transplanted from the grounds of MassMOCA, the bags I recognized from Tijuana, but apparently they are also popular among the Vienna’s poor immigrants from Eastern Europe, and Africa. Hard to capture in words, you could sense Weinberger’s deep sympathy with these scrappy, unwelcome transplants. A message that blended environmentalisms, and social justice.

Against one wall was a large piece of newsprint in which a sprawling fictitious city plan has been depicted, “Course/Drift”. The city has grown smack up against its encircling mountains, represented with topo lines, and is now trying to figure out how to go on growing. The cityscape is labeled, not with traditional names like Main St., or 6th Ave., but with a powerful and loaded vocabulary. (on which I did not take notes, unfortunately) A plaza might be named happiness, taxes, lost, or ball game. A concourse I remember was named “Upper Class Concert”, a little neighborhood block named “Revolution”. It conjured up one of Calvino’s invisible cities, and I stood and stared for a long time.