Anyway I got inspired last weekend to spend a little while coding up a primitive Top Explorers implementation that would run over the API and on top of Pig. And as part of that I resorted to one of my favorite hacks for visualizing Flickr API results, the “standard photo response as slideshow” hack.

Not sure it was ever documented, but you can call it with the form:

http://www.flickr.com/show.gne?api_method=$method_name&method_params=param1|paramvalue1;param2|paramvalue2

The canonical example of this is kittens from galleries, but I was just using it to help visualize variations on recent photos sorted by interestingness.

With any luck I’ll actually have some interesting Top Explorers results soon as well.

tl;dr photosnearme

I recently was added to a “People like John Resig” Twitter list. I am nothing like John Resig. In particular I suck at Javascript.

However, given the annoyance of properly constructing a Flickr geo feed URL (admittedly linked to from the bottom of every places page, but whose counting), I decided to make a script for it while waiting for the kettle this morning.

I broke out the jQuery (thanks John!) and some code from Weather Near Me, and now there is photosnearme for finding the feed of photos for the neighborhood you’re in. YMMV.

ps. I keep forgetting and having to remind myself, Chrome’s ridiculous over aggressive caching makes it totally unsuitable for development purposes. That was probably the hardest part of the script.

pps. I don’t care that RSS is dead, I still like it.

see also: other waiting for the kettle scripts

The “You and X” pages are my favorite Facebook feature I’ve seen in a long time. (I’ll admit I’m a pretty limited user of “the Book” so the bar is low, though a new feature we launched adminonly yesterday on Etsy is making me think about bulking up on ‘Book friends).

Also reminded me of my favorite semi-secret feature of Flickr people tagging: the “You + X” pages.

If you visit the “/photosof/kellan” style people URLs you’ll get prompted “Anyone ever snapped a picture of the two of you?” and if you click on the link you can see pictures of the two of you. (us?) You can also hack the URL to string together a longer list of folks.

Got me thinking about standards. I wrote back a rant to a mailing list of fellow senior hacker, and coders types. Below I’ve included that rant, largely verbatim. I’d been meaning to turn it into a more reasoned blog post, maybe something suitable for posting on a more official outlet, but life is short, and Rod’s post about Quora reminded me to get on it.

As software engineers, as social software engineers, it’s important to have standards. You can debate the how much of what we do can be called engineering, even charitably, but the code we write determines the rules that govern the spaces more and more people spend time in, and while “First, do no harm” might be reaching, a few standards that you should be embarrassed to not meet seem appropriate.

One of those is around data access, data ownership, and sharecropping. This is something Flickr takes very seriously.

The Minimum

With Flickr you can get out, via the API, every single piece of information you put into the system.

Every photo, in every size, plus the completely untouched original. (which we store for you indefinitely, whether or not you pay us) Every tag, every comment, every note, every people tag, every fave. Also your stats, view counts, and referers.

Not the most recent N, not a subset of the data. All of it.

It’s your data, and you’ve granted us a limited license to use it.

Additionally we provide a moderately competently built API that allows you to access your data at rates roughly 500x faster then the rate that will get you banned from Twitter.

Asking people to accept anything else is sharecropping. It’s a bad deal. Flickr helped pioneer “Web 2.0″, and personal data ownership is a key piece of that vision. Just because the wider public hasn’t caught on yet to all the nuances around data access, data privacy, data ownership, and data fidelity, doesn’t mean you shouldn’t be embarrassed to be failing to deliver a quality product.

The ability to get out the data you put in is the bare minimum. All of it, at high fidelity, in a reasonable amount of time.

The bare minimum that you should be building, bare minimum that you should be using, and absolutely the bare minimum you should be looking for in tools you allow and encourage people who aren’t builders to use.

A Reasonable Exchange of Value

Flickr actually goes a bit farther, not only can you get your data out, but it gets enriched as it passes through the system.

If you use the geotagging feature, you don’t just get the lat/long out you put in, but your photo comes back with a whole hierarchy of geographic descriptors, that are pointers into a publicly available gazetteer (Y! GeoPlanet). It would be good if there were pointers into other publicly available gazetteers (if for example Google ever released one) but there isn’t a good concordance service yet (but it’s being worked on)

You get structured access to all the metadata that people have added to your photos, with proper attribution available. (of course there is a working privacy model, so your “friends” aren’t getting data they aren’t supposed to, like your friend requests, and chat logs)

If you used our machine tags vocab, you get extra information pulled in from 3rd party APIs like Open Street Maps, Open Library, Last.fm, various transit administrations, and Foursquare.

Additionally you also have access to the data that was created in aggregate using the data you shared with us, like tag clusters, and the Creative Commons licensed neighborhood shape boundaries.

This isn’t the exhaustive list, just a few of the things Flickr does to respect, and collaborate with the people who share their time and data with us.

I’d certainly love to get a fraction of this data back from other services I use. Imagine getting access to all the data Google has about you, and everything they’ve learned partially based on observing you. I’ve gotten used to being disappointed by most of my fellow practitioners, but I still dream about using good tools that treat me with respect and want to collaborate.

Thanks go to Jesse Vincent, for the useful sharecropping metaphor.

(and I’ll state the obvious this is my personal blog, nothing I post here should be taken as official Flickr or Yahoo communication or policy, unless otherwise noted, that isn’t what they pay me to do.)

“This, is what geotagging photos is all about, it’s about having enough of them, millions and millions, so that they can be thrown through complex analysis, allowing them to be matched up, combined, calculated and computed into a geo-spatal context. It’s also about people sharing the world about them. Start of mini rant: You’ll see that all these advances are made by Google and Microsoft …” – Rev. Dan Catt.

I try not to let it get to me anymore that we’ve been actively de-prioritizing geo as an axis of understanding the human experience as everyone else has been spinning it up..

“Flickr offers the wonderful Recent Activity page that I loved so much I copied it for MetaFilter. It's pretty much the ultimate tool for finding what has happened with your content on the network and I hope other services are watching and following suit. I would love to see an internet-wide tool that worked like this to track stuff people have said about my writing/photos as well as any followups on comments I left on any other blog. Many companies have tried, no one has succeeded yet.” – Matt Haughey

Yay! That almost makes it worth how much pain it was to build that page. The whole post is good, a couple of neat tricks I’d missed for tracking the conversations.

http://code.flickr.com/blog/2010/02/10/5-questions-for-simon-willison/

http://developers.facebook.com/news.php?blog=1&story=358

http://kottke.org/04/10/normalized-data

http://blog.flickr.net/en/2010/01/29/fence-friday-2/

When you’ve been working with a technology for a long time, it’s difficult not to develop Stockholm syndrome. Not sure when I started using MySQL, but I bought my first license in 1998. I think it wasn’t until mid-to-late ’98 when we had to call Monty long distance to Sweden to get help with some tricky issues. Which is to say its been a long time since I thought about how confusing MySQL’s CREATE TABLE syntax can be.

Which is not to say that the documentation isn’t clear:

M indicates the maximum display width for integer types. The maximum legal display width is 255. Display width is unrelated to the range of values a type can contain, as described in Section 10.2, “Numeric Types”. For floating-point and fixed-point types, M is the total number of digits that can be stored.

But last week Flickr had a hiccup. We hit 4,294,967,295 photos. Or as a geek might say it, the largest number that can be represented by a 32-bit unsigned integer. This didn’t exactly catch us by surprise. We’d switched to using 64-bit ids for some things January, Friday the 13th, 2006. That and we got bit a few years ago when we hit 2,147,483,647 photos (that’d be the max signed 32 bit integer). Shortly after that we did a full audit of our tables.

But somehow we went on writing code after that, and we managed to slip a couple of new tables into the mix. And some of those tables ended up with INT(20) columns. Which simply mean we were adding some non-significant zeros to pad the display but truncating photo ids over 4294967295.

INT(5), INT(10), INT(20), and INT(255) all store the same amount of data.

Funny thing is, when I told this story to folks last week, this caught them by surprise. Sophisticated engineers, some of whom had deployed quite large MySQL backed sites. Because they were right, that syntax is dumb. And confusing. And I’d been taking it for granted so long I hadn’t thought about it in a decade. Which is why I’d bother to write a blog post about a popular piece of software, behaving exactly as it’s extensively documented to work.

Also, it’s interesting to note how if you keep making the same mistakes they become easier and easier to fix.

If you’re ever debugging a problem and you see the number 42-mumble-mumble-mumble-7295 you’ve run out of 32-bit storage. If you see 2-mumble-mumble-mumble-647 (2147483647) you’ve run out of signed 32-bit storage. 167-mumble-mumble-15 (16777215) you’ve run out of 24-bits and 65-mumble-mumble-35 (65535) you’ve run out of 16-bits of integers.

Somehow those numbers just jump out at me after all this time, you ignore the numbers in the middle, and notice the significant bits at the front and the end.

Photo from pjern

On an unrelated email thread this morning I got to thinking about how I quantify the Flickr engineering team, and counting things in general.

Depending on how I’m counting I tend to place the Flickr engineering team at ~20 people. In that group I include everyone on our team who writes code (including HTML, CSS, Javascript, PHP, Java, Perl, Python, C, C++, XUL, or Objective-C). Additionally I include our operations team (aka sysadmins aka “service engineering”), our “tech support” team (technical customer care/qa/researchers), and various folks with “manager” in their title.

(a more traditional count would probably put the Flickr engineering team at 5 application/backend engineers, 4 front-end engineers, and 4 technical manager types.)

Which got me thinking about a new metric, the RPE or “roughly per engineer”. Mostly it’s a useful thought tool (for me) to think about what sorts of things scale up with economies of scale, and what doesn’t. Here are a couple of quick RPE metrics I pulled tonight.

• We’ve got roughly 2.5 million Flickr members per engineer.
• Roughly 200 million photos per engineer.
• 28 user facing pages.
• 23 administrative pages.
• 20 API methods, though only 7.5 public API methods.
• 80 API calls per second.
• 250 CPUs.
• 850 annual deploys.
• 16 feature flags.

Photo from siliconmonkey

… I mean Flickr/Upcoming/Delicious. In particular, I mean that brief moment of optimism in the Spring of ’06, on the roof of the Iron Cactus, at the Spread the FUD party, when it looked like Yahoo! had a wedge and the will to solve the social search problem, and magically, I might even get to be a part of that. I said in my cover letter (in silly flowery, cover letter speak)

“The next round of innovation will be about building connections. The explosion of voices, information and ideas is currently outpacing our techniques for coping with them. We need to be helping people and communities find new ways to connect, interact, and work together to make sense of this accelerating decentralization. Innovation has been blossoming at the edges of the Net since the beginning, but innovation is also moving back to the connecting nodes, like Yahoo.”

Which is much on my mind when I hear about Marissa demo’ing social search yesterday.

And I’m deeply puzzled (and not a little disappointed) that anyone would care if Bing or Google can search the public status timelines, if it doesn’t come with social context.

Now the question is can Goog shake their historied failure at all things social.

Photo from Jan Brašna

Aaron’s FlickrApp is a brain dead easy to use subclass of webapp.RequestHandler that turns Flickr into an single-sign on service for Google AppEngine.

As a bonus, you get a valid Flickr auth_token for every signed in user. This makes writing Flickr API apps about the simplest thing ever.

Case in point, I wrote the largely mis-named photosthatmatter app last night in slightly less then 20 minutes, while waiting for dinner to simmer. Shows the most interesting photo from each for your contact in a given time period. Great for catching up on things you missed as they flowed by the first time.

YMMV, but it works for me.

I remember being tired from a long day in the salt mines; that was during a period when I was always tired after work.

I remember there being whiskey, and knowing @maureen, that seems likely.

I’d just won some internal battles regarding delegated auth, and implemented Google AuthSub for the new Blogger Beta, as well as Amazon auth for a side project. So when I wanted to share photos from Flickr to Twitter, I knew it wasn’t going to be over HTTP Basic Auth.

A few weeks earlier @blaine and @factoryjoe had pulled me a into a project called OpenAuth that they’d been talking about for a couple of months — an alternative to yet another auth standard, and a solution for authenticating sites using OpenID.

So one late, damp night along Laguna St. with whiskey, we did a pattern extraction, identifying the minimal possible set of features to offer compatibility against existing best practice API authorization protocols. And wrote down the half pager that became the very first draft of the OAuth spec.

That spec wasn’t the final draft. That came later, after an open community standardization process allowing experts from the security, web, and usability community to weigh in and iterate on the design. But many of those decisions (and some of the mistakes) from that night made it into the final version.

Yesterday, a little over two years later, we finally shipped Flickr2Twitter.

So it was nice yesterday when people commented on the integration:

“Uses OAuth!” “Doesn’t ask for your Twitter password” “Great use of OAuth”.

And I thought to myself, “It better be, this is what OAuth was invented for — literally”.

http://geobloggers.com/2009/04/17/2-every-building-with-a-shoebox-in-its-basement/

Uploaded by straup on 12 Mar 09, 1.49PM EDT.

NewsWire API

The NY Times at their great Times Open event announced their Newswire API, which is a real time stream of their content. Stories, and blog posts, and what not. More interestingly was their discussion about how they’ve built a backend “pinging service” that makes it easy for them to add new types of data to their stream. I’m a dork enough that a Grey Lady firehose sounds pretty awesome.

But they got some flack for it being a snowflake API. From where I sit snowflake APIs look like opening up your data as fast as possible, along any means necessary, and trying not to pre-judge how people will use it, but I’m thankful for the metaphor, as it allowed me to spend the morning envisioning fire hoses of snowflakes.

Still I spent 2007, and 2008 talking about how XMPP was going to be a key piece of building firehoses standardizing and enabling the real time Web, so its a criticism I’m sensitive to. (and I’ve already been skipping conferences in 2009 in the hopes of actually having some time to build it, though thankfully minor details like time haven’t stopped my colleagues at Fire Eagle from launching theirs)

Pandas

Which is all apropos of saying, we launched our own “snowflake” realtime API yesterday. (though actually its just a slight modification of our standard photo response format). And its Panda-shaped. And it is awesome.

Near Real-Time, Every Minute, up to 120 Events

But because the documentation is quirky, I think people missed the significance. These are Flickr real time data APIs.

We’re building streams of photos in real time. Examining the huge stream of data events that happen on Flickr, the social activity, the searching, the meta-data creation, and fishing from that stream to build 3 real time streams. We’re then exposing those streams via a near real time polling based API.

The API pattern is specifically structured around making it easy to call from client side scripting, and the data streams are structured around discovery rather then guided search, but we’re pushing up to 120 discovered photos down these streams each minutes, every minute. Two streams of real-time interestingness, and 1 of lightly interestingnessed geotagged photos.

And they’re named after famous pandas. Really what more do you want?

Whither XMPP

So what’s up with the blossoming real time data APIs? And where is our promised standardization? They’re coming. There has always been a tricky chicken and egg problem. There is so little data out there that is appropriate to expose in a real time fashion, that there is little demand to consume it, so the tools fail to evolve. But I’m seeing tons of work, great toolkits from like Fire Hydrant from FireEagle and Babylon from notifixio.us, and Google’s decision to make XMPP a standard part of their AppEngine toolkit are just I’ve been most excited about recently.

Based on Derek’s NYT Trender and some APIs we haven’t gotten around to releasing, I spent 20 minutes whipping up Flickr Trends yesterday morning.

App Engine is awesome for this kind of stuff.

Favorite’s I’ve found so far:

• snow vs flowers
• party vs wedding
• jumping vs hdr (oooh, 365 vs hdr is more fun.)