Laughing Meme

When you’ve been working with a technology for a long time, it’s difficult not to develop Stockholm syndrome. Not sure when I started using MySQL, but I bought my first license in 1998. I think it wasn’t until mid-to-late ‘98 when we had to call Monty long distance to Sweden to get help with some tricky issues. Which is to say its been a long time since I thought about how confusing MySQL’s CREATE TABLE syntax can be.

Which is not to say that the documentation isn’t clear:

M indicates the maximum display width for integer types. The maximum legal display width is 255. Display width is unrelated to the range of values a type can contain, as described in Section 10.2, “Numeric Types”. For floating-point and fixed-point types, M is the total number of digits that can be stored.

But last week Flickr had a hiccup. We hit 4,294,967,295 photos. Or as a geek might say it, the largest number that can be represented by a 32-bit unsigned integer. This didn’t exactly catch us by surprise. We’d switched to using 64-bit ids for some things January, Friday the 13th, 2006. That and we got bit a few years ago when we hit 2,147,483,647 photos (that’d be the max signed 32 bit integer). Shortly after that we did a full audit of our tables.

But somehow we went on writing code after that, and we managed to slip a couple of new tables into the mix. And some of those tables ended up with INT(20) columns. Which simply mean we were adding some non-significant zeros to pad the display but truncating photo ids over 4294967295.

INT(5), INT(10), INT(20), and INT(255) all store the same amount of data.

Funny thing is, when I told this story to folks last week, this caught them by surprise. Sophisticated engineers, some of whom had deployed quite large MySQL backed sites. Because they were right, that syntax is dumb. And confusing. And I’d been taking it for granted so long I hadn’t thought about it in a decade. Which is why I’d bother to write a blog post about a popular piece of software, behaving exactly as it’s extensively documented to work.

Also, it’s interesting to note how if you keep making the same mistakes they become easier and easier to fix.

If you’re ever debugging a problem and you see the number 42-mumble-mumble-mumble-7295 you’ve run out of 32-bit storage. If you see 2-mumble-mumble-mumble-647 (2147483647) you’ve run out of signed 32-bit storage. 167-mumble-mumble-15 (16777215) you’ve run out of 24-bits and 65-mumble-mumble-35 (65535) you’ve run out of 16-bits of integers.

Somehow those numbers just jump out at me after all this time, you ignore the numbers in the middle, and notice the significant bits at the front and the end.

Photo from pjern

On an unrelated email thread this morning I got to thinking about how I quantify the Flickr engineering team, and counting things in general.

Depending on how I’m counting I tend to place the Flickr engineering team at ~20 people. In that group I include everyone on our team who writes code (including HTML, CSS, Javascript, PHP, Java, Perl, Python, C, C++, XUL, or Objective-C). Additionally I include our operations team (aka sysadmins aka “service engineering”), our “tech support” team (technical customer care/qa/researchers), and various folks with “manager” in their title.

(a more traditional count would probably put the Flickr engineering team at 5 application/backend engineers, 4 front-end engineers, and 4 technical manager types.)

Which got me thinking about a new metric, the RPE or “roughly per engineer”. Mostly it’s a useful thought tool (for me) to think about what sorts of things scale up with economies of scale, and what doesn’t. Here are a couple of quick RPE metrics I pulled tonight.

• We’ve got roughly 2.5 million Flickr members per engineer.
• Roughly 200 million photos per engineer.
• 28 user facing pages.
• 23 administrative pages.
• 20 API methods, though only 7.5 public API methods.
• 80 API calls per second.
• 250 CPUs.
• 850 annual deploys.
• 16 feature flags.

Photo from siliconmonkey

… I mean Flickr/Upcoming/Delicious. In particular, I mean that brief moment of optimism in the Spring of ‘06, on the roof of the Iron Cactus, at the Spread the FUD party, when it looked like Yahoo! had a wedge and the will to solve the social search problem, and magically, I might even get to be a part of that. I said in my cover letter (in silly flowery, cover letter speak)

“The next round of innovation will be about building connections. The explosion of voices, information and ideas is currently outpacing our techniques for coping with them. We need to be helping people and communities find new ways to connect, interact, and work together to make sense of this accelerating decentralization. Innovation has been blossoming at the edges of the Net since the beginning, but innovation is also moving back to the connecting nodes, like Yahoo.”

Which is much on my mind when I hear about Marissa demo’ing social search yesterday.

And I’m deeply puzzled (and not a little disappointed) that anyone would care if Bing or Google can search the public status timelines, if it doesn’t come with social context.

Now the question is can Goog shake their historied failure at all things social.

Photo from Jan Brašna

Aaron’s FlickrApp is a brain dead easy to use subclass of webapp.RequestHandler that turns Flickr into an single-sign on service for Google AppEngine.

As a bonus, you get a valid Flickr auth_token for every signed in user. This makes writing Flickr API apps about the simplest thing ever.

Case in point, I wrote the largely mis-named photosthatmatter app last night in slightly less then 20 minutes, while waiting for dinner to simmer. Shows the most interesting photo from each for your contact in a given time period. Great for catching up on things you missed as they flowed by the first time.

YMMV, but it works for me.

I remember it as a dark and stormy night, that seems unlikely, but I’m sure it was late and chilly and damp.

I remember being tired from a long day in the salt mines; that was during a period when I was always tired after work.

I remember there being whiskey, and knowing @maureen, that seems likely.

I’d just won some internal battles regarding delegated auth, and implemented Google AuthSub for the new Blogger Beta, as well as Amazon auth for a side project. So when I wanted to share photos from Flickr to Twitter, I knew it wasn’t going to be over HTTP Basic Auth.

A few weeks earlier @blaine and @factoryjoe had pulled me a into a project called OpenAuth that they’d been talking about for a couple of months — an alternative to yet another auth standard, and a solution for authenticating sites using OpenID.

So one late, damp night along Laguna St. with whiskey, we did a pattern extraction, identifying the minimal possible set of features to offer compatibility against existing best practice API authorization protocols. And wrote down the half pager that became the very first draft of the OAuth spec.

That spec wasn’t the final draft. That came later, after an open community standardization process allowing experts from the security, web, and usability community to weigh in and iterate on the design. But many of those decisions (and some of the mistakes) from that night made it into the final version.

Yesterday, a little over two years later, we finally shipped Flickr2Twitter.

So it was nice yesterday when people commented on the integration:

“Uses OAuth!” “Doesn’t ask for your Twitter password” “Great use of OAuth”.

And I thought to myself, “It better be, this is what OAuth was invented for — literally”.

Uploaded by straup on 12 Mar 09, 1.49PM EDT.

by Kent Brewster

I tried explaining the title of this blog post to Jasmine this morning. Suffice to say my explanation needed a bit of practice. And more than 140 characters. Or it might just be I’m a bit stir crazy from Winter returning with a vengeance in these here parts. But I wanted to call out a couple of points that might have gotten overshadowed in the good Reverend’s recent post on the Flickr Panda APIs.

NewsWire API

The NY Times at their great Times Open event announced their Newswire API, which is a real time stream of their content. Stories, and blog posts, and what not. More interestingly was their discussion about how they’ve built a backend “pinging service” that makes it easy for them to add new types of data to their stream. I’m a dork enough that a Grey Lady firehose sounds pretty awesome.

But they got some flack for it being a snowflake API. From where I sit snowflake APIs look like opening up your data as fast as possible, along any means necessary, and trying not to pre-judge how people will use it, but I’m thankful for the metaphor, as it allowed me to spend the morning envisioning fire hoses of snowflakes.

Still I spent 2007, and 2008 talking about how XMPP was going to be a key piece of building firehoses standardizing and enabling the real time Web, so its a criticism I’m sensitive to. (and I’ve already been skipping conferences in 2009 in the hopes of actually having some time to build it, though thankfully minor details like time haven’t stopped my colleagues at Fire Eagle from launching theirs)

Pandas

Which is all apropos of saying, we launched our own “snowflake” realtime API yesterday. (though actually its just a slight modification of our standard photo response format). And its Panda-shaped. And it is awesome.

Near Real-Time, Every Minute, up to 120 Events

But because the documentation is quirky, I think people missed the significance. These are Flickr real time data APIs.

We’re building streams of photos in real time. Examining the huge stream of data events that happen on Flickr, the social activity, the searching, the meta-data creation, and fishing from that stream to build 3 real time streams. We’re then exposing those streams via a near real time polling based API.

The API pattern is specifically structured around making it easy to call from client side scripting, and the data streams are structured around discovery rather then guided search, but we’re pushing up to 120 discovered photos down these streams each minutes, every minute. Two streams of real-time interestingness, and 1 of lightly interestingnessed geotagged photos.

And they’re named after famous pandas. Really what more do you want?

Whither XMPP

So what’s up with the blossoming real time data APIs? And where is our promised standardization? They’re coming. There has always been a tricky chicken and egg problem. There is so little data out there that is appropriate to expose in a real time fashion, that there is little demand to consume it, so the tools fail to evolve. But I’m seeing tons of work, great toolkits from like Fire Hydrant from FireEagle and Babylon from notifixio.us, and Google’s decision to make XMPP a standard part of their AppEngine toolkit are just I’ve been most excited about recently.

Based on Derek’s NYT Trender and some APIs we haven’t gotten around to releasing, I spent 20 minutes whipping up Flickr Trends yesterday morning.

App Engine is awesome for this kind of stuff.

Favorite’s I’ve found so far:

• snow vs flowers
• party vs wedding
• jumping vs hdr (oooh, 365 vs hdr is more fun.)

asc: it seems to me that we need to set up a magic email address for “things on fire”

From an amazing set by Ai de ke.

Photos from waterbubblz, dgrubz, murdlebanks, so_phie, elton and picturesofthings

I tried to fit this all into 140 characters. I really did. I couldn’t do it, not even with disemvoweling.

#motrinmom

Chatting with a friend who does information architecture for pharmaceutical advertising she was shocked I hadn’t heard about the “Motrin Mom” twitter-in-a-teapot. I had no idea what she was talking about.

Apparently “Twittering Critics Brought Down [the] Motrin Mom Campaign”. And the entire advertising industry, at least here in New York, is having a fear-of-a-twitter planet moment. Complete with righteous anger about the “irrationality of Twitter”. (um, hello folks, but didn’t you build one of the largest global business by cynically manipulating people’s “irrationality”?)

But the part that really caught me off is this didn’t blip my radar at all. Maybe I was just offline for it, but as far as I can tell the twittering classes I follow didn’t peep about this. I thought Twitter was all about us? (Also, Summize you are already awesome and everything, but if you add “search within people you’re following” and “search within people who follow you” I promise to love you forever)

@flickr

Only tangentially related, I’m sure Tyler Hawkins aka @flickr has a very busy @replies tab.

What I can’t figure out is if all these folks responding to @flickr are really confused about whether Hawkins is a Flickr representative (he isn’t and doesn’t in anyway suggest he might be) or just believe so strongly that “@flickr” address twits will arrive in Flickr’s inbox that reality is irrelevant.

I’m torn on whether the assumption that when you speak you will be heard is the ultimate arrogance (and one particularly prevalent on Twitter), or if rather this proves that we’ve historically worried too much about URIs and that culture has no problem evolving them ad-hoc.

Now if only I had a thesis, rather then a rambling collection of half thoughts. Which is why I wanted to fit this all into 140 characters. Alas.

Historical landmark New York Central Railroad 69th Street Transfer Bridge, with Trump Towers looming in the background.