Blog posts tagged "xmpp"

Is a Firehose of Snowflakes a Nor’easter?

March 4th, 2009

I tried explaining the title of this blog post to Jasmine this morning. Suffice to say my explanation needed a bit of practice. And more than 140 characters. Or it might just be I’m a bit stir crazy from Winter returning with a vengeance in these here parts. But I wanted to call out a couple of points that might have gotten overshadowed in the good Reverend’s recent post on the Flickr Panda APIs.

NewsWire API

Picture 21

The NY Times at their great Times Open event announced their Newswire API, which is a real time stream of their content. Stories, and blog posts, and what not. More interestingly was their discussion about how they’ve built a backend “pinging service” that makes it easy for them to add new types of data to their stream. I’m a dork enough that a Grey Lady firehose sounds pretty awesome.

But they got some flack for it being a snowflake API. From where I sit snowflake APIs look like opening up your data as fast as possible, along any means necessary, and trying not to pre-judge how people will use it, but I’m thankful for the metaphor, as it allowed me to spend the morning envisioning fire hoses of snowflakes.

Still I spent 2007, and 2008 talking about how XMPP was going to be a key piece of building firehoses standardizing and enabling the real time Web, so its a criticism I’m sensitive to. (and I’ve already been skipping conferences in 2009 in the hopes of actually having some time to build it, though thankfully minor details like time haven’t stopped my colleagues at Fire Eagle from launching theirs)


Flickr Panda!

Which is all apropos of saying, we launched our own “snowflake” realtime API yesterday. (though actually its just a slight modification of our standard photo response format). And its Panda-shaped. And it is awesome.

Near Real-Time, Every Minute, up to 120 Events

But because the documentation is quirky, I think people missed the significance. These are Flickr real time data APIs.

We’re building streams of photos in real time. Examining the huge stream of data events that happen on Flickr, the social activity, the searching, the meta-data creation, and fishing from that stream to build 3 real time streams. We’re then exposing those streams via a near real time polling based API.

The API pattern is specifically structured around making it easy to call from client side scripting, and the data streams are structured around discovery rather then guided search, but we’re pushing up to 120 discovered photos down these streams each minutes, every minute. Two streams of real-time interestingness, and 1 of lightly interestingnessed geotagged photos.

And they’re named after famous pandas. Really what more do you want?

Whither XMPP

So what’s up with the blossoming real time data APIs? And where is our promised standardization? They’re coming. There has always been a tricky chicken and egg problem. There is so little data out there that is appropriate to expose in a real time fashion, that there is little demand to consume it, so the tools fail to evolve. But I’m seeing tons of work, great toolkits from like Fire Hydrant from FireEagle and Babylon from, and Google’s decision to make XMPP a standard part of their AppEngine toolkit are just I’ve been most excited about recently.

WeeWar needs XMPP

May 14th, 2008

WeeWar broke in a wave across the office this afternoon. (thankfully late afternoon, or I might have gotten nothing done today). Its a Web-based turn based strategy game, thats very well done. Sort of a “Flickr for Risk”, with a nice value add pro account ($24.95/year), and APIs, social networking features, and a chatty tone.


But I’ve never run into an application that needed an XMPP interface more.

The most fundamental missing functionality is a convenient, light weight way of getting notified that your turn has rolled around again. WeeWar will send you email, but now your inboxes is even more cluttered, and you’re having to check your inbox constantly. (something I try to keep to 1-2 times an hour)


A Jabber interface you could trust to push to you the state changes news, and thereby remove the nagging, “Is it my turn?” and the variable positive reinforcement relationship it sets up with your inbox.

Additionally its a classic app where, if you’re polling, you want to keep the polling time very low, but the actual incident of change is fairly spare, which means WeeWar is going to at some point start resenting their polling based APIs.


Ideally messages would also include an XML payload describing either the changes since your last turn, or the current state of the map, allowing for rich consuming clients to build alternate interfaces to the world.

New Games

Orthogonally, a new games, and new games from your “preferred players” would also be excellent to get pushed out over Jabber.


February 6th, 2008

Learned a new word tonight from MattB, SimonB, and Yoz.

A hashbot is a robot that hangs out on an IRC channel (hence the #) and provides a conversational interface to a resource.

hashbots are the ancestors of Social Software for Robots, and the idea of Twitter/YubNub as Web CLI.

Notes from Social Graph Foo

February 4th, 2008

Here is my quick dump of the notebook, probably useful to no one but me. Names mostly removed to protect the guilty.

I think “Social Graph” is kind of a dumb phrase to apply to the back question of relationships. I promptly re-dubbed the event “Social Foo” and thereby found interesting things to talk about. Kevin Marks proposed “social cloud”, clouds hide details. (operations people get hives when you talk about clouds)

XMPP, OpenID, OAuth are all going to be huge in 2008; DiSo, DataPortability, and Social Graph API aren’t as clear winners to me.

Bowling Alone misses the point. There has been a transformative change from groups to networks. Groups are just a funny form of network.”

“Differentiated role networks”. Differentiated roles, and the failure of monolithic identity and friending were one of the things I went to Sebastopol to talk about this weekend, the people who got it got it, and everyone else wasn’t interested in the hard squishy details of real community. I think this might be the side effect of running social software for social softwares sake vs social software as bath for social media object sharing.

“Relationships can be broken down into 5 types: emotional aid, sociality, major help, minor help, and $$$”

Note to self: try block modeling interactions in high profile/high turn Flickr groups. (central, utata, etc)

No one really understands user expectations. Privacy expectation is currently, “unstable”.

Huge conceptual issues with the difference between public information hand aggregated, and public information computer aggregated. Cognitive dissonance ensues.

Rules, games, and rulesets. Modeling of social software as games. Tension of implicit vs. explicit rules. Mag.nol.ia’s altruism game derived from the cracks board (witnessing altruistic acts is a public good, way to update the Mag rules of game to support this?), Satisfaction’s status update game. Hoping Teresa can bring the quality gaming to BoingBoing’s anemic community. Social games + adversting.

Parody/pastiche as lit analysis. Investigate for web.

Social networks need NPCs. e.g. the Instructables Robot.

Standards works should be done in small groups, with a clear need, that selectively grow the list of participants. No hierarchy of early/late joiners (aka OAuth did it right)

“Everything public” bores me.

Beyond LAMP.

Find a feed for Nathan Eagle’s research.

“locations rights management”

“trusts are largely not transitive”

Language communities are “small world networks”, partitions communities by language. 2-5 hops vs 8 in analyzed network.

The Plaxo way: “We gets ze data Lebowski”

“Twitter is my early warning system. My blood pressure has gone down over the last 18 months”

Identity and sharing can make everyone warm and fuzzy, but also came face to face with sobering consequences that kept me up at night with a bottle of tequila. Re-thinking proposed Flickr features.

XMPP in TiVo

January 11th, 2008

“Today each TiVo polls TiVo’s severs roughly every 15 minutes to check for new scheduled recordings, TiVoCast downloads, Unbox downloads, etc. That’s highly inefficient – nearly all of those polling calls are for nothing. There is nothing waiting to be done. And it introduces a lag when you want to start a download – up to 15 minutes. And it doesn’t scale well as TiVo’s user base keeps growing.

So what’s changed? The polling system is gone. TiVo is using XMPP now instead. What is XMPP? The Extensible Messaging and Presence Protocol – better known as the instant messaging protocol that powers Jabber, Google Talk, and other IM systems.” – Peter St. Andre noticed as interesting announcement coming out of CES. (via aaron)

Google Talk Architecture, and High Availability (HA)

July 29th, 2007


Via the HA blog (an obviously unserved niche in retrospect), a very interesting 30 minute presentation on the Google Talk architecture.

ConnectedUsers * BuddyListSize * OnlineStateChanges

Interestingly people keep independently re-discovering that maintaining presence is the hard part of scaling these systems.

Its something that really came home hard in my talking with Twitter helping with their scaling challenges (so much so that we took a slide out of our “Social Software for Robots” talk to talk about it, and Blaine mentioned it again in his “Scaling Twitter” talk)

So by way of a PSA:

Presence isn’t easy.

Growth in social systems in non-linear. Ignore the network effect at your peril.

Kick the Tires

Also interesting was “Real Life Load Tests”. The GTalk team deployed to Orkut and GMail weeks before actually turning on the UI for the features to be able to monitor the load. These are the practices that make Bill’s recent observation on HA systems possible:

An interesting takeaway is that it’s clearly possible to re-architect data storage on super-busy production systems seemingly no matter where you start from.

For the rest of bullets see the HA blog post.

Crumbling Illusion of Phone Network Security

April 11th, 2007

Tetanus Factory

It’s been “big news” that Twitter and Jott are vulnerable to CallerID spoofing. Can’t speak for the Jott kids, but I know this isn’t news to Twitter kids, given that Blaine and Rabble were demonstrating CallerID spoofing (and related techniques) over Asterisk several years ago.

This is one of those security problems that everyone knows about and quietly agrees not to speak too loudly about because alternatives like Nitesh proposes are usability disasters.

The carriers have designed their networks around the assumption of monopoly practices and zero information sharing, and it means they’re slow and make dumb mistakes. (I always think of it as a nice parallel to the current U.S. administration — organizational tactics constrain your imagination and your coping techniques)

Jabber, btw doesn’t have this problem.

Photo by thristian

Coding a Twitter killbot

March 9th, 2007

j =, pass)

j.received_messages.each do |mesg|
    if mesg.body.match(/austin|sxsw/i)
        sender = mesg.elements['//screen_name'].text
        j.deliver('', "leave #{sender}")

Though truth be told, it isn’t running.