July 10th, 2012

oldtweets is a search engine for the first year of Twitter.

A bunch of folks asked about the how. The Twitter API provides a method for fetching a tweet by ID. So to build an index of the first year of Twitter you need call the api for each ID in the range of IDs 1-20,000,000. 20 million API calls at the rate of 150 calls per hour. Or roughly 15 years of elapsed API time to index year one.

It also helps to know that Twitter is, and has always been, a MySQL shop, and that in the early days there was a theory about scaling databases by using large auto-increment offsets. (I don’t remember what the logic of that was) That started about 6 months in, was turned off for a while, and periodically drifted. So good news the 20 million ID space is very sparse, which significantly cuts down on the elapsed API time. You just need to send tracers into the space to map it.

From there it’s just a question of patience.

The whole things runs on a very small EC2 instance, and it’s on this week’s todo list to get the index running under Upstart, but it hasn’t happened yet. So if it goes away….


I think our history is what makes us human, and the push to ephemerality and disposability “as a feature” is misguided. And a key piece of our personal histories is becoming “the story we want to remember”, aka what we’ve shared. I just wanted my old tweets, as a side effect I got all of them.

Providing an interface to the whole corpus was motivated by the desire that folks would investigate where the social norms arose, exactly like Rabble’s @-reply investigation.

Year 1

I thought year one was a meaningful symbol. It maps to the time when we were figuring out how to use Twitter, and maps to the time when I felt like the service was working best for me and mine as an “ambient intimacy” service.

Additionally after SxSW 2007 the rate of tweeting increased significantly, making the brute force approach even slower.

Tagged: Uncategorized

A new (to me) spam type: the “let me write you a blog post”

June 26th, 2012

Something is going on here, but I’m not sure I can figure out what

from: Sarah Thompson sarahjthompson5@gmail.com

An idea for a blog post: Science and Design Behind Music Production

Hi ,

I’m getting in touch with you because I’m interested in writing an article for your blog. I came across your blog post laughingmeme.org while writing for a website on music production. During my research, I’ve found an increasing focus in terms of design as the tools and technology available today improve our ability to customize how create music and collaborate as musicians

Please let me know if you’d be interested in an article this topic. Thanks, and I look forward to hearing from you soon.



And then a bit earlier

from: Brianna Meiers briannameiers8@gmail.com

An idea for a blog post: Aesthetics of Learning

Hi Kellan,

I found the information on your blog http://laughingmeme.org/ insightful as I was scouring the web for research on historical topics that are relevant to issues in higher education today. One of the biggest hurdles in completing an enriching and useful college degree (especially with different organizations such as online schools bombarding students with options) has been the lack of innovation in learning today. I believe that for students today to truly retain useful knowledge, it’s important to design curriculums keeping in mind aesthetics and learning theory.

I’d love to write a post for you that perhaps blends this topic with something deeper you are interested in for your blog. What do you think? Thanks, and I really look forward to hearing from you.



Tagged: Uncategorized

The ambivalence of backing a successful Kickstarter project

June 26th, 2012

It’s cool to see a project I backed on Kickstarter, Ai Weiwei: Never Sorry up on the Apple trailers page. But also deeply ambivalent as the promised digital download never arrived, which leaves me feeling used and kind of cheated.

Tagged: Uncategorized

Random Walk as Cargo Cult

June 25th, 2012

One of the strangest genres of tech blog post is the:

“we abandoned our old clearly suboptimal solution, and we have a new solution which we’re excited about, and we’ll assert our reasoning as to why this solution is clearly awesome, even though we don’t yet have any data on how it performs, or any discussion of why our decision making process for picking this solution was different then the decision making process for picking the last solution.”

Tagged: Uncategorized

Move fast, break stuff

June 24th, 2012

You know that argument you’ve gotten into about whether pushing out an experimental feature is a good idea that goes something like, “If we offer this we can never take it back, so let’s be careful”? It’s akin to the argument, “It needs to be perfect, because first impressions are everything.” They’re reasonable fears that often morph into stasis, because we forget that change should feel risky and scary, and if it isn’t then we’re moving to slow.

I imagine if you were going to roll out your own currency system that people exchanged real dollars for that would be an example of something you need to make sure you got right because you could never take it back.

One of the reasons Facebook is where it is is because they don’t subscribe to that mental trap. Refreshing to see.

Tagged: Uncategorized

The Dagger and the Coin by Daniel Abraham are solid

June 23rd, 2012

Daniel Abraham’s fantasy trilogy “The Dagger and the Coin”: The Dragon’s Path, The King’s Blood, and forthcoming “The Poisoned Sword” are the most satisfying fantasy I’ve read in recent memory.

It’s familiar and unexpected, engaging and unchallenging, swords and sorcery and fallen empires, and deep history, but done well. Economics, and race dynamics, and religion, and archaeology all play roles. Solid multi viewpoint story telling add some novelty to the heroes journeys.

Also a quite reasonable 12-14 months between new books, the 2nd book lived up the promise of the first, and they’re available on Kindle. Really everything you could ask for if you’re looking for a good fantasy novel. Recommended.

I also tried reading some of this “Poet Books” (officially called the “Long Price Quartet”). They dragged considerably more, but a couple of interesting ideas in there.

“Where the best and brightest come to shine…”

April 2nd, 2012

Made in New York from NASDAQ on Vimeo.

The two hour interview I did for this piece was a nice culmination of the last 4 years I’ve spent thinking about what makes the New York tech scene different and special. In retrospect editing it down to 7 seconds was even better. In the words of the mayor, “NYC is where the best and brightest come to shine”.

Tagged: Uncategorized

Room for disruption

April 2nd, 2012

This February, Amazon again asserted its influence when it pulled nearly 5,000 titles by distributor Independent Publishers Group from its Kindle e-book store. Amazon wanted better terms, and IPG said “no.” Amazon.com trying to wring deep discounts from publishers

A thought, as Amazon’s and the publisers’ relationship gets more bruising seems like there is room to be the company listing all the books in print, and where they’re available for purchase. Could be classically disruptive business (in that it will always be tiny, but with the chance to implode something larger).

Aggregating information from a diverse set of sources all over the world is the sort of thing which even 10 years ago was hard, and is now straightforward as long as you have the right motivational loops in place.

Tagged: Uncategorized

Open 24 hours

March 31st, 2012

Adjusting to NYC Life

We moved (back) to New York July 2008. We drove, though I only made it half way across the country before I had to have Jasmine drop me off at Austin-Bergstrom International Airport because I suddenly had to get back to work.

That first year we were living in a traditional Williamsburg apartment: a zoned light industrial, lightly converted for human habitation loft space (above the Roebling Tea Room in that big brick building). And it was a hot, and sticky Summer. Not like this last Summer. It was a lie awake at night sweating, slightly fevered thoughts blending with the sounds of a city that wasn’t sleeping coming through the open windows. It wasn’t an easy transition that Summer. It was amazing, but not easy. And in circumstances like that you find comfort in odd places.

And one of those places was my laundromat. Really. You see it felt like New York. It was big. It was full of a people who weren’t like me. It was open 24 hours. This was a laundromat for New York. And sitting in the laundromat, at 3am, in shorts and sandals, watching a world swirl by I felt like a New Yorker. I felt like I’d broken free from the normal rules, and I was living a new negotiated existence in a city of possibility.

So odd as it is, I’m going to miss my laundromat. We don’t live around the corner anymore. And the wave of gentrification of which we’re a part has swept us along and left us with our own washer/dryer. The bagel store on the corner is gone as well, rumored to be a Starbucks, so there was never going to be another hot sticky night of laundry and bialy anyway. And I’ll miss it.

Tagged: Uncategorized

because as soon as we started [pinning] for you

March 20th, 2012

“Updated my PHP bindings to work with unofficial Pinterest API v2. And then @curlyjazz and I made a thing: @omgrobot10 https://github.com/kellan/pinterest.api.php” – @kellan

or rather

“… because as soon as we started [pinning] for you it really became our civilization.” – omgrobot10

Saturday Jasmine and I put together omgrobot10, which is really the “Hello, world” of a particular branch of New Aesthetic. It motivated/exercised some code, and it amused us. But something unexpected also fell out of it.

“culture is the interference of two programs being executed in information proximity”

omgrobot10 is continuously repeating a very simple rule set. Spontaneity is added entirely by it’s interactions with people. As expected. What was unexpected was how it would interact with other actors also continuously repeating a very simple rule set. And no, that isn’t a dig at Pinterest users.


You see what I found almost immediately was, “TheDress”.

omgrobot10  omgrobot  on Pinterest 1

“OMG!!! I love this dress! I bought it for a cruise last year, and got many many compliments.”

In the first 24 hours omgrobot10 re-pinned TheDress from 6 different accounts, all for whom it was a fond memory of last year’s cruise. Since then omgrobot10 has re-pinned TheDress 17 times from 17 difference accounts.

What’s interesting is the TheDress also works fine in the noisy and spontaneous space that is the “Pinterest > Everything” page largely populated by humans. But it’s simple expectations clash with the simple expectations of omgrobot10.

There is also DeskOrganizer, who follows a similar agenda as TheDress, a tireless activist for a particular product that it has become fixated with it. But DeskOrganizer seems more content to be patient, play the long game, TheDress wants to get it’s message out now, at least ombrobot10 has only encountered it 4 times.

But both of them have formed a mutually beneficial relationship with omgrobot10. The interference of their programs creating a robotic culture of what looks, from the outside human perspective (and therefore deeply uninformed) , as mutual aid.

See also, the ethnography of robots.

Tagged: Uncategorized

Austin plans, so far (SxSW 2012)

March 5th, 2012


I’m going to be at Kick, Saturday morning, 9:30am

I’m going to be at Frank, at 10am sharp Sunday morning as they don’t take brunch reservations

I’ll be at Made in NY, at the Cedar Room Sunday night, because Etsy’s presence at SxSW this year is low-key/unofficial like.

I’ll be at “New Aesthetic” 9:30 Monday morning, because “Maps, Books, and Spimes” was one of my all time favorite SxSW talks, and this looks like it will be at least as much fun. And I want to see a drone.

I’ll be at World Changing 2.0 at 12:30 on Tuesday because Cameron Sinclair’s “Open Architecture Challenge: Revisioning Decommisioned Military Facilities” was fucking awesome, and I’m looking forward to more.

I’ll be at Bruce Sterling 5pm Tuesday, because I once taught a class on global politics using the text of “Islands in the Net”, and it’s Bruce Sterling!

I’ll be selling gold on black “Just Ship” shirts, with any luck using a prototype Etsy mobile payment system. So if you’ve wanted a “Just Ship” shirt, this is your opportunity.

I might make a pilgrimage to Opal Divine’s as the thing I missed most bitterly at SxSW last year was the indie “Data Drinks”, and I strongly encourage people to organize and publicize “[important technical topic here] Drinks”, rather then trying to communicate then on a panel. (e.g. y’all “Decentralized Web” and “SSO” folks ought to plan something, somewhere in the afternoon with “Mexican Martinis”)

I’m hoping to have some success booking small to medium sized group meals, mixed results so far.

Beyond that I’ll be around hoping to meet great people.

What are your plans?

Tagged: Uncategorized

Remembrance Engines

February 11th, 2012

That history is written by the winners is at least in part an unfortunate artifact of poor storage and retrieval technologies, and a poor backup regime.

I love Photojojo’s TimeCapsule. TwitShift is delightful. And I’m sure in time I’ll come to feel the same way about TimeHop. Like many people of my generation I’ve spent most of my “adult” life bouncing between jobs, cities, countries, etc. At one point Jasmine and I calculated that between the two of us we’d moved 14 times in the previous 10 years. Outsourcing memory to silicon in a life that largely lacks useful time signifiers is immensely helpful. But I’m also a bit uncomfortable with how much I like these services. Besides a certain puritan work ethic guilt, they’re deeply narcissistic.

In particular two features bother me:

  • they present the world devoid of the people I was sharing it with at the time
  • they’re largely constrained to minor modes of participation, e.g. tweets and check-ins.

Rod tweeted out a link yesterday to his 8 years old blog post about Flickr launching, it’s amazing:

It has the standard Friendster-esque friend-browsing capabilities, plus Tribe’s, erm, tribes. So far, so orkut. But what’s super-neato is what’s on top.

First off, you can gradate your friendships. The levels of Acquaintance, Friend, Best Buddy and Soulmate are all available to make the politics of friendship even more precarious. (There’s also a planned-for-the-future level called Enemy which is as-yet unattainable)

Then there’s the funky flash chat-app: an multi-window IRC-lite affair with an emphasis on picture sharing.

This is not the Flickr most people think of, or even that most people remember existed. The Flickr of today is many iterations of lessons learned later, and perfectly binary nature of the digital world largely hides that honing effort, except in the remembrances of those who were there 8 years ago.

I’d like that as a service. Send me what my friends were writing 8 years ago today-ish. Their long form work. We could start with blogs, tease out books and papers later, eventually troll The Archive for projects they were launching.

Over coffee this morning I thought about how you’d do it just for blogs. And I decided I wasn’t going to try to build it this morning as it clearly was going to take longer then 20 minutes.

My first thought was using RSS. It has fairly well understood semantics for permalink and publish date extraction. But you’d really only be able to start at this moment in time, not have the last decade plus of historical record.

You could build a time machine and go back and make sure either good semantic markup/microformats got adopted or that the RSS pagination specs became the norm. But if you’ve got a time machine fixing those issues aren’t even on my top 10 list for you.

Continuing on the time machine line of thought, one of the old aggregators, like Bloglines could offer the service assuming IAC still has the archives around. (oh wait, looks like they sold Bloglines to MerchantCircle, huh)

Not sure how interesting/practical the above are.

So then I started thinking about all the folks over the years who built blog crawlers with logic for doing date and permalink extraction, and even summarization. Some quick Googling turned up nothing useful in terms of documented techniques or code. But I figure someone has just got to have some code lying around, yes?

Could you please build this? (or barring that, send pointers on papers/code/etc)


photo by sweetfineday

Tagged: Uncategorized

Pour Overs

January 29th, 2012

In our 2012, “Most Williamsburg” contest, Mike White has taken an early lead with his bacon infused pour over technique.

“This is how I use my Chemex filters. ”

See also, pourover mac and cheese, and The Awl’s “get off my lawn” defense of NYC’s traditionally terrible coffee.

Tagged: Uncategorized

“They can even stop when they have a map that is just good enough.”

January 28th, 2012

There’s a hilarious answer to the question Why are software development task estimations regularly off by a factor of 2-3? floating around now. Hilarious, deeply knowing, but not terrible accurate or actionable.

But buried in the comments was this gem from Neil K, which I’m going to quote, in it’s entirety, for truth (Neil has, after all, seen more sausage getting made at more of the places whose tools you use then nearly anyone):

This is really good, but if I can offer a suggestion — the analogy could be even more apt with a slight shift. Currently it only shows how people go wrong when they develop software in a naive way — by starting at the beginning, and coding each step to final quality, in order. The story, as written now, makes it look like writing software is just an impossible slog and nobody can do it.

The truth is, software is research. It’s a matter of discovering the solution, not plodding through it. This is implicit in your story, because they keep encountering unexpected problems. But let’s make it explicit.

Imagine, instead, that our intrepid pair is charged with mapping the coastline of California from SF to LA. Mapping is more like software development because it involves discovery, and getting things right at multiple “points”.

The naive mappers start off from SF and it all fails exactly as you outline. A more clever pair of mappers instead decide to hire a boat, and map just a few points on the coastline precisely, just to get a rough estimate, and to survey the coastline for the tricky places. Then they know where to apply their efforts — an intern can be hired to pace out some of the easy bits, and a team of well-equipped hikers can be brought in to handle the hard parts.

They can even stop when they have a map that is just good enough.

Conflicted on: per country censorship, responsibility, and the cost of doing business

January 28th, 2012

Regarding Twitter’s recent changes:

Starting today, we give ourselves the ability to reactively withhold content from users in a specific country — while keeping it available in the rest of the world. We have also built in a way to communicate transparently to users when content is withheld, and why.

I mostly agree with Simon’s excellent post on his experience of adding this feature Flickr, and I18N in the broader Yahoo context, summarized as:

(1) Online companies have (and should have) little to no power to alter a country’s laws. (2) Online companies have an obligation to follow the law in places where they do business. (3) The revolution will not be Twitter-fied (4) The internet transcends borders; most laws do not. (5) Assumptions of malice are generally misguided, moreso in the face of opposing evidence

I’m also deeply appreciative that he took the time to write it.

I also mostly agree with Blaine’s related assertion

Twitter is a corporation, not a radical or even progressive platform. The internet, though, is.

There’s a topic I’m deeply conflicted about in there, and I’m not even sure what it is.

I’m not remotely conflicted that the Internet’s unthinking freak out is counter productive and poorly informed. (probably due to so much of what passes for dialogue these days being conducted in 140 characters and proxy gossip rags)

This however:

Anyone who followed or engaged with the recent SOPA/PIPA protest hopefully left it with an appreciation for the dangers inherent in allowing companies to craft legislation

Creates for me a worrying abrogation of our rights as moral individuals working within a corporate system to improve the world.

When Google went into China refusing to adapt, and later pulled out because they refused to adapt to the limits of that market’s law was that them being the MPAA forcing their corporate vision on unsuspecting foreign citizens or was that them taking a principled stand for a better world? (and that they pulled out also because they got hacked, and because Baidu was probably kicking their ass somewhat undermines my point, but there are so few examples to point to at scale we’ll take this one. If you’ve got a better one, love to hear about it)

I was never comfortable with the line we walked at Flickr, it felt like we accepted too many restrictions as “the price of doing business”, but in a resource constrained tiny organization inside a much larger corporate behemoth there is a moral fig leaf to hide behind. Twitter has significantly less to hide behind. It’s a line that Etsy walks as well. We don’t, for example, allow you to sell vintage Nazi paraphernalia in any jurisdiction.

Additionally Simon, and Heather, and others who actively shaped our international policy were:

  1. Transparent with our community that we were doing it
  2. Transparent when content was being hidden and why (for country restrictions or DMCA or whatnot).[1]

If you’re going to do per country filtering, I can’t think of two better principles to put in place, and these are the principles that Twitter mentioned in their blog post.

Fundamentally this is one of the tensions in centralizing all your communications into a small handful of privately held, globally scaled, profit motivated corporations, and then centralizing people into semi-archaic, geo-political niche regulated markets we like to call countries.

(In the misattributed, misquoted, and then deeply munged words of Mark Twain, “I’m sorry, if I was clearer on what I believed, I would have written less.”)

  1. (aka the fuzzies, or the blacked out images, transparency around this wasn’t perfect, and didn’t arrive all at once. Iteration was needed.)
Tagged: Uncategorized

15 Minutes Spent Sleuthing the Case of the Missing Fab.com Engineering Team

December 29th, 2011

Stumbling onto this Fab.com 2011 retrospective, reminded me of something that’s been puzzling me. While I knew a bunch of great folks going to work for Fab.com, including Lori Dorn and Beth Ferreira, I didn’t know any of their geeks. Which is odd in a small tech scene like New York.

Turns out the geeks are True Sparrow Systems, they appear to be a boutique Rails shop, they’re based in Pune, India, and described as co-founders. And Jason Goldberg did his last company, socialmedian, with them as well.

Mystery solved.

Tagged: Uncategorized


December 4th, 2011

Last March Jasmine started talking to me about Pinterest in terms that made me pay attention. Since then I’ve been keeping an eye on it. A few weeks ago I decided to “cheat” my way into participation by writing a quick script to sync my Etsy favorites to a board.

Per request, I just pushed my PHP library for the unofficial Pinterest API up to Github. It’s a simple thing, more suitable for forking then using as is, and with some serious gaps where I simply haven’t had the time to find (assuming they exist) API methods for simulating the “pinmarklet” (as opposed to uploading), and for backing up pins.


Tagged: Uncategorized

Some *.es domains coming for renewal this week

November 23rd, 2011
  • inexplicabl.es
  • invisibleciti.es
  • otherwher.es
  • sometim.es (I was convinced to keep this one)
  • somewher.es
  • unmentionabl.es

Will you do something cool with it? Are we friends? They’re your for the asking.

Also there really ought to be a site that facilitates offering up domain names you aren’t using to friends with a bit more structure then Twitter/blogging.

Tagged: Uncategorized

Snooping on an iPhone app’s API usage (aka light weight reverse engineering)

November 12th, 2011

I don’t much like using sites that don’t offer APIs. (this is one of the reasons I don’t use Quora anymore, they’ve had plenty of time to offer an API in good faith)

But I do like playing with new sites and that means playing with sites that haven’t opened up an API, yet. It’s funny I keep getting into conversations with folks about, “We’re not sure how to open an API” or “We’ve got an API, but we think we need to rebuild it (and/or outsource maintenance of it to a 3rd party) before we make it public.” For those folks, please see flamework-api, an implementation of the Flickr “API framework” for how easy it can be, and then get over your timidity! But my favorite variant on this is, “Well we have an API for the iPhone app, but it isn’t ready yet for public.” Because that means there is an API, and you can use it.

Some quick notes on reverse engineering an API from an iPhone (mostly because I had to scrape all this back out of my lizard brain this week, and while it’s straightforward, there are a few step)

  1. Grab a proxy, I use Charles, but Burp works just as well and is free.
  2. On a wifi network, fire up the proxy and enable SSL proxying.
  3. Connect to a secure site with a browser via the proxy. (Charles will setup proxying for Firefox automatically)
  4. Using Firefox, drill into Page Info > Security > View Certificate > Details and export the CA certificate, which will be the intermediate proxy’s root cert. (e.g. with Charles it will be CharlesProxySSLProxying.cer) (YMMV with different proxies, and different browsers)
  5. Upload the root cert somewhere you can hit with mobile Safari from your phone
  6. Browse there with your phone, and add the cert.
  7. Throw your phone into “Airplane Mode”, re-enable wifi, connect to the same network your proxy is running on, in the “Choose a Network” menu, drill down and setup Manual proxying pointing at your laptops IP address, and port 8888 if you’re running Charles or 8080 if you’re running Burp (or whatever else your proxy is running at)
  8. Fire up the iPhone app with the API you’re interested in, and sit back and watch the bits flow.

(Reading over this again this morning I noticed I just assumed SSL, my data points suggest that’s a reasonable assumption both as SSL is easy, and as FBOAuth becomes more common, but obviously if the API you’re looking at isn’t running over SSL, skip steps 2-7.)

Tagged: Uncategorized

What apps do you use to explore your city?

September 15th, 2011

It wasn’t really until we were leaving London last week that I figured out that I should have installed the London Cycle app, a 3rd party app that scrapes up-to-the-minute information about availability of Barclays bikes (aka “Boris bikes”), and provides maps and routing.

Nearly every Londoner (or sometimes Londoner) we met had it installed, but none of the varied insipid “top 25 iPhone apps for London” style SEO bait blog posts mentioned anything like it.

So I’m curious, what applications do you know to navigate your city and get the most out of it? They don’t have to be city specific, I’m just curious what the apps you whip out when exploring, or getting to work, or meeting friends. What’s useful? (and where do you live)

We’re reaching a point where the our cityscapes are almost intractable without augmentation and I know that keying up proper augmentations will become a key piece of my next round of urban travel.

In New York I tend to use only a small set of city specific software (much to my chagrin), and mostly rely on big generic apps (in which NYC tends to be well represented)

To navigate New York I use:

Tagged: Uncategorized