Optimizing for Learning

November 8th, 2015

I gave two similar keynotes back to back in San Francisco in September. One at Github Universe, and one at the Code for America Summit. I felt like I was bit rusty after taking over two years off to focus on the inward facing work at Etsy, but folks enjoyed them.

This is the CfA version:

And some photos from the GHU one:

Tagged: Uncategorized

Five years, building a culture, and handing it off.

August 31st, 2015

After 5 years, the time has come. This is my last week as Etsy’s CTO.

Five years ago I joined a tiny site, with a struggling software product. I wasn’t sure how long I’d stay. I was coming off 5 intense years helping scale Flickr. But I had some theories about engineering I wanted to test, and Etsy had a nascent team with some great folks who were up for the challenge, though a number of the odds were stacked against us.

Theory 1: Nothing we “know” about software development should be assumed to be true.

Most of our tools, our mental models, and our practices are remnants of an era (possibly fictional) where software was written by solo practitioners, but modern software is a team sport.

Theory 2: Technology is the product of the culture that builds it.

Great technology is the product of a great culture. Culture gives us the ability to act in a loosely coupled way; it allows us to pursue a diversity of tactics. Uncertainty is the mind-killer and culture creates certainty in the face of the yawning shapeless void of possible solutions that is software engineering.

Culture is what you do, not what you say. It starts at the top. It affects everything. You have a choice about the culture you promote, not about the culture you have.

Theory 3: Software development should be thought of as a cycle of continual learning and improvement rather a progression from start to finish, or a search for correctness.

If you aren’t shipping, you aren’t learning. If it slows down shipping, it probably isn’t worth it. Maturity is knowing when to make the trade off and when not to.

I had some experience with this at Flickr, and I wanted to see how far you could scale it. My private bet was that we’d make it to 50 engineers before things broke down.

Theory 4: You build a culture of learning by optimizing globally not locally.

Your improvement, over time, as a team, with shared tools, practices and beliefs is more important than individual pockets of brilliance. And more satisfying.

Theory 5: If you want to build for the long term, the only guarantee is change.

Invest in your people and your ability to ask questions, not your current answers. Your current answers are wrong, or they will be soon.

Five years ago, continuous deployment was still a heretical idea. The idea you could do it with over 250 engineers was, to me at least, literally unimaginable.

Five years ago, it was crazy to discuss that monitoring, testing, debugging, QA, staged releases, game days, user research, and prototypes are all tools with the same goal, improving confidence, rather than separate disciplines handled by distinct teams.

Five years ago, focusing on detection and response vs prevention in order to achieve better, more reliable, more scalable, and more secure software was unprofessional.

Five years ago, suggesting that better software is written by a diverse team of kind people who care about each other was antithetical to our self-image as an industry.

Five years ago, trusting not only our designers and product managers to code and deploy to production, but trusting everyone in the company to deploy to production, trusting engineers to meet customers, and trusting teams on the edges to use data to make their own decisions went against everything we’d been taught about how we were supposed run a software development company.

Five years ago, rooms of people excitedly talking about their own contribution to a serious outage would have been a prelude to mass firings, rather than a path to profound learning.

And five years ago no one was experimenting in public about how to do this stuff, sharing their findings, and open sourcing code to support this way of working.

Five years ago, it would have seemed ludicrous to think a small team supporting a small site selling crafts could aspire to change how software is built and, in the process, cause us to rethink how the economy works.

These last 5 years testing the theories, making discoveries, growing a team who has changed how software is built, supporting the flowering of a tech scene here in New York — I’m incredibly proud and grateful to have been a part of it. And I’m proud to have been part of growing Etsy from a tiny, not-always-online website to a billion dollar public company that challenges conventional notions of whether you can be led by values, other than selfishness, and be a successful business at the same time.

The goal was always to build a culture — a culture of learning, a culture of generosity, a culture of values. Culture infects everyone. Successfully building a culture ensures when you leave you can hand your work off to people you trust and they will run the thing without you and make it better than you could have imagined.

And building a thing that can be handed off means … you eventually hand it off.

I took a sabbatical this summer. It was awesome. Burn out is a real and pressing phenomenon in our industry and I was certainly feeling its effects. I spent the summer doing things I enjoy: spending time with family, traveling, meeting new people, talking to smart folks about the future, and writing code. By the end of sabbatical, I felt better. I felt stronger and more present.

So I’m handing it off. To my team, you’re all awesome, I love you and I’ll miss you.

And now it’s time for me to find something new.

Tagged: Uncategorized

More thoughts on the Keyboardio Model 01

June 15th, 2015

I’m super excited about the Keyboardio Model 01, and glad to see their Kickstarer is live (go back it now!), even if I squirm to see my bits in the video. Given that I recorded nearly an hour of video about my experience using the keyboard, I tend to think the best bits of the interview didn’t make it into the their final draft.

Here are some more thoughts

I met Jesse nearly 15 years ago at Diesel Cafe in Somerville, MA when I walked up and introduced myself and asked him about his crazy keyboard setup. I’ve always preferred working out of coffee shops to hermetically sealed fortress of solitude like setups and ergonomics is one of those things that gets compromised. Jesse was already bridging the gap, but 15 years ago the results weren’t pretty. (sort of Snow Crash gargoyle-ish)

Most popular ergonomic keyboard designs are worse then useless, and the ones that aren’t seem to assume you’re very stationary spending 8-10 hours a day in your designated parking space (probably with a fancy chair, desk, dual monitor setup, etc) — they’re big, heavy, clunky things.

And at the end of the day they’re still designed for typing prose.

The Model 01 is different. It’s small enough that I can throw it in my low profile Osprey day pack which is my standard navigating the crowded streets of New York bag, along with a laptop, a book, and a notebook. And it’s lovely from the smooth milled maple, the custom keycaps, the rainbow backlit keys, and the solid metal backing. (that said it isn’t light with all that solid maple and metal)

The Model 01 is a tool for a craft, the craft of programming. When I think about great craft people, I think about how their tools become extensions of their hands. You keep your tools sharp and well oiled, you invest the time into customizing your editor (except for the folks always flitting to the latest and greatest which feels like the opposite of craft), but a keyboard that is a sharp, well designed tools, that’s designed to be customized and repaired, and also takes into account how hands actually work is something new.

I’m sure folks will get value from the Model 01 for tasks other programming. But it’s hard to overstate how poorly thought out the current offerings of keyboards are for programming, with our statistically unusual range of characters, forcing a ton of stretching and reaching, the activities that cause RSI.

I’ve been using the keyboard for about 2 weeks. I’m definitely not entirely comfortable with it yet. For some reason my right hand slips left and I have a habit of hitting the Return key instead of H meaning my sentences look like t
is wit
weird spaces in t

I do love the functions keys, the no reach navigation and bracket keys. The delete and space keys come very natural. Still getting used to reaching for the shift keys.

Looking forward to having time to do immersion as I’m right now flipping back and forth between the Model 01 and my laptop keyboard. It’s been a while since I’ve learned a new layout, so many of the alternative layouts like Dvorak promise typing speed increases which is one interesting. I’m looking for typing sustainability not typing speed.

And despite threatening to, I haven’t hooked the butterfly programmable button up to deploy … yet.

Tagged: Uncategorized

“Engineering capacity for an economic revolution”

April 19th, 2015

“Engineering capacity for an economic revolution”

That’s what my LinkedIn profile says I’m up to if you check it. That means a lot of things. One of the things it means is I believe that software is important, doing software well is important, and when we invest at getting good at human centric, thoughtful, sustainable software development we’re increasing capacity in the system, and we should share what we learn. Internally we call this a spirt of generosity.

I think at Etsy we’re currently doing the best work we’ve ever done in digging into the how and why of software development and leadership, and when I look across the team I see an amazing pool of folks who someday (hopefully far in the future) will flow out and change how this industry works.

But this week I’ve been feeling reflective, and it got me thinking about all the folks who are already out there on that journey.

A totally non-comprehensive list of companies where folks who used to work at Etsy are now either founders or senior engineering leaders (managerial or otherwise) includes:

  • Abacus
  • AltspaceVR
  • First Look
  • floship
  • Handy
  • Hinge
  • Julia Computing
  • Justworks
  • liwwa
  • Radico
  • Ringly
  • Signal Scienes
  • SoundCloud
  • Stripe
  • Tumblr

For a small company with a low turnover rate that feels like a high impact list, and I’m proud of the capacity we’ve added to the system.

If I missed some feel free to add them in the comments.

Tagged: Uncategorized

A little bit on the technical track

January 11th, 2015

I’ve got a post I want to write in response to Dan’s post on the technical track which I think does a decent job of laying out the issues, but doesn’t cover all the challenges and I think is poor on the solutions. Barring significantly more focus and free time dropping from the sky, this clip from a recent fireside chat I did has my recent, if somewhat opaque, thinking on the topic and is the closest I’m going to get to a blog post. 2 minutes and 25 seconds.

(also this is about 90 minutes into the fireside chat sitting on really uncomfortable chairs, going to get those replaced before the next extended QA session we do)

Tagged: Uncategorized

2014 End of Year Booklist

December 29th, 2014

Read Rick’s, Rachel’s and Diana’s year end book list, and got inspired to write up my own.

2014 was a year of extenuating circumstances, the most obvious being the arrival of Anwen 9 hours before it began, but numerous others large and small abound. Reading this year was centered around escape, even oblivion, in short doses, and re-reading featured prominently.

(also I finally forwent the urge to build a Kindle exporter as part of this process and just used clippings.io which did the job well, though seems to have hit a problem after downloading nearly 2400 highlights)

Additionally I’ve probably missed a couple which I didn’t highlight anything in (or read exclusively in dead tree)

  1. The Old Ways: A Journey on Foot by Robert Macfarlane

    Pilgrim paths, green roads, drove roads, corpse roads, trods, leys, dykes, drongs, sarns, snickets – say the names of paths out loud and at speed and they become a poem or rite – holloways, bostles, shutes, driftways, lichways, ridings, halterpaths, cartways, carneys, causeways, herepaths.

    In Ireland there are hundreds of miles of famine roads, built by the starving during the 1840s to connect nothing with nothing in return for little, unregistered on Ordnance Survey base maps. In the Netherlands there are doodwegen and spookwegen – death roads and ghost roads – which converge on medieval cemeteries. Spain has not only a vast and operational network of cañada, or drove roads, but also thousands of miles of the Camino de Santiago, the pilgrim routes that lead to the shrine of Santiago de Compostela. In Scotland there are clachan and rathad – cairned paths and shieling paths – and in Japan the slender farm tracks that the poet Bash? followed in 1689 when writing his Narrow Road to the Far North

  2. The Cuckoo’s Calling (Cormoran Strike Book 1), Robert Galbraith (aka JK Rowling)

    displayed a multitudinous mess of life’s unnecessities.

  3. The Silkworm (Cormoran Strike Book 2), Robert Galbraight (aka JK Rowling)

  4. Startide Rising (Uplift Trilogy Book 2), David Brin

  5. The Uplift War (Uplift Trilogy Book 3), David Brin

  6. Tamsin, Peter S. Beagle

    She’s got a daughter, too, my cousin Barbara, and we were always supposed to be lifelong buddies, but the first time we met, when we were maybe two years old, we tried to beat each other to death with our toy fire engines, and it’s been downhill from there.

  7. The Peripheral, William Gibson

    “Terrorism,” said the rental. “We prefer not to use that term,” said Lowbeer, studying her candle flame with something that looked to Netherton to be regret, “if only because terror should remain the sole prerogative of the state.”

  8. Ancillary Justice (Imperial Radch), Ann Leckie

  9. Ancillary Sword (Imperial Radch), Ann Leckie

  10. Country Driving: A Chinese Road Trip, Peter Hessler

  11. The Slow Regard of Silent Things (The Kingkiller Chronicle, Book 1), Patrick Rothfuss

    You might not want to buy this book.

  12. Memory (Vorkosigan Saga Book 10), Lois McMaster Bujold

    There’s no place like home. I didn’t say there was nothing better. I just said there was nothing like it.

  13. The Winter Long: October Daye #8, Seanan McGuire

    “The day I get a vacation is the day the world ends,”

  14. Chimes at Midnight: Book Seven of Toby Daye (October Daye), Seanan McGuire

    “A knight of the land Courts asking a Duchess of the Undersea to save a King of Cats,”

  15. Fool’s Assassin: Book One of the Fitz and the Fool Trilogy, Robin Hobb

    “I will always take your part, Bee. Right or wrong. That is why you must always take care to be right, lest you make your father a fool.

  16. The Girl Who Circumnavigated Fairyland in a Ship of Her Own Making, Catherynne M. Valente

    I’m afraid the whole thing moves around according to the needs of narrative.

  17. The King’s Blood (The Dagger and Coin Book 2), Daniel Abraham

    The day you throw me in a ditch and take control of the company?” “Still not today.

  18. The Dragon’s Path (The Dagger and Coin Book 1), Daniel Abraham

    if everyone benefits, you’ve overlooked something.

  19. The Widow’s House (The Dagger and Coin Book 4), Daniel Abraham

    “The world is burning. Anything that doesn’t end in ashes is worth doing.

  20. Captain Vorpatril’s Alliance (Vorkosigan Saga Book 14), Lois McMaster Bujold

    “The ones still inside, you’ll want to commend. The ones outside, those are the ones I’d promote . . .”

  21. Komarr (Vorkosigan Saga Book 11), Lois McMaster Bujold

    He’s not even a mad scientist. He’s merely a very upset engineer.

  22. A Civil Campaign (Vorkosigan Saga Book 12), Lois McMaster Bujold

    You won’t regret this seemed a much too optimistic statement to add.

    “What about acts of ineptitude?” “A gray area, and don’t tell me you haven’t lived in that twilight before.” “Most of my life, sir. Not that I haven’t leaped up into the blinding light of competence now and then. It’s sustaining the altitude that defeats me.”

  23. Rogues, George R.R. Martin and Gardner Dozois

    the very image of a person with just the average amount to hide.

  24. The Rhesus Chart (Laundry Files Book 5), Charles Stross

    “Hurry up! We can’t stop here; this is black pudding country!”

    The five stages of bureaucratic grieving are: denial, anger, committee meetings, scapegoating, and cover-up.

  25. Farthing (Small Change), Jo Walton

  26. The Wise Man’s Fear (The Kingkiller Chronicle, Book 2), Patrick Rothfuss

    “It’s the questions we can’t answer that teach us the most.

    “Maps don’t just have outside edges. They have inside edges. Holes.

    It was the same scolding any child receives. Stay out of the neighbor’s garden. Don’t tease the Bentons’ sheep. Don’t play tag among the thousand spinning knives of your people’s sacred tree.

  27. The Name of the Wind: The Kingkiller Chronicle: Day One, Patrick Rothfuss

    “I thought you would be older.” “I am,”

  28. Be Slightly Evil: A Playbook for Sociopaths (Ribbonfarm Roughs 1), Venkatesh Rao

    We are low-entropy creatures trying hopelessly to swim upstream in a universe that’s gradually winding down towards a maximum-entropy heat death.

  29. My Real Children, Jo Walton

    She had never felt older than those years when the children were small and so demanding of her attention. She had felt it a new lease on youth when they were grown and gone,

    In November of 1966 there was a flood in Florence, killing six people and damaging some property. Fortunately the weather computers had predicted it well in advance, so most people had evacuated and most works of art were moved to safety. Some frescos were damaged. Pat wrote articles about their restoration and sat on a committee to raise money for it.

    Just the cold contingent universe where things happened for random reasons nobody could understand.

  30. Capital in the Twenty-First Century, Thomas Piketty

    Expert analysis will never put an end to the violent political conflict that inequality inevitably instigates.

  31. Words of Radiance (The Stormlight Archive, Book 2), Brandon Sanderson

    “If I toss something upward, it comes back down.” “Except when it doesn’t.” “It’s a law.” “No,” Syl said, looking upward. “It’s more like . . . more like an agreement among friends.

  32. The Way of Kings (The Stormlight Archive, Book 1), Brandon Sanderson

    Shallan moved her eyes down to the bottom of the page where—separated by a line—the undertext was written in a small, cramped script. Most books dictated by men had an undertext, notes added by the woman or ardent who scribed the book. By unspoken agreement, the undertext was never shared out loud. Here, a wife would sometimes clarify—or even contradict—the account of her husband. The only way to preserve such honesty for future scholars was to maintain the sanctity and secrecy of the writing.

    When he passed, the timid grass pulled back, but the fingermoss was bolder. The clumps would only pull into their shells if he tapped the rock near them.

  33. Moneyball, Michael Lewis

  34. Liar’s Poker (Norton Paperback), Michael Lewis

    Why did investment banking pay so many people with so little experience so much money?

    By the mid-eighties, however, all things black, yellow, and female have disappeared from the photographs. There isn’t a trace of anything but white men in the annual reports.

  35. Healthy Sleep Habits, Happy Child, Marc Weissbluth Md

  36. Flash Boys: A Wall Street Revolt, Michael Lewis

    Every systemic market injustice arose from some loophole in a regulation created to correct some prior injustice.

  37. The Hard Thing About Hard Things: Building a Business When There Are No Easy Answers, Ben Horowitz

    “When we drove off the cliff, we left no skid marks.

    The people who stay will care deeply about how you treat their colleagues.

    Yes, of course. The reason that people leave our service and don’t come back is that we have not been sending them enough spam. That makes total sense to me, too.

    Shock is a great mechanism for behavioral change.

    Hiring scalable execs too early is a bad mistake. There is no such thing as a great executive. There is only a great executive for a specific company at a specific point in time.

  38. Half-Off Ragnarok: Book Three of InCryptid, Seanan McGuire

    Nature’s not talking, possibly because even nature realizes that giving perfect camouflage to apex predators is sort of a dick move.

    (Australia. The only continent designed with a difficulty rating of “ha ha fuck you no.”)

  39. Turn the Ship Around!: A True Story of Turning Followers into Leaders, L. David Marquet

    Most of what we study, learn, and practice in terms of leadership today follows this leader-follower structure. This model has been with us for a long time. It is pervasive. It is the structure depicted in The Iliad, in Beowulf, and in other Western epics. But this model developed during a period when mankind’s primary work was physical. Consequently, it’s optimized for extracting physical work from humans. Those who take orders usually run at half speed, underutilizing their imagination and initiative. While this doesn’t matter much for rowing a trireme, it’s everything for operating a nuclear-powered submarine.

  40. The Privilege of the Sword (The World of Riverside Book 2), Ellen Kushner

    “I do not make the rules,” he said creamily. “This annoys me, and so I comfort myself by breaking them.

    Getting the long sword into the soft scabbard was a bit like getting a bootie onto a baby’s foot: neither was very interested in helping,

    And suddenly, as the night air turned cold and the day sky burned a bright and gallant blue, the world was full of apples. The air smelt of them, sharp and crisp, then underlaid with the sweet rot of groundfall.

    Serious sword-practice made me forget to think in words, so that I didn’t always understand when people spoke to me.

  41. What’s Going on in There?: How the Brain and Mind Develop in the First Five Years of Life, Lise Eliot

  42. Influx, Daniel Suarez

    “Mankind was on the moon in the 1960s, Jon. That was half a century ago. Do you really think the pinnacle of innovation since that time is Facebook? They designed the Saturn V rocket with slide rules.

  43. Abaddon’s Gate (The Expanse Book 3), James S.A. Corey

    She had a sudden vision of Jesus, who’d asked His disciples to keep doing this in remembrance of Him, watching her little congregation as they floated in microgravity and drank reconstituted grape beverage out of suction bulbs. It seemed to stretch the boundaries of what He’d meant by this.

  44. Caliban’s War (The Expanse Book 2), James S.A. Corey

    Owning your own racing ship wasn’t even wealth. It was like speciation.

  45. Leviathan Wakes (The Expanse Book 1), James S.A. Corey

  46. The Tyrant’s Law (The Dagger and Coin Book 3), Daniel Abraham

    “Job is we kill a goddess and save the world. Let’s not complicate it.”

    The citizens of the city wrapped themselves in coats and cloaks, scarves and mittens, until they all seemed part of a single unified race of the chilled.

    “Good that we have the powers of chaos and madness on our side sometimes.

  47. The Golem and the Jinni (P.S.), Helene Wecker

    Gradually the well-kept brownstones gave way to Dutch clapboard houses, their trellises heavy with roses.

    He supposed that eventually Matthew would grow older and lose interest, and take his place with the feral young men who slouched on the neighborhood stoops. Or—even worse—he’d become just another streetcar rider, dull-eyed and unprotesting.

  48. The Boy Kings: A Journey into the Heart of the Social Network, Katherine Losse

    Logging on to Facebook that first day, in retrospect, was the second, and to date the last, time that any technology has captured my imagination.

    “California is a place in which a boom mentality and a sense of Chekhovian loss meet in uneasy suspension; in which the mind is troubled by some buried but ineradicable suspicion that things better work here, because here, beneath the immense bleached sky, is where we run out of continent.

    While they abhorred the idea of being a wage slave, the young men of Silicon Valley were not trying to tear down the capitalist system. They were trying to become its new masters.

  49. Neptune’s Brood, Charles Stross

    Which means I need to brief you on the politics of mermaids.

  50. The Everything Store: Jeff Bezos and the Age of Amazon, Brad Stone

    “How do you plan to handle the narrative fallacy?”

    “When a company comes up with an idea, it’s a messy process. There’s no aha moment,”


remembered at least one book missing from above

Half a King, Joe Abercombie. YA but looking forward to the sequel

Tagged: Uncategorized

Annoyed by “The Peripheral”

November 11th, 2014

I enjoyed reading William Gibson’s latest, as I’ve enjoyed reading nearly everything he’s written, and as I anticipate enjoying seeing him and James Gleick speak tomorrow night.

However as I finished “The Peripheral” the overwhelming sensation I was left with was annoyed.

Below are spoilers.

There were a number of moments that bothered me, but in 3 quick bullets.

  1. The literature didn’t need another implacably evil bearded Arab villain. Even is he is half Swiss.

  2. The literature didn’t need another heroine whose happily-ever-after is signified by getting pregnant.

  3. It goes against everything I look for in a Gibson novel to basely indulge a maybe-the-future-will-take-pity-on-us-and-save-us fantasy. It left me with the sensation that the author is deeply scared, which while reasonable, isn’t what I expect from the typewriting steely eyed prophet of the future.

Lots of good moments, and interesting world building, again the near future work, especially the near future rural poverty in a cheap printing/fabbing future was interesting.

Tagged: Uncategorized

Pumpkin Pudding

October 26th, 2014

Back when my dad drove his big yellow truck, delivering food for the food bank, he sometimes dropped me off at the Live Oak Senior Center industrial kitchen.

And every October I remember peering over the edge into the bowl of an industrial stand mixer taller then I was at a truly giant vat of pumpkin pudding.

And while I’m telling myself this pumpkin flavored Chobani yogurt is kind of a pumpkin pudding, it really isn’t.

Tagged: Uncategorized

“Wolf” narrative considered harmful (also biologically unlikely)

September 2nd, 2014

Rands’ “The Wolf” post floated across my Twitters this morning. My immediate reaction was that it was misguided to the point of being potentially harmful. That was 16 long hours ago, but I’m going to see if I can pull the strands of that thought back together.

Michael mentions he’s had the fortune to work with several “wolves” in his career. I’ve worked with a bunch, including teams compromised of nothing but (I guess you’d call that a “pack”). The problem is the post plays into the fantastical narrative of the lone wolf. But here is the secret. No one is a lone wolf. Or at least not completely. “Wolf” is the story that many people want told about them, and that on good days they tell themselves, but in the dark of the soul, we all need help and have doubts. By celebrating the narrative construct rather then reality you perpetuate the dangerous tendency towards isolation that is inherent in this archetype. I’m playing the long game, and in my perspective sooner or later one of two things is going to happen to your isolated “wolf”: either they’re going to run into a hard enough problem that they’re going to need more help then they know how to ask for, or in their bid to maintain their Harvey Keitel-like facade they’ll eventually isolate themselves from new challenges and dead end their usefulness.

So let’s pull back the curtain a bit and understand the archetype:

  • they’re passionate about solving hard problems, and generally they have a few favorite problems they like to gnaw on. That deep personal investment in the solution is what pushed them to develop their skills past the point of their peers in the first place, and also where they find the personal clarity they need to opt out of the clarifying bureaucratic structure of the larger org. With that passion can come narrowness, often expressed as hyper rationality. Pro move: keep supplying hard problems off center of their primary focus.

  • they’ve developed a toolkit. A preferred language, set of scripts, investigative processes, checklist, whatever. And they come back to it over and over again. That disproportionate impact they have, it isn’t magic, its craftsmanship. They’ve mastered their tools to the point where the tools are an extension of their hands. They might see everything as a problem to be solved in Lisp, or via experimentation or machine learning or wireshark. They might have written their own text editor that lacks features what you and I consider non-negotiable, but doesn’t seem to impair their progress. The best of these folks literally generate tools, turning their own idiosyncratic but highly effective approach into tools that the wider org can use. Pro move: create a space and a culture that celebrates investing in tools. Everyone should hone their craft, and a few great tool builders makes everyone if not a 10x programmer, maybe an 8.5x.

  • they’re convinced they’re right. Or at least, they’re convinced often enough that they’re right that they appear convinced all the time. A banal outcome of this is hill climbing. The toxic outcome is stagnation with a hugely talented individual stuck because they don’t know how not to be right. You owe your high impact individuals to surround them with people who challenge them. If they’re really a “wolf” they’re likely to try to wriggle out of this. Pro move: they care about impact, so make the road to impact littered with smart challenging people.

  • the “well defined IC track for non-managers” is probably the project I’ve found most elusive and difficult to execute on. Rands gave me my first key insight into it late one night after much tequila: you have to make sure the role comes with super powers. Management comes with super powers by default, you have a team to force multiply you, and by convention, most corporations share information with managers at a greater rate then ICs. I haven’t found a one-size fits all approach to super powers, but a thing we’ve done is make sure that while the “senior eng org” (super powered engineers) have their own discussion space, they can also ease drop on manager discussions if they so choose.

If you’ve been maneuvered into calling someone “a wolf”, then I hypothesize you’ve let someone who could be transformational to your organization off the hook too easily. Imagine instead of a wolf you had a Intrinsically Motivated Full Stack Product Hacker.

Tagged: Uncategorized

AWS Redshift: “Missing newline: Unexpected character 0x4f found at location 3”

July 21st, 2014

I was trying to bulk load some data into Redshift this weekend using the copy command and I got this error message that no amount of Googling turned up a hit for. It was particularly strange given the hex value specified was for that exotic character the capital-O (as is OMG).

stl_load_errors listed the offending raw_line as being “.LZO”.

Anyway, this particular error message is documented here for the next person who forgets to toss lzop on to end of their copy command and tries to Google it

to be explicit you’ve got:

copy some_table from 's3://some_bucket/data.lzo' CREDENTIALS 'some_credentials' delimiter '\s' timeformat 'auto';

and you need

copy some_table from 's3://some_bucket/data.lzo' CREDENTIALS 'some_credentials' delimiter '\s' timeformat 'auto' lzop;

Tagged: Uncategorized

Communication Skills

May 17th, 2014

A great deal of negativity has been directed lately at the practice of hiring for culture fit. I find this troubling because its a central tenet of Etsy’s hiring. This pithy insight was shared with me yesterday by one of our senior most engineers, and I slept better last night.

“As time passes, I focus more and more on interviewing for communication skills over technical skills.

Even the most brilliant engineer can get off course. But a great communicator might move slower, but they will always be moving in the right direction.”

(actually I didn’t sleep better last night, but that was the baby’s fault, not anxiety about contributing to oppression.)

Tagged: Uncategorized

Shell trick for the perpetually distracted

May 12th, 2014

Manager time is the schedule that sub-divides your day into 15 minute blocks. It’s the internal clock in your head that knows when you’ve reached the end of a 30 minute meeting without checking the clock. It’s a trained form of ADHD, and it’s the opposite of maker time.

One of the things I find I need in manager time is a quick way to regain my context. I’m way behind on adopting a proper getting things done framework, but it wouldn’t help with this one thing I wanted to post about today anyway.

I keep two copies of most of the Etsy code base locally. The one I use constantly, which is for quick reference when a technical question comes up, and the one I use almost never which is where my various half hearted attempts to compress a quick hack into a 15 minute block live.

git gives me the tools to make it very painful when I get confused about which directory I’m in.

So in prep for hoping to finish up a quick hack during hack week (didn’t happen) I shaved a small attention starved yak. Namely changing the color of my terminal when I’m in workspace, which is the hacking directory. It’s a parallel to prompt hacks I’ve had for years that change based on host, but it uses osascript (command line AppleScript) for simplicity in scripting Terminal.app

set_bg () {
  osascript -e "tell application \"Terminal\" to set current settings of window 1 to settings set \"$1\""

function cd {
  builtin cd "$@"
  if [[ "$pwd/" =~ /workspace/ ]] ; then
    set_bg "Red Sands"
  elif [[ "$old_wd/" =~ /workspace/ ]] ; then
    set_bg "Basic"

Override cd, if we entering workspace, use the profile “Red Sands”, if we’re leaving workspace switch basic to “Basic”.

Tagged: Uncategorized

Explaining to the phone how to be a CTO

December 5th, 2013

The other morning I was sitting stuck in traffic … which an odd experience in New York, but for a few reasons I’ve actually been driving to work lately, profoundly surreal as a New Yorker, and not recommended.

Back to the story.

I was sitting stuck in traffic, and the last thing I’d seen before being stuck in traffic was the question, “How do I learn enough to become a good CTO?”

So below is the first blog post I ever tried to write via Siri. I thought about leaving in all the misidentified words, but the point wasn’t comedy. It still came out quite brief and almost certainly incomplete and incoherent:

It’s a leadership role, which means you need to develop a theory of leadership. It’s a spiritual role which means you need to be able to speak and write. It’s a role that fundamentally trades on credibility, so having been part of a team that did something that actually worked is key both for the reflected credibility but also because it’s important to have the certainty about what something working feels like.

It’s a role we learn by trial and error so you need to find a trusted community to compare notes with. You need to develop an empathy for the customer and interest in the business so whatever interdisciplinary pursuits do that for you is part of your training.

Finally being right more often then you’re wrong helps, but is less critical then you’d imagine.

It isn’t the 1st engineer either literally or metaphorically though that person is sometimes called the CTO. It isn’t the engineering manager in chief, though often you end up doing that job as well (but that’s a VPofE).

Software development scales non-linearly with clarity and excitement so your job is removing unnecessary ambiguity and drag. holding the narrative of what engineering means at YourCompany.

That’s how you learn to be a CTO.

And now that I’m typing this in to a textbox here on web, I’ll add, get good at managing your psychology

Tagged: Uncategorized

Slow Twitter

November 17th, 2013

My brother tweeted recently, “making sure I have a working implementation of Logo is part of setting up the nursery, right?”. I feel the same way about building the just the right photo sharing site and Twitter client. But I’m RESISTING. And I’m very proud of myself for it.

The Twitter client I want (and have wanted for years, but I want it more now) is at least fairly straightforward, and I’m hoping someone has either already built it or will get inspired. I think of it as “Slow Twitter”.

Twitter works great when you’re constantly checking it. Or when you don’t follow that many folks. But once you’re distracted you start missing stuff. In fact you only see the people who tweet constantly. And they’re often the most banal.

What I’m looking for is an app (and I include websites in that word, I’m old fashioned) that shows me who has tweeted since I lasted accessed it, with access to what they’ve tweeted, sorted in reverse order of the likelihood that they would have tweeted during the period of elapsed time aka probably sorted by ascending rolling average tweet frequency.


Tagged: Uncategorized

5 Approaches to Handling Bugs We Tried Before Our Current Approach

November 1st, 2013
  • don’t write bugs
  • once you write a piece of code you own it forever and your top priority is fixing the bugs in it
  • assign teams to own every section of the site, allow them to make sure no unapproved changes happen to those areas
  • have a bug fixing team
  • delete all code over a year old, switch to a new language, rewrite it
Tagged: Uncategorized

A quick knot in a few Flickr threads

October 19th, 2013

Paul Mison’s Maximum Viable Product feels like a more clever (if more cheeky) way to get at what I was trying to say with my notes on Flourishes and Minimal Competence (and Competence found), and threads the needle of recent 4up from Aaron and The Flickr We Lost from Dan.

I don’t have much conclusion as much as a sense there’s a thing there that I’m still trying to get my hands around.

The other piece I wonder at about this is the role of sophisticated testing and measurements and being the user.

I hear that engagement is way up in the wake of Flickr’s changes, and I know I’ve been gone a long time and things have probably changed, but I can’t help but be struck how unsophisticated we were at measuring things at Flickr — in part because we were a tiny team, and in part because we relied on our instincts. It makes me wonder what’s guiding the current work.

Tagged: Uncategorized

Paths to Production Confidence, Part 1 of N

August 8th, 2013

This started out as a short note explaining the unifying theory behind the Etsy development practices. Then it got out of hand (see also, Mark Twain’s “If I had more time…”). As such I’ve made it “Part 1 of N”, where “Parts 2 .. N” will cover the actual practices and how they relate to the philosophy.

Why Write This?

We believe in being as open as possible about how we develop and run Etsy — our current best theories, learnings, practices, and tools. Given that openness I often get questions about the hows and whys of given subsets of our engineering practice, e.g. “How do you do testing? How do you know you have enough testing?”. Or monitoring or deployment or what not.

At times, it can be tricky to do the questions and answers justice when taking them piecemeal, because underlying them all is a single shared philosophical premise, that isn’t necessarily obvious. And while I tend to be a pragmatist, favoring the rough and ready over the theoretical, without understanding the theory you can’t reason correctly about trade offs. This post is an attempt to surface that underlying philosophy and the practices it informs.

The theory is a theory of change, and the philosophy is about finding paths to move from risk to confidence.

A Theory of Change

Etsy is in the change business. As are, definitionally, all startups, nearly all businesses, and most human projects. We’re attempting to add new capacities to the world, and influence behavior around them. And we’re attempting to do it in an uncertain and complex[1] environment; we neither know the exact recipe for success, nor we do expect that recipe to stay the same over time. In fact, as a startup, we believe that our ability to respond to a changing environment is the key success factor for our engineering organization. It’s natural to read that sentence and think of change in terms of product changes, but more prosaic examples might include the ability to add new server capacity when the site slows down, or to replace a hard drive when one fails.

But change is risky. This is something most of us believe intuitively, and it’s worth examining the sources of risk in change.

Why is change risky?

As humans and practitioners why do we associate change with risk? Doing new things inherently contains the risk of doing the wrong thing.

We may for example have reached a ready state in our project. Through a combination of good luck and planning we find ourselves running a system that we understand sufficiently to keep running indefinitely, while a change would implicitly contain the risk of moving from a state of working to a state of not working. Steady state systems are so rare and so often illusory, it’s almost not worth mentioning except we’re fervently entranced with the possibility. Generally the illusion of steady state simply means the needed changes are non-linear, and often the cost of ignoring them will be high.

More practically, very few of us are employed to maintain systems in a stable environment. Even if we hold the pieces we control constant it’s unlikely that our systems will remain stable forever, at which point action is required. Still change is often associated with surprise in a system that hadn’t previously surprised us, and surprise is definitely risky.

The second reason change is risky has to do with how we think about causality, intention and culpability. While we can agree that the ability to choose not to make a change is an illusion, often fear leads people to approximate avoiding change, by avoiding making choices. If I personally avoid making changes to a system and instead wait for outside pressure to force change, or if I simply play the odds and hope that disastrous failure happens rarely enough that it won’t happen on my watch then I can avoid the personal risk of being labelled the root cause of failure. Forced choices avoid the necessity of stating a hypothesis before acting, thereby reducing significantly the opportunity to be personally wrong.

Software development is a complex system existing as it does at the intersection of people, systems, good intentions, confused and changing goals, and overly literal state machines. Past behavior isn’t always an indication of future behavior, and humans are terrible at reasoning about complex systems. As such we’re unlikely to know or have good visibility into whether we’ve reached a steady state and our hypotheses are likely to be wrong. In this uncertain and complex environment we initiate change only when the cost of not making a change overcomes the fear of making it. (e.g. “The server is down” or “You’ll be fired if this feature isn’t done by April 1st”)

As an industry this means though we’re in the change business, often we aren’t very good at it, and we avoid it out of fear.

Different groups attempt to address this tension by:

  • raising the cost of not making change (“you’ll be fired”)
  • distributing those costs broadly (this is one of the key functions bureaucracy and process serve)
  • gaining confidence by addressing the fear

We see Etsy’s engineering practices as spectrum of tools for increasing our confidence in our ability to make change.

Going back to the opening idea of this post, the attempt to answer a question like, “How much testing do you do?”, the answer becomes, “Enough to gain confidence. But testing is just one of the tools we use to gain confidence, so less then a strong testing shop might.” Similarly if someone asks, “How much monitoring is enough?”, the answer is, “We add monitoring until we feel like it gives us confidence, and we’re comfortable striking an 80/20 balance, particularly upfront, because we’re confident if we don’t have the balance right we have other ways of finding out.” In fact how many and how much confidence boosting techniques you need is situational, and depends on how risky your change is. Which speaks to another fundamental piece of our process, small and iterative changes.

Hopefully that starts to explain why, while I think our testing infrastructure (with its try-servers, “Bobs”, static analysis, integration tests, and quality metrics) is awesome, just telling you how we do testing isn’t necessarily going to be useful. Or perhaps it just speaks to my personal penchant for holistic post-modern explanations.

So given a theory(-ish) of risk, change and confidence, what’s the philosophical premise we derive to underly our development practices:

To be able to consistently deliver the kind of resilient and ongoing change the business requires, we deploy a spectrum of confidence gaining techniques.

Or jokingly what we call, “Making failure cheap and easy.”

Before moving on it’s worth calling out that the goal is NOT to be careful. The goal is to be confident. Careful would imply we’re trying to avoid the risk which is fundamental to the change we’re trying to make. Attempting to avoid risk often leads to paralysis, favoring the short term risk avoidance while compromising long term goals. Instead confidence implies you believe to the best of your ability that you understand and have mitigated the risk involved in your change, and are now going to act.

Now, with a little shared theory and philosophy, what does that spectrum of confidence gaining techniques look like?

Our Paths to Confidence:

  • Small and Frequent (and Iterative)
  • Testing
  • Ramp Ups
  • Controls
  • Default Access to Open
  • Monitoring, Metrics and Anomaly Detection
  • People / Culture / Brains

Each of which I’ll talk about in subsequent future posts.

1. Complex systems as defined as something that has many diverse, interdependent, adaptive and connected parts points to the uncertainty. Small perturbations can produce large results, and those results could be failures or successes, but in either case: the potential for surprise is high.

Tagged: Uncategorized

Surviving being senior (tech) management

July 16th, 2013

I’ve got a short list of things I tell people they need to do to survive being senior management. This list has come up a bunch in the last week talking to different folks. So I’m writing it down so I don’t actually have to remember it. That’s sort of unfortunate because there are some alternate versions that exist in a super-positional state, but I think having it written down outweighs the flexibility.

I’m sure the list isn’t especially unique to being senior management but there are a few things that are unique to being senior management, that makes it particularly relevant:

  • it’s a job where your ability to cope with your demons is critical to the success of everyone who works for you. See also Ben Horowitz’s, “What’s The Most Difficult CEO Skill? Managing Your Own Psychology”

  • most people doing the work (at least in tech) are transitioning from maker to manager, and while a few special people are both good at management and enjoy it (and also a few sociopaths), most of us find it a really difficult transition to feel good about consistently and on an ongoing basis.

It’s a simple list. It shouldn’t surprise. This is the minimum. This is my list from having done the job, managed folks doing the job, hired, promoted, and fired folks doing the job, and perhaps most importantly drank with folks doing the job. Your mileage may vary. (But I’d be kind of surprised.)

1. Get some exercise

The ways most of us cope with stress are toxic. They lead to sickness, injury, and reduced cognitive clarity and elasticity. Small amounts of regular exercise help. This is not about getting in shape, this is not about living longer, that’s between you and your work-life balance. But to be an effective manager you need to healthy, functioning, and present, exercise will help with that.

2. Have someone to talk to

There are two variations of have someone to talk to on this list. That’s how important it is. Management brings shit up. It’s a psychological job. Your relationship with your parents is unfortunately relevant, as are just about every other aspect of your personality. Knowing what triggers you, and why, and having someone you trust to talk through it is the only way to do the job well.

It can be a coach, a therapist, a good friend, potentially a very patient and saintly spouse (not recommended). Ongoing, trusted and good at listening are the characteristic you’re looking for.

3. Talk with peers

As distinct from #2, find some folks in your industry, with similar job scope. Get together regularly. Talk shop. But the real shop. The stuff you don’t talk about when the people you work for or the people who work for you are around. This should be off the record. This isn’t a meet up. Start with a small group. Intimacy is the name of the game. Alcohol can help. Ask people, they’ll say yes, everyone needs to talk.

What you’ll find out is everything is fucked up everywhere. And you feel better about your own job. Your problems suck, but boy are you super glad you don’t have their problems. And they’ll feel the same way about you, and your problems.

Perspective is the thin line between a challenging but manageable problem, and chittering balled up in the corner.

4. Have a personal mastery project

Maybe you used to be a coder. Now you’re management overhead. But you really loved coding (and probably because you loved it so much, you spent a lot of time analyzing how folks could do it better, and that’s how you ended up in this mess). You’ve admitted to yourself that you can’t really spend your time writing much code anymore, but you like to keep your hand in the game, carve off a small project here or there for yourself, something that you can look back on after day and say, “Hey, I actually accomplished something today, not just go to meetings.”

You’re almost certainly doing it for the wrong reason. Cut it the fuck out.

There are lots of good reason to stay close to the day to day work (including, but not limited to, you’re an early stage startup, and everybody has to pitch in), and even more failure modes in that directions. But none of those good reasons are about you feeling better, or more in control, or like you “did something real”.

But that doesn’t mean the need you felt to learn, grow, acquire new skills, and generally stretch yourself that were hopefully key traits to getting you this far just go away, or that the sometimes vanishingly abstract accomplishments of your team can be swapped in for that personal satisfaction. For that you need a personal mastery project. Something, quite probably not related to work, where you can prove to yourself that you aren’t actually getting dumber every day (just older), but can still think, reason, and learn.

A side coding project might be it, learning a language, taking a class, practicing classical piano. Something. Something that stretches you, and you can master. By yourself.

Put your own oxygen mask on first, before assisting others

Time is tight, and your schedule is the buckshot mess of manager time. Maybe you’ve mastered time management (block off time, use Google’s auto-reject I’m-busy feature), maybe you haven’t. You’re certainly too busy with work + life to add anything new to your schedule. After all the company depends on you (as does your family). Get over yourself.

You aren’t useful to anyone if you aren’t taking care of yourself. There is an unbound set of things you could be doing better in order to help insure the success of your team that will constantly expand to fill up all the time, but the most important thing you need to be doing is making sure your own oxygen mask is on first.

(hopefully first in a series of posts turning the emails, and chats I have with folks about this work into something public. Keep an eye out for, “Help, I’m a CTO now what do I do?” and “Confidence in the face of risk”)

Tagged: Uncategorized

Novels, Yes And Improv Comedy

May 25th, 2013

So, here’s the insight I’m currently tossing around in my head: The problem is that software isn’t built; it’s written. The final product is not like the Bay Bridge. It’s like a novel. – Don’t build. Compose. – Kurt Leafstrand

No, that’s not right.

As appealing as it is. You see novels, modern novels, are a particularly peculiar form of creativity primarily characterized by being the point of view and output of a single individual. Most software, and most bridges, aren’t like that.

Maybe you’re writing novels, but most of us are doing improv theater.

Mr Chaturong Nakmamee – Koh Tao, Thailand

March 18th, 2013

I met Mr. Chaturong this morning. It was 6:30am and he was waiting with his taxi and his 7yr old son to drive us to our dive spot. It was a peaceful morning, up a small mountain, some dew still on the broad green leaves, away from much of the bustle of Thailand (if Koh Tao can be said to have any bustle at all). He might have been about 40 years old, I didn’t ask. I did ask if he was from Koh Tao, I hadn’t yet met someone from Koh Tao in the 4 days we’d been here, but I kept asking, here is what he answered.

“I’m from Koh Tao, people come from lots of places, all over, to find work with the tourists, but it was different when I was young. When I was young there were only 700 people on the island, it was quiet, no trucks, no motor bikes, we walked everywhere, it was nice, my dad was a fisherman, sometimes made coconuts, everyone was a fisherman or made coconuts. It was quiet.”

If you need a taxi in Koh Tao his number is 089 0049117.