on moving to the Bay Area and engaging with Startups

dave@davepedu.com (dave) — Wed, 12 Jul 2017 00:00:00 +0000

I moved to the San Francisco Bay Area almost 3 years ago now. Having grown up in a small town (~2000 people) and gone to school in a small city (Rochester metropolitan area, New York ~1,000,000), it was not unexpectedly quite the shocking transition.

Most of my observations of difference are fairly mundane - traffic patterns are different, more timid in some ways, and more aggressive in others. Endless cool summer sun is easy to love. So many cultures and cuisines all in one place. A beautiful setting, and at the same time, it’s ground zero for the information technology revolution upending the world.

I saw what the Bay’s tech industry had to offer and found my niche over time. I came here to work for HP, an ancient giant in the hard and software industry, doing web development for marketing websites. I eventually found my way into the Startup Lottery, designing and operating critical core infrastructure for “big data” applications.

Startups are generally small companies that attempt to use computers and code to tackle challenges previously thought as too hard for such a small team. And the Lottery is the right term for how startups ebb and flow in the Bay.

Every so often, a “new thing” becomes the popular “new thing” and a thousand startups pop up around it. Doing “it” directly, supporting “it”, documenting “it”, consulting about “it”, defending from “it”, whatever “it” may be. The Dot-Com boom in the early 2000s, Web 2 and Social media in the later 2000s, early Apps in the 2010s, Big data in the mid 2010s, the list goes on. When it looks like there is not anyone left that wants to pay any more money for the old thing, a new thing pops up and the cycle begins anew. A handful of those startups may survive, and actually become profitable and valuable. And the few investors and few early employees in that company reap what is left of their particular thing, which is still substantially much, even though everyone is paying attention to the “new thing” now.

Once you understand that, you can see why working for a startup in Silicon Valley is like playing the lottery. You probably won’t, but there’s a slim chance you could pick the right company, stick with it, and be rewarded.

Even if that doesn’t happen, working for a startup is rewarding in other ways too. These companies understand they are inherently unstable, and as people generally prefer stable occupations over unstable ones, make up for it with anything from laid-back offices with free snacks to higher salaries. And as different types of perks appeal to different people, each startup tends to have a different personality.

As these companies are usually small, each employee is often in a greater position of responsibility. Large companies tend to let employees hyperfocus in specific areas and make slow changes involving a large number of people. Startups have fewer employees, which means each one must manage a larger slice of the company’s footprint.

In practice, this kind of environment pushes us engineers to think in terms of scale and leverage much earlier in their careers.

The only practical way to operate is to gain this leverage through automation. Tasks that might be handled manually in an older and larger organization quickly become unsustainable. You begin to look at systems not as one-off deployments or fixes, but as processes that must run reliably over and over again with minimal human involvement.

Automation is a force multiplier: the good decisions scale, but so do the bad ones. A poorly designed process that runs once might cause a small inconvenience. A poorly designed process that runs automatically across an entire fleet can cause a disaster in seconds. A process that works well in isolation may fail when invoked thousands of times simultaneously. Because of that, building automation forces a special kind of discipline. You have to research the tools you use, think through edge cases, make predictions with limited information and understand the systems you’re interacting with much more deeply than if you were just executing tasks by hand.

Over time, that kind of work develops a particular sort of experience. In larger organizations, thinking about systems at that scale is often the responsibility of architects or senior infrastructure teams. In a startup environment, those responsibilities fall much earlier in a person’s career. When a small group of engineers is responsible for keeping a large and complex system running, everyone ends up participating in the design decisions that shape how it works.

That puts you in an unusual position. You’re simultaneously operating the systems, building the automation around them, and deciding how they should evolve. The feedback loop between design and operation becomes very tight, or even nonexistent on a bad day. You feel the consequences of your decisions immediately, which encourages you to design things in ways that are easier to operate, monitor, repair, and repeat.

It also builds a habit of intentional experimentation. When your tools can affect large systems very quickly, you learn to test ideas in small steps. Changes get staged, rehearsed, and validated before they touch anywhere important. The process becomes less about heroics and more about methodical iteration.

In that sense, working in a startup environment isn’t just about “moving fast”. It’s about learning how to build systems that can move fast safely. And that combination of large responsibility, heavy reliance on automation, and constant experimentation creates a kind of practical architectural experience that would otherwise take many more years, and a much larger organization, to acquire.

And it’s fun to build and drive big toys using other people’s money.

In my current role, a large part of that philosophy shows up in the tools we create and use to operate our systems. I spend a lot of time working with tools like Ansible and SaltStack, both of which are designed to control and coordinate large groups of machines at once. Each brings its own model for turning a small amount of human effort into changes across an entire environment. Used well, they make it possible for our small team to operate infrastructure of surprising scale.

And honestly? All of this - the Lottery, the scale, the rewards - adds up to a whole lot of fun. The confident, but still nervous, satisfying feeling of seeing your change take effect on exponentially more systems is pretty addicting.

Life on davepedu.com

on moving to the Bay Area and engaging with Startups