Paul Dix Explains Nothing

Water water everywhere and not a drop to drink: The Myth and Truth of the NYC engineer shortage

Earlier this week I had the opportunity to sit with six other people from the NYC technology scene and talk to NYC Council Speaker Christine Quinn and a few members of her staff. Charlie O'Donnell organized the event to help Speaker Quinn engage with the startup scene and hear from us directly what the city can do, if anything, to help foster startups, tech, and innovation here in NYC. The two major themes that emerged from the discussion were: more on the ground interaction between city government and the tech community and that NYC has a major shortage of engineering talent. The resulting talk on the perceived programmer shortage in NYC is the inspiration for this post.

Ask any startup in NYC (or anyone that needs a programmer for that matter) and most will agree that their biggest problem is finding programmers to hire. This was definitely the most prominent topic in the discussion with Speaker Quinn and her staff. The question then became: how do we solve this problem? How can the city government help? The group converged on education as the method to create more programming talent in the city (there wasn't much talk about what can be done to entice already qualified developers to move here). I agree completely that education is what is ultimately needed. An idea that I thought was spot on was required computer science classes for grade school students. We require math and physical sciences. Computers are just as ingrained and important in our society at this point as math and hard sciences. We need something more than teaching kids MS Word, Excel, and Powerpoint. That's like teaching kids how to use a match to light a fire and expecting them to be able to explain the second law of thermodynamics. However, that doesn't solve the immediate need. It's more of a 20 year plan.

The group's idea to address the immediate need for programmers was to put together secondary classes to give training to those currently unemployed people that have interest in picking up new skills. The city can help immensely by providing space and letting those in need know about the availability of these classes. The thinking goes that people could pick up mobile application development or Ruby on Rails in a period of 3 to 6 months and then jump into the NYC startups that are so desperate for these skill sets. I contend that: first, there are more available engineers in this city than hiring startups claim, and second, this probably won't solve the programmer need to the satisfaction of those NYC startups that claim to be so desperate. Hence the myth that there are no available programmers in NYC and the truth that the best programmers are hard to find. But there is hope to fix this, let me explain...

First, my claim that there are more engineers in this city than hiring startups claim. It's not that startups can't find anyone. I'm sure that startups with open positions came across resumes and even interview some candidates. It's just that they end up throwing out most of the resumes they see and giving the "no hire" stamp on most of the candidates they interview. I've been guilty of this myself. I've interviewed many candidates and given far more "no hire" than "hire" recommendations. I've also looked through a stack of resumes and thrown out a significant portion of them. This comes naturally from a startup's desire to only hire the best. This means that while startups are all screaming to the rafters about the shortage of programmers, there are programmers in this city that are available for hire. But there is hope...

Second, my claim that secondary education in mobile app development or Ruby on Rails won't solve the programmer shortage to the satisfaction of NYC startups. This follows naturally from my first point: startups only want to hire the best. If they can't find the best they say that there are no "qualified candidates" out there. While I'm sure that some of the basics of web development or mobile apps can be taught in six months, the student won't be the best. Becoming the best takes years of education, personal study, practice, and work experience. Someone with no previous programming experience that picks up development over the course of six months will still be incredibly green. They will either not pass the resume filter for hiring or will get hung up on questions in the technical interview. In either case the result is the same: no hire and startups still can't find qualified candidates. But there is hope...

The hurdles around retraining engineers that may not have your specific skill set and working with very junior programmers can be overcome. First, startups need to readjust their expectations. You don't need to have the best in everything. You don't even need a programmer that has your specific skill requirement in their resume. The key is to do pair programming with senior people paired up with junior people (or someone picking up something new). Within six months of full time pairing the junior people will have picked up so much that they'll be well on their way to being ready to mentor a new junior hire themselves.

Getting students from these programming courses around the resume filter is also possible. The combination of two things will go a long way: require the students to create two or more projects and have the students put those projects up on Github. The two projects will give them invaluable experience. Further, they should be required to refactor as they go based on feedback from code review done by experienced developers. Requiring them to put the code on Github gives multiple advantages. First, they learn how to use source control as they write code. Second, the code reviews done by other developers can be done right there on the site and associated with each commit. Third, they'll have projects of reference that can be used to show prospective employers what exactly their classes taught them.

For startups to get past the hurdle of hiring junior people they'll need to do a couple of things. First, embrace pair programming as a great method for bringing on junior developers and making them a key part of your team from the beginning (they'll be mid-level and senior before you know it). Second, readjust your expectations. You don't really need the most bad-ass developer you can find. If you did, you'd do whatever it took to poach them from their current job. Mid-level and senior developers paired with junior developers will do just fine for most startup's needs. If you can't find the best, senior, or mid-level developers to pair with junior hires, look to a consulting company. Pivotal Labs (they have a great new office here in NYC) is regularly brought in to pair with developers already in a startup. They do a great job of bringing in solid engineering process and bringing junior developers up to speed quickly (and they're not the only game in town). Finally, if you can't afford a consulting company or hiring senior developers, you'll need to do one of two things. Roll up your own sleeves and pick up some books and a keyboard, or be ok with the fact that you may only be able to convince someone without experience to work on your hot new startup idea (it's how I got started).

There is definitely hope for the engineering shortage in NYC. It requires bringing on inexperienced people or those in need of retraining (C++ or Java to dynamic language anyone?) and helping them achieve their potential while letting them help you get your startup built. I'd like to thank Speaker Quinn, Sarah, and David from her staff for taking the time to talk to us. I hope that we'll come up with some great ideas and collaboration in the future to make innovation a permanent home here in NYC.

What ideas do you have to increase the amount of talent here in the city?

Update: My friend Brody Berg wondered why I didn't write about the "talent vortex" in NYC that is the financial industry. I actually originally meant to, but simply forgot. So here is why I think the competition from the NYC financial market helps fuel the myth that it's impossible to find great talent in NYC.

First, the finance industry generally pays their programmers quite well. Startups that are desperate for these people will just have to get used to it and offer competitive compensation packages. It's kind of like trying to hire programmers in Redmond or Bellevue, WA in the 90's. Microsoft was a huge talent vortex and offered larger salaries and better benefits. The same could have been said for Google from 2003-2008. You either get competitive or find people that these companies have overlooked.

Which brings me to my second point: there are plenty of programmers to hire that the finance industry won't even look twice at. The intersection of the programmers you want and those that work in finance may not be as large as you think. Financial firms want people from name brand schools (preferably Ivy) with high GPAs (3.5 or greater is common). If you don't want to compete on compensation, look for programmers that don't fit their mold. Self-taught (and specially those with secondary education like this article talks about) are the kinds of programmers that will rarely get consideration from a financial firm.

Also, the financial industry wants Java, dotNet, and C++ programmers. Chances are that if you're a startup, this isn't what you're after. Further, the finance industry has laid off quite a few programmers in the last three years. These are those experienced engineers that don't have the skill set that matches yours. They know Java, but they don't know Rails and Javascript. However, these are generally smart people that can pick it up very quickly on the job. The easiest way to find a Rails programmer right now is to poach from another community and teach them Rails.

Competing with the financial firms is tough, but it's certainly doable.

Why Mark Suster is wrong about not hiring job hoppers

Well known VC and entrepreneur Mark Suster recently posted that you should "Never Hire Job Hoppers. Never. They Make Terrible Employees." I almost completely disagree. Before I dissect his argument I should provide a little disclaimer and background.

The disclaimer: Mark is definitely more of an authority on the topic than I am. He's a VC and has run companies while I'm currently bootstrapping my own and haven't hired anyone yet. Additionally, there are many people in management and hiring positions that would agree with Mark (based on my own anecdotal conversations with people in hiring positions).

The background: I'm guilty of being in the category of disloyal, self-serving, job-hopping scum that Mark is writing about. If you check my LinkedIn profile I have 11 positions listed starting in '98. However, I may get a special carve out upon further explanation of my history. I'll lay it out quickly because it's relevant to my argument later.

Two of the positions are current. One is part time consulting and the other is my startup. Ok, so 9 positions to explain now. Still in job hopping group. One I quit to go back to school. Three were temporary summer positions while I was in school. Now I'm down to 5, but that probably still makes me a job hopper. In one I was laid off because the startup ran out of runway. Another, I left because the previously mentioned startup had intellectual property (read: code) that sold and I was needed to help the buyer transition and integrate it. Now I'm down to three unexplained company departures. One I left because it was a contract position and the project was completed. The other two I left for other opportunities: one to join a startup and the other to start my own company.

With all that out of the way let's dive into some of Mark's points. First, here's what he has to say about someone that has been identified as a job hopper due to having too many short positions on their resume:

You won’t even get through my filter (unless I know you or you’re recommended). No matter what your job hopping excuse is.

This isn't an argument against job hopping, it's an argument for finding a better filter. Now, you definitely won't get through Mark's filter, but you'd get through mine. Note that my filter is built for hiring engineers. Here are some filters in no particular order.

• Do you have relevant experience?
• Do you have a Github or other public code profile?
• Do you have a blog where you write about technical things?
• Are you involved in the community (attending user groups, meetups, etc.)?
• Do you have programming languages and technologies listed on your resume that you picked up in your free time?

Most of these filters are designed to tell me if the candidate spends some of their free time improving their skills and advancing their career. Of course, most of those specific filters are only good for finding programmers. They also only apply to looking at resumes and not other things that come in an interview (like personality, culture fit, and more detailed technical stuff). However, people (and not just engineers) that work on their skills and career in their free time are more likely to be job hoppers unless their job is with them and helping to contribute to their growth. So detracting job hopping isn't part of my filter. I'll get into the why when I dig into his primary argument against them (us).

Mark goes on to define a job hopper. Notice the language he uses in his definition:

If you’re 30 and have had 6 jobs since college you’re 98% likely to be a job hopper. You’re probably disloyal. You don’t have staying power. You’re in it more for yourself than your company. OR … you make bad decisions about which companies you join.

There's a little more in his definition, but that's the meaty bit that I want to pick apart. First, the part about being disloyal. "You're in it for yourself more than the company"?! That's exactly what I expect of everyone I work with. They have families to feed, student loans to pay, rent to pay, and they have to be worried about whether their continued employment at the company will benefit their overall career. Everyone is in it for themselves. That's capitalism. The goal is to make is so that everyone's self interests align with that of the company. That means having the right stock options, opportunities for professional growth, acquiring new skills, and clear advancement opportunities. It's a company, not a commune.

Second, the part about making bad decisions about companies to join. As a VC Mark evaluates companies and hopes to find out of 10: 1 home run and maybe 2 base hits. He's ok if 7-8 of 10 companies in his portfolio fail completely. Job seekers don't have that luxury. They have to evaluate the companies out there and pick one. If it's the wrong one then they have to move on to the next. For job seekers to get through a portfolio of 10 companies it can take a decade of changing jobs each year.

Next comes Mark's main argument, which I will boil down to the following: Job hoppers make bad employees at startups because startups need people that are loyal and job hoppers have proven that they are disloyal. First, let's talk about loyalty. I consider loyalty, respect, and love to be things that have to be earned. If you want loyalty without earning it, hire a dog. Giving out a paycheck every two weeks doesn't earn you loyalty. As the old saying goes: money can't buy you love, but it can buy you prostitutes (who probably won't be loyal). Loyalty should be earned by building an environment that is mutually beneficial. Further, you have to be sure to continually assess not only employees, but yourself to see if you're living up to their expectations of what a great workplace should be. Don't hire someone off the street and expect them to be loyal right off the bat just because you agreed to give them a paycheck.

So have job hoppers proven that they are disloyal? For the purposes of Mark's filter the answer is yes, but I personally disagree. I would never fault someone for changing jobs because they felt it was better for their career, or going back to school, or moving to a new city, or because their manager was a jerk, or whatever. Mark's point is that job hopping establishes a pattern of disloyalty and, perhaps, personality flaws. However, you can look at it this way. People have can have a few reasons for staying at a job: it's awesome or they want to leave, but can't find another opportunity. There are many more reasons to leave a job. The truth is that you can't establish whether there is a pattern simply by looking at someone's resume. As my own resume explanations show, there are plenty of reasons to leave jobs. Leave the personality filter for the interview.

Mark then puts job hoppers as not only no-hires, but also as people he wouldn't fund if they were entrepreneurs pitching their startup. Here's that part:

And good VC’s feel the same way. When my company hit the fan in 2001 I could have easily walked and gotten a better paying job. Anyone around then knows that B2B stood for “back-to-banking” and B2C stood for “back-to-consulting.” I took people’s money. It was my job to stay and try and make things better. When I’m looking to fund somebody I care about that loyalty and integrity. I can never know for sure but as I’ve written about before I’m certainly looking for resiliency.

So he views job hoppers as people that have proven that they are disloyal, lack integrity, and have no resiliency. Further, if you're a job hopper that will translate to how you view entrepreneurship. This is so off the mark it hurts my brain. Let me explain why. As an entrepreneur there is one benefit that ranks above all others. Stability and a big paycheck definitely aren't it. It's a sense of ownership. It's a sense of owning your own destiny. I've talked to a bunch of other entrepreneurs and they share that feeling. If you ask job hoppers why they left some of their jobs the story will be the opposite. They left because they weren't given enough ownership. They weren't included in vital decision making processes. When you have the ability to improve your workplace, you have no reason to hop jobs. Most jobs don't actually give their employees this chance. They're filled with bureaucracy and ridiculous power struggles. That's exactly the opposite of the startup and entrepreneurship experience.

I propose that Mark has it backwards. Job hoppers could be exactly the kind of people you want to find for your startup. The likely case is that you're a job hopper because you're willing to take risks, you're looking for something great to work on, you're looking for a job that gives you a sense of ownership, and you're looking for the right team. True, you could be a job hopper because you have serious personality flaws, but save that for the interview. If 2 out of 3 job hoppers have exactly the personality traits that make them well suited for startups and 1 is a disloyal scumbag, you're better off specifically filter for job hoppers.You won't ever find 2/3 odds in a stack of resumes.

Finally, I propose that an argument could be made that startups specifically don't want people that stay in jobs too long. These are people that have shown that they prefer stability over exploring  something new (or maybe they were the lucky few that found the right job on the first few tries). Stability is exactly the opposite of what you're signing up for when you join a startup. The person seeking stability over all else will be the first to leave in favor of a big company or government gig when things get dicey.

The job hopping filter could be used in exactly the opposite way that Mark is suggesting. That shows that it is a poor filter. A filter should give you better than random odds and job hopping just doesn't. It could be good, it could be bad, you just don't know. There are too many reasons for someone to have many entries on the resume. Stick to what's there and don't try to read between the lines.

So who in the hell should you hire? Hire the best. Hire people that can leave your startup at any minute if they wanted to because they're so kick ass that they're constantly getting contacted by interested parties. Then it becomes your job to ensure that you're creating an environment that is equally as good as your people. Create a company that gives your people the most room for growth, creativity, a sense of ownership, and fun. And if you can't hire the best then hire people with any level of experience (novice, intermediate, advanced) that have promise. Then help them become the best so that they can leave any time they want. Hint: they won't. They'll be loyal because you helped them become the best. Your goal should be to help every single employee get to the point where they're the best in their field and are constantly getting job offers.

And what if they aren't loyal once you've helped them become the best? People will eventually leave no matter what you do. Don't view it as a character flaw on their part. Just know that you can pick up more people because you've created a work environment that breeds the best and the word will spread. And as any job hopper can tell you, there are plenty of companies where your skill, ambition, and talent can go to die, but there are very few that help you become the best and give you real reasons to be loyal.

Parse Domains from URLs easily with Domainatrix

I've released another library just in time for the new year. The new library is called Domainatrix and she's used for parsing domain names, canonicalizing URLs, and a few other things. It uses the list of domain names from the Public Suffix List to know what constitutes a subdomain, domain, and public suffix.

Usage couldn't be simpler.

url = Domainatrix.parse("http://www.pauldix.net")
url.public_suffix  # => "net"
url.domain # => "pauldix"
url.canonical # => "net.pauldix"

url = Domainatrix.parse("http://foo.bar.pauldix.co.uk/asdf.html?q=arg")
url.public_suffix # => "co.uk"
url.domain # => "pauldix"
url.subdomain # => "foo.bar"
url.path  # => "/asdf.html?q=arg"
url.canonical # => "uk.co.pauldix.bar.foo/asdf.html?q=arg"

It's simple, but I think quite useful. Enjoy!

Resources for Synchronous Reads, Asynchronous Writes at RubyConf 2009

Here are some links to articles and libraries that I mentioned in my talk at RubyConf 2009.

• RabbitMQ
• Sinatra
• Typhoeus, my Ruby library for performing parallel Ruby HTTP requests
• Great intro to RabbitMQ
• Bunny, a synchronous Ruby library for RabbitMQ
• AMQP, an evented Ruby library for RabbitMQ
• YAJL-Ruby, a superb streaming JSON parser written in Ruby and C
• Good article on Brewer's CAP Theorem
• Vogels' article on Eventually Consistent

Here are links to images used during the talk.

• NYC
• Synchronized
• lyubov_zhivotn_12
• Feathery Friday is Having a Party
• Jason Siefer and Greg Pollack of RailsEnvy from their Advanced Active Record Peepcode
• Highway Insomnia
• Boldly go where no man has gone before
• Terrified Sandwich
• darma queen
• Sinatra
• Distance, Depth, Marble World
• Cathédrale espagnole
• Maneating Antenna'd Neotronic Terror Insect Simulator
• Don't Stop!
• Big Baby
• Otter Cry/Laugh - Tamar Otter and Wildlife Park, Devon

Finally, here are the slides with the images removed.

Synchronous Reads Asynchronous Writes RubyConf 2009

View more presentations from pauldix.

Typhoeus, the best Ruby HTTP client just got better

I've been quietly working on Typhoeus for the last few months. With the help of Wilson Bilkovich and David Balatero I've finished what I think is a significant improvement to the library. The new interface removes all the magic and opts instead for clarity.

It's really slick and includes improved stubing support, caching, memoization, and (of course) parallelism. The Typhoeus readme highlights all of the awesomeness. It should be noted that the old interface of including Typhoeus into classes and defining remote methods has been deprecated. I'll be removing that sometime in the next six months.

In addition to thanking everyone using the library and everyone contributing, I should also thank my employer kgbweb. If you're a solid Rubyist that likes parsing, crawling, and stuff, or a machine learning guy, or a Solr/Lucene/indexing bad ass, let me know. We need you and we're doing some crazy awesome stuff.