Water water everywhere and not a drop to drink: The Myth and Truth of the NYC engineer shortage

Earlier this week I had the opportunity to sit with six other people from the NYC technology scene and talk to NYC Council Speaker Christine Quinn and a few members of her staff. Charlie O'Donnell organized the event to help Speaker Quinn engage with the startup scene and hear from us directly what the city can do, if anything, to help foster startups, tech, and innovation here in NYC. The two major themes that emerged from the discussion were: more on the ground interaction between city government and the tech community and that NYC has a major shortage of engineering talent. The resulting talk on the perceived programmer shortage in NYC is the inspiration for this post.

Ask any startup in NYC (or anyone that needs a programmer for that matter) and most will agree that their biggest problem is finding programmers to hire. This was definitely the most prominent topic in the discussion with Speaker Quinn and her staff. The question then became: how do we solve this problem? How can the city government help? The group converged on education as the method to create more programming talent in the city (there wasn't much talk about what can be done to entice already qualified developers to move here). I agree completely that education is what is ultimately needed. An idea that I thought was spot on was required computer science classes for grade school students. We require math and physical sciences. Computers are just as ingrained and important in our society at this point as math and hard sciences. We need something more than teaching kids MS Word, Excel, and Powerpoint. That's like teaching kids how to use a match to light a fire and expecting them to be able to explain the second law of thermodynamics. However, that doesn't solve the immediate need. It's more of a 20 year plan.

The group's idea to address the immediate need for programmers was to put together secondary classes to give training to those currently unemployed people that have interest in picking up new skills. The city can help immensely by providing space and letting those in need know about the availability of these classes. The thinking goes that people could pick up mobile application development or Ruby on Rails in a period of 3 to 6 months and then jump into the NYC startups that are so desperate for these skill sets. I contend that: first, there are more available engineers in this city than hiring startups claim, and second, this probably won't solve the programmer need to the satisfaction of those NYC startups that claim to be so desperate. Hence the myth that there are no available programmers in NYC and the truth that the best programmers are hard to find. But there is hope to fix this, let me explain...

First, my claim that there are more engineers in this city than hiring startups claim. It's not that startups can't find anyone. I'm sure that startups with open positions came across resumes and even interview some candidates. It's just that they end up throwing out most of the resumes they see and giving the "no hire" stamp on most of the candidates they interview. I've been guilty of this myself. I've interviewed many candidates and given far more "no hire" than "hire" recommendations. I've also looked through a stack of resumes and thrown out a significant portion of them. This comes naturally from a startup's desire to only hire the best. This means that while startups are all screaming to the rafters about the shortage of programmers, there are programmers in this city that are available for hire. But there is hope...

Second, my claim that secondary education in mobile app development or Ruby on Rails won't solve the programmer shortage to the satisfaction of NYC startups. This follows naturally from my first point: startups only want to hire the best. If they can't find the best they say that there are no "qualified candidates" out there. While I'm sure that some of the basics of web development or mobile apps can be taught in six months, the student won't be the best. Becoming the best takes years of education, personal study, practice, and work experience. Someone with no previous programming experience that picks up development over the course of six months will still be incredibly green. They will either not pass the resume filter for hiring or will get hung up on questions in the technical interview. In either case the result is the same: no hire and startups still can't find qualified candidates. But there is hope...

The hurdles around retraining engineers that may not have your specific skill set and working with very junior programmers can be overcome. First, startups need to readjust their expectations. You don't need to have the best in everything. You don't even need a programmer that has your specific skill requirement in their resume. The key is to do pair programming with senior people paired up with junior people (or someone picking up something new). Within six months of full time pairing the junior people will have picked up so much that they'll be well on their way to being ready to mentor a new junior hire themselves.

Getting students from these programming courses around the resume filter is also possible. The combination of two things will go a long way: require the students to create two or more projects and have the students put those projects up on Github. The two projects will give them invaluable experience. Further, they should be required to refactor as they go based on feedback from code review done by experienced developers. Requiring them to put the code on Github gives multiple advantages. First, they learn how to use source control as they write code. Second, the code reviews done by other developers can be done right there on the site and associated with each commit. Third, they'll have projects of reference that can be used to show prospective employers what exactly their classes taught them.

For startups to get past the hurdle of hiring junior people they'll need to do a couple of things. First, embrace pair programming as a great method for bringing on junior developers and making them a key part of your team from the beginning (they'll be mid-level and senior before you know it). Second, readjust your expectations. You don't really need the most bad-ass developer you can find. If you did, you'd do whatever it took to poach them from their current job. Mid-level and senior developers paired with junior developers will do just fine for most startup's needs. If you can't find the best, senior, or mid-level developers to pair with junior hires, look to a consulting company. Pivotal Labs (they have a great new office here in NYC) is regularly brought in to pair with developers already in a startup. They do a great job of bringing in solid engineering process and bringing junior developers up to speed quickly (and they're not the only game in town). Finally, if you can't afford a consulting company or hiring senior developers, you'll need to do one of two things. Roll up your own sleeves and pick up some books and a keyboard, or be ok with the fact that you may only be able to convince someone without experience to work on your hot new startup idea (it's how I got started).

There is definitely hope for the engineering shortage in NYC. It requires bringing on inexperienced people or those in need of retraining (C++ or Java to dynamic language anyone?) and helping them achieve their potential while letting them help you get your startup built. I'd like to thank Speaker Quinn, Sarah, and David from her staff for taking the time to talk to us. I hope that we'll come up with some great ideas and collaboration in the future to make innovation a permanent home here in NYC.

What ideas do you have to increase the amount of talent here in the city?

Update: My friend Brody Berg wondered why I didn't write about the "talent vortex" in NYC that is the financial industry. I actually originally meant to, but simply forgot. So here is why I think the competition from the NYC financial market helps fuel the myth that it's impossible to find great talent in NYC.

First, the finance industry generally pays their programmers quite well. Startups that are desperate for these people will just have to get used to it and offer competitive compensation packages. It's kind of like trying to hire programmers in Redmond or Bellevue, WA in the 90's. Microsoft was a huge talent vortex and offered larger salaries and better benefits. The same could have been said for Google from 2003-2008. You either get competitive or find people that these companies have overlooked.

Which brings me to my second point: there are plenty of programmers to hire that the finance industry won't even look twice at. The intersection of the programmers you want and those that work in finance may not be as large as you think. Financial firms want people from name brand schools (preferably Ivy) with high GPAs (3.5 or greater is common). If you don't want to compete on compensation, look for programmers that don't fit their mold. Self-taught (and specially those with secondary education like this article talks about) are the kinds of programmers that will rarely get consideration from a financial firm.

Also, the financial industry wants Java, dotNet, and C++ programmers. Chances are that if you're a startup, this isn't what you're after. Further, the finance industry has laid off quite a few programmers in the last three years. These are those experienced engineers that don't have the skill set that matches yours. They know Java, but they don't know Rails and Javascript. However, these are generally smart people that can pick it up very quickly on the job. The easiest way to find a Rails programmer right now is to poach from another community and teach them Rails.

Competing with the financial firms is tough, but it's certainly doable.

Comments

[Luke Melia]

Well-written and well-considered post, Paul. I think that your approach can achieve the goal, but not without a loss of velocity, which is a huge pressure at any startup, especially seed-funded companies with a relatively short cash runway.

I'm a big proponent of pair programming, and think that when good developers pair, there is little to know loss of velocity. But pairing when serious mentoring is big goal is a lot slower in the short-term than forgoing the hire altogether.

Start-ups who know they have a long runway can afford to go with your strategy. I personally wouldn't recommend it to most startups.

[Anon]

Also I question startups who won't consider remote workers. If you have a senior technical person to interface with the business, they should be smart enough to divy out tasks. If you hire the wrong remote worker (someone who is jsut gaming the system to fund their own startup - fire their ass). Don't believe me? Try this experiment: put up a posting on the Ruby on Rails LinkedIn group saying you want a remote worker, but will only consider an individual who posts their interest publicly via comments. I guarantee it will blow your mind.

[Paul Dix]

Luke, while I agree that your velocity may take a little bit of a hit, I don't think it will be that bad. I also disagree that it's a bad idea for seed stage startups to hire inexperienced people. I'll give early stage Facebook as an example. The only engineers involved in Facebook at the seed stage were students who were either only half way through their CS degree or weren't even in CS to begin with. They picked up what they needed on the job and went on to become well regarded, experienced engineers. Zuckerberg was more than willing to hire inexperienced engineers because he was inexperienced himself.

Running out of runway will kill any startup, regardless of the experience level of engineers. Experienced engineers may be able to turn out more features, but it doesn't mean that they'll turn out the right ones.

[Brynary]

Late to the party, but a few, somewhat random thoughts:

• I think the definition of "Senior Software Engineer" gets conflated a lot. Pairing at a good shop for 6 months might make you productive, but not Senior. Seniors engineers have been through multiple long-term projects and therefore have the experience from that to bring to the problem at hand.
• I agree with Luke that hiring green engineers is a tough sell when looking at runway. If hiring employee X will be net-negative to productivity for 3 months, neutral for 3 months, and positive after that, it's hard to make that call if you've got 1-1.5 years of cash in the bank. The question of whether they are building the right features is orthogonal -- but engineers with the most experience are also more likely to help the business understand the tradeoff of the different features they can build.
• Finally, I agree with Paul that there are available engineers, but startups are choosing not to hire them. This cuts directly to the value proposition of a green vs. senior engineer. Personally, I subscribe to the theory that a good Senior engineer is 5-10x more valuable than an "average" engineer over time. I think a lot of other engineers managers feel this implicitly as well. But here's the problem: Politically, we can't offer 5x salary for someone who is worth that much. An outsider would think it's crazy to pay a rank-and-file engineer even $200k. So, there are a effectively-infinite number of startups trying to make a bargain hire of an extremely valuable engineer at a price well below their value. (Although I hear there are now local startups offering in the $180s for Sr. Engineers now.) With such an imbalance, traditional hiring strategies breakdown, and even progressive HR policies get lost in the noise.

[Paul Dix]

With regards to the senior thing, what about the following example. Say there's an engineer with 6 years experience delivering projects, but not in the language you're working in. You bring them on and they pick up your language. Is this person senior when they come in? How about after 6 months?

I think they're senior the entire time. They just need a little ramp up time to pick up the language specifics. Having this person pair with one of your existing programmers probably won't reduce your velocity. They may ask basic questions, but those are quickly answered. Most of the pairing time is spent discussing the solution anyway. If you think it will significantly reduce velocity, I'm curious to hear why.

Or how about this. You have a single pair with two good programmers. Call them either mid-level or senior. You hire two green people. So now you have two pairs with one mid/senior and one green. I'm willing to bet that your velocity will increase immediately and go up even more as the green programmers become more proficient. You're still talking about two pairs vs. one pair. And given that you have a proficient programmer in each of the pairs they'll offset the junior person. Besides, some of the value of pairing is simply having someone to talk through a solution with, regardless of their experience level.

Regardless of all this, my overall point is that there are engineers out there that are perfectly able to pick up a project and help build it, it's just that expectations can sometimes be too high on the side of the startups. You either have the money and/or the clout to recruit the best or you have to make a compromise. Most startups fall into the having to make a compromise category. They need to seriously think about how to ensure that they build talent in the organization.

[Brynary]

With your example of an engineer who has a lot of experience (e.g. 6 years) on other similar technologies, yes I'd say many of those are senior engineers (specifically, the ones with the right intangibles). And I've seen these people join highly effective teams and do a great job -- getting above the productivity curve quickly. I wouldn't hesitate to consider those sorts of candidates at any team size.

A green programmer, on the other hand, is a different story. I've also seen situations where trying to bring programmers up to speed who were green slowed down development and created a situation that no one was really happy with. So I'd recommend caution in that approach, but would start to consider it at around 3 pairs team size.

-Bryan

[Kyle]

Thanks for writing this, it was very informative. I would like to consider myself (perhaps arrogantly) a good engineer (above average at least), and I want to share with you what it looks on the other side of things.

Here's the story in a nutshell: My girlfriend is moving to New York for college for up to three years. I live in Portland Oregon right now, and I have good employment at a great company. I've been flirting with the idea of joining her in New York for the duration.. I do kindof like the idea of hanging out in NYC for a while, just to try it (probably won't get another opportunity again, you know?)

I've done a lot of searching through the New York job boards and I haven't found anything that I would describe as approachable. The jobs either sound very strict and unpleasant, are asking for junior developers, are "barely hanging on with a bad idea", or are asking for suspiciously stupid things (the whole "ultra rails ninja wizards" thing).

So I'd like to work in NYC on Ruby/Sinatra/RoR web work, but from my chair the pickings are slim. I feel it's not just a lack of engineering talent.. it's a lack of enticing offers for those engineers. As of now, I have not found work that is sufficient for me to leave my current job and enter another one. The last thing I need if I move to the most expensive city in America is a crap job I can't afford to bail out of.

I'm sure there's plenty of opportunities floating around of course, and you have to dig a little bit.. but I'll need a steady income before I get there, because I won't have the money to look for good work for three months. In that light, if you have any good leads, let me know :-)

BTW, I stumbled onto your site while playing with Typhoeus.. it's a great library. Thank you for including Multipart Form support.. it's the big thing most net libraries miss.

[drew]

Are you serious? You think education is going to produce programmers. Anyone smart enough to program knows paying for college with loans is a waste and huge financial risk. You can't even bankrupt the loans. I have meet some of the best techs without degrees. And getting into 1 of the 5 great computer science colleges is like playing the lottery. If you want to really attract developers, you can start by canceling all housing programs. Why is NYC one of the few cities with a broad rent control program but still is the most expensive in terms of rent? Oh, you can find many $3000 apartments for rent. Half the average salary of a tech. The real thing holding NYC back from being a tech leader is taxes and crazy rent prices. Startups look in places where there is low taxes, land and talent. I love NYC but it will take more exodus and higher vacancy rates before something happens. And god forbid another bailout is made to large property companies that only distort the market and keep prices high.