Over the past couple of weeks, I’ve been pondering the above question for a number of different reasons. For people who really study programming, like I attempt to, there are a number of claims/myths/legends/tales that are commonly held about people who cut code for a living, such as:
- some programmers, the “alphas”, are as much as ten times more efficient than the common programmer;
- there are people who “get” computers, and those who don’t. Cooper splits these into “humans” and “homo logicus”. Those who don’t grok computers are destined to never be able to program;
- there are people who are paid to cut code, and who simply can’t – they rely on auto-complete IDEs, cut’n’paste library code, etc.;
For the purposes of this post, I’ll separate between these different concepts: the “goats” (people who cannot code, at all), the “sheep” (people who code, perhaps professionally, but poorly) and the alphas. Sheep and alphas are collectively referred to as coders.
Saeed Dehnadi’s PhD homepage cropped up recently, on Hacker News I think, and mentions some tests which have had varying degrees of success in differentiating between goats and coders. Somewhat surprisingly, it’s claimed that it’s possible to administer tests to people before they have been taught to code and yet still determine whether or not they will be able to code. The tests are disturbingly simplistic, but although they involve (pseudo-)code, they’re actually designed to determine the mental models people apply to the problem, and in particularly whether people apply a consistent model.
I have to say, I remain a little bit sceptical about all of that. It reminds me of a question a lecturer once asked our class, while working on set theory: “Why to people refer to ‘∀’ as ‘upside down A’, and refer to ‘∃’ as ‘backwards E’? They’re both simply rotated π radians”. I remember thinking to myself that he obviously had little clue how the human mind approached novelty – and in particular attempt to label things with a communicable tag. “`A’ transformed with a π radians rotation about the centre point” not having quite the same ring about it. But maybe there was a point in there, somewhere, about finding single consistent models that can be re-applied.
It’s really tempting to think about this in terms of tests. This is, after all, one of the reasons vendor certification programmes came into life: to reassure employers of whatever description that the person they’re hiring to do a task with a specific product really is able to do what they say they are able to. And it does work, kind of, after a fashion. If what you need is within the narrow scope of the studies of the certification, you can be generally assured that the person does at least know the theory. However, for programming, this is a bit different – frankly, there is no such thing as “narrow scope” when you’re talking about cutting code. Some people, like Paul Graham, go as far as to say questions like “how do you pick good programmers if you’re not a programmer?” are basically unanswerable (mistake 6 in his classic list of 18 mistakes that start-ups make).
It’s also difficult to talk about how you can tell the difference between sheep and alphas (let’s pretend, for a moment, that there is no spectrum in between there – that’s probably not true, but it might be a somewhat valid simplification). How many people read Reg Braithwaite’s discussion of using FizzBuzz during interviews and didn’t recognise the picture he paints? Let me repeat here his main thesis:
He refers back to another classic, Joel Spolsky’s thoughts on the hiring process from back in 2005. So, what are all these people doing applying for jobs that they are effectively incapable of doing, and how many of these people actually end up being hired and contributing to the sheep pool? It’s difficult to know exactly why they are applying, but part of the reason has got to be the modern tools available to programmers: both the editing environments, the IDEs, and the documentation and programming resources available. Some coders will have no idea about the types of tool I’m talking about, having never used the likes of IntelliSense and visual designers.
Let me give you a clue. Take a long hard look at this abomination. Just to be really clear about what’s going on there, they have a tool which can effectively write code based on semantic guessing – the words you’ve written, the types of the variables involved, the context of the code, that kind of thing. This is like a builder using a brick-laying machine to build a house without the need of any intervening thought about such trifles as “how strong should this wall be?”, “am I putting this wall in the right place?” and even “what if I need to later put a door in here?”. Simplifying the coding process is an admirable goal, and in fact has been an ongoing process since we wrote the first assemblers to translate mnemonic instructions into machine code. However, the ideal has always been to raise the level of abstraction to the height at which it makes sense to solve the problem at hand. You don’t want to write an entire database system in assembler, it’s not really a high level enough language. But you may want to dip into assembler in specific parts. Sometimes, it will hide the details of what’s going on underneath from the programmer, and occasionally that will annoy the programmer. In general, you do not what to be writing large pieces of code where you actually have no idea of what’s going on – an alpha would never, ever stand for that; a sheep, on the other hand, would.
Jeff Atwood has another collection of thoughts on his 2007 post about coders who can’t code. The conclusion he reaches is relatively natural, based on his references and the above: you ask people to actually write some code. Amazingly, surprisingly, gob-smackingly, this still isn’t happening – even today. I could name, but won’t, places which are hiring programmers based on not much else but their claimed experiences. I know people who’ve been through such processes, and have seen such myself. Do you need to ask a question of each candidate, make them do something original? No, of course not – you could even simply ask for a sample of code and take it on trust that they wrote it. My experience on the hiring end is that it’s actually quite easy to tell whether someone wrote a piece of code or not, and the most telling piece of information is not the code itself, but the problem it solves – seeing the type of code that a candidate thought was appropriate to display actually says an awful lot about their tastes and sensibilities.
If I was doing this again right now, what would I do? Probably about the same as I did last time: ask people who turn up to do some simple tests. It’s shocking how often people with an otherwise interesting CV just totally bomb even the most simple request, but it’s nothing much more than a filter. Trying to determine higher-level skills is actually fundamentally more difficult, because the more skill you’re attempting to detect the more parochial your test necessarily becomes, to the point where you’re filtering out people who simply don’t have the exact same background / knowledge set as yourself. Much more important is the capacity to learn and apply problem-solving techniques – part of me thinks that asking them to write code in a language they’ve never seen before might be an interesting test, but it would be fantastically difficult to pitch it at the right level.
I’m going to end with a link to a discussion on StackExchange about how to detect a passionate programmer. I’m not sure I agree with much more than about 50% of it, but there are a lot of ideas in there for predictors of programming expertise. Interestingly, there are even some companies out there who claim to be able to help out in the hiring process to clear out goats and sheep. I have a lot of sympathy for people like Eric Smith who are rolling their own tests to attempt to judge skill. I have to say, though: it really shouldn’t be this hard.
