5626463547 | San Jose, CA 95126

On Justifying the Current Practice in the Software Engineering Technical Interview

Why do software engineering interviewers care so much about data structures and algorithms? 

It is a question you have probably asked yourself if you have gone on the market for a software engineering job. I have never myself heard a very convincing answer where the answer is interpreted as a justification of the practice. There are plausible sociohistorical explanations for why interviewers ask questions like, “balance this binary tree” or “implement a function that removes a node from a linked list.” Such questions are arguably analogs of vestigial organs, a legacy of interview questions from the 1950s. But still they remain. And should they? Yessays Soham Metha, the previous Director of Engineer at Box and founder of InterviewKickstart.com.

Metha’s answer to this question is interesting because he acknowledges:

Yes, DS/Algos are rarely coded directly in your daily work – in fact, we categorically know that you’re unlikely to use many of them directly, if ever, in your entire career.

In fact, if I simply want you to sort a dataset at work, I’d categorically expect you to not code Quicksort in 30 minutes. I’d instead expect you to start with SQL or Command line, and I’ll give time for you to learn it, if you don’t already know it.

But given their rare relevance to work, “there need to be some distinct advantages to the process, for it to be so prevalent. And there are.”  The consideration of those putative advantages is the topic of this post.

Metha does not advocate asking such questions to every candidate for any scenario. If a candidate has the domain knowledge you are looking for, you can skip it. If the candidate is a well known computer scientist, don’t waste everyone’s time with these questions. If the interviewer does not know a solution, don’t ask it.

Metha also knows that there are candidates who do not prepare for these sorts of questions but who might be great at the job. His line on this is that there are a lot of candidates who are happy to go through this process and companies should just aim at those people. I will say little about this because it is obviously irrelevant to whether the process is a good process, and it is doubtful that there are a lot of candidates who are happy about the existing process. And in any case, it is not always an employer’s market.

I will move on, now, to consider the three best reasons provided for justifying the current practice of the software engineering technical interview.

Reason 1: It is the best method we currently know of for interviewing

Interviewers are attempting to determine how well a candidate will perform on the job if hired. Metha argues that because the interview process is such a poor predictor of how well someone will actually perform on the job, we are stuck between either asking some questions that have a chance in measuring performance versus random selection of hiring job candidates. And algorithm/data structure questions (ADQs) are best option we have to measure performance. For this argument to justify the continued tradition of ADQs, the following would have to be likely true:

  1. That it is better to choose candidates based on some measures rather than based on randomized hires.
  2. That ADQs do a better job than alternatives to satisfy some measures to determine a candidate’s success on the job.

I accept 1. But it is interesting to see how Metha justifies this. He points to empirical evidence such as that companies like Google have performed tens of thousands of interviews, and they have found that there is no correlation between how well a candidate does on an interview and how well a candidate performs on the job. But, he adds, at least we can say that we tried to hire candidates that will perform well.  I think this justification is self-serving; it serves our own sense of attempting to accomplish something without also serving the candidates or the needs of the companies performing the interviews.

The proposal that we must face with this dilemma (i.e. try our best in a world with bad correlation or randomize) is partly due to basing one’s evidence upon results from a company that performs the very type of interview in question, that would lead one to believe we are stuck with only mediocre predictors. But not only Google experiences the problem of bad correlations; and also not only is it limited to software engineers. Many fields fail to train people how to perform interviews apart from avoiding illegal questions (“are you married?”). As a result, interviewers must assume a standard of what makes for a good interview, and they can use existing normative structures as the basis for whether a candidate is adequate, especially when those norms have been in place for decades. Even if the questions might have been good at one time, it is not necessarily true that they are relevant or good question for today.

But perhaps ADQs are the best option we have, which is the second point.

If the measure is problem-solving, then ADQs are one way of determining problem solving skills. It doesn’t follow that they are the best way. Metha’s argument requires that ADQs are the best.  So are there other measures, and if so, what? Metha considers knowledge-based role questions. If you were applying for a Java job, you might be asked questions about how would one implement an API with Spring. But, he says,

Knowledge-based questions (e.g. Java), while can be argued as work-sample tests, they don’t really test the GMA [general mental ability], and are difficult at best to make new/challenging. Also, Software engineering is not a knowledge-based role – it’s a problem-solving role. if companies want to hire knowledge-based roles, they generally just hire contractors.

The fact that knowledge-based roles go to contractors is irrelevant to whether ADQs are best for determining on the job success, and contractors also get ADQs. If the goal is to predict success on the job, the fact that the questions are new or challenging is also irrelevant. The questions should be only as new and challenging as the job would require when it comes to what is essential to determining how a candidate would do on the job over the long run.  One can ask a Java engineer to implement a tiny feature that has not been implemented on his or her resume. That would indicate whether (1) the engineer knows Java, and (2) whether the engineer can learn quickly how to acquire new skills and be successful within the company. Those can be new and challenging if the interviewer puts forth some effort in to thinking about the question. (Sometimes companies give interviews too little time though, and that should change.) It is also odd that ADQs are propped up as being uniquely suited to be new and challenging. There are many books like Cracking the Interview, that cover commonly asked ADQs: e.g., if this string is an anagram?

The reasons given thus far for dismissing other types of questions are either irrelevant, their alleged flaws either are not flaws or they are also properties of ADQs, and in any case, it is already established that ADQs are bad at measuring performance.

But what about ADQs as a measure of GMA? I agree with Metha that they are one way to measure GMA, at least under a very vague conception of GMA. My worry, however, is that they get hyped as the best way of measuring GMA. If someone has a doctorate in, say, linguistics, but is unable to implement a linked list, would that provide good evidence that this candidate lacks GMA? The current trend in engineering suggests yes. The trend is wrong. What being able to answer that question more likely shows is that the candidate has had to think about that question before and can whiteboard a solution; it does not likely show the candidates lacks the required degree of GMA. The fact that the person has a doctorate in linguistics is itself better evidence of there being high GMA. The ADQs that get asked, if they get asked, can be used as measures of how much computer science knowledge this person has. And then the question has the same goal as the knowledge-based question: how well will this knowledge contribute to the job?

ADQs, then, are unlikely to be the best measure of whether a candidate will perform well on the job as a general rule. They might be highly significant to ask in some cases. Engineers who have to find the quickest path should be questioned about, say, Dijkstra’s algorithm or some alternative. But in this case, the relevance of the ADQ is guided by the required tasks of the job and not a myopic way of determining whether a candidate can solve problems.

Reason 2: ADQs allow for companies to interview at scale

Companies get far too many resumes, and “Spending time on interviewing is the last thought ever. Most engineers don’t ever want to spend time in interviewing.” ADQs help, the thought goes, because interviews can be trained on them quickly. In fact, Metha says, “No formal training is necessary. Pick up any random book or website, read the question, read the answer and deliver it.” It might seem a bit unfair that an interviewer has the luxury to look up material he or she might not already know and then expect the candidate to know it within a few minutes. Metha responds, “Not fair, you say? Of course, but fairness is not the goal. Goal is to spend least amount of time, not to prove whether it’s fair or not. And finally, it is an open and closed case whether someone can answer correctly or not.

My view is that this embodies exactly what is wrong with the interview process. If one complains about having to spend time to invest in hiring the right people, is it odd that the people a company hires shows little correlation with how well someone does in the interview?

Notice as well that the argument for scalability depends on the interviewer having to look something up in a book. But recall that in the previous argument above, ADQs were supposed to be new and challenging. Is it any wonder why interviewees purchase books on the interview process? A successful interview in the process Metha defends is only evidence for how well have a candidate has prepared for the interview. It does not provide evidence, generally speaking, that one is likely going to flourish in the job. In addition, there are thousands of questions one might be asked. There is an immense amount of luck involved in getting the questions for which one prepared. That one fails the ADQ is not great evidence one did not prepare, and certainly not great evidence one are not fit for the job. (Obviously: if the question is as easy as writing a for loop, which is something one will have to do on a job, then failure at that problem would indicate not being suited to succeed on the job.)

Finally, whether this is an open and closed case is not true, in my experience, of whether a candidate gets an offer or not. It certainly it not sufficient. If you come across as a jerk during the interview, the interview is over and no offer will be recommended. But not successfully solving the ADQ might not even be necessary. Your solution might fail, but it shows enough creativity and thoughtfulness that together with other considerations are enough to make an offer.

3. ADQs offer flexibility in the interview process

The final argument has to do with flexibility. If an ADQ is too easy or too hard, adjust to another ADQ. ADQs can be written in any format. ADQs are irrelevant to a candidate’s past. They allow the candidate to prove he can think and code rather than talk; they are not coupled to language.

Metha is correct that ADQs are easy to adjust. But why should that matter if they do not indicate whether one will be successful at the job? ADQs are irrelevant to a candidates’s past. That is true. But companies still require resumes because a candidate’s past is relevant. What else is probably relevant to the job is communication skills. Can you engage with product owners, clients, stake holders, other team members and then execute on delivering a quality product? Or is your worth that you sit alone in a darkly covered cubicle with a hoodie and headphones with the skill to solve interview questions? Because there is an important collection of properties and skills successful employees must have to do a job well and be a good fit with a team, the trend to push ADQs as the single most important gate is irrational.

Concluding comment on the justification of the practice

The interview process as it is widely practiced today, I believe, lacks the justification required to render it a rational process. Of course there are times when ADQs are relevant, and significant to the hiring process. But that they are the standard is is a mistake. And this should lead engineers to start thinking harder about the sort of questions that help create the best teams. I have not here argued for what sort of questions engineers should ask. In fact, I am a pluralist about the sort of questions that should be asked. I ask automation testers different coding questions than I ask frontend Javascript engineers.

As a quick example, I sometimes ask automation testers the following question: you expect an api to return a set of objects containing a primitive int and a reference type (e.g., a string) as an object. The api returns an unordered collection of objects. You need to validate that the objects returns have exactly the values you initially expected. Write a function to compare the objects.

That question will require writing some code. There are slow ways to do it (nested for loops), and there are quicker ways to do it (homework for the reader). This question is language neutral. But importantly, it gets at writing a task that someone will actually have to perform on the job. Shoot for those sorts of questions if you can.

As a parting comment, I leave you with a thought experiment. The thought experiment is that you are applying for the job of answering phone calls for a major phone company. Because you will be directed to speak with English speaking callers, you list on your resume that English is the language in which you are fluent. What would the interview process look like for this job were it to follow the trend in software engineering today? And would that show anything is misguided about the process? I’ll start for you:

Interviewer: So you list that you can speak English. Where on a scale from 1 to 10 would you say is your competence level for speaking English?

Candidate: I don’t know every English word, so… I’d say an 8.

I: And how long have you been speaking English?

C: Since I was one year old.

I: Great. So in the sentence, “It would be nice for you to quickly run to the store,” can you please identify what the modal auxiliary is and can you tell me the grammatical mistake that is present?

C: O_o

I: I’m sorry, but we were looking for “would” and “split infinitive.” Strunk & White, Elements of Style, 4th edition. All of the interviews around here rely heavily on that book. We do in fact recognize that you would almost certainly never need to know this sort of grammar on the job. But we have plenty of candidates happy to answer questions like this. It makes interviews interesting. Next!

2018-02-18T23:54:26+00:00 November 18th, 2017|Comments Off on On Justifying the Current Practice in the Software Engineering Technical Interview