> We know that cognitive ability has one of the strongest correlations with job performance
There is a good argument that the technical interview success is much more strongly correlated to the number of hours spent practicing Leetcode-type questions than to actual cognitive ability.
The technical interview is not the only stage in the interview process. It might be better thought of as a way to filter out applicants who have poor innate ability and/or poor learning skills.
I would argue that being able to study leetcode and then apply that new knowledge you've learned is an important skill in and of itself in this industry. Isn't that what all of highschool and most of college is? Signalling that you a quick learner and a hard worker is important.
>It might be better thought of as a way to filter out applicants who have poor innate ability and/or poor learning skills.
It’s a way to filter out people who haven’t had time to study leetcode style quiz questions. It has no relationship to learning skills or innate ability. There is no way to distinguish between someone who hasn’t spent time grinding algorithms and someone who had with poor learning skills.
>I would argue that being able to study leetcode and then apply that new knowledge you've learned is an important skill in and of itself in this industry. Isn't that what all of highschool and most of college is?
Nope, it demonstrates that you had time to waste, nothing more. Candidates that built actual software to accomplish something are much more impressive because it shows they can write software attached to reality.
And just like the standardized test rigamarole, technical interviews can be gamed by investing in prep courses and materials for those who are willing to dump the money and time on them.
It's like saying "This guy spent 4 years in college trying to get a 4.0 GPA. Clearly, he only did it to game the system and get a good job/career. So I won't hire him"
A person willing to spend time and money preparing to succeed in a scenario that exists in real life (pass interviews), is one of the types of persons I want to hire.
The situation clearly becomes complicated when an ostensibly purely meritocratic measure becomes something that those with means can improve. If there was free and quality standardized test prep / technical interview prep for all, and then it was just a matter of dedicating enough spare time for the latter, then you might have a better case that it's good. Though you're also ignoring the fact that people have a limited amount of time/energy to devote to that, such as in the case as engineers with children.
I have recently passed several technical interviews at multiple large companies. The first one I botched completely. Then I looked at some algorithmic courses from Princeton on Coursera and passed the rest of the interviews in a breeze.
My point is, there _are_ free and good quality courses out there and if you have enough experience with actual work then it is just a matter of brushing up on the foundation. In the end I’m glad that I did.
Well, yes. However if you are already a good programmer then it will not take that much time, and if you are looking for a job then there is nothing wrong in expecting that you would prepare. And I would argue that it's not sunk time either. In my day to day job I do not use that many algorithms because most of the actually needed ones are hidden by the internals of the language, I got rusty and the refresher was useful.
>New data shows studying for the SAT for 20 hours on free Official SAT Practice on Khan Academy is associated with an average score gain of 115 points, nearly double the average score gain compared to students who don’t use Khan Academy. Out of nearly 250,000 test-takers studied, more than 16,000 gained 200 points or more between the PSAT/NMSQT and SAT …
But the company doesn’t really suffer if they don’t hire that disadvantaged developer who doesn’t have the resources to study for the test. Sure they have to spend some more $$$ on recruiting but there aren’t a ton of companies in America existentially threatened by their inability to hire these hypothetical engineers.
You're stating that the company doesn't lose out with false negatives. That's certainly true with big FAANG companies. But I suppose in my original comment, and when I brought up Goodhart's law in a descendent comment, also suggests that by creating a system that can be gamed by prep, you end up creating a situation that encourages false positives. In the end, is a company of Leetcoder elites and new CS grads going to be a more productive company than one comprised of experienced and capable engineers who don't happen to do as well on those tests? Which I suppose is the question for a different study.
In my experience hiring, the diligence to sit down and practice code tests correlates very highly with being a good employee. Just like strong academic achievement at good schools.
That's not to say there aren't gems in the rough who don't interview well, or skipped college, or who didn't go to good schools. But there's a huge time cost to looking for them.
Given that the former is dealing with decreased earnings growth and the latter is dealing with decreased user growth, perhaps all options should be explored.
> A person willing to spend time and money preparing to succeed in a scenario that exists in real life (pass interviews), is one of the types of persons I want to hire.
This brings us to an interesting dilemma. Do you want to hire people who know the problem domain you're solving --- or people who really badly want to be hired?
> There is a good argument that the technical interview success is much more strongly correlated to the number of hours spent practicing Leetcode-type questions than to actual cognitive ability.
Where is that "good argument" substantiated?
People say exactly the same thing about IQ tests, probably because it's something they'd like to be true. But it's not backed up by the evidence.
Ten years ago, I couldn’t program, so I couldn’t do leetcode problems. Now I can. If I go on the market, I’ll probably study a bit, and get even better at leetcode problems. During that whole time, my cognitive ability will not have significantly changed.
There probably is a decent correlation between cognitive ability and performance on these tasks, but it’s obviously mediated by practice.
https://www.gwern.net/DNB-FAQ contains a rather extensive review of this study and its follow-ups, critiques, etc, and comes to the conclusion that Dual n-Back training probably doesn't work to improve fluid intelligence.
That's quite a leap there. There is no contradiction between thinking IQ tests work and thinking that DNB increases IQ; I thought it did, and it was a bitter experience learning first-hand how exactly the Replication Crisis works.
If you think I am an unreliable source, why don't you look at the other meta-analyses like Melby-Lervåg, or Sala? Or look at the criticisms made by researchers like Redick whose experiments turned out to be nulls? (Redick's most recent comment on DNB is titled "The Hype Cycle of Working Memory Training" https://journals.sagepub.com/eprint/U8YNVXKM5SKF6ZY9TQBR/ful... , so you can probably guess what he thinks of DNB now.) Or look at what the field thinks in recent studies - assuming you can find any recent DNB/IQ studies, as everyone has realized that it doesn't do anything.
The process can definitely reject good candidates who are out of practice, but it’d be pretty hard to do the practice and give good performances without some underlying skill.
But look at it that way: a candidate that didn't practice before the interview, is either not so interested, a bit dull about basic human interaction or cannot even understand what an interview is.
It's honestly like coming to an interview without a suit and a tie. Sure if you're a pretentious startup it'll horrify you, but anywhere else it shows that the candidate just doesn't understand what's happening, that he could have just made the cheapest first impression effort and didn't even think about it, he came in sandals and tshirt instead.
Do you want him talking to clients ? Can you afford him not talking to clients ?
Is there anything wrong with preferring people who have spent a lot of time practicing leetcode?
Most jobs prefer people with certain kinds of experience vs raw cognitive ability. Raw cognitive ability doesn’t mean much outside of entry level jobs.
I don’t believe the interview process would be any more efficient or effective for software engineering roles if they started preferring pure “cognitive ability” to leetcode effort.
Assuming all other things are equal, obviously anyone will choose the candidate who can do these questions vs one who cannot.
The problem, obviously, is that all other things are seldom equal IRL. So, the issue now becomes : Would you rather hire someone who has a solid github profile but can't whiteboard algos, or someone who doesn't have any past portfolio but can ace whiteboard algos.
If you give me 20 years, I could probably get a PhD. Given ENOUGH time, anyone could theoretically achieve anything. But the differentiator between candidates happens when the clock is ticking.
I, and most other employers, are not looking for perfect answers. We want to know how you approach a problem under these constraints : short time, high pressure, unseen problem.
We want to know whether you are the type who chokes. It tells us that your desire to look good in front of someone (or conversely, your fear of looking bad in front of someone) fills up your mind more than your desire to solve a problem.
Would you choke if you mom asked you the same question - No, because you don't fear being judged, right ?
We want a similar trait in our developers. We want you to GIVE VERY LITTLE SHIT about what we think of you, at least for the next 30 minutes. We want you to have enough control of your mind to TEMPORARILY ignore our existence and focus on the problem at hand.
Granted, such scenarios occur rarely IRL, but it tells us a lot about how much control the person has over their psyche, how honest they are about their limits, what they do when they reach their limit etc. It's as much a behavioral test as an IQ one.
Having said all that, not all companies need non-choking geniuses. Choking geniuses work fine for many.
> I, and most other employers, are not looking for perfect answers
I wouldn't agree with that. Most of the interviewers I had at the end of the day didn't want the take the risk of "hire" if the answer wasn't like they expect.
> We want to know whether you are the type who chokes. It tells us that your desire to look good in front of someone (or conversely, your fear of looking bad in front of someone) fills up your mind more than your desire to solve a problem.
I think that can be better written in the job requirement. I would think twice before going to an interview which the solely purpose is to make an artificial stressful situation, out of an already quite stressful situation, just to see if I break or not.
Yeah, I can give zero shit if the stakes are low or if I am not interested in the job or the company for example. What happens is not my ego is hurt, but before I go for an interview I am already emotionally invested that I want to work for that prospective company.
All that said, people come from different backgrounds, imagine interviewing someone who is constantly judged in life and/or have struggled with bad managers prior to expect they would be free like a bird in front of failure.
The problem is that you're not paying enough for this. You're just not, even if you're FAANG. A smart dev can likely find a good answer for you---but it will almost always be true that it is a better answer, for them personally, to just go work somewhere else.
We are not paying you to SOLVE the problem with those constraints.
We are simply evaluating how you APPROACH a problem with those constraints.
If given such a problem, you throw your hands up in the air, or give me a lecture about how such a problem does not occur in real life, or how it doesn't represent your best-self, that itself tells a lot about you.
My question, as I repeat in other comments is this : WHY do you feel stressed ? Delve into your psyche for just 1 minute and it will boil down to - fear of being judged as incompetent. All we are looking for is someone who can set aside this fear TEMPORARILY.
You don't feel pressure if your mom throws you this problem, right ? Can you do the same even if a million people are watching, but just for 2 minutes ?
Even if we throw you a NP-Hard problem, can you objectively spend 2 minutes thinking about it, talking another 2 minutes about how you'd approach it, and then saying, 'This looks like it's beyond my skillset' - That clarity of thinking, mastery of fear, and blatant honesty is worth a hundred github apache open source projects.
People who can't do this lack the maturity to understand that in real-life, nobody really gives a damn about our achievements and failures except us (and very close relatives of course).
I think you are kidding yourself if you think the majority of these companies are satisfied with an answer consisting of "This looks like it's beyond my skillset". All things considered, they are going to progress the candidate who can solve the problem because it's an early stage test.
I wouldn't feel pressure if my mom threw me the problem, but I still probably wouldn't be able to come up with a solution (or even approach anything useful in a reasonable amount of time) that would satisfy an interviewer because frankly, I'm terrible at it and haven't spent the time to get good, just like many, many other successful developers.
Stress is only one aspect of these that people dislike.
>All we are looking for is someone who can set aside this fear TEMPORARILY.
Why should I do this? Fear is a survival signal, why should I ignore it? It's not fear of the problem, it's fear of the company. And is this not a justified fear? You say not to be afraid of being judged, but isn't that the entire purpose of the interview?
Now, there is a scenario I can imagine where I have some pride invested and am willing to be a fearless engineering badass, the developer equivalent of a Marine. But the Marines are loyal back in return, never leaving a man behind, etc. I'd also happily attempt such a problem for my mom, because I know our relationship isn't dependent on this particular problem. Is your company going to be as loyal as the Marines, or love me unconditionally like my mom?
Why do you say that FAANG isn’t paying enough for this? They seem to be, since they are hiring more than enough engineers in this manner to build even trillion-dollar companies. I doubt they give two shits about the developers who decide to work elsewhere.
Sure, they hire a good # of devs---who then mysteriously want to go into management, leave to start a startup, or simply complain about their bosses. No one with actual agency and power in the matter chooses to stay in that position, once they understand what it is, what their options are, and aren't handcuffed by a mortgage. Or perhaps the company isn't as dysfunctional as the interview, and they stay---in which case it is just the interview that's miscalibrated, serving as an arbitrary hazing ritual.
Despite any problems they have, they are able to retain enough devs to get their work done. Are they losing out on even $1 of earnings because of their inability to hire devs? Even with the arbitrary hazing ritual in place?
> Is there anything wrong with preferring people who have spent a lot of time practicing leetcode?
If you care about diversity and inclusion, yes. This interview format prefers people with formal education, extra time, good mental health, no kids, etc.
This can easily eliminate some women + caregivers + those with kids, the self taught, people that struggle with mental health or learning disabilities (including ADHD), etc.
Anecdotal story:
Studying LC because I'm self taught led me into burn out and I ended up with an ADHD diagnosis (common for women to get diagnosed at an older age anyway). My ADHD brain wants to do things it likes and studying LC isn't one of them so it led to being burned out faster. A former manager even told me to slow down and I didn't listen!
I struggled for a while with interviews and getting my current job. Each additional interview became extremely traumatizing for me (which is probably compounded by Rejection Sensitive Dysphoria that comes with ADHD). It's been almost a year since I got my job and I still worry about interviewing again in the future.
For me, being a woman in tech with ADHD makes it really hard to walk into an interview feeling confident. It's intimidating to be obviously different then the vast majority of my interviewers and I'm so worried if I'll keep my train of thought and remember things or talk in a way that makes sense. And it spirals into me worrying if I make women in tech look bad because so many people seem to believe women are inherently worse.
I've noticed that exactly. IQ presumably is something inherent.. But personal experience shows that technical interviews have nothing to do with IQ. They can be easily solved by creating a dynamic programming problem pattern matcher in your brain, which is what I created after doing 300 leetcode problems. I did not become smarter in general or at coding at all, I just got good at mapping problems to binary trees or dynamic programming or string search.
I don't get how IQ could be considered inherent. It's a test, so results are defined by talent and training. A person who spent a month solving the types of problems present in an IQ test will certainly get a higher score. By, the way, you did become better at coding by solving Leetcode problems, cause now you can easily spot them in the wild and use a proper algorithm instead of some naive solution.
aren't IQ tests the same?
You just learn how to solve some kind of problems (series of images, series of numbers and so on) etc.
This obviously does not make you "smarter", it just makes you good at solving some kind of puzzles.
But "smart" is ill defined and vague anyway..
That's the reason all real IQ tests (not the ones you find on the internet, the real ones) are secret/closely guarded: Knowing the tests beforehand skews the results.
So: yes, it is the same in that way and has been theorized as one of the reasons for the Flynn effect. Afaik there's also research into new types of IQ tests to mitigate this all the time, but a good IQ test has higher requirements than the typical interviewing test, so it takes longer.
It's really not. Knowing how to solve those leetcode questions has not improved my professional life one bit. I am not a better programmer.. In fact it probably made me worse. At my work I pretty much never deal with the kind of coding that leetcode requires. I'm more interested in writing larger scale software rather than throw away toy problems which completely ignore code quality or maintainability.
In fact, I'm not even good at solving leetcode problems that well so I wasn't able to get a job at the big N companies...although it did help with passing the interview at one of the smaller companies.
It depends- maybe if people devoted enough time to cramming algorithmic problems (more than they did in their CS classes, even) could they have memorized enough solutions and seen the patterns enough to be able to master this type of technical interview? One wonders, can that rote memorization + pattern machine process be trained into anyone- meaning that this is not a measure of intelligence but of brute force, like memorizing Rubik's Cube solutions?
And then the practical concern arises- does that actually make anyone a better programmer, and benefit employers?
There is a good argument that the technical interview success is much more strongly correlated to the number of hours spent practicing Leetcode-type questions than to actual cognitive ability.