The "standardized test" is an artifact of academia. It isn't worth that much to academics to evaluate prospective students, so they do it in a cheap, scalable, low-quality way.
Companies that employ white-collar workers typically invest much *more* per interview than a standardized test. They generally do not want standardized tests because they perceive their own hiring processes as a more expensive but superior way of evaluating prospective employees.
It's very possible that companies in some field have a generally bad system of interviewing, but it's hard to cut through all the noise because 99% of people who complain about the interview process on the internet are people who are bitter that they didn't get some job.
Yes. A better model is a company that offers screening-as-a-service for employers. That way employers can fish in a larger talent pool. The screening-as-a-service company interviews candidates in a resume-blind way, utilizing economies of scale (e.g. AI-administered screens). Hiring managers can "subscribe" in order to get access to a pool of scored and pre-screened candidates.
There have been some companies that do this, like triplebyte, but it hasn’t been successful. I think the problem is that for the best companies, this is a core competency, one you don’t want to outsource. So the employers end up being lower-tier ones, and then it’s hard to convince workers to participate.
Insofar as Triplebyte “failed”, I don’t think it’s because the idea is fundamentally bad. I believe they got some pretty serious companies partnering with them.
This is called "overrating your own skill/topic of expertise." Excel isn't even the most universally used app within the Microsoft Office catalog, let alone all of business, and most of the white collar professionals who use excel only need to use its most basic features and never use it for math above a high school algebra level. An excel test might make sense for an actuary or finance professional, but everyone else should be outsourcing financial/statistical analysis to those professionals. Possibly, being especially good at excel is negatively correlated with HR, management, marketing, sales, etc potential. It might even be a poor signal for finance potential beyond a minimum threshold, since finance success is about understanding the business and creatively enabling other departments, not speedy spreadsheet creation.
This sounds like it would be good right up until it became commonplace, then it would suffer from the same issue that Leetcode does in tech, i.e. people widely practice and cram for a particular skillset that they'll be asked in interviews. The competition keeps getting better so the questions get harder. Eventually they're hard enough where they're almost completely disconnected from what people do in the job on a regular basis, and now everyone has to waste their time memorizing pointless features they'll rarely/never use.
As mentioned by the author, this sort of thing is standard practice in tech. One problem in that world is that interviewers give problems that are in their own category of coding puzzles. These are distinct from what, for example, a software engineer would do on a day-to-day basis. So someone can get good at solving these problems without necessarily being good at the main tasks.
This might be changing a bit; it can also be mitigated by asking questions that are directly related to day-to-day tasks to test for relevant knowledge. But still, it doesn't seem that interviewers are optimizing purely for relevant knowledge/competency.
Here are some reasons that I think can explain this:
1) hiring managers want to hire people they like. They're willing to sacrifice some day 1-readiness for this.
2) as long as a candidate is generally competent for the field in question, the particulars of the job can be taught. Filtering for these competencies too early, especially if they're reasonably easy to teach to someone with foundational knowledge, would limit the pool of candidates to a possibly counterproductive degree.
3) there are competitive forces with which hiring decisions trend toward greater accuracy, but they're not so intense or so precisely relevant to people doing the hiring that they have to hire the most competent person every time. This allows hiring managers to appeal to their own preferences, whether that be in how they conduct interviews, whom they hire, etc. (i.e., justifying point 1).
4) perhaps switching to rigorous testing would be a turnoff to potential employees (possibly even some of the stronger ones), and a lot of employers aren't in a position to alienate a large pool of potential candidates. This is especially true if the gains to be had from more technical hiring practices are small.
I would definitely want to know if anyone in my team was still using vlookup, since XLOOKUP must be 5 years old now (and also replaces index match in all cases that I need it for).
I think it is a good idea but very difficult to operationalise. For example, you mention discount rates issues that I am quite certain many people who are quite proficient with excel would fail because they never come across them, and because only a rather limited fraction of all humans can understand discount rates.
I was on the job market recently, which made me think about this a lot. I do wonder why companies do not use these sort of tests more often, at least as an initial filter. I heard company recruiters will literally just find the first x number of qualified candidates, then send them an interview request. So if you are an early applicant, your odds are much better. For jobs that already had 100s of applicants, it truly seemed like a complete waste of time to apply. It seems that having an initial filter mechanism would be useful in these cases of high applicant volume.
I do recognize that there is likely huge variance in the recruitment process. Some companies may have very effective processes while others have less effective ones.
Another thought is that I know companies are technically allowed to use AI in filtering applications, but there is a catch. You might get in trouble for discrimination, so you have to be very careful in your application of AI. I wonder what percent of companies are currently using AI in their resume screening.
The following source documents a case of an AI discrimination allegation (https://www.heplerbroom.com/blog/legal-implications-ai-in-hiring-process), stating that a job applicant brought an employment discrimination action against Workday, Inc. Then Mobley, "Workday brought a motion to dismiss the complaint, arguing that Mobley failed to state a claim. The District Court denied Workday’s motion to dismiss...The Court further stated, "Given Workday’s allegedly crucial role in deciding which applicants can get their ‘foot in the door’ for an interview, Workday’s tools are engaged in conduct that is at the heart of equal access to employment opportunities."
This case highlights the potential for AI to unintentionally discriminate against candidates. It further indicates that companies that delegate traditional hiring functions to algorithmic decision-making tools may face potential legal exposure for even unintentional discrimination."
I've heard the same thing, from recruiters no less, about simply looking at the first X candidates. This undermines reasons 1 and 2 from my previous post, but it supports 3.
One counter then might be to make some technical question(s) a part of the application process, or a part of the automated interview process. This already happens in tech (i.e., you're sent a timed coding question to do on your own). But this has at least two effects: one, people can easily cheat -- even if you're doing things like using a webcam, monitoring keystrokes, etc; second, it adds friction and some people just won't bother moving forward (reason 4).
You might argue that the second effect is actually positive since there are more applicants than can actually be processed by humans (recruiters). That might be true.
So maybe it's that, as mentioned, hiring managers don't have extreme pressure to always hire the most technically competent person, and recruiters have even less pressure on them to filter for the most competent people (at least not with a great deal of precision).
So, yeah, there probably is some entrepreneurial opportunity here for improvement. But I think there are a lot of other compounding factors, like discrimination laws that you mentioned, or perhaps labour laws and sticky wage effects creating a surplus of applicants.
What you say about the technical tests totally makes sense. It is easier said than done. I also agree that there are other soft skills that recruiters understandably care a lot about, if not more than the hard skills. And it isn't obvious a priori what is the optimal mix.
Don't forget VBA. I would hire anyone who can do rest API call to OpenAI get embedding for a text. Then do a vector semantic similarity search using TEXTSPLIT/VALUE and INDEX/MATCH/MAX.
Some companies do use this. For everyone I’ve hired, I’ve administered excel tests. And I learned I’m not alone as I interviewed for two positions last year (one for a chief of staff role and one for an FP&A consulting role) which both administered excel tests, and I’ve had a couple hiring managers ask what my favorite excel formula was as a way to probe how familiar I was with the tool (ironically, one had never heard of the formula I chose to give: LET)
I ended up landing in management consulting for MBB, but I’m very interested by the fact that MBB does not directly screen for excel ability, as this is the industry where lack of facility with excel makes you functionally unable to do the job.
My best sense of this choice is that: 1) relative to other industries, MBB is relatively immune to bad hires because the up or out setup so quickly forces out people who end up being mishires
2) by directly screening on quant performance on standardized tests, there is some reason to expect new hires can pick up excel
3) as a multifaceted job, over indexing on one skill in the hiring process could hypothetically select for a workforce who is good with data but completely unable to navigate client dynamics (which would also make new hires functionally unable to perform the job).
None of the above though says excel testing shouldn’t exist alongside the battery of assessments the firms administer
Good idea! I'm sure many finance businesses are doing this already, but I work in health care and I can tell you that excel skills are far from consistent in jobs where tabular data is core to the work
Let’s just use a standard IQ test or if there’s some concern about literacy, there are non-verbal IQ tests that are used with young children and other non-readers.
Yep, between IQ and teaming during Q course, they combine to filter pretty good. Not perfect as I had some real shit birds still occasionally, the sort that generally was instantly promoted to the B team and then shortly thereafter HHC/D, but overall you definitely seen the stark intelligence difference when dealing with the regular army. Too bad they mediocritized SF under Obama for the sake of wokeness.
That’s sad to hear. I had President Reagan during my service. My closest team members all had stellar military careers: one ended up graduating from West Point (he tested into their prep school), others joined Special Operations and went on to work off the grid, and still others like myself found our way in the civilian world and became well educated (when that still meant something positive). As you indicated, IQ isn’t something that guarantees you success in the military or life in general, but if you don’t have it, you can’t lean on other abilities when lives are at stake (especially your own).
Yeah during GWOT some people cried about the optempo, something no one cared about when I was deployed for forty out of forty-eight months in the 90s but hey, different breed of soldiers then. So SF expanded the TOO greatly giving each group an additional battalion and vastly expanded the NGuard groups basically overnight instead of via the standard training pipeline under the flimsy guise of "recruitment" which meant allowing women, shims, loners, and reducing minimum qualifications standards.
Also you can't team out anyone anymore unless they are a white straight male because of "diversity" so kicking out the obese black girl with an attitude who can't even pass her PTQ but still gets through the Q course with a "medical profile" is verboten.
I don't know much about the actuarial industry, and I can imagine that Spain is behind on hiring practices, but testing/evaluations like this are widespread in tech in the United States. I would be a bit surprised to hear that other such professional fields are not implementing evaluations here in the US.
"Companies don't have a standard test for these skills. Coding tests are used by "tech" companies but as far as I know they're non-standard"
I am nowhere near his target audience, but I get the impression he wants standardized tests several times a year. Not customized hiring-and-done tests.
The "standardized test" is an artifact of academia. It isn't worth that much to academics to evaluate prospective students, so they do it in a cheap, scalable, low-quality way.
Companies that employ white-collar workers typically invest much *more* per interview than a standardized test. They generally do not want standardized tests because they perceive their own hiring processes as a more expensive but superior way of evaluating prospective employees.
It's very possible that companies in some field have a generally bad system of interviewing, but it's hard to cut through all the noise because 99% of people who complain about the interview process on the internet are people who are bitter that they didn't get some job.
Yes. A better model is a company that offers screening-as-a-service for employers. That way employers can fish in a larger talent pool. The screening-as-a-service company interviews candidates in a resume-blind way, utilizing economies of scale (e.g. AI-administered screens). Hiring managers can "subscribe" in order to get access to a pool of scored and pre-screened candidates.
There have been some companies that do this, like triplebyte, but it hasn’t been successful. I think the problem is that for the best companies, this is a core competency, one you don’t want to outsource. So the employers end up being lower-tier ones, and then it’s hard to convince workers to participate.
Insofar as Triplebyte “failed”, I don’t think it’s because the idea is fundamentally bad. I believe they got some pretty serious companies partnering with them.
https://techcrunch.com/2023/03/16/technical-recruitment-platform-karat-snaps-up-triplebyte-to-add-ai-based-quizzes-for-engineers/
This is called "overrating your own skill/topic of expertise." Excel isn't even the most universally used app within the Microsoft Office catalog, let alone all of business, and most of the white collar professionals who use excel only need to use its most basic features and never use it for math above a high school algebra level. An excel test might make sense for an actuary or finance professional, but everyone else should be outsourcing financial/statistical analysis to those professionals. Possibly, being especially good at excel is negatively correlated with HR, management, marketing, sales, etc potential. It might even be a poor signal for finance potential beyond a minimum threshold, since finance success is about understanding the business and creatively enabling other departments, not speedy spreadsheet creation.
If a standard tests were appropriate to determine all of the activities for the job, then the job is a candidate for full automation and elimination.
This sounds like it would be good right up until it became commonplace, then it would suffer from the same issue that Leetcode does in tech, i.e. people widely practice and cram for a particular skillset that they'll be asked in interviews. The competition keeps getting better so the questions get harder. Eventually they're hard enough where they're almost completely disconnected from what people do in the job on a regular basis, and now everyone has to waste their time memorizing pointless features they'll rarely/never use.
As mentioned by the author, this sort of thing is standard practice in tech. One problem in that world is that interviewers give problems that are in their own category of coding puzzles. These are distinct from what, for example, a software engineer would do on a day-to-day basis. So someone can get good at solving these problems without necessarily being good at the main tasks.
This might be changing a bit; it can also be mitigated by asking questions that are directly related to day-to-day tasks to test for relevant knowledge. But still, it doesn't seem that interviewers are optimizing purely for relevant knowledge/competency.
Here are some reasons that I think can explain this:
1) hiring managers want to hire people they like. They're willing to sacrifice some day 1-readiness for this.
2) as long as a candidate is generally competent for the field in question, the particulars of the job can be taught. Filtering for these competencies too early, especially if they're reasonably easy to teach to someone with foundational knowledge, would limit the pool of candidates to a possibly counterproductive degree.
3) there are competitive forces with which hiring decisions trend toward greater accuracy, but they're not so intense or so precisely relevant to people doing the hiring that they have to hire the most competent person every time. This allows hiring managers to appeal to their own preferences, whether that be in how they conduct interviews, whom they hire, etc. (i.e., justifying point 1).
4) perhaps switching to rigorous testing would be a turnoff to potential employees (possibly even some of the stronger ones), and a lot of employers aren't in a position to alienate a large pool of potential candidates. This is especially true if the gains to be had from more technical hiring practices are small.
I would definitely want to know if anyone in my team was still using vlookup, since XLOOKUP must be 5 years old now (and also replaces index match in all cases that I need it for).
I think it is a good idea but very difficult to operationalise. For example, you mention discount rates issues that I am quite certain many people who are quite proficient with excel would fail because they never come across them, and because only a rather limited fraction of all humans can understand discount rates.
I was on the job market recently, which made me think about this a lot. I do wonder why companies do not use these sort of tests more often, at least as an initial filter. I heard company recruiters will literally just find the first x number of qualified candidates, then send them an interview request. So if you are an early applicant, your odds are much better. For jobs that already had 100s of applicants, it truly seemed like a complete waste of time to apply. It seems that having an initial filter mechanism would be useful in these cases of high applicant volume.
I do recognize that there is likely huge variance in the recruitment process. Some companies may have very effective processes while others have less effective ones.
Another thought is that I know companies are technically allowed to use AI in filtering applications, but there is a catch. You might get in trouble for discrimination, so you have to be very careful in your application of AI. I wonder what percent of companies are currently using AI in their resume screening.
The following source documents a case of an AI discrimination allegation (https://www.heplerbroom.com/blog/legal-implications-ai-in-hiring-process), stating that a job applicant brought an employment discrimination action against Workday, Inc. Then Mobley, "Workday brought a motion to dismiss the complaint, arguing that Mobley failed to state a claim. The District Court denied Workday’s motion to dismiss...The Court further stated, "Given Workday’s allegedly crucial role in deciding which applicants can get their ‘foot in the door’ for an interview, Workday’s tools are engaged in conduct that is at the heart of equal access to employment opportunities."
This case highlights the potential for AI to unintentionally discriminate against candidates. It further indicates that companies that delegate traditional hiring functions to algorithmic decision-making tools may face potential legal exposure for even unintentional discrimination."
I've heard the same thing, from recruiters no less, about simply looking at the first X candidates. This undermines reasons 1 and 2 from my previous post, but it supports 3.
One counter then might be to make some technical question(s) a part of the application process, or a part of the automated interview process. This already happens in tech (i.e., you're sent a timed coding question to do on your own). But this has at least two effects: one, people can easily cheat -- even if you're doing things like using a webcam, monitoring keystrokes, etc; second, it adds friction and some people just won't bother moving forward (reason 4).
You might argue that the second effect is actually positive since there are more applicants than can actually be processed by humans (recruiters). That might be true.
So maybe it's that, as mentioned, hiring managers don't have extreme pressure to always hire the most technically competent person, and recruiters have even less pressure on them to filter for the most competent people (at least not with a great deal of precision).
So, yeah, there probably is some entrepreneurial opportunity here for improvement. But I think there are a lot of other compounding factors, like discrimination laws that you mentioned, or perhaps labour laws and sticky wage effects creating a surplus of applicants.
What you say about the technical tests totally makes sense. It is easier said than done. I also agree that there are other soft skills that recruiters understandably care a lot about, if not more than the hard skills. And it isn't obvious a priori what is the optimal mix.
Don't forget VBA. I would hire anyone who can do rest API call to OpenAI get embedding for a text. Then do a vector semantic similarity search using TEXTSPLIT/VALUE and INDEX/MATCH/MAX.
Love the idea.
Some companies do use this. For everyone I’ve hired, I’ve administered excel tests. And I learned I’m not alone as I interviewed for two positions last year (one for a chief of staff role and one for an FP&A consulting role) which both administered excel tests, and I’ve had a couple hiring managers ask what my favorite excel formula was as a way to probe how familiar I was with the tool (ironically, one had never heard of the formula I chose to give: LET)
I ended up landing in management consulting for MBB, but I’m very interested by the fact that MBB does not directly screen for excel ability, as this is the industry where lack of facility with excel makes you functionally unable to do the job.
My best sense of this choice is that: 1) relative to other industries, MBB is relatively immune to bad hires because the up or out setup so quickly forces out people who end up being mishires
2) by directly screening on quant performance on standardized tests, there is some reason to expect new hires can pick up excel
3) as a multifaceted job, over indexing on one skill in the hiring process could hypothetically select for a workforce who is good with data but completely unable to navigate client dynamics (which would also make new hires functionally unable to perform the job).
None of the above though says excel testing shouldn’t exist alongside the battery of assessments the firms administer
I took SATs at 17 and scored above 90th percentile. 40 years later, took GRE and ... same.
That's just me, and now (60 years later), I'm senile.
I STILL like Excel, though, and fiddle with it every chance I get.
Good idea! I'm sure many finance businesses are doing this already, but I work in health care and I can tell you that excel skills are far from consistent in jobs where tabular data is core to the work
Let’s just use a standard IQ test or if there’s some concern about literacy, there are non-verbal IQ tests that are used with young children and other non-readers.
Trump just signed EO 14281, so we might see a real-world test of whether IQ tests are useful for hiring.
I was in the U. S. Special Forces (where IQ tests must meet a specific standard) and I can assure you that they are predictive of mental abilities.
Yep, between IQ and teaming during Q course, they combine to filter pretty good. Not perfect as I had some real shit birds still occasionally, the sort that generally was instantly promoted to the B team and then shortly thereafter HHC/D, but overall you definitely seen the stark intelligence difference when dealing with the regular army. Too bad they mediocritized SF under Obama for the sake of wokeness.
That’s sad to hear. I had President Reagan during my service. My closest team members all had stellar military careers: one ended up graduating from West Point (he tested into their prep school), others joined Special Operations and went on to work off the grid, and still others like myself found our way in the civilian world and became well educated (when that still meant something positive). As you indicated, IQ isn’t something that guarantees you success in the military or life in general, but if you don’t have it, you can’t lean on other abilities when lives are at stake (especially your own).
Yeah during GWOT some people cried about the optempo, something no one cared about when I was deployed for forty out of forty-eight months in the 90s but hey, different breed of soldiers then. So SF expanded the TOO greatly giving each group an additional battalion and vastly expanded the NGuard groups basically overnight instead of via the standard training pipeline under the flimsy guise of "recruitment" which meant allowing women, shims, loners, and reducing minimum qualifications standards.
Also you can't team out anyone anymore unless they are a white straight male because of "diversity" so kicking out the obese black girl with an attitude who can't even pass her PTQ but still gets through the Q course with a "medical profile" is verboten.
I had no idea how bad things were.
I don't know much about the actuarial industry, and I can imagine that Spain is behind on hiring practices, but testing/evaluations like this are widespread in tech in the United States. I would be a bit surprised to hear that other such professional fields are not implementing evaluations here in the US.
Ah, but that doesn't match his criteria:
"Companies don't have a standard test for these skills. Coding tests are used by "tech" companies but as far as I know they're non-standard"
I am nowhere near his target audience, but I get the impression he wants standardized tests several times a year. Not customized hiring-and-done tests.