How Do You Check If A Test Is Working?

Several companies use a Test while assessing job applicants. Considering the popularity of Tests in assessing job applicants its important to know how to check if a Test is working. Here are some guidelines ...

1. Does the Job Description Need the Test?

This discussion assumes that while choosing to use the Test, a study of the Job Description was done and the Test was chosen since it measures a required competence.  In case this was not done, its a good idea to do this.  Check if the Test measures a competence mentioned in the Job Description. For instance you could use a C Programming Test and a Spoken English Test if the Job Description says "Requires strong C Programming Skills and excellent spoken communication skills".  This is a step that has to be done while choosing a Test.

2. Job roles are Alive, Job Descriptions are Not!

Assuming the Test has been chosen after looking at the Job Description, its important to revisit the Job Description from time to time. Job roles are alive ... they grow with time, they expand, they change, they require newer competencies. Job roles change just as business realities change. But most documented Job Descriptions are rarely revisited. So its a good idea to revist the documented Job Description from time to time to see if the Job Description accurately reflects the job.  For instance a Job Description that says "requires strong Visual Basic skills" may change to "requires strong ASP.net skills" since customers have migrated to a newer technology. In such  a case using Visual Basic Test for the job role would be pointless. And the Test would certainly not work.

3.  Compare Against Another Measure

Assuming point 1 and point 2 are addressed, the best way to check if a Test is working for you is to compare against another measure. The other measure could be a Technology Interview if the Test was  technology Test or  could be a communication skill check by a communications specialist if the Test is a communications Test. For instance, if you are adminstering a C Programming  Test on job applicants then check

Measure A) How many of the job applicants who pass the Test also pass the Technology Interview your team is conducting? (It should be a minimum of 60%)

Measure B) How many of the job applicants who fail the Test also fail the Technology Interview your team is conducting? (It shoul be a minimum of 80%) 

Keep in mind that ideally, both Measure A and Measure B should be 100%.  This indicates that you have a perfect test that knows EXACTLY what your Technology Interviewers are looking for.  But 100% is almost never achievable. This is because of a variety of reasons

- All Interviewers do not ask the same questions
- All Interviewers do not rate consistently - one interviewer might pass what another interviewer fails
- An Interviewer is influenced by external factors - for instance a job applicant's poor communication skills or body language might influence the interviewer's judgement of the job applicants C Programming skill

Factoring for these measurement errors, a 60%  on A and 80% on B is good. Its a good practice to check these measures month on month to ensure that the Tests are working. Many times, a change in the job role causes Interviewers to ask different questions which will lead to very low percentages on Measure A and Measure B. This clearly points to a change in job role or a problem with the Test.

4.  The Selection Criteria

The effectiveness of a Test is completely dependent on the Selection Criteria - the minimum score on the Test that you consider appropriate for the job. Measure A and Measure B hinge completely on the Selection Criteria. You can be cautious and set the selection criteria high but then you end up rejecting job applicants who may fit the role. You can be optimistic and keep the selection criteria low but then you end up interviewing several job applicants who dont fit the role. The Selection Criteria has to be optimised - fixed at a level where both Measure A and Measure B are maximized. Your Test will not work if you dont have the optimal Selection Criteria.

These steps can help in ensuring that Tests you are using are working.