Last week, we had an opportunity to have a quick phone conversation with Prometric’s Vice President (Solution Services) David A Meissner during his visit to New Delhi about scoring, testing environments and results. Following is an edited excerpt:
Why is it not possible for a candidate to get his raw score from Prometric?
There are a couple of reasons. First, the raw score is at best a misleading figure and can create a significant source of concern without going through the full equating process. Without the equating, the raw score does not mean anything. By knowing the raw score, the candidates could come with the feeling that they deserved a higher score if they focused just on the raw score. Our concern which is also shared by the IIMs is that releasing that information will create more confusion than clarity. Secondly, there is the concern that if we release the raw score information we might end up enabling access to sensitive procedures in the scoring routine, thereby undermining the security of the exam. I would also like to say that in other examinations that use similar methods of equating, it is not an unusual practice to not release the raw score.
Why would it take you more than one-and-half month to release the CAT results, despite computerization of the exam?
The reason why there is a seven-week delay in releasing the result is because we are going to be employing 40 test-forms (question papers). Each test-form has to undergo the equating process in order to ensure that candidates receive scores that are reflective of their performance. That takes several weeks to analyze. To commit to a shorter time would have us run the risk of not releasing accurate scores.
How critical is the public knowledge of questions from prior test slots to the fairness of the exam?
In this particular exam we have gone to great lengths to minimize commonality of content between forms, the exception being the equating items. While creating the test forms we have minimized the opportunity for candidates to harvest items and share with others and get any sort of advantage. We have designed discrete test-forms in a way that knowing items will have no effect on the score of subsequent candidates.
How about cloned items, which are similar questions but with different variable values?
Cloned items, or item models are used to facilitate creation of a large number of test questions. You take a base question and change certain variables to get more questions. When we create cloned items we choose questions that wouldn’t be prone to memorization. However in certain areas such as Data Interpretation, there are only a few basic principles that are being assessed that one might say are prone to memorizing and sharing of content. The questions, which are created by professors of the IIMs and IITs undergo a detailed review process before being kept or discarded. While doing that we have to be careful to the challenges you describe.
After the equating was completed for CAT 2009, by what factor did the easiest test and the most difficult test differ?
I don’t have that information and I’m not sure we can release that to the public.
Is there a provision for a large-scale retest in CAT 2010 if things go wrong again at the same scale as last year?
Yes we have built in contingencies in many ways, such as additional workstations and sessions at every location that will remain on standby in case some workstations fail.