A Pearson VUE team from the USA was in Mumbai today. They are the guys that NMIMS University has contracted out the computer-based NMAT to. Their agenda was to create awareness about computer-based testing (CBT) and resolve perceived myths surrounding it. PaGaLGuY caught up with the team comprising Kathleen Gialluca – Senior Research Scientist, Fiona Collins -Managing Director and Mark Poole – Director, Test Security. Gialluca is a PhD in Psychometrics — the science behind all testing.
Here are some excerpts from our conversation.
India is still not at home with CBT like the West is…
On the contrary, India has taken to CBT much better than expected. Two years ago, there were a lot of problems with implementing CBT but they have been ironed out now. Last year and this year too, one hasn’t heard of any issues. CBT is a global phenomenon running successfully and India is now part of it. There needs to be a better job of showcasing CBT in India. Clearly CBTs have to be planned and designed efficiently to avoid any errors the implementation is most important.
There are other issues to worry about in India, like power problems and that Internet has not made its way to every corner of the country.
Well, let me tell you that the power situation in India is much better than that in Turkey, Greece and even the US. Earlier this year, there was a power crisis in the US and nothing could be done. In that sense India was far more prepared to handle emergencies. India has solutions to everything. There is something called the ‘inverter’ used here… one has not really heard of it in the US. And about the reach of the Internet, may be it has not reached every corner of the country but with India having one of the highest number of mobile users, people here are familiar with technology. CBT is all about clicking and people do it with their mobiles all the time. Facebook, Twitter and blogging is so rampant in India, it is unlikely that a person who is taking a CBT has been left out of all this.
The concept of Psychometrics has still not sunk in with candidates appearing for CBTs. Is there an easy way of explaining the term?
Psychometrics is not a mystery, it is a science. Any confusion is because it has now been introduced into admission tests. India has been using CBTs since the 1990s (Microsoft certification tests. etc). Psychometrics implies that different papers are set for different days and efforts are taken to make sure that every paper has the same level of difficulty. So if you take the exam on different days, the questions will differ but the difficult level will be the same — we equalise using statistics. Questions are passed through layers of different people with different capabilities and only then do they become part of a test paper. Psychometrics is also admissible in the court of law today.
But are tests that use Psychometrics still prone to errors?
Not hugely but yes, errors can happen due to incorrectly written questions. Or our reading and understanding of a question being different from that of candidates could introduce errors. But none of this will affect the final outcome. Because the scoring process will take care of it. We go by how the majority tackles a question and often judge the test by that. And if we know that there is a problem, we keep that in mind while scoring to eventually balance it out.
But test takers on one particular day could find the paper easy and on another day difficult, something that Psychometrics already accounts for.
There are two reasons why a question or an entire paper may be perceived as ‘difficult’. One, because the student himself finds the question difficult and two, the question may indeed be a difficult one. As I said before, we take into account mass references when we analyse the aftermath of a question. Individual capabilities play a big role in deciding the toughness of a paper. For instance a non-engineer may find math difficult, and an engineer may not be too comfortable with verbal ability questions and in some cases some people are just smarter than the others. All these situations need to be looked into before deciding whether a question is difficult or not.
But do you get complaints from test-takers? How do you respond?
Of course we get complaints. Every complaint is taken seriously. Besides the majority method I described earlier, we also run the question through many layers of people and check for anomalies and then take a call.
‘Normalisation’ has come to be a rather ugly word for candidates who take business school related CBTs in India.
I guess by normalisation, what you really mean is standardisation. In a standardised test, the questions, the conditions of administering the test, scoring processes and interpretations are consistent. Difficulty or easiness of questions is standardised across all papers and is considered a fair process. In a standarised test, the outcome for all takers can be compared reliably. As I said earlier, a test can have different implications for a candidate, the results can be different if a candidate is sleepy or if he is not prepared or he is feeling sick or he has some other problems weighing on him.
Why were three attempts allowed for the NMAT?
We realised that there were many reasons why a candidate might fair badly in an exam. He could be ill, a problem in the family, too much traffic or just not having the time to prepare because of work why let his entire study effort go wasted? Lets instead give him a chance to improve. The first year that the NMAT went online, we had a candidate who arrived for the test all injured. He was bleeding in the leg and in no position to take the exam. We asked him to come the next day and take it.
But doesn’t it compromise the quality of the intake? Is a candidate who performs well in the test on the third attempt of equal aptitude as someone who does it in the first?
Lots of international exams allow many attempts like the GMAT and GRE. The idea is to allow the person to better himself. In fact in GMAT, one has to send all the scores what if the people on the other side want to take only one of the attempts in consideration and that is one of your weak attempts? Take exams such as those for doctors or nurses — would you not prefer that they worked hard towards covering their weaknesses and do better in every exam they attempted? It is similar with admission tests. We are okay if a person wants to improve his weak points and go ahead. As far as intellect goes, no two papers are alike so for the candidate it is attempting a brand new paper with new questions.
Does a test coaching institute help to become better at doing well in a CBT?
Actually CBT does not require any coaching at all, only regular practice. I doubt that coaching really helps a person to do better but in India coaching is what most candidates prefer.