I am reproducing the article written by Mr Arun
JEE 2010: What Went Wrong?
Arun S Moharir
IIT Bombay
‘What Went Wrong?’ is a famous book written by a celebrated author (Trevor Kletz) presenting minute-by-minute account of major industrial accidents, including
JEE 2010 was conducted at over 1000 centres in
Setting up the actual questions and formulating their possible answers involve domain knowledge and a healthy teacher-like attitude on the part of the paper setters. Given the quality of science faculty IITs have within their large family, this expertise is possibly something easily available. Coming up with the question paper pattern in terms of types of questions, weights assigned to questions in terms of marks for a correct answer and negative marks for a wrong answer, etc. is other expertise the examination conductors have to summon. Innovative formats are considered essential to break the monotony of the examination pattern. It has been an internal judgment of IITs that frequent change in the format is essential to ensure that students with innate PCM competence stand a better chance at selection than those who are heavily coached by the large coaching industry JEE has given rise to over the years. Innovative patterns have been a forte of JEE of late and the patterns it churns out find their way in other competitive examinations too. After all, JEE has been a deserving role model. This frequent change of patterns and the surprise element in each JEE calls upon another skill at articulating the instructions for the candidates so that they can understand the paper setters’ mind and evaluation pattern in quick time, decide their strategy and get-set-go. Although JEE does not test the language skills unlike other competitive examinations such as BITSAT, there is an implicit test of the same in reading and understanding these instructions. In all fairness to the candidates, JEE has always given the candidates extra 15 minutes to read the instructions. The instructions are visible to them from their paper sets distributed 15 minutes prior to the scheduled start of the examination. What is not visible before the start are only the actual questions. The examinees are kids with all of 16-17 years experience at real life, and it is the first, and unfortunately the fiercest, competitive examination of their life. IITs have by far been very sensitive to this fact and genuinely caring in their approach to this tender group.
Another aspect of the JEE is its humongous logistics. This aspect is invisible to the outside world. IITs have been managing the show extremely well with utmost diligence for all these years. For those involved in the conduct of JEE, the faculty and staff of IITs, it is an annual responsibility to be discharged to the best of their abilities. Concessions are neither sought, nor given. This call of duty is in no small measure responsible for the success of the system for such an unenviable period. Except for one occasion when the paper was leaked (IITs promptly cancelled the examination), JEE has an unblemished record.
The system so well designed, tested and run for several years suddenly developed multiple organ failure in 2010. It is in ICU and certainly not out of danger as you read this paper. Diagnosis, and a little prognosis, is all those waiting outside can do. Of course, one can suggest a treatment course and also pray for the best. This is the limited scope of this article.
The first hint of the problem in JEE 2010 came within minutes of the start of examination at 0900 hrs on April 11. Students found that the ORS (Optical Response Sheet) on which answers are to be given had a mismatch with the question paper. While the question paper had questions numbered 30-58 for Mathematics and 59-87 for Physics, the ORS showed 30-58 for Physics and 59-87 for Mathematics. The candidates who noticed the discrepancy asked for clarifications. However, the same must come from the organizing IIT, which was IIT Madras this year. With calls coming from all over
The second paper also had similar misprint. However, instructions were in place by then. The panic the students went through in the first paper certainly must have affected their performance in the second paper, as expressed by many students in the media.
IITs send centre representatives to all centres and most described as unprecedented the chaos they themselves had to go through. Many feel that it was a shame and some feel that it affected the fairness of the examination itself. IIT Madras, in its first reaction, termed it as a ‘minor’ printing error. However, as the gravity of the situation sank in, the damage control exercise began. In its remedial measures published on May 2, the blunder takes the cake and is treated as the most major discrepancy in JEE 2010 conduct. IITs propose to correct all papers by two methods; first assuming the subject order on ORS as CPM and then again as CMP. Whatever the higher score a candidate gets will be treated as his/her legitimate score. The remedy is an admission of the serious impact the misprint can have on the fortunes of a candidate. Unfortunately, the remedy could be worse than the disease. It threatens to bring into the JEE results a lottery element with several surprising winners. The surprising (and also surprised) winners threaten to drive away a sizable number of those who could have made the grade in a fair examination without such an unbecoming blunder.
The blunder is such that it has no solution except admitting it and conducting a re-examination. The blunder occurred because of the lack of diligence on the part of the organizing IIT. For security reasons, other IITs are not in the loop when it comes to proof reading. The order mismatch apparently happened due to copy-paste habit and casualness in checking such a vital part of the paper and ORS. The damage caused is irreversible. The organizers tried to downplay it as a minor error and projecting a factually incorrect picture that all centers were informed of the correction within minutes of the start of the examination.
Another aspect which seems to have missed due attention is the possible fouling of the ORS due to erasing that was required. The ORS is machine-checked. The machines are very sensitive to any blot or unexpected dark spot anywhere on the ORS. Even in ordinary times, the machine rejects about 10% ORS as ambiguous due to inadequately darkened bubbles or undecipherable dark spots on ORS. These answer sheets are then manually corrected by the IIT faculty. With erasure of answers many students were required to do during the examination, the ORS percentage which will need to be manually checked will go up significantly.
The double checking of the ORS proposed as a remedial measure by IITs also needs some elaboration. A small metaphor in the inset with this article would help the reader to appreciate quickly what it could mean. More technical discussion relevant to this blunder in JEE 2010 follows.
A small exercise can reveal the magnitude of impact the misprint can cause. JEE-2010 answers are now available (not officially from IITs but from several coaching classes). Consider only paper 1 and get answers for Physics and Mathematics. Mark these on an ORS. You expect a full score, 84 in each subject in a fair examination. Now correct the same ORS with answers for Physics used to correct the Mathematics part and vice versa. This score is what the candidate who marked the answers in wrong part of ORS will get. The total score is just about 5 this time. This shows that the misprint can cause a genuine genius to get 5 instead of 166 marks. The misprint is not a minor error, but possibly a mountain-sized blunder.
If a deserving candidate can get his gains wiped out by the misprint, a non-deserving candidate similarly holds a chance to get his minimal score blown up significantly if he is also given the advantage of checking ORS in two ways and counting the maximum score as legitimate score. The swing of fortunes may not be as dramatic as above, but the probability is finite and not completely ruled out.
While a newspaper article is not a place for discussing mathematics, statistics and probability theory associated with judging the impact of IITs proposed double evaluation process, it appears that the proposed solution will corrupt the merit list significantly. Use of functions such as ‘maximum of the two quantities’ or ‘absolute value of a quantity’ etc. has a potential to fold the distribution function locally as well as overall. The proposed procedure will mix-up the genuine order and will make a mockery of the examination conducted with the sole purpose of ranking people as fairly as possible. The rank decides the IIT and the discipline an aspirant can get. Rank is what the aspirants strive for. Can it be decided by a childish method proposed by IITs? Can a responsible institution be ramming such a solution down the throat of a nation?
Another possibility which cannot be ignored is as follows. It is inherent in the proposed method that it can only increase the marks of any candidate over and above what he/she deserves. For those towards the tail of the merit list where the actual scores could be in the negative zone, this can cause statistically significant rise in their scores. This will raise the overall subject-wise cut-off levels, which could cause elimination of many deserving candidates. The high cut-off has indeed caused high scoring candidates being eliminated in the past. There is proven record in front of the courts that a candidate scoring 280 odd in an earlier JEE was eliminated while someone with an aggregate of 150 odd got in because of questionable subject-wise cut-offs. With the proposed method certainly bound to raise the cut-offs, genuine candidates who would expect to be in the merit list could find themselves rejected.
IITs should study their proposal and its implications systematically. Most IITs also act as centers for JEE. The corrective announcement about the ORS misprint must have been made at such centers immediately after the discrepancy was noticed. These centers thus must have escaped the impact of the misprint. JEE administration should take one such center as a sample, correct the papers as per the two methods and see the impact on the genuine merit list due to the suggested procedure. The findings would be revealing. A centre normally has 400-600 candidates and this is a statistically significant sample to test the proposed method of double correction. The findings should be shared with all, if IITs care for transparency.
Any post-examination measure cannot eliminate the full impact of the blunder. Doubts will remain and at least some will suffer for no fault of theirs and some will gain for no efforts on their parts. Luck will penetrate the competitive domain and make it corrupt. Re-examination alone can do justice to all, whatever be the logistical hurdles. JEE administration is showing no signs of any scientific temperament in their decisions and is going possibly with only one point agenda, to protect its people at the cost of the future of hundreds of thousands of students.
The second problem became apparent as the candidates came out of the examination halls at 12 noon. IITs allowed the candidates to take away the question papers this time. The cat was thus out and the question paper 1 went under intense scrutiny of thousands of people. Instructions for Section II in this paper drew attention immediately. The pattern was truly jaw-dropping.
The questions in this section had one or more correct answers and the candidates were to mark some or all of the correct answers. It was clearly mentioned that the candidates would get partial credit even if they mark only some of the right answers. It was also added that there is no negative marking, i.e. the candidates would not lose any marks for darkening a bubble against a wrong answer.
Clarity of instructions is very important in any examination. Most competitive examination actually print sample questions and answers to elaborate the answering method and evaluation strategy. IITs chose to be miserly with words. The written instructions clearly invited the candidates to try without the fear of penalty. They gain if they see the correct answers, but do not lose if they happen to mark a wrong answer as right.
The instructions left a gaping loophole, which was pointed out by learned professors the next day. The instructions meant that if a candidate marks all four answers as correct, he gets credit for the correct answers and cannot lose anything for marking the wrong answers. He thus scores full marks for the question. This is the perfect, the legal and the only meaning of the instructions.
There were 15 such questions, each of 3 marks. 45 marks were thus available to the candidates for grab without even reading the questions. They just have to darken all available bubbles. When this was pointed out, JEE management’s reaction was again inadequate. The claim was rubbished initially. However, as internal inputs must have suggested subsequently that the claim is indeed what the instructions mean, a cover-up operation began. A totally different evaluation strategy was proposed out of nowhere. It was said that the candidate would get partial or full credit only if the number of answers she marks as correct out of the given options is not more than the number of actually correct answers. The organizers expected to wriggle out of the tight corner by this devious and illegal explanation. Unfortunately, this interpretation (not supported even remotely by the printed instructions) is also equally, or even more, problematic, as the following would suggest. So far, the media has solely focused on the possibility of someone undeservingly getting full marks by marking all the bubbles. The following possibility with the suggested alternative interpretation by IITM is even worse.
Consider only one question in this section. It has four options A, B, C, D. Out of these, only two are correct, say A and C.
A candidate marking A and B will get 1.5 marks because he could identify one of the two correct options (namely A). He marked a wrong option (B) also as correct. He would not lose for that because there is no negative marking.
A candidate marking A, B and C as correct options gets zero. Actually he also made the same mistake as the first candidate, that of marking B as a correct answer. However, he could identify both the correct answers (A, C). He is thus better than the first candidate in all respects. However, he marked three answers as correct while the post-examination evaluation procedure restricts credit only to those who do not darken more than two bubbles in this case. This better candidate thus effectively got 3 marks for identifying both the correct answers, but lost all three for marking a wrong answer as well. He was thus penalized with three marks, even while the instructions clearly state that there is no negative marking. The question effectively had 100% negative marking for him and other genuine aspirants like him.
The logic can be extended to questions with 1, 3 or all 4 answers correct. The gist of the matter is that these questions with the new interpretation by IITM have implicit negative marking. The negative marking can be 1, 1.5 or 3 marks for each wrong answer depending on whether 3, 2 or 1 answers are correct respectively and depending on how many correct answers a candidate was able to see. The evaluation procedure puts a better candidate at a disadvantage and the best candidate can lose all he gained by identifying all correct answers because he marked one wrong answer as also correct. The procedure supports less gifted candidates more and has the potential to penalize a good candidate by up to 45 marks, the maximum score possible.
In their poor efforts to rubbish the legal claim that the instructions allow a clever candidate to gain 45 marks by marking all bubbles, JEE management not only took liberty to change the rules of the game after the game was over, but devised a rule which now can cause a loss of 45 marks to a candidate who actually reads all questions and attempts them by giving his time at the cost of other questions. His correct answers deserve 45 marks, but he gets zero. Evaluation with new interpretation offered by the JEE Chairman is worse than the disease in this case also. While the original sin allows an undeserving candidate to gain, the subsequent sinful interpretation makes a deserving candidate lose.
This anomaly has not been discussed in media much possibly because of the feeling that JEE managers have decided to go to any extent to avoid re-examination. There is a sense of frustration and helplessness at the attitude displayed by JEE management. However, the anomaly in the new interpretation was brought to the notice of the JAB before their fateful meeting on May 2, which prepared and announced corrective measures for the blunders. So irrefutable is the added damage potential of the new interpretation, that IITs seem to have smugly avoided any comment on it by saying that the issue regarding the Section II, Paper 1 questions is ‘immaterial’. Actually, JAB members certainly see that any remedial action at this stage is ‘immaterial’ and that JEE 2010 has become a farce with this blunder in question format and the printed instructions. How long they insist on ‘no re-examination’ line is to be seen. But, these are institutes where honesty and integrity used to matter. It is hoped that the acceptance of irreparable blunders in JEE 2010 will come sooner than later and re-examination will be ordered. Students should keep warm with their preparation as there is no other just solution but the re-examination for this JEE.
The third shock was delivered publicly at 5 PM on April 11th, as the hassled and confused JEE candidates trooped out of the examination centers with the second paper in their hands. Section IV in this paper had again a blunder similar to the Section II in Paper 1. ‘Partial credit with no negative marking’ type questions allow 48 marks to be earned by darkening all bubbles. IITs will resort to re-interpretation and leave a potential penalty of 48 marks to deserving candidates. These questions worth 48 marks together with earlier questions with 45 marks add up to a blunder affecting 93 marks. If written instructions are followed, undeserving candidates can score 93 free marks in few minutes without reading the questions. If post-examination interpretation by IITs is followed, a genuine and a genius candidate can lose 93 well-deserved marks, that too after having devoted time to these questions and actually having used his PCM knowledge. What is better? We should also remember that 95% of the JEE merit list is crammed in about 90 odd marks, the same as the impact range of these two format errors.
Another error affecting part of the students also came to light. To minimize malpractices, IITs give 10 different sets of the papers. These are different only in the order of the questions within the paper. One of the sets had a problem in the sense that question number 44 was not printed at all. 10% of the students thus did not get to see and attempt this question. In proportion to other shortcomings, which have the magnitude of Himalayan blunders, this can be deemed as a ‘mistake’ and IITs are right in giving affected students advantage by proportionately scaling up their marks. Only small hitch here is what this scaling would mean for those with overall negative score. Will there score become further negative? But, that now sounds a petty issue as it does not affect the tip of the pyramid which matters in an examination which boasts of selecting 1-2% of the candidates. It is like worrying about common cold when the patient is on a ventilator.
The basket of woes seems to have something more as well. Some papers seem to have missed out not just one question, but a full two pages were illegible. Candidates will get advantage of scaling up here as well. It is also a mistake which does not do JEE proud, and has a potential to hurt prospects of good candidates. Who knows, the two pages could be having easier questions where the candidate could have scored proportionately much better than other questions on legible pages. But this also sounds like creating a fuss on relatively minor issues as compared to major blunders JEE-2010 managed to produce. JAB devoted 50% of the remedial measures to these minor mistakes in their response, even as they found other blunders ‘immaterial’.
Another blunder came to light on the next day. The Hindi version of Paper 2 showed 3 marks against questions which were actually of 8 marks worth each. Questions worth 48 marks in this paper were thus shown as worth 18 marks only in the Hindi version. As discussed earlier, students attempt questions as per the weight (marks assigned to correct answer) and negative marking. To manage time and to maximize gains, they might have given less importance to these questions and focused their attention on other questions. These questions would obviously take more time to crack. They would have decided that a question worth 3 marks is not worth giving more than 3-4 minutes. When they found it difficult to arrive at any solution in this reasonable time, they would have just moved on thinking that the loss is just 3 marks. About 20% of the students reportedly give their examination in Hindi, our national language. It is something to be proud of. These many students would suffer. This blunder is no smaller than the earlier three and the damage is also irreversible. In their response, JAB curtly say that the weight of the questions would be restored to 8 instead of 3. That everybody knows. But how does it do justice to the affected students?
Considering the number and type of anomalies, it is clear that the JEE management was incompetent and careless this time. There can be no second judgment on them. It is likely that there are other mistakes in this JEE in the knowledge of IITs. What came out in public is based on what the public saw. For example, IITs have not published correct answers, although it was expected. What faith one can have on a system which tries to cover-up so brazenly? What if the answers the paper setters think are right are actually wrong or if some answer which is deemed to be wrong is actually right? It will change the score of the candidates. Such mistakes have occurred in the past. May be, IITs do not want to disclose answers as yet to avoid further damage to JEE’s reputation.
JEE 2010: What Went Wrong?
JEE 2010 deserves to be scrapped on any one of the following grounds because the damage done cannot be reversed.
- Question Paper – ORS mismatch
- Post examination re-interpretation of evaluation procedure for 45 marks questions in Section II in Paper 1 and/or 48 marks in section IV of paper 2.
- Hindi-English Question Paper mismatch, adversely affecting the prospects of Hindi medium students
Such an action is important to restore the credibility of the JEE system as well as that of the IITs. IITs should be forthright, admit that there were irrecoverable blunders and also that they have tried to cover-up. What were human errors and hence condonable, have become criminal offences because of the subsequent considered manipulations and cover-up attempts. There should be an investigation and fixing of responsibilities. This is necessary to root out the cancer that seems to have found place in these institutions known for zero-tolerance to academic manipulations. Our world-wide reputation can be affected as everyone is watching.
At the same time, the IIT-bashing seen on all fora is quite disturbing, at least for those who have been associated with these institutions. IITs are not all about JEE. There is so much quality and sincerity in many things that they do that the summary judgment on their motives, integrity and caliber based on this aberration may be unfair. Certainly, IITs have put up an incompetent show and have shown insensitivity towards the students and contempt for fairness this time. Surely, they themselves are ashamed. Please forgive them for this, one last time. Do not discount them. They have too much merit to be rejected as mediocre in the scheme of things of this country, or even the world. IITs also should reciprocate by ensuring that they do not give away everything built over 50 years by standing on hollow prestige and denying the stark truth of JEE-2010. The people, the government, the judiciary, anybody for that matter, should support such admission of guilt and offer amnesty to JEE management so that aspirants soon get a fair chance in a calmer atmosphere.
