The detailed report on the PISA 2009+ cycle of tests including the results for the pilot in India in the states of Tamil Nadu (TN) and Himachal Pradesh (HP) has been published by the Australian Council for Educational Research, which conducted the PISA 2009+ Cycle of tests. The entire database is also accessible online in an interactive manner for slicing and dicing the data (look for QTN and QHP in the countries list).
For perspective and comparison with HP and TN, I have picked the top three in the PISA results - Shanghai-China, South Korea and Finland and also USA as a large country (in geography and population terms) and a few other countries like Brazil, Russia, Mexico, Indonesia, Turkey and Argentina as emerging economies with large populations. Of all these countries, four of them, Finland, Mexico, Turkey and USA are part of the OECD. The OECD average is also provided - it is the arithmetic mean of the estimates for the 34 OECD countries that participated in PISA 2009.
Here're some results based on the report:
Extracts from Table B.2.1: Comparing countries’ performance in reading (p 105)
The last column indicates the % of students having proficiency in reading literacy above the baseline level to participate effectively and productively in life (Table B.2.2, p 109). There are seven proficiency levels in reading (Tables 2.2 to 2.6 p 15-17), ranging from Level 6, the highest level, involving sophisticated reading tasks that are generally only able to be completed by highly competent readers; through to Level 1b, involving elementary tasks that require only very basic reading skills. Level 2 is considered the baseline level of proficiency at which students begin to demonstrate the reading skills that will enable them to participate effectively and productively in life.
Mean S.E. Confidence Interval 5th Percentile S.E. of 5th Percentile 95th Percentile S.E. of 95th Percentile % at or over baseline Himachal Pradesh 317 4 309 - 325 192 6.9 443 8.5 10.8 Tamil Nadu 337 5.5 326 - 347 211 4.5 472 14.4 17.3 OECD Average 493 0.5 492 - 494 332 1 637 0.7 81.2 Shanghai-China 556 2.4 551 - 561 417 5.2 679 3.3 95.9 South Korea 539 3.5 532 - 546 400 7.6 658 3.8 94.2 Finland 536 2.3 531 - 540 382 3.4 666 2.6 91.9 USA 500 3.7 493 - 507 339 4.2 656 5.8 82.4 Brazil 412 2.7 406 - 417 262 3 572 4.6 50.4 Russian Federation 459 3.3 453 - 466 310 5.8 607 5.6 72.6 Indonesia 402 3.7 394 - 409 291 5.8 510 5.8 46.6 Mexico 425 2 421 - 429 281 3.9 557 2.4 59.9 Turkey 464 3.5 457 - 471 325 5.1 596 5.4 75.5 Argentina 398 4.6 389 - 407 209 11.3 568 6.7 48.4
The OECD mean reading score is 493 with a standard deviation of 93. The mean reading scores of both HP and TN are 1.7 standard deviations below the OECD mean. Even the 95th percentile scores in HP and TN are below the OECD mean. HP and TN rank significantly lower than the other emerging economies too.
The other metric of the percentage of students at or above the baseline level of proficiency is abysmally low in HP (11%) and TN (17%), and a serious cause for concern. While the OECD average is 81%, the percentage of students at or above the baseline level in countries like Brazil, Indonesia, Mexico and Argentina is in the 50% range. These countries seem to be far better off than HP and TN.
There may be a reason for this. In response to the question posed to students "What language do you speak at home most of the time?", the options being (1) the language of the test or (2) another language, a substantial majority of students said they spoke the language of the test at home in many countries - Finland (95.63%), Mexico (95.48%), Turkey (95.57%), USA (85.56%), Argentina (95.38%), Brazil (96.72%), Russian Federation (90.12%) and Shanghai-China (98.41%). But the corresponding numbers were much lower for Tamil Nadu (67.53%) and Himachal Pradesh (60.5%). I have only picked a few countries for comparison and have not looked at these numbers for all the other countries. Click here for the data.
One might think the low reading scores for TN and HP may be due to the fact that they were being tested in a language that was not their native language. Apparently the PISA 2009+ tests were administered in English, Hindi and Tamil in India (Source p 6). But surprisingly, the test scores of children in both Tamil Nadu and Himachal Pradesh who said the test language was not their first language were higher than the scores of children who said the test language was their first language!
To a similar question posed to schools "About how many students in <national modal grade for 15-year-olds> in your school have a <first language> that is not <the test language>?", the options being (1) 60% or more, (2) 40% to 60%, (3) 20% to 40%, (4) 10% to 20%, (5) 0% to 10%, only a very small percentage of schools said over 60% of students had a first language that is not the test language in many countries- Finland (0.06%), Mexico (5.22%), USA (5.65%), Argentina (1.48%), Brazil (1.67%), Russian Federation (11.01%). In contrast, the corresponding numbers were far higher for Tamil Nadu (75.46%) and Himachal Pradesh (82.76%). Click here for the data.
Another set of data also need to be examined in this context. The scores of students in those schools that offered significant amount of instruction in the first language to students whose first language was not the test language and the scores of students in those schools that offered some some instruction in school subjects through the first language to students whose first language was not the test language.
I'm unable to understand the relationship between the scores, the first language and the test language - other experts may be able to throw some light. There may be an issue about the medium of instruction and the medium of testing for children in India leading to a comprehension problem. This merits further investigation before the PISA 2012 series of tests are administered.
Extracts from Table B.3.1: Comparing countries’ performance in mathematics (p 121)
The last column indicates the % of students having proficiency in mathematical literacy above the baseline level to participate effectively and productively in life (Table B.3.2, p 124). There are six proficiency levels in mathematical literacy (Table 3.2 p 43) ranging from Level 6 (involving advanced mathematical skills and reasoning) through to Level 1 (involving routine mathematical tasks and procedures). Level 2 is considered the baseline level of proficiency at which students begin to demonstrate the mathematical skills that will enable them to participate effectively and productively in life.
Mean S.E. Confidence Interval 5th Percentile S.E. of 5th Percentile 95th Percentile S.E. of 95th Percentile % at or over baseline Himachal Pradesh 338 4.2 330 - 347 223 6.8 458 8.8 11.9 Tamil Nadu 351 5.1 341 - 361 241 6.6 468 13.8 15.2 OECD Average 496 0.5 495 - 497 343 0.9 643 0.8 78 Brazil 386 2.4 381 - 390 261 3 531 5.9 30.9 Shanghai-China 600 2.8 595 - 606 421 7.1 757 4.6 95.2 South Korea 546 4 538 - 554 397 8.4 689 6.5 91.9 Finland 541 2.2 536 - 545 399 4.4 669 3.6 92.2 USA 487 3.6 480 - 494 337 4.3 637 5.9 76.6 Russian Federation 468 3.3 461 - 474 329 5.1 609 7.2 71.5 Indonesia 371 3.7 364 - 379 260 4.9 493 8.6 23.4 Mexico 419 1.8 415 - 422 289 3.2 547 3.3 49.2 Turkey 445 4.4 437 - 454 304 5.2 613 12.2 57.8 Argentina 388 4.1 380 - 396 231 7.9 543 7 36.4
The OECD mean mathematics score is 496 with a standard deviation of 92. The mean mathematics scores of both HP and TN are 1.6 standard deviations below the OECD mean. Even the 95th percentile scores in HP and TN are below the OECD mean. Compared to the other emerging economies, HP and TN rank significantly lower.
The other metric of percentage of students at or above the baseline level of proficiency is abysmally low in HP (12%) and TN (15%), and a serious cause for concern. While the OECD average is 78%, the percentage of students at or above the baseline level in countries like Brazil, Indonesia, Mexico and Argentina is in the 23%-30% range. These countries seem to be just a bit better off than HP and TN in mathematical literacy.
Extracts from Table B.3.3: Comparing countries’ performance in science (p 126)
The last column indicates the % of students having proficiency in scientific literacy above the baseline level to participate effectively and productively in life (Table B.3.4, p 129). There are six proficiency levels in scientific literacy (Table 3.4 p 54) ranging from Level 6 (involving the application of scientific knowledge and knowledge about science to complex life situations) through to Level 1 (where the tasks require only limited scientific knowledge to be applied to familiar situations). Level 2 is considered the baseline level of proficiency at which students begin to demonstrate the science competencies that will enable them to participate actively in life situations related to science and technology.
Mean S.E. Confidence Interval 5th Percentile S.E. of 5th Percentile 95th Percentile S.E. of 95th Percentile % at or over baseline (Table B.3.4) Himachal Pradesh 325 4.2 317 - 333 209 6.8 450 11.1 11.2 Tamil Nadu 348 4.2 340 - 357 252 4.3 461 12.1 15.5 OECD Average 501 0.5 500 - 502 341 1 649 0.7 82 Brazil 405 2.4 401 - 410 275 3.5 554 4.8 45.8 Shanghai-China 575 2.3 570 - 579 430 4.9 700 3.3 96.8 South Korea 538 3.4 531 - 545 399 6.5 665 4.8 93.7 Finland 554 2.3 550 - 559 400 4.2 694 3.6 94 USA 502 3.6 495 - 509 341 4.8 662 6.7 81.9 Russian Federation 478 3.3 472 - 485 331 5.8 628 5.2 78 Indonesia 383 3.8 375 - 390 272 5.4 499 5.4 34.4 Mexico 416 1.8 412 - 419 291 2.8 544 2.8 52.7 Turkey 454 3.6 447 - 461 322 5 587 6.4 47.6 Argentina 401 4.6 392 - 410 228 10.6 564 7.9 70.1
The OECD mean science score is 501 with a standard deviation of 94. The mean science scores of both HP and TN are 1.6 standard deviations below the OECD mean. Even the 95th percentile scores in HP and TN are below the OECD mean. Compared to the other developing economies, HP and TN rank significantly lower.
The other metric of percentage of students at or above the baseline level or proficiency is abysmally low in HP (11%) and TN (16%), and a serious cause for concern. While the OECD average is 82%, the percentage of students at or above the baseline level in countries like Brazil, Indonesia, Mexico and Argentina is in the 34%-70% range. These countries, especially Argentina (70%) seem to be far better off than HP and TN in scientific literacy.
Visible, but unseen so far - what do we make of the PISA 2009+ results in HP and TN?
Despite the caveats about the quality issues in adminstering the PISA tests in India, the results need to be taken seriously. There is no doubt that our children are being tuned towards rote-memorization and are unable to comprehend or understand well enough and reason out. The ASER reports published annually by Pratham and the periodic reports put out by Educational Initiatives have been telling us as much for years.
The PISA 2009+ results provide us an opportunity to crystallise reform. Although we knew things aren't good, we have shied away from figuring out how bad they are. We can't do that anymore. Knowing that we are at the bottom of the list of 74 countries leaves us no room for denying the reality on the grounds that there is no data that says things are bad. The PISA 2012 cycle is only likely to show us how much worse it is when the data from all the other states are also available.
I hope we won't shoot the messenger and question the motives of the OECD which organises the PISA tests or worse, go into a shell and refuse to participate in the 2012 PISA cycle. It would be interesting to see what the MHRD has to say about the PISA results. I am waiting for an MP to ask a question about this in Parliament. The question will need to be phrased cleverly, or else the Minister will get away by saying nothing meaningful.
To improve, we will need to embark on serious and sustained assessment and exam reform to change the way we evaluate what students have learnt. If we begin testing for comprehension, understanding and the ability to think and reason out, there will be an incentive for schools, teachers and students to work towards honing those skills rather than the ability to reproduce the precise expected answer. Exam reform will also require changing the way things students are taught in a classroom and as a result teacher education as well. It will take many years to turn the system around, given the huge inbuilt inertia. But there is no alternative to the long hard way.
Dependence of students' scores on student and school characteristics
The detailed report also describes the variation of student scores as a function of various parameters to try and identify the factors that may be responsible for some students doing better on the tests. I've looked at the data on a couple of such parameters below.
Public Schools vs Private Schools
An index of school type is constructed by combining a classification of the school as public or private and the proportion of core funding received from government agencies. A public school is defined in PISA as one that is managed directly or indirectly by a public education authority, government agency, or governing board appointed by government or elected by public franchise. A government-independent private school is classified as one that is not managed by a government agency and receives less than 50% of its core funding from government agencies.
Reading Reading Mathematics Mathematics Science Science Country Category % %SE Mean SE Mean SE Mean SE Himachal Pradesh Public 85.07 3.08 310 5.41 331 5.45 317 5.84 Himachal Pradesh Private 14.93 3.08 356 14.57 379 12.01 372 14.86 Himachal Pradesh m a a a a a a a a Tamil Nadu Public 60.1 3.32 325 6.72 334 6.09 339 5 Tamil Nadu Private 39.9 3.32 356 9.84 378 8.33 365 7.62 Tamil Nadu m a a a a a a a a OECD Average Public 81.9 0.3 488 0.63 491 0.64 496 0.67 OECD Average Private 17.85 0.3 518 2.02 519 2.04 525 1.93 OECD Average m 1.17 0.36 444 5.18 452 8.47 458 6.11
m - Missing or invalid response category (this category is excluded from computations of group percentages in the international report)
a - The category does not apply in the country concerned. Data therefore missing
The mean scores of students in public schools can be compared to those in private schools through the response to a question posed to schools "Is your school a public or a private school?". In almost all countries in the set that I looked at, students in private schools seem to have done better than students in public schools, except for Indonesia where students in public schools did better.
Students in private schools did better in Tamil Nadu (40% of schools tested were private schools) and Himachal Pradesh (15% of schools tested were private schools) too. Click here for the data.
Reading Habits of Students
The 15-year-old student populations in Tamil Nadu-India and Himachal Pradesh-India were estimated to have among the lowest reading literacy levels of the PISA 2009 and PISA 2009+ participants with more than 80% of students below the baseline of proficiency. Around one-fifth of students in these economies are very poor readers.
Students were assigned a reader profile based on their self-report to questions on two dimensions: effective learning strategies; and frequency of reading a range of materials.
Students with high levels of effective learning strategies were classified as ‘deep’ readers, while those with low levels of learning strategies were classified as ‘surface’ readers. Additionally, students who indicated they read a range of materials regularly were classified as ‘wide’ readers, those who indicated they only read magazines or newspapers regularly were classified as ‘narrow’ readers, and those who indicated they did not read any materials regularly were classified as ‘highly restricted’ readers.
Table 4.9 (p 93)
Deep and Wide Readers Deep and Narrow Readers Deep and highly restricted readers Surface and wide readers Surface and deep readers Surface and highly restricted readers Country or economy % Score % Score % Score % Score % Score % Score Himachal Pradesh 32 321 16 327 18 317 18 311 7 319 9 296 Tamil Nadu 13 360 10 366 14 321 22 336 17 342 23 313 OECD Average 19 546 25 506 29 504 5 462 10 440 13 427
Socioeconomic and Cultural Status
In PISA 2009 an index of social, economic and cultural status was constructed (ESCS). The data used to construct the index came from responses to a variety of items within the student questionnaire. These items included parental levels of education, parental occupation, the number of books in the home and a variety of items measuring family wealth, the presence of cultural possessions in the home, and the presence of educational resources in the home. This index is used in models presented subsequently in this chapter to unpack the amount of variance in performance that is, firstly, due to social, economic and cultural factors; and secondly, due to school level factors.
The socioeconomic profiles of Tamil Nadu-India and Himachal Pradesh-India are very similar (Figure 4.9 p 78). Both populations have low socioeconomic status relative to the OECD average. Socioeconomic status is not a strong predictor of reading performance, as measured by PISA, in these populations
Other indices that summarise responses from students, their parents or school representatives (typically principals) to a series of questions
Student scores can also be compared across a variety of other factors including:
student age, occupational status or parents, educational level of parents, language background, family structure, family wealth (whether students had the following at home: a room of their own, a link to the Internet, a dishwasher (treated as a country-specific item), a DVD player, and three other country-specific items (some items in ST20); and their responses on the number of cellular phones, televisions, computers, cars and the rooms with a bath or shower), home educational resources (whether students had the following resources at home: a desk and a quiet place to study, a computer that students can use for schoolwork, educational software, books to help with students’ school work, technical reference books and a dictionary) and cultural possessions (whether students had the following at home: classic literature, books of poetry and works of art),
relative grade of the students, learning time in school in the test language, students' reading habits and activities, students' approaches to learning, students' attitudes towards school, students' perception of teachers, students' perception of the disciplinary climate in school, students' reports on teacher stimulation of reading, students' reports of how their language of instruction was taught by the teachers, students' use of libraries, students' reports on how they understood and memorised the text,
school and class size, student teacher ratio, proportion of girls enrolled, availability of computers in school, number of teachers, basis for admission to the school, whether resource allocation and curriculum and assessment policies were decided at the school level or at a higher level, teacher shortage, extracurricular activities in the school, student behaviour and teacher behaviour.
There is a wealth of data on all these indices made available in the detailed report on the PISA 2009+ cycle of test as well as the entire database that is accessible online. I have only been able to parse a small segment of the data as described above. There is much more to be gleaned from the data, if you are interested and motivated enough to dig through.
In other posts, I had looked at
- Programme for International Student Assessment: What? Why? How?
- India's participation in PISA 2009+ cycle: The Process
- India's participation in PISA 2009+ cycle: Terminology & Caveats
