There are many terms and concepts used in the PISA 2009+ report, which also lays out some caveats in interpreting the results, for Himachal Pradesh and Tamil Nadu in particular. This post looks at the terminology and the caveats described in the detailed report on the PISA 2009+ cycle of tests on the results for the pilot in India in the states of Tamil Nadu and Himachal Pradesh.
Terminology (quoting from the report)
- Proficiency levels. To help interpret what students’ scores mean in substantive terms, the scale is divided into bands or levels (based on a set of statistical principles) and then descriptions are generated (based on the tasks that are located within each level) to describe the kinds of skills and knowledge needed to successfully complete those tasks.
For PISA 2009, the range of difficulty of tasks allows for the description of seven levels of reading proficiency: the range of difficulty of tasks allows for the description of seven levels of reading proficiency: Level 1b requires the lowest proficiency, then Level 1a, Level 2, Level 3 and so on up to Level 6. Students with a proficiency estimated to be within the range of Level 1b are likely to be able to successfully complete Level 1b tasks, but are unlikely to be able to complete tasks at higher levels. Level 6 reflects tasks that present the greatest challenge in terms of skills and knowledge. Students with scores in this range in reading are likely to be able to complete reading tasks located at that level successfully, as well as all the other reading tasks in PISA. The mathematics and science scales are divided into six proficiency levels, from Level 1 (least proficient) to Level 6 (most proficient). (p 5) - Baseline level of proficiency: Level 2 is considered a baseline level of proficiency at which students begin to demonstrate the skills that will enable them to participate effectively and productively in life. The percent of students at or above the baseline level of proficiency is the percentage of students at or above Level 2.
- Confidence levels: The statistics in this report represent estimates of national performance based on samples of students, rather than values that could be calculated if every student in every country had answered every question. Consequently, it is important to measure the degree of uncertainty of the estimates. The use of confidence intervals provides a way to make inferences about the population means and proportions in a manner that reflects the uncertainty introduced through the sampling of students, and for the tests, the use of a small set of items. From an observed sample statistic it can be inferred that the corresponding population result would lie within the confidence interval in 95 out of 100 replications of the measurement on different samples drawn from the same population. (p 100)
- Standard error: Each estimate has an associated degree of uncertainty, which is expressed through a standard error. The standard errors for most of the statistics used in this report are provided in the data tables within Appendix B (abbreviated as S.E.). Standard errors are used to construct the confidence interval. A 95% confidence interval around a statistic is calculated as the statistics value +/- 1.96 × the standard error. (p 100)
- Statistical Significance: In the tables, figures and text of this report, differences are labelled or described as statistically significant when a difference of that size or greater would be observed less than 5% of the time in a similar sample drawn from the same population, if there were actually no difference in the corresponding population. Where observed differences do not meet this criterion, they are described as being not statistically significantly different, or as ‘statistically the same’. (p 4)
- PISA student populations:
Total population of 15-year-olds is an estimate of the number of 15-year-olds in the country/economy, including those who are not attending educational institutions.
Total population 15-year-olds enrolled in grade 7 or above is the estimate of the PISA eligible population.
Total in national desired population refers to the number of students from the PISA eligible population who are potentially included as part of the PISA sample. - Equity of outcomes within countries: the degree to which there is equality of outcomes among student subpopulations including analyis of
- the differences in educational outcomes between boys and girls;
- the relationship between socioeconomic background and educational outcomes; and
- the relationships between school policies and practice and educational outcomes. - OECD average is the arithmetic mean of the estimates for the 34 OECD countries that participated in PISA 2009. Each country contributes equally to the calculation of the OECD average, regardless of the number of 15-year-olds in its population.
Caveats (quoting from the report)
There seem to have been many issues in the population data provided and the student sampling in Himachal Pradesh (HP) and Tamil Nadu (TN) during the PISA 2009+ cycle. The report lays out the caveats in interpreting the data from these two states in India:
Executive Summary (page xii)
Ten additional partner participants who were unable to participate within the PISA 2009 project timeframe participated in the PISA 2009 study on a reduced and delayed timeline. This is known as the PISA 2009+ project. The ten PISA 2009+ participants administered the same assessments as their PISA 2009 counterparts, the only difference being that the assessments were administered in 2010. The PISA 2009+ countries or economies were adjudicated against the same technical and quality standards as their PISA 2009 counterparts*.
* Himachal Pradesh-India, Miranda-Venezuela and Tamil Nadu-India did not meet all PISA standards (for student sampling) and their results should be interpreted with caution. See ‘Sampling outcomes’ in Appendix A for details.
Data on student populations and exclusions were not supplied or were of insufficient quality to report the following for Himachal Pradesh-India and Tamil Nadu-India [Appendix A - page 101, Table A.2 - Population Coverage Indices - page 102]:
- the number of 15-year-olds in the country/economy, including those who are not attending educational institutions,
- population of 15-year-olds enrolled in grade 7 or above which is the estimate of the PISA eligible population
- number of students from the PISA eligible population who are potentially included as part of the PISA sample
Himachal Pradesh-India and Tamil Nadu-India did not meet PISA standards for student sampling. Due to irregularities in the student sample numbers, it was established after the testing that these economies sampled from student lists that were often incomplete: not all 15-year-olds within the school were listed. It was not possible to determine whether any bias existed in the obtained sample. Caution should be exercised when using the data from Himachal Pradesh-India or Tamil Nadu-India and when interpreting the reported analyses. [Appendix A - Table A. 4 - Student Response Rates & Sample Size - page 104]
In other posts, I had looked at
- Programme for International Student Assessment: What? Why? How?
- India's participation in PISA 2009+ cycle: The Process
- India's participation in PISA 2009+ cycle: Results from Himachal Pradesh and Tamil Nadu
