PISA 2009 Technical Report -

Foreword......Page 5
Programme for International Student Assessment: an Overview......Page 23
Participation......Page 25
Managing and implementing PISA......Page 26
Organisation of this report......Page 28
Test Design and Test Development......Page 29
Digital Reading Assessment (DRA)......Page 30
Paper-based assessment......Page 31
Development timeline......Page 33
The PISA 2009 reading literacy framework......Page 34
First phase of development......Page 35
National item submissions......Page 36
National review of items......Page 37
Preparation of dual (English and French) source versions......Page 38
Field trial selection......Page 39
Field trial design......Page 40
Field trial coder queries......Page 41
Main survey reading item selection......Page 42
Main survey mathematics items......Page 45
Despatch of main survey instruments......Page 46
Review of main survey item analyses......Page 47
The Development of the PISA Context Questionnaires......Page 49
The development of the PISA 2009 Questionnaire Framework......Page 50
Research areas in PISA 2009......Page 51
The field-trial of the PISA 2009 context questionnaires......Page 54
Student and School Questionnaires......Page 55
The implementation of the context questionnaires......Page 56
Sample Design......Page 59
Population coverage, and school and student participation rate standards......Page 60
Coverage of the PISA international target population......Page 61
School response rates......Page 62
Student response rates......Page 63
The sampling frame......Page 64
Stratification......Page 65
School sample selection......Page 68
Special school sampling situations......Page 70
Monitoring school sampling......Page 73
Student samples......Page 77
Definition of school......Page 78
Translation and Verification of the Test and Survey Material......Page 83
Development of source versions......Page 84
Double translation from two source languages......Page 85
Testing languages and translation/adaptation procedures......Page 86
International verification of the national versions......Page 88
Verification of test units......Page 89
Main survey verification......Page 90
Verification of questionnaires......Page 93
Final optical check of test booklets, questionnaire booklets and coding guides......Page 95
Verification of Digital Reading Assessment (DRA) units......Page 97
Summary of items deleted at the national level, due to translation, printing or layout errors......Page 98
Field Operations......Page 99
School Co-ordinators......Page 100
The selection of the school sample......Page 101
Preparation of test booklets, questionnaires and manuals......Page 102
Packaging and shipping materials......Page 103
Preparing for coding......Page 104
Logistics prior to coding......Page 106
Single coding design......Page 108
Multiple coding......Page 111
Managing the coding process......Page 113
Questionnaire coding......Page 114
The main survey review......Page 115
Quality Assurance......Page 117
Field trial and main survey review......Page 118
National Centre Quality Monitor (NCQM) visits......Page 119
Post final optical check......Page 120
Survey Weighting and the Calculation of Sampling Variance......Page 121
Survey weighting......Page 122
The school base weight trimming factor......Page 123
The within-school base weight......Page 124
The within school non-response adjustment......Page 127
The balanced repeated replication variance estimator......Page 128
Countries and economies where all students were selected for PISA......Page 130
Scaling PISA Cognitive Data......Page 131
The mixed coefficients multinomial logit model......Page 132
Combined model......Page 133
National calibrations......Page 134
National reports......Page 135
International calibration......Page 141
Student score generation......Page 142
Booklet effects......Page 143
Analysis of data with plausible values......Page 144
Developing common scales for the purposes of trends......Page 145
Uncertainty in the link......Page 146
Data Management Procedures......Page 149
Introduction......Page 150
Data entry quality control......Page 152
DRA data......Page 154
Review of the test and questionnaire data......Page 155
Next steps in preparing the international database......Page 156
Sampling Outcomes......Page 157
Design effects and effective sample sizes......Page 170
Design effects in PISA for performance variables......Page 173
Summary analyses of the design effect......Page 185
Sampling for the Digital Reading Assessment (DRA) component......Page 187
Scaling Outcomes......Page 189
International characteristics of the item pool......Page 190
Test targeting......Page 191
Domain inter-correlations......Page 196
National item deletions......Page 197
International scaling......Page 199
Generating student scale scores and reliability of the PISA scales......Page 200
Test length analysis......Page 201
Booklet effects......Page 205
Overview of the PISA cognitive reporting scales......Page 213
PISA literacy subscales......Page 214
Observations concerning the construction of the PISA overall literacy scales......Page 215
Framework development......Page 216
Testing time and item characteristics......Page 217
Reading......Page 231
Link error......Page 232
Coding Reliability Studies......Page 235
Consistency analyses......Page 236
International coder review......Page 241
Data Adjudication......Page 249
Implementing the standards – quality assurance......Page 250
Information available for adjudication......Page 251
Data adjudication process......Page 252
Digital Reading Assessment (DRA)......Page 253
Detailed country comments......Page 254
Proficiency Scale Construction......Page 259
Introduction	......Page 260
Stage 1: Identifying possible scales......Page 261
Stage 6: Revising and refining with main survey data......Page 262
Defining proficiency levels......Page 263
Building an item map for print reading......Page 265
Levels of print reading literacy......Page 267
Building an item map for digital reading......Page 274
Levels of digital reading literacy......Page 277
Interpreting the reading literacy levels......Page 278
Scaling Procedures and Construct Validation of Context Questionnaire Data......Page 281
Simple questionnaire indices......Page 282
School questionnaire indices......Page 284
Scaling procedures......Page 286
Describing questionnaire scale indices......Page 288
Student scale indices......Page 289
School questionnaire scale indices......Page 308
Parent questionnaire scale indices......Page 312
The index of economic, social and cultural status......Page 314
Digital Reading Assessment......Page 319
Translation......Page 320
School computer resources survey......Page 321
Test delivery system......Page 322
Online Coding System......Page 323
International Database......Page 327
Student files......Page 328
School file......Page 329
Records included in the database......Page 330
How are students and schools identified?......Page 331
Further information......Page 332
References......Page 335
Annexes......Page 337
Annex A ­– Main study item pool classification......Page 338
Annex B – Contrast coding used in conditioning......Page 346
Annex C – DESIgn EFFECT TABLES......Page 355
Annex D - Changes to core questionnaire items......Page 361
Annex E – Mapping of ISCED to years......Page 366
Annex F – National household possession items......Page 367
Annex G – PISA 2009 TECHNICAL STANDARDS......Page 369
Annex H – pisa Consortium, staff and consultants......Page 383
Annex I – SELECTION OF OECD PISA PUBLICATIONS......Page 386
Annex J – OECD countries included in standardisation of major PISA scales......Page 387