Improving Education
through
Accountability and Evaluation
Lessons from Around the World


October 3-5, 2012
Center for American Studies
Via Michelangelo Caetani 32
00186 Rome, Italy



Tuesday, October 2, 2012
17:00-19:00 Registration

Reception with Light Hors-d'Oeuvres


At: Italian Ministry of Education, University and Research,
Viale Trastevere 76/a, 00196 Roma
access limited to the participants registered online

Wednesday, October 3, 2012
8:30-9:00 Registration
9:00-9:30 Welcoming Speakers
- Room A -

Marco Rossi Doria – Under-Secretary of State, Ministry of Education, University and Research
Annamaria Leuzzi – PON Management Authority, Ministry of Education, University and Research
Douglas Besharov – University of Maryland; Association for Public Policy Analysis and Management
9:30-11:00 Plenary - Session 1: "School Accountability through Performance Monitoring"
Chaired by Jane Hannaway, American Institutes of Research: Association for Education Finance and Policy
- Room A -

School Accountability and Monitoring Systems: Insights from the U.S. and Other Countries
Helen Ladd, Duke University; Association for Public Policy Analysis and Management
Paper

Accountability in Education Around the World: Lessons from International Achievement Tests
Ludger Woessmann, University of Munich
Abstract | Presentation

11:00-11:15 Coffee Break
11:15-13:15 Parallel Sessions

Session 1.1: Building and Interpreting Scientific Evidence

Chair: Kentaro Yamamoto – Educational Testing Service
- Room A -

Review on Evaluation and Assessment Frameworks for Improving School Outcomes
Paulo Santiago – OECD
Presentation

Educational Evaluation and Assessment in the Netherlands
Jaap Scheerens, Melanie Ehren, Peter Sleegers and Renske de Leeuw – University of Twente
Paper

The Heterogeneity of "Private School Effect" in Italy
Tommaso Agasisti, Samuele Murtinu and Piergiacomo Sibiano – Politecnico di Milano
Paper

Experimental Evidence on the Effect of Childhood Investments on Postsecondary Attainment and Degree Completion
Susan Dynarski and Joshua Hyman – University of Michigan; Diane Whitmore Schanzenbach – Northwestern University
Abstract

Is it Live or is it Internet? Experimental Estimates of the Effects of Online Instruction on Student Learning
David Figlio – Northwestern University; Mark Rush and Lu Yin – University of Florida
Paper

The Effects of Student Coaching in College: An Evaluation of a Randomized Experiment in Student Mentoring
Eric Bettinger and Rachel Baker – Stanford University
Paper

Session 1.2: Accountability: School Autonomy and Management
Chair: Helen Ladd - Duke University; Association for Public Policy Analysis and Management
- Room B -

Organizational Turnaround and Educational Performance: The Impact of Performance-Based Monitoring Analysis Systems (PBMAS)
Amanda Rutherford – Texas A&M University
Paper

Development of Performance Measures for a Small After-school Program Serving South Asian Youth in New York City, a Case Study
Swati Desai – State University of New York at Albany
Paper

Examination of School-Based Management in Indonesia
Rita Karam, Georges Vernez and Jeffrey Marshall - RAND
Paper

School Autonomy and Student Achievement: An International Study with a Focus on Italy
Angelo Paletta – Università di Bologna; Maria Magdalena Isac – Joint Research Centre, CRELL
Paper | Presentation

Planning and Working On School Improvement. Preliminary Evidence from the Quality and Merit Project in Italy
Andrea Caputo – INVALSI; Sara Mori – INDIRE; Valentina Rastelli – INVALSI
Paper

A Performance-Based Evaluation Model for Rewarding Merit in Italian Schools
Donatella Poliandri, Paola Muzzioli, Isabella Quadrelli, and Sara Romiti – INVALSI
Paper

Session 1.3: Teachers: Measuring and Ensuring Quality
Chair: Alberto Martini - Università del Piemonte Orientale
- Room C -

Investigating the Role of Human Resources in School Turnaround: Evidence from Two States
Michael Hansen – American Institutes for Research
Paper

Teacher Training, Extra Education and Students' Achievement: The Evaluation of a Program for Promoting Students' Performances in Italian Lower Secondary Schools
Elena Claudia Meroni – Padua University; Giovanni Abbiati – University of Milan
Paper

Under Pressure: Job Security, Resource Allocation, and Productivity in Schools Under No Child Left Behind
Randall Reback – Columbia University; Jonah Rockoff – Columbia University and NBER; Heather Schwartz – RAND Corporation
Paper

A Composite Estimator of Effective Teaching
Kata Mihaly, Daniel McCarrey, Douglas O. Staiger, and J. R. Lockwood – RAND
Paper

Teacher Assessment and Students' Performance in OECD-PISA 2009
Brunella Fiore, Isabella Romeo – University of Milano-Bicocca
Paper

Evaluating Mathematics Knowledge for Teaching as an Outcome of Pre-Service Teacher Education: Lessons from the High Performing Countries in the Teds-M Study
Maria Teresa Tatto – Michigan State University; Michael Rodriguez – University of Minnesota;
Paper

13:15-14:30 Lunch on Premises
14:30-16:00 Plenary - Session 2: "Improving Teacher and Administrator Quality"
Chaired by Bert Creemers - University of Groningen
- Room A -

Educator Effectiveness: Measures, Policies and Debates
Jane Hannaway - American Institutes of Research; Association for Education Finance and Policy
Paper

Improving Teaching Quality: Promoting a Dynamic Approach to Teacher Professional Development
Leonidas Kyriakides - University of Cyprus
Paper

16:00-16:15 Coffee Break
16:15-18:30 Parallel Sessions

Session 2.1: Accountability: Unintended Consequences
Chair: David Figlio - Northwestern University
- Room A -

When Accountability Strategies Collide: Do Policy Changes that Raise Accountability Standards Also Erode Public Satisfaction?
Rebecca Jacobsen, Andrew Saultz and Jeffrey W. Snyder – Michigan State University
Paper

School Accountability: How Can We Reward Schools and Avoid Pupil Selection?
Erwin Ooghe – KU Leuven and IZA; Erik Schokkaert – KU Leuven
Paper

When the Cat Is Near, the Mice Won’t Play: The Effect of External Examiners in Italian Schools
Marco Bertoni – University of Padova; CEP-LSE; Giorgio Brunello – University of Padova; IZA; and CESifo; Lorenzo Rocco – University of Padova
Paper

Incentives, Information, and Ideals: The Use of Economic Theory to Evaluate Educational Accountability Policies
Andrew McEachin - University of Virginia
Paper

The Failure of Educational Accountability to Work as Intended in the United States
Sharon L. Nichols – University of Texas at San Antonio
Paper

Outcome-Based and Competition-Based Policies of School Evaluation: A Comparison of School Performance and Perverse Effects in Two Mid-sized Hungarian Towns
Dàniel Horn – European University Institute; Lajos Bódis – Corvinus University;
Paper

Session 2.2: Translating Research Results Into Practice
Chair: Susan Dynarski - University of Michigan
- Room B -

Exploring Politics, Accountability and Evaluation Use in the Mexican Federal Government Education Programs
Gabriela Pérez Yarahuàn – Universidad Iberoamericana
Paper

The Causal Effect of Class Size on Pupils’ Performance: Evidence from Italian Primary Schools
Larysa Minzyuk and Felice Russo – University of Salento
Paper

Creating a Performance Culture: Anxiety, Climate and Performance Management Reform
Kate Destler - University of Washington
Paper

Career Pathways as a Framework for Program Design and Evaluation
David J. Fein – Abt Associates Inc.
Paper

Moving Matters: The Causal Effect of School Mobility on Student Performance
Amy Ellen Schwartz and Leanna Stiefel – New York University
Paper

Preventing and Re-integrating Early School Leavers: A Comparative Meta Evaluation of Policies Implemented in 7 European Member States.
Anja Meierkord and Massimiliano Mascherini – EUROFUND
Paper | Presentation

Session 2.3: Innovation: Changes to Policy and Curriculum
Chair: Daniele Checchi - Università degli Studi di Milano; IZA
- Room C -

Can Formal-Informal Collaborations Improve Science Literacy in Urban Middle Schools? The Urban Advantage Middle School Science Initiative in New York City
Meryle Weinstein, Emilyn Ruble Whitesell, and Amy Ellen Schwartz – New York University
Paper

Preliminary Evidence from the M@t.abel Teacher Professional Development Program in Italy
Gianluca Argentin – Università di Milano Bicocca; Aline Pennisi – Ministero dell’Economia e delle Finanze; Daniele Vidoni – INVALSI; Giovanni Abbiati – Università degli Studi di Milano; Andrea Caputo – INVALSI
Paper

Improving Reading Comprehension by Fostering Children’s Engagement with Books During Summer Vacation: A Cluster-Randomized Trial Comparing Strategy and Text Structure Instruction in High Poverty Elementary Schools
James S. Kim, Helen Kingston, Lisa Foster – Harvard University; Thomas G. White - University of Virginia
Paper

Standing on the Shoulders of Chess Masters: Using RTCs to Evaluate the Effects of Including Chess in the Italian Primary School Curriculum
Gianluca Argentin – Università di Milano Bicocca; Alberto Martini – Università del Piemonte Orientale; Barbara Romano – Università di Genova; University of Pennsylvania
Paper

Early High School Interventions to Increase Students’ Access to Post-secondary Education: Experimental Impacts from Canada’s Future to Discover Project
Reuben Ford – Social Research and Demonstration Corporation
Paper

Synthesis of Findings from 15 Years of Educational Reform in Thailand: Lessons on Leading Educational Change in East Asia
Philip Hallinger – Hong Kong Institute of Education
Paper

20:00-23:00 Social Dinner (please register at registration desk)
Speech by Elizabeth King, World Bank
Thursday, October 4, 2012
9:00-10:30 Plenary - Session 3
- Room A -

"Promising Innovations Inside and Outside the Classroom"

Chaired by Neil Gilbert - University of California at Berkeley

Digital Information Literacy: Challenges and Opportunities for 21st Century Education
Jean-François Rouet - University of Poitiers; National Center for Scientific Research
Paper

Outgrowing the Mode Effect Study of Paper and Computer Based Testing
Kentaro Yamamoto - Educational Testing Service
Paper

10:30-10:45 Coffee Break
10:45-13:00 Parallel Sessions Session 3.1: Accountability: Achievement Gap
Chair: Daniele Vidoni - INVALSI
- Room A -

Does Accountability Narrow Achievement Gaps? Evidence from the United States in the Era of No Child Left Behind
Sean Reardon, Erica Greenberg, Demetra Kalogrides, Kenneth A. Shores and Rachel A. Valentino – Stanford University
Paper

At-Risk Student Averse: Risk Management and Accountability
Julian Vasquez Heilig, Michelle Young and Amy Williams – The University of Texas at Austin
Paper

A Way to Resilience: How Can Italian Disadvantaged Students and Schools Close the Achievement Gap?
Tommaso Agasisti – Politecnico di Milano; Sergio Longobardi – University of Naples "Parthenope"
Paper

Science Competencies Across PISA OECD Countries: Comparing Exceptionally High and Low Performers
Fabio Alivernini and Sara Manganelli – INVALSI
Paper

Why Do Northern European Secondary Schools Outperform American Secondary Schools?
John H. Bishop – Cornell University and ILR
Paper

Session 3.2: Innovation: School Choice and Market Based Incentives
Chair: Jean-François Rouet, University of Poitiers; National Center for Scientific Research
- Room B -

The Impact of High School Choice on Mediators of Student Success
Sean P. Corcoran, Lori Nathanson and James Kemple – New York University
Paper

Do KIPP Schools Boost Student Achievement?
Philip M. Gleason, Christina Clark Tuttle, Brian Gill, Ira Nichols-Barrer and Bing-ru Teh – Mathematica Policy Research
Paper

Choices for Studying Choice: Assessing Charter School Effectiveness Using Two Quasi-Experimental Methods
Devora H. Davis and Margaret E. Raymond – Stanford University
Paper

Do Charter Schools Improve Student Achievement? Evidence from a National Randomized Study
Melissa A. Clark, Philip Gleason and Christina Tuttle – Mathematica Policy Research; Marsha K. Silverberg – Institute of Education Sciences
Paper

School Choice and School Accountability: Evidence from a Private Voucher Program in Milwaukee, Wisconsin
Deven E. Carlson – University of Oklahoma; Joshua M. Cowen – University of Kentucky; David J. Fleming – Furman University
Paper

Performance Based Incentives for Learning in the Mexican Classroom
Andrew Christensen, Brian Fuller – Foundation Escalera; Victor Steenbergen – London School of Economics; Alison Hamburg – Columbia University
Paper

Session 3.2: Teachers: New Challenges
Chair: Roel Bosker University of Groningen; Groningen Institute for Educational Research
- Room C -

Render unto Primary the Things which are Primary’s: Inherited and Fresh Learning Divides in Italian Lower Secondary Education
Gianfranco De Simone – Fondazione Giovanni Agnelli
Paper

Immigrant Background Peer Effects in Italian Schools
Dalit Contini – University of Torino
Paper
Appendix

Teaching Digital Skills to "Digital Natives": Does School Play a Role?
Gianluca Argentin and Marco Gui – University of Milan-Bicocca; Chiara Tamanini – IPRASE
Paper

Parental Empowerment in Mexico: Randomized Experiment of the "Apoyos a la Gestion Escolar (AGE)" in Rural Primary Schools in Mexico
Paul Gertler – University of California at Berkeley; Harry Anthony Patrinos – World Bank; Eduardo Rodr&iccute;guez-Oreggia – Tecnol&occute;gico de Monterrey, Campus State of Mexico
Paper

Issues in the Evaluation of a Program to Promote Educational Achievements of Ethiopian-Israeli High School Students
Dalia Ben-Rabi, Viacheslav Konstantinov, Ruth Baruj and Miriam Cohen-Navot – Myers-JDC Brookdal
Paper

13:00-14:15 Lunch on Premises
14:15-15:45 Session 4: "Building and Interpreting Scientific Evidence"
Chaired by Alexei Monsarrat – Atlantic Council

Towards Evidence-Based Education
Roel Bosker – University of Groningen
Abstract

What Works in Education: Don't Let Perfection be the Enemy of the Good
Grover Whitehurst – The Brookings Institution
Abstract

15:45-16:00 Coffee Break
16:00-18:15 Parallel Sessions Session 4.1: Accountability: Performance Measures and Evaluation
Chair: Sean P. Corcoran New York University
- Room A -

The Impact of Selection of Student Achievement Measurement Instrument on Teacher Value-Added Measures
James L. Woodworth – Hoover Institute, Stanford University; Wen-Juo Lo – University of Arkansas; Joshua B. McGee – Laura and John Arnold Foundation; Nathan C. Jensen – Northwest Evaluation Association
Paper

One-year Value-added School Effects from Various Models and their Inter-temporal Variability: Evidence from China
Pai Peng and Eckhard Klieme – German Institute for International Educational Research
Paper

Student literacy one year later. On school value added estimation using PISA-OECD
Massimiliano Bratti and Daniele Checchi – Università deglì Studi di Milano; IZA
Paper

Patterns of Value-Added Creation in the Transition from Primary to Lower Secondary Education in Italy
Gianfranco De Simone and Andrea Gavosto – Fondazione Giovanni Agnelli
Paper

The Infrastructure of Accountability: Examining the Governance of Longitudinal Data Systems
Stacey A. Rutledge – Florida State University; Dorothea Anagnostopoulos and Rebecca Jacobsen – Michigan State University
Paper

The Effects of Texas's Targeted Pre-kindergarten Program on Academic Performance
Rodney J. Andrews – The University of Texas at Dallas and The Texas Schools Project; Paul Jargowsky, Rutgers UniversityCamden; Kristin Kuhne – Communities Foundation of Texas
Paper

Session 4.2: Building and Interpreting Scientific Evidence
Chair: John Bishop - Cornell University
- Room B -

Reviewing Systematic Reviews: Meta-Analysis of What Works Clearinghouse Computer-Assisted Reading Interventions
Andrei Streke – Mathematica Policy Research; Tsze Chan – American Institutes for Research Paper

Reinforcing Evidence-Based Policymaking in Education: Methodological Developments at CRELL, Centre for Research on Lifelong Learning Based on Indicators and Benchmarks
Andrea Saltelli – Joint Research Centre, Unit of Applied Statistics and Econometrics
Paper

Evaluation of the Ontario Ministry of Education’s - Student Success / Learning to 18 Strategy
Ungerleider, C., Baumann, R., B&eccute;langer, J., Cartwright, F., Eaton, I., Guerriero, S. and T. Lavin – Directions-EPRG
Paper

How Salient are Performance Incentives in Education? Evidence from North Carolina
Thomas Ahn – University of Kentucky; Jacob L. Vigdor – Duke University and NBER
Paper

Evaluating the "Programa Mais Sucesso Escolar": Lessons Learned from Evaluating the Impact of a Portuguese National Educational Policy for Compulsory Education
M. Clara Barata, M. Manuela Calheiros, Joana Nunes Patr&iccute;cio, João Graça and M. Lu&iccute;sa Lima – ISCTE-Lisbon University Institute and CIS-IUL
Paper

Equity in School: A Challenge for Regional Based Educational Systems
Patrizia Falzetti and Roberto Ricci – INVALSI
Paper

Session 4.3: Teachers: Recruitment, Retention, and Distribution
Chair: Kenneth Wong - Brown University
- Room C -

Teacher Quality Policy When Supply Matters Jesse Rothstein – University of California at Berkeley and NBER
Paper

Moving High-Performing Teachers to Low-Performing Schools: A Randomized Experiment
Steven Glazerman, Ali Protik, Bing-ru Teh and Julie Bruch – Mathematica Policy Research
Paper

Portability of Teacher Effectiveness Across School Settings
Zeyu Xu, Umut Ozek and Matthew Corritore – American Institutes for Research
Paper

Teacher Mobility and Student Learning
Gianna Barbieri – MIUR; Claudio Rossetti – LUISS; Paolo Sestito – Bank of Italy
Paper

Meritocracy for Teachers: Evidence from Colombia
Alejandro Ome – University of Chicago
Paper

18:30-19:00 Reception with Light Hors d’Oeuvres
Speech by Raj Chetty, Harvard University (on video)
Friday, October 5, 2012
9:00-10:30 Rapporteur Session
Chaired by Douglas Besharov, University of Maryland; Association for Public Policy Analysis and Management

Rebecca Maynard, University of Pennsylvania
Gianluca Argentin, University of Milan-Bicocca
Aline Pennisi, Ministry of Economy and Finance
Erich Battistin, Istituto di Ricerca per la Valutazione delle Politiche Pubbliche
10:30-12:30 Panel Session: "Translating Research Results into Practice"
Chaired by Paolo Sestito, Istituto Nazionale per la Valutazione del Sistema Educativo di Istruzione e di Formazione; Banca d’Italia

Francesco Profumo, Italian Minister of Education
Ugo Trivellato, University of Padova
Andreas Schleicher, Organisation for Economic Co-operation and Development (on video)
Pina Antonaci, Istituto "De Pace" di Lecce
12:30 Closure

 




With additional support from





Cooperating agencies