In this document we review the evaluation, led by the American Institutes for Research (AIR), of the U.S. Agency of International Development (USAID) Quality Reading Project (QRP) in Tajikistan and the Kyrgyz Republic. The aim of the review is to provide quality assurance of the research methodology at the midterm stage of the evaluation. Specifically, the review will assess (i) the research design, (ii) data collection and other research procedures, (iii) the Early Grade Reading Assessment (EGRA) instruments, and (iv) analysis and reporting of baseline and midline EGRA findings. This review will serve two purposes: to assess the quality of the QRP’s own internal impact evaluation as well as inform USAID of future research design and EGRA administration considerations.
Main Evaluation Questions
This review is guided by the following questions:
1. To what degree is the research design methodology valid, reliable, and aligned to the USAID Global Education Strategy technical reporting guidance?
2. Do the standardized EGRA tools in Kyrgyz, Tajik, and Russian at grades 1, 2, and 4 meet the quality standards of USAID EGRA guidance? If not, what are the gaps and weaknesses of the tools?1
3. Are there other gaps and/or weaknesses in the current research design? Describe and discuss implications.
4. What is the degree of implementation fidelity with regard to the research design? What are the implications of the implementation fidelity for validity of the research design?
5. Do the baseline and midline EGRA reports produced by the QRP activity for Tajikistan and the Kyrgyz Republic accurately report the findings and in a comparable manner to allow for assessment of change over time from baseline to midline?
We used several methods and tools to address the questions stated above, including:
1. Document review (Appendix B)
2. Semi-structured key stakeholder interviews (Appendix C for interview protocols and Appendix D for list of personnel interviewed)
3. Observation of EGRA implementation guided by the following tools:
a. EGRA test administrator checklist (Appendix E.1)
b. USAID data quality assessment template (Appendix E.2)
c. Checklist for assessing USAID evaluation reports (E.3)
1 Grade 3 was not examined for this report.
EdData II Task Order 15 (DEP-AME)
Review of the Quality Reading Project’s Evaluation at Midterm July 2016 ix
d. EGRA item design specifications checklist (E.4)
4. Data analysis – we replicated analyses from the mid-term impact evaluation
To carry out in-country interviews and observations, Matthew Jukes spent seven days in the Kyrgyz Republic, April 19–28, 2016, and Maria Dzula spent seven days in Tajikistan, May 2–10, 2016.