Tool Review Process
The National Center has established a standard process to evaluate the scientific rigor of commercially available tools to monitor student’s progress.
In 2004, a total of thirteen commercially available student progress monitoring tools were submitted for the TRC review. And in 2005, four new tools submitted for TRC review and two previously reviewed tools provided new information.
All submitters of progress monitoring tools are required to complete an evaluation form developed by the Center with the assistance from the Center’s Technical Review Committee (TRC). Based on the Standards for Educational and Psychological Testing developed by the Joint Committee appointed by the American Educational Research Association (AERA), the American Psychological Association (APA), and the National Council on Measurement Used in Education (NCMUE) and the Individuals with Disabilities Education Act (IDEA), TRC has identified seven standards of technical adequacy that are critical to the Center’s definition of evidence-based student progress monitoring. The evaluation form, the Standard Protocol for Evaluating Progress Monitoring Tools (the protocol), asks for an extensive array of questions about these seven technical adequacy standards as well as detailed descriptive information. Consistent with the methods used by the What Works Clearinghouse, each standard is comprised of two or three essential components and a tool must demonstrate evidence on each critical component in order to be considered sufficient for a particular standard.
In an attempt to ensure the integrity and independence of the evaluation process and final recommendations, all members of the TRC were asked to disclose all contractual obligations and affiliations with educational testing and measurement firms/organizations to avoid any actual or apparent conflict of interests. Upon the receipt of assigned tools, all TRC reviewers were again asked to confirm in writing that they did not have any conflicts of interests with each assignment.
Each submitted tool was randomly assigned to two reviewers who did not have any conflict of interests with the particular tool. Once the reviews were completed, Center staff carefully compared individual recommendations of the two reviewers item by item on the protocol. In an effort to provide the most accurate and comprehensive information for its consumers, the National Center has extended an opportunity to all submitters to provide additional information in response to the TRC’s initial review results. All additional information/materials were forwarded to TRC members for a complete re-review to make final decisions.
When the two independent reviewers made the same recommendation, it became the final decision. When there were discrepancies, on the other hand, the Center arranged discussions among them to reconcile the differences. Discrepancies that could not be resolved by the two reviewers were forwarded to additional reviewers who had specific expertise on the concerns raised from the initial reviews.
Information on the tools that were found sufficient and met the Center’s criteria for scientifically based progress monitoring tools are disseminated through the Center’s various technical assistance and dissemination activities and through this website.
Back to Top