December 4, 2006
The details outlined here relate only to PSAE Day 2 and do not impact Day 1 results or student ACT scores.
As part of its regular data quality assurance processes, ISBE posted the 2006 PSAE scores in mid-October on the IWAS system for the data correction window. During the data correction period two districts notified ISBE Student Assessment with concerns that some of their students received high ACT scores but very low WorkKey scores. While this disparity is possible if students do not do well on Day 2 of the test, these districts were concerned because these results for the particular students involved was unexpected. The districts asked ISBE to look into the situation.
In investigating the situation, ISBE asked ACT to rescore the answer documents for some 160,000 students using keys for all three versions of the Day 2 tests. With that rescoring approximately 1,500 students earned higher Day 2 WorkKeys scores than their originally reported scores. ISBE verified that many of these students tested at schools which had answer documents “alerted” by Harcourt.
On further investigation ISBE found that during scoring Harcourt experienced many “alerts” for Day 2 answer documents. These alerts are part of the test quality assurance process. When resolving these alerts, Harcourt incorrectly determined that some answer documents should be scored using a different form number from the one gridded on the answer documents. This incorrect determination at Harcourt resulted in the Science answer documents being scored with an incorrect answer key. Harcourt then passed this same incorrect form number to ACT to score the Day 2 Reading and Mathematics WorkKeys tests.
ISBE told Harcourt to revisit these test results. Working with ISBE staff, it was discovered that many of these students should not have received scores because the school administered the wrong form of the test (e.g. – a student tested with accommodations used the standard time form rather than the form for accommodations). Schools have been notified of these situations.
There are some side effects in reassigning students to a different (and correct) test version for re-scoring. In particular, each time a PSAE test is given, the results are equated to those of earlier assessments. For equating to be valid, the correct key must have been used when scoring students’ answer documents. ACT then determined that the number of incorrectly scored Day 2 tests was large enough to affect equating, thereby affecting the original scores assigned to all students.
Accordingly, ACT removed all erroneous student test records from the equating sample and re-equated the 2005-2006 data to get correct scores for all grade 11 students. Although the net impact of re-equating and re-scoring is small and many students’ original scores will be unaffected, ACT made the determination to discard all previously generated scores and produce new Day 2 and PSAE scores for all students.
The net effect of the re-equating and re-scoring all 160,000 tests that roughly one PSAE mathematics score in the state changed from “below standards” to “meets standards.” For PSAE reading, 10 scores in the state changed from “below standards” to “meets standards.” For PSAE science, 69 scores in the state changed from “below standards” to “meets standards” and 27 in the state changed from “meets standards” to “below standards.”
The PSAE file is currently being rescored will be reposted in the next few weeks. This reposting will show the corrected scores with the new equating applied. All grade 11 reports and AYP for high schools will not be processed until the reposting has occurred.
Again, this does not impact Day 1 results or student ACT scores.
Illinois State Board of Education