How the Department Made Determinations – Part C 2020

In 2020, the U.S. Department of Education (Department) is continuing to use both results and compliance data in making our determination for each State under sections 616(d) and 642 of the Individuals with Disabilities Education Act (IDEA) for each State’s early intervention program under Part C of the IDEA. We considered the totality of the information we have about a State, including information related to the State’s Federal fiscal year (FFY) 2018 State Performance Plan (SPP)/Annual Performance Report (APR), Indicator C3 Child Outcomes data (Outcomes data) and other data reported in each State’s FFY 2018 SPP/APR; information from monitoring and other publicly available information, such as Specific Conditions on the State’s grant award under Part C; and other issues related to a State’s compliance with the IDEA.

In examining each State’s Outcomes data, we specifically considered the following results elements:

1. Data quality by examining—
   1. the completeness of the State’s data, and
   2. how the State’s FFY 2018 data compared to four years of historic data to identify data anomalies; and
2. Child performance by examining—
   1. how each State’s FFY 2018 data compared with all other States’ FFY 2018 data, and
   2. how each State’s FFY 2018 data compared with its own FFY 2017 data.

Below is a detailed description of how the Office of Special Education Programs (OSEP) evaluated States’ data using the Results-Driven Accountability (RDA) Matrix. The RDA Matrix is individualized for each State and consists of:

1. a Compliance Matrix that includes scoring on SPP/APR Compliance Indicators and other compliance factors;
2. Results Components and Appendices that include scoring on Results Elements;
3. a Compliance Score and a Results Score;
4. an RDA Percentage based on both the Compliance Score and the Results Score; and
5. the State’s 2020 Determination.

The scoring of each of the above evaluation criteria is further explained below in the following sections:

1. 2020 Part C RDA Matrix and Results Score
2. 2020 Part C Compliance Matrix and Compliance Score; and
3. 2020 RDA Percentage and 2020 Determination

In making each State’s 2020 determination, the Department used the FFY 2018 early childhood outcomes data reported by each State under SPP/APR Indicator C3 by considering the following results elements:

1. Data Quality

   1. Data Completeness:

      • Data completeness was calculated using the total number of Part C children who were included in each State’s FFY 2018 Outcomes data and the total number of children the State reported exiting during FFY 2018 in its FFY 2018 IDEA Section 618 Exiting data; and

   2. Data Anomalies:

      • Data anomalies were calculated by examining how the State’s FFY 2018 Outcomes data compared to four years of historic data.

2. Child Performance

   1. Data Comparison:

      • How each State’s FFY 2018 Outcomes data compared with all other States’ FFY 2018 Outcomes data; and

   2. Performance Change Over Time:

      • How each State’s FFY 2018 Outcomes data compared with its own FFY 2017 Outcomes data.

Calculation of each of these results elements and scoring is further described below:

1. Data Quality

   1. Data Completeness:

      • The data completeness score was calculated using the total number of Part C children who were included in your State’s FFY 2018 Outcomes data and the total number of children your State reported exiting during FFY 2018 in its FFY 2018 IDEA Section 618 Exiting data. Each State received a percentage, which was computed by dividing the number of children reported in the State’s FFY 2018 Outcomes data by the number of children the State reported exited during FFY 2018 in the State’s FFY 2018 IDEA Section 618 Exiting Data. This yielded a percentage such that each State received a data completeness score of ‘2’ if the percentage was at least 65%¹; a data completeness score of ‘1’ if the percentage was between 34% and 64%; and a data completeness score of ‘0’ if the percentage were less than 34%. For the two States with approved sampling plans, the State received a ‘2’. (Data Sources: FFY 2018 APR Indicator C3 data and EDFacts School Year (SY) 2018-2019; data extracted 5/27/2020.)

   2. Data Anomalies:

      • The data anomalies score for each State represents a summary of the data anomalies in each State’s FFY 2018 Outcomes data. Publicly available data for the preceding four years reported by and across all States for each of 15 progress categories under Indicator 3 (in the FFY 2014 – FFY 2017 APRs) were used to determine an expected range of responses for each progress category under Outcomes A, B, and C². For each of the 15 progress categories, a mean was calculated using this publicly available data. A lower and upper scoring percentage was set at one standard deviation above and below the mean for category a and two standard deviations above or below the mean for categories b through e. In any case where the low scoring percentage set from one or two standard deviations below the mean resulted in a negative number, the low scoring percentage is equal to 0.
      • If your State's FFY 2018 Outcomes data reported in a progress category fell below the calculated "low percentage" or above the "high percentage" for that progress category for all States, the data in that particular category are statistically improbable outliers and considered an anomaly for that progress category. If your State’s data in a particular progress category was identified as an anomaly, the State received a ‘0’ for that category. A percentage that is equal to or between the low percentage and high percentage for each progress category received 1 point. A State could receive a total number of points between 0 and 15. Thus, a point total of 0 indicates that all 15 progress categories contained data anomalies and a point total of 15 indicates that there were no data anomalies in all 15 progress categories in the State's data. An overall data anomalies score of ‘0’, ‘1’, or ‘2’ is based on the total points awarded. Each State received a data anomalies score of ‘2’ if the total points received in all progress categories were 13 through 15; a data anomalies score of ‘1’ for 10 through 12 points; and a data anomalies score of ‘0’ for zero through nine points. (Data Sources: States’ FFY 2014 through FFY 2017 SPP/APR Indicator C3 data and each State’s FFY 2018 Outcomes data)

2. Child Performance

   1. Data Comparison:

      • The data comparison overall performance score represents how your State's FFY 2018 Outcomes data compares to other States' FFY 2018 Outcomes data. Each State received a score for the distribution of the 6 Summary Statements (SS) for that State compared to the distribution of the 6 Summary Statements in all other States³. The 10th and 90th percentile for each of the 6 Summary Statements was identified and used to assign points to performance outcome data for each Summary Statement. Each Summary Statement outcome was assigned ‘0’, ‘1’, or ‘2’ points.
      • If a State’s Summary Statement value fell at or below the 10th percentile, that Summary Statement was assigned a score of ‘0’. If a State’s Summary Statement value fell between the 10th and 90th percentile, the Summary Statement was assigned ‘1’ point, and if a State’s Summary Statement value fell at or above the 90th percentile, the Summary Statement was assigned ‘2’ points. The points were added across the 6 Summary Statements. A State can receive total points between 0 and 12, with the total points of ‘0’ indicating all 6 Summary Statement values were below the 10th percentile and a total points of 12 indicating all 6 Summary Statements were above the 90th percentile. An overall comparison Summary Statement score of ‘0’, ‘1’, or ‘2’ was based on the total points awarded.
      • The data comparison Overall Performance Score for this results element of ‘0’, ‘1’, or ‘2’ for each State is based on the total points awarded. Each State received an Overall Performance Score of: ‘2’ if the total points across SS1 and SS2 were nine through 12 points; score of ‘1’ for five through eight points; and score of ‘0’ for zero through four points. (Data Sources: All States’ SPP/APR Indicator C3 data from FFY 2018 and each State’s FFY 2018 SPP/APR Indicator C3 data.)

   2. Performance Change Over Time:

      • The Overall Performance Change Score represents how each State’s FFY 2018 Outcomes data compared with its FFY 2017 Outcomes data and whether the State’s data demonstrated progress. The data in each Outcome Area is assigned a value of 0 if there was a statistically significant decrease from one year to the next, a value of 1 if there was no significant change, and a value of 2 if there was a statistically significant increase. The specific steps for each State are described in the State’s RDA Matrix. The scores from all 6 Outcome Areas were totaled, resulting in total points ranging from 0 – 12. The Overall Performance Change Score for this results element of ‘0’, ‘1’, or ‘2’ for each State is based on the total points awarded. Each State received an Overall Performance Change Score of: ‘2’ if the total points were eight or above; a score of ‘1’ for four through seven points; and score of ‘0’ for below three points. Where OSEP has approved a State’s reestablishment of its Indicator C3 Outcome Area baseline data as its data for FFY 2018, because the State has changed its methodology for collecting this outcome data, the State received a score of ‘N/A’ for this element since determining performance change based on the percentages across these two years of data would not be a valid comparison. The points are not included in either the numerator or denominator in the overall calculation of the results score. (Data Source: SPP/APR Indicator C3 data from FFY 2017 and 2018)

Footnotes

¹ In determining the data completeness score, the Department will round up from 64.5% (but no lower) to 65%. Similarly, the Department will round up from 33.5% (but no lower) to 34%.

² The three Child Outcome areas are: Outcome A (Positive social-emotional skills (including social relationships); Outcome B (Acquisition and use of knowledge and skills (including early language/communication)); and Outcome C (Use of appropriate behaviors to meet their need). The five Progress Categories under SPP/APR Indicator C3 are the following:

1. Percent of infants and toddlers who did not improve functioning
2. Percent of infants and toddlers who improved functioning but not sufficient to move nearer to functioning comparable to same-aged peers
3. Percent of infants and toddlers who improved functioning to a level nearer to same-aged peers but did not reach it
4. Percent of infants and toddlers who improved functioning to reach a level comparable to same-aged peers
5. Percent of infants and toddlers who maintained functioning at a level comparable to same-aged peers

Outcomes A, B, and C under SPP/APR Indicator C- each contain these five progress categories for a total of 15 progress categories

³ Each of the three Child Outcome Areas (A, B, and C) are measured by the following two Summary Statements:

1. Of those infants and toddlers who entered or exited early intervention below age expectations in each Outcome, the percent who substantially increased their rate of growth by the time they turned 3 years of age or exited the program.
2. The percent of infants and toddlers who were functioning within age expectations in each Outcome by the time they turned 3 years of age or exited the program.

In making each State’s 2020 determination, the Department used a Compliance Matrix, reflecting the following compliance data:

1. The State’s FFY 2018 data for Part C Compliance Indicators 1, 7, 8A, 8B, and 8C (including whether the State reported valid and reliable data for each indicator); and whether the State demonstrated correction of all findings of noncompliance it had identified in FFY 2017 under such indicators;
2. The timeliness and accuracy of data reported by the State under sections 616, 618, and 642 of the IDEA;
3. The State’s FFY 2018 data, reported under section 618 of the IDEA, for the timeliness of State complaint and due process hearing decisions;
4. Longstanding Noncompliance:
   • The Department considered:
     1. Whether the Department imposed Specific Conditions on the State’s FFY 2019 IDEA Part C grant award and those Specific Conditions are in effect at the time of the 2020 determination, and the number of years for which the State’s Part C grant award has been subject to Specific or Special Conditions; and
     2. Whether there are any findings of noncompliance identified in FFY 2016 or earlier by either the Department or the State that the State has not yet corrected.

The Compliance Matrix indicates a score of ‘0’, ‘1’, or ‘2’ for each of the compliance indicators in item one above and for each of the additional factors listed in items two through four above. Using the cumulative possible number of points as the denominator, and using as the numerator the actual points the State received in its scoring under these factors, the Compliance Matrix reflects a Compliance Score, which is combined with the Results Score to calculate the State’s RDA percentage and determination.

1. Scoring of the Matrix for Compliance Indicators 1, 7, 8A, 8B, and 8C

In the 2020 Part C Compliance Matrix, a State received points as follows for each of Compliance Indicators 1, 7, 8A, 8B, and 8C⁴:

• Two points, if either:
  • The State’s FFY 2018 data for the indicator were valid and reliable, and reflect at least 95%⁵ compliance; or
  • The State’s FFY 2018 data for the indicator were valid and reliable, and reflect at least 90% compliance; and the State identified one or more findings of noncompliance in FFY 2017 for the indicator, and has demonstrated correction of all findings of noncompliance identified in FFY 2017 for the indicator. Such full correction is indicated in the matrix with a “Yes” in the “Full Correction of Findings of Noncompliance Identified in FFY 2017” column.⁶
• One point, if the State’s FFY 2018 data for the indicator were valid and reliable, and reflect at least 75% compliance, and the State did not meet either of the criteria above for two points.
• Zero points, under any of the following circumstances:
  • The State’s FFY 2018 data for the indicator reflect less than 75% compliance; or
  • The State’s FFY 2018 data for the indicator were not valid and reliable;⁷ or
  • The State did not report FFY 2018 data for the indicator.⁸

2. Scoring of the Matrix for Timely and Accurate State-Reported Data

In the 2020 Part C Compliance Matrix, a State received points as follows for Timely and Accurate State-Reported Data⁹:

• Two points, if the OSEP-calculated percentage reflects at least 95% compliance.
• One point, if the OSEP-calculated percentage reflects at least 75% and less than 95% compliance.
• Zero points, if the OSEP-calculated percentage reflects less than 75% compliance.

3. Scoring of the Matrix for Timely State Complaint Decisions and Timely Due Process Hearing Decisions

In the 2020 Part C Compliance Matrix, a State received points as follows for timely State complaint decisions and for timely due process hearings, as reported by the State under section 618 of the IDEA:

• Two points, if the State’s FFY 2018 data were valid and reliable, and reflect at least 95% compliance.
• One point, if the State’s FFY 2018 data reflect at least 75% and less than 95% compliance.
• Zero points, if the State’s FFY 2018 data reflect less than 75% compliance. • Not Applicable (N/A), if the State’s data reflect less than 100% compliance, and there were fewer than ten State complaint decisions or ten due process hearing decisions.

4. Scoring of the Matrix for Long-Standing Noncompliance (Includes Both Uncorrected Identified Noncompliance and Specific Conditions)

In the 2020 Part C Compliance Matrix, a State received points as follows for the Long-Standing Noncompliance component:

• Two points, if the State has:
  • No remaining findings of noncompliance identified by OSEP or the State; in FFY 2016 or earlier, and
  • No Specific Conditions on its FFY 2019 grant award that are in effect at the time of the 2020 determination.
• One point, if either or both of the following occurred:
  • The State has remaining findings of noncompliance, identified by OSEP or the State, in FFY 2016, FFY 2015, and/or FFY 2014, for which the State has not yet demonstrated correction (see the FFY 2018 OSEP Response to the State’s FFY 2018 SPP/APR in the EMAPS SPP/APR reporting tool for specific information regarding these remaining findings of noncompliance); and/or
  • The Department has imposed Specific Conditions on the State’s FFY 2019 Part C grant award and those Specific Conditions are in effect at the time of the 2020 determination.
• Zero points, if either or both of the following occurred:
  • The State has remaining findings of noncompliance identified, by OSEP or the State, in FFY 2013 or earlier, for which the State has not yet demonstrated correction (see the OSEP Response to the State’s FFY 2018 SPP/APR in the EMAPS SPP/APR reporting tool for specific information regarding these remaining findings of noncompliance); and/or
  • The Department has imposed Specific or Special Conditions on the State’s last three (FFYs 2017, 2018, and 2019) IDEA Part C grant awards, and those Specific Conditions are in effect at the time of the 2020 determination.

Footnotes

⁴ A notation of “N/A” (for “not applicable”) in the “Performance” column for an indicator denotes that the indicator is not applicable to that particular State. The points for that indicator are not included in the denominator for the matrix.

⁵ In determining whether a State has met the 95% compliance criterion for these indicators (1, 7, 8A, 8B, and 8C), the Department will round up from 94.5% (but no lower) to 95%. Similarly, in determining whether a State has met the 90% compliance criterion discussed below, the Department will round up from 89.5% (but no lower) to 90%. In addition, in determining whether a State has met the 75% compliance criterion discussed below, the Department will round up from 74.5% (but no lower) to 75%. The Department will also apply the rounding rules to the compliance criteria for 95% and 75% for:

1. the timeliness and accuracy of data reported by the State under sections 616, 618, and 642 of the IDEA;
2. the State’s FFY 2018 data, reported under section 618 of the IDEA, for the timeliness of State complaint and due process hearing decisions.

⁶ A “No” in that column denotes that the State has one or more remaining findings of noncompliance identified in FFY 2017 for which the State has not yet demonstrated correction. An “N/A” (for “not applicable”) in that column denotes that the State did not identify any findings of noncompliance in FFY 2017 for the indicator.

⁷ If a State’s FFY 2018 data for any compliance indicator are not valid and reliable, the matrix so indicates in the “Performance” column, with a corresponding score of “0.” The explanation of why the State’s data are not valid and reliable is contained in the OSEP Response to the State’s FFY 2018 SPP/APR in the EMAPS SPP/APR reporting tool.

⁸ If a State reported no FFY 2018 data for any compliance indicator, the matrix so indicates in the “Performance” column, with a corresponding score of 0.

⁹ OSEP used the Part C Timely and Accurate Data Rubric to award points to states based on the timeliness and accuracy of their 616 and 618 data. A copy of the rubric is contained in the OSEP Response to the State’s FFY 2018 SPP/APR in the the EMAPS SPP/APR reporting tool. On the first page of the rubric, entitled “Part C Timely and Accurate Data-SPP/APR Data” states are given one point for each indicator with valid and reliable data and five points for SPP/APRs that were submitted timely. The total points for valid and reliable SPP/APR data and timely submission are added together to form the APR Grand Total. On page two of the rubric, the State’s 618 data is scored based on information provided to OSEP on 618 data timeliness, completeness and edit checks from EDFacts. The percentage of Timely and Accurately Reported Data is calculated by adding the 618 Data Grand Total to the APR Grand Total and dividing this sum by the total number of points available for the entire rubric. This percentage is inserted into the Compliance Matrix.

Each State’s 2020 RDA Percentage was calculated by adding 50% of the State’s Results Score and 50% of the State’s Compliance Score. The State’s RDA Determination is defined as follows:

1. Meets Requirements

   • A State’s 2020 RDA Determination is Meets Requirements if the RDA Percentage is at least 80%,¹⁰ unless the Department has imposed Specific or Special Conditions on the State’s last three IDEA Part C grant awards (for FFYs 2017, 2018, and 2019), and those Specific Conditions are in effect at the time of the 2020 determination.

2. Needs Assistance

   • A State’s 2020 RDA Determination is Needs Assistance if the RDA Percentage is at least 60% but less than 80%. A State would also be Needs Assistance if its RDA Determination percentage is 80% or above, but the Department has imposed Special or Specific Conditions on the State’s last three IDEA Part C grant awards (for FFYs 2017, 2018, and 2019), and those Specific Conditions are in effect at the time of the 2020 determination.

3. Needs Intervention

   • A State’s 2020 RDA Determination is Needs Intervention if the RDA Percentage is less than 60%.

4. Needs Substantial Intervention

   • The Department did not make a determination of Needs Substantial Intervention for any State in 2020.

Footnotes

¹⁰ In determining whether a State has met this 80% matrix criterion for a Meets Requirements determination, the Department will round up from 79.5% (but no lower) to 80%. Similarly, in determining whether a State has met the 60% matrix criterion for a Needs Assistance determination discussed below, the Department will round up from 59.5% (but no lower) to 60%.