MEMORANDUM

The Government Performance and Results Act (GPRA) of 1993 requires the Social Security Administration (SSA) to develop performance indicators that assess the relevant service levels and outcomes of each program activity. GPRA also requires disclosure of the processes used to verify and validate the measured values used to report on program performance. SSA is committed to ensuring the importance of verifying and validating performance measures, and the Office of the Inspector General (OIG) audits of the performance measures are a means to achieve this. Our objective was to assess the reasonableness of the methodology SSA used to measure the following two Fiscal Year (FY) 2000 GPRA performance indicators:

Our audit was limited to reviewing SSA’s methodology to obtain the data used to determine the number of individuals beginning TWPs or participating in 1619(a) status in FYs 1997 and 2000. We did not assess the reliability of the data SSA used for these two performance indicators.

Based on our audit, SSA’s methodology for measuring the FY 2000 GPRA performance indicators related to its TWP and 1619(a) programs was reasonable. Specifically, we reviewed SSA’s methodology, as well as the data used by SSA, for calculating the number of people beginning TWPs and participating in the 1619(a) program.

Many beneficiaries with disabilities want to work and become independent, and many can work despite their impairments if they receive the support they need. To that end, SSA’s programs contain various work incentive provisions. Currently, however, less than one-half of 1 percent of DI beneficiaries and about 1 percent of SSI recipients actually leave the programs because of work activity. SSA has established performance indicators under GPRA to increase the number of DI adult worker beneficiaries who begin TWPs and to increase the number of SSI disabled recipients, aged 18 to 64, participating in 1619(a) status. For FY 2000, SSA established goals of 10 percent increases relative to the FY 1997 base year numbers for each of these indicators.

Federal regulations provide for TWPs for most DI beneficiaries. The entitlement to a TWP generally begins with the month of entitlement and continues until the impairment is no longer disabling or until 9 TWP months are completed, whichever occurs earlier. The trial work provision is intended to give a disabled beneficiary the opportunity to test his or her ability to work and hold a job. The individual may work for 9 months (not necessarily consecutive) during which the work, regardless of magnitude, will not be used as the basis for determining whether there was demonstrated ability to engage in

substantial gainful activity (SGA). After the end of the TWP, the work may be considered in determining whether the individual demonstrated the ability to engage in SGA. If a beneficiary is determined to be engaging in SGA, SSA no longer considers the individual to be disabled.

A work incentive for SSI recipients is the 1619(a) provision. Section 1619(a) of the Social Security Act provides special SSI benefits to individuals who otherwise would lose eligibility for SSI payments under the regular rules because they engage in SGA. The person must continue to be severely impaired and meet all other eligibility rules.

The TWP and 1619(a) GPRA measures are interim measures that are being used by SSA while it implements the return to work provisions included in the Ticket to Work and Work Incentives Improvement Act of 1999. In SSA’s Annual Performance Plan for FY 2002, the following Strategic Objective was included: "By 2005, increase by 100 percent from 1999 levels, the number of [DI] and of SSI disability beneficiaries who achieve steady employment and no longer receive cash benefits."

Based on our audit, SSA’s methodology for measuring the FY 2000 GPRA performance indicators related to its TWP and 1619(a) programs was reasonable. Specifically, we reviewed SSA’s methodology, as well as the data used by SSA, for calculating the number of people beginning TWPs and participating in the 1619(a) program.

The chart below shows information published by SSA regarding its TWP GPRA goal, which was a 10-percent increase (over FY 1997) in the number of DI beneficiaries beginning TWPs.

To test SSA’s methodology for this measure, we requested from SSA data on its FY 1997 and FY 2000 participation levels. SSA provided a file containing 16,376 beneficiaries with TWP start dates in FY 1997 and a file containing 14,789 beneficiaries with TWP start dates in FY 2000. To test whether beneficiaries included in these files participated in TWPs, we selected a random sample of 100 cases each from the FY 1997 and FY 2000 files.

Of the 100 cases reviewed, we determined that 99 cases were correctly identified, and 1 case was incorrectly identified as having participated in a TWP. In this one case, the Master Beneficiary Record (MBR) for the beneficiary did not show that any TWP months were worked, and this beneficiary’s Detailed Earnings Query (DEQY) did not show any earnings for calendar years (CY) 1997, 1998 or 1999. Projecting the results of our sample to the population, we estimate that 16,212 beneficiaries included in SSA’s FY 1997 TWP file participated in TWPs. (See Appendix C.) Since our estimate of 16,212 beneficiaries participating in TWPs exceeds SSA’s published FY 1997 base year number of 16,000 beneficiaries by only 1.3 percent, we concluded that SSA’s methodology for developing its FY 1997 TWP performance measure baseline data was reasonable.

Of the 100 cases reviewed, we determined that 98 cases were correctly identified, and 2 cases were incorrectly identified as having participated in TWPs. In these two cases, the MBRs for the beneficiaries did not show that any TWP months were worked, and these beneficiaries’ DEQYs did not show any earnings for CY 2000. Projecting the results of our sample to the population, we estimate that 14,493 beneficiaries included in SSA’s FY 2000 file participated in TWPs. (See Appendix C.) Since our estimate of 14,493 beneficiaries is only 2 percent less than SSA’s published number of 14,789, we concluded that SSA’s methodology for developing its FY 2000 TWP performance measure figures was reasonable. However, as reported by SSA, the Agency did not meet its FY 2000 TWP goal of 17,600 TWP participants.

The chart below shows information published by SSA regarding its 1619(a) GPRA goal, which was a 10-percent increase (over FY 1997) in the number of SSI disabled recipients, aged 18-64, participating in 1619(a) status.

To test SSA’s methodology for this measure, we requested from SSA data on its FY 1997 and FY 2000 participation levels. SSA provided us a file containing 19,443 recipients in 1619(a) status in September 1997 and a file containing 25,772 recipients in 1619(a) status in September 2000. These files were based on estimated September 1997 and September 2000 wages and were extracted from the Supplemental Security Record (SSR) in August 1997 and August 2000, respectively. To test whether recipients included in these files were in 1619(a) status, we selected a random sample of 100 cases from each of the FY 1997 and FY 2000 files. Based on the results of our testing of SSA’s September 1997 and September 2000 1619(a) data files, we were unable to reach a conclusion to meet our objective. (See below.) As an alternative audit test, we then created and reviewed 1619(a) participant files for September 1997 and September 2000 based on information on the SSR at later dates.

Based on the results of our sample of SSA’s FY 1997 1619(a) data file—in which we found that only 47 of 100 sampled recipients were in 1619(a) status—we could not support SSA’s reported performance indicator level. This occurred because SSA’s FY 1997 performance level was computed based on estimated wages reported to SSA as of August 1997. Since, over time, those estimated wages were changed to verified wage amounts, many of the recipients identified by SSA to be in 1619(a) status during September 1997 were not actually in 1619(a) status based on their verified wages. Conversely, we recognized that the number of recipients who were actually in 1619(a) status in September could be higher if we identified people in 1619(a) status on a later date—after their estimated wages were verified by SSA.

To determine the actual number of recipients in 1619(a) status in September 1997, we performed additional audit tests. Specifically, using information contained on the SSR in June 2000, we created a file that contained 21,517 recipients in 1619(a) status as of September 1997. To test the reliability of these data, we randomly selected 100 of these 21,517 recipients for review. Based on wage and benefit payment information on the recipients’ SSRs for September 1997, all 100 cases were correctly identified as having participated in 1619(a) status in September 1997. Projecting the results of our sample to the population, we are 90-percent confident that at least 21,028 recipients (which is 6.4 percent greater than SSA’s published FY 1997 base year number of 19,767) were in 1619(a) status in September 1997. (See Appendix C.) Thus, although SSA’s FY 1997 data file of 19,443 cases was based on estimated earnings for September 1997, our review of individuals with verified earnings for September 1997 showed that SSA’s methodology for developing the FY 1997 1619(a) performance indicator was reasonable. Since earnings can take months—and sometimes years—to verify, SSA’s use of estimated earnings was reasonable for developing this performance measure. The estimated earnings data was the best information available to SSA at the time.

Based on the results of our sample of SSA’s FY 2000 1619(a) data file—in which we found that only 62 of 100 sampled recipients were in 1619(a) status—we could not support SSA’s reported performance indicator level. This occurred because SSA’s FY 2000 performance level was computed based on estimated wages reported to SSA as of August 2000. Since, over time, those estimated wages were changed to verified wage amounts, many of the recipients identified by SSA to be in 1619(a) status during September 2000 were not actually in 1619(a) status based on their verified wages. Conversely, we recognized that the number of recipients who were actually in 1619(a) status in September 2000 could be higher if we identified people in 1619(a) status on a later date—after their estimated wages were verified by SSA.

To determine the actual number of recipients in 1619(a) status in September 2000, we performed additional audit tests. Specifically, using information contained on the SSR in November 2001, we created a file that contained 34,184 recipients in 1619(a) status as of September 2000. To test the reasonableness of SSA’s methodology, we randomly selected 100 of these 34,184 recipients for review. Based on wage and benefit payment information on the recipients’ SSRs for September 2000, all 100 cases were correctly identified as having participated in 1619(a) status in September 2000. Projecting the results of our sample to the population, we are 90-percent confident that at least 33,407 recipients (which is 29.6 percent greater than SSA’s published FY 2000 number of 25,772) were in 1619(a) status in September 2000. (See Appendix C.) Thus, although SSA’s FY 2000 data file of 25,772 cases was based on estimated earnings for September 2000, our review of recipients with verified earnings for September 2000 showed that SSA’s methodology for developing the FY 2000 1619(a) performance indicator was reasonable. Since earnings can take months—and sometimes years—to verify, SSA’s use of estimated earnings was reasonable for developing this performance measure. The estimated earnings data was the best information available to SSA at the time.

To assess whether participating in a TWP or being in 1619(a) status resulted in these individuals leaving SSA’s programs due to work activities, we determined the payment status for our FY 1997 sample cases as of October 2001. For this analysis, we used the pay status of the individuals included in our two samples extracted from SSA’s FY 1997 TWP and 1619(a) files. The pay status on the MBRs or SSRs as of October 2001 are shown in the following table:

The table above shows that 78 of the 200 sampled cases (39 percent) did not receive benefit payments in October 2001 due to income or resource issues. This indicates that SSA’s TWP and 1619(a) work incentive programs may be encouraging individuals to return to work.

Based on our audit, SSA’s methodology for measuring the FY 2000 GPRA performance indicators related to its TWP and 1619(a) programs was reasonable. However, we found that—to report on this performance measure in a timely manner—SSA must rely on estimated wages to calculate the number of recipients in 1619(a) status. Since SSA is using estimated wages to calculate this measure, we recommend that SSA include a statement in future performance reports that its "actual" performance levels are based on estimated wage amounts, and, as a result, may change over time as those wage amounts are verified.

In response to our draft report, SSA agreed with our recommendation. Specifically, SSA will include a statement in future performance reports. (See Appendix F for SSA’s comments.)

James G. Huse, Jr.

Appendices

APPENDIX B – Scope and Methodology

APPENDIX C – Sample Results

APPENDIX D – Flowchart for Input of Trial Work Period (TWP) Information to the Master Beneficiary Record and Extraction of TWP Performance Measure Files

APPENDIX E – Flowchart for Input of 1619(a) Information to the Supplemental Security Record and Extraction of 1619(a) Performance Measure Files

APPENDIX F – Agency Comments

APPENDIX G – OIG Contacts and Staff Acknowledgments

Scope and Methodology

Our objective was to assess the reasonableness of the methodology the Social Security Administration (SSA) used to measure the number of disabled beneficiaries who are working.

To test the accuracy and reliability of SSA’s performance data, we:

• identified the data collection process used by SSA in preparing the data for its Annual Performance Plan;

• interviewed personnel at data entry/collection sites to gain an understanding of how data is collected and input into SSA’s systems;

• obtained files from SSA of the individuals who began trial work periods (TWP) and participated in the 1619(a) program in Fiscal Years (FY) 1997 and 2000;

• obtained a random sample of 100 beneficiaries/recipients from each group and tested those records to determine whether the individuals actually began TWPs or received 1619(a) benefits;

• created a file (as of June 2000) of recipients in 1619(a) status in September 1997, and reviewed 100 sample cases; and

• created a file (as of November 2001) of recipients in 1619(a) status in September 2000, and reviewed 100 sample cases.

In conducting this audit, we also:

• reviewed SSA’s Performance and Accountability Report and SSA’s Annual Performance Plan for FY 2000 to determine the baseline data, definition, and data source for the performance indicator;

• researched the Program Operations Manual System, the Code of Federal Regulations, and the Social Security Act; and

• determined the pay status in October 2001 of the beneficiaries/recipients in our random samples based on SSA’s FY 1997 TWP and 1619(a) files.

Our audit was limited to reviewing SSA’s methodology in determining the number of individuals beginning TWPs or participating in the 1619(a) program in FYs 1997 and 2000. We could not assess the reliability of the data SSA used for these two performance indicators. Also, we could not test the completeness of SSA’s FY 1997 or FY 2000 TWP files. Our audit tests were limited because SSA extracted these files from the MBR in January 1998 and January 2001, respectively, but continued to retroactively place individuals in TWPs after creating these files.

We performed our audit between June and December 2001 in Boston, Massachusetts. The entity audited was the Office of Employment Support Programs within the Office of the Deputy Commissioner, Disability and Income Security Programs. We conducted our audit in accordance with generally accepted government auditing standards.

Appendix C

Sample Results

From the Social Security Administration (SSA), we obtained files of individuals who began trial work periods (TWP) in Fiscal Years (FY) 1997 and 2000 and individuals who were in 1619(a) status in September 1997 and September 2000.

Based on the results of our testing of SSA’s September 1997 and September 2000 1619(a) data files that were based on estimated income, we were unable to reach a conclusion to meet our audit objective. As an alternative audit test, we created and reviewed files of recipients in 1619(a) status in September 1997 and September 2000 based on information on the Supplemental Security Record (SSR) at later dates. Our second sample of September 1997 1619(a) data was based on information on the SSR in June 2000, and our second sample of September 2000 1619(a) data was based on information on the SSR in November 2001.

Our sample results and projections are detailed in the tables below.

Note: All projections are at the 90 percent confidence level.

Note: All projections are at the 90-percent confidence level.

Note: All projections are at the 90 percent confidence level.

Note: All projections are at the 90 percent confidence level.

Note: All projections are at the 90 percent confidence level.

Agency Comments

We appreciate the OIG’s efforts in conducting this review. Our comments on the report content and recommendation are attached.

Appendix G

OIG Contacts and Staff Acknowledgments

Acknowledgments

In addition to the persons named above: