A-02-00-10017 Alternate Format

Office of the Inspector General
Larry G. Massanari
Acting Commissioner of Social Security

Inspector General

Performance Measure Review: Reliability of the Data Used to Measure
Disability Claims Processing (A-02-00-10017)

The attached final report presents the results of our audit. Our objective was to assess the reliability of the Social Security Administration's performance measurement data related to the disability claims process of title II Disability Insurance claims and title XVI Supplemental Security Income disability claims. Specifically, this review examined the performance data used to measure Fiscal Year 1999: initial disability claims processing times (days); number of initial disability claims processed; and the number of initial disability claims pending.

Please comment within 60 days from the date of this memorandum on corrective action taken or planned on each recommendation. If you wish to discuss the final report, please call me or have your staff contact Steven L. Schaeffer, Assistant Inspector General for Audit, at (410) 965-9700.

James G. Huse, Jr.

OFFICE OF THE INSPECTOR GENERAL

SOCIAL SECURITY ADMINISTRATION

PERFORMANCE MEASURE REVIEW:

RELIABILITY OF THE DATA USED TO

MEASURE DISABILITY CLAIMS

PROCESSING

November 2001

A-02-00-10017

AUDIT REPORT

Office of the Inspector General

Larry G. Massanari
Acting Commissioner of Social Security

Inspector General

Performance Measure Review: Reliability of the Data Used to Measure Disability Claims Processing (A-02-00-10017)

The Government Performance and Results Act (GPRA) of 1993 requires the Social Security Administration (SSA) to develop performance indicators that assess the relevant service levels and outcomes of each program activity. GPRA also calls for a description of the means employed to verify and validate the measured values used to report on program performance. The objective of this audit was to assess the reliability of SSA's data used to measure the following Fiscal Year (FY) 1999 GPRA performance indicators:

FY 1999 FY 1999
Indicator Plan Actual

Initial disability claims processing times (days) 100 105

Number of initial disability claims processed 2,090,000 2,013,089

Number of initial disability claims pending 408,000 457,823

BACKGROUND

SSA oversees two long-term disability programs. The Disability Insurance (DI) program was established in 1954 under title II of the Social Security Act (Act). The program is designed to provide benefits to wage earners and their families in the event the wage earner becomes disabled. The Supplemental Security Income (SSI) program was created as a result of the Social Security Amendments of 1972 with an effective date of January 1, 1974. SSI, (title XVI of the Act) provides a nationally uniform program of income to financially needy individuals who are aged, blind and/or disabled.

To determine eligibility, the claimant must first file a disability claim with SSA. Personnel in 1 of SSA's approximately 1,300 field offices (FO) conduct the initial interview with disability applicants and assist them in completing the application. The Modernized Claims System (MCS) is used to key in applications for DI. The Modernized SSI Claims System is used to key in applications for SSI. Interviews for both are conducted via the telephone or in person. Initial interviews are made to determine the applicant's non-medical eligibility on the basis of income, resources, and work history. Basic medical information concerning the disability, medical treatments, and identification of treating sources is also obtained.

Upon meeting the non-medical eligibility requirements, SSA sends the claims file to a State Disability Determination Services (DDS) office. SSA is responsible for implementing the general policies governing the development of disability claims under the DI and SSI programs. SSA relies on DDSs for disability determinations under both DI and SSI. In carrying out its obligation, each DDS is responsible for determining claimants' disabilities and ensuring that adequate evidence is available to support its determinations. Once the DDS makes a disability determination, it notifies the FO, and a letter is sent informing the claimant of the determination and of his/her appeal rights.

When a determination is made for a DI claim, MCS updates the Workload Management System (WMS), and staff manually updates the Social Security Administration Claims Control System (SSACCS). The Management Information Initial Claims Record (MIICR) reads clearance information from WMS and SSACCS. MIICR writes data for the completed claim into the MIICR Master File, which creates a file of completed claims for the week. MIICR then creates a monthly file of completed claims and monthly summary reports, which are obtained through the GETRSDHI module. These reports are provided to the Office of Strategic Management for inclusion in SSA's Annual Accountability Report.

When a determination is made for a SSI claim, an initial determination date is posted to the Supplemental Security Record, and claim data are forwarded to the SSI Claims Exception Control System. This system ensures the claim is complete before the data is sent to the SSI Claims Report (SSICR), which is a process that compiles the claims
data for inclusion in various management information reports. These reports are obtained through the GETSSICR module and provide statistics, which are included in the annual Accountability Report.

SSA calculates the initial disability claims processing times (days) for inclusion in the Accountability Report by obtaining monthly figures from the GETRSDHI and GETSSICR modules. In a manual process, title II and title XVI monthly claims are recorded on a spreadsheet and multiplied by their respective number of processing days to get the "total number of processing days." Title II and title XVI monthly processed claims are added to obtain the FY disability processed claims, and title II and title XVI processing days are added to result in the FY disability processing days. These two sums are then divided (total number of processing days by total number of processed claims) to result in the average FY claims processing days.

SSA calculates the number of initial disability claims processed and initial disability claims pending for inclusion in the Accountability Report by obtaining monthly figures from the National Disability Determination Services System's (NDDSS) State Agency Operations Report, FD-14. Claims are summarized and sorted for inclusion in various weekly, monthly, quarterly, semiannual and annual reports. The monthly title II and title XVI processed and pending claims are recorded on a spreadsheet and totaled at the end of the FY. A small number of initial disability Hospital Insurance/Supplementary Medical Insurance (HI/SMI) Federal Employee Medicare Only case claims are added to this figure, which results in the FY processed and pending claims.

RESULTS OF REVIEW

We reviewed the reliability of the data used to measure the disability claims process for initial disability claims processing times (days) and the number of initial disability claims processed and initial disability claims pending. Although the data used to measure the disability claims process was reliable, we found several conditions, which affected these measures. These included an overstatement of the number of processed claims in the Accountability Report and a lack of documentation on the processes used to calculate the number of processed and pending claims. We also found that SSA used a combined measure of both the DI and SSI programs for reporting the initial disability claims average processing time, which does not take into consideration the differences in the DI and SSI programs. SSA also uses inconsistent terminology and definitions for the disability performance indicators.

PERFORMANCE MEASURE DATA WAS RELIABLE

We found the data SSA used in calculating the initial disability claims processing times (days) was reliable. We also found the number of initial disability claims pending was calculated and reported correctly. However, although the data for the number of initial disability claims processed was reliable, it was calculated incorrectly. This resulted in the reporting of an incorrect number of processed claims for FY 1999.

OVERSTATEMENT OF THE NUMBER OF PROCESSED CLAIMS

The FY 1999 goal for the number of initial disability claims processed was 2,090,000 and SSA reported 2,013,089 claims. For pending disability claims, the goal was 408,000 claims, and SSA reported 457,823 claims.

Our review disclosed that the reported number of 457,823 for pending claims was a reliable measurement of performance. However, we found that the number of processed claims was misstated. SSA reported that the actual number of processed claims for FY 1999 was 2,013,089. We calculated that there were 2,012,047 claims. The difference of 1,042 claims represents approximately one-half of one percent of the total. A calculation error occurred when SSA manually added the 12 months of claims posted from their spreadsheets. In the manual calculation, the initial claims were properly posted to a spreadsheet. However, when adding the number of initial disability HI/SMI claims to the total, some HI/SMI claims for two other categories, which were not initial claims-reconsideration and other-were added to the total.

We discussed the improper calculation of the processed claims with SSA staff responsible for the calculation. It was agreed that the 1,042 HI/SMI claims should not have been included in the calculation. Also, as the processed claims for FY 2000 was being calculated and submitted for reporting at the time of our discussion on this matter, we were assured, and we verified, that this same error was not repeated. Further, the figure reported in the FY 2000 Performance and Accountability Report representing the number of initial claims processed in FY 1999 was adjusted to reflect the correct number of 2,012,047 claims.

DOCUMENTATION FOR THE PERFORMANCE MEASURE INDICATORS FOR THE NUMBER OF PROCESSED AND PENDING CLAIMS WAS LACKING

The Office of Management and Budget Circular No. A-123, Management Accountability and Control states, "The documentation for transactions, management controls, and other significant events must be clear and readily available for examination." Furthermore, GPRA requires agencies to ". . . describe the means to be used to verify and validate measured values." While some of the criteria needed to perform the calculations used to generate the performance measures are contained in the Management Information Manual, Part IV, documentation describing the methods used to calculate the number of initial disability claims processed and initial disability claims pending does not exist. The lack of documentation for the calculation of the number of initial disability claims processed and initial disability claims pending does not provide the audit trail necessary to assist in the verification of these performance measure indicators.

THE MEASUREMENT OF THE DI AND SSI INITIAL DISABILITY CLAIMS AVERAGE PROCESSING TIME IS COMBINED

For FY 1999, the performance indicator for the initial disability claims average processing times was 100 days, and SSA's performance was 105 days. SSA calculated this performance by combining both the DI and SSI claims processing times. Historically, there have been differences in the amount of time it takes to process cases in the different programs. In fact, the Code of Federal Regulations established different threshold levels for DI and SSI disability claims processing by DDSs. We found that in FY 1999, the average time to process a title II claim was 94.33 days, as compared to 114.02 days for title XVI. By combining the two types of claims, SSA does not accurately reflect the performance of the two disability programs.

TERMINOLOGY USED IN PERFORMANCE INDICATORS IS INCONSISTENTLY DEFINED

The reporting in the Accountability Report for two of the indicators in this audit may be misleading. The two indicators share a similar name, but measure unrelated statistics and use different data bases to do so. The title of one of the indicators is Initial disability claims processing times (days), and the other is Number of initial disability claims processed. The processing times (days) indicator measures the processing time of all disability claims from the effective filing date to the date of determination. This indicator is based on all claims that were filed at a FO, whether or not they were subsequently entered into the NDDSS. However, the claims processed indicator, although it also refers to "initial disability claims," only includes cases entered into the NDDSS and does not account for those claims that may have been decided at the FO level. This indicator represents NDDS' workload, not the total number of initial disability claims. For FY 1999, SSA calculated the processing times (days) from a data base of 2,647,161 claims, while the number of claims processed was reported to be 2,013,089 claims.

Furthermore, the beginning date for the purposes of calculating processing time is inconsistently determined. The effective date of the application for initial SSI disability claims is generally the application date, which can be as soon as the date that a claimant first contacts SSA or as late as the first day of the month for which a claimant is found eligible for benefits. However, SSA policy stipulates that the application date should be changed to reflect the date of filing if these two dates are in the same month. This allows the Agency to shorten reported processing time without disadvantaging the claimant by withholding a month of payment. SSA staff, however, does not consistently make this change. For cases in which the application date and the date of filing fall into different months, no adjustment is made to the application date because such an adjustment could cost the claimant his/her first monthly payment. The beginning date can therefore reflect the date of filing for some cases or the date of first contact with the Agency in other cases.

It is also possible for an application date to be restricted to a future date reflecting the first day of the month for which a claimant is eligible for benefits based on program requirements. A future application date can result in negative processing times if the application is processed before this date. These negative processing times are counted as zero when calculating average processing time. Errors in restricting the application date, or other errors in changing the application date, may also result in negative processing times or incorrectly shortened processing time. SSA policy allows for changes to be made by FO staff to the application date. This date therefore lacks computer controls, which would help guard against data errors and protect the integrity of processing time data.

CONCLUSIONS AND RECOMMENDATIONS

Our audit found that the data used by SSA to report on the timely processing of initial disability claims processing times (days), the number of initial disability claims processed, and initial disability claims pending was reliable. However, our review showed that: (1) the 2,013,089 processed claims identified in the FY 1999 Accountability Report was overstated by 1,042, which represents one-half of one percent of the number of initial disability claims processed; (2) documentation describing the method used to calculate the number of processed and pending cases does not exist; (3) the measurement of initial disability claims average processing time represents a combined measure of both DI and SSI claims processing rather than individual measures for DI and SSI; and, (4) the term "initial disability claims" is inconsistently applied, as is criteria used to determine the beginning date for measuring processing time.

To ensure the proper reporting of the number of processed and pending claims, and to better reflect actual processing times, we recommend SSA take the following corrective actions:

1. Ensure that the number of initial disability claims processed are calculated and reported correctly;

2. Prepare documentation describing the methods used to calculate the number of initial disability claims processed and initial disability claims pending, including the addition of the HI/SMI claims;

3. Separate the reporting of the initial disability claims average processing times by program, rather than combining both the DI and SSI claims processing
times as one measure so differences between the programs, with their separate customer bases, can be identified;

4. Clearly define performance indicators to reflect the types of initial disability claims actually measured; and,

5. Apply consistent criteria for determining the beginning dates used in processing time.

AGENCY COMMENTS

SSA agreed in principle with all five of our recommendations. SSA further commented on three of the recommendations. In response to Recommendation 1, SSA agreed that an accurate count of disability claims is essential to measuring SSA's performance in the processing of disability claims. However, SSA noted that, at the time of our review, the number of initial disability claims processed was determined by calculation. SSA stated that with the initiation of the Disability Operational Data Store (DODS), the actual number of disability claims is now counted, not calculated. For recommendation 3, SSA agreed with separate reporting of processing times by program for internal purposes, but believes that separate measures are not appropriate for external reporting. Finally, for Recommendation 5, SSA agreed that using consistent criteria in processing time is critical to the overall performance. SSA added, however, that unique differences when a potential beneficiary may apply for and become eligible for benefits between the title II and title XVI programs must be considered when determining the beginning dates used for processing times.

Additionally, SSA noted that the term "decision" is defined in the regulations at 20 C.F.R. § 404.901 as used when deciding a claim at the Administrative Law Judge and the Appeals Council levels and that "determination" should be used at the initial or reconsideration levels of the administrative review process.

OFFICE OF THE INSPECTOR GENERAL RESPONSE

We are pleased that SSA agreed in principle with our recommendations. Regarding SSA's response to Recommendation 1, we note SSA's assertion that the DODS will provide an accurate account of claims. This audit did not review the DODS. Concerning SSA's response to Recommendation 3, we continue to believe that separate reporting of processing times by program is appropriate for both internal purposes and external reporting. In regard to SSA's response to Recommendation 5, we agree that different programs require different policies concerning the start date when determining processing times. Our report noted that SSA used different start dates for different cases within the SSI program. We believe the same criteria for a start date should be used within each program.

We look forward to receiving an implementation plan detailing the steps the Agency will take to improve the performance measures concerning the processing of initial disability claims.

James G. Huse, Jr.

Appendices

APPENDIX A - Scope and Methodology

APPENDIX B - Agency Comments

APPENDIX C - Acronyms

APPENDIX D - Flowchart of Disability Claims Processing Performance Measures

APPENDIX E - OIG Contacts and Staff Acknowledgments

Appendix A

Scope and Methodology

This audit was conducted to assess the reliability of the Social Security Administration's (SSA's) performance data used to measure the initial disability claims processing times (days), the number of initial disability claims processed, and the number of initial disability claims pending.

To test the reliability of SSA's performance data, we performed various audit steps and relied on work performed in two recent Office of the Inspector General audits with similar performance indicators. These audits included:

A Performance Measure Review: Reliability of the Data Used to Measure the Timeliness of Processing Supplemental Security Income Disability Claims . The objective of this audit was to assess the reliability of SSA's performance measurement data for the percent of Supplemental Security Income (SSI) Disability claims decided within 60 days of filing. In this audit, we assessed the reliability of the data by replicating the processing time performance measure statistic for the period March through May 1999 and compared the replicated statistics to the Processing Times Report.

A Performance Measure Review: Reliability of the Data Used to Measure the Timely Processing of Disability Insurance Claims (A-02-99-11001). The objective of this audit was to assess the reliability of SSA's performance measurement data for the percent of Disability Insurance claims decided within 6 months after onset or within 60 days after effective filing date, whichever is later. In this audit, which will be finalized shortly, we assessed the reliability of the data by replicating the processing time performance measure statistic for 3 months, including November 1999, December 1999, and February 2000. We also recalculated the percent of timely cases for each of the 3 months reviewed and compared the results to SSA's Service Delivery Objective 14 reports.

To test the reliability of SSA's performance data for the disability claims processing times (days) indicator for this audit, we:

assessed the reliability of the data by replicating the processing time performance measure statistic for 3 months. Based on available data, this included November 1999, December 1999, and February 2000 for the Management Information Initial Claims Records data, and November 1999, December 1999 and January 2000 for the SSI Claims Report data;

reviewed Management Information Manual (MIM) criteria for calculating processing time and applied such criteria to data reviewed;

compared and reconciled the number of monthly initial disability claims for the data reviewed to the numbers reported by SSA in their monthly Field Office (FO) Initial Disability Claims Report-Processing Times for title II and FO Initial SSI Blind & Disabled Claims Report-Processing Times for title XVI;

reconciled the number of title II and title XVI processed claims as stated in the monthly FO Initial Disability Claims Report-Processing Times and FO Initial SSI Blind & Disabled Claims Report-Processing Times for the entire Fiscal Year (FY) 1999 to SSA's FY 1999 Accountability Report;

compared key dates in the data files to check for relationships between pertinent dates (i.e. beginning dates, end dates, and date of filing);

checked that the data reviewed contained valid entry codes and dates, and;

recalculated and verified the average processing time for initial disability claims as shown in the Accountability Report.

To test the accuracy and reliability of SSA's performance data for FY 1999 for the number of initial disability claims processed and initial disability claims pending indicators for this audit, we:

obtained record layouts for the State Agency Operations Reports (SAOR), which is a completed claims annual report for the National Disability Determination Services System (NDDSS);

identified, extracted, and summarized, through the use of IDEA, the pertinent fields from the SAOR files produced by NDDSS and calculated the number of initial disability claims processed and initial disability claims pending;

reviewed and applied MIM criteria to data files reviewed for determining the number of processed and pending claims;

assessed the reliability of the data by verifying that the data files agreed with the monthly SAOR reports;

verified that the number of initial disability claims processed and initial disability claims pending shown in the Accountability Report agreed with SSA data files.

In conducting this audit, we also:

reviewed SSA's Accountability Report for FY 1999 and SSA's Annual Performance Plan for FY 1999 to determine the baseline data, definitions, and data sources for the performance indicators;

reviewed pertinent Government Performance and Results Act and Office of Management and Budget laws and regulations;

interviewed SSA staff from various departments to gain an understanding of the disability claims process as it applied to the three indicators audited; and,

flowcharted the disability claims process from the claimant's initial contact with SSA to the generation of the reports for the three indicators audited (see Appendix C).

The review of controls over the various stages of the disability claims process was based primarily on previous reviews performed by PricewaterhouseCoopers (PwC), as well as work completed in the two Office of Inspector General reports mentioned at the beginning of this section. PwC is a certified public accounting firm, contracted to perform the FY 1998 and 1999 Financial Statement Audits and audits of certain performance measures related to SSI Aged claims processing. Our work was conducted at OIG's New York FO and SSA Headquarters in Baltimore, Maryland. The field work was conducted from July 2000 to May 2001. The entity audited was the Office of Information Management within the Office of Systems. Our audit was performed in accordance with generally accepted government auditing standards, as it pertains to performance monitoring audits.

Appendix B

Agency Comments

COMMENTS OF THE SOCIAL SECURITY ADMINISTRATION (SSA) ON THE OFFICE OF THE INSPECTOR GENERAL (OIG) DRAFT REPORT, "PERFORMANCE MEASURE REVIEW: RELIABILITY OF THE DATA USED TO MEASURE DISABILITY CLAIMS PROCESSING" A-02-00-10017

We appreciate the opportunity to comment on the draft report. Following are our comments on the recommendations.

Recommendation 1

Ensure that the number of initial disability claims processed are calculated and reported correctly.

SSA Comment

We agree. An accurate count of disability claims is essential to measuring SSA's performance in the processing of disability claims. However, at the time the review was conducted, the number of initial disability claims processed was determined by calculation. With the initiation of the Disability Operational Data Store (DIODS), the actual number of disability claims is now counted and not calculated. We suggest that the advent of DIODS be noted in the subject report.

Also, the problem indicated in the report concerning erroneous inclusion of some Hospital Insurance/Supplementary Medical Insurance (HI/SMI) cases has been corrected.

Recommendation 2

Prepare documentation describing the methods used to calculate the number of initial disability claims processed and initial disability claims pending, including the addition of the HI/SMI claims.

SSA Comment

We agree. Additional documentation regarding calculation methods will be added.

Recommendation 3

Separate the reporting of the initial disability claims average processing times by program, rather than combine both the Disability Insurance and Supplemental Security Income claims processing times as one measure so differences between the programs, with their separate customer bases, can be identified.

SSA Comment

We agree with separate reporting of processing times by program for internal purposes.
We previously provided comments to this recommendation in an earlier OIG report, "Review of the Social Security Administration's Fiscal Year 2000 Annual Performance Plan" (A-02-99-03007), issued in November 1999. We still believe that separate measures are not appropriate for external reporting. We want to note that the Management Information Initial Claims Records and Supplemental Security Income Claims Report, along with the Strategic Plan Report, already report disability claims processing numbers and times by program. With regard to the recommendation mentioned in the November 1999 report, SSA agreed that separate measures for title II and title XVI are reliable and sensible management tools for internal tracking.

Recommendation 4

Clearly define performance indicators to reflect the types of initial disability claims actually measured.

SSA Comment

We agree. Performance indicators should be clearly defined to reflect the types of initial disability claims actually measured in order to have all report categories not only well defined, but also consistent throughout both programs and all reports. The initial disability claims processing time will be included in the Fiscal Year 2002 Performance Plan.

Recommendation 5

Apply consistent criteria for determining the beginning dates used in processing time.

SSA Comment

We agree that using consistent criteria in processing time is critical to the overall performance. However, we must add that there are differences in the title II and title XVI programs in terms of when a potential beneficiary may apply for benefits and become eligible for benefits. These unique differences must be considered when determining the beginning dates used for processing times.

Other Comments

We wish to point out that "decision" is a defined term in the regulations at 20 C.F.R. § 404.901. This item is used when determining a claim at the Administrative Law Judge and the Appeals Council levels and "determination" is used at the initial or reconsideration levels of the administrative review process.

Also, Public Law No. 104-193, section 204(a), amended subparagraphs (A) and (B) of section 1611(c)(7) of the Social Security Act with respect to applications for benefits under title XVI filed on or after August 22, 1996. Generally, section 1611(c)(7) states that the effective date of an application for SSI benefits is the first day of the month following the later of the date the application is filed or the date the individual becomes eligible for benefits. The second sentence in the third paragraph on page 5, under Terminology Used in Performance Indicators is Inconsistently Defined, should be revised. It is not clear whether the effective date of the application or the actual number of days it took to process the decision is being addressed.

Appendix C

Acronyms

DDS Disability Determination Services
DI Disability Insurance
DODS Disability Operational Data Store
FO Field Office
FY Fiscal Year
GPRA Government Performance and Results Act
HI/SMI Hospital Insurance/Supplementary Medical Insurance
MCS Modernized Claims System
MIICR Management Information Initial Claims Records
MIM Management Information Manual
MSSICS Modernized Supplemental Security Income Claims System
NDDSS National Disability Determination Services System
OIG Office of the Inspector General
OIM Office of Information Management
OMB Office of Management and Budget
PwC PricewaterhouseCoopers
RSDHI Retirement, Survivors, Disability and Health Insurance
SA State Agency
SAOR State Agency Operations Report
SDO Service Delivery Objective
SSA Social Security Administration
SSACCS Social Security Administration Claims Control System
SSI Supplemental Security Income
SSICR Supplemental Security Income Claims Report
WMS Work Management System

Appendix D

Flowchart of Disability Claims Processing Performance Measures

Appendix E

OIG Contacts and Staff Acknowledgements

OIG Contacts

Frederick C. Nordhoff, Director, Financial Management and Performance Monitoring Audit Division, (410) 966-6676
Timothy Nee, Deputy Director, (212) 264-5295

Acknowledgements

In addition to those named above:

Arthur Treglia, Senior Auditor-in-Charge
Jackie Patel, Auditor
Denise Ramirez, Program Analyst
Patrick Kennedy, Audit Manager
Annette DeRito, Program Analyst

For additional copies of this report, please visit our web site at www.socialsecurity.gov/oig or contact the Office of the Inspector General's Public Affairs Specialist at (410) 966-1375. Refer to Common Identification Number A-02-01-10017.