To: The Commissioner

From: Inspector General

Subject: Summary of the Office of the Inspector General’s Reviews of the Social Security Administration’s Performance Data (A-02-03-13033)

The attached final report presents the results of our audit. Our objectives were to summarize the results of previous reviews of, and identify common issues related to, the Social Security Administration’s performance data and indicators.

Please comment within 60 days from the date of this memorandum on corrective action taken or planned on each recommendation. If you wish to discuss the final report, please call me or have your staff contact Steven L. Schaeffer, Assistant Inspector General for Audit, at (410) 965-9700.

James G. Huse, Jr.

Mission

We improve SSA programs and operations and protect them against fraud, waste, and abuse by conducting independent and objective audits, evaluations, and investigations. We provide timely, useful, and reliable information and advice to Administration officials, the Congress, and the public.

Authority

The Inspector General Act created independent audit and investigative units, called the Office of Inspector General (OIG). The mission of the OIG, as spelled out in the Act, is to:

Conduct and supervise independent and objective audits and investigations relating to agency programs and operations.
Promote economy, effectiveness, and efficiency within the agency.
Prevent and detect fraud, waste, and abuse in agency programs and operations.
Review and make recommendations regarding existing and proposed legislation and regulations relating to agency programs and operations.
Keep the agency head and the Congress fully and currently informed of problems in agency programs and operations.

To ensure objectivity, the IG Act empowers the IG with:

Independence to determine what reviews to perform.
Access to all information necessary for the reviews.
Authority to publish findings and recommendations based on the reviews.

Vision

By conducting independent and objective audits, investigations, and evaluations, we are agents of positive change striving for continuous improvement in the Social Security Administration's programs, operations, and management and in our own office.

Executive Summary
OBJECTIVE

Our objectives were to summarize the results of previous Office of the Inspector General (OIG) reviews of, and identify common issues related to, the Social Security Administration’s (SSA) performance data and indicators. This report summarizes the major findings and recommendations contained in 34 OIG reports related to the reliability of SSA’s performance data and indicators.

BACKGROUND

The Government Performance and Results Act, the President’s Management Agenda, and Office of Management and Budget guidance have highlighted the importance placed on performance measurement activities in the effective management of public programs. Similarly, we have highlighted the importance of reliably measuring program performance within our statement on SSA’s major management challenges.

The OIG has played a critical role in the oversight of SSA’s program performance. We evaluated the processes and systems SSA used to measure its performance and issued 34 reports assessing the reliability of the data used to measure 53 of SSA’s performance indicators.

RESULTS OF REVIEW

Between FYs 1998 and 2002, we audited 53 of SSA’s performance indicators and found 37 of the indicators (70 percent) were supported by reliable data, and 5 indicators (9 percent) were based on data that were unreliable. We found the data to be unreliable because of their incompleteness or weaknesses in the controls used to collect, analyze or report the data. We were unable to issue an opinion on 11 indicators (21 percent) because of limitations on, or the unavailability of, data. The issues that led to our conclusions are explained in greater detail in the following sections of this report.

We have offered many recommendations in our reviews to help improve SSA’s performance measurement processes. The Agency has taken significant action to implement many of our suggested changes. Over the last few years, SSA has disclosed data limitations, changed performance indicators, and improved data collection systems in response to our recommendations. However, SSA still needs to implement some outstanding recommendations that will help improve its performance measurement processes. SSA especially needs to address recommendations to improve the data used to support 13 of the indicators we could not review or found to be unreliable.

CONCLUSIONS AND RECOMMENDATIONS

SSA needs to continue its efforts toward documenting all performance measurement processes, ensuring its performance data are complete, and implementing adequate internal controls over its data systems and processes. Specifically, SSA should:

• Implement all outstanding recommendations needed to produce reliable performance data and improve SSA’s performance measurement processes.

• Ensure process documentation and internal controls over the collection, processing, and reporting of data are complete and adequate for any new indicators established in the future.

AGENCY COMMENTS

SSA partially agreed with Recommendation 1. According to its records, the Agency considered 7 of the 20 recommendations to be open and the remaining 13 recommendations listed in our draft report to be closed. SSA provided an updated status for all 20 recommendations as part of its comments.

The Agency maintained it is already conducting the activities in Recommendation 2 and will continue to implement this process for any new indicators established in the future. (See Appendix B for the full text of SSA’s comments.)

OIG RESPONSE

We appreciate SSA’s updates on the status of the outstanding recommendations listed in our report and its efforts to resolve those that are still open. The Agency’s update will allow us to close 7 of the 20 recommendations listed in our report. However, we disagree with the Agency’s closing of six recommendations. Our reasons for disagreeing are discussed in detail in the body of this report.

Concerning our Recommendation 2, our audit work in this area has documented multiple instances in which SSA did not have adequate documentation of, and/or internal controls over the collection, processing, and reporting of its performance processes and data. We appreciate the Agency’s efforts to ensure such documentation and controls are in place for new indicators established in the future.

Table of Contents
Page
INTRODUCTION 1
RESULTS OF REVIEW 4
Performance Data Reliability 4
Measurement Process Documentation 5
Performance Measurement Process Completeness 7
Internal Controls in Performance Measurement Systems 8
CONCLUSIONS AND RECOMMENDATIONS 10

APPENDICES
Appendix A – Office of the Inspector General Reports on the Reliability of the Social Security Administration’s Performance Measure Data (Fiscal Years 1998 Through 2002)
Appendix B – Agency Comments
Appendix C – OIG Contacts and Staff Acknowledgments

Acronyms
ALJ Administrative Law Judge
APP Annual Performance Plan
DI Disability Insurance
FY Fiscal Year
GPRA Government Performance and Results Act
OIG Office of the Inspector General
OMB Office of Management and Budget
RJ Reviewing Judge
SSA Social Security Administration
SSI Supplemental Security Income
SSN Social Security Number
VIP Visitor Intake Process

Introduction
OBJECTIVE

Our objectives were to summarize the results of previous Office of the Inspector General (OIG) reviews of, and identify common issues related to, the Social Security Administration’s (SSA) performance data and indicators. This report summarizes the major findings and recommendations contained in 34 OIG reports related to the reliability of SSA’s performance data and indicators.

BACKGROUND

SSA uses performance indicators to measure the performance of its programs and its ability to meet programmatic and strategic goals. The Agency’s performance measurement activities include the implementation of the Government Performance and Results Act (GPRA) of 1993. GPRA seeks to improve the management of Federal programs, as well as their effectiveness and efficiency. GPRA is intended to shift the focus of Federal decisionmaking, management, and accountability from activities and processes to results and outcomes achieved by agency programs. GPRA established a framework through which Federal agencies set goals, measure performance, and report on the extent to which those goals were met. Agencies prepare 5-year strategic plans, annual performance plans (APP), and annual performance reports. The APP provides a direct link between strategic goals and agency performance. The APP identifies (1) the annual performance goals an agency will use to gauge progress toward accomplishing its strategic goals and (2) performance indicators for use in assessing progress.

SSA submitted its first strategic plan, Keeping the Promise, in Fiscal Year (FY) 1997; its second, Mastering the Challenge, in FY 2000; and its third in FY 2003. In February 1998, SSA issued its first APP, which defined performance indicators and goals for FY 1999, and it has released APPs every year since. SSA developed and reported on performance indicators as early as FY 1991—before GPRA was passed—and began reporting on its performance goals as part of its annual accountability report in FY 1995.

The President’s Management Agenda calls for the integration of performance review and budget formulation. As part of this integration process, agencies are being asked to identify high quality outcome indicators that will be used to accurately monitor the programs’ performance. Further, Office of Management and Budget (OMB) guidance states that performance indicators should be measurable objectives against which actual achievement in program activity can be quantitatively compared. GPRA, the President’s Management Agenda, and OMB guidance highlight the importance placed on performance measurement activities in the effective management of public programs. Similarly, we have highlighted the importance of reliably measuring program performance within our description of SSA’s major management challenges.

SCOPE AND METHODOLOGY

The OIG has played a key role in ensuring that SSA’s data systems for measuring performance are reliable. We have used a four-point approach to review SSA’s performance indicators: (1) assess SSA’s system capacity to produce performance data; (2) determine whether reported performance data are reliable; (3) ensure SSA has the appropriate indicators to report on its performance; and (4) ensure the performance indicators fully capture the program segments they are purported to measure.

We have issued 34 reports through FY 2002 covering 53 of SSA’s performance indicators to determine the reliability of the data and indicators SSA uses to measure its performance. (An additional 8 reports covering 18 indicators have or will be issued in FY 2003. We did not include the FY 2003 reports since SSA has not had adequate time to implement the recommendations contained in those reports.) Each report’s findings and recommendations focused on the unique data sources related to specific performance indicators. While our prior reviews were a valuable part of SSA’s performance measurement system and identified weaknesses in data sources and indicators, our focus on individual indicators did not present an overall view of SSA’s performance management.

For this audit, we reviewed and summarized the findings included in the 34 reports on SSA's performance data and indicators we issued between FYs 1998 and 2002 to identify key trends or patterns that continue to be significant issues for SSA. (See Appendix A for a list of the 34 reports.) We confirmed which performance data and indicators addressed by our past reports were still relevant to the Agency at the time of this review. We excluded from our analysis performance data and indicators the Agency no longer uses internally or externally. While we compared the performance indicators discussed in this report to SSA’s FY 2004 APP, this comparison by itself was not sufficient to demonstrate whether all of the indicators we discussed were still in use by SSA internally.

While some of the indicators included in our review were no longer in the FY 2004 APP, the Agency may still use them for internal measurements. The Agency uses additional performance indicators beyond those in its APP in internal reporting systems to manage its programs. For example, the Workload Trend Report contains indicators of SSA’s critical workloads that are not in SSA’s FY 2004 APP. The reliability of internal performance indicators and data used by management is just as critical as it is for those measures reported externally.

Additionally, while the Agency has revised its external performance indicators over time, new indicators may still rely on the same data sets and systems used by the previous external indicators. If this is the case, SSA would still need to address previously identified weaknesses within the performance data and systems to ensure the revised indicators are supported by reliable performance data.

We analyzed the recommendations made to SSA in these reports, determined whether SSA agreed with them, and documented what SSA reported as the status of any corrective actions taken or planned to address our recommendations. We did not independently audit SSA’s assertions of actions taken. In addition, we determined whether there were any significant issues raised in our prior reports that remained unresolved and the potential effects posed by such issues.

The entity audited was the Office of Strategic Management within the Office of Chief Strategic Officer. We conducted our audit at our New York office from September 2002 through February 2003. We performed our audit in accordance with generally accepted government auditing standards.

Results of Review

We have offered many recommendations in our reviews to help improve SSA’s performance measurement processes. SSA has taken action to address weaknesses identified through our work. Over the last few years, SSA has disclosed data limitations, changed performance indicators, and improved data collection systems in response to our recommendations. However, SSA still needs to implement some outstanding recommendations that will help improve its performance measurement processes. SSA especially needs to address recommendations to improve the data used to support 13 indicators we could not review or found to be unreliable.

PERFORMANCE DATA RELIABILITY

The success of SSA’s performance measurement and management activities depends on the quality of the data used to measure and report on program performance. We have reported that SSA has exhibited a strong commitment to measuring its performance and has systems in place to collect data and report on all its performance indicators.

Between FYs 1998 and 2002, we audited 53 of SSA’s performance indicators and found 37 of the indicators (70 percent) were supported by reliable data, and 5 indicators (9 percent) were based on data that were unreliable. We found the data to be unreliable because of their incompleteness or weaknesses in the controls used to collect, analyze or report the data. We were unable to issue an opinion on 11 indicators (21 percent) because of limitations on, or the unavailability of, data. The issues that led us to our conclusions are explained in greater detail in the following sections of this report.

SSA has made progress toward ensuring the reliability of its performance data. Specifically, it reports that corrective action has been taken to ensure the reliability of the data used for 3 of the 16 indicators that we could not review or for which we found the data to be unreliable. While we have not determined the reliability of the data after these recommendations were implemented, the actions taken should aid SSA in producing reliable data. However, the Agency still needs to take corrective actions to address weaknesses in the data used to support the following 13 indicators.

• Percent of original and replacement Social Security number (SSN) cards issued within 5 days of receiving all necessary documentation
• Number of hearings pending
• Hearing processing time (days)
• Percent of hearing decisions made and notices sent within 120 days of filing
• Percent of 800-number calls handled accurately: payment accuracy
• Percent of 800-number calls handled accurately: service accuracy
• Percent of callers who successfully access the 800-number within 5 minutes of their call
• Percent of callers who get through to the 800-number on their first attempt
• 800-number calls handled
• Percent of public with an appointment waiting 10 minutes or less
• Percent of public without an appointment waiting 30 minutes or less
• Number of hearing cases processed per work year
• Number of Supplemental Security Income (SSI) non-disability redeterminations completed

We could not determine the reliability of the data or found the data to be unreliable for these 13 indicators because of weaknesses in the systems and processes used to collect, analyze and/or report the performance measure information.

MEASUREMENT PROCESS DOCUMENTATION

Our reviews found SSA often lacked sufficient documentation of the processes used to collect, analyze, and report performance indicator data, and the complete documentation necessary to recreate performance indicators was not available.

OMB guidance requires that documentation for transactions, management controls, and other significant events be clear and readily available for examination. A significant lack of documentation does not provide the necessary information to manage, assess or change measurement processes, nor does it provide the audit trail necessary to verify the performance measure. Agencies should establish a clear methodology for verifying the underlying data and retain the appropriate documentation to enable an audit of the methodology that supports the performance measures.

Of the 53 indicators we reviewed, 32 lacked the data to permit recreating the measure and/or did not adequately document the measurement process. While we could determine the reliability of the supporting data through alternative testing for most of the indicators, the lack of data prevented us from reaching a conclusion for 11 indicators. We made 28 recommendations to more fully document SSA’s processes to measure its performance. SSA has reported that it implemented 19 of the recommendations. In Table 1, we list the nine recommendations SSA has not implemented.

Table 1
Outstanding Recommendations Related to Documenting
SSA’s Performance Measures
Indicator Recommendation
Percent of disability claims decided within 6 months after onset or within 60 days after the effective filing date, whichever is later Provide an adequate audit trail to document the processes involved in the generation and accumulation of the performance measure.
• Percent of 800-number calls handled accurately: payment accuracy
• Percent of 800-number calls handled accurately: service accuracy Revise procedures to require maintaining detailed support for the facts of all 800-number calls monitored by Regional Office of Quality Assurance.
SSN requests processed Identify and document the processes surrounding the generation and accumulation of performance measure values and establish policies and procedures surrounding the retention of performance measure documentation.
Representative payee actions Identify and document the processes surrounding the generation and accumulation of performance measure values and establish policies and procedures surrounding the retention of performance measure documentation.
Percent of individuals issued Social Security Statements as required by law
Identify and document the processes surrounding the generation and accumulation of performance measure values and establish policies and procedures surrounding the retention of performance measure documentation.
• Percent of initial SSI aged claims processed within 14 days of filing
• SSI aged claims processed Identify and document the processes surrounding the generation and accumulation of performance measure values and establish policies and procedures surrounding the retention of performance measure documentation.
• Percent of Old-Age and Survivors Insurance claims processed by the time the first regular payment is due, or within 14 days from effective filing date, if later
• Old-Age and Survivors Insurance claims processed Identify and document the processes surrounding the generation and accumulation of performance measure values and establish policies and procedures surrounding the retention of performance measure documentation.
Number of SSI non-disability redeterminations completed • Formally document the processes used to collect, analyze, and report the redetermination workload performance measure.
• Maintain an audit trail of the data from which the redetermination workload data are produced.

PERFORMANCE MEASUREMENT PROCESS COMPLETENESS

OMB guidance states that performance goals and indicators should be centered on a program’s core purpose and represent key program characteristics and attributes. While the overall reliability of the indicators was not adversely affected, we reported that SSA omitted certain key components (for example, a population, process, or time period) from its calculations for some performance indicators and, as a result, did not fully measure performance in these instances. Specifically, we found key components omitted in the measurement for 21 indicators. In Table 2, we list four recommendation related to the completeness of the performance measures that SSA has not implemented.

Table 2
Outstanding Recommendations Related to the Completeness
of Performance Measures
Indicator Recommendation
Percent of Disability Insurance (DI) claims decided within 6 months after onset or within 60 days after the effective filing date, whichever is later Either include international DI claims in the measure or fully disclose in the SSA Annual Performance and Accountability Report the number of international DI claims excluded.
Percent of major statistical products that are timely SSA should expand the performance measure to include additional types of reports and include publications produced by divisions other than the Office of Research, Evaluation, and Statistics.
• Percent of employers rating SSA’s overall service as “excellent,” “very good,” or “good”
• Percent of employers rating SSA’s overall service as “excellent” When calculating the percent of respondents providing a rating of “good,” “very good,” or “excellent,” the denominator should include all respondents, regardless of their ratings.7
Hearings cases processed per work year SSA should update the travel formula to ensure it better reflects a measure of the current travel of employees and includes all employees who travel.

INTERNAL CONTROLS IN PERFORMANCE MEASUREMENT SYSTEMS

Our reviews found inadequate controls in the systems used to collect, analyze or report on some indicators. While these inadequacies did not always result in a determination that the affected data were unreliable, we concluded that the reliability could be negatively impacted in the future. Specifically, we reported that systems supporting 20 of the 53 indicators were found to have inadequate controls over the collection, analysis and/or reporting of data. In Table 3, we list the seven recommendations related to internal controls SSA has not implemented.

Table 3
Outstanding Recommendations Related to
Internal Controls
Indicator Recommendation
Percent of original and replacement SSN cards issued within 5 days of receiving all necessary documentation Revise the Modernized Enumeration System interview and batch mode process to accurately capture the time of certification.
• Number of hearings pending
• Hearing processing time (days)
• Percent of hearing decisions made and notices sent within 120 days of filing
• Perform a complete review of the process from the initial taking of the Form 501 in the field offices until input into the Hearing Office Tracking System to ensure that data within the System are completely and accurately captured in a timely manner.
• Ensure that data are automatically transferred from the Modernized Claims System and the Supplemental Security Record to the Hearing Office Tracking System to eliminate the need for the manual input of hearing request dates.
• Develop and present to all staff responsible for Hearing Office Tracking System data management a standardized training course to ensure consistent and accurate entry of data.
Percent of DI claims decided within 6 months after onset or within 60 days after the effective filing date, whichever is later Establish controls in the Modernized Claims System, SSA Claims Control System, Manual Adjustment Credit and Award Processing/Manual Adjustment Credit and Award Data Entry and Management Information Initial Claims Records systems over date fields to limit data entry errors.
• Percent of callers who successfully access the 800 number within 5 minutes of their first call
• Percent of callers who get through to the 800-number on their first attempt
• 800-number calls handled Ensure that controls exist to prevent unauthorized access to the 800-number performance measure data files.
• Percent of public with an appointment waiting 10 minutes or less
• Percent of public without an appointment waiting 30 minutes or less Require that the customer’s SSN and name be included with the data submissions and records to provide an opportunity to test data reliability and follow-up on questionable data.

Conclusions and Recommendations
The data for most of the 53 indicators reviewed were reliable, and SSA reports it has taken actions to address many of the weaknesses identified during our previous audits. Over the last few years, SSA has disclosed data limitations, changed performance indicators, and improved data collection systems in response to our recommendations. However, SSA still needs to implement some outstanding recommendations to improve the Agency’s performance measurement processes. SSA especially needs to address recommendations to improve the data used to support 13 indicators we could not review or for which we found the data to be unreliable.

SSA needs to continue its efforts toward documenting all performance measurement processes, ensuring its performance data are complete, and implementing adequate internal controls over its data systems and processes. Specifically, SSA should:

1. Implement all outstanding recommendations needed to produce reliable performance data and improve SSA’s performance measurement processes.

2. Ensure process documentation and internal controls over the collection, processing, and reporting of data are complete and adequate for any new indicators established in the future.

AGENCY COMMENTS

SSA partially agreed with Recommendation 1. According to its records, the Agency considered 7 of the 20 recommendations to be open and the remaining 13 recommendations listed in our draft report to be closed. SSA provided an updated status for all 20 recommendations as part of its comments.

The Agency maintained it is already conducting the activities in Recommendation 2 and will continue to implement this process for any new indicators established in the future. (See Appendix B for the full text of SSA’s comments.)

SSA stated it is committed to documenting relevant performance measurement processes, ensuring its performance data are complete, and implementing adequate internal controls over its data systems and processes. The Agency was pleased that our report concluded the data for most of the 53 indicators reviewed were reliable. Further, SSA noted that 10 of the 53 performance indicators addressed in our report continue to be external GPRA measures. However, the Agency stated that internal measures—which were once included in its performance plans—should not be the subject of follow-up audits and reviews since these measures are used exclusively by management.

OIG RESPONSE

We appreciate SSA’s comments on our report. We believe the Agency relies on both its external (GPRA) and internal performance indicators to manage its programs and meet its mission. As a result, SSA needs to document relevant performance measurement processes, ensure its performance data are complete, and implement adequate internal controls over its data systems and processes for all of its performance indicators to ensure management’s decisions are made based on reliable information.

We appreciate SSA’s updates on the current status of outstanding recommendations and its efforts to improve its performance measurement processes. These updates will allow us to close 7 of the 20 open recommendations. We disagree with the Agency’s closing of the five recommendations related to the documentation and retention of the processes and data used to measure SSN card requests, representative payee actions, issuance of Social Security Statements, and SSI aged and Old-Age and Survivors Insurance claims processing. SSA’s responses to these recommendations were broad-based policy statements and did not specifically state the steps taken to document and retain the performance measure processes and data. We also disagree with the Agency’s closing of the recommendation to require that the customer’s SSN and name be included with the data submissions used in its field office waiting time studies. We believe this recommendation should remain open until the Agency has implemented the Visitor Intake Process process that will provide information allowing tracking and verification of individual customer and office data.

Concerning Recommendation 2, our audit work in this area has documented multiple instances in which SSA did not have adequate documentation of, and/or internal controls over the collection, processing, and reporting of its performance processes and data. We appreciate the Agency’s efforts to ensure such documentation and controls are in place for new indicators established in the future.

Appendices

Appendix A
Office of the Inspector General Reports on the Reliability of the Social Security Administration’s Performance Measure Data (Fiscal Years 1998 Through 2002)

Report Title and Common Identification Number Date Issued
Performance Measure Review: Timely Issuance of Social Security Number Cards (A-02-97-93003) April 1998
Performance Measure Review: Evaluation of the 13th Annual Social Security Customer Satisfaction Survey Data (A-02-97-01004) January 1999
Performance Measure Review: Periodic Full Medical Continuing Disability Review Data Collection (A-01-98-94003) September 1999
Performance Measure Review: Reliability of the Data Used to Measure Dollar Accuracy of Old-Age and Survivors Insurance Payment Outlays (A-02-98-01001) December 1999
Performance Measure Review: Reliability of the Data Used to Measure Welfare Reform Childhood Disability Reviews (A-01-99-91003) March 2000
Performance Measure Review: Reliability of the Data Used to Measure Social Security Number Request Processing (A-02-99-01009) March 2000
Performance Measure Review: Reliability of the Data Used to Measure Representative Payee Actions (A-02-99-01010) March 2000
Performance Measure Review: Reliability of the Data Used to Measure Personal Earnings and Benefit Estimate Statement Processing (A 02 99 01011) March 2000
Performance Measure Review: Reliability of the Data Used to Measure the Timeliness of Supplemental Security Income Aged Claims Processing (A-02-99-11005) March 2000
Performance Measure Review: Reliability of the Data Used to Measure the Posting of Earnings Items (A-02-99-01008) March 2000
Performance Measure Review: Reliability of the Data Used to Measure the Timeliness of Old-Age and Survivors Insurance Claims Processing (A-02-99-11006) March 2000
Performance Measure Review: Reliability of the Data Used to Measure Continuing Disability Reviews (A-01-99-91002) June 2000
Performance Measure Review: Reliability of the Data Used to Measure the Social Security Administration’s Debt Collection (A-15-99-51006) July 2000
Performance Measure Review: Reliability of the Data Used to Measure Social Security Administration Employee Satisfaction with the Level of Security at Their Facility (A-13-00-10025) September 2000
Performance Measure Review: Reliability of the Data Used to Measure the Timeliness of Processing Supplemental Security Income Disability Claims (A-02-99-11002) December 2000
Management Advisory Report: Performance Measure for the Percent of the Continuing Disability Review Multi-Year Plan Completed (A 01 00 10011) February 2001
Performance Measure Review: Reliability of the Data Used to Measure the Hearing Process (A-02-98-91003) May 2001
Performance Measure Review: Reliability of the Data Used to Measure the Accuracy of Earnings Posted (A-03-00-10004) May 2001
Performance Measure Review: Reliability of the Data Used to Measure Disability Determination Services Decisional Accuracy (A-07-99-21007) June 2001
Performance Measure Review: Reliability of the Data Used to Measure the Percentage of Social Security Numbers Issued Accurately (A 08 99 41003) June 2001
Management Advisory Report: Performance Measure Survey of the Percent of 800-Number Calls Handled Accurately (A-08-01-11024) August 2001
Performance Measure Review: Reliability of the Data Used to Report 800-Number Performance (A-02-00-10019) September 2001
Performance Measure Review: Reliability of the Data Used to Measure Field Office Access (A-04-99-03008) September 2001
Performance Measure Review: Reliability of the Data Used to Measure Disability Claims Processing (A-02-00-10017) November 2001
Performance Measure Review: Reliability of the Data Used to Measure the Timely Processing of Disability Insurance Claims (A-02-99-11001) October 2001
Performance Measure Review: Reliability of the Data Used to Measure Public Knowledge of the Social Security Administration (A-02-01-11015) February 2002
Performance Measure Review: The Social Security Administration’s Transition Planning (A-02-01-11014) March 2002
Performance Measure Review: Reliability of the Data Used to Measure the Quality of the Social Security Administration’s Research (A 02 01 11011) March 2002
Performance Measure Review: Reliability of the Data Used to Measure Employer Satisfaction (A-02-01-11012) April 2002
Performance Measure Review: Reliability of the Data Used to Measure the Office of Hearings and Appeals Decisional Accuracy (A 12 00 10057) April 2002
Performance Measure Review: Increase in the Number of Disabled Beneficiaries Who Are Working (A-01-01-11022) May 2002
Performance Measure Review: Reliability of the Data Used to Measure Electronic Service Delivery (A-14-01-11032) July 2002
Performance Measure Review: Assessing the Methodology Used to Determine the Number of Hearings Cases Processed per Work Year (A 06-01-11037) August 2002
Performance Measure Review: Reliability of the Data Used to Measure Supplemental Security Income Non-Disability Redeterminations Completed (A-02-99-11003) August 2002

Appendix B
Agency Comments

MEMORANDUM

Date: July 30, 2003

To: James G. Huse, Jr.
Inspector General

From: Larry W. Dye
Chief of Staff

Subject: Office of the Inspector General (OIG) Draft Report "Summary of the OIG's Reviews of the Social Security Administration's (SSA) Performance Data" (A-02-03-13033)—INFORMATION

We appreciate OIG’s efforts in conducting this review. Our comments on the draft report content and recommendations are attached.

Staff questions may be referred to Laura Bell on extension 52636.

SSA Response

COMMENTS ON THE OFFICE OF THE INSPECTOR GENERAL (OIG) DRAFT REPORT “SUMMARY OF THE OIG'S REVIEWS OF THE SOCIAL SECURITY ADMINISTRATION'S (SSA) PERFORMANCE DATA" (A-02-03-13033)

Thank you for the opportunity to review and comment on the report content and recommendations. SSA is committed in our efforts toward documenting relevant performance measurement processes, ensuring our performance data are complete, and implementing adequate internal controls over our data systems and processes. We welcome external input, suggestions and the contributions the Inspector General has made, and continues to make, in assisting us in developing a comprehensive performance measurement process. We are pleased that the report concluded that the data for most of the 53 indicators reviewed were reliable, and that it notes that we have completed or are in the process of taking actions to address many of the weaknesses identified during previous audits.

This report summarizes performance measure audits that date as far back as 1998. While the report notes that the measures were all at one time part of an Agency Annual Performance Plan as of Fiscal Year (FY) 2003 only 10 continue to be external Government Performance and Results Act measures. Due to the numerous changes that occur with our performance plans from year to year, we do not believe that following up on prior recommendations that are no longer relevant is an efficient use of resources. Furthermore, we do not believe that internal measures, that may have once been included in the performance plans, should be the subject of follow-up audits and reviews, as they are used exclusively by management, and we need to have the flexibility to use and change them at will to meet our program mission. In addition, we have already provided information on the recommendations either in our response to the original audit or through the quarterly updating and tracking of open recommendations. Finally, we would like to have a more collaborative role in the planning of any future audits of this nature to ensure Agency resources are expended on timely, relevant performance management issues.

Our responses to the specific recommendations are provided below and the charts provide the status of the specific recommendations contained in the draft report.

Recommendation 1

Implement all outstanding recommendations needed to produce reliable performance data and improve SSA’s performance measurement processes.

SSA Response

We partially agree. According to our records, there are only 7 outstanding recommendations as opposed to 20 that are listed in the draft report. Those 7 recommendations are listed below along with each recommendation's current status and implementation plan. The attached chart lists the other 13 recommendations that we consider closed and provides the status of the recommendation that is contained in our Recommendation Tracking Status System.

Outstanding Recommendations Related to Documenting
SSA’s Performance Measures
Indicator Recommendation Status
Percent of disability claims decided within 6 months after onset or within 60 days after the effective filing date, whichever is later* Provide an adequate audit trail to document the processes involved in the generation and accumulation of the performance measure. Recommendation Tracking Status (RTS) March 2003 – The Office of Systems is addressing this issue as it transitions the Management Information Initial Claims Record (MIICR) functionality to the Title II Workload Management Information System (WMI). A General Project Scope Agreement was agreed upon. The detailed functional requirements for the first release have been completed and software changes for release 1 are currently underway. The planned implementation date for the first release is June 30, 2003. However, processing times will not be addressed until Release 3 of this effort, and the final target date has yet to be determined.
Number of SSI non-disability redeterminations completed Formally document the processes used to collect, analyze, and report the redetermination workload performance measure.
RTS March 2003 – The Post-Eligibility Operational Data Store (PEODS) system for redeterminations is a new modernized system that will address many of OIG's concerns. It was provided to all field offices in June 2002 to control their redetermination workload. The PEODS users worked to identify, document and validate requirements for redetermination data, all calculations, report content and format. The Central Office Redetermination Control (CORC) system was terminated as of January 2003 since a review of the performance measure information was verified and agreed upon.
Maintain an audit trail of the data from which the redetermination workload data are produced. RTS March 2003 – As part of the nonfunctional requirements for the SSI Non-Disability Redeterminations workloads, the PEODS has the following data retention requirements:
--There will be an ODS in the production environment that will contain pending data and one month of completion, clearance and deletion data. The data in this ODS is being limited for performance reasons. A copy of this data will also reside on the Management Information Software Facility (MISF) ODS.
--The MISF ODS will also contain all data for the last 3 years.
--The Data Warehouse will house the current fiscal year and 2 years prior. The weekly summaries will be retained for a rolling 26 weeks. Ad hoc queries will be available in the data warehouse.
Outstanding Recommendations Related to the Completeness of Performance Measures
Indicator Recommendation Status
Hearings cases processed per work year SSA should update the travel formula to ensure it better reflects a measure of the current travel of employees and includes all employees who travel. RTS March 2003 -- The Office of Hearings and Appeals' evaluation of the travel formula and necessary changes are expected to be completed by the end of fiscal year 2003.

Outstanding Recommendations Related to Internal Controls
Indicator Recommendation Status
• Number of hearings pending
• Hearing processing time (days)
• Percent of hearing decisions made and notices sent within 120 days of filing
Perform a complete review of the process from the initial taking of the Form 501 in the field offices until input into the Hearing Office Tracking System to ensure that data within the system are completely and accurately captured in a timely manner.

RTS March 2003 – On November 5, 2002, the Office of Disability and Income Security Programs (ODISP) issued a memorandum to the Office of Operations requesting volunteers to form a workgroup to address the issues outlined in this OIG audit. The workgroup has been formed to address these issues and is comprised of representatives from the Office of Hearings and Appeals' (OHA) Office of Management, Office of the Chief Administrative Law Judge (OCALJ), Office of Policy, Planning and Evaluation and Buffalo, New York Hearing Office. The workgroup held it first conference call on January 24, 2003, wherein issues and strategies were discussed. Members of the workgroup submitted suggestions and comments to the OCALJ representative for inclusion in a summary report. Upon completion of the summary report, a joint memorandum will be drafted to the Office of Operations and OHA. A draft of the memorandum is targeted for completion during the third quarter of FY 2003.

Ensure that data are automatically transferred from the Modernized Claims System and the Supplemental Security Record to the Hearing Office Tracking System to eliminate the need for the manual input of hearing request dates. DISP Status – July 2003 – The Accelerated Electronic Disability (AeDib) project includes a Case Processing and Management System (CPMS) that will interface with all of SSA's systems. Data will be propagated from those other SSA systems, eliminating redundant keying of data.

Develop and present to all staff responsible for Hearing Office Tracking System data management a standardized training course to ensure consistent and accurate entry of data. DISP Status – July 2003 – Upon implementation of CPMS, the HOTS will only be used to control Medicare cases. Training needs relative to the implementation of CPMS are being assessed as part of the implementation.

Recommendation 2

Ensure process documentation and internal controls over the collection, processing, and reporting of data are complete and adequate for any new indicators established in the future.

SSA Response

We already document and establish internal controls over the collection, processing, and reporting of data to ensure that our indicators are complete and adequate. We will continue to implement this process for any new indicators that are established in the future.

Closed Recommendations Related to Documenting SSA’s Performance Measures
Indicator Recommendation Status
• Percent of 800-number calls handled accurately: payment accuracy
• Percent of 800-number calls handled accurately: service accuracy* Revise procedures to require maintaining detailed support for the facts of all 800-number calls monitored by Regional Office of Quality Assurance. RTS September 2001 – Closed – SSA disagreed, but suggested OIG co-monitor 800-number calls when the Agency assesses the reliability of the data used to compute the performance measure. We had previously considered documenting the details of monitored calls, but determined it would not be cost-effective to do so and might still produce unreliable results.
SSN requests processed Identify and document the processes surrounding the generation and accumulation of performance measure values and establish policies and procedures surrounding the retention of performance measure documentation. RTS January 2001 – Closed – We agreed in concept the Office of Strategic Management (OSM) is responsible for coordinating the Agency’s GPRA activities. In addition, we continue to work to improve the development and retention of the kind of documentation needed for external audits of our performance measures.
Representative payee actions* Identify and document the processes surrounding the generation and accumulation of performance measure values and establish policies and procedures surrounding the retention of performance measure documentation. RTS January 2001 – Closed – We agreed in concept the Office of Strategic Management (OSM) is responsible for coordinating the Agency’s GPRA activities. In addition, we continue to work to improve the development and retention of the kind of documentation needed for external audits of our performance measures.
Percent of individuals issued Social Security Statements as required by law*
Identify and document the processes surrounding the generation and accumulation of performance measure values and establish policies and procedures surrounding the retention of performance measure documentation. RTS January 2001 – Closed – We agreed in concept the Office of Strategic Management (OSM) is responsible for coordinating the Agency’s GPRA activities. In addition, we continue to work to improve the development and retention of the kind of documentation needed for external audits of our performance measures.
• Percent of initial SSI aged claims processed within 14 days of filing
• SSI aged claims processed Identify and document the processes surrounding the generation and accumulation of performance measure values and establish policies and procedures surrounding the retention of performance measure documentation. RTS January 2001 – Closed – We agreed in concept the Office of Strategic Management (OSM) is responsible for coordinating the Agency’s GPRA activities. In addition, we continue to work to improve the development and retention of the kind of documentation needed for external audits of our performance measures.
• Percent of Old-Age and Survivors Insurance claims processed by the time the first regular payment is due, or within 14 days from effective filing date, if later*
• Old-Age and Survivors Insurance claims processed Identify and document the processes surrounding the generation and accumulation of performance measure values and establish policies and procedures surrounding the retention of performance measure documentation. RTS January 2001 – Closed – We agreed in concept the Office of Strategic Management (OSM) is responsible for coordinating the Agency’s GPRA activities. In addition, we continue to work to improve the development and retention of the kind of documentation needed for external audits of our performance measures.
*This is no longer an external GPRA measure.
Closed Recommendations Related to the Completeness of Performance Measures
Indicator Recommendation Status
Percent of Disability Insurance (DI) claims decided within 6 months after onset or within 60 days after the effective filing date, whichever is later* Either include international DI claims in the measure or fully disclose in the SSA Annual Performance and Accountability Report the number of international DI claims excluded. RTS March 2003 – Closed – SSA does not believe that international DI claims should be included in the measure. The disability timeliness goal is, and has always been, a measure of field office disability claims processing for U.S. States and territories. The process of filing a disability claim is very different for persons who file in foreign countries, and cannot be compared to field office/Disability Determinations Service procedures. Social Security offices do not exist in foreign countries. Applicants must have access to American embassies to conduct business, and the embassy staffs are not SSA employees. While SSA does not agree that international DI claims should be included in the measure, it does agree that SSA's Annual Performance and Accountability Report should disclose the number of international DI claims that are excluded.

The performance measure, "Reliability of Data Used to Measure Timely Processing of Disability Insurance Claims," as referenced in this audit has not been an active measure since 1999. SSA did take action to include international claims in our current measure of the timeliness of disability claims resolving the issue raised in this audit.

Percent of major statistical products that are timely* SSA should expand the performance measure to include additional types of reports and include publications produced by divisions other than the Office of Research, Evaluation, and Statistics. RTS June 2002 – Closed – We disagreed. In the report, OIG acknowledges the Agency made a significant effort in identifying publications for this measure. We have determined that no other SSA statistical publications are produced regularly, have a statutory deadline, or are considered major in terms of usage.
• Percent of employers rating SSA’s overall service as “excellent,” “very good,” or “good”*
• Percent of employers rating SSA’s overall service as “excellent”* When calculating the percent of respondents providing a rating of “good,” “very good,” or “excellent,” the denominator should include all respondents, regardless of their ratings.7 RTS June 2002 – Closed – We disagreed. Basing ratings on substantive responses is acceptable practice, as long as accompanying discussion indicates that the percentages reflect the opinions of those who provided a rating, as was done in the report prepared by the Office of Quality Assurance and Performance Assessment. It should also be noted that the rating questions did not include a choice of "no opinion" as indicated in OIG's report; the choice was "not applicable/service not used," which served as a screening device that fit into the format of the questionnaire.

Closed Recommendations Related to Internal Controls
Indicator Recommendation Status
Percent of original and replacement SSN cards issued within 5 days of receiving all necessary documentation* Revise the Modernized Enumeration System interview and batch mode process to accurately capture the time of certification. RTS April 1998 – Closed – SSA does not agree it is necessary to expand its current processes in order to capture partial day processing time. Staff is examining the need for improvement in the input of certifications of Social Security number applications.
Percent of DI claims decided within 6 months after onset or within 60 days after the effective filing date, whichever is later* Establish controls in the Modernized Claims System, SSA Claims Control System, Manual Adjustment Credit and Award Processing/Manual Adjustment Credit and Award Data Entry and Management Information Initial Claims Records systems over date fields to limit data entry errors. RTS August 2002 – Closed – SSA evaluated the SSN sources used by OIG and determined that the systems applications cited do in fact pass incorrect dates and, when corrections are entered, the updated data is not passed on MIICR. The Office of Systems (OS) components have met and been unable to duplicate the events. OIG's findings were taken from 3 months of MIICR data consisting of 284,974 cases covering the period November and December 1999 and February 2000, with only 9 records reflecting incorrect data. Therefore, it is believed these occurrences were either anomalies, or systems changes subsequent to the review period in the applications noted corrected any erroneous data passing to MIICR.
• Percent of callers who successfully access the 800 number within 5 minutes of their first call
• Percent of callers who get through to the 800-number on their first attempt
• 800-number calls handled Ensure that controls exist to prevent unauthorized access to the 800-number performance measure data files. RTS March 2003 – Closed – OTSO redesigned the front-end to allow a backup to run the jobs. This required moving all the files. During the move, PC file access was made more secure. On the mainframe-side, actions have been taken to change the file names to meet Top Secret and SSA standards. All dataset names have been changed and are under Top Secret security.
• Percent of public with an appointment waiting 10 minutes or less*
• Percent of public without an appointment waiting 30 minutes or less* Require that the customer’s SSN and name be included with the data submissions and records to provide an opportunity to test data reliability and follow-up on questionable data. RTS October 2001 – Closed – The current waiting time study is not meant to be customer specific and generates information needed to assess accessibility only at the national level. Capturing the SSN for such a study would be too resource intensive and would not be justified considering that the Agency is developing the VIP process. When the VIP process is implemented, it will provide information allowing an auditor to track and verify individual customer information and office level data.

Appendix C
OIG Contacts and Staff Acknowledgments
OIG Contacts
Rona Rustigian, Director, (617) 565-1819
Timothy Nee, Deputy Director, (212) 264-5295

Staff Acknowledgments
In addition to those named above:
Jeffrey Shapiro, Program Analyst
Abraham Pierre, Auditor

For additional copies of this report, please visit our web site at www.ssa.gov/oig or contact the Office of the Inspector General’s Public Affairs Specialist at (410) 966-1375. Refer to Common Identification Number A-02-03-13033.

Overview of the Office of the Inspector General

Office of Audit

The Office of Audit (OA) conducts comprehensive financial and performance audits of the Social Security Administration’s (SSA) programs and makes recommendations to ensure that program objectives are achieved effectively and efficiently. Financial audits, required by the Chief Financial Officers' Act of 1990, assess whether SSA’s financial statements fairly present the Agency’s financial position, results of operations and cash flow. Performance audits review the economy, efficiency and effectiveness of SSA’s programs. OA also conducts short-term management and program evaluations focused on issues of concern to SSA, Congress and the general public. Evaluations often focus on identifying and recommending ways to prevent and minimize program fraud and inefficiency, rather than detecting problems after they occur.

Office of Executive Operations

The Office of Executive Operations (OEO) supports the Office of the Inspector General (OIG) by providing information resource management; systems security; and the coordination of budget, procurement, telecommunications, facilities and equipment, and human resources. In addition, this office is the focal point for the OIG’s strategic planning function and the development and implementation of performance measures required by the Government Performance and Results Act. OEO is also responsible for performing internal reviews to ensure that OIG offices nationwide hold themselves to the same rigorous standards that we expect from SSA, as well as conducting investigations of OIG employees, when necessary. Finally, OEO administers OIG’s public affairs, media, and interagency activities, coordinates responses to Congressional requests for information, and also communicates OIG’s planned and current activities and their results to the Commissioner and Congress.

Office of Investigations

The Office of Investigations (OI) conducts and coordinates investigative activity related to fraud, waste, abuse, and mismanagement of SSA programs and operations. This includes wrongdoing by applicants, beneficiaries, contractors, physicians, interpreters, representative payees, third parties, and by SSA employees in the performance of their duties. OI also conducts joint investigations with other Federal, State, and local law enforcement agencies.

Counsel to the Inspector General

The Counsel to the Inspector General provides legal advice and counsel to the Inspector General on various matters, including: 1) statutes, regulations, legislation, and policy directives governing the administration of SSA’s programs; 2) investigative procedures and techniques; and 3) legal implications and conclusions to be drawn from audit and investigative material produced by the OIG. The Counsel’s office also administers the civil monetary penalty program.