Office of the Inspector General
Larry G. Massanari
Acting Commissioner of Social Security

Inspector General

Performance Measure Review: Reliability of the Data Used to Measure the Percentage of Social Security Numbers Issued Accurately (A-08-99-41003)

The attached final report presents the results of our audit. Our objective was to determine the reliability of the data and the accuracy of the estimate the Social Security Administration (SSA) used in Fiscal Year 1998 to measure the percent of Social Security numbers issued accurately. The percent of Social Security numbers issued accurately was one of the performance indicators SSA developed to meet the requirements of the Government Performance and Results Act of 1993.

Please comment within 60 days from the date of this memorandum on corrective actions taken or planned on each recommendation. If you wish to discuss the final report, please call me or have your staff contact Steven L. Schaeffer, Assistant Inspector General for Audit, at (410) 965-9700.


James G. Huse, Jr.

Office of the Inspector General
Larry G. Massanari
Acting Commissioner of Social Security

Inspector General

Performance Measure Review: Reliability of the Data Used to Measure the Percentage of Social Security Numbers Issued Accurately (A-08-99-41003)

The Government Performance and Results Act (GPRA) of 1993 requires the Social Security Administration (SSA) to develop performance indicators that assess the relevant service levels and outcomes of each program activity. GPRA also calls for a description of the means employed to verify and validate the measured values used to report on program performance. SSA has stated the Office of the Inspector General (OIG) plays a vital role in evaluating the data used to measure performance. The objective of this audit was to determine the reliability of the data and the accuracy of the estimate SSA used in Fiscal Year (FY) 1998 for the following GPRA performance indicator.

The Office of Quality Assurance and Performance Assessment (OQA), under the Deputy Commissioner for Finance, Assessment and Management, conducts reviews and special studies to evaluate and assess the integrity and quality of SSA’s programs. One of OQA’s quality assurance reviews is the annual Quality of the Enumeration Process Review. In conducting this Review, OQA continuously selects a sample of Social Security numbers (SSN) issued to determine the accuracy of the enumeration process.

When OQA receives an application, a technician in OQA’s Office of Statistics and Special Area Studies verifies the SSN’s accuracy by running a query of the Alpha-Index file and identifies incomplete/incorrect information in the electronic record of the selected transaction (Numident) by comparing it to the matching application. A pictorial description of OQA’s end-of-line enumeration review can be found in Appendix C of this report.

OQA’s review classifies incorrect enumeration actions as either critical or major errors, defined as follows.

OQA reports the SSN accuracy rates annually in its Report on the Quality of the Enumeration Process to SSA’s Deputy Commissioners for Disability and Income Security Programs, Operations, and Systems. SSA also reports the accuracy rate in its Annual Performance Plan (APP) as a benefit payment performance measure.

SSA’s goal, as stated in the FY 2001 APP, is to achieve a 99.8-percent SSN accuracy rate. SSA has met or exceeded this goal since FY 1997.

The data SSA used to report the FY 1998 SSN accuracy rate appeared to be statistically reliable. However, we believe actual performance could be better reflected if SSA implemented changes in the way it reports SSN errors. Specifically, we believe SSA should disclose it did not consider all types of errors in the accuracy rate computation.

Because SSA excluded relevant data from the SSN accuracy rate measure, it did not follow GPRA and Office of Management and Budget (OMB) guidance on developing APPs. GPRA calls for the establishment of "performance indicators (measures) to be used in measuring or assessing the relevant outputs, service levels, and outcomes of each program activity." Additionally, OMB Circular A-11, Preparation and Submission of Strategic Plans and Annual Performance Plans, part 2, states that APPs "should neither capture the complete array of measures likely to be used in managing programs, nor skimp on what is measured, resulting in a narrowly-drawn or fragmented picture of performance." We believe SSA’s method of measuring and reporting SSN accuracy results in a narrowly drawn picture of performance.

Based on our review of a sample of 105 of 2,301 OQA case files, verification of the electronic SSN error file, and documentation of OQA’s end-of-line review process, the FY 1998 SSN accuracy rate appears reliable. OQA’s error determination was correct for 103 (98 percent) of the 105 cases we reviewed, including all of the critical errors. Two (2 percent) of the cases contained major errors that SSA did not identify. Also, OQA accurately entered the error data for 364 of the 368 records we verified. We did not identify any significant internal control weaknesses in the end-of-line review process, and we concluded OQA’s sampling methodology and projections were statistically sound. However, we believe SSA could modify its sampling methodology to better reflect the population of SSNs issued, as discussed in the Other Matters section of this Report. Finally, we determined two computer programs key to OQA’s review—one computes the SSN accuracy rate, and the other selects the Numidents for OQA’s sample—produced valid results.

The SSN accuracy rate reported in the APP did not reflect all types of SSN errors. Specifically, the Agency did not consider major errors and SSNs issued as a result of false or improper evidentiary documents when calculating the accuracy rate. We believe excluding these errors, without explicit disclosure of such exclusions, is misleading as it implies almost perfect accuracy in SSN processing. In fact, in 1998, 208 (9 percent) of the 2,301 SSNs contained major errors. Additionally, other OIG audits and investigations have noted significant problems with counterfeit documents being presented to fraudulently obtain SSNs. However, OQA did not attempt to identify these occurrences when it conducted its review.

The Agency’s APP neither disclosed that major errors were not considered when calculating the measure nor reported the rate of those errors. If the Agency had included the 208 major errors, it would not have met its 1998 accuracy rate goal of 99.8 percent because the 1998 SSN accuracy rate, based on critical and major errors combined, was 90.7 percent.

The rate at which SSA generates major errors is problematic, as evidenced by the 9-percent major error rate in Calendar Year (CY) 1998 and the 11-percent major error rate in CY 1997. The CY 1998 error rate projected to more than 1.1 million SSNs in the universe.

OQA did not consider improperly assigned SSNs in the accuracy rate computation. Improperly assigned SSNs are numbers SSA assigned based on invalid and/or inappropriate evidentiary documents. Based on other OIG audits and investigations, we learned that detecting counterfeit evidentiary documents can be problematic for SSA. As a result, SSA sometimes assigns an SSN to individuals who should not have received one. Since OQA did not attempt to identify these types of errors during its enumeration review, we do not know whether the 1998 SSN accuracy rate would have dropped if the Agency had included the errors in the accuracy rate computation. To further complete the picture of SSN accuracy, we believe future OQA enumeration reviews should identify improperly assigned SSNs, and the Agency should factor in these errors or at least disclose that they are not included in the accuracy rate.

In examining the SSN applications included in the enumeration end-of-line review, OQA ensured the evidence descriptions and coding were accurate and complete. However, it did not determine whether the evidence submitted with SSN applications was valid. The OQA technician told us she did not validate some types of evidence because she could not physically examine the documents. Thus, she relied on the FO’s determination of the evidence’s validity. We agree that OQA’s validation efforts will be limited to certain types of evidentiary documents. However, the results of another OIG review indicate these efforts are necessary to identify the improper assignment of SSNs. In this review, we found that, from September 1, 1997 to February 28, 1998, SSA issued at least 999 original SSNs based on invalid/inappropriate evidentiary documents. The invalid documents included Immigration and Naturalization Service forms and counterfeit U.S. birth certificates.

Although the FY 1998 99.8-percent SSN accuracy rate was reliable, SSA did not present a complete picture because it did not consider major errors and improperly assigned SSNs when calculating the SSN accuracy rate. The limited information SSA reports may mislead the public to believe SSA made virtually no errors when issuing SSNs. In fact, SSA has generated at least 1 million major errors in SSN records annually since 1997. In addition, in other OIG audits and investigations, we determined that SSA sometimes issues SSNs based on invalid or improper evidence. We do not know the impact of including improperly assigned SSNs in the accuracy rate computation; however, the Agency would not have met its 1998 goal if it had included major errors.

To improve the reporting of future SSN accuracy rates, we recommend SSA include major errors and improperly assigned SSNs when calculating future SSN accuracy rates or include a statement in the APP that specifically discloses the exclusion of such errors from the SSN accuracy rate measure.

SSA agreed with our recommendation. However, the Agency noted the correct volume of SSN applications for FY 1998 was 1.7 million original applications and 10.7 million replacement card applications. Hence, the simple random sample OQA selected adequately reflects the population, and no correction to the sampling methodology is necessary, as we suggested in the draft report. The complete text of SSA’s comments is included in Appendix D.

Through discussion with an OQA manager, we determined that, during our audit, OQA provided us an incorrect figure. Specifically, OQA informed us that, in FY 1998, SSA issued 2.4 million original SSNs. However, in its response to this draft report, OQA corrected that number to only 1.7 million original applications. As a result, we revised the draft report to eliminate our concern that the SSN accuracy rate sample did not represent the population.

OQA’s sampling methodology for the 1998 Quality of the Enumeration Process Review resulted in an examination of lower-risk transactions in almost 87 percent of the cases. We believe SSA could improve its methodology by more fairly distributing the proportion of low-and high-risk applications in the sample. The Agency could achieve this improvement by increasing the number of original applications reviewed for SSN accuracy.

We are concerned about the disproportionate distribution of the sample between original and replacement cards. The Agency based its accuracy assessment primarily on applications that have a lower error risk. In 1998, the sample included 303 original applications and 1,998 replacement card applications. Thus, replacement card applications represented 87 percent (1,998 out of 2,301) of the sample. However, only 20 percent of the critical errors OQA identified were related to applications for a replacement card. We are not inferring that replacement cards are not vulnerable to fraud. However, lower risk of error for a replacement card is inherent because this type of transaction requires little manual data input. The field office representative does not key in as much data for a replacement card because the system automatically completes many of the data fields with information it captures from the existing computer record formed when an individual was originally assigned an SSN.

If SSA approves a proposal submitted by its Enumeration and Systems Team, the Agency would be able to measure the actual rate of wrong number cases reported by individual SSN cardholders. SSA has developed new codes for the different number requested field on the SS-5. These codes correspond to the reason the new SSN is being assigned. One of the six reason codes is for wrong number cases. In addition, SSA plans to add a new field to the Numident to reflect these codes. According to an official of SSA’s Enumeration and Systems Team in the Division of Eligibility and Enumeration Policy, this upgrade will be implemented in the FY 2002 release of the Modernized Enumeration System. OQA could obtain these data during future quality of enumeration process reviews to provide SSA management with another indicator of its performance in correctly issuing SSNs.

James G. Huse, Jr.

Appendices

APPENDIX A – Scope and Methodology

APPENDIX B – Acronyms

APPENDIX C – Flow Chart of 1998 Enumeration Review

APPENDIX D – Agency Comments

APPENDIX E – OIG Contacts and Staff Acknowledgment

Scope and Methodology

We reviewed the sample methodology and procedures employed in conducting the Calendar Year (CY) 1998 enumeration process review to assess its reliability as a data source for Social Security number (SSN) accuracy.

We tested and analyzed the Office of Quality Assurance and Performance Assessment’s (OQA) methodology by fully examining a sample of cases included in the CY 1998 enumeration end-of-line review. We randomly selected 100 cases free from critical error from the CY 1998 Enumeration Quality Assurance System (EQAS), which is the data base containing the results of the review of 2,301 cases. The data base is designed to measure and report on the quality of various aspects of the enumeration process. We verified the accuracy of the data in EQAS by comparing OQA’s data input sheets to a printout from EQAS.

We verified all aspects of OQA’s desk reviews by following the procedures outlined by the OQA staff. These procedures included obtaining an Alpha-Index query for each sampled applicant. We determined whether the OQA case files contained all the required forms, and all of the necessary documents were secured. We also reviewed each OQA case file included in our sample to verify the results were recorded correctly to EQAS.

From the universe of 2,301 cases, we also reviewed all 5 cases for which OQA reported a critical error. According to the enumeration review team’s supervisor, OQA selected 2,414 cases for review. However, 113 of these cases were excluded. The excluded cases included 104 that were generated from benefit claims transactions rather than SSN applications, 7 mislocated SS-5s, and 2 late returns. The following chart provides details of the 105 enumeration transactions we reviewed.

To complete our review, we obtained and reviewed relevant sections of the Program Operations Manual System. We also interviewed knowledgeable Social Security Administration staff to gain an understanding of the enumeration process, the OQA data base, the statistical methods used, and other relevant matters. In addition, for each case finding noted in this report, we analyzed the OQA case file and all pertinent documentation with OQA representatives and discussed the case findings with them. The OQA staff concurred with our conclusions on each of these findings.

We conducted our work at the OQA office in Woodlawn, Maryland, and the Tuscaloosa, Alabama, district office. The entity audited was the Deputy Commissioner for Operations. We conducted audit field work from October 1999 to October 2000 in accordance with generally accepted government auditing standards.

Acronyms

Appendix C

Flow Chart of 1998 Enumeration Review

Agency Comments

Contacts and Staff Acknowledgments

In addition to those named above:

Ellen Justice, Auditor
Kimberly Beauchamp, Writer-Editor, Policy, Planning and Technical Services Division