Office of
the Inspector General
Larry G. Massanari
Acting Commissioner of Social Security
Inspector General
Performance Measure Review: Reliability of the Data Used to Measure the Percentage of Social Security Numbers Issued Accurately (A-08-99-41003)
The attached final report presents the results of our audit. Our objective was to determine the reliability of the data and the accuracy of the estimate the Social Security Administration (SSA) used in Fiscal Year 1998 to measure the percent of Social Security numbers issued accurately. The percent of Social Security numbers issued accurately was one of the performance indicators SSA developed to meet the requirements of the Government Performance and Results Act of 1993.
Please comment
within 60 days from the date of this memorandum on corrective actions taken
or planned on each recommendation. If you wish to discuss the final report,
please call me or have your staff contact Steven L. Schaeffer, Assistant Inspector
General for Audit, at (410) 965-9700.
James
G. Huse, Jr.
OFFICE OF
THE INSPECTOR GENERAL
SOCIAL SECURITY ADMINISTRATION
PERFORMANCE MEASURE REVIEW:
RELIABILITY OF THE DATA USED TO
MEASURE THE PERCENTAGE OF
SOCIAL SECURITY NUMBERS
ISSUED ACCURATELY
June
2001
A-08-99-41003
AUDIT REPORT
Office of the Inspector General
Larry G. Massanari
Acting Commissioner of Social Security
Inspector General
Performance Measure Review: Reliability of the Data Used to Measure the Percentage of Social Security Numbers Issued Accurately (A-08-99-41003)
The Government Performance and Results Act (GPRA) of 1993 requires the Social Security Administration (SSA) to develop performance indicators that assess the relevant service levels and outcomes of each program activity. GPRA also calls for a description of the means employed to verify and validate the measured values used to report on program performance. SSA has stated the Office of the Inspector General (OIG) plays a vital role in evaluating the data used to measure performance. The objective of this audit was to determine the reliability of the data and the accuracy of the estimate SSA used in Fiscal Year (FY) 1998 for the following GPRA performance indicator.
Percent of Social Security numbers issued accurately:
Percent without error – 99.8
BACKGROUND
The Office of Quality Assurance and Performance Assessment (OQA), under the Deputy Commissioner for Finance, Assessment and Management, conducts reviews and special studies to evaluate and assess the integrity and quality of SSA’s programs. One of OQA’s quality assurance reviews is the annual Quality of the Enumeration Process Review. In conducting this Review, OQA continuously selects a sample of Social Security numbers (SSN) issued to determine the accuracy of the enumeration process.
Transactions that do not involve actual SSN applications―enumeration-at-birth (EAB) and claims―are excluded from the sample selection. OQA electronically transmits a list of the selected SSNs to the applicable field offices (FO). FO representatives then fax the associated Form SS-5 (Application for a Social Security Card) to OQA. If the Form is not available at the FO, OQA requests the Form from SSA’s records center. According to the Enumeration Review Team Supervisor, historically, OQA has obtained 90 percent of the SS-5s from the FO.
When OQA receives an application, a technician in OQA’s Office of Statistics and Special Area Studies verifies the SSN’s accuracy by running a query of the Alpha-Index file and identifies incomplete/incorrect information in the electronic record of the selected transaction (Numident) by comparing it to the matching application. A pictorial description of OQA’s end-of-line enumeration review can be found in Appendix C of this report.
OQA’s review classifies incorrect enumeration actions as either critical or major errors, defined as follows.
Critical error - SSA issued multiple SSNs to the same individual or issued someone else’s SSN to the applicant.
Major error - SSA issued the SSN accurately but made some type of error in the individual’s SSN record (Numident). Examples include omitting the applicant’s maiden name or entering the wrong birth date.
OQA reports the SSN accuracy rates annually in its Report on the Quality of the Enumeration Process to SSA’s Deputy Commissioners for Disability and Income Security Programs, Operations, and Systems. SSA also reports the accuracy rate in its Annual Performance Plan (APP) as a benefit payment performance measure.
SSA’s goal, as stated in the FY 2001 APP, is to achieve a 99.8-percent SSN accuracy rate. SSA has met or exceeded this goal since FY 1997.
RESULTS OF REVIEW
The data SSA used to report the FY 1998 SSN accuracy rate appeared to be statistically reliable. However, we believe actual performance could be better reflected if SSA implemented changes in the way it reports SSN errors. Specifically, we believe SSA should disclose it did not consider all types of errors in the accuracy rate computation.
Because SSA excluded relevant data from the SSN accuracy rate measure, it did not follow GPRA and Office of Management and Budget (OMB) guidance on developing APPs. GPRA calls for the establishment of "performance indicators (measures) to be used in measuring or assessing the relevant outputs, service levels, and outcomes of each program activity." Additionally, OMB Circular A-11, Preparation and Submission of Strategic Plans and Annual Performance Plans, part 2, states that APPs "should neither capture the complete array of measures likely to be used in managing programs, nor skimp on what is measured, resulting in a narrowly-drawn or fragmented picture of performance." We believe SSA’s method of measuring and reporting SSN accuracy results in a narrowly drawn picture of performance.
THE DATA USED TO MEASURE SSN ACCURACY APPEAR RELIABLE
Based on our review of a sample of 105 of 2,301 OQA case files, verification of the electronic SSN error file, and documentation of OQA’s end-of-line review process, the FY 1998 SSN accuracy rate appears reliable. OQA’s error determination was correct for 103 (98 percent) of the 105 cases we reviewed, including all of the critical errors. Two (2 percent) of the cases contained major errors that SSA did not identify. Also, OQA accurately entered the error data for 364 of the 368 records we verified. We did not identify any significant internal control weaknesses in the end-of-line review process, and we concluded OQA’s sampling methodology and projections were statistically sound. However, we believe SSA could modify its sampling methodology to better reflect the population of SSNs issued, as discussed in the Other Matters section of this Report. Finally, we determined two computer programs key to OQA’s review—one computes the SSN accuracy rate, and the other selects the Numidents for OQA’s sample—produced valid results.
SSA DID NOT CONSIDER ALL TYPES OF SSN ERRORS WHEN CALCULATING THE ERROR RATE
The SSN accuracy rate reported in the APP did not reflect all types of SSN errors. Specifically, the Agency did not consider major errors and SSNs issued as a result of false or improper evidentiary documents when calculating the accuracy rate. We believe excluding these errors, without explicit disclosure of such exclusions, is misleading as it implies almost perfect accuracy in SSN processing. In fact, in 1998, 208 (9 percent) of the 2,301 SSNs contained major errors. Additionally, other OIG audits and investigations have noted significant problems with counterfeit documents being presented to fraudulently obtain SSNs. However, OQA did not attempt to identify these occurrences when it conducted its review.
SSA Did Not Account For Major SSN Errors
Although OQA computed two error rates—one based on the number of critical errors detected and another based on the number of major errors the Agency made in issuing SSNs―SSA reported an SSN accuracy rate that only accounted for the number of critical errors identified. Figure 1 illustrates SSA’s calculation.
Figure 1: 1998 SSN Accuracy Rate
The Agency’s APP neither disclosed that major errors were not considered when calculating the measure nor reported the rate of those errors. If the Agency had included the 208 major errors, it would not have met its 1998 accuracy rate goal of 99.8 percent because the 1998 SSN accuracy rate, based on critical and major errors combined, was 90.7 percent.
The rate at which SSA generates major errors is problematic, as evidenced by the 9-percent major error rate in Calendar Year (CY) 1998 and the 11-percent major error rate in CY 1997. The CY 1998 error rate projected to more than 1.1 million SSNs in the universe.
We believe major errors should be considered in SSA’s accuracy rate calculation because major errors can result in an additional or incorrect action in subsequent enumeration transactions or claims actions. For example, if an applicant’s name was recorded incorrectly during the SSN application process, subsequent earnings reported for that person may not be properly credited. We believe inclusion of these errors or, at a minimum, specific disclosure of the exclusion is necessary to provide a complete picture of SSA’s SSN accuracy rate.
SSA Did Not Account For Improperly Assigned SSNs
OQA did not consider improperly assigned SSNs in the accuracy rate computation. Improperly assigned SSNs are numbers SSA assigned based on invalid and/or inappropriate evidentiary documents. Based on other OIG audits and investigations, we learned that detecting counterfeit evidentiary documents can be problematic for SSA. As a result, SSA sometimes assigns an SSN to individuals who should not have received one. Since OQA did not attempt to identify these types of errors during its enumeration review, we do not know whether the 1998 SSN accuracy rate would have dropped if the Agency had included the errors in the accuracy rate computation. To further complete the picture of SSN accuracy, we believe future OQA enumeration reviews should identify improperly assigned SSNs, and the Agency should factor in these errors or at least disclose that they are not included in the accuracy rate.
In examining the SSN applications included in the enumeration end-of-line review, OQA ensured the evidence descriptions and coding were accurate and complete. However, it did not determine whether the evidence submitted with SSN applications was valid. The OQA technician told us she did not validate some types of evidence because she could not physically examine the documents. Thus, she relied on the FO’s determination of the evidence’s validity. We agree that OQA’s validation efforts will be limited to certain types of evidentiary documents. However, the results of another OIG review indicate these efforts are necessary to identify the improper assignment of SSNs. In this review, we found that, from September 1, 1997 to February 28, 1998, SSA issued at least 999 original SSNs based on invalid/inappropriate evidentiary documents. The invalid documents included Immigration and Naturalization Service forms and counterfeit U.S. birth certificates.
CONCLUSION AND RECOMMENDATION
Although the FY 1998 99.8-percent SSN accuracy rate was reliable, SSA did not present a complete picture because it did not consider major errors and improperly assigned SSNs when calculating the SSN accuracy rate. The limited information SSA reports may mislead the public to believe SSA made virtually no errors when issuing SSNs. In fact, SSA has generated at least 1 million major errors in SSN records annually since 1997. In addition, in other OIG audits and investigations, we determined that SSA sometimes issues SSNs based on invalid or improper evidence. We do not know the impact of including improperly assigned SSNs in the accuracy rate computation; however, the Agency would not have met its 1998 goal if it had included major errors.
To improve the reporting of future SSN accuracy rates, we recommend SSA include major errors and improperly assigned SSNs when calculating future SSN accuracy rates or include a statement in the APP that specifically discloses the exclusion of such errors from the SSN accuracy rate measure.
AGENCY COMMENTS
SSA agreed with our recommendation. However, the Agency noted the correct volume of SSN applications for FY 1998 was 1.7 million original applications and 10.7 million replacement card applications. Hence, the simple random sample OQA selected adequately reflects the population, and no correction to the sampling methodology is necessary, as we suggested in the draft report. The complete text of SSA’s comments is included in Appendix D.
OIG RESPONSE
Through discussion with an OQA manager, we determined that, during our audit, OQA provided us an incorrect figure. Specifically, OQA informed us that, in FY 1998, SSA issued 2.4 million original SSNs. However, in its response to this draft report, OQA corrected that number to only 1.7 million original applications. As a result, we revised the draft report to eliminate our concern that the SSN accuracy rate sample did not represent the population.
OTHER MATTERS
SSA’S 1998 SSN ACCURACY RATE WAS PRIMARILY BASED ON A REVIEW OF LOWER RISK APPLICATIONS
OQA’s sampling methodology for the 1998 Quality of the Enumeration Process Review resulted in an examination of lower-risk transactions in almost 87 percent of the cases. We believe SSA could improve its methodology by more fairly distributing the proportion of low-and high-risk applications in the sample. The Agency could achieve this improvement by increasing the number of original applications reviewed for SSN accuracy.
We are concerned about the disproportionate distribution of the sample between original and replacement cards. The Agency based its accuracy assessment primarily on applications that have a lower error risk. In 1998, the sample included 303 original applications and 1,998 replacement card applications. Thus, replacement card applications represented 87 percent (1,998 out of 2,301) of the sample. However, only 20 percent of the critical errors OQA identified were related to applications for a replacement card. We are not inferring that replacement cards are not vulnerable to fraud. However, lower risk of error for a replacement card is inherent because this type of transaction requires little manual data input. The field office representative does not key in as much data for a replacement card because the system automatically completes many of the data fields with information it captures from the existing computer record formed when an individual was originally assigned an SSN.
ADDITIONAL ERROR INFORMATION MAY BE CAPTURED IN FY 2002
If SSA approves a proposal submitted by its Enumeration and Systems Team, the Agency would be able to measure the actual rate of wrong number cases reported by individual SSN cardholders. SSA has developed new codes for the different number requested field on the SS-5. These codes correspond to the reason the new SSN is being assigned. One of the six reason codes is for wrong number cases. In addition, SSA plans to add a new field to the Numident to reflect these codes. According to an official of SSA’s Enumeration and Systems Team in the Division of Eligibility and Enumeration Policy, this upgrade will be implemented in the FY 2002 release of the Modernized Enumeration System. OQA could obtain these data during future quality of enumeration process reviews to provide SSA management with another indicator of its performance in correctly issuing SSNs.
James G. Huse, Jr.
Appendices
APPENDIX A – Scope and Methodology
APPENDIX B – Acronyms
APPENDIX C – Flow Chart of 1998 Enumeration Review
APPENDIX D – Agency Comments
APPENDIX E – OIG Contacts and Staff Acknowledgment
Appendix A
Scope and Methodology
We reviewed the sample methodology and procedures employed in conducting the Calendar Year (CY) 1998 enumeration process review to assess its reliability as a data source for Social Security number (SSN) accuracy.
We tested and analyzed the Office of Quality Assurance and Performance Assessment’s (OQA) methodology by fully examining a sample of cases included in the CY 1998 enumeration end-of-line review. We randomly selected 100 cases free from critical error from the CY 1998 Enumeration Quality Assurance System (EQAS), which is the data base containing the results of the review of 2,301 cases. The data base is designed to measure and report on the quality of various aspects of the enumeration process. We verified the accuracy of the data in EQAS by comparing OQA’s data input sheets to a printout from EQAS.
We verified all aspects of OQA’s desk reviews by following the procedures outlined by the OQA staff. These procedures included obtaining an Alpha-Index query for each sampled applicant. We determined whether the OQA case files contained all the required forms, and all of the necessary documents were secured. We also reviewed each OQA case file included in our sample to verify the results were recorded correctly to EQAS.
From the universe of 2,301 cases, we also reviewed all 5 cases for which OQA reported a critical error. According to the enumeration review team’s supervisor, OQA selected 2,414 cases for review. However, 113 of these cases were excluded. The excluded cases included 104 that were generated from benefit claims transactions rather than SSN applications, 7 mislocated SS-5s, and 2 late returns. The following chart provides details of the 105 enumeration transactions we reviewed.
Office of the Inspector General Sample |
|||||||
Region |
Cases |
Original |
Duplicate |
On-Line |
Batch |
Major and Critical Errors |
Critical Errors |
Atlanta |
15 |
2 |
13 |
8 |
7 |
7 |
0 |
Boston |
5 |
1 |
4 |
1 |
4 |
0 |
0 |
Chicago |
19 |
4 |
15 |
14 |
5 |
2 |
0 |
Dallas |
13 |
2 |
11 |
5 |
8 |
6 |
1 |
Denver |
6 |
1 |
5 |
2 |
4 |
4 |
0 |
Kansas City |
5 |
1 |
4 |
2 |
3 |
0 |
0 |
New York |
12 |
1 |
11 |
8 |
4 |
6 |
2 |
Philadelphia |
6 |
2 |
4 |
2 |
4 |
1 |
0 |
San Francisco |
16 |
4 |
12 |
13 |
3 |
4 |
2 |
Seattle |
8 |
2 |
6 |
4 |
4 |
0 |
0 |
Total |
105 |
20 |
85 |
59 |
46 |
30 |
5 |
To complete our review, we obtained and reviewed relevant sections of the Program Operations Manual System. We also interviewed knowledgeable Social Security Administration staff to gain an understanding of the enumeration process, the OQA data base, the statistical methods used, and other relevant matters. In addition, for each case finding noted in this report, we analyzed the OQA case file and all pertinent documentation with OQA representatives and discussed the case findings with them. The OQA staff concurred with our conclusions on each of these findings.
We conducted our work at the OQA office in Woodlawn, Maryland, and the Tuscaloosa, Alabama, district office. The entity audited was the Deputy Commissioner for Operations. We conducted audit field work from October 1999 to October 2000 in accordance with generally accepted government auditing standards.
Appendix B
Acronyms
APP |
Annual Performance Plan |
CY |
Calendar Year |
EAB |
Enumeration-at-Birth |
EQAS |
Enumeration Quality Assurance System |
FO |
Field Office |
FY |
Fiscal Year |
GPRA |
Government Performance and Results Act |
OIG |
Office of the Inspector General |
OMB |
Office of Management and Budget |
OQA |
Office of Quality Assurance and Performance Assessment |
SSA |
Social Security Administration |
SSN |
Social Security Number |
Appendix C
Flow Chart of 1998 Enumeration Review
Appendix D
Agency Comments
for FY 1998 is 1.7 million original applications and 10.7 million replacement card applications. Therefore, the percentage of original SSNs for FY 1998 was 13.7 percent, and the percentage for the Agency’s sample was 13.2 percent. Hence, the simple random sample selected does reflect the population, and no correction to the sampling methodology, as suggested in "Other Matters" or Appendix D is necessary.
Please let us know if we may be of further assistance. Staff questions may be referred to
Dan Sweeney on extension 51957.
Appendix E
Contacts and Staff Acknowledgments
OIG Contacts
Kimberly
A. Byrd, Acting Director, Operational Audits Division (205) 801-1605
Reginia
Grider, Senior Auditor (205) 801-1608
Acknowledgments
In addition to those named above:
Ellen
Justice, Auditor
Kimberly
Beauchamp, Writer-Editor, Policy, Planning and Technical Services Division