MEMORANDUM
Date: June 6, 2003
To: The Commissioner
From: Inspector General
Subject: Performance Indicator Audit: Payment Accuracy (A-15-02-11086)
We contracted with PricewaterhouseCoopers (PwC) to evaluate the data used to measure 18 of the Social Security Administration's performance indicators established to comply with the Government Performance and Results Act. Attached is the final report presenting the results of two of the performance indicators PwC reviewed. The objective of this audit was to assess the reliability of the data used to measure the accuracy of Old-Age, Survivors and Disability Insurance and Supplemental Security Income benefit payments.
Please comment within 60 days from the date of this memorandum on corrective action taken or planned on each recommendation. If you wish to discuss the final report, please call me or have your staff contact Steven L. Schaeffer, Assistant Inspector General for Audit, at (410) 965-9700.
James G. Huse, Jr.
OFFICE
OF
THE INSPECTOR GENERAL
SOCIAL SECURITY ADMINISTRATION
PERFORMANCE
INDICATOR AUDIT:
PAYMENT ACCURACY
June 2003
A-15-02-11086
AUDIT
REPORT
Mission
We improve SSA programs and operations and protect them against fraud, waste,
and abuse by conducting independent and objective audits, evaluations, and investigations.
We provide timely, useful, and reliable information and advice to Administration
officials, the Congress, and the public.
Authority
The Inspector General Act created independent audit and investigative units, called the Office of Inspector General (OIG). The mission of the OIG, as spelled out in the Act, is to:
Conduct and supervise independent and objective audits and investigations
relating to agency programs and operations.
Promote economy, effectiveness, and efficiency within the agency.
Prevent and detect fraud, waste, and abuse in agency programs and operations.
Review and make recommendations regarding existing and proposed legislation
and regulations relating to agency programs and operations.
Keep the agency head and the Congress fully and currently informed of problems
in agency programs and operations.
To ensure objectivity, the IG Act empowers the IG with:
Independence to determine what reviews to perform.
Access to all information necessary for the reviews.
Authority to publish findings and recommendations based on the reviews.
Vision
By conducting independent and objective audits, investigations, and evaluations, we are agents of positive change striving for continuous improvement in the Social Security Administration's programs, operations, and management and in our own office.
MEMORANDUM
To: Office of the Inspector General
From: PricewaterhouseCoopers LLP
Date: May 30, 2003
Subject: Performance Indicator Audit: Payment Accuracy (A-15-02-11086)
The Government Performance and Results Act (GPRA) of 1993 requires the Social
Security Administration (SSA) to develop performance indicators that assess
the relevant service levels and outcomes of each program activity set forth
in its budget. GPRA also calls for a description of the means employed to verify
and validate the measured values used to report on program performance.
The objective of this audit was to assess the reliability of the data used to
measure the following Fiscal Year (FY) 2002 GPRA performance indicators:
1.1.1 Performance Indicator FY 2002 Goal
Percent of OASDI payment outlays "free" of overpayments and underpayments
(based on non-medical factors of eligibility)
Overpayments 99.8%
Underpayments 99.8%
Percent of SSI payment outlays "free" of overpayments and underpayment
(based on non-medical factors of eligibility)
Overpayments 94.7%
Underpayments 98.8%
See Appendix A for a description of the audit scope and methodology.
BACKGROUND
SSA offers retirement and long-term disability programs to the general public.
Old-Age, Survivors and Disability Insurance (OASDI) is authorized under title
II of the Social Security Act. Through the OASDI program, eligible workers and
sometimes their family receive monthly benefits if they retire at an appropriate
age or are found to have a disability that either prevents them from engaging
in substantial gainful activity for at least 12 months or can be expected to
result in death. Supplemental Security Income (SSI) is authorized under title
XVI of the Social Security Act and provides monthly payments to aged and disabled
individuals based on financial need and medical requirements.
SSA measures payment accuracy through Stewardship and Index of Dollar Accuracy
(IDA) reviews. Stewardship reviews measure the quality and accuracy of recurring
payments provided by SSA and are reported to Congress in the Stewardship Reports.
IDA reviews focus on the accuracy of initial claim payments. This performance
indicator audit evaluated payment accuracy for the OASDI and SSI Stewardship
reviews.
SSI Stewardship reviews began in 1975, Retirement and Survivors Insurance (RSI)
Stewardship reviews began in 1981, and Disability Insurance (DI) Stewardship
reviews began in 1997. SSA reports the RSI and DI results as one measure (OASDI)
in the accuracy reviews. The Stewardship reviews are conducted for a sample
of beneficiaries who received recurring payments during the review month. The
review focuses on verifying the factors that lead to monthly payments. See Appendix
B for a workflow and description of the review.
The Stewardship review is an examination of all payments made on the sampled
beneficiaries' Social Security numbers (SSN). SSA treats sampled SSNs as "accounts"
and reviews all payments associated with those accounts. The OASDI Stewardship
review examines the account to determine whether the payment is accurate. The
SSI Stewardship review examines the account to determine whether the recipient
is eligible to receive SSI payments and if the payment is accurate.
Within SSA, the Office of Quality Assurance and Performance Assessment (OQA)
randomly selects OASDI and SSI beneficiaries each month to participate in the
review. The Assistance and Insurance Program Quality Branch, or regional OQA
offices, receive the sample participants' information from OQA, schedule and
conduct interviews with the recipients if necessary, and determine whether there
is an over or underpayment based on non-medical factors of eligibility. The
results of the review are entered into the SSI Quality Assurance (QA) System
or the RSI QA System. OQA calculates the performance indicator based on the
outcome of the review.
Data from FYs 2001 and 2002 was not available for inclusion in this report.
We used the latest available data, FY 2000, to recalculate the performance indicators.
Our review encompassed four performance indicators.
1. Percent of OASDI payment outlays free of overpayments.
2. Percent of OASDI payment outlays free of underpayments.
3. Percent of SSI payment outlays free of overpayments.
4. Percent of SSI payment outlays free of underpayments.
The formula used to calculate each performance indicator follows.
Overpayment (or underpayment) dollars
OASDI Payment Accuracy = 1 - -------------------------------------------------
* 100
All OASDI payments
Overpayment (or underpayment) dollars
SSI Payment Accuracy = 1 - -------------------------------------------------
* 100
All SSI payments
The payment accuracy rate performance indicators are estimates derived from
sampled cases and will not generally equal the true accuracy rates for all payments.
Accordingly, SSA calculates the precision, or margin of error, for each of the
estimates. The margin of error provides a range of values that are expected
to contain the true population accuracy rate with some level of confidence.
Appendix C provides more information on SSA's methodology for calculating the
margin of error. The margin of error is not directly part of the payment accuracy
rate indicator. Rather, it provides information to understand how close the
estimated accuracy rate is expected to be to the true accuracy rate.
As part of the financial statement audit, PricewaterhouseCoopers (PwC) examined
SSA's payment accuracy for both the OASDI and SSI programs. PwC's FY 2001 financial
statement audit included a review of the OASDI sample selection methodology
and the SSI selection programs. PwC recommended SSA integrate their selection
programs, improve the documentation, and use the t-distribution instead of the
normal distribution to calculate the margin of error. In FY 2002, the PwC financial
statement audit included a review of the SSI sample selection methodology and
the OASDI selection programs. The financial statement audit included a review
of any changes made to the sample selection methodology and programs. To avoid
duplication of effort with the financial statement audit, this performance audit
focused on replication of the payment accuracy performance indicators. Our evaluation
of SSA's methodology and programs builds on findings from the financial statement
audit for both OASDI and SSI.
RESULTS OF REVIEW
We evaluated payment accuracy for both the OASDI and SSI programs. We found
SSA's methodology for calculating accuracy rate performance indicators to be
appropriate and statistically valid. We successfully replicated the FY 2000
performance indicators from SSA's Performance and Accountability Report for
FY 2001. We also examined the more detailed Payment Accuracy (Stewardship) Reports
that SSA prepares and issues to Congress. Overall, we found the performance
indicators to be appropriate GPRA measures. They are direct measures of outcomes
and support SSA's strategic goal, "…to ensure the integrity of Social
Security programs, with zero tolerance for fraud and abuse."
SSA provides the margin of error in the Stewardship Reports, but does not include
this information in the final Performance and Accountability Report. While SSA
is not required to include this information in the Performance and Accountability
Report, the margin of error provides valuable information on the precision of
the statistical estimates.
We found that SSA's margin of error, as reported in the Stewardship reports,
is not calculated appropriately for the OASDI payment accuracy rates. When calculating
the margin of error, OQA did not weight the error dollars to the total population.
In addition, the margin of error equations contain an incorrect factor for both
the OASDI and SSI programs. After recalculation, the margin of error for OASDI
payment accuracy rates decreased and the margin of error for SSI payment accuracy
rates slightly increased.
Neither the lack of a published margin of error nor the incorrect margin of
error calculations change the fundamental result that the performance indicators
are appropriate and statistically valid. However, these deficiencies do represent
areas for improvement. Additionally, we noted that SSA could improve its documentation
by saving the sample data used to calculate the performance indicators.
PERFORMANCE INDICATOR DATA WAS RELIABLE
This audit consisted of recalculating the payment accuracy rates for the OASDI program, which is a combination of the RSI and DI programs, and the SSI program. We recalculated the dollar-weighted percentage of overpayments and underpayments for the RSI, DI, and SSI programs, based on FY 2000 data from OQA. Our recalculations matched the payment accuracy reported by SSA for FY 2000.
OQA does not separate and retain data used to calculate the accuracy rates
After calculating the official payment accuracy rates, OQA does not separate and retain the data. In addition, subsequent changes can be made to the data. To complete our evaluation, we recalculated the accuracy rates using the most current Stewardship review data for FY 2000. Our results matched OQA's original results for all accuracy rates except DI underpayment. OQA retains a summary diagnostic report of the data used to calculate the payment accuracy rates. From a review of this diagnostic report we inferred that a record had changed between the original data and the data used in our validation. After adjusting for this record, our recalculated accuracy rates matched OQA's original accuracy rates.
Margin of error is not published with the accuracy rate estimates
SSA conducts the Stewardship Review for a relatively small sample of cases
and uses the sample data to develop an accuracy estimate for all recurring payments.
SSA also calculates a margin of error for each accuracy estimate. Margin of
error is an important element in assessing the precision of a statistical estimate.
Results with a large margin of error do not provide meaningful information on
the true accuracy rate for the entire population.
The SSI Stewardship Report identifies margin of error (termed 'precision') for
the SSI performance indicators. The OASDI Stewardship Report identifies margin
of error (termed 'precision') for the separate RSI and DI overpayment/underpayment
accuracy rates but not for the overall OASDI performance indicator. The FY 2001
Performance and Accountability Report does not provide the margin of error results
for either SSI or OASDI. As a result, the general public is not informed of
the precision of the accuracy rates.
Margin of error is not calculated correctly
Through a series of interviews, we found that OQA calculates the margin of
error for the accuracy rates using a statistical technique known as random group.
Our understanding of this approach is found at Appendix C. We agree that this
technique is reasonable and appropriate. However, during our review, we found
two issues with OQA's application of the procedure.
1. We found an inconsistency in the margin of error calculations for the OASDI
program. OQA indicated it weights error dollars when using the random group
approach to calculate the margin of error. However, OQA used unweighted error
dollars to calculate the margin of error reported in the Stewardship report.
After weighting the error dollars and recalculating the margin of error, we
found that the margin of error is smaller than originally provided in the Stewardship
reports.
2. OQA also uses an incorrect factor in calculating the margin of error (normal
distribution as opposed to t-distribution) . As a result, each margin of error
(including both OASDI and SSI) has been understated by 15 percent. In other
words, the margin of error is larger than originally provided in the Stewardship
reports. Appendix C provides a more detailed explanation of this difference.
The following table shows the margin of error values reported by SSA in its
Stewardship Reports and the recalculated values. Note that both issues above
apply to the OASDI margin of error calculations and impact the margin of error
in opposite directions. The net effect reflected in the table below is that
the first issue has a much greater impact than the second issue on the final
margin of error values.
Accuracy Rate Margin of Error at 95% Confidence
Estimated Accuracy Rate SSA Reported Margin of Error Recalculated Margin of
Error
DI Overpayment 99.62% +/- 0.41% +/- 0.28%
DI Underpayment 99.61% +/- 1.11% +/- 0.78%
RSI Overpayment 99.96% +/- 0.30% +/- 0.04%
RSI Underpayment 99.89% +/- 0.37% +/- 0.16%
SSI Overpayment 93.60% +/- 0.56% +/- 0.65%
SSI Underpayment 98.60% +/- 0.22% +/- 0.25%
We recalculated the margin of error for all over and underpayment performance indicators. The recalculated margin of error for DI and RSI over and underpayments was lower than SSA previously reported. For example, SSA calculated the DI overpayment as 99.62 +/- 0.41 percent. We recalculated the true accuracy rate as 99.62 +/- 0.28 percent. The recalculated margin of error for the SSI over and underpayments was slightly higher than SSA previously reported. While these margin of error discrepancies do not change the estimated accuracy rates, an incorrect margin of error provides misleading information on the precision of the estimated accuracy rates.
OTHER MATTERS
The Office of Management and Budget's (OMB) Circular No. A-11 defines payment
accuracy as:
"Erroneous payments are payments made under the programs listed in Exhibit
57 (for SSA, these programs are OASI, DI, and SSI) that should not have been
made or were made for an incorrect amount…Examples of erroneous payments
include payments to ineligible persons or the wrong organizations, payments
in the wrong amount, payments for ineligible services, duplicate or other overpayments,
and payments for services never received. Erroneous payments may be due to procedural
or administrative errors made by the payor (e.g., providing incorrect account
numbers in payment instructions), or errors or fraud by payees or claimants
(e.g., under reporting of income by beneficiary). Covered payments include overpayments
and underpayments made by the Federal Government, its direct contractors, and
by States or other grant recipients administering Federal programs."
In the General Accounting Office's (GAO) October 2001 executive guide, Strategies
to Manage Improper Payments, GAO provides examples of improper payments as "…inadvertent
errors, payments for unsupported or inadequately supported claims, payments
for services not rendered, payments to ineligible beneficiaries, and payments
resulting from outright fraud and abuse by program participants and/or federal
employees."
SSA's policy and operation manual defines an overpayment as "…the
total amount an individual received for any period which exceeds the total amount
which should have been paid for that period." SSA has three performance
indicators to measure payment accuracy in its FY 2002 Annual Performance Plan:
OASDI payment accuracy
SSI payment accuracy, excluding unpreventable errors
SSI payment accuracy, including preventable and unpreventable errors
SSA's FY 2000 title II Stewardship report states that for OASDI payment accuracy,
"Error dollars refer to incorrect payment made to a case as a whole."
Overpayments are defined as payment in a sample month that "…was more
than the amount that should have been paid." SSA's FY 2000 title XVI Stewardship
report describes SSI payment accuracy as the correct amount, or what should
have been paid, compared to the amount that was paid to the recipient. In addition,
SSA measures the SSI accuracy rate exclusive of unavoidable errors, where unavoidable
errors are errors that "…result from limitations in the Agency's computer
systems and/or limitations placed on SSA by law."
For example, SSA does not include payments made after a beneficiary's death
as overpayments for both the OASDI and SSI performance indicators. OMB and GAO
definitions would include these in the payment accuracy calculations. Also,
SSA does not include errors based on medical factors of eligibility for the
OASDI and SSI payment accuracy calculations while OMB and GAO definitions would
include these in the calculation. Thus, SSA's definition of payment accuracy
is narrower than OMB and GAO's definitions.
1.1.1.1 CONCLUSIONS AND RECOMMENDATIONS
Our overall conclusion is that the performance indicators are accurate and appropriate GPRA measures. OQA calculates the SSI, RSI, and DI payment accuracy appropriately. OQA also compiles the OASDI payment accuracy from the RSI and DI results appropriately. Our audit identified four opportunities for improvement. Our recommendations are as follows.
1. Retain sample data used to calculate accuracy rates
SSA should retain the exact set of data used to generate the final accuracy
estimates. This will improve the ability of third parties to review and validate
accuracy rates in the future.
2. Show the margin of error in future Performance and Accountability Reports
It is a best practice to include the margin of error when reporting statistical
estimates. The margin of error is an important element in determining the value
of statistical estimates. This is especially true for the payment accuracy performance
indicators because even very small differences in the accuracy rates result
in tens of millions of dollars in total program costs. For example, the Stewardship
report estimates that every 0.1 percent difference in the accuracy rates is
approximately $54 million for the DI program and $348 million for the RSI program.
We note that the FY 2001 Performance and Accountability Report references the
OASDI and SSI Stewardship Report, and the Stewardship Reports contain margin
of error data. However, these Stewardship Reports are not generally available
to the public and we do not believe that simply referencing them represents
sufficient documentation. We recommend that margin of error information be directly
reported with the overpayment/underpayment accuracy rate performance indicators.
3. Accurately calculate margin of error
SSA should correct both identified issues with the margin of error calculations:
a) Calculate OASDI margin of error using weighted error dollars consistent with
the approach for the SSI accuracy calculations.
b) Calculate the margin of error using the appropriate value from the t-distribution rather than from the normal distribution. PwC identified this issue and made the same recommendation during the FY 2001 financial statement audit of the OASDI accuracy rates. OQA agreed with our findings and recommendations but this occurred after the FY 2000 payment accuracy results had been published. We understand that SSA has made the change for FY 2001. We recommend this change be made for the SSI program as well.
4. Measure payment accuracy in accordance with OMB and GAO definitions
We recommend that SSA modify its payment accuracy performance indicators so
that they are consistent with OMB and GAO definitions. In addition, we recommend
that SSA measure preventable errors as a separate payment accuracy performance
indicator for both OASDI and SSI. Finally, we recommend that SSA specifically
document the error types included in each payment accuracy performance indicator.
AGENCY COMMENTS
In response to our draft report, SSA agreed with recommendations 1 through 3 and agreed in part with recommendation 4. The full text of SSA's comments can be found in Appendix E.
With respect to recommendation 4, SSA agrees that it should measure payment
accuracy in accordance with OMB and GAO definitions and believes that its payment
accuracy measures are in accordance with both OMB's and GAO's guidelines.
SSA disagreed with part 2 of recommendation 4, that it should establish a separate
performance indicator for preventable and unpreventable errors in OASI. Further,
the Agency stated it has already established separate performance indicators
for preventable and unpreventable errors for the SSI program.
SSA agrees with the 3rd part of recommendation 4 to specifically document the error types included in each payment accuracy performance indicator. However, SSA states the documentation already exists in SSA's quality assurance manuals, OASDI and SSI payment accuracy (stewardship) reports and SSA's Annual Performance Plan.
PWC RESPONSE
Although SSA states it measures payment accuracy in accordance with OMB and GAO definitions, we continue to believe that its definition is more narrow than OMB and GAO definitions. We recommend the Agency continue to work with OMB and GAO to refine the definition of its payment accuracy calculation.
With respect to the technical comments, we only looked at the two performance indicators listed on page one. We based our performance indicators names on the APP at the time of the audit.
1.1.1.1.1 Appendices
APPENDIX A - Scope and Methodology
APPENDIX B - Flowcharts and Descriptions
APPENDIX C - Statistical Methodology for Margin of Error Calculation
APPENDIX D - Acronyms
APPENDIX E - Agency Comments
Appendix A
Scope and Methodology
We conducted this audit to examine Social Security Administration's (SSA) Fiscal
Year (FY) 2002 payment accuracy performance indicators. SSA developed these
performance indicators to meet the requirements of Government Performance and
Results Act (GPRA). Because FY 2002 payment accuracy data and results were not
available at the time of this audit, we used the latest available data (FY 2000)
in our evaluation of the controls and the final reported performance indicators.
In addition, we evaluated differences in methodology between FYs 2000 and 2002.
We performed our testing from May 1, 2002 through July 18, 2002 as follows.
Reviewed the sampling selection and weighting procedure;
Recalculated the sampling estimates and sampling error estimates;
Discussed the process and techniques used to calculate the payment accuracy
rates with SSA staff;
Obtained documentation on the sampling methodology, including the sampling selection
and the weighting procedure;
Discussed the techniques employed by SSA to calculate the sampling error estimates
with Office of Quality Assurance and Performance Assessment (OQA) staff;
Reviewed the sampling methodology and verified that SSA procedures were statistically
sound;
Obtained payment accuracy data, including Social Security number, amount paid,
dollar error, and sampling weight;
Recalculated the payment accuracy rates; and
Verified that the recalculated payment accuracy rates were the same as the accuracy
reported by SSA.
In conducting this audit, we also:
Reviewed SSA's Accountability Report for FY 1999, Annual Performance Plan for
FY 2001, and Revised Final Performance Plan for FY 2002 to determine the baseline
data, definition, and data source for the performance indicator;
Reviewed GPRA and Office of Management and Budget guidance related to GPRA;
and
Interviewed OQA and Office of Strategic Management policy and program staff
to document the methodologies and procedures used to produce performance data
for this indicator.
Our audit was limited to testing at SSA's Headquarters in Woodlawn, Maryland. The procedures we performed were in accordance with the American Institute of Certified Public Accountants' Statement on Standards for Consulting Services and the General Accounting Office's Government Auditing Standards ("Yellow Book") for performance audits.
Appendix B
Flowcharts and Descriptions
Title II and title XVI payment accuracy:
Select cases to review.
Review of cases.
Calculate payment accuracy.
Select cases to review - title XVI:
The Office of Statistics and Special Area Studies (OSSAS) extracts two segments
from the Supplemental Security Record (SSR).
OSSAS selects field offices (FO) by cluster based on travel costs for a reviewer
to conduct an in-home interview.
OSSAS selects cases to review from the list of FOs in the previous stage.
OSSAS uploads sample on SSI QA System.
SSI QA System sends electronic notification to Assistance and Insurance Program
Quality Branch (AIPQB), or the Regional OQA offices.
Review cases - title XVI:
AIPQB receives electronic notification and system outputs, including the SSI
Record Display (SSIRD), the Summary Earnings Query (SEQY), and other alerts,
and FOs receive notice to send folder to AIPQB.
AIPQB creates a review folder and adds all documentation.
AIPQB contacts recipients to set appointment and obtain releases.
AIPQB reviews and collects case information from recipients.
AIPQB reviewer completes SSA-8538 summary and recalculates benefit payment.
AIPQB requests third-party verification.
AIPQB accesses other matches, such as the Immigration and Naturalization Service
(INS), Office of Child Support Enforcement (OCSE), and Veterans Affairs (VA).
AIPQB reviewer attempts to verify recipient eligibility and benefit amount.
Does AIPQB identify any deficiencies? If AIPQB does not detect any deficiencies,
they return the claim folder to the FO and update the SSI QA data base. If AIPQB
does identify deficiencies, they do the following.
AIPQB completes the benefit calculation worksheet to calculate the over or underpayment.
AIPQB updates the SSI QA data base.
AIPQB completes SSA-93 form and forwards the form and claim folder to the FO.
Does the FO dispute the review? If the FO disputes the review, they resolve
their differences with AIPQB. AIPQB maintains the review folder for 18 months,
retains the SSA-8538 for 5 years, disposes of the case file, and updates the
SSI QA data base. If the FO does not dispute the review, the FO must correct
the deficiency within 30 days. AIPQB ensures the correction was made within
30 days, maintains the review folder for 18 months, retains the SSA-8538 for
5 years, disposes of the case file, and updates the SSI QA data base.
Select cases to review - title II:
The Office of Software Design and Development (OSDD) uses the RETAP program
to select 1 of every 500 cases.
OSSAS selects FOs by cluster based on travel costs for a reviewer to conduct
an in-home interview.
OSSAS selects cases to review from the list of FOs in the previous stage.
OSSAS uploads sample on RMAS.
RMAS sends electronic notification to AIPQB.
Review cases - title II:
AIPQB requests claim folders from the program service center (PSC).
AIPQB gathers queries, including a query of the NUMIDENT and the Master Beneficiary
Record (MBR) and the Detailed Earnings Query (DEQY), SEQY and Full Claims Earnings
Record (FCER).
AIPQB receives folder and begins desk review.
AIPQB creates a review folder with all documentation.
AIPQB reviewer completes SSA-2930 summary and recalculates benefit payment.
Is the beneficiary under 16 or over 70? If yes, the AIPQB contacts the beneficiary
by telephone. If no, AIPQB schedules an in-person interview. The reviewer compares
information from the beneficiary to information on the SSA-2930 form.
Does AIPQB identify any deficiencies? If no, AIPQB enters the results of the
review into the RSI QA system. The Central OQA Headquarters (HQ) selects 10
percent of cases to review for AIPQB consistency. AIPQB returns the case file
to PSC. If yes, the following occurs.
AIPQB completes the benefit calculation worksheet to calculate the over/under
payment.
AIPQB completes a SSA-93 form and forwards the file to Central OQA Headquarters
for a consistency review.
The Central OQA HQ forwards SSA-93 form and file to PSC.
Does PSC rebut findings? If yes, the PSC and AIPQB resolve the differences.
AIPQB maintains a review folder for 18 months, retains the SSA-2930 for 5 years,
and disposes of the case file. If no, the operating component performs the corrective
action within 45 days. AIPQB ensures the correction was made within 45 days.
AIPQB maintains a review folder for 18 months, retains the SSA-2930 for 5 years,
and disposes of the case file.
Calculation of payment accuracy for title II and XVI:
OQA pulls accuracy data from the RSI QA system and the SSI QA system.
OQA weights cases to sample size and universe.
OQA calculates the accuracy rate by dividing the projected dollar errors by
the total payments and subtracting the results from one.
OQA creates and publishes the stewardship report.
Appendix C
Statistical Methodology for Margin of Error Calculation
Social Security Administration (SSA) determines the margin of error (or precision)
using a statistical technique known as the random group approach. SSA performs
this technique by taking the full set of sample data, called the parent sample,
and partitioning it into 10 groups. Each sample record is included in 1 of the
10 groups based on the seventh digit of the Social Security number for that
record.
First, the Office of Quality Assurance and Performance Assessment (OQA) estimates
the weighted dollar accuracy, , from the parent sample. OQA reports this estimate
in the Stewardship reports.
Second, OQA separates the data into 10 groups and estimates the weighted dollar
accuracy for each group, . SSA then calculates the margin of error as
Where
Under the random group approach, OQA uses a small number of groups to calculate
the margin of error. According to statistical literature, if the number of groups
is small, it is more appropriate to use the t-distribution than the normal distribution.
As a result, OQA should calculate the margin of error as:
Where
We recommend using the t-distribution (where t = 2.26) instead of the normal
distribution (where z = 1.96) because OQA uses a small sample size to estimate
the margin of error under the random group approach. As a result, OQA results
understate the margin of error by a factor of 1.15 (2.26/1.96) or 15 percent.
Appendix D
Acronyms
AIPQB Assistance and Insurance Program Quality Branch
DEQY Detailed Earnings Query
DI Disability Insurance
FCER Full Claims Earnings Record
FO Field Office
FY Fiscal Year
GAO General Accounting Office
GPRA Government Performance and Results Act
HQ Headquarters
IDA Index of Dollar Accuracy
INS Immigration and Naturalization Service
MBR Master Beneficiary Record
OASDI Old-Age, Survivors and Disability Insurance
OCSE Office of Child Support Enforcement
OMB Office of Management and Budget
OQA Office of Quality Assurance and Performance Assessment
OSDD Office of System Design and Development
OSSAS Office of Statistics and Special Area Studies
PSC Program Service Center
PwC PricewaterhouseCoopers LLP
QA Quality Assurance
RSI Retirement and Survivors Insurance
SEQY Summary Earnings Query
SSA Social Security Administration
SSI Supplemental Security Income
SSIRD SSI Record Display
SSN Social Security number
SSR Supplemental Security Record
VA Department of Veterans Affairs
Appendix E
Agency Comments
SOCIAL SECURITY
MEMORANDUM
Date: May 30, 2003
To: James G. Huse, Jr.
Inspector General
From: Larry W. Dye
Chief of Staff
Subject: Office of the Inspector General (OIG) Draft Report, "Performance Indicator Audit: Payment Accuracy" (A-15-02-11086)-INFORMATION
We appreciate OIG's efforts in conducting this review. Our comments on the draft report content and recommendations are attached.
Staff questions may be referred to Laura Bell at extension 52636.
SSA Response
COMMENTS ON THE OFFICE OF THE INSPECTOR GENERAL (OIG) DRAFT REPORT "PERFORMANCE INDICATOR AUDIT: PAYMENT ACCURACY" (AUDIT NO. A-15-02-11086
We appreciate the opportunity to review and comment on the draft report, and are pleased with your conclusion that the performance indicators are accurate and appropriate Government Performance Results Act (GPRA) measures.
Our responses for the specific recommendations are provided below. We are also
providing a technical comment that should be included to enhance the accuracy
of the report.
Recommendation 1
The Social Security Administration (SSA) should retain sample data used to calculate accuracy rates.
SSA Response
We agree, and have already taken action to retain the exact set of data used to generate the final accuracy estimates.
Recommendation 2
SSA should show the margin of error in future Performance and Accountability Reports (PAR).
SSA Response
We agree, and we will include the margin of error when we submit the Title
II and Title XVI stewardship data for inclusion in the next Performance and
Accountability Report (PAR).
Recommendation 3
SSA should accurately calculate margin of error.
SSA Response
We agree, and are now calculating the margin of error using the appropriate
value from the T-distribution for both the Old Age and Survivors Disability
Insurance (OASDI) and Supplemental Security Income (SSI) payment accuracy reviews.
PricewaterhouseCoopers (PwC) was advised of the action we took on this recommendation
in the August 2002 update to PwC's Fiscal Year 2001 Management Letter.
Recommendation 4
SSA should measure payment accuracy in accordance with the OMB and GAO definitions. SSA should measure preventable errors as a separate payment accuracy performance indicator for both OASDI and SSI. SSA should specifically document the error types included in each payment accuracy performance indicator.
SSA Response
We agree that SSA should measure payment accuracy in accordance with OMB and GAO definitions, and we believe that our payment accuracy measures are in accordance with both OMB's and GAO's guidelines. The data definitions that are included in our fiscal year (FY) 2003 Annual Performance Plan (APP) were reviewed with OMB to ensure compliance with their guidelines. We also believe that they meet GAO's general definition as GAO has been very actively involved in OASDI and SSI payment issues.
With respect to the second part of the recommendation, "SSA should measure preventable errors as a separate payment accuracy performance indicator for both OASDI and SSI" and the specific items regarding payments after death and disability determination decision errors based on medical factors, we do not agree.
The Agency has already established separate performance indicators for preventable and unpreventable errors for the SSI program. We are required by statue, regulation or court order to make certain payments that may later be determined to be recoverable overpayments. We emphasize that these overpayments were not due to Agency or beneficiaries caused error, but were required to be made by statue, regulation or court order.
As OASI program overpayment accuracy rate for fiscal years (FYs) 2001 and 2000 were 99.98 percent and 99.96 percent, respectively. The OASI underpayment accuracy rates for FYs 2001 and 2000 were 99.78 percent and 99.89 percent, respectively. Even if SSA devotes additional resources to improve the rate of payment accuracy in the OASI program, any improvement in payment accuracy may not be discernible in SSA's quality assurance reviews. Therefore, we believe it is not necessary or appropriate for us to establish a separate performance indicator for preventable and unpreventable errors in OASI.
With respect to errors based on medical factors, we already have a separate performance indicator for Disability Determination Services medical decisional accuracy and do not believe medical accuracy should be part of this measure. Due to the fundamental differences in the decision making process at the initial claims and appeals levels, a combined indicator would result in an inaccurate measure. The separate indicators provide data that is more realistic and useful for managing the disability program. Our position is that the medical improvement review standard and prospective ineligibility are part of the current law, and that payments made in accordance with current law are, by definition, not erroneous. The current law was enacted in the mid-1980s following extensive debate and public comment around these controversial issues and after several years of moratorium on continuing disability reviews.
The legislative history in the area is very clear. Prior to the 1984 amendments, the law had permitted termination of benefits using a standard which included correcting prior error on medical issues. That is, a person could be told he or she was eligible for benefits one day and ineligible the next, without any changes in their medical condition. The new law required the much higher stand of evidence of medical improvement before a termination could be effectuate. The law eliminated what was seen as a kind of double jeopardy. This is especially sensitive issue given the subjective nature of the allow/deny decision and the fact that two adjudicators can reach opposites decisions on the same record.
We will continue to carry the medical decisional accuracy separately as we recognize its strategic importance to the Agency.
The third part of the recommendation calls for SSA to specifically document
the error types included each payment accuracy performance indicator. We agree
that it is appropriate to have documentation of the types of error included
in SSA's payment accuracy review, and the documentation already exists in SSA's
quality assurance manuals. Specifically, the manuals define the specific types
of errors that are to be recorded in the review process to measure the percentage
of dollars paid that are accurately paid, overpaid or underpaid. Our payment
accuracy review system has been audited several times by both GAO and OIG to
ensure that it utilizes an appropriate and comprehensive methodology and provides
accurate projections of overpayment and underpayment error. In addition our
OASDI and SSI payment accuracy (stewardship) reports include the basic types
of error that occur, and SSA's APP contains the data definitions of how the
error rates are calculated for the payment accuracy performance indicators.
Technical Comments
Page 1 shows two payment accuracy measures for FY 2002, but page 7 references
three measures. To be consistent, page 1 should probably show all three measures,
and the names of the SSI payment accuracy measures should be cited exactly the
way they are printed in the Performance Plan for FY 2003 and Revised Performance
Plan for Fiscal Year 2002.
Overview of the Office of the Inspector General
Office of Audit
The Office of Audit (OA) conducts comprehensive financial and performance audits
of the Social Security Administration's (SSA) programs and makes recommendations
to ensure that program objectives are achieved effectively and efficiently.
Financial audits, required by the Chief Financial Officers Act of 1990, assess
whether SSA's financial statements fairly present the Agency's financial position,
results of operations, and cash flow. Performance audits review the economy,
efficiency, and effectiveness of SSA's programs. OA also conducts short-term
management and program evaluations focused on issues of concern to SSA, Congress,
and the general public. Evaluations often focus on identifying and recommending
ways to prevent and minimize program fraud and inefficiency.
Office of Executive Operations
The Office of Executive Operations (OEO) provides four functions for the Office
of the Inspector General (OIG) - administrative support, strategic planning,
quality assurance, and public affairs. OEO supports the OIG components by providing
information resources management; systems security; and the coordination of
budget, procurement, telecommunications, facilities and equipment, and human
resources. In addition, this Office coordinates and is responsible for the OIG's
strategic planning function and the development and implementation of performance
measures required by the Government Performance and Results Act. The quality
assurance division performs internal reviews to ensure that OIG offices nationwide
hold themselves to the same rigorous standards that we expect from the Agency.
This division also conducts employee investigations within OIG. The public affairs
team communicates OIG's planned and current activities and the results to the
Commissioner and Congress, as well as other entities.
Office of Investigations
The Office of Investigations (OI) conducts and coordinates investigative activity
related to fraud, waste, abuse, and mismanagement of SSA programs and operations.
This includes wrongdoing by applicants, beneficiaries, contractors, physicians,
interpreters, representative payees, third parties, and by SSA employees in
the performance of their duties. OI also conducts joint investigations with
other Federal, State, and local law enforcement agencies.
Counsel to the Inspector General
The Counsel to the Inspector General provides legal advice and counsel to the
Inspector General on various matters, including: 1) statutes, regulations, legislation,
and policy directives governing the administration of SSA's programs; 2) investigative
procedures and techniques; and 3) legal implications and conclusions to be drawn
from audit and investigative material produced by the OIG. The Counsel's office
also administers the civil monetary penalty program.