EXECUTIVE
SUMMARY
OBJECTIVE
The objective of this evaluation was to benchmark private
and public business practices in measuring payment accuracy.
BACKGROUND
The Government Performance and Results Act (GPRA), enacted
in August 1993, seeks to improve public confidence by systematically
holding Federal agencies accountable for achieving program results.
Agencies must set performance goals, measure performance against
the goals, and report publicly on their performance. GPRA also
calls for agencies to have strategic plans in place by September
30, 1997. Additionally, GPRA requires agencies to provide a basis
for comparing results and for verifying and validating measured
values. Similarly, Executive Order 12862 requires benchmarking
customer service standards against the best in business. This benchmarking
exercise, by examining the procedures and performances of other
organizations in monitoring payment accuracy, will address both
the GPRA and Executive Order concerns.
The Social Security Administration (SSA) annually conducts
payment accuracy reviews of its title II Old-Age, Survivors, and
Disability Insurance; and title XVI Supplemental Security Income
(SSI) programs, including the Index of Dollar Accuracy (IDA) reviews
and Stewardship reviews. These reviews are used to determine SSA`s
payment accuracy rate through a quality review of a selection of
sampled cases. The IDA measures payment accuracy for title II initial
claims, title XVI initial claims, and field office (FO) redeterminations.
Initial claims include original entitlements and related retroactive
benefit decisions. The title II review does not examine payments
made to individuals due to disabilities. The title XVI review does
not include reviews of the initial disability decisions made by
State Disability Determination Services.
The Stewardship reviews of title II and title XVI programs
are not limited to just initial cases or redeterminations. The
Stewardship reviews are based on a sample drawn from the entire
universe of Retirement and Survivors Insurance (RSI) beneficiaries
who received title II payments and from the entire universe of
SSI recipients who received title XVI payments. The Stewardship
reviews identify the overall payment accuracy, as well as the rate
of overpayments and underpayments made by SSA.
FINDINGS
ORGANIZATIONS ISSUING REPETITIVE PAYMENTS MONITORED THE
ACCURACY OF PAYMENTS
ORGANIZATIONS USED TWO METHODS TO MEASURE THE ACCURACY
OF THEIR PAYMENTS
MOST ORGANIZATIONS MEASURED BOTH THE AMOUNT OF DOLLARS
IN ERROR AND THE NUMBER OF CASES WITH ERRORS
ORGANIZATIONS TREATED OVERPAYMENTS AND UNDERPAYMENTS DIFFERENTLY
TWO TYPES OF PAYMENT ACCURACY RATES WERE USED
MOST ORGANIZATIONS DID NOT PROJECT THE COSTS OF ERRORS
MOST ORGANIZATIONS WERE REQUIRED BY A GOVERNMENTAL OR PROFESSIONAL
ORGANIZATION TO MEASURE THE ACCURACY OF THEIR PAYMENTS
SSA’S PAYMENT ACCURACY REVIEW PROCESS IS SIMILAR TO
MANY OF THE ORGANIZATIONS CONTACTED, BUT A FEW DIFFERENCES WERE
NOTED
CONCLUSIONS AND RECOMMENDATIONS
The practices used by private and public organizations have
highlighted methods that could help SSA measure payment accuracy
more efficiently and effectively. We recommend that SSA consider
the following changes to its procedures:
Report case payment accuracy rates.
Report a netted and non-netted payment accuracy dollar
rate.
Eliminate life-cycle accuracy rates as a performance measure.
Reconsider sample sizes.
AGENCY COMMENTS
SSA did not agree with our first three recommendations. The
Agency stated that the addition of either a case payment accuracy
rate or a non-netted payment accuracy dollar rate would misrepresent
the extent of errors in the payment process. Additionally, the
Agency believed that the life-cycle accuracy rate presented a more
realistic measurement of the impact of payment errors. SSA noted
in response to our final recommendation that it had recently reduced
sample sizes and would continue to do so as long as statistical
precision was not compromised.
OIG RESPONSE
OIG believes that the addition of both a case payment accuracy
rate and a non-netted payment accuracy dollar rate would provide
a clearer indication of how accurately the Agency disburses payments
to its customers. While we agree that the life-cycling accuracy
rate does show a projected cost to the Agency, we do not believe
that it is an accurate performance measure of current payment accuracy.
Lastly, we appreciate the Agency’s efforts to reduce the sample
sizes for the Stewardship and IDA reviews and encourage consideration
of further reductions.
INTRODUCTION
OBJECTIVE
The objective of this evaluation was to benchmark private
and public business practices in measuring payment accuracy.
BACKGROUND
GPRA, enacted in August 1993, seeks to improve public confidence
by systematically holding Federal agencies accountable for achieving
program results. Agencies must set performance goals, measure performance
against the goals, and report publicly on their performance. GPRA
also calls for agencies to have a strategic plan in place by September
30, 1997. These plans are to contain a discussion of outcome-related
goals and objectives, how the goals and objectives are to be achieved,
and the program evaluations used to establish and revise the goals
and objectives.
GPRA also requires agency heads to submit annual reports
on program performance to the President and Congress beginning
March 31, 2000, and requires the Office of Management and Budget
to require each agency to prepare an annual performance plan which,
beginning with Fiscal Year 1999, will be included in a Federal
Government’s performance plan for the overall budget. The
required annual performance plans must have: 1) objective, quantifiable,
and measurable performance goals; 2) performance indicators to
be used in measuring the relevant outputs, service levels and outcomes;
and 3) a means for comparing, verifying, and validating data.
Within SSA, the Office of Strategic Management (OSM) is the
lead component responsible for developing the Agency`s strategic
and performance plans. The OSM is currently updating the Agency`s
strategic plan and leading the effort to develop the Agency`s
first performance plan. The actual task of developing and presenting
a performance plan to the Agency`s Executive Staff has been
given to an intercomponent work group. At this time, the work group
has completed a draft performance plan that will be presented to
the Executive Staff. Three of the performance measures contained
in the draft are measures of payment accuracy in SSA`s three
programs: Old-Age and Survivors Insurance, Disability Insurance,
and SSI.
In September 1993, the National Performance Review recommended
that all Government agencies develop and publish customer service
standards. President Clinton responded to that recommendation
with Executive Order 12862, which calls for the Federal Government
to be customer-driven. To achieve this goal, each Federal agency
is required to have customer service standards equal to the "best
in business."
Executive Order 12862 requires benchmarking customer service
standards against the best in business. Additionally, GPRA requires
agencies to provide a basis for comparing results and for verifying
and validating measured values. This benchmarking exercise, by
looking at the procedures and performances of other organizations
in monitoring payment accuracy, will address both the Executive
Order and GPRA concerns.
SSA annually conducts payment accuracy reviews of its title
II and title XVI programs, including the IDA reviews and Stewardship
reviews. These reviews are used to determine SSA`s payment
accuracy rate through quality reviews of a selection of sampled
cases.
IDA Reviews
SSA annually conducts IDA reviews for title II initial claims
and title XVI initial claims and FO redeterminations. Initial claims
include original entitlements and related retroactive benefit decisions.
The title II review does not examine payments made to individuals
due to disabilities. The title XVI review does not include reviews
of the initial disability decisions made by State Disability Determination
Services.
The title II review examines initial entitlement decisions
for RSI claims. The RSI program provides benefits to retired workers,
their families, and dependent survivors. SSA processes over 3 million
such claims a year. The review is used to identify and make recommendations
for changes needed to address issues of payment errors. The reviews
develop two payment accuracy rates. There is the initial payment
accuracy rate and a life-cycle accuracy rate, which shows the dollar
accuracy of adjudicative decisions over the expected life of the
beneficiary.
The title II review examines a sample of payments selected
through the following method of probability sampling. Counties
are clustered into groups to minimize travel costs. Each cluster
is assigned a probability of selection based on Regional Office
of Program and Integrity Reviews’ (ROPIR) desire to visit
the counties in that cluster. For example, a cluster with counties
that minimizes a certain ROPIR’s travel costs will be assigned
a higher probability of selection (90 to 95 percent) than the clusters
of counties that would involve higher travel costs. Given the higher
probability of selection, the desired clusters are more likely
to be selected, but all clusters have the potential to be selected.
The cases are chosen from the 48 contiguous States and the District
of Columbia. SSA selects approximately 2,500 cases a year for this
review.
The title XVI IDA review examines payments for the SSI program,
which provides a minimal level of income for the aged, blind, and
disabled with limited income and resources. Each year, SSA processes
approximately 2 million initial SSI claims and 1 million FO redeterminations
with benefit payments totaling over $20 billion annually.
As with the title II review, the title XVI review is used to identify
and correct payment errors. Dollar payment accuracy rates are derived
from the data collected for this review.
In the title XVI IDA review, a stratified cluster design
is used to select the sample. Four strata are used, which are defined
by the travel costs for ROPIR staff. The first strata contains
FOs which involve the least amount of travel expense. Approximately
80 percent of the sample is drawn from the first strata. The remaining
20 percent of the sample is drawn from strata II to IV, which have
higher travel costs. Each year, the title XVI review examines approximately
1,000 initial awards, 1,000 FO redeterminations with a change,
and 1,000 FO redeterminations without a change.
The process used for both the title II and title XVI IDA
reviews are similar. In both cases, quality assurance staff review
a case to determine if the benefit payment amount was based on
accurate information and was calculated correctly. All of the beneficiaries
or recipients in the sample under review are contacted to verify
information within the case file. The title II beneficiaries are
either called or visited, while title XVI recipients are all visited
by a quality assurance staff member.
Stewardship Reviews
SSA uses another review to measure the accuracy of payments
made to its beneficiaries and recipients. The Stewardship reviews
of title II and title XVI programs are not limited to initial cases
or redeterminations. The Stewardship reviews are based on a sample
drawn from the entire universe of RSI beneficiaries who received
a payment and SSI recipients who received a payment. The Stewardship
reviews identify the payment accuracy for overpayments and underpayments.
METHODOLOGY
This review was a benchmarking exercise which examined private
and public business processes. Thirty-one organizations were contacted;
nine were private organizations and the remainder were State or
Federal agencies. All of the organizations made repetitive payments
or monitored organizations that made repetitive payments. An effort
was made to interview only organizations that made repetitive payments
such as those made by SSA. The organizations contacted made payments
to individuals who were retired, disabled, or in need of income
assistance. The organizations provided the following type of payments:
State income assistance, unemployment, State or Federal pensions,
private pensions, private disability, Federal disability, and private
annuity payments. Many of the organizations were identified through
the International Benchmarking Clearinghouse of the American Productivity
and Quality Center, of which SSA is a member. The private businesses
are very well-known entities and could easily be classified as
the best in business.
A detailed discussion guide was used to interview the individuals
responsible for the payment accuracy reviews within their organizations.
The discussion guide addressed the processes used by the organizations
to measure the accuracy of the repetitive payments made to customers,
the steps used to analyze the information reviewed, and the formulas
used to determine payment accuracy rates. The discussion guide
also obtained demographic information about each organization.
Eight of the organizations also submitted reports and other documentation
related to their payment accuracy reviews.
The data from the discussion guides was entered into a data
base and also reviewed manually. Similarities and differences in
the review processes used by organizations were identified. The
practices used by the organizations were also compared to the practices
used by the Office of Program and Integrity Reviews in conducting
the IDA and Stewardship reviews. Part of the analysis was aided
by SAS, a statistical computer software.
Our review was conducted from December 1996 to April 1997.
It was conducted in accordance with the Quality Standards for
Inspections issued by the President`s Council on Integrity
and Efficiency.
FINDINGS
ORGANIZATIONS
ISSUING REPETITIVE PAYMENTS MONITORED THE ACCURACY OF PAYMENTS
We spoke with 28 organizations that issue repetitive payments
and 3 organizations that regulate or monitor organizations that
issue repetitive payments. Of the 31 organizations contacted,
22 were Government agencies and 9 were private businesses. The
Government agencies consisted of State public assistance agencies,
State unemployment departments, State insurance departments, State
and Federal pension divisions, and a Federal disability insurance
division. The private businesses consisted of insurance companies,
financial and investment firms, and the pension divisions of large
organizations.
All of these organizations monitored the accuracy of payments
released to customers. However, they reviewed payment accuracy
with different frequency. Half of the organizations measured the
accuracy of their payments on a monthly basis. This was the most
common time frame reported by the organizations. Other reviews
were conducted on a weekly, quarterly, or yearly basis.
ORGANIZATIONS
USED TWO METHODS TO MEASURE THE ACCURACY OF THEIR PAYMENTS
The organizations contacted reported using two distinct methods
in measuring the accuracy of payments. Two-thirds of the organizations
we spoke with recalculated the amount of the benefit that their
customers should have received and compared it to the amount that
the customers were actually paid. The other third reviewed information
from their computer systems used for making the repetitive payments.
Many of these organizations performed a reconciliation from month
to month with the balances provided by their payment system.
The basic model used to recalculate a customer’s correct
payment amount in order to compare it to the actual payment made
was quite similar for the organizations that used this methodology.
In these organizations, audit or quality control staff manually
recalculated the amount of the payment a customer should have received.
Once this figure was computed, it was compared to the amount of
money already paid to a customer which was provided through the
organization’s computer system or a canceled check.
The organizations that recalculated the amount of a customer’s
payment started with the information available within the customer’s
case folder or computer file. For some organizations, the information
within the file was sufficient to do the recalculation. Many (58
percent), however, verified information that was missing or unclear
in the client’s files. They contacted individuals, such as
employers, bank officials, and school officials, to verify information
that affected the amount a customer was eligible to receive. These
organizations also determined whether the customer was eligible
to receive a payment at all. Almost all of the organizations that
manually recalculated the payment amount due a customer also redetermined
the overall eligibility of the customer.
The organizations that relied on their computer systems to
measure payment accuracy used different methods. Half of the organizations
that relied solely on their computer files performed a reconciliation
of their computer files from month to month. They took the balance
of all payments made for the previous month, added any new payments,
subtracted any terminated payments, and compared that result to
the current month’s balance on their computer system.
Other organizations that relied on their computer systems
to measure payment accuracy examined a few case files to determine
the payments due to the customers. They compared these few figures
to the amounts in the computer system that generated the payments.
They were checking the controls within the computer system to ensure
that it processed payments accurately. One Government organization
sent letters to their customers asking them to verify that they
received the amount shown as payable by the computer payment system.
Private businesses were more likely than Government agencies
to review their computer system than to manually recalculate payments
and compare the recalculation to a payment already made. Two-thirds
of the private businesses reviewed computer files and system controls.
Only 18 percent of the Government agencies used this methodology.
Most of the Organizations Reviewed a Sample of Payments
Eighty-four percent of the organizations reviewed a sample
of payments when measuring their payment accuracy. Most of these
organizations chose a random sample of cases. The size of the samples
differed for each organization, but most of them were equal to
or less than 500 cases. Overall, 70 percent of the samples consisted
of 500 or fewer cases. The remaining samples, for the most part,
were between 1,000 and 2,000 cases. There were 2 organizations
with samples larger than 2,000 cases. One organization, a private
financial company, reported reviewing 133,000 cases when measuring
the accuracy of payments issued to customers.
All of the Government agencies selected a sample of cases
for their payment accuracy reviews. The private businesses were
almost evenly split between reviewing the entire universe or reviewing
a sample of payments. Fifty-six percent of these organizations
reviewed the entire universe of payments and the remaining 44 percent
reviewed only a sample of payments.
Thirty-one percent of the organizations that used a sample
reported excluding some cases from review after they had been chosen
for the sample. Cases were generally excluded due to the inability
of the organizations to contact the customer under review. For
example, organizations excluded cases because the customer refused
to cooperate with the review process. Others were excluded because
the customer could not be located to provide additional information.
Also, customers that moved out of State during the review period
were excluded by some State agencies. Finally, some organizations
excluded cases that had checks sent back uncashed.
Some cases were excluded from review even before a sample
was drawn. The reason a claim would be excluded from review was
that no payment was made. Most of the organizations we spoke with
(81 percent) only reviewed a claim if a payment was made.
Almost half of the organizations that used samples stratified
their samples to ensure that certain subgroups were represented.
The organizations desired to examine specific subgroups such as
recently approved initial claims, geographic regions and counties,
and disability claims. Stratification ensured that organizations
had an adequate number of these type of cases within their samples
to examine and project back to the universe of all payments. Generally,
strata were taken for groups that would have been too small for
analysis if a simple random sample was drawn from the universe.
Amount of Staff Hours Needed to Conduct the Payment
Accuracy Reviews Differed among Organizations Contacted
The amount of staff hours needed to complete a payment accuracy
review varied from organization to organization. One of the organizations
needed only 1 hour to complete its payment accuracy review, while
another organization reported having used 24,000 staff hours. Generally,
organizations that manually recalculated their customers’ authorized
payment amounts and compared those to actual payments sent to the
customers, used more staff hours than organizations that relied
on computer systems to monitor payment accuracy. The former organizations
tended to need hundreds or thousands of staff hours while the latter
organizations reported having used 40 staff hours or less
to complete their payment accuracy reviews.
The method used to measure the accuracy of payments appears
to be related to the number of staff hours needed to complete a
payment accuracy review. Organizations that reviewed computer systems’ controls
needed fewer hours to do so than organizations that manually recalculated
payment amounts. This is true even though they were not necessarily
reviewing smaller samples. Most of the organizations that reviewed
their computer systems to measure payment accuracy examined samples
of hundreds or thousands. Most of the organizations that manually
recalculated payment amounts reviewed samples of under 125 cases.
MOST
ORGANIZATIONS MEASURED BOTH THE AMOUNT OF DOLLARS IN ERROR
AND THE NUMBER OF CASES WITH ERRORS
Most of the organizations contacted (67 percent) reported
having measured both case and dollar errors when they reviewed
payment accuracy. Thirteen percent measured only case errors, 10
percent measured only dollar errors, and 10 percent did not
measure either case or dollar errors. The organizations that did
not measure errors used their computer systems to reconcile the
current month’s balance of payments to the previous month’s
balance. They were looking only to see if the overall balance was
correct and were not concerned with individual dollar or case errors.
One State agency reported that the measurement of dollar
errors could identify which areas were in most need of improvement.
It particularly measured the causes of overpayments, in an attempt
to isolate and correct errors in the payment system. One person
from a private insurance company stated why he thought it was important
to record the number of cases in error. He said that one practice
he found ineffective is, ". . . concealing the errors you
make in the large volume you do. If you only have a $10,000 error
and you paid out a million dollars, you look good. But, you may
have made hundreds of errors and you are really not doing all that
well."
ORGANIZATIONS
TREATED OVERPAYMENTS AND UNDERPAYMENTS DIFFERENTLY
In most of the organizations we spoke with, it was possible
for customers to have both an overpayment, where they were paid
more than they should have, and an underpayment, where they were
paid less than they should have, during the period being reviewed.
A majority of these organizations (52 percent) treated each instance
of an underpayment or overpayment as separate errors. For example,
if a customer was overpaid $100 and also underpaid $100, there
would be a dollar error of $200.
The remaining offices (48 percent) did not look at an overpayment
or underpayment as two separate errors. In these organizations,
if one customer had both an overpayment and an underpayment within
the review period, all overpayment errors were added and underpayment
errors were subtracted to create one net error dollar amount. For
example, if a customer was overpaid $100 and also underpaid $100,
there would be no dollar error since the two figures netted out.
Private businesses were more likely than Government agencies
to report overpayments and underpayments as separate errors. Sixty
percent of private businesses who had customers with both an overpayment
and an underpayment in a single claim reported overpayments and
underpayments as separate errors. Forty-five percent of the Government
agencies reported these types of payments as separate errors.
TWO
TYPES OF PAYMENT ACCURACY RATES WERE USED
All of the organizations that performed manual reviews of
cases in reviewing payment accuracy created payment accuracy rates.
Conversely, all but two of the organizations that relied on a computer
system to review payment accuracy did not compute accuracy rates.
The organizations that computed a payment accuracy rate used two
basic formulas. One of the formulas focused on the amount of dollars
paid in error and the other focused on the number of errors made
in the handling of customers cases.
Eighty-three percent of the organizations that computed accuracy
rates used a dollar payment accuracy rate. The basic formula used
in evaluating the amount of dollars in error was the total amount
of dollars in error divided by the total amount of dollars in the
sample (or universe if the entire universe was being examined).
This formula provided the percentage of dollars in error. This
number subtracted from the number one gave the percentage of dollars
paid correctly.
The organizations contacted had different thresholds on what
constituted a dollar error, but all were low. About half of the
organizations contacted considered a payment in error if it was
one cent different than it should have been. All but one of the
remaining organizations considered a payment in error if it was
one dollar or five dollars different from what the payment should
have been. The last organization had an error threshold of $25.
Some organizations reported calculating more than just the
overall dollar payment accuracy rate. Organizations also looked
at specific types of dollar errors. They examined the percentage
of dollars in the sample that were overpayments and those that
were underpayments separately. The organizations that combined,
or netted, overpayments and underpayments together did not compute
these separate dollar error rates.
Thirty-nine percent of the organizations that computed payment
accuracy rates used a case payment accuracy rate. The basic formula
used in evaluating the number of errors was the total number of
payment errors divided by the total number of cases in the sample
(or universe if the entire universe was being examined). This formula
provided the case error rate. The case error rate subtracted from
the number one yielded the case payment accuracy rate.
Organizations reported the need for specific information
related to the type of errors involved in the process of making
payments to customers. They calculated the percentage of errors
caused by their employees versus the percentage of errors caused
by the customers. (Customers were often required to report personal
information to determine eligibility for payment.) They also examined
the percentage of payments not made in a timely fashion, the percentage
of payments not received by the customer, the percentage of payments
not stopped in time resulting in an overpayment, and the percentage
of cases with incorrect information regardless if the information
caused a payment error or not. These figures were collected to
monitor and improve specific aspects of the payment process.
The number of organizations using each type of payment rate
was not mutually exclusive. Some of the organizations (22 percent)
that used payment accuracy rates reported using both a dollar payment
accuracy rate and a case payment accuracy rate (see graph on next
page). These organizations believed that both rates were necessary
to ensure the proper use of funds and to ensure a high level of
quality customer service.
MOST
ORGANIZATIONS DID NOT PROJECT THE COSTS OF ERRORS
Eighty-three percent of the organizations did not project
how much an error found during the review would have cost the organization
if it had not been corrected. The organizations that reported projecting
such costs did so for varying lengths of time. One organization
projected how much errors cost the organization for each year.
Another organization calculated the cost of errors, but only up
to the time they were found during the review. They went back and
identified when the error began and then calculated the costs for
the organization from that time to the time the error was discovered.
None of the organizations calculated how much errors cost for a
period greater than a year.
MOST
ORGANIZATIONS WERE REQUIRED BY A GOVERNMENTAL OR PROFESSIONAL
ORGANIZATION TO MEASURE THE ACCURACY OF THEIR PAYMENTS
Three-quarters of the organizations contacted were required
by some governmental or professional body to monitor the accuracy
of payments sent to customers. Government agencies were twice as
likely as private businesses to be required to measure their payment
accuracy. Eight-six percent of the Government agencies were required
to conduct payment accuracy reviews, while 44 percent of the private
businesses were required to conduct reviews. While these governmental
or professional bodies required many of the organizations to measure
payment accuracy, not all mandated which process to use.
Sixty-one percent of the organizations required to measure
accuracy had specific procedures to follow that were dictated by
an outside organization. The organizations that mandated procedures
dictated the type of measure to be used, the sample size, and the
procedures to be used during the review. The remaining organizations
that were required to measure payment accuracy were left to their
own discretion on how to conduct their review.
SSA’S
PAYMENT ACCURACY REVIEW PROCESS IS SIMILAR TO MANY OF THE
ORGANIZATIONS CONTACTED, BUT A FEW DIFFERENCES WERE NOTED
As noted in the background section of this report, SSA conducts
payment accuracy reviews for title II and title XVI payments. Similar
to many of the organizations contacted, SSA selects a sample of
payments and manually reviews and recalculates each customer’s
payment amount to compare it to what actually has been paid. Another
similarity is the use of stratification to ensure that certain
subgroups are large enough for analysis in the sample reviewed.
Also, SSA does exclude some cases from review as did some of the
organizations contacted for this study.
SSA Reports Only a Dollar Accuracy Rate
SSA reports on the percentage of dollars paid correctly through
dollar payment accuracy rates. While not all organizations reported
the use of a case payment accuracy rate, three-fourths of the private
businesses that calculated accuracy rates did. Some of the businesses
that used case payment accuracy rates stated that they used this
information to let their staff know how they were performing and
to train them in areas that consistently had errors.
SSA Combines Overpayments and Underpayments as One
Error
In calculating the overall amount of dollars in error, SSA
combines, or nets, overpayments and underpayments within a case
and reports one error dollar amount per case. A majority of the
organizations interviewed (52 percent) who could have claims with
both an overpayment and an underpayment during the review period
and a majority of the private businesses (60 percent) with such
claims did not net such payments. They reported the overpayments
and underpayments in each case as separate errors. While not an
overwhelming majority, many businesses believed it is important
to identify all dollar errors separately rather than netting. These
private businesses reported that identifying such errors separately
provided information that helped improve all aspects of the payment
process.
SSA Projects the Costs of Errors Through "Life-Cycling;" Most
Organizations Do Not Make Projections
SSA uses a process known as "life-cycling" to estimate
the magnitude of an error made during the initial claims process.
The life-cycle accuracy rate reported by SSA in its title II IDA
reviews projects the costs of errors over an actuarial life expectancy
of 17 years. The actuarial projection of 17 years is an average
model based on the type of error, when the error should be corrected
and the estimated life of a beneficiary. Most organizations (83
percent) contacted for this review reported that they do not make
projections on how much errors cost if they went uncorrected. Those
that did project the costs of errors did not project out to the
potential life of the customer.
SSA Reviews Are More Time Consuming and Its Samples
Are Larger Than Other Organizations
SSA’s IDA and Stewardship reviews are more time consuming
than other organizations’ reviews. SSA required approximately
80,000 staff hours to complete the 1995 title XVI IDA review alone.
The average number of staff hours for the organizations reviewed
was 1,963 hours, with a median of 300 hours. The average is much
larger than the median due to 2 large outliers of 8,870 and 24,000 hours.
Regardless, the time needed for one of the IDA reviews is three
times larger than the highest number of hours needed by the organizations
we contacted.
SSA may require more hours because it reviews a larger sample
than most of the organizations contacted. Overall, 70 percent of
our respondents’ samples consisted of 500 or fewer cases.
The remaining samples, for the most part, were between 1,000 and
2,000 cases. There were 2 organizations with samples larger than
2,000 cases. They reviewed samples of 16,000 and 133,000 respectively.
SSA reviews samples of approximately 2,500 and 3,000 cases for
the title II and the title XVI IDA reviews, respectively, and another
1,500 cases for each of the Stewardship reviews.
CONCLUSIONS
AND RECOMMENDATIONS
GPRA was established to ensure that Government agencies have
adequate information to measure and improve their performance.
GPRA seeks to improve Federal program effectiveness by focusing
on results and service quality and to help Federal managers improve
service delivery by providing them with information about program
results and service quality. Additionally, it seeks to improve
public confidence by providing data on program performance.
Benchmarking business practices reveals the processes that
other organizations use and value as part of their operations.
A review of many organizations’ policies and procedures helps
to show the standards currently being used. This information is
valuable as a point of comparison. In its proposed strategic plan,
SSA highlights benchmarking other organization’s practices
as a useful method to achieve its goal to make SSA program management
the best in business.
Our review of 31 organizations’ practices highlighted
the types of information the organizations believe are necessary
and useful in measuring the accuracy of their payments. Their practices
help to establish some standards on the type of performance data
used by organizations to help ensure the provision of correct payment
amounts to individuals. Their measurements of payment accuracy
also identify problems in the payment process.
The practices used by private and public organizations have
highlighted methods that could help SSA measure payment accuracy
more efficiently and effectively. We recommend that SSA consider
the following changes to its procedures.
Report Case Payment Accuracy Rates:
SSA has performance measures that are related to payment accuracy.
One of these measures, providing proper stewardship of funds,
can be measured through SSA’s dollar payment accuracy rate.
A measure of the dollars properly spent provides the public with
an indication of SSA’s ability to properly manage title
II and title XVI funds.
An additional performance measure that could provide a
clear display of the level of service customers are receiving
is a case payment accuracy rate. The use of this performance
measure would provide an indication of SSA’s ability to
meet its goal of providing world-class customer service. Similarly,
the absence of a case payment accuracy rate deprives the Agency
of a useful measurement of payment performance. This is particularly
true since the dollar payment accuracy rate currently does not
give an accurate indication of the number of errors. This is
especially true when the netting of overpayments and underpayments
is used. Netting effectively masks the effect of some errors.
A measurement of the total number of errors can highlight
areas that are in particular need of improvement, or identify
staff who require additional training. While the dollar payment
accuracy rate provides a measure of the Agency’s stewardship
of SSA funds, the case payment accuracy rate would show the Agency’s
ability to provide quality service. Such an indicator would help
assure the public that SSA is providing service that is equal
to the best in business.
Report a Netted and Non-netted Payment Accuracy Dollar
Rate: SSA currently nets, or combines, overpayments
and underpayments within a case and reports one error dollar
amount per case. These dollar amounts are used in calculating
SSA’s dollar payment accuracy rates. This measure does
not indicate the total amount of dollars that were paid incorrectly.
Reporting overpayment and underpayment amounts separately would
provide a clear indication of the total amount of dollars erroneously
paid.
While the use of data with netted dollar errors indicates
the overall amount of dollars that were released appropriately,
it alone does not provide decision makers and the public with
a performance measure that indicates the accuracy of the process
used to make payments. SSA has created performance measures for
the percent of payments without overpayments and those without
underpayments in the Business Plan: Fiscal Years 1998-2002.
These measures provide useful information and should be included
with other payment accuracy performance measures in all publications
reporting on the accuracy of the payment process.
Eliminate Life-Cycle Accuracy Rates as a Performance
Measure: Only 17 percent of the organizations
reported projecting the cost of errors and none of them reported
projecting to the actuarial life of the error. At most, the
amount of an error was projected for the year in which it occurred
or up to the time it was identified. SSA should reconsider
the value of life-cycling as a measurement of performance.
While the life-cycle payment accuracy rate can provide information
on the magnitude of a payment error made today over the projected
life of a claimant, it does not indicate how many dollars were
paid erroneously today or how many errors occurred in the payment
process. Life-cycling information can be useful in indicating
which reasons for errors have the greatest effect over time,
but it is not an accurate performance measure of current payment
accuracy.
Reconsider Sample Sizes: SSA devotes
significantly more staff hours to measure payment accuracy than
other organizations. Sample size may play a role in this. Almost
all of the organizations contacted used smaller samples than
SSA and, accordingly, required less staff hours to complete their
payment accuracy reviews. This is true even though many of these
organizations use methods similar to SSA’s, including contacting
the paid customer during the review. Even given the significant
amount of payments SSA makes, its use of large samples in the
IDA and Stewardship reviews may not be necessary.
SSA has taken action to reduce the sample sizes reviewed,
particularly in the title II and the title XVI IDA reviews. The
samples have been cut approximately in half over the last 2 years.
Even further reductions may be appropriate. The additional precision
gained by large sample sizes reduces significantly once samples
get larger than a few hundred observations, regardless of the
size of the universe. SSA may be able to save resources in this
area by reducing sample sizes.
The data used for SSA’s payment accuracy reviews are
used for other analyses. For example, the data for the Stewardship
reviews are used to determine profiling for SSI redeterminations.
SSA should review all of the needs for the payment accuracy data
and consider any methods that will allow for a reduction in the
samples used for payment accuracy measurement.
AGENCY COMMENTS
SSA did not agree with our first three recommendations. The
Agency stated that the addition of either a case payment accuracy
rate or a non-netted payment accuracy dollar rate would misrepresent
the extent of errors in the payment process. Additionally, the
Agency responded that the life-cycle accuracy rate presented a
more realistic measurement of the impact of payment errors. SSA
noted in response to our final recommendation that it had recently
reduced the sample sizes of both the SSI Stewardship and the SSI
IDA reviews. The Agency stated that it will continue to attempt
to reduce sample sizes as long as statistical precision is not
compromised. (See Appendix B for the full text of the Agency’s
comments.)
OIG RESPONSE
OIG believes that the addition of both a case payment accuracy
rate and a non-netted payment accuracy dollar rate would provide
a clearer indication of how accurately the Agency disburses payments
to its customers. While the current netted dollar rate does measure
proper stewardship of funds, a case payment accuracy rate and a
non-netted payment accuracy rate would measure the actual number
of payment errors and the total amount of dollars in error, respectively.
We believe, as do many of the organizations we spoke with, that
this type of information is necessary to monitor and improve the
payment process. While we agree that the life-cycling accuracy
rate does show a projected cost to the Agency, we do not believe
that it is an accurate performance measure of current payment accuracy.
Lastly, we appreciate the Agency’s efforts to reduce the sample
sizes for the Stewardship and IDA reviews and encourage consideration
of further reductions.
APPENDICES
APPENDIX A
DESCRIPTION OF PARTICIPATING ORGANIZATIONS
Thirty-one organizations were interviewed for this benchmarking
exercise.
Twenty-eight of the organizations issued repetitive payments
to customers and three monitored organizations that issued repetitive
payments to customers. All of the organizations reviewed the accuracy
of payments made to customers.
Twenty-two of the organizations were Government agencies
and nine were private businesses. The organizations were contacted
since they monitored payments that were similar to SSA payments,
i.e., payments were paid due to retirement or disability. The governmental
organizations contacted consisted of State public assistance agencies,
State unemployment departments, State insurance departments, State
and Federal pension agencies, a Federal disability division, and
a national social security organization. The private businesses
consisted of insurance companies, financial and investment firms,
and pension divisions of large organizations. These businesses
were well-known organizations that are equal to the best in business.
Twenty-nine percent of the organizations served customers
on a national level and 19 percent served customers on an international
level. The remaining organizations were State organizations that
served customers within their State. The organizations served from
4,500 to 60,000,000 customers per year. The average number of customers
served was 4,677,763. Most of the organizations (71 percent)
issued payments on a monthly basis. The remaining organizations
issued payments to their customers on a weekly or bi-monthly basis.
Three-quarters of the organizations reported that the amount
of a payment was subject to change after a customer started receiving
their payments. The reasons for changes were generally related
to changes in the customers’ living arrangements, work status,
number of dependents, or health status.
The number of payments released each payment cycle ranged
from 700 to 3,450,193. The average number of payments was 595,666.
The average amount of a payment for each organization ranged from
$75 to $1,184. The mean payment amount for all of the organizations
was $532. The total amount of dollars released per payment cycle
was between $750,000 and $2,400,000,000, with an average of $352,191,426.
APPENDIX C
MAJOR CONTRIBUTORS TO THIS REPORT
Office of the Inspector General
Scott Patterson, Director, Evaluations and Technical Services
Tim Nee, Acting Deputy Director
Arthur Treglia, Senior Auditor
|
www.socialsecurity.gov
