of the Inspector General
Larry G. Massanari
Acting Commissioner of Social Security
Management Advisory Report: Performance Measure Survey of the Percent of 800-Number Calls Handled Accurately (A-08-01-11024)
The attached final report presents the results of our audit. The objective of our planned audit was to assess the reliability of the Social Security Administration’s (SSA) performance data used to measure the percent of 800-number calls handled accurately in Fiscal Year 1999. Achieving accuracy rates of 90 percent for service calls and 95 percent for payment calls were two of the performance goals SSA developed to meet the requirements of the Government Performance and Results Act of 1993.
within 60 days from the date of this memorandum on corrective action taken or
planned on each recommendation. If you wish to discuss the final report, please
call me or have your staff contact Steven L. Schaeffer, Assistant Inspector
General for Audit, at (410) 965-9700.
James G. Huse, Jr.
THE INSPECTOR GENERAL
SOCIAL SECURITY ADMINISTRATION
PERFORMANCE MEASURE SURVEY
OF THE PERCENT OF
MANAGEMENT ADVISORY REPORT
We improve SSA programs and operations and protect them against fraud, waste, and abuse by conducting independent and objective audits, evaluations, and investigations. We provide timely, useful, and reliable information and advice to Administration officials, the Congress, and the public.
The Inspector General Act created independent audit and investigative units, called the Office of Inspector General (OIG). The mission of the OIG, as spelled out in the Act, is to:
Conduct and supervise independent and objective audits and investigations relating to agency programs and operations.
Promote economy, effectiveness, and efficiency within the agency.
Prevent and detect fraud, waste, and abuse in agency programs and operations.
Review and make recommendations regarding existing and proposed legislation and regulations relating to agency programs and operations.
Keep the agency head and the Congress fully and currently informed of problems in agency programs and operations.
To ensure objectivity, the IG Act empowers the IG with:
Independence to determine what reviews to perform.
Access to all information necessary for the reviews.
Authority to publish findings and recommendations based on the reviews.
By conducting independent and objective audits, investigations, and evaluations, we are agents of positive change striving for continuous improvement in the Social Security Administration's programs, operations, and management and in our own office.
the Inspector General
Larry G. Massanari
Acting Commissioner of Social Security
Management Advisory Report: Performance Measure Survey of the Percent of 800-Number Calls Handled Accurately (A-08-01-11024)
The Government Performance and Results Act of 1993 (GPRA), Public Law 103-62, requires the Social Security Administration (SSA) to develop performance indicators that assess the relevant service levels and outcomes of each program activity. GPRA also calls for a description of the means employed to verify and validate the measured values used to report on program performance. The objective of our planned audit was to assess the reliability of data SSA used to measure the following Fiscal Year (FY) 1999 GPRA performance indicator.
Percent of 800-number calls handled accurately:
Payment accuracy – 95.0
Service accuracy – 90.0
The Agency’s Performance and Accountability Report for Fiscal Year 2000 (Accountability Report) contains performance indicators related to its strategic goal of delivering customer-responsive, world-class service. One of the performance indicators cited in the Accountability Report relates to the percent of 800-number calls handled accurately during FY 1999. Specifically, the indicator provides goals of 95 percent and 90 percent accuracy in responding to payment-related and service-related calls, respectively.
To improve service and convenience for SSA’s customers, the Agency implemented a toll-free, 800-number in October 1988 for public use in contacting SSA’s teleservice centers (TSC). Using this number, among other items, callers can request applications for Social Security numbers, Social Security statements, verification of monthly benefit amounts, Medicare information, and information on local field offices and Medicare carriers. In FY 1999, SSA responded to 58.8 million calls through the 800-number, the largest amount of calls served in the number’s history.
To ensure the accuracy of information SSA employees provide to 800-number callers, the Office of Quality Assurance and Performance Assessment (OQA) conducts an evaluation of the national 800-number service. The results of this evaluation serve as the direct source of the 800-number accuracy rates reported in SSA’s Accountability Report.
In performing the 800-number evaluation, OQA uses staff in Regional OQAs (ROQA) to remotely monitor a random sample of 800-number calls every day. Each ROQA samples calls from all 10 regions. OQA assesses call accuracy based on whether teleservice representatives (TSR) and/or SPIKES adhered to national procedures and operating guidelines when responding to callers’ questions. The Agency considers calls to be handled accurately when neither payment errors nor service errors occur in the sample calls.
The SSA National 800 Number Teleservice Evaluation Monitoring Instructions provide explicit definitions and examples of all types of errors that could result when SSA employees respond to 800-number calls. In general, when an SSA response to a call has the reasonable potential to affect payment or eligibility, SSA includes any resulting errors in the payment accuracy rate. The service accuracy rate represents the percentage of calls that are free from errors that do not have a reasonable potential to improperly affect payment or eligibility of benefits. Service errors involve situations that can (1) result in inconveniences to the public, (2) cause additional SSA workloads, or (3) result in situations where information is improperly released from SSA records or reports.
In documenting their evaluations, ROQA staff prepares a separate 800-number teleservice evaluation data entry form for each successfully monitored 800-number call. This form uses alpha and alphanumeric codes to capture call data and error results. ROQAs submit the completed data entry forms to OQA’s Office of Statistics and Special Area Studies, Division of Public Service Evaluation. For each monitored 800-number call an SSA employee handles incorrectly, the ROQA also completes an evaluation feedback form. The monitor describes his/her specific findings on the feedback form.
OQA provides the results of the 800-number evaluations and recommendations for improvements in 800-number service quality to the Associate Commissioner for the Office of Telephone Services (OTS). In its Accountability Report, SSA reported the Agency met its FY 1999 payment accuracy goal with a 95.4 percent rate. However, the Agency did not meet its service accuracy goal, achieving a rate of only 81.8 percent in FY 1999.
SCOPE AND METHODOLOGY
In developing an approach and methodology for our planned audit, we conducted research into OQA’s process for evaluating 800-number accuracy. Specifically, we examined SSA’s Accountability Report, OQA’s FY 1999 evaluation report of 800-number accuracy, and the 800-number evaluation monitoring instructions. In addition, we held discussions with 800-number coordination officials in OQA, the Birmingham TSC, OTS, and the Atlanta ROQA. Furthermore, with the assistance of our statistician, we reviewed OQA’s sampling plan to determine whether it appeared statistically sound. Finally, we reviewed SSA’s methodology for calculating both 800-number accuracy rates. We performed this survey work in Birmingham, Alabama, from September 2000 through January 2001.
Based on our survey results, we determined that we could not accomplish our audit objectives. Accordingly, we ceased further work and are issuing this management advisory report.
We were unable to audit the FY 1999 performance measure because SSA did not maintain adequate source documentation. Specifically, in performing their 800-number evaluations, ROQA personnel did not routinely document the callers’ questions and/or requests. Additionally, unless ROQA personnel identified an error made by a TSR/SPIKE, they did not document SSA personnel’s response to the caller. As a result of this significant gap in documentation, we could not determine whether ROQA personnel properly assessed the accuracy of all calls included in the sample, a key step in our determination of data reliability.
Until SSA provides evidence of 800-number call facts sampled, it cannot meet the (1) GPRA requirement that Agencies verify and validate measured values or (2) requirements outlined in section II of Office of Management and Budget (OMB) Circular A-123, Management Accountability and Control.
During our survey, we concluded that SSA’s sampling plan for this measure appeared statistically sound. However, the Agency’s method for computing the 800-number service accuracy rate was not consistent with the definition reflected in its Annual Performance Plan. The definition stated the service accuracy rate represented SSA’s success in responding to questions not related to payments. However, OQA computed the service accuracy rate based on all calls, including those related to payment. To ensure SSA properly reflects the 800-number accuracy rates, it should either exclude the number of calls related to payment from the service accuracy rate computation or revise the definition of this measure in its Annual Performance Plan.
ROQA STAFF DID NOT ALWAYS DOCUMENT CALL FACTS WHEN ASSESSING THE ACCURACY OF 800-NUMBER RESPONSES
The scope of ROQA’s 800-number review was limited to the verbal information exchanged during the monitored telephone conversation between the TSR/SPIKE and the caller. According to an OQA official, ROQA monitors did not prepare written documentation of the facts of all calls—particularly those that were "simple" in nature, such as the location of a field office. In most cases, the monitors only prepare a written document if the TSR/SPIKE who handled the call made an error. The team leader of the OQA 800-number evaluation told us auditors would have a difficult time ascertaining the nature of the call and how the call was handled from these documents because most monitors write in shorthand or provide sketchy details. We also determined the Agency did not tape record any 800-number calls it received. Accordingly, our ability to recreate the facts of the sampled calls was severely affected.
SSA developed a note-taking form for monitors to record call facts. However, use of this form is not mandatory. When we requested one copy of a completed form from the Atlanta ROQA, the 800-number coordinator told us none of the monitors could provide one because they did not use the form. The coordinator explained the monitors’ knowledge/experience/resources negate the necessity of taking notes.
All the permanent monitors in the Atlanta ROQA who participated in the FY 1999 evaluation were former TSRs with a minimum of 5 years experience evaluating 800-number accuracy. The monitors have instant access to the same resources the TSR/SPIKE uses to provide customer information/assistance. For example, if a caller requests the address of the nearest field office, the monitor accesses the location information data base during the call to assess the accuracy of the TSR/SPIKE’s response. According to the 800-number coordinator in the Atlanta ROQA, the accuracy of many calls can be assessed on-the-spot because the errors the TSR/SPIKE makes are obvious. Also, she agreed that we would not have an audit trail for all calls monitored by the Atlanta ROQA in FY 1999 because the staff only documented the facts of the 800-number calls SSA handled erroneously.
THE SERVICE RATE COMPUTATION WAS NOT CONSISTENT WITH THE DEFINITION
In computing the FY 1999 service accuracy rate reported in the Accountability Report, OQA divided the number of nonpayment-related sample calls free from errors by the total number of sampled calls, including payment-related calls. The OQA 800-number review Team Leader confirmed that the base portion of the computation included calls related to payments. However, in reporting the 800-number accuracy measure, SSA defines service accuracy as "a measure of whether 800 number representatives respond correctly to inquiries related to issues other than eligibility and payment of benefits."
The service accuracy rate reflected in the Accountability Report should correspond to the definition. Therefore, in future 800-number evaluations, we believe OQA should compute the service accuracy rate by dividing the number of sample calls free from service-related errors by the total number of sampled service-related calls (that is, exclude all payment-related calls from the denominator). This change would ensure the two 800-number accuracy rates are comparable. The payment accuracy rate reflected in the Accountability Report is computed by dividing the sample calls free from payment-related errors by the payment universe. If SSA chooses to continue calculating the service-related accuracy rate in the same manner, it should revise the definition of this performance measure in future Annual Performance Plans and reports.
CONCLUSIONS AND RECOMMENDATIONS
As a result of insufficient documentation, we could not assess the reliability of the data SSA used to measure the accuracy of its 800-number service. Through preliminary research, we determined that ROQA personnel who monitored sampled calls did not document caller questions and/or requests or TSR/SPIKE responses. Therefore, we do not have sufficient data on which to base a reliability assessment.
Also, the FY 1999 800-number service accuracy rate does not properly reflect the measure’s definition. SSA included payment-related calls in the service accuracy rate computation, although the definition states service accuracy is a measure of the accuracy of responses to non-payment questions.
For future 800-number evaluations, we recommend that SSA:
SSA provided an alternative to our first recommendation. Specifically, the Agency suggested OIG co-monitor 800-number calls with ROQA personnel when assessing the reliability of the data used to compute this performance measure. SSA stated it had previously considered documenting the details of ROQA-monitored calls; however, the Agency determined it would not be cost-effective to do so and might still produce unreliable results. In response to our second recommendation, SSA agreed to revise the reported definition of the service accuracy rate to provide a clearer description of how it is computed.
We acknowledge the Agency’s determination that the requirements of our first recommendation may not be cost-effective or reliable. Accordingly, we accept the Agency’s proposed alternative and agree to co-monitor a sample of 100 calls during the testing phase of our next review of this performance measure.
James G. Huse, Jr.
Appendix A – Acronyms
Appendix B – Agency Comments
C – OIG Contacts and Staff Acknowledgments
Government Performance and Results Act of 1993
Office of Management and Budget
Office of Quality Assurance and Performance Assessment
Office of Telephone Services
Regional Office of Quality Assurance and Performance Assessment
Social Security Administration
COMMENTS ON THE OFFICE OF THE INSPECTOR GENERAL (OIG) DRAFT MANAGEMENT ADVISORY REPORT – PERFORMANCE MEASURE SURVEY OF THE PERCENT OF 800-NUMBER CALLS HANDLED ACCURATELY (A-08-01-11024)
Revise procedures to require maintaining detailed support of all 800 number calls monitored by Regional Office of Quality Assurance and Performance Assessment Offices (ROQA) to ensure compliance with Government Performance and Results Act of 1993 and Office of Management and Budget (OMB) standards. Such documentation should permit audit assurance of the discussion held between the caller and the TSC/SPIKE.
We disagree. We have considered the suggested expansion of ROQA reviews of the 800 number several times since we began monitoring calls in 1989. Each time we have reached the same conclusion, i.e., not to require a detailed summary of all 800 number calls included in the determination of the reported accuracy rates. ROQA staff are currently required to document all incorrect call situations in order to provide Operations with information for training and corrective action purposes. The ROQA staffs are also instructed to furnish Operations with positive feedback on call situations that are exceptionally well handled.
Written summaries on all correct calls would result in an unwise expenditure of resources. Since just a single ROQA employee monitors each sampled call, the employee could describe the nature of the call in whatever terms he/she wanted. When the employee determined a particular call was "correct," the written summary would basically be written to support that position. Therefore, "correct call" summaries would not provide "evidence" of 800 number call facts that would allow a third party to determine the correctness of ROQA staff’s accuracy determination. Currently, for correct calls, ROQA staff enter in the data base the reason for the call, the program(s) involved and the response given by the 800 number personnel. There is no other information that could be collected for these correct calls that would assist the Agency in its management of the 800 number system. However, for purposes of validating its audit objectives, OIG may wish to consider co-monitoring calls with ROQA analysts. This would allow OIG auditors to assess the accuracy of the same calls to provide a measure of consistency with OQA findings.
Revise the 800-number service accuracy rate computation or definition to ensure the reported results correspond to the definition and the reported payment accuracy rate.
We do not believe a revision of the computation of the service accuracy rate to exclude payment-related calls would provide an accurate measure of service accuracy. Service errors, i.e., major failures in service that have no reasonable potential to affect payment or eligibility, are recorded on payment-related as well as non-payment related calls. For example, 800 number representatives can provide accurate payment-related information, but fail to ask the caller to provide all required identifying information in order to release information from SSA records. Therefore, all 800 number calls (including payment-related calls) should be included in the denominator of the service accuracy computation.
We agree that the definition of service accuracy reflected in the Accountability Report needs to be modified to more clearly state how the service accuracy rate is computed. In future reports, we will revise the definition to reflect that service errors include major service delivery failures that do not have a reasonable potential to improperly affect payment or eligibility and state that service accuracy reflects the percentage of calls free of such failures.
OIG Contacts and Staff Acknowledgments
Kimberly A. Byrd, Acting Director, Operations Audit Division, (205) 801-1605
In addition to those named above:
Grider, Auditor-In-Charge, (205) 801-1608
Ellen Justice, Auditor
Kimberly Beauchamp, Writer/Editor
Brennan Kraje, Statistician
For additional copies of this report, please contact Office of the Inspector General’s Public Affairs Specialist at (410) 966-5998. Refer to Common Identification Number A-08-01-11024.
Overview of the Office of the Inspector General
Office of Audit
The Office of Audit (OA) conducts comprehensive financial and performance audits of the Social Security Administration’s (SSA) programs and makes recommendations to ensure that program objectives are achieved effectively and efficiently. Financial audits, required by the Chief Financial Officers Act of 1990, assess whether SSA’s financial statements fairly present the Agency’s financial position, results of operations, and cash flow. Performance audits review the economy, efficiency, and effectiveness of SSA’s programs. OA also conducts short-term management and program evaluations focused on issues of concern to SSA, Congress, and the general public. Evaluations often focus on identifying and recommending ways to prevent and minimize program fraud and inefficiency.
Office of Executive Operations
OEO supports the OIG by providing information resource management; systems security; and the coordination of budget, procurement, telecommunications, facilities and equipment, and human resources. In addition, this office is the focal point for the OIG’s strategic planning function and the development and implementation of performance measures required by the Government Performance and Results Act. OEO is also responsible for performing internal reviews to ensure that OIG offices nationwide hold themselves to the same rigorous standards that we expect from SSA, as well as conducting investigations of OIG employees, when necessary. Finally, OEO administers OIG’s public affairs, media, and interagency activities, coordinates responses to Congressional requests for information, and also communicates OIG’s planned and current activities and their results to the Commissioner and Congress.
Office of Investigations
The Office of Investigations (OI) conducts and coordinates investigative activity related to fraud, waste, abuse, and mismanagement of SSA programs and operations. This includes wrongdoing by applicants, beneficiaries, contractors, physicians, interpreters, representative payees, third parties, and by SSA employees in the performance of their duties. OI also conducts joint investigations with other Federal, State, and local law enforcement agencies.
Counsel to the Inspector General
The Counsel to the Inspector General provides legal advice and counsel to the Inspector General on various matters, including: 1) statutes, regulations, legislation, and policy directives governing the administration of SSA’s programs; 2) investigative procedures and techniques; and 3) legal implications and conclusions to be drawn from audit and investigative material produced by the OIG. The Counsel’s office also administers the civil monetary penalty program.