Office of the Inspector General
Larry G. Massanari
Acting Commissioner of Social Security

Inspector General

Management Advisory Report: Performance Measure Survey of the Percent of 800-Number Calls Handled Accurately (A-08-01-11024)

The attached final report presents the results of our audit. The objective of our planned audit was to assess the reliability of the Social Security Administration’s (SSA) performance data used to measure the percent of 800-number calls handled accurately in Fiscal Year 1999. Achieving accuracy rates of 90 percent for service calls and 95 percent for payment calls were two of the performance goals SSA developed to meet the requirements of the Government Performance and Results Act of 1993.

Please comment within 60 days from the date of this memorandum on corrective action taken or planned on each recommendation. If you wish to discuss the final report, please call me or have your staff contact Steven L. Schaeffer, Assistant Inspector General for Audit, at (410) 965-9700.

James G. Huse, Jr.

MANAGEMENT ADVISORY REPORT

Office of the Inspector General
Larry G. Massanari
Acting Commissioner of Social Security

Inspector General

Management Advisory Report: Performance Measure Survey of the Percent of 800-Number Calls Handled Accurately (A-08-01-11024)

The Government Performance and Results Act of 1993 (GPRA), Public Law 103-62, requires the Social Security Administration (SSA) to develop performance indicators that assess the relevant service levels and outcomes of each program activity. GPRA also calls for a description of the means employed to verify and validate the measured values used to report on program performance. The objective of our planned audit was to assess the reliability of data SSA used to measure the following Fiscal Year (FY) 1999 GPRA performance indicator.

Percent of 800-number calls handled accurately:

To improve service and convenience for SSA’s customers, the Agency implemented a toll-free, 800-number in October 1988 for public use in contacting SSA’s teleservice centers (TSC). Using this number, among other items, callers can request applications for Social Security numbers, Social Security statements, verification of monthly benefit amounts, Medicare information, and information on local field offices and Medicare carriers. In FY 1999, SSA responded to 58.8 million calls through the 800-number, the largest amount of calls served in the number’s history.

To ensure the accuracy of information SSA employees provide to 800-number callers, the Office of Quality Assurance and Performance Assessment (OQA) conducts an evaluation of the national 800-number service. The results of this evaluation serve as the direct source of the 800-number accuracy rates reported in SSA’s Accountability Report.

In performing the 800-number evaluation, OQA uses staff in Regional OQAs (ROQA) to remotely monitor a random sample of 800-number calls every day. Each ROQA samples calls from all 10 regions. OQA assesses call accuracy based on whether teleservice representatives (TSR) and/or SPIKES adhered to national procedures and operating guidelines when responding to callers’ questions. The Agency considers calls to be handled accurately when neither payment errors nor service errors occur in the sample calls.

The SSA National 800 Number Teleservice Evaluation Monitoring Instructions provide explicit definitions and examples of all types of errors that could result when SSA employees respond to 800-number calls. In general, when an SSA response to a call has the reasonable potential to affect payment or eligibility, SSA includes any resulting errors in the payment accuracy rate. The service accuracy rate represents the percentage of calls that are free from errors that do not have a reasonable potential to improperly affect payment or eligibility of benefits. Service errors involve situations that can (1) result in inconveniences to the public, (2) cause additional SSA workloads, or (3) result in situations where information is improperly released from SSA records or reports.

In documenting their evaluations, ROQA staff prepares a separate 800-number teleservice evaluation data entry form for each successfully monitored 800-number call. This form uses alpha and alphanumeric codes to capture call data and error results. ROQAs submit the completed data entry forms to OQA’s Office of Statistics and Special Area Studies, Division of Public Service Evaluation. For each monitored 800-number call an SSA employee handles incorrectly, the ROQA also completes an evaluation feedback form. The monitor describes his/her specific findings on the feedback form.

OQA provides the results of the 800-number evaluations and recommendations for improvements in 800-number service quality to the Associate Commissioner for the Office of Telephone Services (OTS). In its Accountability Report, SSA reported the Agency met its FY 1999 payment accuracy goal with a 95.4 percent rate. However, the Agency did not meet its service accuracy goal, achieving a rate of only 81.8 percent in FY 1999.

In developing an approach and methodology for our planned audit, we conducted research into OQA’s process for evaluating 800-number accuracy. Specifically, we examined SSA’s Accountability Report, OQA’s FY 1999 evaluation report of 800-number accuracy, and the 800-number evaluation monitoring instructions. In addition, we held discussions with 800-number coordination officials in OQA, the Birmingham TSC, OTS, and the Atlanta ROQA. Furthermore, with the assistance of our statistician, we reviewed OQA’s sampling plan to determine whether it appeared statistically sound. Finally, we reviewed SSA’s methodology for calculating both 800-number accuracy rates. We performed this survey work in Birmingham, Alabama, from September 2000 through January 2001.

Based on our survey results, we determined that we could not accomplish our audit objectives. Accordingly, we ceased further work and are issuing this management advisory report.

We were unable to audit the FY 1999 performance measure because SSA did not maintain adequate source documentation. Specifically, in performing their 800-number evaluations, ROQA personnel did not routinely document the callers’ questions and/or requests. Additionally, unless ROQA personnel identified an error made by a TSR/SPIKE, they did not document SSA personnel’s response to the caller. As a result of this significant gap in documentation, we could not determine whether ROQA personnel properly assessed the accuracy of all calls included in the sample, a key step in our determination of data reliability.

Until SSA provides evidence of 800-number call facts sampled, it cannot meet the (1) GPRA requirement that Agencies verify and validate measured values or (2) requirements outlined in section II of Office of Management and Budget (OMB) Circular A-123, Management Accountability and Control.

During our survey, we concluded that SSA’s sampling plan for this measure appeared statistically sound. However, the Agency’s method for computing the 800-number service accuracy rate was not consistent with the definition reflected in its Annual Performance Plan. The definition stated the service accuracy rate represented SSA’s success in responding to questions not related to payments. However, OQA computed the service accuracy rate based on all calls, including those related to payment. To ensure SSA properly reflects the 800-number accuracy rates, it should either exclude the number of calls related to payment from the service accuracy rate computation or revise the definition of this measure in its Annual Performance Plan.

The scope of ROQA’s 800-number review was limited to the verbal information exchanged during the monitored telephone conversation between the TSR/SPIKE and the caller. According to an OQA official, ROQA monitors did not prepare written documentation of the facts of all calls—particularly those that were "simple" in nature, such as the location of a field office. In most cases, the monitors only prepare a written document if the TSR/SPIKE who handled the call made an error. The team leader of the OQA 800-number evaluation told us auditors would have a difficult time ascertaining the nature of the call and how the call was handled from these documents because most monitors write in shorthand or provide sketchy details. We also determined the Agency did not tape record any 800-number calls it received. Accordingly, our ability to recreate the facts of the sampled calls was severely affected.

SSA developed a note-taking form for monitors to record call facts. However, use of this form is not mandatory. When we requested one copy of a completed form from the Atlanta ROQA, the 800-number coordinator told us none of the monitors could provide one because they did not use the form. The coordinator explained the monitors’ knowledge/experience/resources negate the necessity of taking notes.

All the permanent monitors in the Atlanta ROQA who participated in the FY 1999 evaluation were former TSRs with a minimum of 5 years experience evaluating 800-number accuracy. The monitors have instant access to the same resources the TSR/SPIKE uses to provide customer information/assistance. For example, if a caller requests the address of the nearest field office, the monitor accesses the location information data base during the call to assess the accuracy of the TSR/SPIKE’s response. According to the 800-number coordinator in the Atlanta ROQA, the accuracy of many calls can be assessed on-the-spot because the errors the TSR/SPIKE makes are obvious. Also, she agreed that we would not have an audit trail for all calls monitored by the Atlanta ROQA in FY 1999 because the staff only documented the facts of the 800-number calls SSA handled erroneously.

In computing the FY 1999 service accuracy rate reported in the Accountability Report, OQA divided the number of nonpayment-related sample calls free from errors by the total number of sampled calls, including payment-related calls. The OQA 800-number review Team Leader confirmed that the base portion of the computation included calls related to payments. However, in reporting the 800-number accuracy measure, SSA defines service accuracy as "a measure of whether 800 number representatives respond correctly to inquiries related to issues other than eligibility and payment of benefits."

The service accuracy rate reflected in the Accountability Report should correspond to the definition. Therefore, in future 800-number evaluations, we believe OQA should compute the service accuracy rate by dividing the number of sample calls free from service-related errors by the total number of sampled service-related calls (that is, exclude all payment-related calls from the denominator). This change would ensure the two 800-number accuracy rates are comparable. The payment accuracy rate reflected in the Accountability Report is computed by dividing the sample calls free from payment-related errors by the payment universe. If SSA chooses to continue calculating the service-related accuracy rate in the same manner, it should revise the definition of this performance measure in future Annual Performance Plans and reports.

As a result of insufficient documentation, we could not assess the reliability of the data SSA used to measure the accuracy of its 800-number service. Through preliminary research, we determined that ROQA personnel who monitored sampled calls did not document caller questions and/or requests or TSR/SPIKE responses. Therefore, we do not have sufficient data on which to base a reliability assessment.

Also, the FY 1999 800-number service accuracy rate does not properly reflect the measure’s definition. SSA included payment-related calls in the service accuracy rate computation, although the definition states service accuracy is a measure of the accuracy of responses to non-payment questions.

For future 800-number evaluations, we recommend that SSA:

1. Revise procedures to require maintaining detailed support of all 800-number calls monitored by ROQAs to ensure compliance with GPRA and OMB standards. Such documentation should permit audit assurance of the discussion held between the caller and the TSC/SPIKE.
2. Revise the 800-number service accuracy rate computation or definition to ensure the reported results correspond to the definition and the reported payment accuracy rate.

SSA provided an alternative to our first recommendation. Specifically, the Agency suggested OIG co-monitor 800-number calls with ROQA personnel when assessing the reliability of the data used to compute this performance measure. SSA stated it had previously considered documenting the details of ROQA-monitored calls; however, the Agency determined it would not be cost-effective to do so and might still produce unreliable results. In response to our second recommendation, SSA agreed to revise the reported definition of the service accuracy rate to provide a clearer description of how it is computed.

We acknowledge the Agency’s determination that the requirements of our first recommendation may not be cost-effective or reliable. Accordingly, we accept the Agency’s proposed alternative and agree to co-monitor a sample of 100 calls during the testing phase of our next review of this performance measure.

James G. Huse, Jr.

Appendices

Appendix A

Acronyms

In addition to those named above:

For additional copies of this report, please contact Office of the Inspector General’s Public Affairs Specialist at (410) 966-5998. Refer to Common Identification Number A-08-01-11024.

The Office of Audit (OA) conducts comprehensive financial and performance audits of the Social Security Administration’s (SSA) programs and makes recommendations to ensure that program objectives are achieved effectively and efficiently. Financial audits, required by the Chief Financial Officers Act of 1990, assess whether SSA’s financial statements fairly present the Agency’s financial position, results of operations, and cash flow. Performance audits review the economy, efficiency, and effectiveness of SSA’s programs. OA also conducts short-term management and program evaluations focused on issues of concern to SSA, Congress, and the general public. Evaluations often focus on identifying and recommending ways to prevent and minimize program fraud and inefficiency.

OEO supports the OIG by providing information resource management; systems security; and the coordination of budget, procurement, telecommunications, facilities and equipment, and human resources. In addition, this office is the focal point for the OIG’s strategic planning function and the development and implementation of performance measures required by the Government Performance and Results Act. OEO is also responsible for performing internal reviews to ensure that OIG offices nationwide hold themselves to the same rigorous standards that we expect from SSA, as well as conducting investigations of OIG employees, when necessary. Finally, OEO administers OIG’s public affairs, media, and interagency activities, coordinates responses to Congressional requests for information, and also communicates OIG’s planned and current activities and their results to the Commissioner and Congress.

The Office of Investigations (OI) conducts and coordinates investigative activity related to fraud, waste, abuse, and mismanagement of SSA programs and operations. This includes wrongdoing by applicants, beneficiaries, contractors, physicians, interpreters, representative payees, third parties, and by SSA employees in the performance of their duties. OI also conducts joint investigations with other Federal, State, and local law enforcement agencies.

The Counsel to the Inspector General provides legal advice and counsel to the Inspector General on various matters, including: 1) statutes, regulations, legislation, and policy directives governing the administration of SSA’s programs; 2) investigative procedures and techniques; and 3) legal implications and conclusions to be drawn from audit and investigative material produced by the OIG. The Counsel’s office also administers the civil monetary penalty program.