A-02-00-10019 Alternate Format

Office of the Inspector General
Larry G. Massanari
Acting Commissioner of Social Security

Inspector General

Performance Measure Review: Reliability of the Data Used to Report 800-Number Performance (A-02-00-10019)

The attached final report presents the results of our audit. Our objective was to determine the reliability of the data used by the Social Security Administration (SSA) in Fiscal Year 1999 for the following performance indicators, and to assess how the indicators will assist SSA to achieve world class service:

Percent of callers who successfully access the 800-number within 5 minutes of their first call.

Percent of callers who get through to the 800-number on their first attempt.

800-number telephone calls handled.

Please comment within 60 days from the date of this memorandum on corrective action taken or planned on each recommendation. If you wish to discuss the final report, please call me or have your staff contact Steven L. Schaeffer, Assistant Inspector General for Audit, at (410) 965-9700.

James G. Huse, Jr.

OFFICE OF

THE INSPECTOR GENERAL

SOCIAL SECURITY ADMINISTRATION

PERFORMANCE MEASURE REVIEW:

RELIABILITY OF THE DATA

USED TO REPORT

800-NUMBER PERFORMANCE

September 2001

A-02-00-10019

AUDIT REPORT

Mission

We improve SSA programs and operations and protect them against fraud, waste, and abuse by conducting independent and objective audits, evaluations, and investigations. We provide timely, useful, and reliable information and advice to Administration officials, the Congress, and the public.

Authority

The Inspector General Act created independent audit and investigative units, called the Office of Inspector General (OIG). The mission of the OIG, as spelled out in the Act, is to:

Conduct and supervise independent and objective audits and investigations relating to agency programs and operations.

Promote economy, effectiveness, and efficiency within the agency.

Prevent and detect fraud, waste, and abuse in agency programs and operations.

Review and make recommendations regarding existing and proposed legislation and regulations relating to agency programs and operations.

Keep the agency head and the Congress fully and currently informed of problems in agency programs and operations.

To ensure objectivity, the IG Act empowers the IG with:

Independence to determine what reviews to perform.

Access to all information necessary for the reviews.

Authority to publish findings and recommendations based on the reviews.

Vision

By conducting independent and objective audits, investigations, and evaluations, we are agents of positive change striving for continuous improvement in the Social Security Administration's programs, operations, and management and in our own office.
Office of the Inspector General

Larry G. Massanari
Acting Commissioner of Social Security

Inspector General

Performance Measure Review: Reliability of the Data Used to Report 800-Number Performance (A-02-00-10019)

The Government Performance and Results Act (GPRA) of 1993 requires the Social Security Administration (SSA) to develop performance indicators that assess the relevant service levels and outcomes of each program activity. GPRA also requires disclosure of the processes used to verify and validate the measured values used to report on program performance. SSA is committed to ensuring the importance of verifying and validating performance measures, and the Office of the Inspector General audits of the performance measures are a means to achieve this. The objective of this audit was to determine the reliability of the data used by SSA in Fiscal Year (FY) 1999 for the following GPRA performance indicators and to assess how the indicators will assist SSA to achieve world class service:

Indicator FY 1999 Goal FY 1999 Performance

Percent of callers who successfully access the 800-number within 5 minutes of their first call 95.0 percent 95.8 percent

Percent of callers who get through to the 800-number on their first attempt 90.0 percent 92.9 percent 800-number telephone calls handled 55.5 million 58.8 million

The within "5-minute" access measures the number of call attempts from a specified telephone number, referred to as individual callers, that select either the automated or live agent option. This does not include callers who reach the 800-number but terminate the call prior to making a selection.

The "first attempt" measure refers to the number of call attempts that reach either the automated or live agent option on the first attempted call of the day, or any subsequent attempt after a previously successful call.

The "calls handled" measure refers to the number of call attempts that select either the automated menu or live agent option and complete their business prior to terminating the call. This does not include callers who reach the 800-number but terminate the call either prior to selecting an option or after waiting in queue for an agent.

BACKGROUND

SSA implemented a national toll-free 800-number telephone service (1-800-SSA-1213) in 1989 as an alternative to field office contact for customers to use when conducting basic Social Security business. Today, there are 3,000 teleservice representatives, also known as live agents, who staff the 800-number system, which is comprised of 37 teleservice centers (TSC). In FY 1999, the 800-number received over 78 million calls, with an average of 1.5 million calls per week.

After accessing the 800-number, callers are prompted to select an option, in English or Spanish and to either select from an automated menu or to select to speak with a live agent. The live agent option is available on business days between the hours of 7 AM and 7 PM local time, and automated service is available daily 24 hours a day. Callers using the automated service encounter multiple menus. During FY 1999, approximately 70 percent of the callers chose to speak with a live agent, while 30 percent chose the automated menu.

A call to the 800-number is routed from the caller’s local network to the AT&T network, that then delivers it to SSA's 800-number network. If a caller selects the live agent option, the call is routed to one of SSA’s 22 Automatic Call Distributors (ACDs). The ACDs operate like computerized switchboards to steer and deliver incoming calls among the 37 TSCs, depending on the day of the week, time of day, and distribution of call volume among the various TSCs. When a live agent is not available to take the call at a TSC, the ACD puts the call in queue.

RESULTS OF REVIEW

While we replicated the "5-minute" and "first attempt" access rates, we cannot opine on their reliability because limitations in the source data do not allow us to determine whether SSA is measuring the performance it intends. Specifically, the data does not identify all individual callers, which is how SSA measures the access rates. While we replicated the "calls handled" measure, we found errors in the measurement process that could affect its reliability. We found SSA lacks formal documentation of the processes used to collect, analyze, and report performance information. Further, we found weaknesses in the access controls applicable to processing and maintaining the 800-number performance measures. We also believe that the definitions used for the 800-number performance measures tend to emphasize access, as opposed to service, which would better address SSA’s goal to achieve world class service.

DATA LIMITATIONS HAVE AN UNKNOWN EFFECT UPON 800-NUMBER ACCESS MEASURES

SSA relies upon the Automatic Number Identification (ANI) file, which is the detailed daily record of all calls made to SSA’s 800-number provided by AT&T, to calculate the "5-minute" and "first attempt" access performance measures. As shown in Appendix D, SSA receives this file daily, and applies daily and monthly data processing applications to develop the access rate measures. Separate systems subsequently capture and report data on calls handled in the automated and live agent options.

The Office of Information Management (OIM) receives the AT&T ANI file of raw telephone call data, and forwards a copy to the Office of Telecommunications and Systems Operations (OTSO). OTSO performs daily and monthly data processing routines to produce the performance measurements. In its daily operations, OTSO edits out two types of call records: (1) calls that hang up after reaching SSA’s 800-number up-front network prompt and (2) calls from non-digital local exchange companies in which the ANI system can not read or record the entire originating telephone number. These calls are edited out because they do not allow SSA to track the experience of individual callers, a condition necessary to the calculation of two of SSA’s 800-number performance measures. After completing the edit checks, OTSO counts and sorts the remaining call records according to various call disposition categories and calculates the daily "5-minute" and "first attempt" performance measures. OTSO also calculates the "5-minute" and "first attempt" performance measures twice in a monthly process: once for the current month and again cumulatively for the FY to date. OTSO maintains summary-level files of the daily performance measure counts and access rates and deletes the original AT&T ANI record.

Applying the automated processing routines used by SSA to calculate the "5-minute" and "first-time" access measures, we replicated the reported FY 1999 measures. However, we found that limitations in the AT&T file prevent accurately measuring the "first attempt" and "5-minute" access rate performance measures. Specifically, while SSA has defined the access measures to be the experience of unique call attempts, referred to as individual callers, telecommunications architecture limits the ability to identify all individual callers. These limitations occur with calls made from certain PBX systems and from nonequal access areas.

PBX Systems Prevent Identification of Individual Calls

Calls made from locations with a PBX system can be recorded in one of three ways on

the AT&T record. All outgoing calls can be assigned:

one general number,

random numbers, or

the individual number from which the call was made.

The first two scenarios prevent the capturing and recording of a caller’s individual extension, and thus the identification of an individual caller. Since SSA’s measurements require identification of individual callers to track "first attempt" and "5-minute" access, these limitations in the recording of originating numbers can affect the reliability of the reported measures. As shown in Figure 1, the inability to identify all individual callers could either overstate or understate actual reported performance.

Figure 1

Potential Effect of Calls from PBX-Based Telephone Systems upon the Access Measurements

Method of Call Routing Through the PBX	Potential Effect	Measures Affected
Single Lead Number: Different calls from different telephone lines served by the PBX system are recorded as one unique number on the AT&T file.	Over reports the number of successful calls because a caller within 5-minutes of a previously unsuccessful caller could be classified as successful. Underreports the number of successful calls to the 800-number because one successful first attempt will prevent any subsequent call from the same number within a 24-hour period being classified successful.	"5-minute" access "first attempt" access
Multiple Numbers: Different calls from the same telephone served by the PBX system can be recorded as different unique telephone numbers on the AT&T file.	Can either over or underreport the number of successful calls to the 800-number depending upon the number of attempts made by callers before successfully reaching SSA.	"5-minute" and "first attempt" access

As Figure 1 shows, the "5-minute" access rate can be overstated if a call is routed through a single lead number because a caller who successfully accesses the number within five minutes of another caller’s unsuccessful access would be counted as a successful call. At the same time, if calls are made from a location that assigns the same number to all calls, the first caller from that location will be identified in order to determine "first attempt" access, and subsequent callers from that location will not be recognized as individual callers. Consequently, if the first caller successfully reaches the 800-number on the first attempt, any other callers assigned that number for that day will not be categorized as a successful "first attempt" caller. The effect of this would be to understate the "first attempt" access rate.

As Figure 1 also shows, when calls are assigned random numbers, the effect could be to inflate or deflate reported performance. For instance, a caller may unsuccessfully access the 800-number on the first nine attempts, but place a successful call on the tenth attempt 10 minutes later. If each attempt were assigned a different originating number, the tenth call would be over reported as a successful "5 minute" access call. Then, if a second caller successfully accessed the 800-number on the caller’s first attempt, assuming no other calls for the day, the successful "first attempt" rate for the day would be underreported. The understatement occurs since SSA would calculate the "first attempt" rate as 2 successful first attempts out of 11 unique callers, or 18 percent, rather than correctly measuring 1 successful first attempt out of 2 unique callers, or 50 percent.

SSA officials did not believe that such calls would materially affect the performance measures, but neither SSA nor AT&T could estimate the extent to which calls originate from PBX systems. Consequently, we could not determine the effect upon the performance measure.

Nonequal Access Exchange Areas Prevent Identification of Individual Calls

Calls made from nonequal access exchanges are not identified on the ANI file as a complete originating telephone number. Therefore, these calls cannot be associated with an individual caller and are excluded from the access performance measure calculations. While the source data does not permit identifying originating numbers or individual callers to be included in the calculations of the access measures, these calls do represent callers that are served. Consequently, excluding them could cause an underreporting of actual performance. SSA staff estimated that these types of calls represent about five percent of all calls. We were unable to determine the effect of including these calls in the measurement calculation.

"Calls Handled" Calculation Errors Could Affect

Reliability of the Performance Measure

"Calls handled" represents a workload measure that is derived monthly through a series of manual and computerized calculations using multiple data sources. We tested the processing routines that produce the "calls handled" measure, and replicated the process SSA used to report the FY 1999 measure. However, we found errors in the processing routine used to calculate the "calls handled" measure that could affect its reliability.

The ANI file records all calls to the 800-number, and identifies whether the caller selected either the automated or live agent menu, or terminated the call prior to selecting an option. Callers choosing the automated menu are tracked daily through the Automated Service Report system, and monthly through the Plasa Reports. Customers who choose the live agent menu are tracked through the Automated Call Distributor system and these calls are processed daily in the Handled Report and monthly in the FTS 2000 Usage Cost Reports. Multiple automated and manual processes that use various data sources are used to develop calls handled by live agents and in the automated menu system, which are combined to report the calls handled measure. As shown in Figure 2, we found errors in these processing routines:

Figure 2

Effect of Processing Errors on "Calls Handled" Performance Measure

Processing Error	Effect of the Processing Error
Calls received in the automated menu system as reported in the Plasa Report include calls that encountered a busy signal.	Overstates the "calls handled" measure
Calls abandoned in the automated menu system as reported in the Handled Report reflect callers that may have selected more than one option within the automated menu. This inflated number is subtracted from actual calls received on the Plasa Report.	Understates the "calls handled" measure

We determined that the cumulative effect of the latter two errors understated the FY 1999 "calls handled" measure by approximately 100,000 calls, or less than one percent. While the effect of these processing errors appears to be minimal during the period audited, the effect of the errors on future periods cannot be determined.

SSA LACKS SUFFICIENT DOCUMENTATION OF THE PERFORMANCE MEASURE PROCESS

We found that SSA lacked sufficient documentation of the processes used to collect, analyze, and report performance indicator data. Further, complete documentation necessary to recreate the performance measure was not available. Office of Management and Budget (OMB) Circular A-123, Management Accountability and Control, states that documentation for transactions, management controls, and other significant events must be clear and readily available for examination. Consequently, agencies should establish a clear methodology for verifying the underlying data and retain the appropriate documentation to enable an audit of the methodology that supports the performance measures.

SSA did not have a comprehensive process map documenting the flow of performance measure data from the receipt of original, unedited 800-number data from an external source, through SSA edit checks and sorts, to the final calculation and reporting of performance measures. At our request, SSA documented the processes by which the performance measure data is collected, analyzed, and reported. However, documenting the process was not only time consuming, it also disclosed that the rationale for the steps in the process was not always known or understood. These processes are depicted in Appendices D and E.

We also found that two different offices within SSA develop 800-number performance information. OTSO produces the GPRA performance measures, and OIM uses the same call data to produce performance data for internal reports. OTSO and OIM perform different edit checks to the data to develop their statistics, which could result in inconsistent information. For instance, OIM deletes calls recorded as complete with a zero call duration on the basis that they represent an illogical record. However, OTSO includes these records in its performance measure calculations. We also found that OTSO had not always accounted for all automated menu calls when developing the "calls handled" measure. For instance, during the month of January 2000, over 60 percent of the calls were erroneously attributed to the live agent option rather than the automated menu. While this did not affect the overall "calls handled" performance measure, which relies upon the monthly Plasa Report, it did distort daily management information on the distribution of 800-number workload between live agent and automated menu options. Additionally, the number of calls attributed daily to the automated menu is overstated because OTSO performs manual adjustments to increase the number of calls received by certain submenu options rather than simply reporting the number of calls. This has the effect of counting a call more than once.

We also found that OTSO erroneously calculates calls abandoned in the live agent menu when reporting monthly data on the Commissioner's Tracking Report. The calls abandoned in the live agent option is taken from the FTS 2000 Usage Cost Report, which includes calls that are abandoned in the automated menu. Consequently, calls abandoned in the live agent menu are overstated. We determined that calls abandoned in the live agent menu were overstated by 298,333 calls, or 6 percent, in FY 1999.

After OTSO calculates the daily performance measures, the original data of individual calls to the 800-number is deleted, and only summary counts are maintained for developing subsequent year-to-date statistics. OIM maintains original call files for the most recent 30 to 60-day period, after which previous files are deleted. Consequently, a complete audit trail is not maintained to allow recalculation of the yearly performance measures. SSA staff did not know why the two units were producing similar performance information, nor why different edit checks were applied.

WEAKNESSES EXIST IN ACCESS CONTROLS OVER PERFORMANCE MEASURE PROCESSING

OMB Circular A-127, Financial Management Systems, requires that a system of internal controls be established to ensure that data used to produce reports is reliable. We found that controls over access to 800-number program files and report listings were weak. SSA stores the files under a dollar sign user identification number that allows unlimited access to the files and sufficient authority for the average user to modify or delete key programs or reported measures without detection. While our field work did not disclose that any unauthorized changes or deletions had occurred, we believe SSA should restrict access to the files to reduce the risk of accidental modification or deletion, and help ensure the validity and completeness of the performance measure.

PERFORMANCE MEASURES EMPHASIZE ACCESS RATHER THAN SERVICE

Figure 3 shows that during FY 1999, SSA reported that 78.7 million calls were placed to the 800-number, of which 6.9 million callers encountered a busy signal prior to reaching the 800-number prompt. Of those making a selection, 13 million terminated the call prior to being served, and 58.8 million completed their business.

Figure 3

Disposition of Calls to the 800-Number

Disposition of Calls FY 1999	Total calls (million)	Percent
Reached the 800-number	78.7	100.0
Encountered a busy signal	6.9	8.8
Discontinued subsequent to being connected to either the live agent or automated option	13.0	16.5
Handled to completion	58.8	74.7

The two access performance measures focus on whether the customer is connected to one of the two menu options, which may or may not result in actually being served. A call meets the performance measure if the caller chooses a live agent or the automated menu, not whether business was conducted or even completed to the customer's satisfaction. No consideration is given to how long the caller must wait to be given information via the automated menu or to speak with a live agent, or whether the caller subsequently hangs up prior to being served. The performance measure for "calls handled," attempts to measure service by only counting callers that actually complete their business through either the automated or live agent option. However, it does not report on the extent of customer service, such as the length of time that a caller is on hold prior to being served.

The February 1999 Office of Quality Assurance and Assessment’s (OQA) semiannual 800-Number Customer Survey (OQA Survey) found that 59 percent of callers who selected a live agent were placed on hold, and that 9 percent of those hung up prior to speaking to a live agent. While the caller is the one who opts to terminate a call prior to completion, the reasons callers terminate a call are not known. Nevertheless, calls terminated prior to completion may negatively impact caller satisfaction. OQA released a report in August 2000 on how well the 800-number automated services met customer needs. Limited information was obtained that showed automated service callers hang up before the message was completed primarily because of a desire to speak with a representative.

The OQA Survey also found that caller satisfaction is greatly affected by access and the extent to which a caller’s issue is handled completely. The OQA Survey reported that overall satisfaction declines as the access rating goes down, and that complete call handling has a profound effect on the overall satisfaction rating. The Survey found that 98 percent of the callers who got through right away and whose call issue was handled completely, either by an automated service or a representative, were satisfied with service overall. However, the Survey further found that callers' satisfaction rating dropped to less than 60 percent when callers remembered having to call more than once and being unable to complete their business.

The September 1999 Social Security Advisory Board report, How the Social Security Administration Can Improve Its Service to the Public, raised the question as to whether the public regards the "5-minute" access rate as an acceptable level of service. The report further suggested that performance measurement should emphasize the percent of calls that conduct business, as well as the percent that achieve access.

CONCLUSIONS AND RECOMMENDATIONS

While we were able to replicate the measures reported by SSA to report on its
800-number performance, limitations in the data do not allow us to conclude as to whether the measures reliably report the performance SSA intends to measure. SSA attempts to measure the experience of each caller, however, SSA does not disclose that data limitations prevent identification of an individual caller in certain scenarios. The lack of formal documentation of the processes used to collect, analyze, and report performance information does not comply with OMB guidance in Circular A-123. Weaknesses in access controls applicable to the data processing used to report performance do not ensure the integrity of the performance data. We believe that the definitions used by SSA for its 800-number performance measures tend to emphasize access, as opposed to service. To ensure greater compliance with GPRA reporting requirements, we recommend that SSA:

Disclose in the Accountability Report and other appropriate documents the data limitation that prevents identifying individual callers from PBXs and nonequal access exchange areas in the calculation of the access measures;

Work with the new contractor for managing the 800-number to quantify the effect upon the access performance measures of the inability to identify individual callers;

Revise the process for calculating calls handled in the automated menu system to exclude calls that receive a busy signal, and to reflect only individual calls, as opposed to options selected;

Formally document the processes by which the 800-number performance measurements are derived;

Ensure that controls exist to prevent unauthorized access to the 800-number performance measure data files;

Consolidate responsibility for 800-number performance reporting within one office, and ensure that appropriate edit checks and processing routines are applied prior to performance measure calculation;

Revise the process for calculating calls abandoned in automation to reflect individual calls; and

Develop indicators to effectively measure the quality of service received by customers served by the 800-number that would more accurately reflect SSA’s performance goal of achieving world class service.

AGENCY COMMENTS

SSA agreed with five (recommendations 1, 4, 5, 6, and 8) of our eight recommendations, and reported that action had already been taken as a result of the transition to a new 800-number contractor to address two others (recommendations 3 and 7). In response to recommendation 2, "Work with the new contractor for managing the 800-number to quantify the effect upon the access performance measures of the inability to identify individual callers," SSA advised that present technology does not permit determining the effect, but that SSA will revisit the issue periodically.

OIG RESPONSE

We are pleased that SSA agreed with our recommendations, and has, or plans to, implement all our suggested improvements. Regarding recommendation 2, "…to quantify the effect upon the access performance measures of the inability to identify individual callers," we agree that SSA should periodically assess the extent to which technology would allow identification of individual callers. Until such time as SSA’s technology is able to do so, SSA should disclose the inability to identify individual callers when reporting the performance measure.

In responding to recommendation 3, "revise the process for calculating calls handled in the automated menu system to exclude calls that receive a busy signal, and to reflect only individual calls, as opposed to options selected," SSA stated that the reports produced by the new 800-number contractor used to calculate the performance measures more accurately reflect the number of calls handled. These reports will identify calls as opposed to options selected. Additionally, the revised method of calculating calls handled will exclude calls that encountered a busy signal, which were previously included.

SSA noted that the planned OQA survey on 800-number callers involved only callers that selected the automated service, and that the report was issued in August 2000.

Our report has been clarified to note this.

James G. Huse, Jr.

Appendices

APPENDIX A - Scope and Methodology

APPENDIX B - Acronyms

APPENDIX C - Agency Comments

APPENDIX D - Flowchart of the 800-Number First-Attempt and 5-Minute

Access Performance Measurements

APPENDIX E- Flowchart of the 800-Number Calls Handled Performance Measure

APPENDIX F- OIG Contacts and Staff Acknowledgements

Appendix A

Scope and Methodology

As part of the Office of the Inspector General’s (OIG) on-going role to verify and validate the Social Security Administration’s (SSA) performance measure reporting, we conducted this audit to assess the reliability of the data used to report the following SSA 800-number performance measurements:

percent of callers who successfully access the 800-number within 5-minutes of their first call;

percent of callers who get through to the 800-number on their "first attempt", and

number of 800-number "calls handled".

To meet our objectives, we documented, based upon analysis of SSA documents and discussion with officials from the Office of Telephone Services (OTS) and the Office of Telecommunications and Systems Operations (OTSO), the infrastructure of the 800-number system, definitions of the 800-number measurements, and the processes for capturing, analyzing, and reporting the data used in the measurements. To obtain an understanding of the 800-number call routing process, we conducted a walk-through of OTS’ Voice Network Control Center, which is responsible for overseeing and routing call traffic. To identify the key internal controls in the data collection and reporting process, we flow-charted the process that collects, analyzes, and reports 800-number data from receipt of AT&T Automatic Number Identification (ANI) data to the reporting of the performance measurements. We also reviewed OIG, the U.S. General Accounting Office, and the Social Security Advisory Board reviews of SSA and other 800-number providers to document any previously reported weaknesses in 800-number operations and performance reporting.

We reviewed and confirmed the program language used by SSA to perform the edit checks on the AT&T ANI file, and calculations of the performance measure access rates. We were unable to identify any prior audits of the AT&T ANI.¹ To confirm the recording of originating telephone numbers on the AT&T ANI record, we placed 300 calls on one day from 20 different numbers around the country. We then traced these calls to that day’s ANI record and confirmed that the call date, call time, originating number, and call disposition had been properly recorded.

Because OTSO does not retain the daily detail call records, but rather maintains summary level data of its daily processing, we performed tests of OTSO’s processing in order to rely upon its summary files. We entered "dummy" records into each of the programs on 2 days to ensure that these records were processed correctly. Using data from the AT&T ANI file, we performed parallel simulation of the control tests and the edit checks applied in the daily access rate calculation programs for the period January 1, 2000 through January 31, 2000, and we replicated the daily measures. These tests provided assurance for us to rely upon the functioning of the edit routines and access rate calculations. After establishing the reliability of OTSO’s daily record calculations, we recalculated the "first attempt" and "5-minute" access rates for Fiscal Year (FY) 1999 using OTSO’s files.

We reviewed and confirmed the program language used by SSA to perform both the daily and the monthly calculation of the calls handled workload. This measure represents the monthly sum of calls handled at both the automated and live agent options. We performed parallel simulation of the automated and manual edit checks and calculations applied to derive the daily consolidated calls handled number for January 2000. We traced the monthly calls handled number for FY 1999 to the source documents. To obtain assurance as to the reasonableness of the breakout between calls handled in each option, we reconciled calls offered in both live agent and automated options to the number of ANI complete calls per the handled report.

Our work was conducted at the OIG New York Field Office and SSA Headquarters in Baltimore, Maryland from November 1999 through October 2000. The entities audited were OTS and OTSO. We conducted our audit in accordance with generally accepted government auditing standards, as applicable to a performance audit.

Appendix B

Acronyms

ACD	Automatic Call Distributor
ANI APP	Automatic Number Identification Annual Performance Plan
FY	Fiscal Year
GPRA	Government Performance and Results Act
OIG	Office of the Inspector General
OIM	Office of Information Management
OMB	Office of Management and Budget
OQA	Office of Quality Assurance and Performance Assessment
OTS	Office of Telephone Services
OTSO	Office of Telecommunications and Systems Operations
SSA	Social Security Administration
TSC	Teleservice Center

Appendix C

Agency Comments

COMMENTS ON THE OFFICE OF INSPECTOR GENERAL (OIG) DRAFT REPORT,

"PERFORMANCE MEASURE REVIEW: RELIABILITY OF THE DATA USED TO REPORT 800-NUMBER PERFORMANCE" (A-02-00-10019)

Recommendation 1

Disclose in the Accountability Report and other appropriate documents the data limitation that prevents identifying individual callers from PBXs and nonequal access exchange areas in the calculation of the access measures.

Comment

We agree and will include the recommended disclosure in future reports.

Recommendation 2

Work with the new contractor for managing the 800-number to quantify the effect upon the access performance measures of the inability to identify individual callers.

Comment

The suggested technology does not presently exist to put this recommendation in place. However, we will keep abreast of available technology and will revisit the issue periodically to determine if technology is available that could assist us in identifying the origin of all calls to the 800 number.

Recommendation 3

Revise the process for calculating calls handled in the automated menu system to exclude calls that receive a busy signal, and to reflect only individual calls, as opposed to options selected.

Comment

This recommendation was addressed with the transition from carrier AT&T to WorldCom. The revised Automated Service Report (ASR) counts individual calls to accurately reflect the Handled Calls count. It should be noted that busy calls have never been a part of the automated system; calls that encounter a busy signal do not enter the automated process.

Recommendation 4

Formally document the processes by which the 800-number performance measurements are derived.

Comment

We concur. The Division of Integrated Telecommunications Management (DITM) will write formal documentation after implementation of the Call Center Network Solution. The target date for completing the performance measurement documentation is late October 2001.

Recommendation 5

Ensure that controls exist to prevent unauthorized access to the 800-number performance measure data files.

Comment

We concur. We will change the file names for the performance measure data set names from dollar sign names to Top Secret protected "AIS" data set names. The data set name change will ensure that the data sets cannot be accessed by any real time or batch request that does not have the correct Top Secret Profile. We expect to institute the new data set names with the first daily run of fiscal year 2002.

Recommendation 6

Comment

We agree. The Office of Systems will assess the appropriate placement of the responsibility for 800 number performance reporting. The responsibility for reporting will be in place in one office by the end of the year.

Recommendation 7

Revise the process for calculating calls abandoned in automation to reflect individual calls.

Comment

As with recommendation 3 above, this recommendation was addressed with the transition to our new carrier, WorldCom. DITM no longer makes manual calculations in the automated abandonment count, as was the case with AT&T. All calls are counted as an individual call in the ASR Primary Report, while additional selections are counted in the ASR Subsequent Report. Each call is counted once.

Recommendation 8

Develop indicators to effectively measure the quality of service received by customers served by the 800-number that would more accurately reflect SSA's performance goal of achieving world class service.

Comment

We agree that indicators should be developed to more accurately reflect our performance goal of providing world class service. The Office of Operations (DCO) is in the process of developing new performance metrics that DCO expects to have in place at the beginning of fiscal year 2003. These new metrics will replace the current performance indicators (i.e., the 5-minute and first time access goals) and will focus on our responsiveness to answering all calls.

Changes to our automated services based on customer input and our own analysis is a fluid process, and we are constantly looking for ways to improve customer access and service over the national 800 number. DCO has conducted focus groups to obtain input on the service provided. On an ongoing basis, DCO and the Office of Quality Assurance and Performance Assessment (OQA) assess the level of satisfaction with service provided over the national 800 number. When appropriate, changes are made to address issues/concerns raised by our callers. For example, we are in the process of making several major enhancements to the national 800 number automated service scripts.

On April 13, 2001, Phase I was implemented to separate touch-tone callers from voice response callers and provide distinct touch-tone or voice response call paths. Callers will hear an English and Spanish touch-tone screening prompt during the main greeting. This change will help ensure that callers reach the option selected.

Phase II is scheduled to be implemented by the end of July 2001. This phase will restructure the current 800 number menus to make them user-friendlier for our callers and add a new automated option to provide information about the Password/Password Request Code applications currently in development. The enhancements planned will provide callers with the ability to hear the list of options or message again, plus the option to return to the main menu. The return to main menu feature will provide callers who complete an automated service application with the opportunity to return to the main menu, where they can select another automated service or, during business hours, choose to speak with an agent.

Phase III is scheduled to be implemented on October 1, 2001. During this phase, we will add several new Password/Password Request Code automated service options which will allow callers to request, change, replace or block their password, hear informational messages about the password process and obtain monthly benefit amount information.

Appendix D

Flowchart of the 800-Number First Attempt
and "5-Minute" Access Performance Measurements

Flow Chart of 800 Number First Attempt and 5-Minute Performance Measurements

Appendix E

Flowchart for the Processing of 800-Number Calls Handled Performance Measure

Appendix F

OIG Contacts and Staff Acknowledgments

OIG Contacts

Frederick C. Nordhoff, Director Financial Management and Performance Monitoring Audit Division (410) 966-6676
Timothy Nee, Deputy Director (212) 264-5295

Acknowledgments

In addition to those named above:

John P. Harrison, Senior Auditor
Jeffrey Shapiro, Senior Analyst
Carol Ann Frost, EDP Auditor
Annette DeRito, Program Analyst

For additional copies of this report, please visit our web site at www.socialsecurity.gov/oig or contact Office of the Inspector General’s Public Affairs Specialist at (410) 966-1375. Refer to Common Identification Number A-02-00-10019.

Overview of the Office of the Inspector General

Office of Audit

The Office of Audit (OA) conducts comprehensive financial and performance audits of the Social Security Administration’s (SSA) programs and makes recommendations to ensure that program objectives are achieved effectively and efficiently. Financial audits, required by the Chief Financial Officers Act of 1990, assess whether SSA’s financial statements fairly present the Agency’s financial position, results of operations, and cash flow. Performance audits review the economy, efficiency, and effectiveness of SSA’s programs. OA also conducts short-term management and program evaluations focused on issues of concern to SSA, Congress, and the general public. Evaluations often focus on identifying and recommending ways to prevent and minimize program fraud and inefficiency.

Office of Executive Operations

The Office of Executive Operations (OEO) supports the Office of the Inspector General (OIG) by providing information resource management; systems security; and the coordination of budget, procurement, telecommunications, facilities and equipment, and human resources. In addition, this office is the focal point for the OIG’s strategic planning function and the development and implementation of performance measures required by the Government Performance and Results Act. OEO is also responsible for performing internal reviews to ensure that OIG offices nationwide hold themselves to the same rigorous standards that we expect from the Agency, as well as conducting employee investigations within OIG. Finally, OEO administers OIG’s public affairs, media, and interagency activities and also communicates OIG’s planned and current activities and their results to the Commissioner and Congress.

Office of Investigations

The Office of Investigations (OI) conducts and coordinates investigative activity related to fraud, waste, abuse, and mismanagement of SSA programs and operations. This includes wrongdoing by applicants, beneficiaries, contractors, physicians, interpreters, representative payees, third parties, and by SSA employees in the performance of their duties. OI also conducts joint investigations with other Federal, State, and local law enforcement agencies.

Counsel to the Inspector General

The Counsel to the Inspector General provides legal advice and counsel to the Inspector General on various matters, including: 1) statutes, regulations, legislation, and policy directives governing the administration of SSA’s programs; 2) investigative procedures and techniques; and 3) legal implications and conclusions to be drawn from audit and investigative material produced by the OIG. The Counsel’s office also administers the civil monetary penalty program.