SOCIAL
SECURITY ADMINISTRATION
PERFORMANCE INDICATOR AUDIT:
800-Number Access
September 2006 A-02-06-16108
AUDIT REPORT
Mission
We improve SSA programs and operations and protect them against fraud, waste, and abuse by conducting independent and objective audits, evaluations, and investigations. We provide timely, useful, and reliable information and advice to Administration officials, the Congress, and the public.
Authority
The Inspector General Act created independent audit and investigative units, called the Office of Inspector General (OIG). The mission of the OIG, as spelled out in the Act, is to:
Conduct and supervise independent and objective audits and investigations
relating to agency programs and operations.
Promote economy, effectiveness, and efficiency within the agency.
Prevent and detect fraud, waste, and abuse in agency programs and operations.
Review and make recommendations regarding existing and proposed legislation
and regulations relating to agency programs and operations.
Keep the agency head and the Congress fully and currently informed of problems
in agency programs and operations.
To ensure objectivity, the IG Act empowers the IG with:
Independence to determine what reviews to perform.
Access to all information necessary for the reviews.
Authority to publish findings and recommendations based on the reviews.
Vision
By conducting independent and objective audits, investigations, and evaluations,
we are agents of positive change striving for continuous improvement in the
Social Security Administration's programs, operations, and management and in
our own office.
SOCIAL SECURITY
MEMORANDUM
Date: September 25, 2006 Refer To:
To: The Commissioner
From: Inspector General
Subject: Performance Indicator Audit: 800-Number Access (A-02-06-16108)
We contracted with PricewaterhouseCoopers, LLP (PwC) to evaluate 15 of the Social Security Administration’s performance indicators established to comply with the Government Performance and Results Act. Attached is the final report presenting the results of one of the performance indicators PwC reviewed. For the performance indicator included in this audit, PwC’s objectives were to:
This report contains the results of the audit for the following indicators:
Please provide within 60 days a corrective action plan that addresses each
recommendation. If you wish to discuss the final report, please call
me or have your staff contact Steven L. Schaeffer, Assistant Inspector General
for Audit, at
(410) 965-9700.
Patrick P. O’Carroll, Jr.
Attachment
MEMORANDUM
Date: September 18, 2006
To: Inspector General
From: PricewaterhouseCoopers, LLP
Subject: Performance Indicator Audit: 800-Number Access (A-02-06-16108)
OBJECTIVE
The Government Performance and Results Act (GPRA) of 1993 requires the Social Security Administration (SSA) to develop performance indicators that assess the relevant service levels and outcomes of each program activity. GPRA also calls for a description of the means employed to verify and validate the measured values used to report on program performance.
Our audit was conducted in accordance with generally accepted government auditing standards for performance audits. For the performance indicators included in this audit, our objectives were to:
(FY) 2005 Performance and Accountability Report (PAR).
BACKGROUND
We audited the following performance indicators as stated in the SSA FY 2005 PAR:
Performance Indicator |
FY 2005 Goal |
FY 2005 Actual Results |
Optimize the Speed in Answering 800-Number Calls |
330 Seconds |
296 Seconds |
Optimize the 800-Number Agent Busy Rate |
10% |
10%* |
* The result reported in the FY 2005 PAR for the Agent Busy Rate (ABR) was 10 percent. However, the ABR actual result for FY 2005 was 9.5 percent. SSA management rounded the result of this indicator to the nearest whole number using the standard rounding convention of rounding down numbers less than or equal to .4 and rounding up numbers greater than or equal to .5.
SSA began using telephone service centers as a mechanism to interface with
the public in the mid-seventies and established the National 800 Number Network
in
October 1988. The Call Center Network Solutions (CCNS) architecture that
is used today for the National 800-Number Network was installed in 2001 to
provide the general public with greater access to SSA programs, such as the
Disability Insurance program, authorized by Title II of the Social Security
Act, and the Supplemental Security
Income program, authorized by Title XVI of the Social Security Act. The
National 800-Number Network was also implemented to provide the public with
the ability to request changes to their individual records (i.e. address, earnings,
etc). Although the general public has a variety of other service options
when obtaining information or conducting business with SSA (i.e. internet,
field offices, etc.), the majority of the customers conduct their business
with SSA via the telephone. In
FY 2005, SSA estimated that 57 million customers would access the National
800-Number Network to conduct their business with the Agency.
When SSA customers dial the National 800-Number Network (800-SSA-1213) and
request agent assistance, they are connected to an SSA agent in 1 of the 44
geographical SSA locations. These locations consist of 36 Teleservice
Centers
(TSC), 6 Program Service Centers (PSC), and 2 components of the SSA Office
of Central Operations (OCO). There
are approximately 4,000 agents at the 36 TSCs and the 2 components of the OCO. These
agents have the primary responsibility of answering the 800-number incoming
calls. During times when the call volume exceeds the agents’ capabilities,
SSA management has the ability to deploy additional personnel trained to act
as agents to supplement the full time TSC personnel. These personnel
(known as "SPIKES") have primary responsibilities outside of the
TSC function. These responsibilities cover a variety of other administrative
duties (i.e. claims processing, collections or mailings). By using the
assistance of "SPIKES", SSA management has the ability to efficiently
operate the National 800-Number Network by reducing the length of time callers
remain in queue, and reducing the number of agent busy messages given to callers. There
are approximately 2,000 "SPIKES" in the 6 PSCs (co-located at the
TSC sites).
The 800-Number Access performance indicators are linked to SSA’s strategic objective to “…improve service with technology.” The strategic objective is linked to the SSA’s strategic goal “…to deliver high quality, citizen-centered service…” which encompasses the Agency’s traditional and electronic services to applicants for benefits, current beneficiaries, and future beneficiaries.
RESULTS OF REVIEW
Overall, we found the nature of the performance indicators to be meaningful GPRA measurements. However, SSA management could not provide PricewaterhouseCoopers, LLP (PwC) access to the systems used to generate the results of the performance indicators, as the systems are owned and operated by a contractor (Verizon). In addition, SSA management was unable to provide adequate documentation that supported their conclusion that the Verizon systems used to generate the performance indicator results were sufficiently controlled to ensure the completeness, accuracy, and validity of the data. As a result, PwC was unable to recalculate, and could not conclude on the accuracy of the results reported in the PAR for the performance indicators included in this report.
Specific to the Average Speed of Answer (ASA) performance indicator, we found the description of the data definition in the PAR to be inaccurate.
Indicator Backgrounds
Optimize the Speed in Answering 800-Number Calls
When an SSA customer calls the SSA National 800-Number Network, he/she is presented with a series of service selection announcements. The callers have the option of conducting their business through automated prompts or via agent assistance. If the customer chooses agent assistance, the vendor supplied Intelligent Contact Manager (ICM) application determines the most efficient routing destination for the call. The ICM considers agent availability, estimated wait time in queue and the number of calls currently in queue before routing calls to an available agent. If there are agents available to handle the incoming call, the call is immediately transferred and answered by an agent without waiting in queue. These calls report a zero ASA and are included in the calculation for the ASA. The ASA indicates how timely the agents and SPIKES are answering the incoming calls to the National 800-Number Network.
When there are no agents available to handle the incoming call, the call is placed in queue and the wait time calculation begins. Once an agent becomes available to handle the call, the system automatically routes the call to the available agent. Once the agent answers the call, the wait time calculation ends.
The Automatic Call Distributor (ACD) systems record the call data. The
call data is downloaded from the ACDs to a Cisco Webview Application every
30 minutes. The Cisco Webview Application is a vendor-supplied application
that maintains and produces reports on the call data recorded by the ACDs on
a real-time basis.
Every 60 minutes, the Office of Telephone Services (OTS) Voice Network
Team (VNT) downloads the system generated reports of the call data from the
Cisco Webview Application into spreadsheets for the recording and tracking
of the hourly and year to date (YTD) ASA calculation. The ACDs automatically
calculate the YTD ASA calculation. By tracking the ASA, SSA is able to
manage the National 800-Number Network through internal forecasting projections.
Performance Indicator Calculation
Average Speed of Answer |
= |
Answer Wait Times of All Calls |
“Answer Wait Times of All Calls” is defined as the cumulative wait
times from calls being placed in queue waiting for agent assistance. “All
Calls Answered by Agents”is defined as the cumulative number of calls
that were answered by agents. For additional details on the calculation
of this performance indicator, please refer to the flowcharts in Appendix C.
Optimize the 800-Number Agent Busy Rate
If the call queues are filled to capacity and there are no available agents to handle the incoming call, the caller receives a busy message. A busy message is a voice announcement that informs the caller that the system is filled to capacity and asks the caller to call back at a later time. The performance indicator measures the ABR based upon the number of calls that receive busy messages.
The vendor supplied ICM application monitors the system capacity to ensure
that all calls are getting through to the National 800-Number Network. When
the system is operating at full capacity and there is no available agent or
queue space, the ICM routes the call to the network Menu Routing System (MRS)
platform which tells the Voice Response Unit (VRU) to play a busy message that
instructs the caller to call back at a later time. The ICM records the
call data for the calls receiving the busy messages. The call data is
downloaded from the ICM to the Cisco Webview Application
every 30 minutes. Every 60 minutes, the OTS VNT downloads the call
data from a Cisco Webview Application into spreadsheets for the recording and
tracking of the ABR. The OTS VNT performs the calculation for the ABR
based upon the call data downloaded from the Cisco Webview application. The
OTS VNT compiles the cumulative daily ABR to calculate the YTD ABR which is
ultimately reported in the PAR.
Performance Indicator Calculation
Agent Busy Rate |
= |
Number of Busy Messages |
“Number of Busy Messages” is defined as the number of blocked calls in the network that generate a busy message to the caller. “Number of Calls Offered to Agents” is defined as the number of callers that requested an agent. For additional details on the calculation of this performance indicator, please refer to the flowcharts in Appendix C.
Findings
Internal Controls and Data Reliability
The ACDs, ICM, and the Cisco Webview Application that recorded the call data for the ASA and ABR calculations were owned and operated by Verizon. SSA could not provide PwC access to perform any internal control tests on the Verizon owned applications. In addition, SSA neither had the ability to conduct systems control reviews, nor was it able to assess the effectiveness of the control environment or thereliability of the data produced from the Verizon systems. Finally, Verizon did not provide SSA with adequate control attestation documentation (such as a Statement of Auditing Standards #70 report) to provide SSA management, or the auditor, assurance that the controls related to the performance indicator data were adequately designed and operating effectively. As a result, PwC was not able to conclude on the reliability of the data that was used to calculate these performance indicator results. SSA management concurred with this finding, and is currently working with Verizon to request that a Statement of Auditing Standards #70 report be provided.
SSA management informed PwC that they had performed extensive testing on the Verizon ACDs and ICMs during the installation of the CCNS architecture for the National 800-Number Network in 2001. Management performed this testing to ensure the Verizon systems were accurately recording all of the pertinent call information; including total call times, ABRs, and caller wait times. However, SSA management stated that the results of this testing were not retained beyond 3 years following installation of the CCNS architecture.
Data Retention
The Cisco Webview Application system generated reports were maintained by
OTS
for 180 days. However, at the time of our request, the call data and
reports had been deleted from the OTS systems. PwC was informed that
the call data that was deleted from the OTS systems were also stored on the
SSA mainframe. However, SSA management stated that the call data maintained
on the SSA mainframe was not saved in a format that could be used by the auditor
in recalculating the performance indicators, and that management was unable
to provide the data to PwC. As a result, PwC was unable to recalculate
the performance indicator results, and could not verify the accuracy of results
presented in the PAR. It should be noted that SSA management is currently
working on a project to save the data to the SSA mainframe in a format that
will be usable by management and auditors.
Accuracy of PAR Presentation and Disclosure
The data definition published in the PAR for the performance indicator, “Optimize the Speed in Answering 800-Number Calls” was inaccurate. The wording in the PAR defined the wait time calculation for the ASA performance indicator as “Wait time begins from the time callers first hear the message that they will be connected with the next available agent, and ends when an agent answers.” However, our testing revealed that the Verizon ACDs were tracking and recording the wait time calculation based upon the time between when the caller first heard their estimated wait time in queue, until anagent answered the call. If the caller did not receive an announcement of their estimated wait time in queue, the ACD recorded the call as going straight to the agent without waiting in queue.
SSA management stated that the ACDs were set to provide the caller with an estimated wait time for those incoming calls that were estimated to be answered by agents in greater than 1 minute. The ACDs automatically estimated the queue wait times based upon agent availability and queue capacity. Our research revealed that this calculation was commonly used in call center measurements. Although PwC recognized that this calculation methodology was valid and in-line with call center industry standards, the definition as currently stated in the PAR did not accurately define how SSA calculated the ASA, and could cause the reader to reach the inappropriate conclusion. SSA management concurred with this finding, and is currently updating the data definition for this indicator.
In addition, we noted that the linkage in the PAR between the ASA performance indicator and the indicator "Percent of individuals who do business with SSA rating the overall service as ‘excellent’, ‘very good’, or ‘good’" could be improved. This performance indicator is a compilation of three separate service delivery options to the public: 800 number service, field office telephone service, and office visits. SSA had made numerous technological enhancements to the National 800-Number Network in efforts to improve the service to the public. As such, there have been “very favorable ratings they received for their courtesy, helpfulness, job knowledge and the clarity of their explanations.” This link would complement the ASA performance indicator by informing the reader that the Agency is committed to continuous improvements, and that 85 percent of the public was satisfied with their interactions with SSA via the various service delivery options. By excluding a reference to this complementary indicator, the reader may not develop a complete understanding of the relatively high level of public satisfaction through interactions with SSA.
RECOMMENDATIONS
For the performance indicators included in this report, we recommend SSA:
Specific to the performance indicator, “Optimize the Speed in Answering 800-Number Calls,” we recommend SSA:
Agency Comments
The Agency agreed with all of the recommendations. The Agency’s comments are included in Appendix D.
Appendices
APPENDIX A – Acronyms
APPENDIX B – Scope and Methodology
APPENDIX C – Process Flowcharts
APPENDIX D – Agency Comments
Appendix A
Acronyms
ABR |
Agent Busy Rate |
ACD |
Automatic Call Distributor |
ASA |
Average Speed of Answer |
CCNS |
Call Center Network Solutions |
FY |
Fiscal Year |
GAO |
Government Accountability Office |
GPRA |
Government Performance and Results Act |
ICM |
Intelligent Contact Manager |
OCO |
Office of Central Operations |
OCSO |
Office of the Chief Strategic Officer |
OTS |
Office of Telephone Services |
PAR |
Performance and Accountability Report |
PSC |
Program Service Center |
PwC |
PricewaterhouseCoopers |
SSA |
Social Security Administration |
TSC |
Teleservice Center |
U.S.C. |
United States Code |
VNT |
Voice Network Team |
YTD |
Year to Date |
Appendix B -
Scope and Methodology
We updated our understanding of the Social Security Administration’s
(SSA) Government Performance and Results Act (GPRA) processes. This was
completed through research and inquiry of SSA management. We also requested
SSA to provide various documents regarding the specific programs being measured
as well as the specific measurement used to assess the effectiveness and efficiency
of the related program.
Through inquiry, observation, and other substantive testing, including testing of source documentation, we performed the following:
As part of this audit, we documented our understanding, as conveyed to us by Agency personnel, of the alignment of the Agency’s mission, goals, objectives, processes, and related performance indicators. We analyzed how these processes interacted with related processes within SSA and the existing measurement systems. Our understanding of the Agency’s mission, goals, objectives, and processes were used to determine if the performance indicators appear to be valid and appropriate given our understanding of SSA’s mission, goals, objectives and processes.
We followed all performance audit standards in accordance with generally accepted government auditing standards. In addition to these steps, we specifically performed the following to test the indicators included in this report:
Optimize The Speed in Answering 800-Number Calls
Optimize The 800-Number Agent Busy Rate
Appendix C - Process Flowchart for 800-Number Access
-- IMAGE OMITTED IN THIS TEXT-ONLY VERSION --
Legend -
Flowchart of Optimize the Speed of Answering 800-Number Calls - Process Flowchart (Data Path)
-- IMAGE OMITTED IN THIS TEXT-ONLY VERSION --
Flowchart of Optimize the Speed in Answering 800-Number Calls – Narrative
Flowchart of Optimize the 800-Number Agent Busy Rate - Process Flowchart (Data Path)
-- IMAGE OMITTED IN THIS TEXT-ONLY VERSION --
Flowchart of Optimize the 800-Number Agent Busy Rate – Narrative
Data reported in the PAR.
Appendix D - Agency Comments
SOCIAL SECURITY
MEMORANDUM 0609-0011261 |
Date: |
September 14, 2006 |
Refer To: S1J-3 |
To: |
Patrick P. O'Carroll, Jr. |
From: |
Larry W. Dye /s/ |
Subject: |
Office of the Inspector General (OIG) Draft Report, "Performance Indicator Audit: 800 Number Access" (A-02-06-16108)--INFORMATION |
We appreciate OIG’s efforts in conducting this review. Our comments on the draft report’s recommendations are attached.
Please let me know if you have any questions. Staff inquiries may be
directed to
Ms. Candace Skurnik, Director, Audit Management and Liaison Staff, at extension
54636.
Attachment:
SSA Response
COMMENTS ON THE OFFICE OF THE INSPECTOR GENERAL’S (OIG) DRAFT
REPORT, “PERFORMANCE INDICATOR AUDIT: 800 NUMBER ACCESS”
(A-02-06-16108)
Thank you for the opportunity to review and provide comments on this draft report.
Recommendation 1
Work with Verizon to request access to systems for verification and/or adequate control attestation documentation (such as a Statement of Auditing Standards #70 report) be provided to SSA management.
Comment
We agree. We will address this issue in the technical specifications of the next procurement, which will be in fiscal year (FY) 2007.
Recommendation 2
Perform ongoing control tests related to the generation, storage, and calculation of performance indicator data. These tests should be designed to ensure that data used to calculate the results of the performance indicators are complete, accurate, valid, and that access to the data is appropriately restricted.
Comment
We agree. We will address this issue in the technical specifications of the next procurement, which will be in FY 2007.
Recommendation 3
Retain all documentation related to the testing procedures and results of those tests of the control environment related to the performance indicator data.
Comment
We agree. We will address this issue in the technical specifications of the next procurement, which will be in FY 2007.
Recommendation 4
Archive the Cisco Webview Application source data used in calculating the
performance indicator results and populating the internal management reports. Additionally,
call data maintained on the SSA mainframe should be kept in a usable format
to provide support for the indicator results reported in the Performance and
Accountability Report (PAR).
Comment
We agree. Currently the Cisco Webview Application source data is already archived on the SSA mainframe. Regarding the need to have the call data maintained in a usable format, we will improve the tracking mechanism used to capture performance data so that third-parties can easily retrieve and analyze data.
Recommendation 5
Specific to the performance indicator, “Optimize the Speed in Answering 800-Number Calls,” revise the performance indicator data definition in the PAR so it more accurately reflects what is being measured specific to when call wait time calculations begin and end.
Comment
We agree. We have revised the definition of “answer wait time” which will be included in the Agency’s FY 2007-2008 Annual Performance Plan and reported in the FY 2007 PAR:
The answer
wait time of all calls divided by the number of all calls answered by
agents - Wait time begins from the time the call is placed in queue and ends
when
an agent answers. Calls that go straight to an agent without waiting
in the queue
have a zero wait time but are included in the average speed of answer (ASA)
calculation. ASA does not include callers who hang up after being in
queue.
Recommendation 6
Specific to the performance indicator, “Optimize the Speed in Answering 800-Number Calls,” include a narrative linkage in the description of this indicator to the “Percent of people who do business with SSA rating the overall service as “excellent,” “very good,” or “good”” performance indicator in the PAR.
Comment
We agree. We will include this linkage in the FY 2006 PAR.
Overview of the Office of the Inspector General
The Office of the Inspector General (OIG) is comprised of our Office of Investigations
(OI), Office of Audit (OA), Office of the Chief Counsel to the Inspector
General (OCCIG), and Office of Resource Management (ORM). To ensure
compliance with policies and procedures, internal controls, and professional
standards, we also have a comprehensive Professional Responsibility and Quality
Assurance program.
Office of Audit
OA conducts and/or supervises financial and performance audits of the Social
Security Administration’s (SSA) programs and operations and makes recommendations
to ensure program objectives are achieved effectively and efficiently. Financial
audits assess whether SSA’s financial statements fairly present SSA’s
financial position, results of operations, and cash flow. Performance
audits review the economy, efficiency, and effectiveness of SSA’s programs
and operations. OA also conducts short-term management and program evaluations
and projects on issues of concern to SSA, Congress, and the general public.
Office of Investigations
OI conducts and coordinates investigative activity related to fraud, waste,
abuse, and mismanagement in SSA programs and operations. This includes
wrongdoing by applicants, beneficiaries, contractors, third parties, or SSA
employees performing their official duties. This office serves as
OIG liaison to the Department of Justice on all matters relating to the investigations
of SSA programs and personnel. OI also conducts joint investigations
with other Federal, State, and local law enforcement agencies.
Office of the Chief Counsel to the Inspector General
OCCIG provides independent legal advice and counsel to the IG on various matters,
including statutes, regulations, legislation, and policy directives. OCCIG
also advises the IG on investigative procedures and techniques, as well as
on legal implications and conclusions to be drawn from audit and investigative
material. Finally, OCCIG administers the Civil Monetary Penalty program.
Office of Resource Management
ORM supports OIG by providing information resource management and systems security. ORM
also coordinates OIG’s budget, procurement, telecommunications, facilities,
and human resources. In addition, ORM is the focal point for OIG’s
strategic planning function and the development and implementation of performance
measures required by the Government Performance and Results Act of 1993.
-- footnotes follow --
Public Law No. 103-62, 107 Stat. 285 (codified as amended in scattered sections of 5 United States Code (U.S.C.), 31 U.S.C. and 39 U.S.C.).
31 U.S.C. § 1115(a)(4).
31 U.S.C. § 1115(a)(6).
Government Accountability
Office (GAO), GAO-03-273G, Assessing Reliability of Computer Processed
Data, October 2002, p. 3.
Social Security Administration Performance and Accountability Report Fiscal Year 2005, p. 80.
The Social Security Act, §§ 201-234, 42 U.S.C. §§ 401-434.
The Social Security Act, §§ 1601-1637, 42 U.S.C. §§ 1381-1383f.
Office of the Inspector General, Social Security Administration, Performance Indicator Audit: Overall Service Rating, A-15-05-15118 (October 2005).
Social Security Administration Performance and Accountability Report Fiscal Year 2005, p. 80.
Social Security Administration Performance and Accountability Report Fiscal Year 2005, p. 79.
Social Security Administration Performance and Accountability Report Fiscal Year 2005, p. 67.
GAO, Social Security Administration: Additional Actions Needed in Ongoing Efforts to Improve 800-Number Service, GAO-05-735 (August 2005), p. 10.
Social Security Administration Performance and Accountability Report Fiscal Year 2005, p. 79.
American Institute of Certified Public Accountants, AU 324 Service Organizations, Statement on Auditing Standards No. 70 is the authoritative guidance that allows service organizations to disclose their control activities and processes to the organizations that use their services and the organizations’ auditors in a uniform reporting format.
Social Security Administration Performance and Accountability Report Fiscal Year 2005, p. 80.
Social Security Administration Performance and Accountability Report Fiscal Year 2005, p. 83.
Ibid.