MEMORANDUM

Date: January 23, 2003

To: The Commissioner

From: Inspector General

Subject: Performance Indicator Audit: Electronic Access (A 15-02-11083)

We contracted with PricewaterhouseCoopers (PwC) to evaluate the data used to measure 18 of the Social Security Administration's (SSA) performance indicators established to comply with the Government Performance and Results Act (GPRA). The attached final report presents the results of three of the performance indicators PwC reviewed. The objective of this audit was to assess the reliability of the data used to measure the following Fiscal Year 2002 GPRA performance indicators:

1. Percent of States with which SSA has electronic access to Human Services and Unemployment Information.

2. Percent of States with which SSA has electronic access to Vital Statistics and other material information.

3. Milestones/deliverables demonstrating progress in increasing electronic access to information held by other Federal agencies, financial institutions, and medical providers.

Please comment within 60 days from the date of this memorandum on corrective action taken or planned on each recommendation. If you wish to discuss the final report, please call me or have your staff contact Steven L. Schaeffer, Assistant Inspector General for Audit, at (410) 965-9700.

James G. Huse, Jr.

OFFICE OF
THE INSPECTOR GENERAL

SOCIAL SECURITY ADMINISTRATION

PERFORMANCE INDICATOR
AUDIT:
ELECTRONIC ACCESS

January 2003

A-15-02-11083

AUDIT REPORT

MEMORANDUM

To: Office of the Inspector General

From: PricewaterhouseCoopers LLP

Date: January 2, 2003

Subject: Performance Indicator Audit: Electronic Access (A-15-02-11083)

The Government Performance and Results Act (GPRA) of 1993 requires the Social Security Administration (SSA) to develop performance indicators that assess the relevant service levels and outcomes of each program activity set forth in its budget. GPRA also calls for a description of the means employed to verify and validate the measured values used to report on program performance. The objective of this audit was to assess the reliability of the data used to measure the following Fiscal Year (FY) 2002 GPRA performance indicators:

Performance Indicator FY 2002 Goal

1. Percent of States with which SSA has electronic access to Human Services (HS) and Unemployment Information (UI).
68%
2. Percent of States with which SSA has electronic access to Vital Statistics (VS) and other material information. 14%
3. Milestones/deliverables demonstrating progress in increasing electronic access to information held by other Federal Agencies, financial institutions, and medical providers. See background section of this report.
See Appendix A for a description of the audit scope and methodology.

BACKGROUND

SSA offers retirement and long-term disability programs to the general public. Old-Age, Survivors, and Disability Insurance (OASDI) is authorized under title II of the Social Security Act. Through the OASDI program, eligible workers and sometimes their family receive monthly benefits if they retire at an appropriate age or are found to have a disability that either prevents them from engaging in substantial gainful activity for at least 12 months or can be expected to result in death. Supplemental Security Income (SSI) is authorized under title XVI of the Act and provides monthly payments to aged and disabled individuals based on financial need and medical requirements.

SSA requires data from a variety of sources to process title II and title XVI claims accurately. Online access to key sources of external data helps improve services and efficiency, decrease administrative costs, and reduce or prevent overpayments. SSA developed three performance indicators in support of this strategy. The first two indicators address increasing online access to various State agencies' records. The third indicator addresses access to Federal agencies, medical providers, and financial institutions.

Performance indicator #1 measures the percentage of State agencies with which SSA has established access to online HS and UI records. SSA calculates this by dividing the number of HS or UI agencies that are accessible electronically by 100 (1 HS and 1 UI agency per State). Performance indicator #2 measures the percentage of agencies with which SSA has access to online VS records. This is calculated by dividing the number of VS agencies SSA can access electronically by 50 (1 agency per State).

"SSA's Access to State Records Online" (SASRO) is an intranet-based SSA report used to track and report the progress of this initiative. Each time a new connection is made to a State agency, the report is updated to reflect the change. SSA uses the report as the data source for performance indicators #1 and #2.

SSA is also working with the medical and financial community to develop a solution for accessing and transferring data electronically. For FY 2002, SSA defined several milestones and deliverables to increase electronic access to this information. Performance indicator #3 addresses the completion of these goals for FY 2002. The goals include:

1. The completion of the California Electronic Medical Evidence (EME)/Public Key Infrastructure (PKI) pilot.
2. Analysis of three alternatives for Internet transmission of medical information.
3. Development of an implementation plan as it relates to the EME pilot.
4. Begin the project with financial institutions to check their records to determine applicants/recipients eligibility for benefits by publishing final regulations, preparing a statement of work, and developing a schedule for the pilot.

Completion of the first three goals will assist SSA in building the foundation for exchanging electronic information with the medical community. The fourth goal will establish the basis for obtaining online access to records held by financial institutions.
Appendix B provides a flowchart and description for the process of establishing electronic access to State agencies.

RESULTS OF REVIEW

We evaluated the processes and controls that support all three FY 2002 performance indicators. We also reviewed and confirmed the accuracy of the underlying data for performance indicators #1 and #2. Additionally, we reviewed the work products and controls related to the goals listed in performance indicator #3. We determined the data for performance indicator #1 was not reliable based on our evaluation of the information we obtained from SSA and third parties. We also determined that performance indicator #1 was not calculated correctly. We determined the underlying data for performance indicator #2 was reliable, although it was not calculated correctly. We also noted that the management controls related to these performance indicators were inadequate.

We found that the goals listed in performance indicator #3 were partially completed for FY 2002. SSA completed the first three goals but did not complete the goal to "Begin the project with financial institutions to check their records to determine applicants/recipients eligibility for benefits by publishing final regulations, preparing a statement of work, and developing a schedule for the pilot".

We determined if each performance indicator was an appropriate GPRA measure. We found that while performance indicators #1 and #2 are quantitative, they are not outcome or output oriented as prescribed by GPRA. However, performance indicators #1 and #2 measure progress in completing a key SSA initiative. Additionally, they address several Government-wide initiatives including the Government Paperwork Elimination Act and the President's Management Agenda item for expanding electronic Government. Similarly, performance indicator #3 addresses these issues although it does not measure outputs or outcomes and is neither qualitative nor quantitative. Office of Management and Budget (OMB) Circular No. A-11, Part 6: Preparation and Submission of Strategic Plans, Annual Performance Plans, and Annual Program Performance Reports, section 210.5 states that "…performance goals that represent milestones in achieving the general goals of a strategic plan may be appropriate." We therefore found that all three performance indicators were appropriate measures for the Annual Performance Plan (APP).

PERFORMANCE INDICATOR #1 DATA WAS NOT RELIABLE

The SASRO report states that SSA has electronic access to 34 HS and 29 UI agencies. We tested this data by examining 31 HS and 29 UI legal agreements and sending third party confirmation requests to all State agencies listed in the SASRO report. We note that SSA could not provide legal agreements for three HS agencies.
Based on our tests, we determined that the SASRO report overstated the number of HS agencies by one. Pennsylvania's HS informed us that SSA no longer has access to its data despite being included in the SASRO report. In addition, nine State agencies did not reply to our confirmation request and we were not able to confirm or refute their inclusion in the SASRO report.

PERFORMANCE INDICATOR #2 DATA WAS RELIABLE

The SASRO report states that SSA has electronic access to five VS agencies. We examined the legal agreements and obtained independent third party confirmations for each VS agency. Our audit substantiated SSA's SASRO report regarding the number of VS agencies with which SSA has established electronic access.

PERFORMANCE INDICATORS #1 AND #2 RESULTS WERE CALCULATED INCORRECTLY

SSA defines performance indicator #1 as the "percent of State HS and UI agencies from which data are available online out of a total of 100 agencies (i.e., 50 HS and 50 UI agencies)". However, SSA includes the District of Columbia (DC) HS in its SASRO report. Based on SSA's calculation, counting agencies from DC can potentially result in a score greater than 100 percent for performance indicator #1. To take credit for electronic access to DC, SSA must change the denominator for performance indicator #1 from 100 to 102. This would change the reported data for this performance indicator from 63 percent to 62 percent.

Similarly, performance indicator #2 is defined as the "percent of State VS agencies from which data are available online out of a total of 50 agencies." This number should actually be 53 to account for DC, Virgin Islands (VI), and Puerto Rico (PR). This would change the reported data for this performance indicator from 10 percent to 9 percent.

MANAGEMENT CONTROLS FOR PERFORMANCE INDICATORS #1 AND #2 WERE INADEQUATE

We evaluated key control points in SSA's process to establish electronic access with State agencies. We found that SSA was missing three State agency legal agreements at SSA Headquarters in Baltimore, Maryland. In addition, we found that the regional data coordinators did not communicate updates to SSA Headquarters on a timely basis. Our audit found that SSA did not have electronic access to 1 of the 68 State agencies listed in the SASRO report. We investigated this issue and determined that Pennsylvania's HS terminated the agreement with SSA in early 2002 due to account inactivity. SSA is currently working with Pennsylvania's HS to re-establish the connection with a new agreement. The current SASRO report did not reflect this update.

We were also unable to confirm whether SSA had electronic access to nine State agencies' data. OMB Circular No. A-123, Management Accountability and Control, states at section II, page 6, "The documentation for transactions, management controls, and other significant events must be clear and readily available for examination." We found that SSA regional coordinators did not maintain an up-to-date list of contacts at the State agencies and could not provide an alternate means of verification.

PERFORMANCE INDICATOR #3 GOALS WERE PARTIALLY COMPLETED FOR FY 2002

We evaluated SSA's completion of the goals listed for performance indicator #3. We found that SSA completed the first three goals but did not complete the goal to "Begin the project with financial institutions to check their records to determine applicants/recipients eligibility for benefits by publishing final regulations, preparing a statement of work, and developing a schedule for the pilot". By FY 2002 year-end, SSA had only published the proposed regulations and still needed OMB approval to issue the final regulations.

OTHER MATTERS

We evaluated each performance indicator within the context of SSA's APP. While we found that these performance indicators were appropriate measures, we noted several areas for improvement.

Performance indicator #1 states, "Percent of States with which SSA has electronic access to human services and unemployment information." This statement implies that the performance indicator represents the percentage of States with which SSA has electronic access to both HS and UI. The performance indicator should instead, read, "Percent of State agencies with which SSA has electronic access to human services or unemployment information."

We also noted that Workers' Compensation (WC) is a State agency that is tracked in the SASRO report. There are, however, no performance indicators that track access to WC data. Additionally, we found no justification for grouping HS and UI in the same performance indicator. A separate performance indicator for each agency would more precisely measure the intent of the underlying strategic objective.

Performance indicator #2 states, "Percent of States with which SSA has electronic access to vital statistics and other material information." This performance indicator only measures percent of VS agencies. The last clause in this statement "…and other material information," is not representative of anything being measured and should be deleted.

The data definition for performance indicator #3 lists several goals that will help SSA increase electronic access to financial institutions and medical providers. However, it does not list any goals related to increasing electronic access to Federal agencies. This is inconsistent with the performance indicator, which specifically states, "…increasing electronic access to information held by other Federal agencies, financial institutions, and medical providers."

SSA should resolve this difference by removing the term "Federal agencies" or by adding additional goals related to increasing electronic access to Federal agencies.

In addition, we found that performance indicator #3 did not include an adequate description for each goal. For example, the first goal of this performance indicator states, "Evaluation of the California EME/PKI pilot..." The APP does not define this pilot or explain how it relates to the underlying performance indicator. Although our audit concluded that this goal does "…demonstrate progress in increasing electronic access to information held by external agencies…", the APP should include plain language definitions for each goal and succinctly define how they contribute to the overall strategic objective.

CONCLUSIONS AND RECOMMENDATIONS

We found the data used by SSA to report electronic access to HS and UI agencies for performance indicator #1

was not reliable. We found that performance indicators #1 and #2 were calculated incorrectly and had inadequate management controls. We found that the goals for performance indicator #3 were partially completed. Finally, we found that the performance indicators were appropriate measures for the AAP although we identified several opportunities for improvement. Our recommendations are as follows:

1. Account for DC when calculating performance indicator #1 and DC, VI, and PR when calculating performance indicator #2

SSA should include DC in the data definition for performance indicator #1 and DC, VI, and PR in the data definition for performance indicator #2. Additionally, SSA should change the denominator to account for DC for performance indicator #1 and DC, VI, and PR for performance indicator #2.

2. Improve and formalize management controls SSA should develop more rigorous policies and procedures in obtaining and tracking electronic access to State agencies. The policies and procedures should specifically include:
" Requiring regional coordinators to send initial legal agreements to Headquarters immediately after they are established.
" Requiring regional coordinators to maintain State agency contacts.

3. Create performance indicators for each type of State agency

Each type of State agency should be tracked separately through individual performance indicators. SSA should develop a performance indicator for HS, UI, VS, and WC to more precisely report the progress for each State agency.

4. Remove "…other material information," from performance indicator #2

SSA should remove this clause from performance indicator #2. The performance indicator only measures VS. The additional wording is extraneous and may be misleading.

5. Provide adequate descriptions for performance indicator #3 goals

SSA should provide plain language descriptions for each of the goals listed in performance indicator #3. Additionally, SSA should provide an explanation as to how these goals relate to the underlying performance indicator and strategic objective.

6. Resolve discrepancy between performance indicator #3 and its data definition

SSA should either remove the phrase "…Federal Agencies…" from this performance indicator or add goals that relate to increasing electronic access to Federal agencies. Currently, the data definition is inconsistent with the wording of the performance indicator.

AGENCY COMMENTS

SSA plans to remove the electronic access performance indicators from its list of FY 2003 GPRA measures to focus on its most significant priorities. SSA plans to continue to monitor these performance indicators internally.

SSA agreed with most of our recommendations. SSA disagreed with Recommendation 2 since Regional Data Exchange staff are required to maintain copies of their agreements. SSA also disagreed with Recommendation 5 since it will exclude electronic access performance indicators from its list of GPRA measures. The full text of SSA's comments can be found in Appendix D.

PWC RESPONSE

We agree that SSA should remove electronic access performance indicators from its list of FY 2003 GPRA performance indicators while continuing to monitor the measures internally. This allows SSA to focus on its most significant priorities while continuing to track progress in increasing electronic access to information and records.

For Recommendation 2, we agree that Regional Data Exchange staff can maintain copies of the agreements. However, we note that the Regional Data Exchange Coordinators should provide timely updates to the SASRO report to ensure its accuracy and completeness. We agree that SSA does not need to improve the performance indicator descriptions (Recommendation 5) since the electronic access performance indicators will not be included in the list of FY 2003 GPRA measures. SSA indicated it disagreed with a portion of Recommendation 1, however its comments are consistent with our recommendation. We reworded the recommendation to make it more precise. Appendix B was changed to reflect SSA's suggested rewording.

Appendices
APPENDIX A - Scope and Methodology
APPENDIX B - Flowcharts and Descriptions
APPENDIX C - Acronyms
APPENDIX D - Agency Comments

Appendix A

Scope and Methodology

We conducted this audit to examine three performance indicators related to the Social Security Administration's (SSA) initiative to increase electronic access to external agencies. We verified the underlying data and calculations for performance indicators #1 and #2, and we assessed the completion of the goals for performance indicator #3. Our audit was performed from May 1, 2002 through August 1, 2002 as follows:

Reviewed and verified all available legal agreements between SSA and State agencies listed in SSA's Access to State Records Online (SASRO) report;

Performed third party confirmations with all State agencies listed in the SASRO report;

Tested the accuracy of the performance indicator's calculation;

Reviewed all final deliverables and related work products for goals for performance indicator #3;

Ensured that deliverables adequately addressed the stated objectives of performance indicator #3 and ensured that the goals were completed or on-schedule to be completed for Fiscal Year (FY) 2002; and

Reviewed the process for internally reporting the status of each performance indicator.

In conducting this audit, we also:

Reviewed SSA's Performance and Accountability Report for FY 2001, SSA's Annual Performance Plan for FY 2001, and SSA's Revised Performance Plan for FY 2002 to determine related milestones, definition, and data source for each performance indicator;

Reviewed pertinent sections of the Government Performance and Results Act of 1993 and Office of Management and Budget circulars and ensured that each performance indicator conformed to the guidelines; and

Interviewed the respective project team for each milestone to gain an understanding of the goal, the process to complete each goal, and the outcome of the overall project.

Our audit was limited to testing at SSA's Headquarters in Woodlawn, Maryland, and transmitting third party confirmations via fax. The procedures we performed were in accordance with the American Institute of Certified Public Accountants' Statement on Standards for Consulting Services and the General Accounting Office's Government Auditing Standards ("Yellow Book") for performance audits.

Appendix B

Flowcharts and Descriptions

State Agency Electronic Access Process

State Agency Electronic Access Process:

Social Security Administration (SSA) contacts the State agency to establish electronic access to the State agency's data.
SSA determines if obtaining electronic access is feasible.
If electronic access is not feasible, SSA cannot obtain electronic access to the State agency's data.
If electronic access is feasible, SSA must establish a legal agreement to proceed.
SSA uses a standard legal agreement when State agency agrees.
If the standard legal agreement is used, the SSA Regional Commissioner and the State Commissioner sign the agreement.
If the standard agreement is not used, SSA and the State agency negotiate mutually acceptable language. The SSA Regional Commissioner and the State Commissioner sign the new agreement.
SSA sends representatives to the State agency to identify and address technical issues.
If the State agency is IBM-mainframe compatible, SSA and the agency exchange system information and establish a connection through the mainframe.
If the State agency is not IBM-mainframe compatible, SSA and the State agency choose one of three alternatives: an Internet Secure Socket Layer exchange, a Virtual Private Network exchange, or an extranet solution.
SSA and the State agency determine the appropriate method and establish access to the State agency.
The State agency is added to SSA's Access to State Records Online Report.
The SSA Office of Automation Support updates the intranet website listing of State agencies.

Appendix C

Acronyms

APP Annual Performance Plan
DC District of Columbia
EME Electronic Medical Evidence
FY Fiscal Year
GPRA Government Performance and Results Act
HS Human Services
OASDI Old Age, Survivors, and Disability Insurance
OMB Office of Management and Budget
PKI Public Key Infrastructure
PR Puerto Rico
SASRO SSA's Access to State Records Online
SSA Social Security Administration
SSL Secure Socket Layer
UI Unemployment Information
VI Virgin Islands
VPN Virtual Private Network
VS Vital Statistics
WC Workers' Compensation
Appendix D
Agency Comments

MEMORANDUM

Date: 12/24/02

To: James G. Huse, Jr.
Inspector General

From: Larry W. Dye
Chief of Staff

Subject: Office of the Inspector General (OIG) Draft Report, "Performance Indicator Audit: Electronic Access" (A-15-02-11083)-INFORMATION

We appreciate OIG's efforts in conducting this review. Our comments on the report content and recommendations are attached. Staff questions can be referred to Laura Bell on extension 52636.

COMMENTS OF THE SOCIAL SECURITY ADMINISTRATION (SSA) ON THE OFFICE OF THE INSPECTOR GENERAL (OIG) DRAFT REPORT, "REVIEW OF THE ELECTRONIC ACCESS PERFORMANCE INDICATOR" (A-15-02-11083)

We appreciate the opportunity to review and comment on this audit. We think it is important to say up front why we have deleted the electronic access performance indicators as external indicators and emphasize that we will continue to monitor the data internally. We have reduced the Agency's overall number of external indicators in order to be able to focus our performance planning on the most significant Agency priorities. We still consider the electronic access indicators to be important, and therefore we will continue monitoring them internally.

Our comments on the specific recommendations follow:

Recommendation 1

Account for the District of Columbia (DC) and territories when calculating performance indicators #1 and #2.

SSA Comment

We agree that the District of Columbia (DC) should be added to the calculation of online access to Human Service (HS) and Unemployment Insurance (UI) Agencies. However, we do not agree that Puerto Rico and the Virgin Islands should be included. Online access to both HS and UI data was negotiated to assist in the area of Supplemental Security Income (SSI) high risk, in an effort to reduce/eliminate payment errors and overpayments. SSA does not pay SSI in either PR or VI. We also agree to include DC and the territories in the calculation of access to Vital Statistics.

Recommendation 2

Improve and formalize management controls.

SSA Comment

SSA disagrees with the request to have copies of the agreements mailed to SSA. Regional coordinators routinely send in copies of the agreements they negotiate with their State agencies. However, SSA does not see the Headquarters staff as the record keeper for the State agreements. The Disclosure Policy staff assists the regions with their agreements by reviewing State requested amendments, wording changes, etc. The Regional Data Exchange staff is required to maintain copies and controls of their agreements.

Recommendation 3

Create performance indicators for each type of State Agency.

SSA Comment

SSA agrees with this recommendation, which is exactly the information captured by SSA's Office of Automation Support. SSA will continue to capture and monitor the data for each State Agency individually, even though the performance indicators for electronic access will no longer be part of SSA's external Government Performance and Results Act (GPRA) documents."

Recommendation 4

Remove..."other material information," from performance indicator #2

SSA Comment

SSA agrees with this recommendation. Although this will no longer be a GPRA performance indicator in FY 2003 and FY 2004, we will continue to track progress in getting electronic access to VS information.

Recommendation 5

Provide adequate descriptions for performance indicator #3 goals.

SSA Comment

We do not agree. This performance indicator is no longer an external GPRA measure for FY 2003 and FY 2004. We will continue to plan and track these activities internally. The Agency is being encouraged by the OMB to have fewer and more meaningful indicators "to tell our story." SSA will be moving to an aggregate measure of representative transactions rather than separate measures.

Recommendation 6

Resolve discrepancy between performance indicator #3 and its data definition.

SSA Comment

We agree that the phrase "Federal Agencies" should be removed.

Other Matters

Appendix B,

Both the flowchart and the description of the process should be changed. While it is essentially correct, there is a detail that needs to be clarified.

The seventh bullet which currently reads:

If the standard agreement is not used SSA Office of General Counsel draft a new agreement with the State agency's counsel. The SSA Regional Commissioner and the State Commissioner sign the new agreement.

Should be changed to:

If the standard agreement is not used, SSA and the State agency negotiate mutually acceptable language. The SSA Regional Commissioner and the State Commissioner sign the new agreement.

The pertinent box in the flow chart should be changed to "SSA/Agency develops (or negotiates) new language."