OFFICE OF
THE INSPECTOR GENERAL

SOCIAL SECURITY ADMINISTRATION

PERFORMANCE INDICATOR AUDIT:
MANAGEMENT INFORMATION SYSTEMS
AND MAINFRAME PROTECTION

September 2006            A-15-06-16112

 

Mission

By conducting independent and objective audits, evaluations and investigations, we inspire public confidence in the integrity and security of SSA’s programs and operations and protect them against fraud, waste and abuse.  We provide timely, useful and reliable information and advice to Administration officials, Congress and the public.

Authority

The Inspector General Act created independent audit and investigative units, called the Office of Inspector General (OIG).  The mission of the OIG, as spelled out in the Act, is to:

     To ensure objectivity, the IG Act empowers the IG with:

Vision

We strive for continual improvement in SSA’s programs, operations and management by proactively seeking new ways to prevent and deter fraud, waste and abuse.  We commit to integrity and excellence by supporting an environment that provides a valuable public service while encouraging employee development and retention and fostering diversity and innovation.

MEMORANDUM

Date:       September 18, 2006                                                                                                                            Refer To:

To:         The Commissioner
             
From:      Inspector General
                 
Subject:  Performance Indicator Audit:  Management Information Systems and Mainframe Protection (A-15-06-16112)

We contracted with PricewaterhouseCoopers, LLP (PwC) to evaluate 15 of the Social Security Administration’s performance indicators established to comply with the Government Performance and Results Act.  The attached final report presents the results of three of the performance indicators PwC reviewed.  For the performance indicators included in this audit, PwC’s objectives were to:

This report contains the results of the audit for the following indicators:

Please provide within 60 days a corrective action plan that addresses each recommendation.  If you wish to discuss the final report, please call me or have your staff contact Steven L. Schaeffer, Assistant Inspector General for Audit, at
(410) 965-9700.
Patrick P. O’Carroll, Jr

Attachment

 

MEMORANDUM    

Date:       September 6, 2006

To:        Inspector General

From:     PricewaterhouseCoopers, LLP

Subject: Performance Indicator Audit:  Management Information Systems and Mainframe Protection (A-15-06-16112)

 

OBJECTIVE

The Government Performance and Results Act (GPRA) of 1993 requires the Social Security Administration (SSA) to develop performance indicators that assess the relevant service levels and outcomes of each program activity.   GPRA also calls for a description of the means employed to verify and validate the measured values used to report on program performance.  

Our audit was conducted in accordance with generally accepted government auditing standards for performance audits.  For the performance indicators included in this audit, our objectives were to:

  1. Assess the effectiveness of internal controls and test critical controls over data generation, calculation, and reporting processes for the specific performance indicator.

 

  1. Assess the overall reliability of the performance indicator’s computer processed data.  Data are reliable when they are complete, accurate, consistent and are not subject to inappropriate alteration.
  1. Test the accuracy of results presented and disclosed in the Fiscal Year (FY) 2005 Performance and Accountability Report (PAR).

 

  1. Assess if the performance indicator provides a meaningful measurement of the program it measures and the achievement of its stated objective.

BACKGROUND

We audited the following performance indicators as stated in the SSA FY 2005 PAR:

Performance Indicator

FY 2005 Goal

FY 2005 Actual Reported Results

Enhance efforts to improve financial performance using Managerial Cost Accountability System (MCAS).

15%

5%

Improve workload information using Social Security Unified Measurement System (SUMS).

46%

42%

Maintain zero outside infiltrations of SSA's programmatic mainframes.

0 infiltrations

0 infiltrations

MCAS and SUMS Projects
SSA is developing two new systems to enhance the monitoring and reporting of financial and performance data.  MCAS and SUMS will be a key enabler to allow SSA to monitor and report progress toward achieving its strategic goals and objectives and tracking resource expenditures.

SSA Information Systems
SSA employees process a tremendous amount of sensitive personal data through the SSA mainframe applications on a daily basis.  To ensure the integrity and security of this data, SSA has invested heavily in the development and implementation of multiple layers of electronic security.  As a result, SSA management has implemented numerous intrusion detection and prevention controls to identify and address threats to the SSA systems.  SSA management continuously monitors the security of the SSA mainframe environment, and the networks that surrounds it.

RESULTS OF REVIEW

We did not identify any significant findings related to the internal controls, data reliability, meaningfulness, accuracy of presentation, or disclosure of the information related to the indicators "Enhance efforts to improve financial performance using Managerial Cost Accountability System (MCAS)" and "Improve workload information using Social Security Unified Measurement System (SUMS)."We identified findings related to

internal controls, meaningfulness, and accuracy of presentation and disclosure of the information contained in the PAR for the indicator "Maintain zero outside infiltrations of SSA's programmatic mainframes." 

Enhance efforts to improve financial performance using Managerial Cost Accountability System (MCAS)

Indicator Background

“MCAS focuses on critical performance and financial information needed by managers and employees, and promotes performance accountability for Social Security programs.  As stewards of the Social Security Trust Fund, SSA must also model appropriate information management processes to ensure accountability for workloads.  The Agency’s MCAS includes a number of projects designed to update the cost analysis system, reporting systems, workload measurement systems, and system access.  The integration of financial and performance management systems will allow the Agency to routinely assess performance and financial information in order for local managers to make more timely and efficient day-to-day decisions."

Performance Indicator Calculation

Performance %

=

A methodology which weights individual projects to create a combined percentage is used to track the overall completion of this initiative.

 

 

The SUMS/MCAS project plan tracking and releases as reported to the SUMS/MCAS Executive Steering Committee are the data sources for this calculation.

Findings

We did not identify any significant findings related to the internal controls, data reliability, accuracy of presentation, meaningfulness, or disclosure of the information related to this indicator contained in the PAR.

Improve workload information using Social Security Unified Measurement Systems (SUMS)

Indicator Background

"The Agency has recognized the need to improve the quality, consistency and access to information that is used by managers and analysts throughout SSA to manage work and account for resources.  The objective of SUMS is to create a system for SSA operational components that counts and measures all work in a consistent manner regardless of where the work is processed.  This system provides access to information needed to meet changing business requirements, support process reviews and comply with government standards.  Access to web based reports and workload control listings and other information are available on demand, eliminating the need for paper reports."

SUMS is considered a key enabler in monitoring and reporting on SSA's progress toward achieving its strategic goals and objectives and tracking resource expenditures.  The objective of this system is to count and to measure work in a consistent manner at all organizational levels.  It provides the detailed information that managers need to monitor service, forecast workloads, and make informed decisions on how best to manage work and resources.

Performance Indicator Calculation

Performance %

 

=

A methodology which weights individual projects to create a combined percentage used to track the overall completion of this initiative.  Completion percentages are also attributed to cross cutting projects, including Time Allocation and the Customer
Service Record to derive an overall SUMS completion percentage.

 

 

 

 

The SUMS/MCAS project plan tracking and releases as reported to the SUMS/MCAS Executive Steering Committee are the data sources for this calculation.

Findings

We did not identify any significant findings related to the internal controls, data reliability, accuracy of presentation, meaningfulness, or disclosure of the information related to this indicator contained in the PAR.

Maintain zero outside infiltrations of SSA's programmatic mainframes

Indicator Background

SSA maintains an Intrusion Protection Team (IPT) specifically created to prevent outside infiltrations of systems.  The IPT uses numerous software tools to immediately detect attempts to infiltrate SSA’s network and underlying systems.  Additionally, software controls at all levels of SSA systems are used to prevent unauthorized access to SSA systems. 

SSA created this performance indicator to document the Agency’s success in protecting the mainframe computers, on which SSA’s sensitive programmatic data resides.  According to SSA security management and the PAR, the indicator is intended to measure infiltrations from outside of SSA, and not infiltrations from authorized internal users who manage to elevate their privileges and perform unauthorized actions.  In addition, an infiltration is further defined as “…unauthorized access that requires a cleanup or restoration of backup files to a state prior to the infiltration.”  Also the indicator is intended to only measure infiltrations of the mainframe computers.  Infiltrations that are related to non-mainframe systems, including SSA’s Intranet, network, and distributed systems are excluded for reporting purposes within this indicator.

Performance Indicator Calculation

Total Mainframe Infiltrations

=

Count of the times that Mainframe Infiltrations are detected from the period of October 1, 2004 to September 30, 2005.

 

 

The count of mainframe infiltrations is maintained in the Change Asset and Problem Reporting System (CAPRS).

Findings

Internal Controls and Data Reliability

We found the policies and procedures related to the formal process to capture, store, and calculate the results of the performance indicator were not adequate.  The documentation did not accurately describe the process in place during FY 2005 and all components of the indicator calculation were not included.  Office of Management and Budget (OMB) Circular A-123, Management Accountability and Control, requires, "...documentation for transactions, management controls, and other significant events must be clear and readily available for examination. …"

It should be noted that SSA management was in the process of updating the documentation related to this indicator during the time of the audit.  As the calculation of this indicator is not based on computerized data, we did not complete an analysis of data reliability.

Accuracy of PAR Presentation and Disclosure

The intent of the indicator is to highlight SSA’s success in preventing mainframe infiltrations.  We believe this is an important goal and its success is very relevant to the Agency.  However, it is not possible to state that undetected infiltrations did not occur.  Therefore, management cannot measure or fully assert that an outside infiltration has not occurred. 

We also noted inconsistencies in the descriptions of the indicator.  Based on the title of the indicator, internal infiltrations would not be included in the calculation of this indicator; however, the definition, as described in the FY 2005 PAR, is unclear with regard to inclusion of internal infiltrations:

An infiltration is an unauthorized access that requires a cleanup or restoration of back-up files to a state prior to the infiltration.  This measure is a count of the number of times that an infiltration of mainframes is detected. (emphasis added)

Finally, we believe that the data definition too narrowly defines a mainframe infiltration and could omit important events such as unauthorized access which results in disclosure of sensitive SSA information or misuse of data that occurs but does not require clean up or restoration activities.  The Federal Information Processing Standards Publication (FIPS PUB) 200 defines an incident as

An occurrence that actually or potentially jeopardizes the confidentiality, integrity, or availability of an information system or the information the system processes, stores, or transmits or that constitutes a violation or imminent threat of violation of security policies, security procedures, or acceptable use policies. 

Additionally, the indicator excludes infiltrations of SSA’s Intranet, network and distributed systems which maintain important Agency information.

Performance Indicator Meaningfulness

SSA management does not provide a clear statement in the PAR of how preventing outside infiltrations of the mainframe relates to the Agency goal “To ensure superior Stewardship of Social Security programs and resources,” or the Agency objective of “Efficiently manage Agency finances and assets, and effectively link resources to performance outcomes.”

CONCLUSION AND RECOMMENDATIONS

SSA management indicated that the performance indicator “Maintain Zero Outside Infiltrations of SSA’s Programmatic Mainframes” will be significantly updated in the 2006 PAR.  As such we recommend SSA:

  1. Document the policies and procedures used to prepare and disclose the results of the performance indicator.

 

  1. Ensure the performance indicator definitions and reported results are meaningful, complete, and consistent with the title by:

Zero outside infiltrations of SSA’s programmatic mainframes were detected.

 

  1. Articulate and disclose the linkage of the performance indicator to the Agency’s strategic goals and objectives.

AGENCY COMMENTS

SSA agreed with our recommendations.  See Appendix D for the Agency’s comments.

 

Appendices

APPENDIX A – Acronyms

APPENDIX B – Scope and Methodology

APPENDIX C – Process Flowcharts

APPENDIX D – Agency Comments

Appendix A -- Acronyms

APP

Annual Performance Plan

CAPRS

Change Asset and Problem Reporting System

DCS

Deputy Commissioner of Systems

DMZ

Demilitarized Zone

US-CERT

United States Computer Emergency Readiness Team

FIPS PUB

Federal Information Processing Standards Publication

FY

Fiscal Year

GPRA

Government Performance and Results Act

IPT

Intrusion Protection Team

MCAS

Managerial Cost Accountability System

OCIO

Office of Chief Information Officer

OCSO

Office of the Chief Strategic Officer

PAR

Performance and Accountability Report

SSA

Social Security Administration

SRT

Security Response Team

SUMS

Social Security Unified Measurement System

VPN

Virtual Private Network


Appendix B -- Scope and Methodology

We updated our understanding of the Social Security Administration’s (SSA) Government Performance and Results Act (GPRA) processes.  This was completed through research and inquiry of SSA management.  We also requested SSA to provide various documents regarding the specific programs being measured as well as the specific measurement used to assess the effectiveness and efficiency of the related program. 

Through inquiry, observation, and other substantive testing, including testing of source documentation, we performed the following:

As part of this audit, we documented our understanding, as conveyed to us by Agency personnel, of the alignment of the Agency’s mission, goals, objectives, processes, and related performance indicators.  We analyzed how these processes interacted with related processes within SSA and the existing measurement systems.  Our understanding of the Agency’s mission, goals, objectives, and processes were used to determine if the performance indicators being used appear to be valid and appropriate given our understanding of SSA’s mission, goals, objectives and processes.

We followed all performance audit standards in accordance with generally accepted government auditing standards.  In addition to the steps above, we specifically performed the following to test the indicators included in this report:
Management Information Systems, Management Cost Accountability Systems (MCAS) and Social Security Unified Measurement Systems (SUMS)

Maintain zero outside infiltrations of SSA’s programmatic mainframeS

Appendix C -- Flowchart of Management Information Systems, MCAS and SUMS

(flowchat graphic eliminated from text-only version of this document)

Management Information Systems, MCAS and SUMS

Flowchart and Mainframe Protection

(flowchat graphic eliminated from text-only version of this document)

Mainframe Protection

Appendix D -- Agency Comments

SOCIAL SECURITY

 

MEMORANDUM                                                                                                  

                                                                                                                 


Date:

September 5, 2006

Refer To: S1J-3

                                                                                                                                                           


To:

Patrick P. O'Carroll, Jr.
Inspector General

From:

Larry W. Dye  /s/
Chief of Staff

Subject:

Office of the Inspector General (OIG) Draft Report, “Performance Indicator Audit:  Management Information Systems and Mainframe Protection” (A-15-06-16112)--INFORMATION

 

We appreciate OIG’s efforts in conducting this review.  Our comments on the draft report are attached.

Please let me know if you have any questions.  Staff inquiries may be directed to
Ms. Candace Skurnik, Director, Audit Management and Liaison Staff, at extension 54636.

Attachment:
SSA Comments

 


COMMENTS ON THE OFFICE OF THE INSPECTOR GENERAL’S (OIG) DRAFT REPORT, “PERFORMANCE INDICATOR AUDIT:  MANAGEMENT INFORMATION SYSTEMS AND MAINFRAME PROTECTION” (A-15-06-16112)

Thank you for the opportunity to review and provide comments on this draft report.  The report notes that the auditors did not identify any significant findings related to two of the three performance indicators included in this audit: "Enhance efforts to improve financial performance using the Managerial Cost Accountability System" and "Improve workload information using Social Security Unified Measurement System."  However, the report includes significant findings related to the performance indicator "Maintain zero outside infiltrations of SSA's programmatic mainframes." 

We are reviewing the performance indicator “Maintain Zero Infiltrations of SSA’s Programmatic Mainframes,” as well as the data definition and the linkage of the indicator to the Agency’s Goals and Objectives.  In this regard, we are taking an in-depth look at the existing tools and techniques to determine the Agency’s ability to monitor, record and report meaningful measurements to include infiltrations of the Agency’s intranet, network and distributed systems. 

We have the following comments on the report’s recommendations.

Recommendation 1

Document the policies and procedures used to prepare and disclose the results of the performance indicator.

Comment

We agree.  We documented the policies and procedures used to prepare and disclose the results of the performance indicator and provided them to OIG and PricewaterhouseCoopers (PwC).  PwC indicated the policies and procedures sufficiently document the processes.  

Recommendation 2

Ensure the performance indicator definitions and reported results are meaningful, complete, and consistent with the title by:
- Clearly documenting the inclusion or exclusion of internal infiltrations in the calculation of the indicator results;
- Revising the performance indicator results to clarify that it measures only detected infiltrations.  As an example, the indicator actual performance results could be documented as follows:
Zero outside infiltrations of SSA’s programmatic mainframes were detected.
- Broadening the indicator data definition to include infiltrations resulting in disclosure or misuse of sensitive SSA data; and,
- Expanding the calculation of indicator results to include infiltrations of the Agency's intranet, network, and distributed systems.

Comment

This recommendation contains 4-items.  We agree with the first item  and are performing an in-depth review to ensure the performance indicator definitions and reported results are meaningful, complete and consistent.

We also agree with the second item included in this recommendation.  We are revising the performance indicator to clarify that it measures only detected infiltrations.

Regarding the third item included in this recommendation, we are determining the technical aspects and feasibility of including infiltrations resulting in disclosure or misuse of sensitive data.  Currently, we are unsure of the available methodologies, tools and techniques.  If this section of the recommendation cannot be implemented using the existing processes, an evaluation and cost analysis will be required. 

About the fourth item included in this recommendation, we are determining if, using existing technologies, methodologies and tools, the results can be measured to include infiltrations of the Agency’s intranet, network and distributed systems.  If this section of the recommendation cannot be implemented using the existing processes, this will also require an evaluation and cost analysis. 

Recommendation 3

Articulate and disclose the linkage of the performance indicator to the Agency’s strategic goals and objectives.

Comment

We agree.  We will update the Performance and Accountability Report and Information Resources Management plan to articulate the linkage of the performance indicator to the Agency’s strategic goals and objectives. 

[In addition to the comments above, SSA provided a technical comment, which has been addressed in the final report.]

Overview of the Office of the Inspector General
The Office of the Inspector General (OIG) is comprised of our Office of Investigations (OI), Office of Audit (OA), Office of the Chief Counsel to the Inspector General (OCCIG), and Office of Resource Management (ORM).  To ensure compliance with policies and procedures, internal controls, and professional standards, we also have a comprehensive Professional Responsibility and Quality Assurance program.

Office of Audit
OA conducts and/or supervises financial and performance audits of the Social Security Administration’s (SSA) programs and operations and makes recommendations to ensure program objectives are achieved effectively and efficiently.  Financial audits assess whether SSA’s financial statements fairly present SSA’s financial position, results of operations, and cash flow.  Performance audits review the economy, efficiency, and effectiveness of SSA’s programs and operations.  OA also conducts short-term management and program evaluations and projects on issues of concern to SSA, Congress, and the general public.

Office of Investigations
OI conducts and coordinates investigative activity related to fraud, waste, abuse, and mismanagement in SSA programs and operations.  This includes wrongdoing by applicants, beneficiaries, contractors, third parties, or SSA employees performing their official duties.  This office serves as OIG liaison to the Department of Justice on all matters relating to the investigations of SSA programs and personnel.  OI also conducts joint investigations with other Federal, State, and local law enforcement agencies.

Office of the Chief Counsel to the Inspector General
OCCIG provides independent legal advice and counsel to the IG on various matters, including statutes, regulations, legislation, and policy directives.  OCCIG also advises the IG on investigative procedures and techniques, as well as on legal implications and conclusions to be drawn from audit and investigative material.  Finally, OCCIG administers the Civil Monetary Penalty program.

Office of Resource Management
ORM supports OIG by providing information resource management and systems security.  ORM also coordinates OIG’s budget, procurement, telecommunications, facilities, and human resources.  In addition, ORM is the focal point for OIG’s strategic planning function and the development and implementation of performance measures required by the Government Performance and Results Act of 1993.

-- footnotes follow --

Public Law Number 103-62, 107 Stat. 285 (codified as amended in scattered sections of 5 United States Code (U.S.C.), 31 U.S.C. and 39 U.S.C.).

31 U.S.C. § 1115(a)(4).

31 U.S.C. § 1115(a)(6).

GAO-03-273G, Assessing Reliability of Computer Processed Data, October 2002, p. 3.

SSA, PAR FY 2005 p. 99.

Id. p. 84.

Id. p. 98.

Id. pp. 35 and 42.

Id. p. 99.

Id. p. 99.

Id. p. 84.

Id. p. 35.

Id. p. 84.

Id. p. 99.

OMB Circular A-123, Attachment II, Establishing Management Controls, June 21, 1995. Note:  OMB Circular A-123 Revised December 21, 2004, did not become effective until FY 2006 and therefore was not in place during the time period of the review.

SSA, PAR FY 2005 p. 99.

FIPS PUB 200, Minimum Security Requirements for Federal Information and Information Systems, March 2006, p. 7.