OFFICE OF
THE INSPECTOR GENERAL
SOCIAL SECURITY ADMINISTRATION
PERFORMANCE INDICATOR AUDIT:
MANAGEMENT INFORMATION SYSTEMS
AND MAINFRAME PROTECTION
September 2006 A-15-06-16112
Mission
By conducting independent and objective audits, evaluations and investigations, we inspire public confidence in the integrity and security of SSA’s programs and operations and protect them against fraud, waste and abuse. We provide timely, useful and reliable information and advice to Administration officials, Congress and the public.
Authority
The Inspector General Act created independent audit and investigative units, called the Office of Inspector General (OIG). The mission of the OIG, as spelled out in the Act, is to:
To ensure objectivity, the IG Act empowers the IG with:
Vision
We strive for continual improvement in SSA’s programs, operations
and management by proactively seeking new ways to prevent and deter fraud,
waste and abuse. We commit to integrity and excellence by supporting
an environment that provides a valuable public service while encouraging
employee development and retention and fostering diversity and innovation.
MEMORANDUM
Date: September 18, 2006 Refer To:
To: The Commissioner
From: Inspector General
Subject: Performance Indicator Audit: Management Information
Systems and Mainframe Protection (A-15-06-16112)
We contracted with PricewaterhouseCoopers, LLP (PwC) to evaluate 15 of the Social Security Administration’s performance indicators established to comply with the Government Performance and Results Act. The attached final report presents the results of three of the performance indicators PwC reviewed. For the performance indicators included in this audit, PwC’s objectives were to:
This report contains the results of the audit for the following indicators:
Please provide within 60 days a corrective action plan that addresses each
recommendation. If you wish to discuss the final report, please call
me or have your staff contact Steven L. Schaeffer, Assistant Inspector General
for Audit, at
(410) 965-9700.
Patrick P. O’Carroll, Jr
Attachment
Date: September 6, 2006
To: Inspector General
From: PricewaterhouseCoopers, LLP
Subject: Performance Indicator Audit: Management Information Systems and Mainframe Protection (A-15-06-16112)
OBJECTIVE
The Government Performance and Results Act (GPRA) of 1993 requires the Social Security Administration (SSA) to develop performance indicators that assess the relevant service levels and outcomes of each program activity. GPRA also calls for a description of the means employed to verify and validate the measured values used to report on program performance.
Our audit was conducted in accordance with generally accepted government auditing standards for performance audits. For the performance indicators included in this audit, our objectives were to:
BACKGROUND
We audited the following performance indicators as stated in the SSA FY 2005 PAR:
MCAS and SUMS Projects
SSA is developing two new systems to enhance the monitoring and reporting of
financial and performance data. MCAS and SUMS will be a key enabler
to allow SSA to monitor and report progress toward achieving its strategic
goals and objectives and tracking resource expenditures.
SSA Information Systems
SSA employees process a tremendous amount of sensitive personal data through
the SSA mainframe applications on a daily basis. To ensure the integrity
and security of this data, SSA has invested heavily in the development and
implementation of multiple layers of electronic security. As a result,
SSA management has implemented numerous intrusion detection and prevention
controls to identify and address threats to the SSA systems. SSA management
continuously monitors the security of the SSA mainframe environment, and
the networks that surrounds it.
RESULTS OF REVIEW
We did not identify any significant findings related to the internal controls, data reliability, meaningfulness, accuracy of presentation, or disclosure of the information related to the indicators "Enhance efforts to improve financial performance using Managerial Cost Accountability System (MCAS)" and "Improve workload information using Social Security Unified Measurement System (SUMS)."We identified findings related to
internal controls, meaningfulness, and accuracy of presentation and disclosure of the information contained in the PAR for the indicator "Maintain zero outside infiltrations of SSA's programmatic mainframes."
Enhance efforts to improve financial performance using Managerial Cost Accountability System (MCAS)
Indicator Background
“MCAS focuses on critical performance and financial information needed by managers and employees, and promotes performance accountability for Social Security programs. As stewards of the Social Security Trust Fund, SSA must also model appropriate information management processes to ensure accountability for workloads. The Agency’s MCAS includes a number of projects designed to update the cost analysis system, reporting systems, workload measurement systems, and system access. The integration of financial and performance management systems will allow the Agency to routinely assess performance and financial information in order for local managers to make more timely and efficient day-to-day decisions."
Performance Indicator Calculation |
Performance % |
= |
A methodology which weights individual projects to create a combined percentage is used to track the overall completion of this initiative. |
The SUMS/MCAS project plan tracking and releases as reported to the SUMS/MCAS Executive Steering Committee are the data sources for this calculation.
Findings
We did not identify any significant findings related to the internal controls, data reliability, accuracy of presentation, meaningfulness, or disclosure of the information related to this indicator contained in the PAR.
Improve workload information using Social Security Unified Measurement Systems (SUMS)
Indicator Background
"The Agency has recognized the need to improve the quality, consistency and access to information that is used by managers and analysts throughout SSA to manage work and account for resources. The objective of SUMS is to create a system for SSA operational components that counts and measures all work in a consistent manner regardless of where the work is processed. This system provides access to information needed to meet changing business requirements, support process reviews and comply with government standards. Access to web based reports and workload control listings and other information are available on demand, eliminating the need for paper reports."
SUMS is considered a key enabler in monitoring and reporting on SSA's progress toward achieving its strategic goals and objectives and tracking resource expenditures. The objective of this system is to count and to measure work in a consistent manner at all organizational levels. It provides the detailed information that managers need to monitor service, forecast workloads, and make informed decisions on how best to manage work and resources.
Performance Indicator Calculation |
Performance % |
= |
A methodology which weights individual projects
to create a combined percentage used to track the overall completion
of this initiative. Completion
percentages are also attributed to cross cutting projects, including
Time Allocation and the Customer |
The SUMS/MCAS project plan tracking and releases as reported to the SUMS/MCAS Executive Steering Committee are the data sources for this calculation.
Findings
We did not identify any significant findings related to the internal controls, data reliability, accuracy of presentation, meaningfulness, or disclosure of the information related to this indicator contained in the PAR.
Maintain zero outside infiltrations of SSA's programmatic mainframes
Indicator Background
SSA maintains an Intrusion Protection Team (IPT) specifically created to prevent outside infiltrations of systems. The IPT uses numerous software tools to immediately detect attempts to infiltrate SSA’s network and underlying systems. Additionally, software controls at all levels of SSA systems are used to prevent unauthorized access to SSA systems.
SSA created this performance indicator to document the Agency’s success in protecting the mainframe computers, on which SSA’s sensitive programmatic data resides. According to SSA security management and the PAR, the indicator is intended to measure infiltrations from outside of SSA, and not infiltrations from authorized internal users who manage to elevate their privileges and perform unauthorized actions. In addition, an infiltration is further defined as “…unauthorized access that requires a cleanup or restoration of backup files to a state prior to the infiltration.” Also the indicator is intended to only measure infiltrations of the mainframe computers. Infiltrations that are related to non-mainframe systems, including SSA’s Intranet, network, and distributed systems are excluded for reporting purposes within this indicator.
Performance Indicator Calculation |
Total Mainframe Infiltrations |
= |
Count of the times that Mainframe Infiltrations are detected from the period of October 1, 2004 to September 30, 2005. |
The count of mainframe infiltrations is maintained in the Change Asset and Problem Reporting System (CAPRS).
Findings
Internal Controls and Data Reliability
We found the policies and procedures related to the formal process to capture, store, and calculate the results of the performance indicator were not adequate. The documentation did not accurately describe the process in place during FY 2005 and all components of the indicator calculation were not included. Office of Management and Budget (OMB) Circular A-123, Management Accountability and Control, requires, "...documentation for transactions, management controls, and other significant events must be clear and readily available for examination. …"
It should be noted that SSA management was in the process of updating the documentation related to this indicator during the time of the audit. As the calculation of this indicator is not based on computerized data, we did not complete an analysis of data reliability.
Accuracy of PAR Presentation and Disclosure
The intent of the indicator is to highlight SSA’s success in preventing mainframe infiltrations. We believe this is an important goal and its success is very relevant to the Agency. However, it is not possible to state that undetected infiltrations did not occur. Therefore, management cannot measure or fully assert that an outside infiltration has not occurred.
We also noted inconsistencies in the descriptions of the indicator. Based on the title of the indicator, internal infiltrations would not be included in the calculation of this indicator; however, the definition, as described in the FY 2005 PAR, is unclear with regard to inclusion of internal infiltrations:
An infiltration is an unauthorized access that requires a cleanup or restoration of back-up files to a state prior to the infiltration. This measure is a count of the number of times that an infiltration of mainframes is detected. (emphasis added)
Finally, we believe that the data definition too narrowly defines a mainframe infiltration and could omit important events such as unauthorized access which results in disclosure of sensitive SSA information or misuse of data that occurs but does not require clean up or restoration activities. The Federal Information Processing Standards Publication (FIPS PUB) 200 defines an incident as
An occurrence that actually or potentially jeopardizes the confidentiality, integrity, or availability of an information system or the information the system processes, stores, or transmits or that constitutes a violation or imminent threat of violation of security policies, security procedures, or acceptable use policies.
Additionally, the indicator excludes infiltrations of SSA’s Intranet, network and distributed systems which maintain important Agency information.
Performance Indicator Meaningfulness
SSA management does not provide a clear statement in the PAR of how preventing outside infiltrations of the mainframe relates to the Agency goal “To ensure superior Stewardship of Social Security programs and resources,” or the Agency objective of “Efficiently manage Agency finances and assets, and effectively link resources to performance outcomes.”
CONCLUSION AND RECOMMENDATIONS
SSA management indicated that the performance indicator “Maintain Zero Outside Infiltrations of SSA’s Programmatic Mainframes” will be significantly updated in the 2006 PAR. As such we recommend SSA:
Zero outside infiltrations of SSA’s programmatic mainframes were detected.
AGENCY COMMENTS
SSA agreed with our recommendations. See Appendix D for the Agency’s comments.
Appendices
APPENDIX A – Acronyms
APPENDIX B – Scope and Methodology
APPENDIX C – Process Flowcharts
APPENDIX D – Agency Comments
Appendix A -- Acronyms
APP |
Annual Performance Plan |
CAPRS |
Change Asset and Problem Reporting System |
DCS |
Deputy Commissioner of Systems |
DMZ |
Demilitarized Zone |
US-CERT |
United States Computer Emergency Readiness Team |
FIPS PUB |
Federal Information Processing Standards Publication |
FY |
Fiscal Year |
GPRA |
Government Performance and Results Act |
IPT |
Intrusion Protection Team |
MCAS |
Managerial Cost Accountability System |
OCIO |
Office of Chief Information Officer |
OCSO |
Office of the Chief Strategic Officer |
PAR |
Performance and Accountability Report |
SSA |
Social Security Administration |
SRT |
Security Response Team |
SUMS |
Social Security Unified Measurement System |
VPN |
Virtual Private Network |
Appendix B --
Scope and Methodology
We updated our understanding of the Social Security Administration’s (SSA) Government Performance and Results Act (GPRA) processes. This was completed through research and inquiry of SSA management. We also requested SSA to provide various documents regarding the specific programs being measured as well as the specific measurement used to assess the effectiveness and efficiency of the related program.
Through inquiry, observation, and other substantive testing, including testing of source documentation, we performed the following:
As part of this audit, we documented our understanding, as conveyed to us by Agency personnel, of the alignment of the Agency’s mission, goals, objectives, processes, and related performance indicators. We analyzed how these processes interacted with related processes within SSA and the existing measurement systems. Our understanding of the Agency’s mission, goals, objectives, and processes were used to determine if the performance indicators being used appear to be valid and appropriate given our understanding of SSA’s mission, goals, objectives and processes.
We followed all performance audit standards in accordance with generally accepted
government auditing standards. In addition to the steps above, we specifically
performed the following to test the indicators included in this report:
Management Information Systems, Management Cost Accountability Systems
(MCAS) and Social Security Unified Measurement Systems (SUMS)
Maintain zero outside infiltrations of SSA’s programmatic mainframeS
Appendix C -- Flowchart of Management Information Systems, MCAS and SUMS
(flowchat graphic eliminated from text-only version of this document)
Management Information Systems, MCAS and SUMS
Flowchart and Mainframe Protection
(flowchat graphic eliminated from text-only version of this document)
Mainframe Protection
Appendix D -- Agency Comments
SOCIAL SECURITY
MEMORANDUM |
Date: |
September 5, 2006 |
Refer To: S1J-3 |
To: |
Patrick P. O'Carroll, Jr. |
From: |
Larry W. Dye /s/ |
Subject: |
Office of the Inspector General (OIG) Draft Report, “Performance Indicator Audit: Management Information Systems and Mainframe Protection” (A-15-06-16112)--INFORMATION |
We appreciate OIG’s efforts in conducting this review. Our comments on the draft report are attached.
Please let me know if you have any questions. Staff inquiries may be
directed to
Ms. Candace Skurnik, Director, Audit Management and Liaison Staff, at extension
54636.
Attachment:
SSA Comments
COMMENTS ON THE OFFICE OF THE INSPECTOR GENERAL’S (OIG) DRAFT REPORT, “PERFORMANCE INDICATOR AUDIT: MANAGEMENT INFORMATION SYSTEMS AND MAINFRAME PROTECTION” (A-15-06-16112)
Thank you for the opportunity to review and provide comments on this draft report. The report notes that the auditors did not identify any significant findings related to two of the three performance indicators included in this audit: "Enhance efforts to improve financial performance using the Managerial Cost Accountability System" and "Improve workload information using Social Security Unified Measurement System." However, the report includes significant findings related to the performance indicator "Maintain zero outside infiltrations of SSA's programmatic mainframes."
We are reviewing the performance indicator “Maintain Zero Infiltrations of SSA’s Programmatic Mainframes,” as well as the data definition and the linkage of the indicator to the Agency’s Goals and Objectives. In this regard, we are taking an in-depth look at the existing tools and techniques to determine the Agency’s ability to monitor, record and report meaningful measurements to include infiltrations of the Agency’s intranet, network and distributed systems.
We have the following comments on the report’s recommendations.
Recommendation 1
Document the policies and procedures used to prepare and disclose the results of the performance indicator.
Comment
We agree. We documented the policies and procedures used to prepare and disclose the results of the performance indicator and provided them to OIG and PricewaterhouseCoopers (PwC). PwC indicated the policies and procedures sufficiently document the processes.
Recommendation 2
Ensure the performance indicator definitions and reported results are meaningful,
complete, and consistent with the title by:
- Clearly documenting the inclusion or exclusion of internal infiltrations
in the calculation of the indicator results;
- Revising the performance indicator results to clarify that it measures only
detected infiltrations. As an example, the indicator actual performance
results could be documented as follows:
Zero outside infiltrations of SSA’s programmatic mainframes were
detected.
- Broadening the indicator data definition to include infiltrations resulting
in disclosure or misuse of sensitive SSA data; and,
- Expanding the calculation of indicator results to include infiltrations of
the Agency's intranet, network, and distributed systems.
Comment
This recommendation contains 4-items. We agree with the first item and are performing an in-depth review to ensure the performance indicator definitions and reported results are meaningful, complete and consistent.
We also agree with the second item included in this recommendation. We are revising the performance indicator to clarify that it measures only detected infiltrations.
Regarding the third item included in this recommendation, we are determining the technical aspects and feasibility of including infiltrations resulting in disclosure or misuse of sensitive data. Currently, we are unsure of the available methodologies, tools and techniques. If this section of the recommendation cannot be implemented using the existing processes, an evaluation and cost analysis will be required.
About the fourth item included in this recommendation, we are determining if, using existing technologies, methodologies and tools, the results can be measured to include infiltrations of the Agency’s intranet, network and distributed systems. If this section of the recommendation cannot be implemented using the existing processes, this will also require an evaluation and cost analysis.
Recommendation 3
Articulate and disclose the linkage of the performance indicator to the Agency’s strategic goals and objectives.
Comment
We agree. We will update the Performance and Accountability Report and Information Resources Management plan to articulate the linkage of the performance indicator to the Agency’s strategic goals and objectives.
[In addition to the comments above, SSA provided a technical comment, which has been addressed in the final report.]
Overview of the Office of the Inspector General
The Office of the Inspector General (OIG) is comprised of our Office of Investigations
(OI), Office of Audit (OA), Office of the Chief Counsel to the Inspector
General (OCCIG), and Office of Resource Management (ORM). To ensure
compliance with policies and procedures, internal controls, and professional
standards, we also have a comprehensive Professional Responsibility and Quality
Assurance program.
Office of Audit
OA conducts and/or supervises financial and performance audits of the Social
Security Administration’s (SSA) programs and operations and makes recommendations
to ensure program objectives are achieved effectively and efficiently. Financial
audits assess whether SSA’s financial statements fairly present SSA’s
financial position, results of operations, and cash flow. Performance
audits review the economy, efficiency, and effectiveness of SSA’s programs
and operations. OA also conducts short-term management and program evaluations
and projects on issues of concern to SSA, Congress, and the general public.
Office of Investigations
OI conducts and coordinates investigative activity related to fraud, waste,
abuse, and mismanagement in SSA programs and operations. This includes
wrongdoing by applicants, beneficiaries, contractors, third parties, or SSA
employees performing their official duties. This office serves as
OIG liaison to the Department of Justice on all matters relating to the investigations
of SSA programs and personnel. OI also conducts joint investigations
with other Federal, State, and local law enforcement agencies.
Office of the Chief Counsel to the Inspector General
OCCIG provides independent legal advice and counsel to the IG on various matters,
including statutes, regulations, legislation, and policy directives. OCCIG
also advises the IG on investigative procedures and techniques, as well as
on legal implications and conclusions to be drawn from audit and investigative
material. Finally, OCCIG administers the Civil Monetary Penalty program.
Office of Resource Management
ORM supports OIG by providing information resource management and systems security. ORM
also coordinates OIG’s budget, procurement, telecommunications, facilities,
and human resources. In addition, ORM is the focal point for OIG’s
strategic planning function and the development and implementation of performance
measures required by the Government Performance and Results Act of 1993.
-- footnotes follow --
Public Law Number 103-62, 107 Stat. 285 (codified as amended in scattered sections of 5 United States Code (U.S.C.), 31 U.S.C. and 39 U.S.C.).
31 U.S.C. § 1115(a)(4).
31 U.S.C. § 1115(a)(6).
GAO-03-273G, Assessing Reliability of Computer Processed Data, October 2002, p. 3.
SSA, PAR FY 2005 p. 99.
Id. p. 84.
Id. p. 98.
Id. pp. 35 and
42.
Id. p. 99.
Id. p. 35.
Id. p. 84.
SSA, PAR FY 2005 p. 99.
FIPS PUB 200, Minimum Security Requirements for Federal Information and Information Systems, March 2006, p. 7.