OFFICE OF
THE INSPECTOR GENERAL

SOCIAL SECURITY ADMINISTRATION

PERFORMANCE INDICATOR AUDIT:
MANAGEMENT INFORMATION SYSTEMS
AND MAINFRAME PROTECTION

September 2006            A-15-06-16112

Mission

By conducting independent and objective audits, evaluations and investigations, we inspire public confidence in the integrity and security of SSA’s programs and operations and protect them against fraud, waste and abuse.  We provide timely, useful and reliable information and advice to Administration officials, Congress and the public.

Authority

The Inspector General Act created independent audit and investigative units, called the Office of Inspector General (OIG).  The mission of the OIG, as spelled out in the Act, is to:

• Conduct and supervise independent and objective audits and investigations relating to agency programs and operations.
• Promote economy, effectiveness, and efficiency within the agency.
• Prevent and detect fraud, waste, and abuse in agency programs and operations.
• Review and make recommendations regarding existing and proposed legislation and regulations relating to agency programs and operations.
• Keep the agency head and the Congress fully and currently informed of problems in agency programs and operations.

     To ensure objectivity, the IG Act empowers the IG with:

• Independence to determine what reviews to perform.
• Access to all information necessary for the reviews.
• Authority to publish findings and recommendations based on the reviews.

Vision

We strive for continual improvement in SSA’s programs, operations and management by proactively seeking new ways to prevent and deter fraud, waste and abuse.  We commit to integrity and excellence by supporting an environment that provides a valuable public service while encouraging employee development and retention and fostering diversity and innovation.

Please provide within 60 days a corrective action plan that addresses each recommendation.  If you wish to discuss the final report, please call me or have your staff contact Steven L. Schaeffer, Assistant Inspector General for Audit, at
(410) 965-9700.
Patrick P. O’Carroll, Jr

Attachment

MEMORANDUM    

Date:       September 6, 2006

To:        Inspector General

From:     PricewaterhouseCoopers, LLP

Subject: Performance Indicator Audit:  Management Information Systems and Mainframe Protection (A-15-06-16112)

OBJECTIVE

The Government Performance and Results Act (GPRA) of 1993 requires the Social Security Administration (SSA) to develop performance indicators that assess the relevant service levels and outcomes of each program activity.   GPRA also calls for a description of the means employed to verify and validate the measured values used to report on program performance.  

Our audit was conducted in accordance with generally accepted government auditing standards for performance audits.  For the performance indicators included in this audit, our objectives were to:

1. Assess the effectiveness of internal controls and test critical controls over data generation, calculation, and reporting processes for the specific performance indicator.

1. Assess the overall reliability of the performance indicator’s computer processed data.  Data are reliable when they are complete, accurate, consistent and are not subject to inappropriate alteration.

1. Test the accuracy of results presented and disclosed in the Fiscal Year (FY) 2005 Performance and Accountability Report (PAR).

1. Assess if the performance indicator provides a meaningful measurement of the program it measures and the achievement of its stated objective.

BACKGROUND

We audited the following performance indicators as stated in the SSA FY 2005 PAR:

MCAS and SUMS Projects
SSA is developing two new systems to enhance the monitoring and reporting of financial and performance data.  MCAS and SUMS will be a key enabler to allow SSA to monitor and report progress toward achieving its strategic goals and objectives and tracking resource expenditures.

SSA Information Systems
SSA employees process a tremendous amount of sensitive personal data through the SSA mainframe applications on a daily basis.  To ensure the integrity and security of this data, SSA has invested heavily in the development and implementation of multiple layers of electronic security.  As a result, SSA management has implemented numerous intrusion detection and prevention controls to identify and address threats to the SSA systems.  SSA management continuously monitors the security of the SSA mainframe environment, and the networks that surrounds it.

RESULTS OF REVIEW

We did not identify any significant findings related to the internal controls, data reliability, meaningfulness, accuracy of presentation, or disclosure of the information related to the indicators "Enhance efforts to improve financial performance using Managerial Cost Accountability System (MCAS)" and "Improve workload information using Social Security Unified Measurement System (SUMS)."We identified findings related to

internal controls, meaningfulness, and accuracy of presentation and disclosure of the information contained in the PAR for the indicator "Maintain zero outside infiltrations of SSA's programmatic mainframes." 

Enhance efforts to improve financial performance using Managerial Cost Accountability System (MCAS)

Indicator Background

“MCAS focuses on critical performance and financial information needed by managers and employees, and promotes performance accountability for Social Security programs.  As stewards of the Social Security Trust Fund, SSA must also model appropriate information management processes to ensure accountability for workloads.  The Agency’s MCAS includes a number of projects designed to update the cost analysis system, reporting systems, workload measurement systems, and system access.  The integration of financial and performance management systems will allow the Agency to routinely assess performance and financial information in order for local managers to make more timely and efficient day-to-day decisions."

The SUMS/MCAS project plan tracking and releases as reported to the SUMS/MCAS Executive Steering Committee are the data sources for this calculation.

Findings

We did not identify any significant findings related to the internal controls, data reliability, accuracy of presentation, meaningfulness, or disclosure of the information related to this indicator contained in the PAR.

Improve workload information using Social Security Unified Measurement Systems (SUMS)

Indicator Background

"The Agency has recognized the need to improve the quality, consistency and access to information that is used by managers and analysts throughout SSA to manage work and account for resources.  The objective of SUMS is to create a system for SSA operational components that counts and measures all work in a consistent manner regardless of where the work is processed.  This system provides access to information needed to meet changing business requirements, support process reviews and comply with government standards.  Access to web based reports and workload control listings and other information are available on demand, eliminating the need for paper reports."

SUMS is considered a key enabler in monitoring and reporting on SSA's progress toward achieving its strategic goals and objectives and tracking resource expenditures.  The objective of this system is to count and to measure work in a consistent manner at all organizational levels.  It provides the detailed information that managers need to monitor service, forecast workloads, and make informed decisions on how best to manage work and resources.

The SUMS/MCAS project plan tracking and releases as reported to the SUMS/MCAS Executive Steering Committee are the data sources for this calculation.

Findings

We did not identify any significant findings related to the internal controls, data reliability, accuracy of presentation, meaningfulness, or disclosure of the information related to this indicator contained in the PAR.

Maintain zero outside infiltrations of SSA's programmatic mainframes

Indicator Background

SSA maintains an Intrusion Protection Team (IPT) specifically created to prevent outside infiltrations of systems.  The IPT uses numerous software tools to immediately detect attempts to infiltrate SSA’s network and underlying systems.  Additionally, software controls at all levels of SSA systems are used to prevent unauthorized access to SSA systems. 

SSA created this performance indicator to document the Agency’s success in protecting the mainframe computers, on which SSA’s sensitive programmatic data resides.  According to SSA security management and the PAR, the indicator is intended to measure infiltrations from outside of SSA, and not infiltrations from authorized internal users who manage to elevate their privileges and perform unauthorized actions.  In addition, an infiltration is further defined as “…unauthorized access that requires a cleanup or restoration of backup files to a state prior to the infiltration.”  Also the indicator is intended to only measure infiltrations of the mainframe computers.  Infiltrations that are related to non-mainframe systems, including SSA’s Intranet, network, and distributed systems are excluded for reporting purposes within this indicator.

The count of mainframe infiltrations is maintained in the Change Asset and Problem Reporting System (CAPRS).

Findings

Internal Controls and Data Reliability

We found the policies and procedures related to the formal process to capture, store, and calculate the results of the performance indicator were not adequate.  The documentation did not accurately describe the process in place during FY 2005 and all components of the indicator calculation were not included.  Office of Management and Budget (OMB) Circular A-123, Management Accountability and Control, requires, "...documentation for transactions, management controls, and other significant events must be clear and readily available for examination. …"

It should be noted that SSA management was in the process of updating the documentation related to this indicator during the time of the audit.  As the calculation of this indicator is not based on computerized data, we did not complete an analysis of data reliability.

Accuracy of PAR Presentation and Disclosure

The intent of the indicator is to highlight SSA’s success in preventing mainframe infiltrations.  We believe this is an important goal and its success is very relevant to the Agency.  However, it is not possible to state that undetected infiltrations did not occur.  Therefore, management cannot measure or fully assert that an outside infiltration has not occurred. 

We also noted inconsistencies in the descriptions of the indicator.  Based on the title of the indicator, internal infiltrations would not be included in the calculation of this indicator; however, the definition, as described in the FY 2005 PAR, is unclear with regard to inclusion of internal infiltrations:

An infiltration is an unauthorized access that requires a cleanup or restoration of back-up files to a state prior to the infiltration.  This measure is a count of the number of times that an infiltration of mainframes is detected. (emphasis added)

Finally, we believe that the data definition too narrowly defines a mainframe infiltration and could omit important events such as unauthorized access which results in disclosure of sensitive SSA information or misuse of data that occurs but does not require clean up or restoration activities.  The Federal Information Processing Standards Publication (FIPS PUB) 200 defines an incident as

An occurrence that actually or potentially jeopardizes the confidentiality, integrity, or availability of an information system or the information the system processes, stores, or transmits or that constitutes a violation or imminent threat of violation of security policies, security procedures, or acceptable use policies. 

Additionally, the indicator excludes infiltrations of SSA’s Intranet, network and distributed systems which maintain important Agency information.

Performance Indicator Meaningfulness

SSA management does not provide a clear statement in the PAR of how preventing outside infiltrations of the mainframe relates to the Agency goal “To ensure superior Stewardship of Social Security programs and resources,” or the Agency objective of “Efficiently manage Agency finances and assets, and effectively link resources to performance outcomes.”

CONCLUSION AND RECOMMENDATIONS

SSA management indicated that the performance indicator “Maintain Zero Outside Infiltrations of SSA’s Programmatic Mainframes” will be significantly updated in the 2006 PAR.  As such we recommend SSA:

1. Document the policies and procedures used to prepare and disclose the results of the performance indicator.

2. Ensure the performance indicator definitions and reported results are meaningful, complete, and consistent with the title by:

• Clearly documenting the inclusion or exclusion of internal infiltrations in the calculation of the indicator results;

• Revising the performance indicator results to clarify that it measures only detected infiltrations.  As an example, the indicator actual performance results could be documented as follows:

Zero outside infiltrations of SSA’s programmatic mainframes were detected.

• Broadening the indicator data definition to include infiltrations resulting in disclosure or misuse of sensitive SSA data; and,

• Expanding the calculation of indicator results to include infiltrations of the Agency's intranet, network, and distributed systems.

1. Articulate and disclose the linkage of the performance indicator to the Agency’s strategic goals and objectives.

AGENCY COMMENTS

SSA agreed with our recommendations.  See Appendix D for the Agency’s comments.

Appendices

APPENDIX A – Acronyms

APPENDIX B – Scope and Methodology

APPENDIX C – Process Flowcharts

APPENDIX D – Agency Comments

Appendix A -- Acronyms

Appendix B -- Scope and Methodology

We updated our understanding of the Social Security Administration’s (SSA) Government Performance and Results Act (GPRA) processes.  This was completed through research and inquiry of SSA management.  We also requested SSA to provide various documents regarding the specific programs being measured as well as the specific measurement used to assess the effectiveness and efficiency of the related program. 

• Reviewed prior SSA, Government Accountability Office, and other reports related to SSA GPRA performance and related information systems.
• Reviewed applicable laws, regulations and SSA policy.
• Met with the appropriate SSA personnel to confirm our understanding of each individual performance indicator. 
• Flowcharted the processes.  (See Appendix C).
• Tested key controls related to manual or basic computerized processes (e.g., spreadsheets, databases, etc.).
• Conducted and evaluated tests of the automated and manual controls within and surrounding each of the critical applications to determine whether the tested controls were adequate to provide and maintain reliable data to be used when measuring the specific indicator.
• Identified attributes, rules, and assumptions for each defined data element or source document.
• Recalculated the metric or algorithm of key performance indicators to ensure mathematical accuracy.
• For those indicators with results that SSA determined using computerized data, we assessed the completeness and accuracy of that data to determine the data's reliability as it pertains to the objectives of the aud

Through inquiry, observation, and other substantive testing, including testing of source documentation, we performed the following:

As part of this audit, we documented our understanding, as conveyed to us by Agency personnel, of the alignment of the Agency’s mission, goals, objectives, processes, and related performance indicators.  We analyzed how these processes interacted with related processes within SSA and the existing measurement systems.  Our understanding of the Agency’s mission, goals, objectives, and processes were used to determine if the performance indicators being used appear to be valid and appropriate given our understanding of SSA’s mission, goals, objectives and processes.

We followed all performance audit standards in accordance with generally accepted government auditing standards.  In addition to the steps above, we specifically performed the following to test the indicators included in this report:
Management Information Systems, Management Cost Accountability Systems (MCAS) and Social Security Unified Measurement Systems (SUMS)

• Reviewed documentation related to project development, implementation and management activities.
• Reviewed the projects to determine whether they were developed in accordance with Agency policies regarding application software development.
• Reviewed each of the projects and, as applicable, found they were released into production during the time frame reported in the Fiscal Year 2005 Performance and Accountability Report by obtaining their software release documentation and/or observing the use of the system in production

Maintain zero outside infiltrations of SSA’s programmatic mainframeS

• Assessed the reliability of the data by inquiring of appropriate personnel as to the sources of the data included on, and the process for reviewing, the United States Computer Emergency Readiness Team (US-CERT) reports.
• Reviewed the cumulative, September 30, 2005, US-CERT report
• Interviewed various SSA personnel (including the Intrusion Protection Team, SSA Security Response Team, Chief Security Officer, Virtual Private Network and Modems Administration and Support teams, Top Secret Administrators and Security Officer) responsible for protecting the mainframe to gain an understanding of the tools and processes implemented to protect, monitor and report on SSA’s systems security.
• Performed (during SSA’s FY 2005 Financial Statement Audit) penetration testing, firewall assessments, mainframe operating system and Top Secret configuration reviews.

Appendix C -- Flowchart of Management Information Systems, MCAS and SUMS

(flowchat graphic eliminated from text-only version of this document)

Management Information Systems, MCAS and SUMS

• Portions of the MCAS and SUMS application were developed prior to FY 2005.
• The current year goal is reported in the APP.
• SSA management approves the goal and the APP is published.
• Each application scheduled for completion during the current year is planned, tested, and completed, as appropriate.
• Each completed application is moved into production.
• Milestone completed is reported to the OCSO.
• The percent complete of the overall goal is updated to reflect the application that has been completed and is in production.
• Meetings are held to discuss overall progress of projects as well as deviations which are included on the monthly tracking report.  These meetings include:  Bi-weekly project manger meetings, Monthly Executive Steering Committee Meeting, Monthly meeting with the DCS, Monthly Executive Staff meeting with the Commissioner.
• Updated completion calculation is reported in the PAR for the MCAS and SUMS indicators.

Flowchart and Mainframe Protection

(flowchat graphic eliminated from text-only version of this document)

Mainframe Protection

• Activity occurs through outside points of entry.  This includes the VPN and vendor access.
• SSA and IBM monitor port settings (including remote access points), the Internet firewalls, the Intranet, and the DMZ.
• Is activity unusual or suspicious?
  • No - No action is taken.
  • Yes - Alert is forwarded to the IPT.
• CAPRS ticket is created by IPT and/or SRT.
• Did mainframe infiltration occur?
  • No - Document activity noted from the alert and reasoning behind decision in CAPRS and close ticket.
  • Yes - Mainframe infiltration is contained and received by IPT and CAPRS ticket is closed.
• Risk Management Program personnel, IPT, and SRT meet with management to discuss security of SSA systems on a monthly basis.  In addition, a monthly incident report is produced for management and US-CERT.
• Infiltration included on US-CERT report; if no infiltrations are noted, this is recorded on the report.
• OCIO provides the number of mainframe infiltrations detected to OCSO on a monthly basis and at year end.
• Results of the indicator are reported in the PAR on an annual basis.

Appendix D -- Agency Comments

SOCIAL SECURITY

We appreciate OIG’s efforts in conducting this review.  Our comments on the draft report are attached.

Please let me know if you have any questions.  Staff inquiries may be directed to
Ms. Candace Skurnik, Director, Audit Management and Liaison Staff, at extension 54636.

Attachment:
SSA Comments

COMMENTS ON THE OFFICE OF THE INSPECTOR GENERAL’S (OIG) DRAFT REPORT, “PERFORMANCE INDICATOR AUDIT:  MANAGEMENT INFORMATION SYSTEMS AND MAINFRAME PROTECTION” (A-15-06-16112)

Thank you for the opportunity to review and provide comments on this draft report.  The report notes that the auditors did not identify any significant findings related to two of the three performance indicators included in this audit: "Enhance efforts to improve financial performance using the Managerial Cost Accountability System" and "Improve workload information using Social Security Unified Measurement System."  However, the report includes significant findings related to the performance indicator "Maintain zero outside infiltrations of SSA's programmatic mainframes." 

We are reviewing the performance indicator “Maintain Zero Infiltrations of SSA’s Programmatic Mainframes,” as well as the data definition and the linkage of the indicator to the Agency’s Goals and Objectives.  In this regard, we are taking an in-depth look at the existing tools and techniques to determine the Agency’s ability to monitor, record and report meaningful measurements to include infiltrations of the Agency’s intranet, network and distributed systems. 

We have the following comments on the report’s recommendations.

Recommendation 1

Document the policies and procedures used to prepare and disclose the results of the performance indicator.

Comment

We agree.  We documented the policies and procedures used to prepare and disclose the results of the performance indicator and provided them to OIG and PricewaterhouseCoopers (PwC).  PwC indicated the policies and procedures sufficiently document the processes.  

Recommendation 2

Ensure the performance indicator definitions and reported results are meaningful, complete, and consistent with the title by:
- Clearly documenting the inclusion or exclusion of internal infiltrations in the calculation of the indicator results;
- Revising the performance indicator results to clarify that it measures only detected infiltrations.  As an example, the indicator actual performance results could be documented as follows:
Zero outside infiltrations of SSA’s programmatic mainframes were detected.
- Broadening the indicator data definition to include infiltrations resulting in disclosure or misuse of sensitive SSA data; and,
- Expanding the calculation of indicator results to include infiltrations of the Agency's intranet, network, and distributed systems.

Comment

This recommendation contains 4-items.  We agree with the first item  and are performing an in-depth review to ensure the performance indicator definitions and reported results are meaningful, complete and consistent.

We also agree with the second item included in this recommendation.  We are revising the performance indicator to clarify that it measures only detected infiltrations.

Regarding the third item included in this recommendation, we are determining the technical aspects and feasibility of including infiltrations resulting in disclosure or misuse of sensitive data.  Currently, we are unsure of the available methodologies, tools and techniques.  If this section of the recommendation cannot be implemented using the existing processes, an evaluation and cost analysis will be required. 

About the fourth item included in this recommendation, we are determining if, using existing technologies, methodologies and tools, the results can be measured to include infiltrations of the Agency’s intranet, network and distributed systems.  If this section of the recommendation cannot be implemented using the existing processes, this will also require an evaluation and cost analysis. 

Recommendation 3

Articulate and disclose the linkage of the performance indicator to the Agency’s strategic goals and objectives.

Comment

We agree.  We will update the Performance and Accountability Report and Information Resources Management plan to articulate the linkage of the performance indicator to the Agency’s strategic goals and objectives. 

[In addition to the comments above, SSA provided a technical comment, which has been addressed in the final report.]

Overview of the Office of the Inspector General
The Office of the Inspector General (OIG) is comprised of our Office of Investigations (OI), Office of Audit (OA), Office of the Chief Counsel to the Inspector General (OCCIG), and Office of Resource Management (ORM).  To ensure compliance with policies and procedures, internal controls, and professional standards, we also have a comprehensive Professional Responsibility and Quality Assurance program.

Office of Audit
OA conducts and/or supervises financial and performance audits of the Social Security Administration’s (SSA) programs and operations and makes recommendations to ensure program objectives are achieved effectively and efficiently.  Financial audits assess whether SSA’s financial statements fairly present SSA’s financial position, results of operations, and cash flow.  Performance audits review the economy, efficiency, and effectiveness of SSA’s programs and operations.  OA also conducts short-term management and program evaluations and projects on issues of concern to SSA, Congress, and the general public.

Office of Investigations
OI conducts and coordinates investigative activity related to fraud, waste, abuse, and mismanagement in SSA programs and operations.  This includes wrongdoing by applicants, beneficiaries, contractors, third parties, or SSA employees performing their official duties.  This office serves as OIG liaison to the Department of Justice on all matters relating to the investigations of SSA programs and personnel.  OI also conducts joint investigations with other Federal, State, and local law enforcement agencies.

Office of the Chief Counsel to the Inspector General
OCCIG provides independent legal advice and counsel to the IG on various matters, including statutes, regulations, legislation, and policy directives.  OCCIG also advises the IG on investigative procedures and techniques, as well as on legal implications and conclusions to be drawn from audit and investigative material.  Finally, OCCIG administers the Civil Monetary Penalty program.

Office of Resource Management
ORM supports OIG by providing information resource management and systems security.  ORM also coordinates OIG’s budget, procurement, telecommunications, facilities, and human resources.  In addition, ORM is the focal point for OIG’s strategic planning function and the development and implementation of performance measures required by the Government Performance and Results Act of 1993.

-- footnotes follow --