OFFICE OF
THE INSPECTOR GENERAL

SOCIAL SECURITY ADMINISTRATION

The Social Security Administration’s Post-Implementation Review Process

June 2010

A-14-10-30105

QUICK RESPONSE
EVALUATION

MEMORANDUM

Date: June 22, 2010 Refer To:

To: The Commissioner

From: Inspector General

Subject: The Social Security Administration’s Post-Implementation Review Process
(A-14-10-30105)

The attached final quick response evaluation presents the results of our review. Our objective was to assess the Social Security Administration’s Post-Implementation Review Framework and provide matters of consideration.

If you wish to discuss the final report, please call me or have your staff contact Steven L. Schaeffer, Assistant Inspector General for Audit, at (410) 965-9700.

/s/

Patrick P. O’Carroll, Jr.

June 2010

Mission

By conducting independent and objective audits, evaluations and investigations, we inspire public confidence in the integrity and security of SSA’s programs and operations and protect them against fraud, waste and abuse. We provide timely, useful and reliable information and advice to Administration officials, Congress and the public.

Authority

The Inspector General Act created independent audit and investigative units, called the Office of Inspector General (OIG). The mission of the OIG, as spelled out in the Act, is to:

 Conduct and supervise independent and objective audits and investigations relating to agency programs and operations.
 Promote economy, effectiveness, and efficiency within the agency.
 Prevent and detect fraud, waste, and abuse in agency programs and operations.
 Review and make recommendations regarding existing and proposed legislation and regulations relating to agency programs and operations.
 Keep the agency head and the Congress fully and currently informed of problems in agency programs and operations.

To ensure objectivity, the IG Act empowers the IG with:

 Independence to determine what reviews to perform.
 Access to all information necessary for the reviews.
 Authority to publish findings and recommendations based on the reviews.

Vision

We strive for continual improvement in SSA’s programs, operations and management by proactively seeking new ways to prevent and deter fraud, waste and abuse. We commit to integrity and excellence by supporting an environment that provides a valuable public service while encouraging employee development and retention and fostering diversity and innovation.

Background
OBJECTIVE

The objective was to assess the Social Security Administration’s (SSA) Post-Implementation Review (PIR) Framework and provide matters of consideration.

BACKGROUND

The Office of Management and Budget (OMB) defines a PIR as a diagnostic tool to evaluate the overall effectiveness of an agency’s capital planning and acquisition process. A PIR should be conducted on completed and terminated projects by an independent review team. OMB Circular A-130, Management of Federal Information Resources, requires that Federal agencies "…conduct post-implementation reviews of information systems and information resources management processes to validate estimated benefits and costs, and document effective management practices for broader use." In addition, OMB recommended that agencies consider various factors when conducting PIRs, including strategic and mission impact and effectiveness; customer and user satisfaction; investment performance; and evaluations of accuracy, timeliness, and quality of project information. The objectives of a PIR are to
• identify how accurately a capital investment project meets the agency’s objectives, expected benefits, and the strategic goals;
• ensure continual improvement of the agency's capital programming process based on lessons learned; and
• minimize the risk of repeating mistakes by providing quality services to business partners and customers.

In addition to OMB guidance, a Government Accountability Office (GAO) executive guide provides an Information Technology Investment Management framework to evaluate and assess how well an agency is selecting and managing its information technology (IT) resources. The guide incorporates accepted or best practices in IT investment management, as well as the reported experiences of Federal agencies and other organizations in creating their own investment management process.
In July 2007, we issued a report that stated SSA's Office of the Chief Information Officer (OCIO) had a PIR policy that generally met OMB’s PIR requirements. Although the policy was in place, SSA was not conducting PIRs. In addition, efforts by various Agency components to evaluate IT projects were not coordinated or integrated to form a system that, as a whole, would meet OMB’s PIR requirements.
In September 2008, GAO issued a report on whether SSA’s investment management approach was consistent with leading investment management best practices. GAO concluded that SSA had established most (82 percent) of the basic practices needed to manage its projects as investments, including many of the foundational practices for selecting and controlling IT investments. However, in reference to SSA’s PIR process, GAO reported that the Agency had not implemented all the policies and procedures for the key practices. For example, the Agency did not evaluate quantitative data, which limited its ability to determine whether investments met benefit expectations or identify lessons learned for improving the investment management process.

In April 2009, SSA contracted with Booz Allen Hamilton to develop a framework for conducting PIRs (referred to as the PIR Framework). Booz Allen Hamilton developed the Framework and used it to perform a PIR on SSA’s iClaim application. Agency staff stated that the Framework was a work in progress and had not been formally approved. Although the Framework was not formally approved, SSA continued to perform PIRs using it.

For additional Background and Scope and Methodology, see Appendix B.

Results of Review
In our 2007 report, we stated SSA’s PIR policy generally met OMB’s requirements. However, our current evaluation found SSA’s PIR process, as described in its PIR Framework, needed enhancements to provide an effective PIR process that meets Federal and SSA requirements. We identified three areas in which the Framework needed improvement.

1. The Framework should include all PIR requirements.
2. The Framework should have incorporated some of the more common policies and procedures identified by GAO to ensure an effective and consistent PIR process.
3. The Framework should integrate with SSA’s IT Capital Planning and Investment Control (CPIC) and Project Management processes.

SSA’s Framework Should Include all PIR Requirements

SSA’s PIR Framework needed to be enhanced to meet Federal and Agency PIR requirements. OMB Circular A-130 requires that Federal agencies conduct PIRs to validate estimated benefits and costs and document effective management practices. SSA’s PIR policy generally met OMB’s requirements (see a summary of SSA’s PIR policy in Appendix B). However, SSA’s PIR Framework narrowly focused on validating service requirements and performance metrics and excluded all other areas for performing an effective PIR, as required by OMB guidance and SSA policy.

To meet OMB requirements, SSA’s PIR Framework should include the following steps:

• Conduct PIRs also for terminated projects.
• Compare estimated project costs to actual costs.
• Validate planned functionality and evaluate technical capability.
• Validate anticipated benefits, such as cost savings.
• Evaluate mission and program impact.
• Evaluate customer and user satisfaction.
• Identify gaps or deficiencies in the process used to develop and implement the investment.
• Provide lessons learned for improving future decision-making processes.

In addition to the above steps to address Federal requirements, SSA PIR Framework should also incorporate the following Agency requirements.

• Identify reasons for not achieving projected benefits.
• Evaluate and validate the original business assumptions.
• Determine how well the project met time schedules and implementation dates.
• Use SSA existing financial and project managements systems and information for conducting PIR.

Without incorporating these requirements, SSA’s PIR Framework would not provide a comprehensive evaluation of how effective and efficient an IT project was implemented and managed and whether it delivered expected benefits within budget and expected timeframes.

SSA should consider enhancing its PIR Framework to meet Federal requirements, as specified in OMB Circular A-130, and the Capital Programming Guide. The primary objectives of SSA’s PIR Framework should
• identify how accurately a capital investment project meets the Agency’s objectives, expected benefits, and the strategic goals;
• ensure continual improvement of the Agency's capital programming process based on lessons learned; and,
• minimize the risk of repeating mistakes by providing quality services to business partners and customers.

SSA’s OCIO staff stated the Agency plans to incorporate more elements into its Framework as the Framework matures.

The Framework Should Incorporate Some of the More Common Policies and Procedures as Identified by GAO to Ensure an Effective PIR Process.

SSA’s Framework needed to be enhanced to incorporate some of the more common policies and procedures for conducting an effective PIR, as described by GAO. SSA’s PIR policy has incorporated some of these more common policies and procedures, but its PIR Framework only provided a high-level description of the PIR process. The following table compares GAO’s common policies and procedures to SSA’s PIR policy and Framework.

GAO’s Common Policies and Procedures Compared to
SSA’s PIR Policy and Framework
GAO’s Common Policies and Procedures for an Effective PIR
SSA PIR
Policy
SSA’s PIR
Framework
Who conducts and participates in a PIR SSA policy requires a competent and objective team to conduct a PIR.

Policy was silent on PIR participants. The Framework was silent on this attribute.
Types and sizes of investments for which a PIR is conducted SSA policy stated that a PIR is assigned by the Chief Information Officer at project approval.

Policy was silent on the types and sizes of investments that require a PIR. The Framework was silent on this attribute.
Appropriate timing to conduct a PIR
SSA policy stated that a PIR is normally conducted 3 to 12 months after the system becomes operational, but does not provide guidance for the timing of conducting a PIR for long-term projects.

Policy needs to clarify what the term “operational” means for long-term projects. The Framework was silent on this attribute. SSA’s PIR Framework was developed based on the iClaim project, which has multiple releases.

The Framework needs to clarify how to conduct a PIR on a system with multiple releases.
What information is presented in a PIR SSA policy prescribes what information should be provided by a PIR. See Appendix B. The Framework narrowly focuses on validating service requirements and performance metrics and excludes all other areas documented in SSA’s policy.
The criteria and procedures for tailoring the standard PIR process SSA policy was silent on this attribute. The Framework was silent on this attribute.
How conclusions, lessons learned, and recommended management action steps are to be disseminated to executives and others SSA policy was silent on this attribute. The Framework was silent on this attribute.

According to GAO, an organization needs to document its policies and procedures for conducting PIRs. SSA should consider incorporating GAO’s more common policies and procedures in its PIR policy and Framework. For example, SSA needs to determine how to divide the long life-cycle projects, such as Title II System, into manageable segments so a PIR can be conducted in an appropriate timeframe. By incorporating these common policies and procedures in its PIR policies and Framework, the Agency will conduct more efficient and effective PIRs.

SSA’s PIR Framework Needed to Fully Use Existing CPIC and Project Management Documents, Information, and Processes

SSA’s PIR Framework did not fully use existing CPIC and Project Management documents, information, and processes. OMB’s Capital Programming Guide requires that agencies ensure each asset is evaluated consistently. In addition, the organization should have a documented methodology for conducting PIRs. The methodology chosen must be in alignment with the organization’s planning process and build on the organization’s experiences.

SSA has an existing process, Post Release Review (PRR), for validating a project’s technical requirements, functionality, and customer satisfaction. SSA also has established cost accounting and project management systems to track IT project budgets and some actual costs. The Office of Systems’ (OS) Systems Planning and Reporting System (SPARS) provides a repository for all IT planning proposals and related project information, such as resource estimates, dates, and history of changes. OS also uses the Resource Accounting System (RAS) to track actual labor hours in OS. In addition, SSA has implemented an Earned Value Management (EVM) system to track cost and schedule performance for its major IT investment projects. Although SSA has processes readily available to assist in performing a PIR, the Agency’s PIR Framework does not fully use the documents and information obtained through these systems. For example, because of the limited scope of the current PIR Framework, labor costs, and other cost and budget data already documented in and accumulated through the Agency’s PRR, SPARS, RAS, and EVM system may not be included in PIR results.

Therefore, to meet all Federal and Agency PIR requirements, SSA should consider enhancing its Framework to provide additional user guidance on which documents and information should be considered and analyzed in the PIR process. Further, the Framework should use, to the extent possible, existing evaluation processes and information systems. SSA should consider expanding the capability of the OS systems to better meet its PIR needs. For example, the OS RAS only tracks OS and certain contractors’ actual labor hours. SSA should consider expanding RAS to all Agency components to capture all costs related to an IT project.

Matters for Consideration
Federal agencies are required to effectively manage their capital assets to ensure scarce public resources are spent wisely. A PIR is not only a tool to evaluate how effectively an IT project meets Agency goals; it is also a tool that should be used for identifying reasons for project failures. Further, PIRs are important to ensure continuous improvement in SSA’s IT investment decision and management processes and help avoid repeating mistakes in future IT projects.

SSA has implemented numerous IT projects to assist the Agency in meeting its mission and goals. Without an effective PIR process, SSA will be unable to validate estimated benefits and costs and document effective management practices for its IT projects. In the past, SSA terminated some important projects; however, no comprehensive reviews had been conducted to determine why these projects failed.

Our review found that, although OMB issued and documented PIR requirements, it has not provided guidance on the process of conducting effective PIRs. We commend the Agency’s efforts to develop a PIR policy and process. However, based on our review, we are providing the Agency suggestions that we believe will help improve the Agency’s PIR process and ensure its success. The Agency should consider (1) incorporating all Federal requirements into its PIR Framework; (2) incorporating GAO’s more common PIR policies and procedures to help ensure each IT investment is evaluated consistently and PIRs are conducted efficiently and effectively; and (3) integrating existing CPIC and Project Management information into its PIR process.

Appendices
APPENDIX A – Acronyms
APPENDIX B – Background and Scope and Methodology
APPENDIX C - OIG Contacts and Staff Acknowledgments

Appendix A
Acronyms
CPIC Capital Planning and Investment Control
EVM Earned Value Management
GAO Government Accountability Office
IT Information Technology
OCIO Office of the Chief Information Officer
OIG Office of the Inspector General
OMB Office of Management and Budget
OS Office of Systems
PIR
Post Implementation Review
PRR Post Release Review
RAS
Resource Accounting System
SPARS

SSA Special Publication

Systems Planning and Reporting System
Social Security Administration
Capital Planning and Investment Control
Earned Value Management
Government Accountability Office
Information Technology
Office of the Chief Information Officer
Office of the Inspector General
Office of Management and Budget
Office of Systems
Post Implementation Review
Post Release Review
Resource Accounting System
Systems Planning and Reporting System
Social Security Administration

Appendix B
Background and Scope and Methodology

The Social Security Administration’s Post-Implementation Review Policy

The Social Security Administration’s (SSA) policy requires a Post-Implementation Review to produce the following.
• Assessment of the project’s effectiveness in meeting the original objectives.
• Determination of the project benefits that have been achieved, whether achieved benefits match projected benefits, and the reasons for any discrepancies.
• Evaluations of whether the original business assumptions used to justify the project were valid.
• Comparison of the actual costs incurred against projected costs, using the Agency’s official financial accounting, cost allocation, and budgeting systems to verify the information.
• Determination of how well the project met time schedules and implementation dates.
• Assessment of technical capability (for example, conformance to recognized systems development methodology, architecture compliance, contractor performance, and oversight).
• Identification of all decisions, changes, actions, and results that occurred throughout the project’s life cycle, as well as other relevant project information, such as the business case, updated cost-benefit analyses, and Earned Value Management (EVM) System documentation.
• Determination of management and user perspectives on the project.
• Evaluation of issues that still require attention.
 
• Documentation of lessons learned and providing insights to improve the decision-making and oversight in its Information Technology Capital Planning and Investment Control
Scope and Methodology
To accomplish our objectives, we reviewed the following applicable Federal laws, Office of Management and Budget (OMB) guidance, and Government Accountability Office (GAO) common policies and procedures:
 Clinger Cohen Act of 1996
 OMB Circular A-130, Management of Federal Information Resources,
November 28, 2000
 OMB Capital Programming Guide Version 2.0, June 2006
 GAO Executive Guide, Information Technology Investment Management, A Framework for Assessing and Improving Process Maturity, GAO-04-394G
We also obtained, reviewed, and compared the following SSA documents against the above referenced criteria:

 DRAFT - SSA’s Post-Implementation Review Framework, August 2009.
 Social Security Administration FY 2010 Information Technology Capital Plan, September 2008.
 Social Security Administration FY 2010 Information Technology Capital Planning and Investment Control Process, February 2009.
Further, we interviewed personnel from SSA’s Office of the Chief Information Officer and reviewed an Office of the Inspector General report, Social Security Administration's Management of Information Technology Projects (A-14-07-17099), July 26, 2007.

The results of our review are based on the above information provided by SSA. We performed our review during January through April 2010 in Baltimore, Maryland. The entity reviewed was the Office of the Chief Information Officer. We conducted our review in accordance with the Council of the Inspectors General on Integrity and Efficiency’s Quality Standards for Inspections.

Appendix C
OIG Contacts and Staff Acknowledgments
OIG Contacts

Brian Karpe, Division Director, Information Technology Audit Division

Grace Chi, Acting Audit Manager

Acknowledgments

In addition to those named above:

Tina Nevels, Auditor-in-Charge

For additional copies of this report, please visit our web site at www.socialsecurity.gov/oig or contact the Office of the Inspector General’s Public Affairs Staff Assistant at (410) 965-4518. Refer to Common Identification Number
A-14-10-30105.

DISTRIBUTION SCHEDULE

Commissioner of Social Security
Office of Management and Budget, Income Maintenance Branch
Chairman and Ranking Member, Committee on Ways and Means
Chief of Staff, Committee on Ways and Means
Chairman and Ranking Minority Member, Subcommittee on Social Security
Majority and Minority Staff Director, Subcommittee on Social Security
Chairman and Ranking Minority Member, Committee on the Budget, House of Representatives
Chairman and Ranking Minority Member, Committee on Oversight and Government Reform
Chairman and Ranking Minority Member, Committee on Appropriations, House of Representatives
Chairman and Ranking Minority, Subcommittee on Labor, Health and Human Services, Education and Related Agencies, Committee on Appropriations,
House of Representatives
Chairman and Ranking Minority Member, Committee on Appropriations, U.S. Senate
Chairman and Ranking Minority Member, Subcommittee on Labor, Health and Human Services, Education and Related Agencies, Committee on Appropriations, U.S. Senate
Chairman and Ranking Minority Member, Committee on Finance
Chairman and Ranking Minority Member, Subcommittee on Social Security Pensions and Family Policy
Chairman and Ranking Minority Member, Senate Special Committee on Aging
Social Security Advisory Board

Overview of the Office of the Inspector General
The Office of the Inspector General (OIG) is comprised of an Office of Audit (OA), Office of Investigations (OI), Office of the Counsel to the Inspector General (OCIG), Office of External Relations (OER), and Office of Technology and Resource Management (OTRM). To ensure compliance with policies and procedures, internal controls, and professional standards, the OIG also has a comprehensive Professional Responsibility and Quality Assurance program.
Office of Audit
OA conducts financial and performance audits of the Social Security Administration’s (SSA) programs and operations and makes recommendations to ensure program objectives are achieved effectively and efficiently. Financial audits assess whether SSA’s financial statements fairly present SSA’s financial position, results of operations, and cash flow. Performance audits review the economy, efficiency, and effectiveness of SSA’s programs and operations. OA also conducts short-term management reviews and program evaluations on issues of concern to SSA, Congress, and the general public.
Office of Investigations
OI conducts investigations related to fraud, waste, abuse, and mismanagement in SSA programs and operations. This includes wrongdoing by applicants, beneficiaries, contractors, third parties, or SSA employees performing their official duties. This office serves as liaison to the Department of Justice on all matters relating to the investigation of SSA programs and personnel. OI also conducts joint investigations with other Federal, State, and local law enforcement agencies.
Office of the Counsel to the Inspector General
OCIG provides independent legal advice and counsel to the IG on various matters, including statutes, regulations, legislation, and policy directives. OCIG also advises the IG on investigative procedures and techniques, as well as on legal implications and conclusions to be drawn from audit and investigative material. Also, OCIG administers the Civil Monetary Penalty program.
Office of External Relations
OER manages OIG’s external and public affairs programs, and serves as the principal advisor on news releases and in providing information to the various news reporting services. OER develops OIG’s media and public information policies, directs OIG’s external and public affairs programs, and serves as the primary contact for those seeking information about OIG. OER prepares OIG publications, speeches, and presentations to internal and external organizations, and responds to Congressional correspondence.
Office of Technology and Resource Management
OTRM supports OIG by providing information management and systems security. OTRM also coordinates OIG’s budget, procurement, telecommunications, facilities, and human resources. In addition, OTRM is the focal point for OIG’s strategic planning function, and the development and monitoring of performance measures. In addition, OTRM receives and assigns for action allegations of criminal and administrative violations of Social Security laws, identifies fugitives receiving benefit payments from SSA, and provides technological assistance to investigations.