Office of the Inspector General
Jo Anne B. Barnhart
Commissioner

Inspector General

Review of the Social Security Administration’s Cost Effectiveness Measurement System (A-07-00-10028)

The attached final report presents the results of our audit. Our objective was to assess the accuracy and use of the Social Security Administration’s Cost Effectiveness Measurement System data.

Please comment within 60 days from the date of this memorandum on corrective action taken or planned on each recommendation. If you wish to discuss the final report, please call me or have your staff contact Steven L. Schaeffer, Assistant Inspector General for Audit, at (410) 965-9700.

James G. Huse, Jr.

Vision

Office of the Inspector General

Jo Anne B. Barnhart
Commissioner

Inspector General

Review of the Social Security Administration’s Cost Effectiveness Measurement System (A-07-00-10028)

Our objective was to assess the accuracy and use of the Social Security Administration’s (SSA) Cost Effectiveness Measurement System (CEMS) data.

In September 1982, SSA developed CEMS as a tool to measure the operating costs and cost effectiveness of each State Disability Determination Services (DDS). CEMS was also envisioned to provide the basis for establishing formal cost standards for DDSs. In 1997, SSA began revising CEMS to take advantage of technological enhancements and to make the system more user-friendly. SSA spent approximately $690,000 to revise CEMS, and it was released for production in August 1999. SSA also took responsibility for CEMS maintenance, which was previously performed by an outside contractor at an annual cost of approximately $300,000. In Fiscal Year (FY) 1999, SSA spent an estimated $106,000 to maintain CEMS (see Appendix D).

SSA’s instructions require DDSs to: (1) input data into CEMS by the 45th day after the close of each quarter; (2) reconcile CEMS and Form SSA-4513 (Report of Obligations) data; and (3) provide CEMS reconciliation documents to the Regional Offices (RO) who, in turn, provide the documents to the CEMS Administrator in SSA’s Central Office (see Appendix C).

Our audit was designed to review CEMS data for FYs 1999 and 2000, which should have been available by May 2000 and May 2001, respectively. However, as of June 2001, CEMS data for FYs 1999 and 2000 were not available. Therefore, we reviewed the latest CEMS data available, which was for FY 1998.

To achieve our objective we:

• reviewed sections of SSA’s Program Operations Manual System (POMS);

• reviewed the CEMS Financial Procedures Handbook, A Guide to Using CEMS 2.0, and the CEMS Data Element Glossary;

• interviewed SSA, Office of Disability (OD) and Office of Information Management staff in Baltimore, Maryland;

• interviewed the 10 SSA RO CEMS Coordinators and obtained information on the reconciliation and use of CEMS data; and

• compared selected FY 1998 CEMS data to the corresponding amounts reported on the Report of Obligations to validate the accuracy of CEMS data (see Appendix A).

Our audit was conducted between September 2000 and July 2001 in Kansas City, Missouri and Baltimore, Maryland. The entity audited was OD under the Deputy Commissioner for Disability and Income Security Programs. Our audit was conducted in accordance with generally accepted government auditing standards.

Our audit disclosed that CEMS has not achieved its intended purpose of measuring the relative cost effectiveness of each DDS. Our audit also disclosed monetary differences between CEMS and Form SSA-4513 data. Furthermore, CEMS data is not available timely and is not widely used by SSA. Lastly, we found that unauthorized users can access CEMS data.

CEMS was developed by SSA as a tool to measure the costs of operating each DDS, and was to be used to develop a methodology for determining the relative cost effectiveness of each DDS. CEMS was envisioned to have a significant impact on the financial management of the disability determination process by providing the basis for establishing formal cost standards for DDS operations. CEMS information was to provide SSA with the information necessary to plan, budget, control, coordinate, and evaluate the disability program.

SSA planned to use CEMS to provide cost data in a format that permitted reasonable comparisons of DDS costs. CEMS was to employ cost modeling techniques that improved the cost standard process and that fairly reflected local conditions in each State. These techniques included case weighting, indexing of medical costs, and a separate subsystem to record detailed medical information. However, OD discontinued these techniques in 1991 because it could not derive a method of indexing costs that all DDSs felt fairly reflected their local conditions.

For FY 1998, we compared the data on the Form SSA-4513 to the same data reported in CEMS. This comparison identified a difference of approximately $20 million (see Appendix B). We requested the reconciliation documents for the 52 DDSs from the CEMS Administrator to determine whether there were valid reasons for the $20 million discrepancy. The CEMS Administrator did not have the reconciliation documents. According to CEMS instructions, the DDS is instructed to provide reconciliation documents to its SSA RO CEMS Coordinator. The RO CEMS Coordinator is then supposed to forward the reconciliation documents to the CEMS Administrator in SSA’s Central Office, who is responsible for maintaining the documentation.

Since the CEMS Administrator did not have the reconciliation documents, we contacted the 10 RO CEMS Coordinators for the documents. Five of the 10 RO CEMS Coordinators (Regions 1, 2, 3, 4 and 6) stated that they received and retained the reconciliations from all 28 of their respective DDSs. The remaining 5 RO CEMS Coordinators were unable to provide the reconciliations for the following reasons:

• The Region 5 CEMS Coordinator received reconciliations from only two of the six DDSs in the region;

• The CEMS Coordinators from Regions 8, 9 and 10 did not receive reconciliations from any of their 14 DDSs; and

• The Region 7 CEMS Coordinator received but did not retain the reconciliations from the region’s four DDSs.

Without the reconciliation documents from all 52 DDSs, we were unable to determine the reasons for the $20 million discrepancy. Therefore, we were unable to determine the accuracy of CEMS data for FY 1998.

As of June 2001, CEMS data for FYs 1999 and 2000 were not available. The CEMS data for FYs 1999 and 2000 should have been available by May 2000 and May 2001, respectively. The FY 1999 and FY 2000 CEMS data were not available for our review because several DDSs had not entered the required information and CEMS does not allow review of data until all DDSs have completed input.

Even though the redesigned CEMS was placed in production in August 1999 and was installed at 52 DDSs, the system continues to experience problems. OD stated that some of the problems require Office of Systems’ (OS) resolution, however, the necessary OS resources were not available. The problems that CEMS continues to face are:

• Part of the CEMS software resides on the individual workstations. Because of the new design of CEMS, each workstation requires a separate installation, program updates must be installed on each workstation, and any additional software or other changes that are made to the workstation can result in the system becoming non-compatible with CEMS. While CEMS has been installed in the 52 DDSs, some programming problems still exist and required updates have not been installed. The above problems require OS’ assistance.

• While the redesigned CEMS is more user friendly, the system itself is much more complex than its predecessor. Installation and maintenance of CEMS involves a significant time investment on the part of local area network coordinators. In addition, a significant time investment is required by the CEMS Administrator to ensure that systems problems are identified and resolved.

CEMS was developed as a tool that SSA could use to measure the cost effectiveness of each State DDS. However, CEMS has not achieved its intended purpose because SSA never developed formal cost standards or methodologies for determining the cost effectiveness of each DDS. Therefore, the usefulness of CEMS management information is limited.

• Six of the 10 RO CEMS Coordinators stated they had not used CEMS data for over 1 year. One Coordinator indicated that CEMS information was used. Three Coordinators indicated that, if the information were timely, it would be used.

• OD staff stated that CEMS data are used for limited purposes, such as estimating the cost of proposed legislation. However, its use is limited because the information is not accurate and some of the information could be obtained from other sources. CEMS information is not used for budget-related purposes and therefore, is not shared with other SSA components, such as the Deputy Commissioner for Finance, Assessment, and Management.

We found that DDS’ data entry logons, including passwords, are posted on an Intranet site and available to anyone with Intranet access. This site also contains instructions for installing CEMS. The one CEMS safeguard is that once the CEMS information is input and locked by the DDSs, only the CEMS Administrator can alter the information. However, with the passwords available on the Intranet, there is the possibility for unauthorized alteration of CEMS data prior to the data being locked.

• By utilizing the instructions provided on the Intranet site, we successfully installed CEMS. Then, using the passwords at the Intranet site, we accessed both the data entry and analysis applications of CEMS.

Our objective was to assess the accuracy and use of CEMS data; however, the reconciliation information needed to assess the accuracy of CEMS data was not available. Although we identified a discrepancy of approximately $20 million between the FY 1998 CEMS and the Form SSA-4513 data, the unavailability of reconciliation information prevented us from determining whether this discrepancy was valid. As far as the use of CEMS data, we found that CEMS data are not widely used by SSA and the data are not timely. Furthermore, we found that unauthorized users can access—and possibly alter—CEMS data.

We also found that CEMS has not achieved its intended purpose because SSA was unsuccessful in developing formal cost standards or methodologies for determining the cost effectiveness of each DDS. CEMS could be a valuable management information tool if SSA could design the system to allow for measuring the cost effectiveness of DDSs. Such a system is needed—given the complexity and importance of monitoring DDS performance. However, in its current design, CEMS does not meet its intended purpose and the data are not timely enough to be valuable to the Agency. Therefore, we question whether the monies spent by SSA to maintain CEMS—and the time expended by the DDSs on the labor-intensive input of CEMS data—are justified.

We recommend that SSA perform a cost-benefit analysis to determine whether CEMS should continue based on a comparison of CEMS maintenance, oversight, and input costs to the use of CEMS data. If the cost-benefit analysis supports the continuation of CEMS, take the following actions:

1. Resolve CEMS program problems and ensure that CEMS is properly installed at all DDSs.
2. Update the CEMS Financial Handbook Guidelines to correspond with the revised CEMS system and publish the guidelines in the POMS section reserved for CEMS.
3. Ensure that SSA and DDS staff follows CEMS guidelines.
4. Remove CEMS passwords and installation instructions from the Intranet site and issue new passwords to authorized users.
5. Develop formal cost standards or methodologies for determining the cost effectiveness of each DDS.

In response to our draft report, SSA decided to discontinue CEMS. Specifically, SSA stated that a cost benefit analysis would only duplicate the work performed by the auditors and would not support the continuation of CEMS. SSA further stated it is already taking the necessary steps to discontinue the system. (See Appendix E for SSA’s comments.)

James G. Huse, Jr.

Appendices

APPENDIX A – Acronyms

APPENDIX B – Comparison of Cost Effectiveness Measurement System and Report of Obligations Data for Fiscal Year 1998

APPENDIX C – Overview of the Cost Effectiveness Measurement System Process

APPENDIX D – Estimation of the Fiscal Year 1999 Costs to Maintain the Cost Effectiveness Measurement System

APPENDIX E – Agency Comments

APPENDIX F – OIG Contacts and Staff Acknowledgments

Appendix A

Rona Rustigian, Director, Disability Program Audit Division (617) 565-1819
Mark Bailey, Deputy Director, Disability Program Audit Division (816) 936-5591

In addition to those named above:

Ronald Bussell, Lead Auditor
Kenneth Bennett, Auditor
Richard Reed, Auditor
Cheryl Robinson, Writer-Editor

For additional copies of this report, please visit our web site at www.socialsecurity.gov/oig or contact Office of the Inspector General's Public Affairs Specialist at (410) 966-1375. Refer to Common Identification Number A-07-00-10028.