Office of the Inspector General
Jo Anne B. Barnhart
Commissioner

Inspector General

Review of the Social Security Administration’s Cost Effectiveness Measurement System (A-07-00-10028)

The attached final report presents the results of our audit. Our objective was to assess the accuracy and use of the Social Security Administration’s Cost Effectiveness Measurement System data.

Please comment within 60 days from the date of this memorandum on corrective action taken or planned on each recommendation. If you wish to discuss the final report, please call me or have your staff contact Steven L. Schaeffer, Assistant Inspector General for Audit, at (410) 965-9700.

James G. Huse, Jr.

OFFICE OF

THE INSPECTOR GENERAL

SOCIAL SECURITY ADMINISTRATION

REVIEW OF THE SOCIAL

SECURITY ADMINISTRATION’S

COST EFFECTIVENESS

MEASUREMENT SYSTEM

February 2002

A-07-00-10028

AUDIT REPORT

 

Mission

We improve SSA programs and operations and protect them against fraud, waste, and abuse by conducting independent and objective audits, evaluations, and investigations. We provide timely, useful, and reliable information and advice to Administration officials, the Congress, and the public.

Authority

The Inspector General Act created independent audit and investigative units, called the Office of Inspector General (OIG). The mission of the OIG, as spelled out in the Act, is to:

Conduct and supervise independent and objective audits and investigations relating to agency programs and operations.

Promote economy, effectiveness, and efficiency within the agency.

Prevent and detect fraud, waste, and abuse in agency programs and operations.

Review and make recommendations regarding existing and proposed legislation and regulations relating to agency programs and operations.

Keep the agency head and the Congress fully and currently informed of problems in agency programs and operations.

To ensure objectivity, the IG Act empowers the IG with:

Independence to determine what reviews to perform.

Access to all information necessary for the reviews.

Authority to publish findings and recommendations based on the reviews.

Vision

By conducting independent and objective audits, investigations, and evaluations, we are agents of positive change striving for continuous improvement in the Social Security Administration's programs, operations, and management and in our own office.

Office of the Inspector General

Jo Anne B. Barnhart
Commissioner

Inspector General

Review of the Social Security Administration’s Cost Effectiveness Measurement System (A-07-00-10028)

OBJECTIVE

Our objective was to assess the accuracy and use of the Social Security Administration’s (SSA) Cost Effectiveness Measurement System (CEMS) data.

BACKGROUND

In September 1982, SSA developed CEMS as a tool to measure the operating costs and cost effectiveness of each State Disability Determination Services (DDS). CEMS was also envisioned to provide the basis for establishing formal cost standards for DDSs. In 1997, SSA began revising CEMS to take advantage of technological enhancements and to make the system more user-friendly. SSA spent approximately $690,000 to revise CEMS, and it was released for production in August 1999. SSA also took responsibility for CEMS maintenance, which was previously performed by an outside contractor at an annual cost of approximately $300,000. In Fiscal Year (FY) 1999, SSA spent an estimated $106,000 to maintain CEMS (see Appendix D).

SSA’s instructions require DDSs to: (1) input data into CEMS by the 45th day after the close of each quarter; (2) reconcile CEMS and Form SSA-4513 (Report of Obligations) data; and (3) provide CEMS reconciliation documents to the Regional Offices (RO) who, in turn, provide the documents to the CEMS Administrator in SSA’s Central Office (see Appendix C).

SCOPE AND METHODOLOGY

Our audit was designed to review CEMS data for FYs 1999 and 2000, which should have been available by May 2000 and May 2001, respectively. However, as of June 2001, CEMS data for FYs 1999 and 2000 were not available. Therefore, we reviewed the latest CEMS data available, which was for FY 1998.

To achieve our objective we:

Our audit was conducted between September 2000 and July 2001 in Kansas City, Missouri and Baltimore, Maryland. The entity audited was OD under the Deputy Commissioner for Disability and Income Security Programs. Our audit was conducted in accordance with generally accepted government auditing standards.

RESULTS OF REVIEW

Our audit disclosed that CEMS has not achieved its intended purpose of measuring the relative cost effectiveness of each DDS. Our audit also disclosed monetary differences between CEMS and Form SSA-4513 data. Furthermore, CEMS data is not available timely and is not widely used by SSA. Lastly, we found that unauthorized users can access CEMS data.

CEMS HAS NOT ACHIEVED ITS INTENDED PURPOSE

CEMS was developed by SSA as a tool to measure the costs of operating each DDS, and was to be used to develop a methodology for determining the relative cost effectiveness of each DDS. CEMS was envisioned to have a significant impact on the financial management of the disability determination process by providing the basis for establishing formal cost standards for DDS operations. CEMS information was to provide SSA with the information necessary to plan, budget, control, coordinate, and evaluate the disability program.

SSA planned to use CEMS to provide cost data in a format that permitted reasonable comparisons of DDS costs. CEMS was to employ cost modeling techniques that improved the cost standard process and that fairly reflected local conditions in each State. These techniques included case weighting, indexing of medical costs, and a separate subsystem to record detailed medical information. However, OD discontinued these techniques in 1991 because it could not derive a method of indexing costs that all DDSs felt fairly reflected their local conditions.

DIFFERENCES EXISTED IN CEMS AND REPORT OF OBLIGATION DATA

For FY 1998, we compared the data on the Form SSA-4513 to the same data reported in CEMS. This comparison identified a difference of approximately $20 million (see Appendix B). We requested the reconciliation documents for the 52 DDSs from the CEMS Administrator to determine whether there were valid reasons for the $20 million discrepancy. The CEMS Administrator did not have the reconciliation documents. According to CEMS instructions, the DDS is instructed to provide reconciliation documents to its SSA RO CEMS Coordinator. The RO CEMS Coordinator is then supposed to forward the reconciliation documents to the CEMS Administrator in SSA’s Central Office, who is responsible for maintaining the documentation.

Since the CEMS Administrator did not have the reconciliation documents, we contacted the 10 RO CEMS Coordinators for the documents. Five of the 10 RO CEMS Coordinators (Regions 1, 2, 3, 4 and 6) stated that they received and retained the reconciliations from all 28 of their respective DDSs. The remaining 5 RO CEMS Coordinators were unable to provide the reconciliations for the following reasons:

Without the reconciliation documents from all 52 DDSs, we were unable to determine the reasons for the $20 million discrepancy. Therefore, we were unable to determine the accuracy of CEMS data for FY 1998.

CEMS INFORMATION IS NOT AVAILABLE TO USERS TIMELY

As of June 2001, CEMS data for FYs 1999 and 2000 were not available. The CEMS data for FYs 1999 and 2000 should have been available by May 2000 and May 2001, respectively. The FY 1999 and FY 2000 CEMS data were not available for our review because several DDSs had not entered the required information and CEMS does not allow review of data until all DDSs have completed input.


SSA
REGION

DDSs THAT HAD NOT ENTERED ALL FY 1999 DATA INTO CEMS

DDSs THAT HAD NOT ENTERED ALL FY 2000

DATA INTO CEMS

Region 1

 

Maine
New Hampshire
Rhode Island

Region 2

 

New York

Region 3

 

Delaware

Region 4

 

Tennessee
Kentucky
Mississippi

Region 5

Indiana

Indiana
Minnesota

Region 6

   

Region 7

 

Missouri

Region 8

Colorado
North Dakota
South Dakota

Colorado
North Dakota
South Dakota
Utah

Region 9

 

Arizona

California
Nevada
Arizona

Region 10

 

Washington


Even though the redesigned CEMS was placed in production in August 1999 and was installed at 52 DDSs, the system continues to experience problems. OD stated that some of the problems require Office of Systems’ (OS) resolution, however, the necessary OS resources were not available. The problems that CEMS continues to face are:

CEMS INFORMATION IS NOT WIDELY USED

CEMS was developed as a tool that SSA could use to measure the cost effectiveness of each State DDS. However, CEMS has not achieved its intended purpose because SSA never developed formal cost standards or methodologies for determining the cost effectiveness of each DDS. Therefore, the usefulness of CEMS management information is limited.

UNAUTHORIZED USERS CAN ACCESS CEMS

We found that DDS’ data entry logons, including passwords, are posted on an Intranet site and available to anyone with Intranet access. This site also contains instructions for installing CEMS. The one CEMS safeguard is that once the CEMS information is input and locked by the DDSs, only the CEMS Administrator can alter the information. However, with the passwords available on the Intranet, there is the possibility for unauthorized alteration of CEMS data prior to the data being locked.

CONCLUSIONS AND RECOMMENDATIONS

Our objective was to assess the accuracy and use of CEMS data; however, the reconciliation information needed to assess the accuracy of CEMS data was not available. Although we identified a discrepancy of approximately $20 million between the FY 1998 CEMS and the Form SSA-4513 data, the unavailability of reconciliation information prevented us from determining whether this discrepancy was valid. As far as the use of CEMS data, we found that CEMS data are not widely used by SSA and the data are not timely. Furthermore, we found that unauthorized users can access—and possibly alter—CEMS data.

We also found that CEMS has not achieved its intended purpose because SSA was unsuccessful in developing formal cost standards or methodologies for determining the cost effectiveness of each DDS. CEMS could be a valuable management information tool if SSA could design the system to allow for measuring the cost effectiveness of DDSs. Such a system is needed—given the complexity and importance of monitoring DDS performance. However, in its current design, CEMS does not meet its intended purpose and the data are not timely enough to be valuable to the Agency. Therefore, we question whether the monies spent by SSA to maintain CEMS—and the time expended by the DDSs on the labor-intensive input of CEMS data—are justified.

We recommend that SSA perform a cost-benefit analysis to determine whether CEMS should continue based on a comparison of CEMS maintenance, oversight, and input costs to the use of CEMS data. If the cost-benefit analysis supports the continuation of CEMS, take the following actions:

  1. Resolve CEMS program problems and ensure that CEMS is properly installed at all DDSs.
  2. Update the CEMS Financial Handbook Guidelines to correspond with the revised CEMS system and publish the guidelines in the POMS section reserved for CEMS.
  3. Ensure that SSA and DDS staff follows CEMS guidelines.
  4. Remove CEMS passwords and installation instructions from the Intranet site and issue new passwords to authorized users.
  5. Develop formal cost standards or methodologies for determining the cost effectiveness of each DDS.

AGENCY COMMENTS

In response to our draft report, SSA decided to discontinue CEMS. Specifically, SSA stated that a cost benefit analysis would only duplicate the work performed by the auditors and would not support the continuation of CEMS. SSA further stated it is already taking the necessary steps to discontinue the system. (See Appendix E for SSA’s comments.)

James G. Huse, Jr.

Appendices

APPENDIX A
– Acronyms

APPENDIX B – Comparison of Cost Effectiveness Measurement System and Report of Obligations Data for Fiscal Year 1998

APPENDIX C – Overview of the Cost Effectiveness Measurement System Process

APPENDIX D – Estimation of the Fiscal Year 1999 Costs to Maintain the Cost Effectiveness Measurement System

APPENDIX E – Agency Comments

APPENDIX F – OIG Contacts and Staff Acknowledgments

Appendix A

Acronyms

CEMS

Cost Effectiveness Measurement System

DDS

Disability Determination Services

Form SSA-4513

Report of Obligations

FY

Fiscal Year

OD

Office of Disability

OS

Office of Systems

POMS

Program Operations Manual System

RO

Regional Office

SSA

Social Security Administration


Appendix B

Comparison of Cost Effectiveness Measurement System and Report of Obligations Data for Fiscal Year 1998

 

Disability Determination Services

Total Costs

from Data Validation Forms

Totals from SSA Form 4513 (Excluding Non Federal Workload and AOP Pilot Project Costs)

Difference - CEMS Data Over (Under) SSA Form 4513



Regional Difference

Connecticut

$11,917,994

$12,130,470

$(212,476)

Maine

6,179,353

6,348,552

(169,199)

Massachusetts

26,399,434

25,540,922

858,512

New Hampshire

3,326,350

3,170,030

156,320

Rhode Island

6,586,767

5,817,300

769,467

Vermont

2,312,068

2,310,171

1,897

Region 1

$1,404,521

New Jersey

35,535,964

34,584,779

951,185

New York

138,441,018

135,013,533

3,427,485

Puerto Rico

12,338,682

12,506,016

(167,334)

Region 2

$4,211,336

District of Columbia

3,585,580

3,598,715

(13,135)

Delaware

3,560,018

3,835,725

(275,707)

Maryland

17,671,044

17,360,520

310,524

Pennsylvania

59,768,458

58,648,569

1,119,889

Virginia

25,399,686

25,378,422

21,264

West Virginia

15,088,633

15,040,418

48,216

Region 3

$1,211,051

Alabama

28,183,657

28,234,731

(51,074)

Florida

60,044,943

62,201,716

(2,156,773)

Georgia

44,361,768

42,456,708

1,905,060

Kentucky

31,169,720

31,679,505

(509,785)

Mississippi

19,534,184

19,512,394

21,790

North Carolina

37,336,378

37,428,329

(91,951)

South Carolina

19,847,026

19,580,816

266,210

Tennessee

29,937,575

29,570,817

366,758

Region 4

$(249,766)

 

 

Disability Determination Services


Total Costs from Data Validation Forms

Totals from SSA Form 4513 (Excluding Non Federal Workload and AOP Pilot Project Costs)

Difference - CEMS Data Over (Under) SSA Form 4513



Regional Difference

Illinois

62,033,920

62,532,675

(498,755)

Indiana

24,407,163

24,727,373

(320,210)

Michigan

57,233,873

55,946,442

1,287,431

Minnesota

16,952,982

16,193,461

759,521

Ohio

57,993,236

58,085,459

(92,223)

Wisconsin

19,409,237

18,271,667

1,137,570

Region 5

$2,273,334

Arkansas

15,024,703

15,465,516

(440,813)

Louisiana

30,507,918

30,069,372

438,546

New Mexico

9,413,859

9,419,605

(5,746)

Oklahoma

12,840,671

12,880,525

(39,854)

Texas

81,258,292

77,604,283

3,654,009

Region 6

$3,606,142

Iowa

11,437,668

11,431,271

6,397

Kansas

13,164,714

11,877,523

1,287,191

Missouri

30,065,501

28,274,386

1,791,115

Nebraska

5,564,680

5,564,680

(0)

Region 7

$3,084,703

Colorado

13,003,090

12,814,572

188,518

Montana

3,428,753

3,434,660

(5,907)

North Dakota

1,621,275

1,490,384

130,891

South Dakota

2,227,807

2,352,957

(125,150)

Utah

6,417,846

6,530,944

(113,098)

Wyoming

1,890,196

1,804,360

85,836

Region 8

$161,090

Arizona

17,223,035

17,536,440

(313,405)

California

161,147,442

156,880,555

4,266,887

Hawaii

4,218,715

4,190,100

28,615

Nevada

5,843,926

5,833,111

10,815

Region 9

$3,992,912

Alaska

4,024,595

4,094,017

(69,422)

Idaho

3,938,063

3,936,797

1,266

Oregon2

17,545,701

17,545,723

(22)

Washington

23,819,097

23,272,795

546,302

Region 10

$478,124

Totals

$1,352,184,258

$1,332,010,811

$20,173,447



Appendix C

Overview of the Cost Effectiveness Measurement System Process

Appendix D

Estimate of the Fiscal Year 1999 Costs to Maintain the Cost Effectiveness Measurement System

COST EFFECTIVENESS MEASUREMENT SYSTEM MAINTENANCE COSTS

Entity

Explanation of Cost Estimation

Cost Estimate

Disability Determination Services (DDS)

The June 8, 2000 Federal Register provided the results of a Social Security Administration (SSA) survey of DDSs related to CEMS. According to the survey, CEMS input required 6 hours of DDS staff time, quarterly. This approximates to 1,248 DDS staff hours (52 DDSs times 4 quarters times 6 hours). Based on these hours, the survey estimated DDS input costs of $25,985 annually.

$25,985

Regional Office (RO) Cost Effectiveness Measurement System (CEMS) Coordinators

One RO CEMS Coordinator estimated that he spent 2 hours each quarter per DDS to perform his duties. This equates to a total estimated annual burden of 416 hours for all 52 DDSs (52 DDSs times 2 hours times 4 quarters). Using the 1999 General Schedule $25.78 hourly wage for a Grade 13 (the grade of the RO CEMS Coordinator we contacted), we calculated estimated costs of $10,724 ($25.78 times 416 hours).

10,724

CEMS Administrator

According to the current CEMS Administrator, he spends as much as 50 percent of his time on CEMS-related activities. Using the 1999 General Schedule annual salary for a Grade 13 position, we calculated estimated costs of $26,896 ($53,792 times 50 percent).

Office of Systems (OS) Support

According to OS, it spent 1,256 hours on CEMS-related activities during Fiscal Year (FY) 1999. OS also provided the average annual salary of its employees. Using this information, we estimated an average hourly rate of $33.75 for OS employees and an estimated annual cost of $42,390.

NOTE: In the OS cost estimate, we did not use the 10,438 hours OS spent on developing and improving the CEMS during FY 1999 since it is not a recurring expenditure.

42,390

Total Estimated Cost

$105,995


Appendix E

Agency Comments

COMMENTS ON THE OFFICE OF THE INSPECTOR GENERAL (OIG) DRAFT REPORT, "REVIEW OF THE SOCIAL SECURITY ADMINISTRATION’S COST EFFECTIVENESS MEASUREMENT SYSTEM" (A-07-00-10028)

Thank you for the opportunity to review and comment on this draft report. Our comments on the report recommendations are detailed below.

Recommendations

Perform a cost-benefit analysis (CBA) to determine whether the Cost Effectiveness Measurement System (CEMS) should continue based on a comparison of CEMS maintenance, oversight, and input costs to the use of CEMS data.

If the cost-benefit analysis supports the continuation of CEMS, take the following actions:

a) Resolve CEMS program problems and ensure that CEMS is properly installed at all Disability Determination Services (DDSs); b) update the CEMS Financial Handbook Guidelines to correspond with the revised CEMS system and publish the guidelines in the Program Operations Manual System (POMS) section reserved for CEMS; c) ensure that SSA and DDS staff follow CEMS guidelines; d) remove CEMS passwords and installation instructions from the Intranet site and issue new passwords to authorized users; and e) develop formal cost standards or methodologies for determining the cost effectiveness of each DDS.

Comment

We believe that a CBA would only duplicate the work the auditors have already done and would not support the continuation of CEMS. We believe that the audit findings are sufficient enough to support a case for discontinuation of CEMS, and we are already taking the necessary steps to discontinue the system.

We will begin exploring alternatives to CEMS for more cost-effective methods to accomplish the original goals of CEMS.

Appendix F

OIG Contacts and Staff Acknowledgments

OIG Contacts

Rona Rustigian, Director, Disability Program Audit Division (617) 565-1819
Mark Bailey, Deputy Director, Disability Program Audit Division (816) 936-5591

Staff Acknowledgments

In addition to those named above:

Ronald Bussell, Lead Auditor
Kenneth Bennett, Auditor
Richard Reed, Auditor
Cheryl Robinson, Writer-Editor

For additional copies of this report, please visit our web site at www.socialsecurity.gov/oig or contact Office of the Inspector General's Public Affairs Specialist at (410) 966-1375. Refer to Common Identification Number A-07-00-10028.

Overview of the Office of the Inspector General

Office of Audit

The Office of Audit (OA) conducts comprehensive financial and performance audits of the Social Security Administration’s (SSA) programs and makes recommendations to ensure that program objectives are achieved effectively and efficiently. Financial audits, required by the Chief Financial Officers Act of 1990, assess whether SSA’s financial statements fairly present the Agency’s financial position, results of operations, and cash flow. Performance audits review the economy, efficiency, and effectiveness of SSA’s programs. OA also conducts short-term management and program evaluations focused on issues of concern to SSA, Congress, and the general public. Evaluations often focus on identifying and recommending ways to prevent and minimize program fraud and inefficiency.

Office of Executive Operations

The Office of Executive Operations (OEO) provides four functions for the Office of the Inspector General (OIG) – administrative support, strategic planning, quality assurance, and public affairs. OEO supports the OIG components by providing information resources management; systems security; and the coordination of budget, procurement, telecommunications, facilities and equipment, and human resources. In addition, this Office coordinates and is responsible for the OIG’s strategic planning function and the development and implementation of performance measures required by the Government Performance and Results Act. The quality assurance division performs internal reviews to ensure that OIG offices nationwide hold themselves to the same rigorous standards that we expect from the Agency. This division also conducts employee investigations within OIG. The public affairs team communicates OIG’s planned and current activities and the results to the Commissioner and Congress, as well as other entities.

Office of Investigations

The Office of Investigations (OI) conducts and coordinates investigative activity related to fraud, waste, abuse, and mismanagement of SSA programs and operations. This includes wrongdoing by applicants, beneficiaries, contractors, physicians, interpreters, representative payees, third parties, and by SSA employees in the performance of their duties. OI also conducts joint investigations with other Federal, State, and local law enforcement agencies.

Counsel to the Inspector General

The Counsel to the Inspector General provides legal advice and counsel to the Inspector General on various matters, including: 1) statutes, regulations, legislation, and policy directives governing the administration of SSA’s programs; 2) investigative procedures and techniques; and 3) legal implications and conclusions to be drawn from audit and investigative material produced by the OIG. The Counsel’s office also administers the civil monetary penalty program.