Office of
the Inspector General
Jo Anne B. Barnhart
Commissioner
Inspector General
Review of the Social Security Administration’s Cost Effectiveness Measurement System (A-07-00-10028)
The attached final report presents the results of our audit. Our objective was to assess the accuracy and use of the Social Security Administration’s Cost Effectiveness Measurement System data.
Please comment within 60 days from the date of this memorandum on corrective action taken or planned on each recommendation. If you wish to discuss the final report, please call me or have your staff contact Steven L. Schaeffer, Assistant Inspector General for Audit, at (410) 965-9700.
James G. Huse, Jr.
OFFICE OF
THE INSPECTOR GENERAL
SOCIAL SECURITY ADMINISTRATION
REVIEW OF THE SOCIAL
SECURITY ADMINISTRATION’S
COST EFFECTIVENESS
MEASUREMENT SYSTEM
February
2002
A-07-00-10028
AUDIT REPORT
Mission
We improve SSA programs and operations and protect them against fraud, waste, and abuse by conducting independent and objective audits, evaluations, and investigations. We provide timely, useful, and reliable information and advice to Administration officials, the Congress, and the public.
Authority
The Inspector General Act created independent audit and investigative units, called the Office of Inspector General (OIG). The mission of the OIG, as spelled out in the Act, is to:
Conduct and supervise independent and objective audits and investigations relating to agency programs and operations.
Promote economy, effectiveness, and efficiency within the agency.
Prevent and detect fraud, waste, and abuse in agency programs and operations.
Review and make recommendations regarding existing and proposed legislation and regulations relating to agency programs and operations.
Keep the agency head and the Congress fully and currently informed of problems in agency programs and operations.
To ensure objectivity, the IG Act empowers the IG with:
Independence to determine what reviews to perform.
Access to all information
necessary for the reviews.
Authority to
publish findings and recommendations based on the reviews.
Vision
By conducting independent and objective audits, investigations, and evaluations, we are agents of positive change striving for continuous improvement in the Social Security Administration's programs, operations, and management and in our own office.
Office of the Inspector General
Jo Anne
B. Barnhart
Commissioner
Inspector General
Review of the Social Security Administration’s Cost Effectiveness Measurement System (A-07-00-10028)
OBJECTIVE
Our objective was to assess the accuracy and use of the Social Security Administration’s (SSA) Cost Effectiveness Measurement System (CEMS) data.
BACKGROUND
In September 1982, SSA developed CEMS as a tool to measure the operating costs and cost effectiveness of each State Disability Determination Services (DDS). CEMS was also envisioned to provide the basis for establishing formal cost standards for DDSs. In 1997, SSA began revising CEMS to take advantage of technological enhancements and to make the system more user-friendly. SSA spent approximately $690,000 to revise CEMS, and it was released for production in August 1999. SSA also took responsibility for CEMS maintenance, which was previously performed by an outside contractor at an annual cost of approximately $300,000. In Fiscal Year (FY) 1999, SSA spent an estimated $106,000 to maintain CEMS (see Appendix D).
SSA’s instructions require DDSs to: (1) input data into CEMS by the 45th day after the close of each quarter; (2) reconcile CEMS and Form SSA-4513 (Report of Obligations) data; and (3) provide CEMS reconciliation documents to the Regional Offices (RO) who, in turn, provide the documents to the CEMS Administrator in SSA’s Central Office (see Appendix C).
SCOPE AND METHODOLOGY
Our audit was designed to review CEMS data for FYs 1999 and 2000, which should have been available by May 2000 and May 2001, respectively. However, as of June 2001, CEMS data for FYs 1999 and 2000 were not available. Therefore, we reviewed the latest CEMS data available, which was for FY 1998.
To achieve our objective we:
Our audit was conducted between September 2000 and July 2001 in Kansas City, Missouri and Baltimore, Maryland. The entity audited was OD under the Deputy Commissioner for Disability and Income Security Programs. Our audit was conducted in accordance with generally accepted government auditing standards.
RESULTS OF REVIEW
Our audit disclosed that CEMS has not achieved its intended purpose of measuring the relative cost effectiveness of each DDS. Our audit also disclosed monetary differences between CEMS and Form SSA-4513 data. Furthermore, CEMS data is not available timely and is not widely used by SSA. Lastly, we found that unauthorized users can access CEMS data.
CEMS HAS NOT ACHIEVED ITS INTENDED PURPOSE
CEMS was developed by SSA as a tool to measure the costs of operating each DDS, and was to be used to develop a methodology for determining the relative cost effectiveness of each DDS. CEMS was envisioned to have a significant impact on the financial management of the disability determination process by providing the basis for establishing formal cost standards for DDS operations. CEMS information was to provide SSA with the information necessary to plan, budget, control, coordinate, and evaluate the disability program.
SSA planned to use CEMS to provide cost data in a format that permitted reasonable comparisons of DDS costs. CEMS was to employ cost modeling techniques that improved the cost standard process and that fairly reflected local conditions in each State. These techniques included case weighting, indexing of medical costs, and a separate subsystem to record detailed medical information. However, OD discontinued these techniques in 1991 because it could not derive a method of indexing costs that all DDSs felt fairly reflected their local conditions.
DIFFERENCES EXISTED IN CEMS AND REPORT OF OBLIGATION DATA
For FY 1998, we compared the data on the Form SSA-4513 to the same data reported in CEMS. This comparison identified a difference of approximately $20 million (see Appendix B). We requested the reconciliation documents for the 52 DDSs from the CEMS Administrator to determine whether there were valid reasons for the $20 million discrepancy. The CEMS Administrator did not have the reconciliation documents. According to CEMS instructions, the DDS is instructed to provide reconciliation documents to its SSA RO CEMS Coordinator. The RO CEMS Coordinator is then supposed to forward the reconciliation documents to the CEMS Administrator in SSA’s Central Office, who is responsible for maintaining the documentation.
Since the CEMS Administrator did not have the reconciliation documents, we contacted the 10 RO CEMS Coordinators for the documents. Five of the 10 RO CEMS Coordinators (Regions 1, 2, 3, 4 and 6) stated that they received and retained the reconciliations from all 28 of their respective DDSs. The remaining 5 RO CEMS Coordinators were unable to provide the reconciliations for the following reasons:
Without the reconciliation documents from all 52 DDSs, we were unable to determine the reasons for the $20 million discrepancy. Therefore, we were unable to determine the accuracy of CEMS data for FY 1998.
CEMS INFORMATION IS NOT AVAILABLE TO USERS TIMELY
As of June 2001, CEMS data for FYs 1999 and 2000 were not available. The CEMS data for FYs 1999 and 2000 should have been available by May 2000 and May 2001, respectively. The FY 1999 and FY 2000 CEMS data were not available for our review because several DDSs had not entered the required information and CEMS does not allow review of data until all DDSs have completed input.
|
DDSs THAT HAD NOT ENTERED ALL FY 1999 DATA INTO CEMS |
DDSs THAT HAD NOT ENTERED ALL FY 2000 DATA INTO CEMS |
Region 1 |
Maine |
|
Region 2 |
New York |
|
Region 3 |
Delaware |
|
Region 4 |
Tennessee |
|
Region 5 |
Indiana |
Indiana |
Region 6 |
||
Region 7 |
Missouri |
|
Region 8 |
Colorado |
Colorado |
Region 9 |
Arizona |
California |
Region 10 |
Washington |
Even though the redesigned CEMS was placed in production in August 1999 and was installed at 52 DDSs, the system continues to experience problems. OD stated that some of the problems require Office of Systems’ (OS) resolution, however, the necessary OS resources were not available. The problems that CEMS continues to face are:
CEMS INFORMATION IS NOT WIDELY USED
CEMS was developed as a tool that SSA could use to measure the cost effectiveness of each State DDS. However, CEMS has not achieved its intended purpose because SSA never developed formal cost standards or methodologies for determining the cost effectiveness of each DDS. Therefore, the usefulness of CEMS management information is limited.
UNAUTHORIZED USERS CAN ACCESS CEMS
We found that DDS’ data entry logons, including passwords, are posted on an Intranet site and available to anyone with Intranet access. This site also contains instructions for installing CEMS. The one CEMS safeguard is that once the CEMS information is input and locked by the DDSs, only the CEMS Administrator can alter the information. However, with the passwords available on the Intranet, there is the possibility for unauthorized alteration of CEMS data prior to the data being locked.
CONCLUSIONS AND RECOMMENDATIONS
Our objective was to assess the accuracy and use of CEMS data; however, the reconciliation information needed to assess the accuracy of CEMS data was not available. Although we identified a discrepancy of approximately $20 million between the FY 1998 CEMS and the Form SSA-4513 data, the unavailability of reconciliation information prevented us from determining whether this discrepancy was valid. As far as the use of CEMS data, we found that CEMS data are not widely used by SSA and the data are not timely. Furthermore, we found that unauthorized users can access—and possibly alter—CEMS data.
We also found that CEMS has not achieved its intended purpose because SSA was unsuccessful in developing formal cost standards or methodologies for determining the cost effectiveness of each DDS. CEMS could be a valuable management information tool if SSA could design the system to allow for measuring the cost effectiveness of DDSs. Such a system is needed—given the complexity and importance of monitoring DDS performance. However, in its current design, CEMS does not meet its intended purpose and the data are not timely enough to be valuable to the Agency. Therefore, we question whether the monies spent by SSA to maintain CEMS—and the time expended by the DDSs on the labor-intensive input of CEMS data—are justified.
We recommend that SSA perform a cost-benefit analysis to determine whether CEMS should continue based on a comparison of CEMS maintenance, oversight, and input costs to the use of CEMS data. If the cost-benefit analysis supports the continuation of CEMS, take the following actions:
AGENCY COMMENTS
In response to our draft report, SSA decided to discontinue CEMS. Specifically, SSA stated that a cost benefit analysis would only duplicate the work performed by the auditors and would not support the continuation of CEMS. SSA further stated it is already taking the necessary steps to discontinue the system. (See Appendix E for SSA’s comments.)
James G. Huse, Jr.
Appendices
APPENDIX A – Acronyms
APPENDIX B – Comparison of Cost Effectiveness Measurement System and
Report of Obligations Data for Fiscal Year 1998
APPENDIX C – Overview of the Cost Effectiveness Measurement System Process
APPENDIX D – Estimation of the Fiscal Year 1999 Costs to Maintain the Cost Effectiveness Measurement System
APPENDIX E – Agency Comments
APPENDIX F – OIG Contacts and Staff Acknowledgments
Appendix A
Acronyms
CEMS |
Cost Effectiveness Measurement System |
DDS |
Disability Determination Services |
Form SSA-4513 |
Report of Obligations |
FY |
Fiscal Year |
OD |
Office of Disability |
OS |
Office of Systems |
POMS |
Program Operations Manual System |
RO |
Regional Office |
SSA |
Social Security Administration |
Appendix
B
Comparison of Cost Effectiveness Measurement System and Report of Obligations Data for Fiscal Year 1998
Disability Determination Services |
Total Costs from Data Validation Forms |
Totals from SSA Form 4513 (Excluding Non Federal Workload and AOP Pilot Project Costs) |
Difference - CEMS Data Over (Under) SSA Form 4513 |
|
Connecticut |
$11,917,994 |
$12,130,470 |
$(212,476) |
|
Maine |
6,179,353 |
6,348,552 |
(169,199) |
|
Massachusetts |
26,399,434 |
25,540,922 |
858,512 |
|
New Hampshire |
3,326,350 |
3,170,030 |
156,320 |
|
Rhode Island |
6,586,767 |
5,817,300 |
769,467 |
|
Vermont |
2,312,068 |
2,310,171 |
1,897 |
|
Region 1 |
$1,404,521 |
|||
New Jersey |
35,535,964 |
34,584,779 |
951,185 |
|
New York |
138,441,018 |
135,013,533 |
3,427,485 |
|
Puerto Rico |
12,338,682 |
12,506,016 |
(167,334) |
|
Region 2 |
$4,211,336 |
|||
District of Columbia |
3,585,580 |
3,598,715 |
(13,135) |
|
Delaware |
3,560,018 |
3,835,725 |
(275,707) |
|
Maryland |
17,671,044 |
17,360,520 |
310,524 |
|
Pennsylvania |
59,768,458 |
58,648,569 |
1,119,889 |
|
Virginia |
25,399,686 |
25,378,422 |
21,264 |
|
West Virginia |
15,088,633 |
15,040,418 |
48,216 |
|
Region 3 |
$1,211,051 |
|||
Alabama |
28,183,657 |
28,234,731 |
(51,074) |
|
Florida |
60,044,943 |
62,201,716 |
(2,156,773) |
|
Georgia |
44,361,768 |
42,456,708 |
1,905,060 |
|
Kentucky |
31,169,720 |
31,679,505 |
(509,785) |
|
Mississippi |
19,534,184 |
19,512,394 |
21,790 |
|
North Carolina |
37,336,378 |
37,428,329 |
(91,951) |
|
South Carolina |
19,847,026 |
19,580,816 |
266,210 |
|
Tennessee |
29,937,575 |
29,570,817 |
366,758 |
|
Region 4 |
$(249,766) |
Disability Determination Services |
|
Totals from SSA Form 4513 (Excluding Non Federal Workload and AOP Pilot Project Costs) |
Difference - CEMS Data Over (Under) SSA Form 4513 |
|
Illinois |
62,033,920 |
62,532,675 |
(498,755) |
|
Indiana |
24,407,163 |
24,727,373 |
(320,210) |
|
Michigan |
57,233,873 |
55,946,442 |
1,287,431 |
|
Minnesota |
16,952,982 |
16,193,461 |
759,521 |
|
Ohio |
57,993,236 |
58,085,459 |
(92,223) |
|
Wisconsin |
19,409,237 |
18,271,667 |
1,137,570 |
|
Region 5 |
$2,273,334 |
|||
Arkansas |
15,024,703 |
15,465,516 |
(440,813) |
|
Louisiana |
30,507,918 |
30,069,372 |
438,546 |
|
New Mexico |
9,413,859 |
9,419,605 |
(5,746) |
|
Oklahoma |
12,840,671 |
12,880,525 |
(39,854) |
|
Texas |
81,258,292 |
77,604,283 |
3,654,009 |
|
Region 6 |
$3,606,142 |
|||
Iowa |
11,437,668 |
11,431,271 |
6,397 |
|
Kansas |
13,164,714 |
11,877,523 |
1,287,191 |
|
Missouri |
30,065,501 |
28,274,386 |
1,791,115 |
|
Nebraska |
5,564,680 |
5,564,680 |
(0) |
|
Region 7 |
$3,084,703 |
|||
Colorado |
13,003,090 |
12,814,572 |
188,518 |
|
Montana |
3,428,753 |
3,434,660 |
(5,907) |
|
North Dakota |
1,621,275 |
1,490,384 |
130,891 |
|
South Dakota |
2,227,807 |
2,352,957 |
(125,150) |
|
Utah |
6,417,846 |
6,530,944 |
(113,098) |
|
Wyoming |
1,890,196 |
1,804,360 |
85,836 |
|
Region 8 |
$161,090 |
|||
Arizona |
17,223,035 |
17,536,440 |
(313,405) |
|
California |
161,147,442 |
156,880,555 |
4,266,887 |
|
Hawaii |
4,218,715 |
4,190,100 |
28,615 |
|
Nevada |
5,843,926 |
5,833,111 |
10,815 |
|
Region 9 |
$3,992,912 |
|||
Alaska |
4,024,595 |
4,094,017 |
(69,422) |
|
Idaho |
3,938,063 |
3,936,797 |
1,266 |
|
Oregon2 |
17,545,701 |
17,545,723 |
(22) |
|
Washington |
23,819,097 |
23,272,795 |
546,302 |
|
Region 10 |
$478,124 |
|||
Totals |
$1,352,184,258 |
$1,332,010,811 |
$20,173,447 |
Appendix
C
Overview of the Cost Effectiveness Measurement System Process
Estimate of the Fiscal Year 1999 Costs to Maintain the Cost Effectiveness Measurement System
COST EFFECTIVENESS MEASUREMENT SYSTEM MAINTENANCE COSTS |
||
Entity |
Explanation of Cost Estimation |
Cost Estimate |
Disability Determination Services (DDS) |
The June 8, 2000 Federal Register provided the results of a Social Security Administration (SSA) survey of DDSs related to CEMS. According to the survey, CEMS input required 6 hours of DDS staff time, quarterly. This approximates to 1,248 DDS staff hours (52 DDSs times 4 quarters times 6 hours). Based on these hours, the survey estimated DDS input costs of $25,985 annually. |
$25,985 |
Regional Office (RO) Cost Effectiveness Measurement System (CEMS) Coordinators |
One RO CEMS Coordinator estimated that he spent 2 hours each quarter per DDS to perform his duties. This equates to a total estimated annual burden of 416 hours for all 52 DDSs (52 DDSs times 2 hours times 4 quarters). Using the 1999 General Schedule $25.78 hourly wage for a Grade 13 (the grade of the RO CEMS Coordinator we contacted), we calculated estimated costs of $10,724 ($25.78 times 416 hours). |
10,724
|
CEMS Administrator |
According to the current CEMS Administrator, he spends as much as 50 percent of his time on CEMS-related activities. Using the 1999 General Schedule annual salary for a Grade 13 position, we calculated estimated costs of $26,896 ($53,792 times 50 percent). |
|
Office of Systems (OS) Support |
According to OS, it spent 1,256 hours on CEMS-related activities during Fiscal Year (FY) 1999. OS also provided the average annual salary of its employees. Using this information, we estimated an average hourly rate of $33.75 for OS employees and an estimated annual cost of $42,390. NOTE: In the OS cost estimate, we did not use the 10,438 hours OS spent on developing and improving the CEMS during FY 1999 since it is not a recurring expenditure. |
42,390 |
Total Estimated Cost |
$105,995 |
Appendix E
Agency Comments
COMMENTS ON THE OFFICE OF THE INSPECTOR GENERAL (OIG) DRAFT REPORT, "REVIEW OF THE SOCIAL SECURITY ADMINISTRATION’S COST EFFECTIVENESS MEASUREMENT SYSTEM" (A-07-00-10028)
Thank you for the opportunity to review and comment on this draft report. Our comments on the report recommendations are detailed below.
Recommendations
Perform a cost-benefit analysis (CBA) to determine whether the Cost Effectiveness Measurement System (CEMS) should continue based on a comparison of CEMS maintenance, oversight, and input costs to the use of CEMS data.
If the cost-benefit analysis supports the continuation of CEMS, take the following actions:
a) Resolve CEMS program problems and ensure that CEMS is properly installed at all Disability Determination Services (DDSs); b) update the CEMS Financial Handbook Guidelines to correspond with the revised CEMS system and publish the guidelines in the Program Operations Manual System (POMS) section reserved for CEMS; c) ensure that SSA and DDS staff follow CEMS guidelines; d) remove CEMS passwords and installation instructions from the Intranet site and issue new passwords to authorized users; and e) develop formal cost standards or methodologies for determining the cost effectiveness of each DDS.
Comment
We believe that a CBA would only duplicate the work the auditors have already done and would not support the continuation of CEMS. We believe that the audit findings are sufficient enough to support a case for discontinuation of CEMS, and we are already taking the necessary steps to discontinue the system.
We will begin exploring alternatives to CEMS for more cost-effective methods to accomplish the original goals of CEMS.
Appendix
F
OIG Contacts and Staff Acknowledgments
OIG Contacts
Rona
Rustigian, Director, Disability Program Audit Division (617) 565-1819
Mark Bailey, Deputy Director, Disability Program Audit Division (816) 936-5591
Staff Acknowledgments
In addition to those named above:
Ronald
Bussell, Lead Auditor
Kenneth Bennett, Auditor
Richard Reed, Auditor
Cheryl Robinson, Writer-Editor
For additional copies of this report, please visit our web site at www.socialsecurity.gov/oig or contact Office of the Inspector General's Public Affairs Specialist at (410) 966-1375. Refer to Common Identification Number A-07-00-10028.
Overview of the Office of the Inspector General
Office of Audit
The Office of Audit (OA) conducts comprehensive financial and performance audits of the Social Security Administration’s (SSA) programs and makes recommendations to ensure that program objectives are achieved effectively and efficiently. Financial audits, required by the Chief Financial Officers Act of 1990, assess whether SSA’s financial statements fairly present the Agency’s financial position, results of operations, and cash flow. Performance audits review the economy, efficiency, and effectiveness of SSA’s programs. OA also conducts short-term management and program evaluations focused on issues of concern to SSA, Congress, and the general public. Evaluations often focus on identifying and recommending ways to prevent and minimize program fraud and inefficiency.
Office of Executive Operations
The Office of Executive Operations (OEO) provides four functions for the Office of the Inspector General (OIG) – administrative support, strategic planning, quality assurance, and public affairs. OEO supports the OIG components by providing information resources management; systems security; and the coordination of budget, procurement, telecommunications, facilities and equipment, and human resources. In addition, this Office coordinates and is responsible for the OIG’s strategic planning function and the development and implementation of performance measures required by the Government Performance and Results Act. The quality assurance division performs internal reviews to ensure that OIG offices nationwide hold themselves to the same rigorous standards that we expect from the Agency. This division also conducts employee investigations within OIG. The public affairs team communicates OIG’s planned and current activities and the results to the Commissioner and Congress, as well as other entities.
Office of Investigations
The Office of Investigations (OI) conducts and coordinates investigative activity related to fraud, waste, abuse, and mismanagement of SSA programs and operations. This includes wrongdoing by applicants, beneficiaries, contractors, physicians, interpreters, representative payees, third parties, and by SSA employees in the performance of their duties. OI also conducts joint investigations with other Federal, State, and local law enforcement agencies.
Counsel to the Inspector General
The Counsel to the Inspector General provides legal advice and counsel to the Inspector General on various matters, including: 1) statutes, regulations, legislation, and policy directives governing the administration of SSA’s programs; 2) investigative procedures and techniques; and 3) legal implications and conclusions to be drawn from audit and investigative material produced by the OIG. The Counsel’s office also administers the civil monetary penalty program.