A-15-04-14074 - Alternate Format

OFFICE OF
THE INSPECTOR GENERAL

SOCIAL SECURITY ADMINISTRATION

Disability Determination Services
Net Accuracy Rate--Allowances
and Denials Combined

November 2004

A-15-04-14074

AUDIT REPORT

Mission

We improve SSA programs and operations and protect them against fraud, waste, and abuse by conducting independent and objective audits, evaluations, and investigations. We provide timely, useful, and reliable information and advice to Administration officials, the Congress, and the public.

Authority

The Inspector General Act created independent audit and investigative units, called the Office of Inspector General (OIG). The mission of the OIG, as spelled out in the Act, is to:

Conduct and supervise independent and objective audits and investigations relating to agency programs and operations.
Promote economy, effectiveness, and efficiency within the agency.
Prevent and detect fraud, waste, and abuse in agency programs and operations.
Review and make recommendations regarding existing and proposed legislation and regulations relating to agency programs and operations.
Keep the agency head and the Congress fully and currently informed of problems in agency programs and operations.

To ensure objectivity, the IG Act empowers the IG with:

Independence to determine what reviews to perform.
Access to all information necessary for the reviews.
Authority to publish findings and recommendations based on the reviews.

Vision

By conducting independent and objective audits, investigations, and evaluations, we are agents of positive change striving for continuous improvement in the Social Security Administration's programs, operations, and management and in our own office.
MEMORANDUM

Date: November 10, 2004

To: The Commissioner

From: Acting Inspector General

Subject: Performance Indicator Audit: Disability Determination Services Net Accuracy Rate--Allowances and Denials Combined (A-15-04-14074)

We contracted with PricewaterhouseCoopers, LLP (PwC) to evaluate 16 of the Social Security Administration's performance indicators established to comply with the Government Performance and Results Act. The attached final report presents the results of the performance indicator PwC reviewed. For the performance indicator included in this audit, PwC's objectives were to:
Test critical controls over the data generation and calculation processes for the specific performance indicator,
Assess the overall adequacy, accuracy, reasonableness, completeness, and consistency of the performance indicator and supporting data, and
Determine if each performance indicator provides meaningful measurement of the program and the achievement of its stated objectives.

This report contains the results of the audit for the following indicator:

Disability Determination Services Net Accuracy Rate (allowances and denials combined).

Please provide within 60 days a corrective action plan that addresses each recommendation. If you wish to discuss the final report, please call me or have your staff contact Steven L. Schaeffer, Assistant Inspector General for Audit, at (410) 965 9700.

Patrick P. O'Carroll, Jr.

MEMORANDUM

Date: October 8, 2004

To: Acting Inspector General

From: PricewaterhouseCoopers LLP

Subject: Performance Indicator Audit: Disability Determination Services Net Accuracy Rate--Allowances and Denials Combined (A-15-04-14074)

The Government Performance and Results Act (GPRA) of 1993 requires the Social Security Administration (SSA) to develop performance indicators that assess the relevant service levels and outcomes of each program activity. GPRA also calls for a description of the means employed to verify and validate the measured values used to report on program performance.

To enhance the practical use of performance information, the Office of Management and Budget (OMB), in collaboration with other Federal agencies, developed the Program Assessment Rating Tool (PART), which is comprised of assessment criteria on program performance and management. PART establishes a high, "good government" standard of performance and is used to rate programs in an open, public fashion. The indicator included in this audit report, "DDS Net Accuracy Rate (allowances and denials combined)," is a PART measure.

OBJECTIVE

For the performance indicator included in this audit, our objectives were to:

1. Test critical controls over the data generation and calculation processes for the specific performance indicator.
2. Assess the overall adequacy, accuracy, reasonableness, completeness, and consistency of the performance indicator and supporting data.
3. Determine if each performance indicator provides meaningful measurement of the program and the achievement of its stated objectives.

We audited the following performance indicator as stated in SSA's Fiscal Year (FY) 2003 Performance and Accountability Report (PAR):

Performance Indicator FY 2003 Goal FY 2003 Reported Results
DDS Net Accuracy Rate (allowances and denials combined) 97% Not Available

BACKGROUND

Section 221 (a)(2) of the Social Security Act directs the Commissioner to promulgate regulations specifying performance standards for the disability determination function to assure effective and uniform administration of the Disability Insurance Program. Initial disability determinations are completed by the Disability Determination Services (DDS), which are State agencies or other responsible jurisdictions. Title 20 of the Code of Federal Regulations (CFR), section 404.1643 (b), specifies DDS performance criteria, including the required rate of decisional accuracy for initial disability claims of 99 percent for combined Title II and Title XVI cases.

Section 221(c) of the Social Security Act also directs the Commissioner to review favorable and unfavorable determinations made by the DDSs. SSA's Quality Assurance (QA) Initial Disability Determination Reviews, which are conducted by the regional disability quality branches (DQB) within the Office of Quality Assurance and Performance Assessment (OQA), are used to determine the accuracy of the initial allowance and denial determinations made by the DDSs. The results of the reviews are compiled and used to calculate the DDS Net Accuracy Rate reported in SSA's FY 2003 PAR. Appendix C provides an overview of the OQA process for determining the accuracy rate.

SSA's OQA regional Disability Quality Branches (DQB) review the DDS determinations selected by the National Disability Determination Services System Automated Sample Selection Process module. To ensure the appropriateness of the disability determination, the review replicates the DDS eligibility determination process as closely as possible. Medical consultation is obtained by one of the disability reviewers (a non-physician OQA disability examiner) in essentially all cases. The results of the reviews are tracked entirely within the Disability Case Adjudication and Review System (DICARS). Reviewers input findings into DICARS. The DICARS results are uploaded to the OQA National Datafile on SSA's mainframe for subsequent calculation of the indicator.

There are three types of deficiencies identified by DQB. Group I deficiencies are substantive deficiencies that affect or have the potential to affect the determination of eligibility. Group II deficiencies are substantive deficiencies that affect only the onset date, ending date, or cessation date but do not affect the determination. Group III technical deficiencies are instances of noncompliance with procedural requirements that do not affect the determination. Group I decisional deficiencies are those in which all documentation is present and a different medical decision should have been made. Group I deficiencies identified by the reviewers are returned to the appropriate DDS with detailed directions for the required corrective action. If the DDS agrees with DQB's assessment, it will correct the deficiency and return the case to DQB for completion of the review and final case input. The DDSs may dispute Group I deficiencies; if OQA agrees with the dispute, the deficiency is rescinded.

The DDS Net Accuracy Rate reflects Group I deficiencies in initial determinations only. The Net Accuracy Rate is based on the number of deficient cases with changed disability decisions. In addition, deficient cases not corrected within 90 days from the end of the period covered by the report, are counted as changed decisions.

RESULTS OF REVIEW

DDS Net Accuracy Rate (allowances and denials combined)

FY 2003 Goal: 97%
Actual FY 2003 Performance: Not available.

Due to the length of time required to gather, validate, analyze, and report on this performance indicator, SSA did not include the actual FY 2003 performance results in the FY 2003 PAR. The data and final calculation results were provided for our review in January 2004. Refer to Appendix D for the final report provided to PricewaterhouseCoopers, LLP. OQA reported a 96.1 percent Net Accuracy Rate in its final report. At the time of this audit, SSA had not reported the result externally.

The DDS Net Accuracy Rate indicator is an outcome measure representing a direct evaluation of whether SSA made the right decision in the disability determination process. The indicator directly supports the strategic objective of "Make the right decision in the disability process as early as possible."

Generally, we found this performance indicator to be meaningful. However, some improvements could be made to clarify and improve the usefulness of this performance indicator. In addition, we believe SSA should strengthen the logical access privileges given to employees to ensure that the data supporting this performance indicator is not inadvertently deleted or changed without appropriate approval.

Title 20 of the CFR, section 404.1643 (b), establishes the target level for decision accuracy at 99 percent. However, SSA's goal for the performance indicator was set lower than the established regulatory target at 97 percent. The PAR did not explain why SSA set the goal below the target established by regulation. SSA did not meet the internal goal or the regulatory target during the year under review.

We found the narrative included in the PAR related to the performance indicator should have been more informative. The narrative explained that SSA did not meet its FY 2002 goal and it provided the actual results for FY 2002, but did not state the
FY 2002 goal. Trend performance was shown for FY 1999 through FY 2001, but the text explained that FY 2002 data was not comparable to the FY 1999 through FY 2001 data due to a change in the performance measure definition. No explanation was provided to aid the user of this information to understand the practicality or relevance of the FY 1999 through FY 2001 data.

The PAR narrative states that the Net Accuracy Rate, which is the accuracy rate for allowances and denials combined, is more meaningful as an indicator of "…correctness of DDS decision-making.…" Allowances in the disability adjudication process are more frequently decided earlier in the process, thus there is more opportunity for error in the denial determination. The combined accuracy rate for a DDS could potentially increase while its denial accuracy rate continues to decrease. OQA measures and tracks accuracy rates separately for allowances and denials. By including the individual error rates for the two types of decisions, SSA could improve the meaningfulness of the indicator. Also, the narrative did not include the margin of error (i.e., quantification of sampling error) that results from extrapolating the sample error rate to the entire population of cases reviewed.

We also found that the accuracy rate only covered the initial disability determinations made by the DDSs, even though the DDSs were also responsible for making the disability determinations for: a) Continuing Disability Reviews (CDR) and
b) Reconsideration requests. The sampling process included DDS decisions related to the CDRs and Reconsideration determinations; however, this information was excluded from the Net Accuracy Rate. The lack of clarity in the title of the performance indicator and in the "data definition" section of the PAR may lead to confusion regarding whether this performance indicator reflects the accuracy of all DDS workloads as opposed to a single DDS workload.

The documentation of the sampling methodology used by OQA could be improved. The documentation lacks a complete and comprehensive description of the statistical methodology and statistical formulas to sample and project the results of the DQB reviews. In addition, SSA's computer programs did not contain descriptive narratives. Specifically, the programs did not include enough imbedded comments to describe its function. There was an absence of documented policies and procedures used to prepare the performance measure information and management indicated that resource constraints prevented adequate documentation of the programs and methodology. Finally, documentation describing the automated and manual controls involved in the calculation and reporting of the performance indicator did not exist.

We tested a selection of DICARS mainframe datasets used to calculate the indicator and found that seven SSA employees had the "All" access designation within the Top Secret security software to these datasets. This level of access would allow users to create, delete and update any of the data (or datasets) contained within the datasets we reviewed without appropriate review or approval of the changes.

RECOMMENDATIONS

We recommend SSA:

1. Consider reporting interim results in the PAR, or change the measurement period for this indicator to enable SSA to report current year results in the PAR.
2. Disclose in the PAR how the performance indicator contrasts with the target set forth in the CFR, and the reason why it differs from the CFR target.
3. Ensure that the performance indicator titles, definitions, and goals are explicit, complete, and consistent.
4. Provide comprehensive explanations on the relevance of non-comparable data included in the PAR narrative.
5. Consider including the accuracy rates for allowances and denials.
6. Include in the PAR the margin of error (i.e., quantification of sampling error) that results from projecting the sample error rate to the entire population of cases reviewed.
7. Maintain documentation that describes how the performance indicator goals were established, document the policies and procedures used to prepare and report the results of the performance indicators, and keep a complete audit trail.
8. Ensure that SSA personnel do not have the ability, through inappropriate access, to directly modify, create or delete the datasets used to calculate the results of this indicator.

AGENCY COMMENTS

SSA agreed with five of the recommendations in this report; however, it disagreed with two recommendations and had clarifying comments to a third recommendation. The full text of SSA's comments can be found in Appendix E.

Recommendation 2

Specific to Recommendation 2, SSA disagreed. SSA stated the targeted level for Net Accuracy is not in conflict with the regulations. In fact, Net Accuracy is not mentioned in the regulations.

PwC RESPONSE

We disagree with the SSA's comment. Title 20 of the CFR, section 404.1643 (b), establishes the target level for decision accuracy at 99 percent. The definition of decision accuracy rate contained in the CFR is consistent with the data definition of this performance indicator contained in the PAR. Therefore, we continue to recommend that SSA disclose in the PAR (a) how the performance indicator goal of 97 percent contrasts with the decision accuracy rate target of 99 percent set forth in the CFR, and (b) the reason why the two targets differ.

Recommendation 5

Specific to Recommendation 5, SSA disagreed with the need to have multiple external GPRA measures for DDS accuracy. SSA stated that reporting allowances and denials separately would not provide a more meaningful indicator and would simply be redundant. If anything, it would cause more confusion for the public regarding the Quality Assurance (QA) process.

PwC RESPONSE

We continue to recommend that SSA consider including the accuracy rates for both allowances and denials. By including the individual error rates for the two types of decisions, SSA could improve the meaningfulness of the indicator.

Recommendation 7

Specific to Recommendation 7, SSA responded that SSA already documents and establishes internal controls over the collection, processing and reporting of data to ensure that our indicators are complete and adequate. SSA will continue to implement this process for any new indicators that are established in the future. The full text of SSA's comments can be found in Appendix E.

PwC RESPONSE

While the agency has documented certain aspects of the DDS Net Accuracy Rate performance reporting process, these documents do not contain detailed information that explains the statistical methodology and statistical formulas used to sample and project the results of the DQB reviews. In addition, SSA's computer programs did not contain enough imbedded information to allow an unfamiliar reader to understand the function of the program. Finally, policies and procedures were not documented that described (a) how the performance measure information was prepared, or (b) the automated and manual controls involved in the calculation and reporting of the performance indicator. By comprehensively documenting the process for calculating and reporting the performance indicators, SSA will further ensure that the indicator reporting life cycle is completed in a consistent and controlled manner.

Technical Comments

The full text of SSA's comments can be found in Appendix E.

PwC RESPONSE

PwC has updated the report. Regarding Appendix C, page 2, some of the modified text in SSA's comments was too voluminous to be included in the flow chart. However, all of the text was included in the narrative description of the process flow.

Appendices

APPENDIX A - Acronyms
APPENDIX B - Scope and Methodology
APPENDIX C - Process Flowcharts
APPENDIX D - Table 2 of the FY 2003 Net Accuracy Report
APPENDIX E - Agency Comments

Appendix A
Acronyms

CDR Continuing Disability Review
CFR Code of Federal Regulations
DDS Disability Determination Services
DICARS Disability Case Adjudication and Review System
DQB Disability Quality Branches
FY Fiscal Year
GAO Government Accountability Office
GPRA Government Performance and Results Act
NDDSS National Disability Determination Services System
OMB Office of Management and Budget
OQA Office of Quality Assurance and Performance Assessment
PAR Performance and Accountability Report
PART Program Assessment Rating Tool
Pub. L. Public Law
QA Quality Assurance
SSA Social Security Administration
U.S.C. United States Code

Appendix B
Scope and Methodology

We updated our understanding of the Social Security Administration's (SSA) Government Performance and Results Act (GPRA) processes. This was completed through research and inquiry of SSA management. We also requested SSA to provide various documents regarding the specific programs being measured, as well as the specific measurement used to assess the effectiveness and efficiency of the related program.

Through inquiry, observation and other substantive testing, we performed the following as applicable:

Reviewed prior SSA, Government Accountability Office, and other reports related to SSA GPRA performance and related information systems.
Met with the appropriate SSA personnel to confirm our understanding of each individual performance indicator.
Flowcharted the associated processes (see Appendix C).
Conducted and evaluated tests of the automated and manual controls within and surrounding the critical application to determine whether the tested controls were adequate to provide and maintain reliable data to be used when measuring the specific indicator.
Identified and extracted data elements from relevant systems and obtained source documents for detailed testing selections and analysis.
Identified attributes, rules, and assumptions for each defined data element or source document.
For those indicators with results that SSA determined using computerized data, we assessed the completeness and accuracy of that data to determine the data's reliability as it pertains to the objectives of the audit.
Tested the adequacy, accuracy, reasonableness, consistency, and completeness of the selection.
Recalculated the metric or algorithm of key performance indicators to ensure mathematical accuracy.

As part of this audit, we documented our understanding, as conveyed to us by SSA personnel, of the alignment of SSA's mission, goals, objectives, processes, and related performance indicators. We analyzed how these processes interacted with related processes within SSA and the existing measurement systems. Our understanding of SSA's mission, goals, objectives, and processes were used to determine if the performance indicators being used appear to be valid and appropriate. We followed the applicable performance audit standards.

In addition to the steps mentioned, we specifically performed the following to test the indicator included in this report:

Obtained an understanding and reviewed the Office of Quality Assurance and Performance Assessment (OQA) statistical methodology (including sample weighting and error estimation) for performing Quality Assurance Initial Disability Determination Reviews through interviews and meetings with appropriate OQA personnel.

Selected 45 cases from the OQA National Datafile on SSA's mainframe and reviewed each claim folder to ensure proper recording of review results in Disability Case Adjudication and Review System and whether the review properly established the beneficiary's disability status based on the evidence available in the claims folder.

Appendix C
Flowchart of DDS Net Accuracy Rate

DDS Net Accuracy Rate

Disability Determination Services (DDS) makes decisional input into State system.
Data is then uploaded to National Disability Determination Services System (NDDSS).
The automated sample selection process within NDDSS selects the files for review.
Case data is subsequently downloaded to Disability Case Adjudication and Review System (DICARS).
Case files are forwarded to the Disability Quality Branches (DQB) for review.
If documentation in the file is sufficient to support the disability determination, DQB approves the determination.
If documentation in file is insufficient to support the proposed disability determination, the case is returned for additional documentation. If the documentation supports a different determination, the case is returned for correction of the determination.
If a case is returned to the DDS, the DDS must return the corrected case to the DQB for completion of the review within 90 days.
Once the additional documentation or the corrected determination is provided to DQB, the DQB completes the review.
Review data and final results are entered into DICARS.
Results are uploaded to the Office of Quality Assurance and Performance Assessment National Datafile.
Results are summarized in the Net Accuracy Report. Determinations in which the determination did not change; i.e., the decision to allow or deny remained the same, are not counted as deficiencies for the purpose of the calculation of net accuracy. All other deficiencies; i.e., all deficiencies in which the determination changed, as well as the delinquent returns, are counted as changed decisions for the calculation of net accuracy.

Appendix D
Table 2 of the FY 2003 Net Accuracy Report

PricewaterhouseCoopers received the final Fiscal Year (FY) 2003 Disability Determination Services (DDS) Net Accuracy Report from the Office of Quality Assurance and Performance Assessment (OQA) on January 18, 2004. The report has undergone OQA's clearance process. It has been approved by all appropriate levels within OQA, and has been provided to all users. OQA's Associate Commissioner is aware of the results; however, he is not part of the report clearance process.

Table 2 includes the DDS Net Accuracy at the national level for FY 2003.

Appendix E
Agency Comments

SOCIAL SECURITY

MEMORANDUM

Date: September 24, 2004

To: Patrick P. O'Carroll, Jr.
Acting Inspector General

From: Larry W. Dye
Chief of Staff

Subject: Office of the Inspector General (OIG) Draft Report, "Performance Indicator Audit: Disability Determination Services Net Accuracy Rate - Allowances and Denials Combined" (A-15-04-14074)--INFORMATION

We appreciate OIG's efforts in conducting this review. Our comments on the draft report content and recommendations are attached.

Let me know if we can be of further assistance. Staff inquiries may be directed to Candace Skurnik, Director, Audit Management and Liaison Staff on extension 54636.

Attachment:
SSA Response

COMMENTS ON THE OFFICE OF THE INSPECTOR GENERAL (OIG) DRAFT PERFORMANCE INDICATOR AUDIT: "DISABILITY DETERMINATION SERVICES NET ACCURACY RATE - ALLOWANCES AND DENIALS COMBINED" (A-15-04-14074)

Thank you for the opportunity to review and comment on the draft report. We are pleased that the review found that the indicator is both meaningful and directly supports the strategic objective of "Make the right decision in the disability process as early as possible." We welcome external input, suggestions, and the contributions the Inspector General has made, and continues to make, in assisting us in developing a comprehensive performance measurement process. We too are committed in our efforts toward documenting relevant performance measures, ensuring our performance data are complete, and implementing adequate internal controls over our data systems and processes.

We strongly suggest rewording the second sentence of the third full paragraph on page 4 which reads, "The disability adjudication process involves a larger number of steps for denials than allowances, making denials inherently more error prone." The adjudication process, or sequential evaluation, is not different for allowances and denials. It is true that allowances are more frequently decided earlier in the process-thus there is more opportunity for error in the denial determination. However, we find the current language in the report to be potentially misleading to the outside reader.

Our responses to the specific recommendations are provided below.

Recommendation 1

Consider reporting interim results in the Performance and Accountability Report (PAR) or change the measurement period for this indicator to enable SSA to report current year results in the PAR.

Response

We agree. We understand OIG's comments supporting the usefulness of reporting the net accuracy measurement. However, net accuracy reports cannot be issued until three months after OQA has cited the original error. As a result, the net accuracy report for the previous fiscal year will not be available until mid-January. We will provide interim performance results in the FY 2004 PAR.

Recommendation 2

Disclose in the PAR how the performance indicator contrasts with the target set forth in the Code of Federal Regulations (CFR) and the reason why it differs from the CFR target.

Response

We disagree. The targeted level for Net Accuracy is not in conflict with the regulations. In fact, Net Accuracy is not mentioned in the regulations.

The references made in the report to regulation 20 CFR 404.1643 are incorrect. This regulation governs Performance Accuracy and its corresponding decisional accuracy--not "Net Accuracy." The concept and development of Net Accuracy came into existence in the 1990s, years after the regulations were published. Specifically, the regulation reads:

(b) Target level. The State agency initial performance accuracy target level for combined title II and title XVI cases is 97 percent with a corresponding decision accuracy rate of 99 percent.

(c) Intermediate Goals. These goals will be established annually by SSA's regional commissioner after negotiation with the State and should be used as stepping stones to progress towards our targeted level of performance.

(d) Threshold levels. The State agency initial performance accuracy threshold level for combined title II and title XVI cases is 90.6 percent."

Accordingly, there is a distinction between Performance Accuracy and Net Accuracy. Performance accuracy is the official accuracy standard mandated by the regulation 20 CFR 404.1643. Performance accuracy is the percentage of cases that do not have to be returned to State agencies for decision correction or the development of additional documentation. The State agency performance accuracy target level for combined Title II and Title XVI initial claims is 97 percent with a corresponding decision accuracy rate of 99 percent. Regulations mandate that the deficient case still be counted as an error even if the decision does not change. The performance standard is not just about "getting it right," it is about "doing it right." Consistent application of policy ensures equal treatment.

On the other hand, Net Accuracy is defined as the percentage of correct initial State agency disability determinations, i.e., all cases reviewed and determined to be correct, plus those corrected deficient cases whose decision did not change. Net accuracy counts as deficient those returned deficiencies whose decisions change upon correction, in addition to other returned deficiencies that are not corrected within 90 days from the end of the reporting period.

Net Accuracy is an alternate method of reporting DDS accuracy based on the "correctness" of the DDS disability determination. The Agency believes that Net Accuracy provides the public with a more accurate picture of the initial DDS determination.

Clearly we have made progress and are moving toward the ultimate target of 99 percent. But, at present, we are still in the intermediate goal category.

Recommendation 3

Ensure that the performance indicator titles, definitions, and goals are explicit, complete, and consistent.

Response

We agree. We will work with the sponsoring components to ensure performance titles, definitions, and goals are explicit, complete, and consistent in the Annual Performance Plan (APP) and PAR.

Recommendation 4

Provide comprehensive explanations on the relevance of non-comparable data included in the PAR narrative.

Response

See response to Recommendation 3. We agree that the historical data from 1999 through 2002 do not have any relevance since the performance measure was changed in FY 2003. The FY 2004 PAR will only present the data for FY 2003 and FY 2004.

Recommendation 5

Consider including the accuracy rates for allowances and denials.

Response

We disagree with the need to have multiple external GPRA measures for DDS accuracy. Reporting allowances and denials separately would not provide a more meaningful indicator and would simply be redundant. If anything, it would cause more confusion for the public regarding the Quality Assurance (QA) process. Changing how we report this indicator will not change how the Agency approaches tracking, analyzing, or taking appropriate action. The Agency tracks allowances and denials separately, analyzes trends, and takes appropriate action (e.g., providing feedback, training, staffing, funding, etc.) to improve the quality of the decision, not the accuracy.

We consider the current measure to be the most complete measure of accuracy. The Agency believes that reporting the combined Net Accuracy rate for allowances and denials provides a truer picture to the public of the overall correctness of the initial disability determination.

Recommendation 6

Include in the PAR the margin of error (i.e., quantification of sampling error) that results from projecting the sample error rate to the entire population of cases reviewed.

Response

We agree. Providing a margin of error would add more meaning to the performance measure and we will include the margin of error when we submit the data for inclusion in the next PAR.

Recommendation 7

Maintain documentation that describes how the performance indicator goals were established, document the policies and procedures used to prepare and report the results of the performance indicators, and keep a complete audit trail.

Response

SSA already documents and establishes internal controls over the collection, processing and reporting of data to ensure that our indicators are complete and adequate. We will continue to implement this process for any new indicators that are established in the future.

As an example, we would note that the comments in our computer programs are adequate in that they satisfactorily meet the needs of our users. The majority of the computer programs are written in COBOL, which uses many English terms, and is known for its ability to be followed easily by programmers. Also, most programs contain a short description in the "Program Description" section at the beginning of the listing, along with an explanation of the options that the program uses. In addition, each program contains a "Revision Log" section, where all revisions are documented, including the date of change and the nature of the change itself-in essence, an "audit trail." Outside of the programs themselves, the systems staff maintains an Excel spreadsheet of all COBOL programs. This spreadsheet lists the program name, a short description, the programmer name, and the input and output datasets used. We have discovered that the spreadsheet was not current and are working to update this immediately.

Recommendation 8

Ensure that SSA personnel do not have the ability, through inappropriate access, to directly modify, create or delete the datasets used to calculate the results of this indicator.

Response

We agree. We have implemented this recommendation. The report noted that seven OQA employees have the "All" access designation within the Top Secret security software to the datasets used to calculate the results of this indicator. Six of these seven comprise the OQA/Office of Disability Program Quality (OQA/ODPQ) Systems Staff. They are responsible for all programming pertaining to the Disability Quality Branch's (DQB) review data after it is transmitted to the IBM mainframe. This includes updating, maintaining, and reporting from the databases where the data are housed. All must have full access to be able to do their assigned tasks. Additionally, each has been an SSA employee for at least 20 years, most of which time has been spent working on this data. The seventh employee has been an SSA employee for over 25 years, part of which was on the OQA/ODPQ Systems Staff. He has since moved to a different job, although still within OQA. Because he no longer works with the mainframe datasets that contain Disability Case Adjudication and Review System (DICARS) data, his access has been completely removed. Further, the fact that this review, as well as the many others that have preceded it, has taken random samples from the database to ensure that the coded review results accurately reflect the case outcome, has established that access has been used appropriately and the data have not been manipulated.

Technical Comments

Page 2, Background, second paragraph, second sentence should be changed to read, "SSA's Quality Assurance (QA) Initial Disability Determination Reviews, which are conducted by the regional disability quality branches (DQB) within the Office of Quality Assurance and Performance Assessment (OQA)..."

Page 3, second paragraph should be changed to read, "The DDS Net Accuracy Rate reflects Group I deficiencies in initial determinations only. The Net Accuracy Rate is based on the number of deficient cases with changed disability decisions. In addition, deficient cases not corrected within 90 days from the end of the period covered by the report, are counted as changed decisions."

Page 4, third full paragraph, fourth sentence, delete "Therefore." As noted above in the response to recommendation 5, while the Agency tracks allowances and denials separately, this is done to analyze trends and provide feedback, training, staffing, and funding to improve the quality of the decision, not to measure accuracy. Thus, the report mischaracterizes our action.

For accuracy, Appendix C, page 2 should be revised, as follows:

DDS Net Accuracy Rate

Overview of the Office of the Inspector General

The Office of the Inspector General (OIG) is comprised of our Office of Investigations (OI), Office of Audit (OA), Office of the Chief Counsel to the Inspector General (OCCIG), and Office of Executive Operations (OEO). To ensure compliance with policies and procedures, internal controls, and professional standards, we also have a comprehensive Professional Responsibility and Quality Assurance program.

Office of Audit

OA conducts and/or supervises financial and performance audits of the Social Security Administration's (SSA) programs and operations and makes recommendations to ensure program objectives are achieved effectively and efficiently. Financial audits assess whether SSA's financial statements fairly present SSA's financial position, results of operations, and cash flow. Performance audits review the economy, efficiency, and effectiveness of SSA's programs and operations. OA also conducts short-term management and program evaluations and projects on issues of concern to SSA, Congress, and the general public.

Office of Investigations

OI conducts and coordinates investigative activity related to fraud, waste, abuse, and mismanagement in SSA programs and operations. This includes wrongdoing by applicants, beneficiaries, contractors, third parties, or SSA employees performing their official duties. This office serves as OIG liaison to the Department of Justice on all matters relating to the investigations of SSA programs and personnel. OI also conducts joint investigations with other Federal, State, and local law enforcement agencies.

Office of the Chief Counsel to the Inspector General

OCCIG provides independent legal advice and counsel to the IG on various matters, including statutes, regulations, legislation, and policy directives. OCCIG also advises the IG on investigative procedures and techniques, as well as on legal implications and conclusions to be drawn from audit and investigative material. Finally, OCCIG administers the Civil Monetary Penalty program.

Office of Executive Operations

OEO supports OIG by providing information resource management and systems security. OEO also coordinates OIG's budget, procurement, telecommunications, facilities, and human resources. In addition, OEO is the focal point for OIG's strategic planning function and the development and implementation of performance measures required by the Government Performance and Results Act of 1993.