Date: July 11, 2008

To: The Commissioner

From: Inspector General

Subject: Performance Indicator Audit: Disability Determination Services Processing (A-02-07-17131)

We contracted with PricewaterhouseCoopers, LLP (PwC) to evaluate 13 of the Social Security Administration's (SSA) performance indicators established to comply with the Government Performance and Results Act. Attached is the final report presenting the results of two of the performance indicators PwC reviewed. For the performance indicators included in this audit, PwC's objectives were to:

Assess the effectiveness of internal controls and test critical controls over data generation, calculation, and reporting processes for the specific performance indicator.
Assess the overall reliability of the performance indicator's computer-processed data. Data are reliable when they are complete, accurate, consistent and not subject to inappropriate alteration.
Test the accuracy of results presented and disclosed in SSA's Fiscal Year 2007 Performance and Accountability Reports.
Assess if the performance indicator provides a meaningful measurement of the program it measures and the achievement of its stated objective.

This report contains the results of the audit for the following indicators:

Disability Determination Services (DDS) net accuracy rate (allowances and denials combined).
Maintain the number of initial disability claims pending in the DDS (at/below FY 2007/2008 goal).

Please provide within 60 days a corrective action plan that addresses each recommendation. If you wish to discuss the final report, please call me or have your staff
Patrick P. O'Carroll, Jr.

OFFICE OF
THE INSPECTOR GENERAL

SOCIAL SECURITY ADMINISTRATION

PERFORMANCE INDICATOR AUDIT:
DISABILITY DETERMINATION
SERVICES PROCESSING

July 2008

A-02-07-17131

AUDIT REPORT

Mission

By conducting independent and objective audits, evaluations and investigations, we inspire public confidence in the integrity and security of SSA's programs and operations and protect them against fraud, waste and abuse. We provide timely, useful and reliable information and advice to Administration officials, Congress and the public.

Authority

The Inspector General Act created independent audit and investigative units, called the Office of Inspector General (OIG). The mission of the OIG, as spelled out in the Act, is to:

Conduct and supervise independent and objective audits and investigations relating to agency programs and operations.
Promote economy, effectiveness, and efficiency within the agency.
Prevent and detect fraud, waste, and abuse in agency programs and operations.
Review and make recommendations regarding existing and proposed legislation and regulations relating to agency programs and operations.
Keep the agency head and the Congress fully and currently informed of problems in agency programs and operations.

To ensure objectivity, the IG Act empowers the IG with:

Independence to determine what reviews to perform.
Access to all information necessary for the reviews.
Authority to publish findings and recommendations based on the reviews.

Vision

We strive for continual improvement in SSA's programs, operations and management by proactively seeking new ways to prevent and deter fraud, waste and abuse. We commit to integrity and excellence by supporting an environment that provides a valuable public service while encouraging employee development and retention and fostering diversity and innovation.

MEMORANDUM

Date: June 12, 2008

To: Inspector General

From: PricewaterhouseCoopers, LLP

Subject: Performance Indicator Audit: Disability Determination Services (DDS) Processing (A-02-07-17131)

OBJECTIVE

The Government Performance and Results Act of 1993 (GPRA) requires that the Social Security Administration (SSA) develop performance indicators that assess the relevant service levels and outcomes of each program activity. GPRA also calls for a description of the means employed to verify and validate the measured values used to report on program performance.

Our audit was conducted in accordance with generally accepted government auditing standards for performance audits. For the performance indicators included in this audit, our objectives were to:

1. Assess the effectiveness of internal controls and test critical controls over the data generation, calculation, and reporting processes for the specific performance indicator.

2. Assess the overall reliability of the performance indicator's computer-processed data. Data are reliable when they are complete, accurate, consistent and not subject to inappropriate alteration.

3. Test the accuracy of results presented and disclosed in SSA's Fiscal Year
(FY) 2007 Performance and Accountability Report (PAR).

4. Determine whether the performance indicator provides a meaningful measurement of the program it measures and the achievement of its stated objective.

BACKGROUND

We audited the following performance indicators as stated in the SSA FY 2007 PAR.

Performance Indicator
FY 2007 Goal
FY 2007 Reported Results

Disability Determination Services (DDS) net accuracy rate (allowances and denials combined) 97% 97%* **
Maintain the number of initial disability claims pending in the DDS (at/below FY 2007/2008 goal) 577,000 555,317

* The actual number is rounded to the nearest whole number using the standard rounding convention of rounding up numbers that are .5 or higher and rounding down numbers that are four or less.

** The performance indicator calculation reported in the PAR is through June 2007.

SSA administers the Old-Age and Survivors Insurance (OASI), Disability Insurance (DI), and Supplemental Security Income (SSI) programs. The OASI program, authorized by Title II of the Social Security Act, provides income for eligible workers and eligible members of their families and survivors. The DI program, also authorized by Title II of the Social Security Act, provides income for eligible workers with qualifying disabilities and eligible members of their families before those workers reach retirement age. The SSI program, authorized by Title XVI of the Social Security Act, was designed as a needs-based program to provide or supplement the income of aged, blind, and/or disabled individuals with limited income and resources.

To determine eligibility for both Title II and XVI programs, applicants must first file a claim with SSA. This is accomplished via the Internet, immediate claims taking via the telephone, or through an appointment or walk-in visit to 1 of SSA's approximately
1,300 field offices (FO). Interviews are conducted by FO personnel with applicants via the telephone or in person to determine the applicants' non-medical eligibility. If an applicant is filing for benefits based on disability, basic medical information concerning the disability, medical treatments, and identification of treating sources is obtained.

FO personnel input the applicant's information into the Modernized Claims System (MCS) for OASI and DI claims or the Modernized SSI Claims System (MSSICS) for SSI claims. The SSI Records Maintenance System (SSIRMS) is the back-end processing unit of the MSSICS application for SSI claims. This establishes the application and/or the claim's protective filing date. A relatively minor number of OASI and DI claims are input through the SSA Claims Control System (SSACCS). SSACCS is used to process claims that cannot be fully processed through MCS. For example, when a Title II record is established, the MCS application allows for entry of up to 11 claimants on the specific record. Additional claimants to a single MCS record would need to be recorded on SSACCS.

DI and SSI disability claims are sent to the appropriate State DDS office for review of medical information and determination of benefits. The State DDS offices input case determinations into the National Disability Determination Services System (NDDSS).

SSA has developed performance measures to determine the efficiency and accuracy of those reviews and determinations.

RESULTS OF REVIEW

For both indicators, SSA systems personnel had direct data access that would allow them to update production performance indicator data and weaknesses existed in the operating effectiveness of access controls related to transactions. Because of these internal controls weaknesses, we found the data used to support these indicators to be unreliable.

For the indicator, "DDS Net Accuracy Rate (allowances and denials combined)," management had not corrected issues raised in a previous audit report related to the meaningfulness and the accuracy of the presentation and disclosure of information.

For the indicator, "Maintain the number of initial disability claims pending in the DDS (at/below FY 2007/2008 goal)," an audit trail for transactions processed through the SSACCS application did not exist.

DDS Net Accuracy Rate (allowances and denials combined)

Indicator Background

Title 20 of the Code of Federal Regulations (C.F.R.), section 404.1643 (b), specifies DDS performance criteria, including the targeted rate of decisional accuracy for initial disability claims of 99 percent for combined Title II and Title XVI cases.

Section 221(c) of the Social Security Act also directs the Commissioner to review favorable and unfavorable determinations made by the DDSs. SSA's Quality Performance Initial Disability Determination Reviews, which are conducted by the regional disability quality branches (DQB) in the Office of Quality Performance (OQP), are used to determine the accuracy of the initial allowance and denial determinations made by the DDSs. The results of the reviews are compiled and used to calculate the DDS Net Accuracy Rate reported in SSA's FY 2007 PAR. Appendix C provides an overview of the OQP process for determining the accuracy rate.

The Automated Sample Selection Process module of NDDSS is used by DQB personnel to sample initial disability decisions made at each DDS. The Automated Sample Selection Process module is a Quality Assurance database that uses specific sampling features to determine the sample selection of cases to review. To ensure the appropriateness of the disability determination, the disability examiner (DE) reviews the sample case to determine whether the evidentiary record supports the determination and the evidence and determination conform to SSA operating policies and procedures. The results of the reviews are tracked entirely within the Disability Case Adjudication and Review System (DICARS). DE reviewers input findings into DICARS and the results are uploaded to the OQP National Datafile on SSA's mainframe for subsequent calculation of the indicator.

There are three groups of deficiencies identified by each DQB.

Group I deficiencies are substantive deficiencies that have the potential to affect the determination of eligibility.
Group II deficiencies are substantive deficiencies that affect only the onset, ending, or cessation date but do not affect the determination.
Group III technical deficiencies are instances of noncompliance with procedural requirements that do not affect the determination.

The DDS Net Accuracy Rate reflects only Group I deficiencies in initial determinations. Group I deficiencies identified by the reviewers are returned to the appropriate DDS with detailed directions for the required corrective action. If the DDS agrees with DQB's assessment, they will correct the deficiency and return the case to DQB for completion of the review and final case input. The DDSs may dispute any deficiency cited by DQB by using the Request for Program Consultation (RPC). The Office of Disability Programs (ODP) is the final arbiter. After staffing the RPC with a multi-component forum, if ODP disagrees with the DDS dispute, ODP will affirm the DQB's assessment; however, if ODP disagrees with the DQB's assessment, the deficiency is rescinded.

The Net Accuracy Rate is based on the number of deficient cases with changed disability decisions. In addition, deficient cases not corrected within 90 days from the end of the quarterly period covered by the report are counted as deficiencies.

Performance Indicator Calculation
DDS Net Accuracy Rate = 1-
(Number of corrected deficient cases which result in changed decisions
+
Number of deficient cases not corrected w/n 90 days)
Number of disability determinations reviewed
Findings
Internal Controls and Data Reliability

Our review of access controls revealed the following exceptions.

Two users had excessive access to the NDDSS Customer Information Control System (CICS) transactions and did not require this access to perform their job responsibilities. CICS is a transaction processing system designed for both on-line and batch activity. These transactions may create, update, and delete performance indicator data.
Programmers had update access to NDDSS production datasets and did not require this access to perform their job responsibilities.

The SSA Information System Security Handbook (ISSH) states, "Access to all SSA functions associated with software or enterprise systems must be managed based on need-to-know and least privilege. This specifically includes changes/updates to software, production jobs, and supporting hardware deployments. This access control maintenance policy must be applied across the SSA enterprise." In addition, Office of Management and Budget Circular A-130 requires that agencies implement the practice of least privilege whereby user access is restricted to the minimum necessary to perform his or her job; and enforce a separation of duties so steps in a critical function are divided among different individuals. It also emphasizes the importance of management controls - such as individual accountability requirements, separation of duties enforced by access controls, and limitations on the processing privileges of individuals - to prevent and detect inappropriate or unauthorized activities.

These issues, which were also noted during the FY 2007 financial statement audit, could result in the accidental or inappropriate alteration of the data used to support the performance indicator. It should be noted that, during the audit, SSA management removed the excessive application business user and programmer access to the NDDSS application. However, because this internal control weakness existed during the period of review, we did not find the performance indicator data to be reliable.

Meaningfulness and Accuracy of PAR Presentation and Disclosure

Generally, we found this performance indicator to be meaningful. However, some improvements could be made to clarify and improve the usefulness of this performance indicator. The PAR narrative states that the Net Accuracy Rate, which is the accuracy rate for allowances and denials combined, is an indicator of "…correct initial State disability determinations.…" Some allowances in the disability adjudication process are more frequently decided earlier in the process. This is due to the nature of the disability meeting certain criteria that require little to no additional research or supporting documentation, and therefore the probability of error in those cases is considerably low. In addition, there are certain expedients that are permitted in allowances cases that are not permitted in less than fully favorable or denial cases (for example, if the DDS can make a fully favorable allowance before all 12-month medical evidence of record [MER] are received, they are not required to wait for the receipt of the additional medical evidence. However, if the determination is less than fully favorable or a denial, the DDS must attempt to obtain all 12-month MER.). Denial determinations tend to require more research, supporting documentation and decision making; thus, creating a higher probability for error. The allowances and denials are added together to determine the combined accuracy rate for a DDS. Therefore, SSA's allowances accuracy rate could potentially increase while its denial accuracy rate decreased. OQP measures and tracks accuracy rates separately for allowances and denials. By including the individual error rates for the two types of decisions, SSA could improve the meaningfulness of the indicator.

We also found that the accuracy rate only covered the initial disability determinations made by the DDSs, even though the DDSs were also responsible for making the disability determinations for (a) Continuing Disability Reviews (CDR) and (b) Reconsideration requests. DQB staff also reviewed the accuracy of DDS decisions related to the CDRs and Reconsideration determinations; however, this information was excluded from the Net Accuracy Rate. In addition, only Group I initial disability determinations are included in the performance indicator count. The title of the performance indicator and the "data definition" section of the PAR do not clearly convey whether this performance indicator reflects the accuracy of all DDS workloads, including CDRs and reconsideration requests, as opposed to a single DDS workload. We found the data definition in the PAR specific to net accuracy to be misleading as the indicator is conceptually similar to the accuracy terms defined in Title 20 of the CFR, Section 404.1643 (b); however, the accuracy targets in the PAR of 97 percent differ from the accuracy targets in the CFR of 99 percent. Because of the conceptual similarities, the information in the PAR should clearly articulate the rationale for the differing accuracy targets.

Lastly, the narrative in the PAR did not include the margin of error (that is, quantification of sampling error) that results from extrapolating the sample error rate to the entire population of cases reviewed.

Finally, we obtained an extraction of the detailed data and recalculated the allowance and denial rates with insignificant differences. As a result, we have reasonable assurance that the data reported in the PAR for this indicator are complete, accurate, and consistent. However, the data cannot be considered reliable as the potential for inappropriate alteration existed since systems personnel and business users had update access to the applications used to support the calculation of the performance indicator.

Maintain the number of initial disability claims pending in the DDS
(at/below FY 2007/2008 goal)

Indicator Background

Upon determining an applicant has met the non-medical eligibility requirements, SSA sends the DI and SSI initial claims file to the DDS. The DDS is responsible for determining the status of a claimant's disability and ensuring adequate evidence is available to support the determination. When a claim determination is made by the DDS, the status is entered into the NDDSS as completed. If the DDS has not completed its review, the status of the claim in the NDDSS is pending. The data within NDDSS are automatically transferred to the Disability Insurance Operational Data Store (DIODS). For the FY 2007 PAR, the total number of pending initial disability claims was counted and reported as of September 28, 2007 on the State Agency Operations Report (SAOR).

Performance Indicator Calculation
Total Claims Pending for Title II and Title XVI
=
Total Workloads of Initial
Closed Pending Claims as of
September 28, 2007

Findings

Internal Controls and Data Reliability

Our review of access controls revealed the following exceptions.

Two users had excessive access to the MCS CICS transactions and did not require this level of access to perform their job responsibilities.
One programmer had excessive access to the MSSICS CICS front end screens, and that access was not being monitored by SSA management.
One programmer had update access to the SSIRMS production datasets and did not require this access to perform their job responsibilities.
Users and programmers had excessive update access to NDDSS transactions and datasets. Refer to the findings section on page 5 for additional details.

The SSA Information System Security Handbook (ISSH) states, "Access to all SSA functions associated with software or enterprise systems must be managed based on need-to-know and least privilege. This specifically includes changes/updates to software, production jobs, and supporting hardware deployments. This access control maintenance policy must be applied across the SSA enterprise." In addition, Office of Management and Budget Circular A-130 requires that agencies implement the practice of least privilege whereby user access is restricted to the minimum necessary to perform his or her job; and enforce a separation of duties so that steps in a critical function are divided among different individuals. It also emphasizes the importance of management controls - such as individual accountability requirements, separation of duties enforced by access controls, and limitations on the processing privileges of individuals - to prevent and detect inappropriate or unauthorized activities.

These issues, which were noted during the FY 2007 financial statement audit, could result in the accidental or inappropriate alteration of the data used to support the performance indicator. It should be noted that, during the audit, SSA management removed the excessive application business user and programmer access to the NDDSS application. However, because this internal control weakness existed during the period of review, we did not find the performance indicator data to be reliable.

Additionally, we found the DIODS data used to classify the disability claims as pending were not archived and maintained. SSA management stated the detailed data were not maintained because of limited data storage space and lack of personnel resources.

Although we found inappropriate access issues, and SSA did not maintain the archived information, we were able to perform alternative procedures to assess the accuracy of the performance indicator calculation. SSA was able to provide a copy of the system code used to generate the indicator results for our review. We concluded the code was designed to calculate the indicator results as described by SSA management. In addition, we selected numerous cases from DIODS, and compared the case information to the corresponding records in the Supplemental Security and Master Beneficiary Records. Also, we were able to observe the final calculation of this indicator on a real-time basis. We compared the final reported results of this indicator as reported in the PAR with the final data recorded on the SAOR report (which includes final indicator results). Our testing resulted in no exceptions with the code, the data in DIODS, or the results recorded in the PAR.

As a result of these tests, we have reasonable assurance that the data reported in the PAR for this indicator are complete, accurate, and consistent. However, the data cannot be considered reliable as the potential for inappropriate alteration existed since systems personnel and business users had update access to the applications used to support the calculation of the performance indicator.

In addition, we noted that an audit trail for transactions processed through the SSACCS was not created or reviewed. This could prevent management from reviewing and identifying inappropriate or unauthorized transactions being processed through SSACCS.

CONCLUSION AND RECOMMENDATIONS

For both indicators, we recommend SSA:

1. Restrict access to CICS screens and datasets for both systems based on the concept of least privilege access.

Specific to the performance indicator "DDS Net Accuracy Rate (allowances and denials combined)," we recommend SSA:

2. Disclose in the PAR the accuracy rates for allowances and denials.

3. Ensure that the performance indicator titles, definitions, and goals are explicit, complete, and consistent.

4. Disclose in the PAR how the performance indicator data definition is different from the terms in the C.F.R. and include the expected completion date for when the DDS will meet the C.F.R. target.

5. Disclose in the PAR the margin of error (that is, quantification of sampling error) that results from the projecting the sample error rate to the entire population of cases reviewed.

Specific to the performance indicator "Maintain the number of initial disability claims pending in the DDS (at/below FY 2007/2008 goal)," we recommend SSA:

6. Maintain an audit trail for SSACCS that captures the user identification, terminal, date, and time the transaction was processed. Policies and procedures should be implemented requiring a review of the audit trail for inappropriate access or processing of transactions. In lieu of making these changes to SSACCS, SSA should ensure that the SSACCS replacement system is configured with the appropriate audit trail controls. SSA management should follow-up on any suspicious activity and retain evidence of reviews and follow-up activities.

AGENCY COMMENTS AND PwC RESPONSE

The Agency agreed with Recommendations 1, 3, and 6; partially agreed with Recommendation 4; and disagreed with Recommendations 2 and 5. For Recommendation 4, SSA agreed it would be useful to include language in the PAR on the difference between the "DDS Net Accuracy Rate" performance measure and the C.F.R target but noted it could not provide an expected completion date for when the DDS accuracy rate and C.F.R target would meet. For Recommendation 2, SSA disagreed with disclosing accuracy rates for allowance and denials in the PAR. SSA stated that the net accuracy figure was the most appropriate for a high-level Agency performance report. Lastly, in disagreeing with Recommendation 5, SSA stated that information on the margin of error was so technical in nature that it would be difficult for the public to understand if it was included in the PAR. The Agency's comments are included in Appendix D.

In response, we believe that fully implementing Recommendations 2, 4 and 5 would strengthen the presentation on the DDS's accuracy rate in the PAR and provide useful information to the public.

Appendices
APPENDIX A - Acronyms
APPENDIX B - Scope and Methodology
APPENDIX C - Process Flowcharts
APPENDIX D - Agency Comments

Appendix A
Acronyms

CDR Continuing Disability Review
C.F.R. Code of Federal Regulations
CICS Customer Information Control System
DDS Disability Determination Services
DI Disability Insurance
DICARS Disability Case Adjudication and Review System
DIODS Disability Insurance Operational Data Store
DQB Disability Quality Branch
FO Field Office
FY Fiscal Year
GAO Government Accountability Office
GPRA Government Performance and Results Act
ISSH Information Systems Security Handbook
MCS Modernized Claims System
MER Medical Evidence of Record
MSSICS Modernized Supplemental Security Income Claims System
NDDSS National Disability Determination Services System
OASI Old-Age and Survivors Insurance
ODD Office of Disability Determinations
ODP Office of Disability Programs
OQP Office of Quality Performance
PAR Performance and Accountability Report
RPC Request for Program Consultation
SAOR State Agency Operations Report
SMS Strategic Management Staff
SSA Social Security Administration
SSACCS Social Security Administration Claims Control System
SSI Supplemental Security Income
SSIRMS Supplemental Security Income Records Maintenance System
TSC Tele-Service Center
U.S.C. United States Code

Appendix B
Scope and Methodology

To meet our audit objective, we updated our understanding of the Social Security Administration's (SSA) Government Performance and Results Act (GPRA) processes. This was completed through research and discussions with SSA management. We also requested SSA to provide various documents regarding the specific programs being measured, as well as the specific measurement used to assess the effectiveness and efficiency of the related program.

Through inquiry, observation, and other substantive testing, including testing of source documentation, we performed the following.

Reviewed prior SSA, Office of the Inspector General and other reports related to SSA's GPRA performance and related information systems.
Reviewed applicable laws, regulations and SSA policy.
Met with the appropriate SSA personnel to confirm our understanding of the performance indicator.
Flowcharted the processes. (See Appendix C.)
Tested key controls related to manual or basic computerized processes (for example, spreadsheets, databases).
Conducted and evaluated tests of the automated and manual controls within and surrounding each of the critical applications to determine whether the tested controls were adequate to provide and maintain reliable data to be used when measuring the specific indicator.
Identified attributes, rules, and assumptions for each defined data element or source document.
Recalculated the metric or algorithm of the performance indicator to ensure mathematical accuracy.
Assessed the completeness and accuracy of the data to determine the data's reliability as it pertains to the objectives of the audit and intended use of the data.

As part of this audit, we documented our understanding, as conveyed to us by Agency personnel, of the alignment of the Agency's mission, goals, objectives, and related performance indicators. We analyzed how these processes interacted with related processes within SSA and the existing measurement systems. We then determined whether the performance indicator appeared to be valid and appropriate given SSA's mission, goals, objectives.

We followed all performance audit standards in accordance with generally accepted government auditing standards.
In addition to these steps, we specifically performed the following to test the performance indicators included in this report.

Specific to the performance indicator "Disability Determination Services (DDS) Net Accuracy Rate (allowances and denials combined)"

Inspected relevant policies and procedures, as necessary.
Obtained an understanding and reviewed the Office of Quality Performance (OQP) statistical methodology (including sample weighting and error estimation) for performing Quality Assurance Initial Disability Determination Reviews through interviews and meetings with appropriate OQP personnel.
Completed a general computer controls review as it relates to National Disability Determination Services System (NDDSS).
Reviewed the process for controlling access to the Disability Case Adjudication and Review System (DICARS) datasets storing the indicator data and tested the appropriateness of the access privileges granted to the datasets for a selection of SSA personnel.
Compared the Net Accuracy Report results for the performance indicator for Fiscal Year (FY) 2007 to the number reported in the Performance Accountability Report (PAR).

Specific to the performance indicator "Maintain the number of initial disability claims pending in the DDS (at/below FY 2007/2008 goal)"

Inspected relevant policies and procedures, as necessary.
Determined the adequacy of the programming logic used by SSA to count the number of initial disability claims pending in the DDS.
Updated our understanding of the Social Security Administration Claims Control System (SSACCS).
Completed an application controls review for the Modernized Claims System (MCS).
o Inspected a selection of users to determine whether their access to MCS transactions and datasets was appropriate.
o Performed Computer Assisted Audit Tests over MCS data to determine whether programmed edits and validations were operating as intended.
o Inspected a selection of sysouts to determine whether the data processed completely.
o Inspected a selection of disability records to determine whether the disability decision was accurately transferred from NDDSS to MCS.
Completed an application controls review for NDDSS.
o Inspected a selection of users to determine whether their access to NDDSS transactions and datasets was appropriate.
o Inspected a selection of sysouts to determine whether the data processed completely.
o Inspected the interface records from NDDSS to the application Disability Operational Data Store.
Completed an application review for Modernized Supplemental Security Income Claims System (MSSICS) and Supplemental Security Income Records Maintenance System (SSIRMS)
o Inspected a selection of users to determine whether their access to MSSICS transactions and SSIRMS datasets was appropriate.
o Performed Computer Assisted Audit Tests over MSSICS data to determine whether programmed edits and validations were operating as intended.
o Inspected a selection of sysouts to determine whether the data processed completely.
o Inspected a selection of disability records to determine whether the disability decision was accurately transferred from NDDSS to MSSICS.
Reviewed the process for controlling access to the Disability Insurance Operational Data Store datasets storing the indicator data and tested the appropriateness of the access privileges granted to the datasets for a selection of SSA personnel.
Compared the State Agency Operations Report results of the performance indicator for FY 2007 to the number reported in the PAR.

Appendix C
Disability Determination Services Net Accuracy Rate (allowances and denials combined) - Flowchart
Disability Determination Services Net Accuracy (allowances and denial combined) - Narrative
DDS makes decisional input into State system.
Data is then uploaded to the NDDSS.
The automated sample selection process module within NDDSS selects cases for quality review.
If the case is selected for review, case data are downloaded to the DICARS and case files are sent to the DQB for review. Also, the DDS is notified that the case locked to prevent modification by the DDS.
Selected claims are reviewed by the DQB disability examiners by replicating the DDS process as closely as possible. Medical staff are selectively referred if the case warrants their review.
If documentation in the file is sufficient to support the disability determination, DQB approves the determination.
If documentation in the file is insufficient to support the proposed disability determination, the case is returned for additional documentation. If the documentation supports a different determination, the case is returned for correction of the determination.
If a case is returned to the DDS, the DDS must return the corrected case to the DQB for completion of the review within 90 days. Group I deficiencies with a recommendation from the DQB to change the decision which are not returned within 90 days will be considered a Net Accuracy Error.
If the DDS disagrees with the actions requested by the DQB, the DDS may immediately submit a Request for Program Consultation (RPC) for resolution of the disagreement. After the RPC is staffed in a multi-component forum (usually within a few days), the Office of Disability Programs (ODP) is the final arbiter and makes the decision to affirm or rescind the DQB's deficiency.
If the DDS does not respond to a DQB return within 90 days, the OQP reviews the case and decides whether the deficiency will be upheld.
Upon receiving additional documentation or the corrected determination from the DDS, the DQB completes the review.
Review data and final results are entered into DICARS.
Results are uploaded to the OQP National Datafile.

Results are summarized in the Net Accuracy Report. Deficiencies in which the determination changed and delinquent returns are counted as net accuracy errors in the calculation of net accuracy.

SMS publishes the reported Government Performance and Results Act (GPRA) results in the annual Performance Accountability Report (PAR).

Maintain the Number of Initial Disability Claims Pending in the DDS (at/below FY 2007/2008 goal) - Flowchart

Maintain the Number of Initial Disability Claims Pending in the DDS (at/below FY 2007/2008 goal) - Narrative
Claimant contacts the Social Security Administration (SSA) via an FO in-person visit, mail, or telephone call to the FO or TSC or online via the ISBA.
If the FO or TSC can interview the claimant, the FO or TSC will verify non-medical factors.
If the FO or TSC is not available to interview the claimant, the FO or TSC will set up an in-office or telephone interview.
During the interview, the FO personnel's review determines whether the claimant is eligible for Title II and/or Title XVI benefits. If the claimant does not qualify for Title II and/or Title XVI benefits, the claimant can continue or stop the filing of the application.
Claimants that are eligible for Title II or Title XVI benefits complete the application form. The FO personnel enter the Title II application into the MCS or SSACCS. The FO personnel enter the Title XVI application in MSSICS.
The FO personnel review non-medical issues and determine the claimant's effective filing date.
If the determination is a technical denial, the FO personnel will enter the decision.
If the determination is not a technical denial, a medical folder is created for the claimant and sent to the State DDS for the review of medical factors and disability determination.
The NDDSS receives the claimant's data from MCS, SSACCS and MSSICS.
The NDDSS provides the total number of pending disability claims to the DIODS.
The DIODS produces the pending disability claims count on a weekly basis and is reported on the SAOR.
ODD staff analyzes the SAOR report to identify anomalies and corrects errors, if applicable.
The year-end SAOR, dated September 28, 2007, was used to record the performance indicator results reported in the FY 2007 PAR.

Appendix D
Agency Comments

SOCIAL SECURITY

MEMORANDUM

Date: June 9, 2008

To: Patrick P. O'Carroll, Jr.
Inspector General

From: David V. Foster /s/
Acting Chief of Staff

Subject: Office of the Inspector General (OIG) Draft report, "Performance Indicator Audit: Disability Determination Services Processing" (A-02-07-17131)--INFORMATION

We appreciate OIG's efforts in conducting this review. Our response to the report findings and recommendations are attached.

Please let me know if we can be of further assistance. Staff inquiries may be directed to Ms. Candace Skurnik, Director, Audit Management and Liaison Staff, at extension 54636.

SSA Response

COMMENTS ON THE OFFICE OF THE INSPECTOR GENERAL (OIG) DRAFT REPORT, "PERFORMANCE INDICATOR AUDIT: DISABILITY DETERMINATION SERVICES PROCESSING" (A-02-07-17131)

Thank you for the opportunity to review and provide comments on this draft report. Our response to the specific recommendations is as follows.

For both performance indicators:

Recommendation 1

Restrict access to Customer Information Control System screens and datasets for both systems based on the concept of least privilege access.

Comment

We agree. As indicated in the report, we have removed the excessive application business user and programmer access to the National Disability Determination Services System application.

With regard to the performance indicator "Disability Determination Services (DDS) Net Accuracy Rate," it was noted in the narrative findings that this deficiency had been corrected during the audit period. As such, it should be equally noted in the "Conclusion and Recommendations" that this has been remedied.

Specific to the performance indicator, "DDS Net Accuracy Rate (allowances and denials combined)":

Recommendation 2

Disclose in the Performance Accountability Report (PAR) the accuracy rates for allowances and denials.

Comment

We disagree. State DDSs make decisions at a high level of accuracy in all cases, regardless of the case outcome. We believe that the overall rate is the most appropriate performance measure indicator. Identifying the combined allowance and denial net accuracy rate provides the public with an accurate depiction of the entire universe of initial cases, but also corresponds with Federal Regulations that monitor the combined allowance and denial accuracy rate. Additional information is available for those who want detailed performance data; however, the reported net accuracy figure is the most appropriate for a high-level Agency performance report.

Recommendation 3

Ensure that the performance indicator titles, definitions, and goals are explicit, complete, and consistent.

Comment

We agree. It is important that SSA's performance measures are clear and precise, and that the accompanying definitions accurately reflect how and what is being measured. As noted on page 6 of the report in footnote 17, we have agreed to change the title of the DDS Net Accuracy Rate goal to "DDS Initial Net Accuracy Rate." In order to maintain consistency with previous PARs, clarifying the definition of the performance indicator to state that the net accuracy rate only covers initial disability determinations is a better solution than changing the indicator.

Recommendation 4

Disclose in the PAR how the performance indicator data definition is different from the terms in the Code of Federal Regulations (CFR) and include the expected completion date for when the DDSs will meet the CFR target.

Comment

We agree with the first part of the recommendation. We agree that language addressing the difference between the data definition for this performance indicator and the terminology in the CFR would be meaningful. We will develop language for inclusion in the fiscal year 2008 PAR.

We disagree with the second part of the recommendation regarding the expected completion date for when the DDSs will meet the CFR target. We set the targets based on resources and priorities. We realize that there is an optimal level of performance as shown in the CFR, but we do not anticipate reaching this target in the near future. As we revisit the target each year and again consider resources and priorities, we will strive to meet that level, but cannot give you an expected completion date.

Recommendation 5

Disclose in the PAR the margin of error (that is, quantification of sampling error) that results from projecting the sample error rate to the entire population of cases reviewed.

Comment

We disagree. In our monthly Net Accuracy reports, we include statistics on the sampling error estimates. We do not believe that additional statistical qualification should be included in the PAR. The information is so technical in nature that it would be very difficult for the public to understand. As a result, there would be little value added.

Specific to the performance indicator, "Maintain the number of initial disability claims pending in the DDS (at/below fiscal year 2007/2008 goal):"

Recommendation 6

Maintain an audit trail for SSA Claims Control System (SSACCS) that captures the user identification, terminal, date, and time the transaction was processed. Policies and procedures should be implemented requiring a review of the audit trail for inappropriate access or processing of transactions. In lieu of making these changes to SSACCS, SSA should ensure that the SSACCS replacement system is configured with the appropriate audit trail controls. SSA management should follow-up on any suspicious activity and retain evidence of reviews and follow-up activities.

Comment

We partially agree. We are phasing out SSACCS. Therefore, it is cost-prohibitive to maintain an audit trail for this system's transactions. The Office of Management and Budget's Circular A-11, section 230.2e states, "Performance data need not be perfect to be reliable, particularly if the cost and effort to secure the best performance data will exceed the value of any data so obtained However, we agree that the replacement system should be designed with appropriate audit trail controls.

[SSA also included one technical comment, which was addressed in the text of the report.]

Overview of the Office of the Inspector General
The Office of the Inspector General (OIG) is comprised of an Office of Audit (OA), Office of Investigations (OI), Office of the Counsel to the Inspector General (OCIG), Office of External Relations (OER), and Office of Technology and Resource Management (OTRM). To ensure compliance with policies and procedures, internal controls, and professional standards, the OIG also has a comprehensive Professional Responsibility and Quality Assurance program.

Office of Audit
OA conducts financial and performance audits of the Social Security Administration's (SSA) programs and operations and makes recommendations to ensure program objectives are achieved effectively and efficiently. Financial audits assess whether SSA's financial statements fairly present SSA's financial position, results of operations, and cash flow. Performance audits review the economy, efficiency, and effectiveness of SSA's programs and operations. OA also conducts short-term management reviews and program evaluations on issues of concern to SSA, Congress, and the general public.

Office of Investigations
OI conducts investigations related to fraud, waste, abuse, and mismanagement in SSA programs and operations. This includes wrongdoing by applicants, beneficiaries, contractors, third parties, or SSA employees performing their official duties. This office serves as liaison to the Department of Justice on all matters relating to the investigation of SSA programs and personnel. OI also conducts joint investigations with other Federal, State, and local law enforcement agencies.

Office of the Counsel to the Inspector General
OCIG provides independent legal advice and counsel to the IG on various matters, including statutes, regulations, legislation, and policy directives. OCIG also advises the IG on investigative procedures and techniques, as well as on legal implications and conclusions to be drawn from audit and investigative material. Also, OCIG administers the Civil Monetary Penalty program.

Office of External Relations
OER manages OIG's external and public affairs programs, and serves as the principal advisor on news releases and in providing information to the various news reporting services. OER develops OIG's media and public information policies, directs OIG's external and public affairs programs, and serves as the primary contact for those seeking information about OIG. OER prepares OIG publications, speeches, and presentations to internal and external organizations, and responds to Congressional correspondence.

Office of Technology and Resource Management
OTRM supports OIG by providing information management and systems security. OTRM also coordinates OIG's budget, procurement, telecommunications, facilities, and human resources. In addition, OTRM is the focal point for OIG's strategic planning function, and the development and monitoring of performance measures. In addition, OTRM receives and assigns for action allegations of criminal and administrative violations of Social Security laws, identifies fugitives receiving benefit payments from SSA, and provides technological assistance to investigations.