OFFICE OF
THE INSPECTOR GENERAL

SOCIAL SECURITY ADMINISTRATION

ELECTRONIC FILE ASSEMBLY

June 2009

A-07-09-19069

EVALUATION REPORT

Mission

By conducting independent and objective audits, evaluations and investigations, we inspire public confidence in the integrity and security of SSA's programs and operations and protect them against fraud, waste and abuse. We provide timely, useful and reliable information and advice to Administration officials, Congress and the public.

Authority

The Inspector General Act created independent audit and investigative units, called the Office of Inspector General (OIG). The mission of the OIG, as spelled out in the Act, is to:

Conduct and supervise independent and objective audits and investigations relating to agency programs and operations.
Promote economy, effectiveness, and efficiency within the agency.
Prevent and detect fraud, waste, and abuse in agency programs and operations.
Review and make recommendations regarding existing and proposed legislation and regulations relating to agency programs and operations.
Keep the agency head and the Congress fully and currently informed of problems in agency programs and operations.

To ensure objectivity, the IG Act empowers the IG with:

Independence to determine what reviews to perform.
Access to all information necessary for the reviews.
Authority to publish findings and recommendations based on the reviews.

Vision

We strive for continual improvement in SSA's programs, operations and management by proactively seeking new ways to prevent and deter fraud, waste and abuse. We commit to integrity and excellence by supporting an environment that provides a valuable public service while encouraging employee development and retention and fostering diversity and innovation.

MEMORANDUM

Date: June 26, 2009

To: The Commissioner

From: Inspector General

Subject: Electronic File Assembly (A-07-09-19069)

OBJECTIVES

Our objectives were to (1) assess the results of the Social Security Administration's (SSA) Electronic File Assembly (ePulling) pilot project and (2) determine whether the assessment procedures were effective in deciding when the Office of Disability Adjudication and Review's (ODAR) hearing offices were ready to implement ePulling.

BACKGROUND

SSA's Deputy Commissioner for ODAR oversees SSA's national hearings and appeals operation of over 6,000 employees, including over 1,100 administrative law judges (ALJ). ODAR consists of a Headquarters complex that houses the Agency's Appeals Council and its support staff, 10 regional offices, 141 hearing offices and 2 national Hearing Centers. ODAR conducts due process hearings and appellate reviews, and issues decisions on appealed Retirement, Survivors, Disability, and Supplemental Security Income determinations. ODAR issues over 500,000 hearing decisions each year and is considered one of the largest administrative judicial systems in the world.

The number of cases awaiting a decision from an ALJ has risen from over 463,000 at the end of Fiscal Year (FY) 2002 to over 767,000 at the end of January 2009. To address this backlog of disability cases, ODAR has implemented, or is implementing, several initiatives to improve hearing office procedures, increase adjudicatory capacity, and increase efficiency with automation and improved business processes.

EPulling is one of ODAR's initiatives. ODAR expects file preparation using ePulling to be an improvement over its traditional file preparation process, which requires manual organization of documents in the electronic folder (EF). Specifically, hearing office staff, typically Senior Case Technicians (SCT), review the EF and identify the relevant documents for the ALJ to consider in making a decision. The SCTs manually enter relevant information into the EF, such as the type of document (that is, medical record), the source of the document (that is, medical provider), and the document's beginning and ending dates. The SCTs then use this information to sort and reorder the documents, identify duplicate documents, and prepare an exhibit list for the hearing. The exhibit list identifies the documents in the EF that will be presented and entered into the record at the hearing.

File preparation using ePulling software automates the manual organization of documents in the EF. Specifically, ePulling software

identifies and records relevant information (type of document, the source of the document, and the to and from dates of the document) for each page in the EF;

identifies potential duplicate documents and sequentially numbers pages in the EF; and

allows staff reviewing the EF to identify a specific type of document (such as X rays).

ODAR expects ePulling to increase the efficiency of the EF preparation process and reduce the time it takes to prepare a case for hearing. ODAR estimates that ePulling may result in an annual reduction of 402 work-years and a savings of $16.6 million. ODAR is piloting ePulling at the National Hearing Center in Falls Church, Virginia, and Hearing Offices in Tupelo, Mississippi; St. Louis, Missouri; Mobile, Alabama; Minneapolis, Minnesota; and Richmond, Virginia.

For our review, we interviewed ODAR's ePulling Project Manager as well as members of the ePulling cadres (the hearing office directors, group supervisors, ALJs, decision writers, and employees who prepared cases using ePulling) in the Tupelo, Mississippi, and Minneapolis, Minnesota, Hearing Offices. We also analyzed information ODAR collected on the ePulling pilot project including accuracy rates and case preparation times. Finally, we analyzed ODAR's methodology for assessing the results of ePulling. Additional information on our scope and methodology can be found in Appendix B.

RESULTS OF REVIEW

Based on our assessment of the results of the ePulling pilot project, ODAR is facing challenges with the accuracy of the ePulling software, which in turn is increasing ePulling's case preparation times. In addition, ODAR needs to establish a sufficient assessment methodology for measuring ePulling's impact on the hearings process. This assessment methodology is critical to future decisions on expanding the use of ePulling to other hearing offices.

THE ePULLING PILOT PROJECT

From June through December 2008, ePulling was used to prepare the files for 773 cases. To monitor the ePulling pilot, ODAR (1) assessed ePulling's accuracy in extracting page information from documents in the EF and (2) collected information from the pilot site participants on case preparation times using ePulling. We analyzed the information ODAR collected and conducted interviews with selected hearing offices.

Accuracy

For each page in the EF, the contractor was supposed to design the ePulling software to accurately identify the type of document (that is, a medical record), source of the document (that is, a medical provider), and beginning and ending dates of the document. According to the contract, the software should consistently identify this page-level information correctly at least 90 percent of the time.

The 773 cases prepared using ePulling contained 250,938 pages. For each page, ODAR assessed the ePulling software's accuracy in identifying four categories of information: type of document, document's source, document's beginning date, and document's ending date. ODAR found that multiple corrections were required for the pages processed. In fact, 433,790 corrections were required for the 250,938 pages processed. As shown in Table 1 on the following page, ePulling accurately identified page-level information about 57 percent of the time.

Table 1: ePulling Accuracy Rates
Page-Level Information Identified by the ePulling Software Total Pages Processed Pages Processed Accurately Pages Requiring Correction Accuracy Rate
Type of Document 250,938 163,267
87,671 65%
Source of Document 250,938 170,650
80,288 68%
Beginning Date of Document 250,938 122,453
128,485 49%
Ending Date of Document 250,938 113,592
137,346 45%
Totals 1,003,752
569,962
433,790 57%

ODAR worked with the software contractor to make enhancements aimed at improving the accuracy of the ePulling software. Since June 2008, the contractor has made five enhancements to the ePulling software. To determine whether the enhancements improved the accuracy of the software, ODAR processed the same 10 cases through each ePulling software enhancement. In addition to the 10 cases repeatedly tested, ODAR selected 5 to 10 new cases to test each enhancement. Using the accuracy rates derived from these 15 to 20 cases, ODAR computed the accuracy of the ePulling software enhancements. As shown in Table 2, following the January 2009 enhancement, the accuracy rates computed from this limited number of cases in the areas of type of document (65 percent) and source of document (75 percent) show accuracy rates comparable to the rates ODAR calculated for the 773 cases (see Table 1). In the area of dates of document (64 percent), the accuracy rates were slightly better for the 15 to 20 cases as compared to the accuracy rates reported for the 773 cases, which was in the 40-percent range. However, accuracy rates declined between June 2008 and January 2009. In addition, the accuracy rates based on the accuracy rates in Tables 1 and 2 show that the ePulling software accuracy was not within the 90-percent target goal.

Table 2: ePulling Software Enhancement Accuracy Rates
ePulling Software Enhancement Type of Document Source of Document Beginning and Ending Dates of Document
June 2008 76% 84% 96%
July 2008 70% 82% 80%
August 2008 70% 82% 80%
October 2008 66% 68% 84%
January 2009 65% 75% 64%

Case Preparation Times

We interviewed six employees in the Tupelo, Mississippi, Hearing Office and three employees in the Minneapolis, Minnesota, Hearing Office who prepared cases using ePulling. All nine employees stated that ePulling increased case preparation time when compared to the traditional EF preparation process.

The employees in the Tupelo Hearing Office stated it takes between 3 and 4 hours to prepare a file using ePulling compared to about 1.5 to 2 hours using the traditional EF preparation process.

The employees in the Minneapolis Hearing Office stated it takes about 3 hours to prepare a file using ePulling while it takes 1 to 2 hours using the traditional EF preparation process.

The employees stated the additional time is attributable to problems the ePulling software has in correctly identifying and classifying documents (based on the type, source, and dates of the documents). As a result, most documents processed by the ePulling software require time-consuming correction. However, the hearing office staff we interviewed were receptive to using ePulling and stated that if the accuracy problems were corrected, the software would be beneficial to the hearings process.

In addition to our interviews, we analyzed data ODAR collected on case preparation times at the pilot sites. For 523 of the 773 cases where ePulling was used for file preparation, ODAR employees recorded the time expended to prepare each case using ePulling. ODAR's compilation of the information showed that the average case preparation times for the pilot sites ranged from 2.5 to 4.9 hours per case. The following table shows the case preparation times by pilot site for the 523 cases.

Table 3: Case Preparation Times by ePulling Pilot Site
Pilot Sites Tupelo, MS Minneapolis, MN Mobile, AL St. Louis,
MO Richmond, VA Falls Church,
VA
Number of Cases Prepared 231 81 69 56 53 33
Average Case Preparation Time
(hours) 2.50 2.75 3.25 4.25 4.55 4.90

According to ODAR, the average traditional EF preparation times do not exist by pilot site. Comparison, by pilot site, between case preparation times using ePulling and traditional EF preparation times would have shown the true impact on file preparation times at each of the six pilot sites.

To assess the impact ePulling has on EF preparation times, ODAR informed us it is using 3 hours as the average traditional EF preparation time. As shown in Table 3, four of the six pilot sites are exceeding the 3-hour traditional EF preparation time when preparing cases using ePulling. The 3-hour average was obtained from an April 2008 Office of Quality Performance (OQP) survey of the ODAR business process. OQP derived the 3-hour average case preparation time based on interviews with a limited number of ODAR's workforce. Specifically, OQP interviewed one individual at each of eight hearing offices and several individuals from ODAR's Headquarters in Falls Church, Virginia. None of the eight hearing offices included in the survey participated in the ePulling pilot. In addition, OQP stated that when the survey was conducted, ODAR was facing performance issues with eView and the Case Processing Management System (CPMS). As a result, case preparation times may have been higher than normal. Therefore, we could not determine whether the OQP 3 hour estimate of the time to prepare a traditional EF can be improved upon.

To ensure ePulling does not increase EF preparation time, SSA should consider developing an estimate for the time it takes to prepare a case under the traditional EF preparation process based on an analysis of quantifiable historical data. We believe SSA should consider whether historical data can corroborate or improve the current OQP 3-hour case preparation time estimate. SSA may be able to improve the assessment of ePulling's impact on hearing office productivity by analyzing quantifiable historical data related to the time it takes to prepare a case under the traditional EF preparation process.

ASSESSMENT PROCEDURES FOR ePULLING

We found that ODAR had not established adequate assessment procedures for identifying and measuring ePulling's impact on hearing office productivity. Without adequate assessment procedures, ODAR cannot accurately identify when hearing offices are ready to implement ePulling. First, and as previously discussed in this report, SSA may be able to improve the assessment of the impact of ePulling EF preparation times by improving the 3-hour average traditional EF preparation time estimate. Second, management information needed to assess ePulling's impact to hearing office productivity was not collected. Third, the ePulling test case selection process may not result in case preparation times reflective of the hearing office workload.

Impact on Hearing Office Productivity

ODAR has not established procedures to ensure that management information needed to adequately assess the success of ePulling was collected. For example, ODAR expects ePulling's filtering function-that allows the identification of a specific type of document (such as X rays)-to result in significant savings for decision writers and ALJs. However, procedures were not established to measure ePulling's impact on decision writers' and ALJs' ability to process claims timely. Therefore, ePulling's impact on the productivity of the entire hearing office could not be measured. However, during our evaluation, SSA arranged for Booz Allen Hamilton (BAH) to perform an impact assessment of ePulling.

BAH began identifying a methodology for measuring the success of the ePulling initiative in December 2008. To provide ODAR the information needed to shape the future of ePulling, BAH will work with ODAR to

identify all parts of the ODAR business process where the potential exists for ePulling to provide benefits,
develop a methodology to measure the benefit in the ePulling pilot sites, and
gather information to measure ePulling's impact on ODAR's entire business process.

ePulling Test Case Selection Process

To assess the impact of ePulling on the hearings process, it is critical for ODAR to identify the difference between the amount of time hearing office staff spend preparing an ePulling case as compared to the amount of time staff spend under the traditional EF preparation process. Under the ePulling pilot, the cases selected for preparation were not based on any statistical process but rather left to the judgment of each pilot site. ODAR provided a general guideline that only EF cases containing 300 pages or less be included in the pilot. ODAR could not provide the percentage of total cases processed by hearing offices that were 300 pages or less. Therefore, the pilot testing of cases with 300 pages or less may not represent the hearing office workload from a case size perspective. Further, measuring the time captured from only smaller sized cases may not be appropriate to assess ePulling's impact on hearing office productivity.

CONCLUSION AND RECOMMENDATIONS

Based on the 773 cases prepared using the ePulling software, ODAR faces challenges with the accuracy of the software, which results in increased ePulling case preparation times. For the ePulling software to benefit the hearings process, it cannot result in increased case preparation times.

ODAR has arranged for BAH to develop an assessment methodology to measure the success of the ePulling initiative including the impact on hearing office productivity. Our report offers additional procedures that should be included in the assessment methodology.

We recommend that SSA:

1. Perform a complete assessment of the ePulling pilot project results before expanding the use of ePulling to other hearing offices. The assessment should ensure that ePulling will not adversely affect file preparation time or any other aspect of the hearings process.

2. Consider if historical data can corroborate or improve upon the current 3-hour case preparation time estimate used to assess ePulling's impact on hearing office productivity.

3. Determine whether the ePulling pilot testing should also include cases with more than 300 pages.

AGENCY COMMENTS

SSA agreed with our recommendations. The Agency's comments are included in
Appendix C.

Patrick P. O'Carroll, Jr.

Appendices
APPENDIX A - Acronyms

APPENDIX B - Scope and Methodology

APPENDIX C - Agency Comments

APPENDIX D - OIG Contacts and Staff Acknowledgments

Appendix A
Acronyms

ALJ Administrative Law Judge
BAH Booz Allen Hamilton
CPMS Case Processing Management System
ePulling Electronic File Assembly
EF Electronic Folder
FY Fiscal Year
ODAR Office of Disability Adjudication and Review
OQP Office of Quality Performance
SCT Senior Case Technician
SSA Social Security Administration

Appendix B
Scope and Methodology

We evaluated the Social Security Administration's (SSA) Electronic File Assembly (ePulling) pilot project results and assessment procedures to determine whether they were sufficient to decide when the Office of Disability Adjudication and Review's (ODAR) hearing offices were ready to implement ePulling. To accomplish our objectives, we:

Reviewed the budgeted and expended SSA funding to develop ePulling.
Reviewed the user requirements established for the ePulling project.
Reviewed system development documentation to obtain an understanding of how ePulling integrates into existing SSA computer systems.
Reviewed ODAR's estimation of expected time savings using ePulling versus the current process.
Analyzed ODAR's methodology for testing and assessing the ePulling pilot.
Determined how hearing offices, personnel, and ODAR case workload were selected to participate in the ePulling pilot.
Analyzed the accuracy rate and processing time of cases processed using ePulling.
Reviewed the ePulling training provided to the pilot hearing offices' staff.
Interviewed ODAR staff at the Tupelo, Mississippi, and Minneapolis, Minnesota, Hearing Offices to obtain their experiences with ePulling.
Identified outstanding problems with the performance of ePulling software and ODAR's planned resolution.
Determined whether SSA has procedures in place to communicate ePulling software deficiencies with the contractor for correction.

We performed our field work in the Kansas City, Missouri, Office of Audit; SSA Headquarters in Baltimore, Maryland; and Hearing Offices in Tupelo, Mississippi, and Minneapolis, Minnesota, from September 2008 through January 2009. We conducted this evaluation in accordance with the Quality Standards for Inspections issued by the President's Council on Integrity and Efficiency.

Appendix C
Agency Comments

MEMORANDUM

Date: June 01, 2009

To: Patrick P. O'Carroll, Jr.
Inspector General

From: James A. Winn
Chief of Staff

Subject: Office of the Inspector General (OIG) Draft Report, "Electronic File Assembly" (A-07-09-19069)--INFORMATION

Thank you for the opportunity to review and comment on the draft report. We appreciate OIG's efforts in conducting this review. We have attached our response to the report findings and recommendations.

Please let me know if we can be of further assistance. You may direct staff inquiries to Candace Skurnik, Director, Audit Management and Liaison Staff, at (410) 965-4636.

COMMENTS ON THE OFFICE OF THE INSPECTOR GENERAL (OIG) DRAFT REPORT, "ELECTRONIC FILE ASSEMBLY" (A-07-0-19069)

In general, we agree with the report and recommendations and provide responses to the specific recommendations below. We agree with the basic finding of the report that we need to perform a complete assessment of the project before expanding the use of the software in other offices. Staff from the Office of Disability Adjudication Review, the Office of Systems, and contractors from Booz-Allen Hamilton will visit all of the pilot offices in June 2009 to further assess the effectiveness of the software. Consistent with the basic findings of the report, this assessment will be a critical element in deciding if we should continue to develop ePulling.

Recommendation 1

Perform a complete assessment of the ePulling pilot project results before expanding the use of ePulling to other hearing offices. The assessment should ensure that ePulling would not adversely affect file preparation time or any other aspect of the hearings process.

Comment

We agree. We will not expand the use of ePulling until we are convinced that it will have a positive impact on hearing office productivity. This has been our position since the beginning of the pilot, and we remain committed to it. Our Office of Systems and the vendor have made numerous software enhancements, and we will assess the effect of these enhancements on productivity before we make a decision on expansion.

Recommendation 2

Consider if historical data can corroborate or improve upon the current 3-hour case preparation time estimate used to assess ePulling's impact on hearing office productivity.

Comment

We agree in principle. As the report notes, we are trying to identify accurate measures of the impact of ePulling on hearing office productivity. We have restructured our data collection surveys to obtain more complete information, and we are working with Booz-Allen Hamilton to identify assessment metrics. One element of our work with Booz-Allen Hamilton may include consideration of historical data in identifying metrics to assess the impact on hearing office productivity. We also are planning a study aimed at measuring the time it takes an ePulling cadre member to pull the same case using the normal pulling method versus ePulling.

Recommendation 3

Determine whether the ePulling pilot testing should also include cases with more than 300 pages.

Comment

We agree. After discussing this recommendation with the pilot offices, we have included a cohort of larger cases. We have not yet analyzed the effect of including these larger cases.

Appendix D
OIG Contacts and Staff Acknowledgments
OIG Contacts
Mark Bailey, Director, Kansas City Audit Division
Ron Bussell, Audit Manager
Acknowledgments
In addition to those named above:
Ken Bennett, IT Specialist
Doug Kelly, Senior Auditor

For additional copies of this report, please visit our website at www.ssa.gov/oig or contact the Office of the Inspector General's Public Affairs Specialist at (410) 965-3218. Refer to Common Identification Number A-07-09-19069.

Overview of the Office of the Inspector General
The Office of the Inspector General (OIG) is comprised of an Office of Audit (OA), Office of Investigations (OI), Office of the Counsel to the Inspector General (OCIG), Office of External Relations (OER), and Office of Technology and Resource Management (OTRM). To ensure compliance with policies and procedures, internal controls, and professional standards, the OIG also has a comprehensive Professional Responsibility and Quality Assurance program.

Office of Audit
OA conducts financial and performance audits of the Social Security Administration's (SSA) programs and operations and makes recommendations to ensure program objectives are achieved effectively and efficiently. Financial audits assess whether SSA's financial statements fairly present SSA's financial position, results of operations, and cash flow. Performance audits review the economy, efficiency, and effectiveness of SSA's programs and operations. OA also conducts short-term management reviews and program evaluations on issues of concern to SSA, Congress, and the general public.

Office of Investigations
OI conducts investigations related to fraud, waste, abuse, and mismanagement in SSA programs and operations. This includes wrongdoing by applicants, beneficiaries, contractors, third parties, or SSA employees performing their official duties. This office serves as liaison to the Department of Justice on all matters relating to the investigation of SSA programs and personnel. OI also conducts joint investigations with other Federal, State, and local law enforcement agencies.

Office of the Counsel to the Inspector General
OCIG provides independent legal advice and counsel to the IG on various matters, including statutes, regulations, legislation, and policy directives. OCIG also advises the IG on investigative procedures and techniques, as well as on legal implications and conclusions to be drawn from audit and investigative material. Also, OCCIG administers the Civil Monetary Penalty program.

Office of External Relations
OER manages OIG's external and public affairs programs, and serves as the principal advisor on news releases and in providing information to the various news reporting services. OER develops OIG's media and public information policies, directs OIG's external and public affairs programs, and serves as the primary contact for those seeking information about OIG. OER prepares OIG publications, speeches, and presentations to internal and external organizations, and responds to Congressional correspondence.

Office of Technology and Resource Management
OTRM supports OIG by providing information management and systems security. OTRM also coordinates OIG's budget, procurement, telecommunications, facilities, and human resources. In addition, OTRM is the focal point for OIG's strategic planning function, and the development and monitoring of performance measures. In addition, OTRM receives and assigns for action allegations of criminal and administrative violations of Social Security laws, identifies fugitives receiving benefit payments from SSA, and provides technological assistance to investigations.