MEMORANDUM

Date: September 30, 2009

To: The Commissioner

From: Inspector General

Subject: Quick Response Evaluation: Self-Help Personal Computer Pilot (A-14-10-11001)

The attached final report presents the results of our review. Our objective was to determine the effectiveness of the Social Security Administration’s Self-Help Personal Computer pilot. Specifically, this evaluation focused on the impact of Self-Help Personal Computers on electronic services’ use and field office performance as well as security concerns related to the public’s use of Agency computers.

If you wish to discuss the final report, please call me or have your staff contact Steven L. Schaeffer, Assistant Inspector General for Audit, at (410) 965-9700.

Patrick P. O'Carroll, Jr.

QUICK RESPONSE
EVALUATION

Self-Help Personal Computer Pilot

A-14-10-11001

September 2009

Mission

By conducting independent and objective audits, evaluations and investigations, we inspire public confidence in the integrity and security of SSA’s programs and operations and protect them against fraud, waste and abuse. We provide timely, useful and reliable information and advice to Administration officials, Congress and the public.

Authority

The Inspector General Act created independent audit and investigative units, called the Office of Inspector General (OIG). The mission of the OIG, as spelled out in the Act, is to:

 Conduct and supervise independent and objective audits and investigations relating to agency programs and operations.
 Promote economy, effectiveness, and efficiency within the agency.
 Prevent and detect fraud, waste, and abuse in agency programs and operations.
 Review and make recommendations regarding existing and proposed legislation and regulations relating to agency programs and operations.
 Keep the agency head and the Congress fully and currently informed of problems in agency programs and operations.

To ensure objectivity, the IG Act empowers the IG with:

 Independence to determine what reviews to perform.
 Access to all information necessary for the reviews.
 Authority to publish findings and recommendations based on the reviews.

Vision

We strive for continual improvement in SSA’s programs, operations and management by proactively seeking new ways to prevent and deter fraud, waste and abuse. We commit to integrity and excellence by supporting an environment that provides a valuable public service while encouraging employee development and retention and fostering diversity and innovation.

Background
OBJECTIVE

The objective of this evaluation was to determine the effectiveness of the Social Security Administration’s (SSA) Self-Help Personal Computer (PC) pilot. Specifically, this evaluation focused on the impact of Self-Help PCs on electronic services’ (eServices) use and field office (FO) performance, as well as security concerns related to the public’s use of Agency computers.

BACKGROUND

SSA has administered its services primarily through face-to-face contact with the public at one of its FOs or by telephone. However, SSA is facing an avalanche of retirement and disability claims at the same time it must address large backlogs due to years of increasing workloads and limited resources. To keep FOs from being overwhelmed, the Agency must find alternative methods to conduct business with the public. One of these alternatives is through the use of the Internet.

In September 2008, SSA issued its Strategic Plan, which included the Agency’s plan for long-term improvement of customer service to the public. One of SSA’s strategic goals is to improve retiree and other core services, which includes the objectives of dramatically increasing baby boomers’ use of SSA’s on-line retirement services and improving service for individuals who visit FOs.

One of the initiatives to address this strategic goal is SSA’s Self-Help PC pilot, which offers FO visitors the option of using FO PCs to conduct their business. Users can apply for retirement and disability benefits, request a Proof of Income letter, change their address, appeal a disability decision, and more. Self-Help PCs connect to the Agency’s existing eServices on SSA’s Website—applications not specifically designed for the FO environment.

The Self-Help PC pilot is part of the Commissioner’s Space Modernization and Reception Transformation (SMART) service initiative. The goal is to make FO space more efficient for conducting business. The pilot also supports the E Government Act of 2002, which requires that Federal agencies support the efforts of the General Services Administration to develop, maintain, and promote an integrated Internet-based system of delivering Federal information and services to the public.

Self-Help PCs were available to the public as early as October 2007. A national rollout began in February 2009 and was phased in through March 2009 to 58 sites. The Agency provided between one and three PCs to each site.

One of the expectations of the pilot was to determine the amount of assistance the public needs to complete Internet transactions. Accordingly, the 58 pilot sites were divided into 2 models, attended and unattended. Attended models had designated SSA employees to assist the public at Self-Help PCs, while unattended models did not have employees readily available for assistance. As a result of space or potential security issues, only six sites were able to be set up as unattended models. These six sites comprised three level-1 FOs and three level-2 FOs. The other 52 sites were attended models that comprised 38 level-1 FOs and 14 level-2 FOs.

To assess the amount of assistance needed, as well as determine the impact of Self-Help PCs on FO workflow and waiting times, the Agency planned several evaluation techniques. At attended sites, the Agency designed surveys for employees to complete every time a Self-Help PC was used. At unattended sites, surveys were available for users to complete if they wanted to provide the Agency feedback. The Agency also planned to track usage electronically and gather user feedback from a survey conducted by the Office of Quality Performance (OQP). Proper evaluation of the pilot supports the Clinger-Cohen Act of 1996, which requires that the Agency design and implement a process to maximize the value, assess and manage the risks of its information technology acquisitions, and provide a means for senior management to obtain timely information regarding the progress of an information system investment.

Results of Review
Based on our review of SSA’s Self-Help PC pilot documentation and interviews with Headquarters and FO personnel responsible for administering the pilot, we could not evaluate the pilot’s effectiveness because user feedback and usage information were insufficient. Surveys were the primary mechanism for collecting Self-Help PC user feedback; however, we determined employee surveys conducted at attended sites were not consistently completed, and the responses may not accurately reflect the overall user experience. Surveys from unattended sites provided limited feedback, with half the sites not receiving any completed surveys over a 12-week period. Furthermore, the Agency has been unable to accurately track usage through surveys or electronic usage logs. As such, improving the Self-Help PC pilot evaluation process is critical to SSA’s ability to assess and manage the implementation of such a key information technology investment and assist the Agency in meeting the requirements of the Clinger-Cohen Act of 1996. We believe the Agency can improve its evaluation techniques by obtaining Office of Management and Budget (OMB) approval to directly question Self-Help PC users and by using Visitor Intake Process (VIP) data in conjunction with employee surveys to better analyze Self-Help PC usage.

With respect to security, the Self-Help PCs we tested were physically secure, and the Agency had taken steps to ensure the public’s use of its computers did not threaten SSA’s production environment. The Agency also developed signs/posters to ensure pilot site visitors were aware of Self-Help PC functions, that use of the PCs was limited to SSA business, and of how to protect their personal information. According to FO personnel, the Self-Help PCs have helped with office workloads.

PILOT EVALUATION TECHNIQUES

In reviewing SSA’s survey practices, we noted opportunities for the Agency to improve its overall approach for assessing the Self-Help PC pilot. We believe the fundamental areas of focus should be to

bull; obtain feedback that reflects the overall user experience; and
bull; accurately monitor Self-Help PC usage.

User Feedback

As noted in the Background Section, the 58 Self-Help PC pilot sites were divided into 52 attended sites and 6 unattended sites. Attended sites had designated SSA employees to assist the public at Self-Help PCs, while unattended sites did not have employees readily available for assistance. The Agency used surveys as the primary method of capturing feedback from all Self-Help PC pilot sites, but these surveys differed between attended and unattended sites, as discussed below.

Attended Self-Help PC Pilot Sites

The Agency established a procedure at attended sites for designated employees to complete an on-line survey each time the Self-Help PC was used. Self-Help PC users were not asked to complete the surveys. Rather, assigned employees completed surveys based on their observations and provided information on

bull; the transaction’s start and end time;
bull; the type of transaction;
bull; whether the user was able to complete the transaction, and if not, why;
bull; whether the user inquired about a transaction not available on the Self-Help PC;
bull; if employee assistance was necessary and if so, for how long; and
bull; any other comments offered by users.

During our visit to an attended site, we interviewed five FO employees who had been designated to assist users on Self-Help PCs. Although Agency policy requires that the designated FO employee complete a survey every time a visitor uses a Self-Help PC, we found this was not consistently done. We also compared the number of employee surveys submitted to FO traffic. As shown in the chart below, the number of employee surveys submitted varies considerably among similarly sized offices. In addition, when compared to average weekly office traffic, the number of surveys completed is low.

Number of Employee Survey Responses Submitted Level-1 FOs Level-2 FOs

Number of
Pilot Sites Average Weekly Office Traffic
Number of
Pilot Sites Average Weekly Office Traffic
100+ 7 1,185 0 0
50-99 3 1,076 2 535
25-49 8 1,340 2 445
1-24 19 1,197 8 432
0 0 0 1 558

Although the Agency’s goal was to capture information that represents the entire population of Self-Help PC users at attended sites, our visit to an attended site, combined with the variance in the number of employee survey responses submitted among similarly sized offices, suggests unplanned “selectiveness.” As a result, the survey responses may not fully reflect the overall user experience.

While we acknowledge and support the importance of gathering employee feedback on the Self-Help PC pilot, we believe users should be more involved in providing feedback. In the current process, designated employees at attended sites record their observations regarding user experiences with Self-Help PCs. However, users may not communicate all their thoughts to the designated employees, and as a result of breaks or assisting other users, the designated employees may not be present for the entire transaction. Furthermore, the validity of general conclusions drawn from user comments may be questionable because the responses may reflect unintended biases for or against the pilot.

Unattended Self-Help PC Pilot Sites

At the six unattended pilot sites, paper surveys were available for users to complete if they wanted to provide the Agency with feedback. Only 34 surveys were collected over a 12-week period, and 3 of the 6 sites did not have any surveys returned. As a result, this method provided limited feedback.
Feedback from employee surveys may not fully and accurately reflect the overall user experience at attended Self-Help PC pilot sites, and the Agency collected limited feedback from unattended sites. If the collected feedback does not fully and accurately reflect users’ experiences with Self-Help PCs, the Agency may not be able to ensure it is maximizing the value of the PCs in accordance with the Clinger-Cohen Act of 1996.

We discuss the process used by the Agency to collect Self-Help PC usage information in the section below.

Self-Help PC Usage

At the 52 attended Self-Help PC pilot sites, the Agency used the employee surveys discussed above as the primary mechanism to determine usage. We determined these surveys were not consistently completed every time a visitor used a Self-Help PC. To further evaluate the collection of employee surveys as a means to track Self-Help PC usage, we obtained a VIP report listing Self-Help PC use at an attended pilot site over a 4 day period. The report provided information including the date, user name, SSN, transaction start and end times, and transaction type for FO visitors indicating they intended to use a Self-Help PC when they checked in. The VIP report indicated that surveys were only completed for one-third of Self-Help PC users over the period. SSA staff stated the VIP report may be misleading because visitors may indicate in VIP that they planned to use a Self-Help PC but never completed the transaction using the PC.

At the six unattended Self-Help PC pilot sites, the Agency did not implement employee surveys because employees were not designated to assist users. Instead, paper surveys were available for users to complete if they wanted to provide the Agency feedback. This method did not provide any indication of the Self-Help PC users who chose not to complete a survey and therefore this process could not fully track usage.

Electronic usage logs are ideal to track Self-Help PC usage because they can ensure data are recorded for every transaction at attended and unattended sites. SSA planned to track use of Self-Help PCs with software used by the Agency to track visits to its Website. The Agency assigned Internet Protocol (IP) addresses to the Self Help PCs and believed it could track usage by identifying the Webpage visits from the IP addresses of Self-Help PCs. System reboots have prevented this method from accurately determining usage. To protect the exposure of personally identifiable information, the Self-Help PCs are configured to reboot after being idle and after each user logs off. Upon rebooting, the computers automatically access SSA’s Website, which is captured in the electronic logs. The Agency stated the electronic logs can capture visits to the Retirement Estimator and Apply for Disability WebPages; however, the logs cannot distinguish between Webpage visits for the other nine transactions available on Self-Help PCs and those initiated by system reboots. Agency staff stated they will try to address this issue.
An accurate account of Self-Help PC usage is necessary for senior management to determine whether Self-Help PCs maximize value to the Agency in accordance with the Clinger-Cohen Act of 1996. Our analysis showed the Agency has been unable to accurately track usage through its surveys or electronic usage logs.

Below we discuss the specific areas where we believe SSA can improve its data collection and evaluation strategy for the pilot.

TECHNIQUES TO IMPROVE DATA GATHERING FOR PILOT EVALUATION

Collection of User Feedback

The majority of feedback collected by the Agency comes from employee surveys at attended sites. As previously indicated, we acknowledge and support the importance of gathering employee feedback. However, we believe surveys should also be administered directly to users because the survey data submitted by employees may not fully reflect the overall user experience.

It should be noted that SSA was prohibited from questioning Self-Help PC pilot users directly without prior OMB approval. Obtaining OMB approval normally takes over 4 months, and because the Self-Help PC pilot was part of a Commissioner initiative, SSA chose to not seek approval to expedite implementation of the pilot. However, had SSA obtained OMB approval to directly question Self-Help PC users and prepared a customer survey, feedback could be more accurately obtained. For instance, a survey could automatically initiate on the Self-Help PC before a user logs off or at fixed time intervals. This would ensure a more even distribution of responses and offer Self-Help PC users at attended sites a chance to provide feedback from their own perspective. Furthermore, by requiring that the user respond or decline the survey, this method could also track Self-Help PC usage.

As an alternative survey approach, the Agency included a question about the Self-Help PC pilot on an OMB-approved survey to measure visitor satisfaction at the Agency’s busiest FOs. The survey was contracted by OQP in December 2008 and asked whether visitors attempted to use Self Help PCs, and if so, how they rated their experience on a scale of excellent to very poor. Although the Agency intended for the survey to include 12 of the 58 Self-Help PC pilot sites, Self-Help PCs were in place at only 4 of the offices as a result of pilot delays. Only a few responders indicated they had used the computers. As such, these customer satisfaction surveys did not provide enough data to support meaningful analysis of the pilot.

Use of VIP Data

The VIP system contains information about Self-Help PC pilot transactions for FO visitors who indicated they planned to use a Self-Help PC. While there may be scenarios where VIP does not reflect actual use of the Self-Help PC, VIP data can be used in conjunction with employee surveys to more accurately determine usage given that surveys alone do not provide an accurate account of Self-Help PC usage. Furthermore, VIP data includes users’ names and Social Security numbers, which would allow the Agency to examine user demographics to better analyze the results of the pilot.

SECURITY OF SELF-HELP PCs

SSA used several techniques to ensure Self-Help PCs were safe for the Agency and its customers. SSA standards state the computers should be physically secured, and print traffic should be routed to printers accessible only by SSA staff. During both our site visits, we found the computers were physically secured. We were unable to print at one site and could print to a secured printer at the second site. The Agency also developed consistent signs to inform customers that use of Self-Help PCs was limited to SSA business and the need to protect their personal information.

Self-Help PCs were isolated from the rest of SSA’s network. During our site visits, we were able to run a diagnostic report on the computers we tested and verify their IP addresses were consistent with the isolated ranges established by the Agency for the pilot. We also accessed an email and messenger application on both the Self-Help PCs we tested; however, Agency controls prevented us from sending or receiving emails or connecting to the messenger service. Agency staff stated these applications could not be removed from the Self-Help PCs without compromising the functionality of SSA’s eServices.

Furthermore, we verified the Agency was using software that, upon rebooting, restored the computer to the configuration established by the Office of Telecommunications and Systems Operations. This ensured system stability as well as eliminated threats like malicious code.

During our testing, all Websites were blocked except those belonging to Federal agencies. However, based on discussions with Agency staff and SSA’s Self-Help PC Website, there was a period of several days when systems activities inadvertently allowed Self Help PCs to access other Websites. Although this allowed Self-Help PCs to be used for purposes other than Agency business, it did not threaten SSA’s production environment.

The Agency designed a “Click When Finished” button to ensure that no personal information remained on the computer after a session. This button always remains on-screen, and clicking the button is supposed to reboot the computer. On our visit to an unattended site, however, the button did not function until after we manually rebooted the computer. Furthermore, at both the pilot sites we visited, users were discouraged by Agency staff from clicking the button because doing so did not complete the application and caused some users to lose the information they entered. Agency staff stated that if a new configuration is established for Self-Help PCs, they will address this issue. FO staff at the attended site also stated that the on-line applications automatically timed out after a certain amount of idle time, which also caused users to lose information.

FIELD OFFICE OBSERVATIONS

The FO staff we interviewed believed the Self-Help PC pilot was worthwhile and important to promote eServices. At the attended site, the staff noted the pilot saved time by allowing one employee to assist two visitors at once. They also explained that they were able to increase Self-Help PC usage through encouragement, by letting visitors know designated employees are available to assist with any problems, and by providing staff with incentives.

Employees at both the attended and unattended sites commented that when claimants were filing for disability benefits, it was beneficial for them to provide their own explanations rather than having an SSA employee describe their situation on the application. Based on their experience with the public and the Self-Help PC pilot, FO staff suggested the Agency allow Proof of Income letters to be printed immediately at the FO rather than the existing process of having the letters mailed within 2 weeks. They also suggested the Agency adjust data validation to prevent errors resulting from punctuation mistakes and abbreviations, as well as allow users to list multiple doctors under the same location on the disability application, generating only one letter of inquiry for medical records. We recognize the Self-Help PC pilot was designed to use SSA’s existing eServices, and implementing these suggestions goes beyond the scope of the pilot. However, we believe these suggestions support SSA’s objective to improve service for FO visitors and it is important to provide the Agency with the feedback we received from front-line staff.

Matters for Consideration
We applaud the Agency for its efforts to initiate measures to prevent FOs from being overwhelmed by large backlogs and an increasing number of retirement and disability claims. Such initiatives are critical for the Agency to find alternative methods to conduct its business with the public. As SSA moves forward with this initiative, we believe the information contained in this report will assist the Agency in its evaluation.

Given the Self-Help PC pilot is part of the Commissioner’s SMART service initiative, great care should be taken in creating the parameters for evaluating the effectiveness of the initiative in achieving the intended goals. The current assessment structure lacks substantial firsthand user feedback, relying heavily on FO employee surveys. While we believe it is important to gather and consider employee feedback, we also believe users should be more involved in providing feedback.

Based on the Agency’s current evaluation techniques, there is not a sufficient basis to determine the effectiveness of the Self-Help PC pilot. We believe the above information provides useful insight on areas that should be addressed as a part of the Agency evaluation process for the Self-Help PC pilot. For example, by systematically implementing an electronic survey for users to complete, the Agency could obtain a better assessment of usage and users’ feedback at both attended and unattended sites. We also support the Agency in further pursuing the use of electronic usage logs, which would ensure usage data is recorded for every Self-Help PC transaction. The VIP system may also be useful for tracking usage and could provide user demographics, allowing the Agency to better analyze the pilot. Improving evaluation techniques would assist the Agency in meeting the requirements of the Clinger-Cohen Act of 1996.

We encourage the Agency to explore alternatives to timeouts and the “Click When Finished” button that will facilitate the security of personally identifiable information, while minimizing user impact. We also encourage the Agency to consider the suggestions provided by the FO staff we interviewed. Allowing Proof of Income letters to be printed immediately at the FO would be more convenient for customers. Furthermore, enhancing eServices with better data validation and allowing multiple doctors to be listed under one location on the disability application would increase customer satisfaction.

Appendices
APPENDIX A – Acronyms
APPENDIX B – Scope and Methodology
APPENDIX C – OIG Contacts and Staff Acknowledgments

Appendix A
Acronyms

eServices Electronic Services
FO Field Office
IP Internet Protocol
OMB Office of Management and Budget
OQP Office of Quality Performance
OPSOS Office of Public Service and Operations Support
PC Personal Computer
PRA Paperwork Reduction Act of 1995
Pub. L. No. Public Law Number
SMART Space Modernization and Reception Transformation
SSA Social Security Administration
U.S.C. United States Code
VIP Visitor Intake Process

Appendix B
Scope and Methodology

The objective of this evaluation was to review the effectiveness of the Social Security Administration’s Self-Help Personal Computer (PC) pilot. Specifically, this evaluation focused on the impact of Self-Help PCs on electronic services’ (eServices) usage and field office (FO) performance and security concerns related to the public’s use of Agency computers.

To assess the impact of Self-Help PCs on eServices’ use and FO performance as well as the security concerns related to the public’s use of Agency computers, we

bull; reviewed the E Government Act of 2002 and the Clinger-Cohen Act of 1996;
bull; reviewed Agency documentation regarding the pilot;
bull; examined the results of surveys completed by FO staff;
bull; interviewed Headquarters personnel from the Offices of Public Service and Operations Support (OPSOS), Telecommunications and Systems Operations (OTSO), and Quality Performance; and
bull; visited an attended and unattended pilot site to observe the pilot’s implementation, interview staff, and test the security of Self-Help PCs.

We performed our review during June and July 2009 in Baltimore, Maryland, and Louisville, Kentucky. The entities reviewed were OPSOS and OTSO. We conducted our review in accordance with the President’s Council on Integrity and Efficiency’s Quality Standards for Inspections.

Appendix C
OIG Contacts and Staff Acknowledgments
OIG Contacts
Brian Karpe, Acting Director, Information Technology Audit Division
Mary Ellen Moyer, Audit Manager
Acknowledgments
In addition to those named above:
Phil Rogofsky, Audit Manager
Michael Zimmerman, Auditor
Cheryl Dailey, Auditor

For additional copies of this report, please visit our web site at www.socialsecurity.gov/oig or contact the Office of the Inspector General’s Public Affairs Staff Assistant at (410) 965-4518. Refer to Common Identification Number A-14-10-11001.

Overview of the Office of the Inspector General

The Office of the Inspector General (OIG) is comprised of an Office of Audit (OA), Office of Investigations (OI), Office of the Counsel to the Inspector General (OCIG), Office of External Relations (OER), and Office of Technology and Resource Management (OTRM). To ensure compliance with policies and procedures, internal controls, and professional standards, the OIG also has a comprehensive Professional Responsibility and Quality Assurance program.

Office of Audit
OA conducts financial and performance audits of the Social Security Administration’s (SSA) programs and operations and makes recommendations to ensure program objectives are achieved effectively and efficiently. Financial audits assess whether SSA’s financial statements fairly present SSA’s financial position, results of operations, and cash flow. Performance audits review the economy, efficiency, and effectiveness of SSA’s programs and operations. OA also conducts short-term management reviews and program evaluations on issues of concern to SSA, Congress, and the general public.

Office of Investigations
OI conducts investigations related to fraud, waste, abuse, and mismanagement in SSA programs and operations. This includes wrongdoing by applicants, beneficiaries, contractors, third parties, or SSA employees performing their official duties. This office serves as liaison to the Department of Justice on all matters relating to the investigation of SSA programs and personnel. OI also conducts joint investigations with other Federal, State, and local law enforcement agencies.

Office of the Counsel to the Inspector General
OCIG provides independent legal advice and counsel to the IG on various matters, including statutes, regulations, legislation, and policy directives. OCIG also advises the IG on investigative procedures and techniques, as well as on legal implications and conclusions to be drawn from audit and investigative material. Also, OCIG administers the Civil Monetary Penalty program.

Office of External Relations
OER manages OIG’s external and public affairs programs, and serves as the principal advisor on news releases and in providing information to the various news reporting services. OER develops OIG’s media and public information policies, directs OIG’s external and public affairs programs, and serves as the primary contact for those seeking information about OIG. OER prepares OIG publications, speeches, and presentations to internal and external organizations, and responds to Congressional correspondence.

Office of Technology and Resource Management
OTRM supports OIG by providing information management and systems security. OTRM also coordinates OIG’s budget, procurement, telecommunications, facilities, and human resources. In addition, OTRM is the focal point for OIG’s strategic planning function, and the development and monitoring of performance measures. In addition, OTRM receives and assigns for action allegations of criminal and administrative violations of Social Security laws, identifies fugitives receiving benefit payments from SSA, and provides technological assistance to investigations.