Office of the Inspector General
Larry G. Massanari
Acting Commissioner of Social Security

Inspector General

Performance Measure Review: Reliability of the Data Used to Measure Field Office Access (A-04-99-03008)

The attached final report presents the results of our audit. Our objective was to assess the reliability and integrity of the Social Security Administration's (SSA) performance data used to measure waiting times for field office access. The amount of time it takes to serve customers visiting field offices is a performance indicator designed to chart SSA’s progress in achieving world-class service. SSA desires to ensure a customer with an appointment waits no longer than 10 minutes before being served. SSA’s goal is to serve customers without an appointment within 30 minutes of their arrival.

Please comment within 60 days from the date of this memorandum on corrective action taken or planned on each recommendation. If you wish to discuss the final report, please call me or have your staff contact Steven L. Schaeffer, Assistant Inspector General for Audit, at (410) 965-9700.

James G. Huse, Jr.

PERFORMANCE MEASURE REVIEW:

By conducting independent and objective audits, investigations, and evaluations, we are agents of positive change striving for continuous improvement in the Social Security Administration's programs, operations, and management and in our own office.

Office of the Inspector General

Larry G. Massanari
Acting Commissioner of Social Security

Inspector General

Performance Measure Review: Reliability of the Data Used to Measure Field Office Access (A-04-99-03008)

The Government Performance and Results Act of 1993 (GPRA), codified in part at 31 U.S.C. §§ 1115-1119 (2001), requires the Social Security Administration (SSA) to develop performance indicators that assess the relevant service levels and outcomes of each program activity. GPRA also calls for a description of the means employed to verify and validate the measured values used to report on program performance. The objective of this audit was to assess the reliability and integrity of SSA's data used to measure the following Fiscal Year (FY) 1999 GPRA performance indicators.

Percent of public with an appointment waiting 10 minutes or less: 85 percent

These "waiting time" indicators and goals remain unchanged in the FY 2000 and 2001 Annual Performance Plans. The Office of Public Service and Operations Support (OPSOS), located in the Office of the Deputy Commissioner for Operations, is the sponsor of record and has overall responsibility for implementing the customer service waiting time performance measures.

The Agency’s Strategic Plan for FYs 1997 through 2002 contains a number of performance indicators related to its strategic goal of delivering customer-responsive, world-class service. SSA designed the two performance indicators identified above to chart the Agency’s progress in providing timely service to customers visiting SSA field offices. After analyzing the waiting time data submitted for FY 1999, SSA reported that 84.6 percent of its customers with an appointment waited 10 minutes or less to be served, and 71.6 percent of customers without an appointment waited 30 minutes or less to be served.

In September 1999, the Social Security Advisory Board (Board) questioned the validity of SSA’s reported waiting time results. Field office managers reported to Board members that waits of 2 to 4 hours were not uncommon in many offices, especially urban offices. Field office managers suggested the self-reported waiting time results used to calculate SSA’s percentages might not reflect the customer’s actual experience. While reports of long customer waiting times were anecdotal, the Board said it heard them too consistently from employees nationwide to ignore them. Board members themselves reported observing crowded field office waiting rooms that, in their opinion, affirmed the reports of long waiting times. The Board concluded SSA’s methods of measuring field office waiting times might rely too heavily on data that could easily be manipulated to make performance look better than it really is.

SSA could not provide documentation to support the data collected and used to report results for the FY 1999 waiting time performance measures. Therefore, we were unable to accomplish the portion of our audit objective dealing with the reliability of the waiting time data. However, we did evaluate SSA’s procedures for collecting and analyzing the waiting time data. Based on this review, we share the Board’s concern over the validity of the field office waiting time study (FOWTS) sampling process and the accuracy of the study’s results. Specifically, we are concerned because SSA did not adequately control the sampling process and allowed field offices to deviate from the designed sampling parameters. We are also concerned because faulty data processing procedures and programming logic could have adversely impacted both the integrity of the FOWTS data and the accuracy of the waiting time percentages. As a result, the published percentages of customers served with and without appointments in the targeted time frames of 10 or 30 minutes, respectively, may have been unreliable.

By the time of our audit, SSA had destroyed the source documents used to calculate FY 1999 waiting time percentages. As a result, we were unable to determine the reliability of the performance measure data SSA used to report on the waiting times of customers visiting field offices.

SSA used the FOWTS to determine the waiting times of customers visiting field offices. Each quarter, every SSA field office was scheduled to record waiting time data during a specified 1-hour period. The field office manually recorded each customer’s waiting time on Form SSA-3198, Field Office Waiting Time Study (Form). The Form was described as a "mark sense form" on which information was entered by filling in circles that corresponded with the recorded information. The Form captured waiting time data for up to three interviews, along with other information regarding the customer’s visit.

Field offices mailed the completed Forms to the Office of Information Management (OIM), where they were electronically scanned to produce a computerized data file. The data were copied to floppy disks and subsequently imported into the cumulative FOWTS data base. Once OIM processed the raw data, waiting time statistics and quarterly or annual reports were generated.

OIM’s practice was to retain the Forms for only one quarter and then to discard the source documentation. Without the original documents from FY 1999, we could not determine whether the Forms field offices sent were scanned and appropriately included in the raw data files. Therefore, we could not assess the reliability of the data SSA used to report on the FY 1999 waiting time performance measures.

While we could not assess the overall reliability of the FY 1999 data, we did evaluate the FOWTS data collection process and determined it was poorly controlled. We identified several field offices that did not report FOWTS data (non-reporters). Additionally, we detected field offices that (1) did not follow the sampling plan, (2) had varied collection procedures, and (3) modified their sampling periods. We also noted low numbers of customers served by appointment during scheduled sampling periods. By allowing field offices to deviate from the established sampling plan and vary their methods for data collection, the overall quality and completeness of the sampling process was adversely impacted. As a result, the integrity of the FOWTS data and the accuracy of the published waiting time percentages may be in question.

Our review of the FY 1999 FOWTS data files disclosed that many field offices were not included in the waiting time calculations. SSA staff did not ensure that field offices submitted waiting time data as requested. Staff members merely scanned the Forms they received. In FY 1999, 1,292 field offices were scheduled to conduct a FOWTS each quarter, for a total of 5,168 annual sample periods. However, the FY 1999 FOWTS data file contained only 3,086 (59.7 percent) of the scheduled field office tests.

In its Annual Performance Plan Status Report, SSA explained how the waiting time data were collected and calculated. The definition implies the waiting time percentages SSA reports are based on field offices’ 100-percent participation in the FOWTS. However, from our analysis of the data, we know this is not the case. To avoid misleading the user of the waiting time information, SSA needs to make the reader aware the percentages reported represent the results of only those offices that submitted data during the reporting period. The percentage of reporting field offices should also be disclosed.

There were inconsistencies in the number and types of offices represented that may impact comparability between quarters. The following chart presents the number and percentage of field offices that submitted FOWTS information by quarter. These percentages represent all the Form submissions scanned into the FY 1999 data file. However, as explained later, all of these submissions were not included in the waiting time calculations.

Table 1: Percentage of Offices with FOWTS Data Submissions

Table 2: Number of Quarters Field Offices Submitted FOWTS Data

Of the population of 1,292 field offices, only 399 (31 percent) submitted waiting time data for each of the 4 quarters, while 238 (18 percent) did not submit waiting time data for the entire year. The large volume of non-reporters could have impacted the validity of the results.

According to representatives in the Office of Research, Evaluation, and Statistics (ORES), the FOWTS sampling methodology was designed so a similar population of urban, suburban, and rural field offices would be sampled during any given period. Even though SSA did not ensure 100 percent compliance, we determined the mix of offices responding was representative of the entire field office population.

As shown in Table 3, from a population of 1,292 field offices, SSA classified 267 (20.6 percent) as urban, 515 (39.9 percent) as suburban, and 510 (39.5 percent) as rural. When examining the classification of offices reporting by quarter, the mix is similar in composition.

Table 3: Classification of Field Offices Reporting by Quarter

However, we noted 49 offices (15 urban, 14 suburban, and 20 rural) that indicated they did not receive visitors during the sampling period. Seven of these offices reported they did not have visitors for two quarters. While some of the smaller rural offices may not have received any visitors during the sample periods, this inactivity seems unlikely for larger facilities in densely populated metropolitan areas. Therefore, although they submitted reports, some of these offices might actually have been non-reporters.

Another concern we had with this sample was that the Agency appeared to give no weight to the number of employees field offices had when it assigned the offices to a particular classification. Because urban offices are typically located in densely populated areas, a logical assumption would be that more staff is needed to serve customers’ needs. However, we found 2 "urban" offices with only 12 and 13 staff members, while other "urban" offices had as many as 66 staff members. In addition, SSA classified some offices as "suburban," even though they had 54 staff members. While some staffing differences may be explained, others may indicate an office that is misclassified.

OPSOS analyzed the FOWTS submissions for the first three quarters of FY 2000. We reviewed this analysis, and the FY 2000 results closely parallel the FY 1999 data. Given the number of non-reporters identified for FY 1999 and the first three quarters of FY 2000, SSA should do more to ensure that all field offices are participating in the process. As the sampling plan is designed, without full participation, the results may be distorted.

After reviewing the field office sample schedule and the FOWTS data files, we determined that some field offices failed to collect waiting time data during their assigned study periods. At the beginning of each fiscal year, ORES generates the FOWTS sample days and times for each SSA field office. ORES systematically selects and randomly assigns the sample days and times to each field office to capture waiting times experienced in different types of offices at different times of the day and month. When we compared scheduled times for the 1,292 field offices to the FOWTS data files, we noted differences in actual and assigned study times for 261 of the offices. The 261 field offices conducted a total of 298 studies as follows:

• study was taken on the assigned day, but the wrong time (118),
• study was taken on the wrong day at the assigned time (145),
• study was taken on the wrong day at the wrong time (27), and
• study was taken over multiple days and times (8).

For 298 studies to be conducted by 261 offices outside the assigned study period, some of the offices must have missed their assigned day and/or time on multiple occasions. The Management Information Manual (MIM) provides instructions for making up a missed sample period. The MIM states if a field office misses its scheduled sample period, it is to notify its regional management information staff and sample at the same time during the next week. The FY 1999 FOWTS data files showed 124 offices failed to conduct the study on the scheduled day, and 43 conducted their alternate studies on the prescribed day of the following week.

The MIM states the validity of the waiting time study depends on each field office adhering to its assigned sample schedule. By not adhering to the sample schedule, field offices remove the randomness from the sample. Field offices cannot conduct the study when convenient; they must adhere to the set schedule to ensure the integrity of the FOWTS.

Field offices used different methods of collecting waiting time data, which may have biased results. The MIM allows field office managers to customize the "generic" sampling instructions to fit local operating practices. However, this latitude has resulted in a variety of interpretations on how the field offices should collect the data.

In 1995, we reported similar inconsistencies in FOWTS data collection practices. To determine whether the inconsistencies still existed, we asked 64 field offices to document the actual FOWTS procedures followed at their offices. Our study showed that, while most of the field offices began measuring a customer’s waiting time when they walked through the front door, five offices (8 percent) did not start measuring waiting time until the customer reached the reception counter. Because of the large volume of customers at some field offices and the lack of staff, we were told it is sometimes difficult to capture data when an individual enters the office. These customers may have an initial wait before they are served at the reception counter that is not factored into the data collected.

We also identified significant variations in how field offices processed the Forms. The 64 field offices we sampled identified the following 5 methods.

1. In 24 offices, the customer received the Form from the greeter (guard or receptionist) to give to the first interviewer. If more than one interview was needed, the first interviewer passed the Form to the next interviewer.
2. In 15 offices, the customer received the Form from the greeter, and the customer carried the Form from interview to interview.
3. In 14 offices, the customer was not given the Form. The interviewers maintained and recorded the waiting time information.
4. For five offices, the field office assigned one individual to observe each customer and capture the waiting times of all visitors that arrived during the study period.
5. The remaining six offices did not use the original Form to capture waiting time data. Instead, they used an alternate form such as a photocopy or locally prepared log. At the end of the study, the data from this alternate form were transcribed, usually by the office manager, onto the original Form.

These different collection methods and practices of recording start times could have had an impact on the quality and accuracy of the data collected. Certain procedures could easily have been manipulated. In fact, field office representatives acknowledged abuses do occur for fear of being faulted if customer service results are unsatisfactory.

Of the 48,661 field office visits included in the FY 1999 FOWTS, only 3,263 (6.7 percent) were by appointment. Of the 1,054 field offices that reported for at least 1 quarter, 338 (32 percent) claimed they did not have any customers with scheduled appointments during 1 or more sampling periods. When appointments are encouraged as the preferred way of conducting business, it is difficult to rationalize how 32 percent of the field offices reporting would not have any appointments scheduled during the planned sampling periods.

SSA’s FY 1999 Accountability Report noted that with more stewardship responsibilities and increases in redeterminations, continuing disability reviews and alerts, more appointments were made in 1999 than in the past. While we were told that FY 1999 appointment data were not available, we determined from national data maintained by the Philadelphia Region Information Management and Analysis Team that there were over 4.1 million appointments scheduled and kept in the last 10 months of FY 2000—an average of 2,467 appointments per hour. The FY 1999 FOWTS data reflect an average of 0.95 appointments per hour. Field offices are notified of their four quarterly sampling periods at the beginning of each year. Therefore, we believe managers may be limiting appointments during these periods to ensure appointment times are met and to free up staff during the sampling period to more effectively provide service to customers arriving without an appointment.

We also noted from our sample of 64 offices that some offices include telephone appointments along with office appointments when reporting waiting times for customers with appointments. This practice appears contrary to the intent of the performance measure and may have distorted reported results.

Deficiencies in processing field office data as well as in the FOWTS program could have adversely impacted both the integrity of the FOWTS data and the accuracy of the waiting time percentages SSA published. Therefore, we were concerned to note SSA did not maintain an updated classification list for field offices when designing the annual field office sample. We were also surprised to find that SSA did not control data submissions and used the FOWTS program to override bad or omitted data by arbitrarily assigning values or discriminating against certain data. These conditions could have impacted the validity of the resulting data on which SSA based its annual waiting time percentages.

SSA used outdated field office classifications when it established field office sample days and times for the annual FOWTS. Having a representative mix of field offices by classification is statistically important for a valid FOWTS sample. Because ORES used outdated classifications, the field office sample selection may not have achieved the representative mix desired for FOWTS.

ORES had the responsibility of assigning FOWTS sample days and times to individual offices. To do this, ORES grouped field offices by their urban, suburban, or rural classifications based on each field office’s service area population density. ORES grouped the offices in this manner to ensure it selected an equal percentage of offices within each classification to test during the same sample period.

Because the sampling mix was important to the FOWTS process, we tested the field office classifications ORES used. We requested a 5-percent random sample of field offices to identify their office classification as urban, suburban, or rural. Of the 64 field offices sampled, 25 offices (39 percent) reported a classification different from the list ORES maintained. Twenty of the offices considered their classifications to be higher, while the remaining five offices reported themselves in a lower classification. These responses were based on the responders’ opinions of their service areas, and they illustrate the changes that may occur with population shifts over a 10-year period.

ORES classified the FOWTS sample using a listing initially prepared with 1990 census data extracted from the Profiling System Database (PSDB). We were told the ORES classification listing was not periodically updated to account for changes that could occur in an office’s service area density due to population shifts. ORES was unaware that population density data contained in the PSDB were updated annually. The only changes ORES routinely made to its listing were to add new field offices using the service area’s population density data at the time of the office’s opening and to remove closed field offices from the list. By not maintaining current field office information in its classification listing, the ORES field office study mix did not reflect the actual field office population.

OPSOS should have ensured that an independent analysis was performed to determine whether the FOWTS data the field offices submitted were complete and logical. OIM merely scanned the data submitted on the individual Forms without any further analysis. OIM only intervened when a Form did not scan. In those instances, the employee scanning the document might have corrected the data or, in the worst situations, returned the Forms to the field office for correction. However, the faulty Forms were not tracked to ensure they were corrected and returned. In fact, SSA did not maintain any records on the number of Forms it received from field offices and reconcile that information to the number of Forms scanned to ensure all the available data were counted.

Additionally, OIM had no way of determining the total number of customers served during the sampling periods. The number of visitors during each of the four study periods varied within each field office and among field offices. Accordingly, SSA relied on each office to adhere to the study design and submit Forms for each visitor. OIM had no independent means by which to verify the totality of the field office submissions.

From our 5-percent sample of 64 field offices, only 5 reported having a tracking system to ensure the Forms distributed to each visitor during the sample period were returned to the designated staff member. These offices reported either comparing the number of Forms collected to the number of visitors that signed in during the study hour or numbering the Forms before giving them to each visitor and accounting for them at the end of the study. These results indicate that most field offices did not keep customer logs to ensure all customers who entered their offices during the sampling period were actually counted and included in the FOWTS data submitted to OIM.

While SSA did not maintain the original Forms, we did analyze FOWTS data records containing information on each of the Forms scanned during FY 1999. Our intent was to recreate the FOWTS calculation to test the accuracy of the reported percentages. In reviewing the data, we noted deficiencies in certain records that could have impacted the calculation’s accuracy. Of the 48,661 records included in the FOWTS file, we eliminated 11,973 records from our calculation because

• 8,234 were outside the sampling period of October 1, 1998 through September 30, 1999,
• 38 had invalid office codes or no office codes,
• 68 indicated no visitors were received during the sampling period,
• 785 indicated visitors left without service, and
• 2,848 contained some type of omission or error.

The FOWTS program substituted artificial data to force records into the waiting time calculation. For FY 1999, 775 records did not contain the time the customer entered the field office, and 985 records did not show a start time for the first interview. However, the field offices completed additional fields that indicated the customer did not leave without service. To force these records into the waiting time calculation, the FOWTS program automatically performed processing steps whereby the missing elements of one key data field were filled using elements of another key data field. For example, for the 775 records with a blank "Time Entered Office" field, the program inserted the start time of the first interview. This practice was problematic because it resulted in understated waiting times for those customers who may have waited before they were seen for the first interview. Similarly, for the 985 records with a blank "Start Time 1^st Interview" field, the program used the time the customer entered the office as the start time. This process of forcing data reduced the overall waiting times for these customers.

We also detected records with unreadable or illogical ("bad") data in key fields, which were force processed. Specifically, we noted bad data in 358 "Time Entered Office" fields, 126 "Appointment Time" fields, and 196 "Start Time 1^st Interview" fields. Further, 408 records contained negative waiting times to the first interview. Negative waiting times occurred when the recorded time of the customer’s first interview was before the field office documented the customer actually arrived. The illogical ordering of time can result in a portion of the customer’s waiting time being excluded from the waiting time calculation. Stars may have represented some of the unrecognizable data contained in these key fields. The FOWTS program made assumptions when processing these data. Depending on where the star was in the data field, the program either substituted a value for the star or eliminated it from the data field. The remaining value became the customer’s newly recorded time.

The FOWTS program used the key data fields on each record to determine whether the customer had an appointment and to identify the associated waiting times. Although the Form captured additional data, the following fields were key in the FOWTS calculation of waiting time percentages.

• Time Entered Office
• Appointment Time (if applicable)
• Start and Stop Time First Interview
• Subject of First Interview
• Start Time Second Interview
• Subject of Second Interview

As discussed above, the data in these key fields were often questionable or corrupt. However, OIM did not perform a quality review of the data to enhance their accuracy. Since the Form does not contain identifying information (customer name and Social Security number) and the field offices sampled did not maintain some type of visitor’s log, there was no way of identifying a customer or group of customers served during the sample period. Without this information, SSA had no way of independently validating the data submitted or possibly correcting omissions/bad data. Instead, the FOWTS program automatically recreated data fields or eliminated records from the waiting time calculations. As a result, SSA relied on the processing of partial data and recreated data to determine whether they met the Agency’s waiting time goals.

Because SSA no longer maintained the original Forms, we could not compare the data on the forms to the FOWTS data files. Accordingly, we could not determine whether the bad data could be resolved or contact the affected customers. Nevertheless, from the number of data errors, omissions, and limits noted during our review, we must question the overall reliability of the data as an accurate measure of customer waiting times. Force processing records with problem data into the waiting time calculations by making assumptions about a customer’s waiting time may have resulted in times being grossly understated or exaggerated. When asked about the force processing steps, SSA staff could not explain why the FOWTS program was designed to perform in this manner. They also could not explain why the waiting time percentages include certain groups of records but not others or why we could not replicate the results outside the FOWTS program.

In addition to forcing data into the calculation, we noted the FOWTS program discriminated against select data. In some cases, the length of the first interview was excluded from the total waiting time. For example, if a customer did not have an appointment and the subject of the first interview was other than to schedule an appointment or obtain a Social Security number, the second interview was considered the in-depth interview. However, when calculating the total waiting time until the second interview (in-depth interview), the FOWTS program did not capture the elapsed time during the first interview. Thus, the waiting time calculated did not reflect the actual length of time the customer had to wait before being seen for their in-depth interview.

In addition, the FOWTS program eliminated groups of customers from the calculation of the overall percentage of customers without appointments that were served within 30 minutes. The waiting times for most customers without an appointment and whose business was completed with their first contact (no second interview), was only included in the calculation if the customer’s business involved a title II and/or title XVI claim and the interview lasted 15 minutes or longer. Also, the waiting times for those customers without an appointment whose first interview was a Social Security number matter were eliminated from the calculations to determine the overall percentage of customers without appointments who were served within 30 minutes. For FY 1999, 29,778 records represented customers with no appointment and only 1 interview. Of these records, 13,358 customers discussed a Social Security number matter. Therefore, SSA eliminated 44.8 percent of the records from the waiting time calculation.

On the other hand, SSA counted some waiting times twice in the FOWTS calculation. This duplication occurred when a customer did not have an appointment and the subject of the first and second interview was a title II or title XVI claim. In this situation, SSA added the waiting time between when the customer entered the office and the start of their first interview to the waiting time between when the customer entered the office and the start of the second interview.

We attempted to replicate the waiting time calculation using the FY 1999 data but, we were unsuccessful. Because we could not get a clear and definitive description of the original FOWTS programming logic, we used a logical presentation of the data eliminating the bad and incomplete data previously discussed. However, we could not successfully duplicate SSA’s reported percentages outside the FOWTS program. We discussed our logic with the FOWTS programmer, but the differences could not be resolved.

Lacking a way to identify the correct information that should be recorded, SSA cannot be certain the times artificially created and the times substituted by the FOWTS program did not distort the waiting times for those records. The individual who originally developed the FOWTS program had retired, and no documentation could be found explaining how the waiting time calculations and the program logic evolved. Without knowing the reasoning behind each step and each calculation, SSA is unable to ensure the adequacy of the program used to generate the waiting time percentages and cannot rely on the results.

In response to this audit and its own concerns with the manual FOWTS, SSA has developed an Intranet-based application that automates the front-end collection of waiting time data. The Automated Waiting Time Study was initiated in March 2001, piloted in the Atlanta region from May through July 2001, and released for Nation-wide use on
July 30, 2001. This application eliminates the need for the Forms to be manually prepared, mailed to Headquarters, and scanned to create a data file.

While we have not audited the Automated Waiting Time Study, we have discussed its functionality with the contractor and regional development team. As designed, the new application processes data nightly rather than quarterly providing for the quick identification of non-reporters. The application also eliminates the possibility of records being processed with empty fields, negative waiting times, hours outside of normal working hours, and "stars" due to scanning problems. With the practice of purging incomplete entries from the transmitted data, the new application should eliminate the past practice of force processing data. However, the program logic that discriminated data will continue to affect the calculated waiting time results. While SSA’s automation of waiting time data submissions is a step forward, many of the conditions we presented in the report will remain. Examples follow.

• SSA will continue to use the same program logic to process the reported waiting time statistics.

• Customer-identifying information will not be captured.

• Field offices can bypass warning messages and complete the FOWTS on an unscheduled date and time.

• One-hour sampling periods will not be enforced.

• Instead of capturing system times, interview times will be manually input.
• The Automated Waiting Time Study Coordinator, often the office manager, can make "corrections" or changes to the waiting time records before submitting them for automated processing.

• Incomplete entries are purged overnight, eliminating them from the FOWTS calculation, but no audit trail of these records is maintained.

We understand the Automated Waiting Time Study was only intended as an interim solution for the data integrity problems previously noted. However, we remain concerned about the limitations listed above and the resulting reliability of the data used to support the Agency’s performance measures for field office access.

In the future, SSA plans to develop a Visitor Intake Process (VIP) to capture information on customer waiting times for all customers. OPSOS indicates that this advanced system will capture customer identifiers and system waiting time data. The data collected by VIP is to be available in "real time" and valid down to the individual field office level. SSA plans to pilot VIP in FY 2002, dependent on annual budget and personnel constraints.

We identified significant problems with the reliability of the data collection, recording, and processing of FY 1999 FOWTS data.

• SSA destroyed source documents used to calculate the FY 1999 waiting time percentages.

• Of a required 100-percent sample, only 59.7 percent of scheduled office tests were conducted. In addition, the percentage of participating field offices varied by quarter. A review by OPSOS showed that FY 2000 field office participation was similar.

• Only 31 percent of field offices submitted waiting time data for all four quarters during FY 1999.

• A number of urban and suburban field offices reported no visitors, some for more than one quarter.

• For those offices that participated, many did not conduct the study at the assigned sample time or the alternative sample time outlined in the MIM.

• A survey of field offices revealed that data collection procedures varied, possibly leading to inconsistent data collection.

• The assigned sample times were not consistent in terms of length—11 percent of the field office tests were for periods more or less than 1 hour.

• Outdated field office classification listings were used to design the waiting time sample.

• SSA neither determined whether field offices completed the FOWTS nor reconciled the number of Forms scanned with the number of Forms submitted or visitors seen.
  
  
• Data files included a large number of records with omissions and faulty data.

• The FOWTS program was designed to make assumptions about omissions and faulty data, forcing data into the waiting time calculations. It also eliminated a high number of records from the waiting time statistics for unexplained reasons.

Considering the overall problems we identified in the FOWTS sampling process and the FOWTS program, we question the validity of the resulting FY 1999 waiting time data and SSA’s ability to accurately compute customers’ waiting times. SSA’s recent actions have improved the process, and future plans in implementing the VIP will improve the data collection process even more. However, while waiting for the VIP to be developed and implemented, SSA should take additional steps to improve the integrity of the current FOWTS process and the reported results. Therefore, we recommend that SSA:

1. Retain the data submissions, including information on all purged records, to allow data verification for at least 2 fiscal years.
2. Disclose in waiting time reports that the percentages are not based on a 100-percent response but only represent those offices that submitted data during the reporting period.
3. Redefine the published measures so the user fully understands what will be measured and reported.
4. Oversee the FOWTS sampling process to ensure that field offices:

• Participate in quarterly sampling, as directed.
• Sample during the assigned times or according to the approved alternate schedule.
• Follow consistent data collection methodologies.

1. Document the sampling methodology and adjust sampling times so they are consistent with the office opening and closing times.
2. Obtain the PSDB revisions and update field office classifications before establishing sample days and times for the annual FOWTS.
3. Establish controls over data submissions and follow-up on questionable entries. SSA should routinely perform data analysis to ensure its integrity. For example, non-reporters should be identified and pursued. Data submissions that appear illogical should be clarified. Field offices reporting no appointments during a sampling period should be questioned. SSA should clearly indicate through its interaction with field offices that the quality of the FOWTS data submissions has importance.
4. Evaluate and document the FOWTS programming logic. SSA should ensure FOWTS-generated statistics represent the customer workloads it intends to measure.
5. Require that the customer’s Social Security number and name be included with the data submissions and records to provide an opportunity to test data reliability and follow-up on questionable data.

SSA expressed concern that some of the criticisms contained in the draft report appeared to be based on speculation and anecdotal information. Nevertheless, SSA recognized the need to improve the process for collecting waiting time study information and agreed with seven of our nine recommendations.

SSA believes its procedures already address this Recommendation 8. SSA indicated it regularly evaluates its reporting needs and revises FOWTS as the needs change. SSA also disagreed with Recommendation 9 to add customer-identifying information to the automated FOWTS process currently in use. SSA stated that capturing the SSN for each individual would be too resource intensive and would not be justified. However, SSA noted future plans to include the customer name and SSN in its planned VIP pilot. See Appendix C for the full text of SSA’s comments.

We are pleased SSA agreed with most of our recommendations. We believe field office access is of utmost importance to the public’s perception of world-class service, and we are encouraged that SSA plans to focus on these highly visible measures and the data supporting them.

Regarding SSA’s concern that some of the criticisms expressed in the draft report appeared to be based on speculation and anecdotal information, we drew our information from the best available sources. Our conclusion that managers may be limiting scheduled appointments was based on statements previously made by the Social Security Advisory Board, regional management, and field office personnel. For example, while reports of long customer waiting times were anecdotal, the Advisory Board reported that it heard those reports too consistently from managers and employees Nationwide to ignore them. Board members themselves reported observing crowded field office waiting rooms, as we did, and, in the Board’s opinion, this affirmed reports of long waiting times. The Board concluded SSA’s methods of measuring field office waiting times might rely too heavily on data that could easily be manipulated to make performance look better than it really is. Based on conversations with individuals intimately involved in the FOWTS, our own analysis, and a general lack of controls over FOWTS data collection, we concurred with the Board’s assessment that manipulation was possible.

The intent of our recommendation to evaluate and document the FOWTS programming logic was to ensure FOWTS-generated statistics represented the customer workloads SSA intended to measure. To understand exactly what FOWTS data were being captured and how they were being used, we attempted to obtain software documentation explaining the FOWTS programming logic. As we noted in our report, OIM provided us the FOWTS program and explained how several individual sections of the program code were working. However, OIM, OPSOS and OWA were unable to provide written documentation to explain how the FOWTS program evolved into its current state and why certain programming steps were taking place. We continue to be concerned with the lack of program documentation because the automated FOWTS uses the same programming code as the old, manual FOWTS process, and several of the deficiencies we reported will persist. Therefore, we still believe SSA should analyze and document the FOWTS programming logic to assess its validity and ensure that appropriate data are being measured and included in the waiting time statistics reported by SSA.

Our recommendation to require that SSNs and names be included with data submissions is needed to allow for internal and independent testing of data reliability. We are disappointed SSA does not plan to revise the automated FOWTS to include customer-identifying information. We continue to believe this information is needed to ensure the integrity of the process by allowing for data verification. We acknowledge the automated FOWTS is an interim solution; however, the individuals responsible for developing the automated FOWTS told us including the customer’s SSN and name would not be difficult. These data fields were not included initially because they were not part of the original system specifications. We believe that capturing customer-identifying information is justifiable and should be required as soon as possible. Without this information SSA will be unable to verify and validate the waiting time performance goals as required under the Government Performance and Results Act.

James G. Huse, Jr.

The objective of this review was to assess the reliability and integrity of the Social Security Administration’s (SSA) performance data used to measure waiting times for field office access. Because SSA could not provide documentation to support the data collected and used to report results for the Fiscal Year (FY) 1999 waiting time performance measures, we were unable to accomplish the portion of our audit objective dealing with the reliability of the waiting time data. However, we did evaluate SSA’s procedures for collecting and analyzing the waiting time data. Without formal documentation, we relied primarily on discussions with officials from the Offices of Public Service and Operations Support (OPSOS), Information Management (OIM), and Research, Evaluation and Statistics (ORES) to detail the Field Office Waiting Time Study (FOWTS) sampling and data collection processes.

In performing our audit, we:

• Interviewed SSA personnel involved in the formulation of the performance indicator as well as those involved in the collection, recording, and analysis of the data.

• Reviewed the Management Information Manual (MIM) related to the FOWTS.

• Reviewed our prior audit work related to FOWTS.

• Reviewed the Advisory Board report and internal SSA reports and documentation related to field office waiting times.

• Observed the scanning process used to generate the electronic data files for the FOWTS program.

• Obtained FY 1999 data files used for generating the FOWTS waiting time statistics.

• Analyzed FY 1999 FOWTS data files to determine the percentages of non-responding field offices and the extent of faulty data used in calculating waiting times.

• Reviewed the analysis completed by OPSOS to determine whether FY 2000 rates of response were similar to FY 1999.

• Surveyed a random sample of five-percent of field offices to gather information about FOWTS data collection practices.

• Interviewed SSA personnel regarding alternative measuring systems that could capture waiting time data in field offices.

• Interviewed SSA personnel regarding the development of the FOWTS program and how the program calculated the waiting time statistics.

• Obtained data from ORES regarding the methodology SSA used to assign sampling times to the field offices. We reviewed the data to determine whether sampling times were uniform and evenly distributed over the year.

• Attempted to replicate the waiting time percentages outside the FOWTS program.

• Gathered limited background information and interviewed SSA staff on SSA’s various efforts to automate the collection of waiting time data.

We performed our audit in Baltimore, Maryland, and Atlanta, Georgia, from
November 1999 to May 2001. We conducted our audit in accordance with generally accepted government auditing standards.

Acronyms

We appreciate the effort of the OIG in conducting this review, but are concerned that some of the criticisms expressed in the draft audit appear to be based on speculation and anecdotal information. For example, OIG relies on the number of employees assigned to an office to conclude that the office has been inappropriately categorized, but no direct correlation has been identified between staffing, the number of office visits, and an office's classification. The field office (FO) classifications used for the report are based on the population density of a FO service area. Also, OIG’s conclusion that managers "may be limiting" scheduled appointments to improve office waiting time reports appears to be speculative.

The Agency recognized a need to improve the process for collecting waiting time study information before this review began. Since 1994, we have been involved in testing various automated queuing and management information systems that have shown some potential for obtaining better information on customer access and waiting times. We have also taken steps to identify non-responders and offices that submitted reports with edits and/or rejects. The Agency rolled out a WEB-based waiting time study application this year that provides consistent and immediate feedback so that offices can better focus on the importance of doing the studies timely and accurately. In addition, the Agency plans to pilot a new automated Visitor Intake Process (VIP) in fiscal year 2002. Implementation will be dependent on the success of the pilot and the availability of resources.

Our comments on the report recommendations follow below.

Retain the data submissions, including information on all purged records, to allow data verification for at least two fiscal years (FY).

We will retain the data submissions made by our FOs through our WEB-based waiting time study application, allowing data verifications for two FYs. The data are designed to be valid only at the national level, although, as noted, we eventually plan to capture individual customer and detailed office level data in the development of the VIP.

Disclose in waiting time reports that the percentages are not based on a 100-percent response but only represent those offices that submitted data during the reporting period.

We agree and will ensure that any publications using the data from the waiting time studies reflect that the information is based on data derived from sampling.

Redefine the published measures so the user fully understands what will be measured and reported.

We agree. As the VIP system is tested, we will reevaluate the published measures to ensure that the data needed to fully assess visitor access to the services in our field offices are captured.

Oversee the Field Office Waiting Time Study (FOWTS) sampling process to ensure that field offices participate in quarterly sampling, as directed; sample during the assigned times or according to the approved alternate schedule; and follow consistent data collection methodologies.

We agree. The absence of current and accurate management information has been a concern to the Agency. We believe that the new WEB-based application, while perhaps not ideal, is a significant improvement. The management information this application provides will be consistent and immediate, which will help maintain a focus on conducting the studies timely and accurately.

Document the sampling methodology and adjust sampling times so they are consistent with the office opening and closing times.

We agree and will ensure that future sampling times consider office opening and closing times.

Obtain the Profiling Systems Database (PSDB) revisions and update FO classifications before establishing sample days and times for the annual FOWTS.

We agree. Updated information from the PSDB will be used to determine office classifications before establishing the sample days and times for the next FOWTS.

Establish controls over data submissions and follow-up on questionable entries. SSA should routinely perform data analysis to ensure its integrity. For example, nonreporters should be identified and pursued. Data submissions that appear illogical should be clarified. FOs reporting no appointments during a sampling period should be questioned. SSA should clearly indicate through its interaction with FOs that the quality of the FOWTS data submissions has importance.

We agree. The WEB-based application that is now being used has eliminated the need for completing the "mark sense" form (SSA-3198) and has automated the front end process of the FOWTS. Reports of nonresponders are available immediately and edits/rejects are processed online, virtually eliminating the errors associated with the manual FOWTS process.

Evaluate and document the FOWTS programming logic. SSA should ensure FOWTS-generated statistics represent the customer workloads it intends to measure.

Our current procedures already address this recommendation. The FOWTS was developed to capture information required for specific reporting needs. We regularly evaluate those needs and make revisions in the FOWTS as warranted.

Require that the customer’s Social Security number (SSN) and name be included with the data submissions and records to provide an opportunity to test data reliability and follow-up on questionable data.

We disagree. The current waiting time study is not meant to be customer specific and generates information needed to assess accessibility only at the national level. Capturing the SSN for such a study would be too resource intensive and would not be justified considering that the Agency is developing the VIP process. When the VIP process is implemented, it will provide information allowing an auditor to track and verify individual customer information and office level data.

OIG Contacts and Staff Acknowledgments

In addition to those named above: