Audit Report - A-02-96-02204

Office of the Inspector General Evaluation Report: Test of Satisfaction Scales - A-02-96-02204 - 10/23/96

This report presents the results of our test of different overall satisfaction scales used in the questionnaire for the annual Customer Satisfaction Survey. This inspection was conducted at the request of the Executive Staff in support of the Agency`s implementation of the Government Performance and Results Act (GPRA).

We evaluated the effect of changing the highest rating, or "top box," on the overall satisfaction scale from the traditional rating of "very good" to "excellent." In their discussions on GPRA, the Executive Staff felt that excellent was a more appropriate customer service goal for a world-class organization. However, because of the importance of longitudinal analysis to ongoing monitoring of customer satisfaction, the Social Security Administration (SSA) needed to be assured that a change to "excellent" as the top box would not preclude longitudinal analysis.

We found no statistically significant difference in top box satisfaction ratings among each of the three samples and the top box rating for the last annual Customer Satisfaction Survey. The SSA`s customers rated "excellent" and "very good" similarly as the top box on an overall satisfaction scale. Because there are no statistically significant differences, we determined that the scale can be changed without jeopardizing the integrity of longitudinal analysis.

BACKGROUND

In 1984, the General Accounting Office (GAO) began annual surveys of how well SSA serves the public. These surveys were conducted in response to congressional concerns that the quality of SSA service would decline as SSA reduced staff and modernized services. In its January 1986 report, GAO recommended that SSA continue conducting customer satisfaction surveys. Each Commissioner since 1986 has requested that the Office of the Inspector General conduct these surveys as an independent evaluation of how well the Agency serves the public.

Data from these surveys are used as performance measures. More specifically, data on overall satisfaction, readability of mail, and staff courtesy are used as performance measures reported under GPRA. Other data from the surveys are used to assess Agency performance in meeting the standards in the customer service pledge. The SSA is required to report its progress in meeting these standards under Executive Order 12682, "Setting Customer Service Standards."

SCOPE OF THE EVALUATION

To conduct this evaluation, we replicated the methodological procedures used in the annual customer satisfaction surveys. More specifically, we selected 3 samples of 300 SSA customers using the same selection procedures and criteria as in the annual customer satisfaction survey. In these samples, the customers had a transaction on their automated file on either May 31 or June 3, 1996.

One of three post-card sized questionnaires (Appendix A) were mailed to each customer on June 22, 1996. The questionnaires were identical except for the scale to the last question on overall satisfaction. Each of the three samples got a different scale. One, referred to as V5, was the traditional five point scale with "very good" as the top box. The other two were four and five point scales beginning with "excellent" as the top box. They are referred to as E4 and E5. The full scales are:

• Ø V5--very good, good, fair, poor, very poor;
• Ø E4--excellent, good, average, poor; and
• Ø E5--excellent, good, average, poor, very poor.
• We sent postcard reminders on July 13, 1996 and second questionnaires on July 31, 1996 to customers who had not responded.

The premise for this inspection was there could be no statistically significant differences among top box scores if SSA is to consider changing the scale. We decided to test the concept of change by conducting control tests of three samples of SSA customers. Assuming at least a 70 percent response rate for each of the three samples, there would be no significant difference among the top box ratings if the ratings were within +/- 6 percent of each other and last years rating of 46 percent (at a 95 percent confidence level). Top box scores for recent customer satisfaction surveys ranged from 43 percent in 1993 to 46 percent this year.

Back to top

RESULTS OF THE EVALUATION

As can be seen in Table 1 (below), top box ratings for each of the three samples cluster in the mid-40 percent range as did the top box rating for the last Customer Satisfaction Survey (S12). Additionally, since all top box ratings fall within a 2 percentage point range, and well within the smallest confidence interval, it is easy to determine that there are no statistically significant differences among the top box ratings.

Table 1

Top Box Ratings

Respondents also had similar ratings on the traditional satisfaction scale. The traditional satisfaction rating, as developed by GAO in the original SSA Customer Satisfaction Survey, consisted of adding the ratings for the top two boxes i.e., "very good" and "good." The overall satisfaction rating for the last Customer Satisfaction Survey was 79 percent. For the three samples, the top two box ratings were 82 percent (V5), 82 percent (E4), and 84 percent (E5). Although consistently higher than the last survey`s rating, they are not statistically significantly different.

Respondents were equally consistent in their ratings of the quality of service on their last contacts with SSA. Customers were asked if they called a local office and/or the 800 number, and if they visited a field office (FO) all within the last year. Satisfaction ratings with the service they received during their last call or visit were similar. (See Table 2.)

Table 2

Satisfaction Ratings for Last Call or Visit

CONCLUSION AND RECOMMENDATION

We found that SSA`s customers rate "excellent" and "very good" similarly as the top box on an overall satisfaction scale. More specifically, we found no statistically significant differences among three test surveys and the last annual Customer Satisfaction Survey. Accordingly, we are confident that ongoing longitudinal analysis among annual customer satisfaction surveys will remain valid if the scale is changed. We recommend that SSA move ahead with its plan to shift to a top box satisfaction scale using "excellent" as the top box on the scale.

David C. Williams

Back to top

Appendix A

SURVEY QUESTIONNAIRE WITH WITH THREE SCALES

HOW WELL IS SOCIAL SECURITY SERVING YOU?

The Office of the Inspector General has been asked to look at how well the Social Security Administration is serving the public. To do this we need your help. Please complete this questionnaire and mail it back to us. We would like you to answer the questions even if you are not receiving benefits.

This survey is completely voluntary. Your answers will not affect your eligibility for Social Security or any other government benefits and will be kept confidential.

If you have questions about this survey, you can call us toll-free at 1-800-772-8246. If you have any questions about your Social Security claim or benefits, call 1-800-772-1213.

Thank you for helping us.

PAPERWORK REDUCTION ACT STATEMENT

The Paperwork Reduction Act of 1995 requires us to notify you that this information collection is in accordance with the clearance requirements of section 3507 of the Paperwork Reduction Act of 1995. We may not conduct or sponsor, and you are not required to respond to, a collection of information unless it displays a valid OMB control number.

TIME IT TAKES TO COMPLETE THIS FORM: We estimate that it will take you about 2 to 3 minutes to complete this survey. This includes the time it will take to read the instructions, gather any necessary facts and fill out the form. If you have comments or suggestions on this estimate,
write to the Social Security Administration, ATTN: Reports Clearance Officer, 1-A-21 Operations Building, Baltimore, MD 21235. Send only comments relating to our "time it takes" estimate to the office listed above. All requests for Social Security cards and other claims-related information should be sent to your local Social Security office, whose address is listed undeer Social Security Administration in the U.S. Government section our your telephone directory.

PLEASE ANSWER THE QUESTIONS, DETACH THIS SIDE, & MAIL THE OTHER HALF

Office of the Inspector General OMB No. 0960-0526

1. Did you visit a Social Security office in the last 12 months?

¨ No--GO TO QUESTION 2

¨ Yes--IF YES; Were you satisfied with SSA`s service during your last visit?

¨ No ¨ Yes

2. Did you call the Social Security 800 number in the last 12 months?

¨ No--GO TO QUESTION 3

¨ Yes -- IF YES; Were you satisfied with SSA`s service during your last call?

¨ No ¨ Yes

3. Did you call a local Social Security office in the last 12 months?

¨ No-- GO TO QUESTION 4

¨ Yes-- IF YES; Were you satisfied with SSA`s service during your last call?

¨ No ¨ Yes

4. Overall, how would you rate the service that Social Security has given you? (_only one)

V5-¨ Very Good ¨ Good ¨ Fair ¨ Poor ¨ Very Poor

E4-¨ Excellent ¨ Good ¨ Average ¨ Poor

E5-¨ Excellent ¨ Good ¨ Average ¨ Poor ¨ Very Poor