The Social Security Administration (SSA) established
the Office of Program and Integrity Reviews (OPIR) to perform quality
assessment reviews of SSA programs and operations. OPIR is headed
by the Associate Commissioner for Program and Integrity Reviews who
reports directly to the Deputy Commissioner for Finance, Assessment,
and Management and Chief Financial Officer. OPIR has 4 headquarters
offices and 10 regional offices. These offices are responsible
for conducting quality assurance reviews and special studies of SSA
programs. These special studies and reviews generally contain recommendations
for corrective changes in programs, policies, procedures, or legislation
aimed at quality and productivity improvement and/or program simplification.
Between October 1, 1992 and September 30, 1994, OPIR had approximately
1,500 staff and an operating budget of $90 million. We estimate
that 25 percent of OPIR’s resources were dedicated to special
studies with the remainder primarily dedicated to quality review
initiatives.
The objectives of our audit were to determine whether
OPIR`s special studies were: (1) used by management to improve
SSA programs; (2) cost-effective and efficient; and (3) performed
in accordance with established standards.
SUMMARY OF FINDINGS
We were not able to meet our intended objectives because
OPIR has not implemented appropriate management controls necessary
for assuring accountability for special studies. OPIR has not developed
a system for tracking special studies and proposed recommendations,
established standards for conducting its special studies, or established
policies for assuring the statistical validity of its studies. As
a result, we were unable to ascertain either the cost-effectiveness
or efficiency of OPIR’s special studies or their usefulness
to Agency managers. Therefore, we are reporting on the conditions
that we found relating to weaknesses in management controls and the
need for OPIR to establish management controls, special study standards,
and statistical policies.
First, our attempt to determine the cost-effectiveness
and efficiency of OPIR’s special studies, required us to identify
an inventory of all special studies conducted during our audit period
of Fiscal Years (FYs) 1993 and 1994. We found that OPIR has not implemented
effective management controls that maintain an inventory of its special
studies and the resulting recommendations and their current implementation
status. The Office of Management and Budget (OMB) issued Circular
A-123 to implement the requirements set forth by the Federal Managers’ Financial
Integrity Act of 1982 (FMFIA), Public Law (P.L.) 97-255. The Circular
requires that management controls be implemented to ensure that managers
are accountable for their operations and for the proper stewardship
of Federal resources. Management has not committed to these principles
and as a result, we were unable to evaluate OPIR’s special study
performance or the value of its studies to the Agency.
Second, we extended our procedures to determine whether
standards were employed over the conduct of individual special studies.
We originally intended to review a statistical sample of special
studies that were conducted during FYs 1993 and 1994. However, due
to the lack of management controls, we were unable to identify the
universe of OPIR’s special studies. As an alternative, we selected
a judgmental sample of 10 special studies and found that the standards
for performing OPIR’s special studies are informally established
by the staff who conduct the review and are not documented to assure
compliance. Management has not implemented professionally accepted
or recognized study standards or established policies or procedures
for conducting OPIR’s special studies.
Third, in reviewing OPIR’s special studies, we
found that two of the studies selected for review did not comply
with minimal OMB statistical sampling requirements. These two studies
involved the collection of information from the public to evaluate
SSA’s effectiveness in making referrals to other government
services. During the conduct of these studies, OPIR used inappropriate
sample selection and data collection methods which resulted in a
response rate well below OMB’s minimum response rate requirements.
As a result, these two studies did not comply with OMB’s minimal
statistical standards that are required under section 1320.9 of Title
5 of the Code of Federal Regulations (CFR). These deficiencies occurred
because OPIR has not implemented adequate statistical policies and
procedures for conducting special studies.
RECOMMENDATIONS
As a result of our review, we are making several recommendations.
To establish effective management controls in OPIR’s operations,
we are recommending that OPIR improve its management control program
and establish special study and statistical policies and procedures.
Specifically, we are recommending that OPIR:
1. develop and maintain a workload
tracking system which includes monitoring prior and current assignments
and resources
expended;
2. develop and maintain a system for tracking the
current status of OPIR recommendations incorporating any prior
recommendations that have not been implemented;
3. establish OPIR as a management control area in
SSA’s FMFIA program;
4. develop and implement policies and procedures
for conducting OPIR studies, including requirements for maintaining
adequate study documentation for examination; and
5. implement statistical policies and procedures,
such as those published in the General Accounting Office’s
(GAO) Using Statistical Sampling publication (GAO/PEMD-10.1.6)
and modify reporting formats to include detailed statistical information,
such as response rates and confidence intervals, in all reports
where statistical sampling is used.
AGENCY COMMENT
SSA decided not to comment on the draft report. The
Agency has engaged a consultant to review a number of issues related
to OPIR’s operations. The Agency believes that it would be beneficial
for the consultant to consider our findings and recommendations before
providing specific comments.
OPIR’s mission is to evaluate and assess the integrity
and quality of SSA programs with emphasis on the prevention of program
and systems abuse, the elimination of waste, and the increase of
efficiency. To accomplish its mission, OPIR conducts quality assurance
reviews and special studies. These reviews and studies generally
contain recommendations for corrective changes in programs, policies,
procedures, or legislation aimed at quality and productivity improvement
and/or program simplification.
OPIR is headed by the Associate Commissioner for Program
and Integrity Reviews who reports directly to the Deputy Commissioner
for Finance, Assessment, and Management and Chief Financial Officer.
OPIR has 4 headquarters offices and 10 regional offices
that conduct its quality assurance reviews and special studies. The
10 Offices of Regional Program and Integrity Reviews support
quality assurance and special study activities that are formulated
by OPIR’s headquarters offices. OPIR is organized as follows:
(CHART UNAVAILABLE AT THIS TIME)
During FYs 1993 and 1994, OPIR, on average, had approximately
426 headquarters staff and 1,125 field staff in its 10 regional offices
and an operating budget of $90 million. Between October 1, 1992 and
September 30, 1994, we estimate that 25 percent of OPIR’s resources
were dedicated to special studies with the remainder primarily dedicated
to quality review initiatives.
Special studies are conducted within each of OPIR’s
major offices. Special studies generally refer to one-time studies
that focus on a particular area or issue of SSA’s programs.
These special studies cover the full range of legal requirements,
policies, procedures, systems, and operations for the Retirement
and Survivors Insurance (RSI), Disability Insurance, and Supplemental
Security Income (SSI) programs. They also cover service delivery
issues that relate to SSA’s field and teleservice offices including
customer satisfaction surveys of its customers. These special studies
sometimes result in formal reports that contain recommendations or
informational reports that summarize the data gathered during the
study.
Examples of OPIR’s quality review initiatives
include the annual quality reviews of the accuracy of disability
decisions in addition to the Index of Dollar Accuracy and Stewardship
reviews that provide payment accuracy measures for the RSI and SSI
programs. Another initiative involves OPIR staff visits to SSA’s
field offices for the purpose of improving operational issues.
Our audit was conducted in accordance with generally
accepted government auditing standards. The objectives of our audit
were to determine whether OPIR`s reviews were: (1) used by management
to improve SSA programs; (2) cost-effective and efficient; and (3) performed
in accordance with established standards.
To determine whether OPIR special studies were cost-effective
and used by management, we designed audit steps to review management
and operational controls and special study workloads. We also designed
steps to review the policies and procedures used to perform reviews
and report on study results and recommendations. Our audit procedures
included a review of:
applicable SSA and OPIR policies and procedures;
OPIR’s systems for measuring, reporting, and
monitoring program performance relating to special studies;
OPIR work plans and other available management
information;
OMB, GAO, and other standards for implementing
and maintaining management controls; and
OMB statistical standards and other guidelines
for statistical sampling.
We could not achieve our objectives because OPIR records
were not complete. Therefore, we performed alternative audit procedures
to identify the universe of OPIR’s FYs 1993 and 1994 workloads,
that included reviewing work plans, OPIR’s correspondence records,
management’s report files, and other available information.
Because of the limitations in this data, we were unable to:
identify the total inventory of special studies
conducted during the audit period;
determine the status of recommendations that resulted
from special studies conducted during the audit period; and
determine the resource and operational costs attributable
to special studies to measure the cost-effectiveness and efficiency
of OPIR’s special study operations.
We also designed procedures to review OPIR’s management
controls over operations and controls that related to the conduct
of its special studies. Our assessment showed that management controls
were not properly designed and/or implemented, therefore, we could
not rely on them.
We continued to perform additional audit steps to determine
whether compensating procedures existed in the management of individual
special studies and to identify standards that were used by OPIR
when conducting special studies. We selected a judgmental sample
of 10 studies that were either initiated, in progress, or completed
during FYs 1993 and 1994. For the 10 studies, we:
reviewed draft and/or final reports;
reviewed project statements, highlight reports,
monthly status reports, and other study documentation, where available,
and
interviewed OPIR management and staff responsible
for conducting the 10 selected studies.
Several of the 10 selected studies were conducted by
regional staff. We requested interviews with regional staff who conducted
the studies, but were informed that individuals who worked on these
studies were no longer employed with OPIR. Therefore, we relied on
headquarters’ managers and staff who were involved with these
studies to assist us.
We conducted our audit between November 1995 and May
1996 at SSA headquarters in Baltimore, Maryland.
We were not able to meet our intended objectives because
our assessment of OPIR’s policies and procedures showed that
OPIR has not implemented appropriate management controls necessary
for assuring accountability for special studies. OPIR has not developed
a system for tracking special studies and proposed recommendations,
established standards for conducting its special studies, or established
policies for assuring the statistical validity of its studies. As
a result, we were unable to ascertain either the cost-effectiveness
or efficiency of OPIR’s special studies and their usefulness
to Agency managers. Therefore, we are reporting on the conditions
that we found relating to weaknesses in management controls and the
need for OPIR to establish management controls, special study standards,
and statistical policies.
OPIR has not implemented effective management controls
to ensure accountability for its special study operations. OPIR does
not have systems that maintain an inventory of its special studies,
the recommendations resulting from studies, or the status of the
recommendations within the Agency. OMB Circular A-123 requires that
management controls be implemented to ensure that managers are accountable
for their operations and for the proper stewardship of Federal resources.
Since management has not committed to these principles, we were unable
to evaluate OPIR’s special study performance or the value of
its special studies to the Agency.
We requested a complete inventory of OPIR’s workload
to identify special studies conducted during FYs 1993 and 1994. However,
OPIR was unable to readily produce a comprehensive accounting of
its special study workload for studies that were initiated, ongoing,
or completed during the audit period. We extended our audit steps
and attempted to quantify OPIR’s study workload by reviewing
other available information such as management’s report files,
correspondence files, available status reports, and work plan documents.
Despite these efforts, we were still unable to reasonably reconstruct
OPIR’s workload because the available information was incomplete
and inconsistent.
We also found that OPIR does not have a mechanism that
provides an inventory of OPIR’s study recommendations or a method
for following up to assure that study recommendations are implemented
within the Agency. OPIR also does not accumulate information that
allows management to determine the costs or resources invested when
conducting these special studies. Therefore, we could not evaluate
the usefulness of these studies to the Agency, the resulting impact
of study recommendations, or whether budgetary resources were used
in the most efficient and effective manner. As a result, we cannot
attest to the value of OPIR’s special study efforts to the Agency
or to the impact that its study efforts have on improving SSA’s
operations because OPIR does not have management controls to track
this information.
OPIR’s primary controls for managing studies and
monitoring performance are through the use of project statements,
highlight reports, and monthly status reports. The project statements
are prepared at the beginning of a study and summarize the study’s
purpose and review methodology. The highlight reports serve as an
informal, internal mechanism to promote staff communication and awareness
of current OPIR activities, while the monthly status reports provide
management with the current status of studies. These documents are
used to manage studies only while the studies are in progress and
are not intended to be used as historical documents. Each manager
has the discretion to retain or dispose of this information. These
documents could not be utilized by management to construct an inventory
of studies and recommendations or be used to monitor the implementation
or status of study recommendations. Therefore, these documents are
insufficient for the purposes of evaluating OPIR’s performance.
FMFIA requires agencies to establish internal accounting
and administrative controls in accordance with standards prescribed
by the Comptroller General. It also requires agencies to evaluate
their controls and annually report on the status of compliance with
the requirements of FMFIA. To implement FMFIA, OMB issued Circular
A-123 to provide guidance to Federal managers on improving accountability
and effectiveness of Federal operations by establishing, assessing,
correcting, and reporting on management controls.
The Circular states that Federal agency management
has a fundamental responsibility for being accountable for the proper
stewardship of Federal resources. It requires management to develop
and implement appropriate, cost-effective management controls in
their program and administrative functions. It also requires managers
to assess and report on the adequacy of those controls in their operations.
OMB Circular A-123 integrates management control activities into
other management requirements and policies, such as the Government
Performance and Results Act of 1993, P.L. 103-62. This integration
emphasizes management’s responsibility to design and implement
an appropriate management structure that helps ensure accountability
for results, and includes appropriate, cost-effective controls.
Management controls are operational checks and balances
which ensure that a task will be carried out as planned in the most
efficient and effective way possible. The control environment represents
the collective effect of various factors on establishing, enhancing,
or mitigating the effectiveness of specific policies and procedures.
One of these factors is management’s control methods for monitoring
and following up on performance, including internal auditing. Management
control activities are not considered to be stand-alone management
practices, but rather are woven into the day-to-day operational responsibilities
of agency managers.
The expectation that managers are responsible for implementing
management controls is common to basic management practices that
are recognized in the Federal community. For example, a Public
Administration Review article explains the concept of management
controls as follows:
"The concept of management control is
quite simple. Management control encompasses all activities
designed
to assure that an organization accomplishes its objectives effectively
and efficiently (1) within the planned time frame, (2) within
approved cost limitations, and (3) with the planned quality and
quantity of output. The control process spans the whole gamut
of management activities from deciding what the organization
should do or what it should emphasize, to allocating funds, monitoring
activities, conducting reviews, making midcourse corrections,
and evaluating organizational and individual performance. As
such, it differs from internal accounting control because of
its emphasis on results."
We believe that controls have not been established
because management has not recognized the importance of management
controls and made a commitment to implementing them to ensure accountability
for its operations. This is evidenced by the exclusion of OPIR from
SSA’s annual FMFIA program. As a result, SSA has not conducted
a periodic management control review of OPIR’s operations to
determine whether controls are adequate and working as intended,
and whether new controls are needed. These reviews are required by
OMB Circular A-123 to meet the requirements of FMFIA and should be
reasonably comprehensive so that weaknesses in controls are identified
and corrected. The FMFIA program seeks to ensure that strong management
controls exist in Federal operations thereby decreasing the likelihood
of fraud, waste, and mismanagement.
We originally intended to review a statistical sample
of special studies that were conducted during FYs 1993 and 1994.
However, due to the lack of management controls, we were unable to
identify the universe of OPIR’s special studies. Alternatively,
we selected a judgmental sample of 10 special studies from OPIR’s
work plans and correspondence records that were conducted during
the period under audit to assess whether standards were employed.
Our review of these 10 studies showed that the standards for performing
OPIR’s special studies are informally established by the staff
who conduct the review and not documented to assure compliance. Management
has not implemented professionally accepted or recognized study standards
or established policies or procedures for conducting OPIR’s
special studies.
For the 10 studies we reviewed, we asked OPIR to provide
all study documentation or working papers in order to evaluate management`s
involvement, the basis for decisions, and the support for statistical
sampling and study methodologies used during the conduct of these
studies. We were provided study documentation for three of the special
studies because no documentation existed for the other seven studies.
We found that the available study documentation for the three studies
did not provide sufficient information for us to assess that standards
were employed during the conduct of these studies.
We also requested statistical documentation, when applicable,
from OPIR staff who conducted these studies and from OPIR’s
statistical specialists. These specialists provide technical assistance
and oversight for statistical sampling performed across the various
OPIR components. OPIR did not provide sufficient supporting statistical
documentation that would allow us to verify either the validity of
the sampling methods used or the basis for conclusions drawn from
the samples.
We interviewed OPIR’s management regarding the
retention of study documents. They stated that they do not require
staff to maintain documentation for historical purposes and if information
is maintained, it is at the sole discretion of the staff conducting
the study. Management relies on the latest version of the study report
as the documentation of the results of the study and the statistical
sampling methodology used to conduct the review.
OMB Circular A-123 requires that management controls
be established consistent with the Comptroller General’s "Standards
for Internal Controls in the Federal Government." Two specific
management control standards address supervisory and documentary
techniques that are essential to providing assurance that internal
control objectives are achieved. The supervision standard requires
supervisors to continuously review and approve the assigned work
of their staff. This requires supervisors to clearly communicate
the duties, responsibilities, and accountabilities assigned to each
staff member, in addition to systematically reviewing and approving
each member’s work.
At the same time, the documentation standard requires
documents to be complete and accurate to facilitate tracing the event
and related information from before it occurs, while it is in process,
to after it is completed. It goes on further to state that "documentation
of internal controls systems and transactions and other significant
events be purposeful and useful to managers in controlling their
operations, and to auditors or others involved in analyzing operations." Other
professionally recognized standards for conducting studies, audits,
or inspections, such as the Internal Auditing Standards, generally
require that documentation of the work performed be maintained so
that an independent party can ascertain whether the evidence supports
the studies’ significant conclusions and judgements.
As a result of OPIR having no documentary evidence
that supports its work performed during these studies, senior management
cannot attest to the quality and accuracy of its work. Since OPIR’s
mission is to assist the Agency in making improvements in SSA programs
and it can recommend significant program changes resulting from its
work, OPIR should be in a position to defend the methodologies used
to reach its conclusions both internally and externally to the Agency.
In reviewing OPIR’s special studies, we found
that two of the studies selected for review did not comply with minimal
OMB statistical requirements. For the other eight studies, the reports
and supporting documentation did not contain sufficient detail to
determine the validity of the statistical methods used. The two studies
were the "Evaluation of SSA Field Office Referral Services" (FO
Referral) and the "Evaluation of Teleservice Center Referral
Services" (TSC Referral), both of which involved collecting
information from the public to evaluate SSA’s effectiveness
in making referrals to other government services.
When Federal agencies perform studies that involve
the collection of information from the public, they must comply with
OMB regulations. Under the authority of 5 CFR § 1320.9,
OMB requires effective and efficient statistical survey methodology.
These two studies did not comply with OMB standards because OPIR
utilized inappropriate sample selection and data collection method
which resulted in a response rate well below OMB’s minimum response
rate requirements. These deficiencies occurred because OPIR has not
implemented adequate statistical policies and procedures.
OMB requires that statistical surveys be conducted
using appropriate statistical methods that will produce valid and
reliable results that can be generalized to the universe. OMB also
issued specific guidance in January 1979 regarding minimal response
rates when using statistical sampling. OMB’s minimal requirements
for response rates are stated as follows:
"Survey Response Rates and Sample Design.
It is expected that data collections based on statistical methods
will have a response rate of at least 75 percent. Proposed data
collections having an expected response rate of less than 75 percent
require a special justification. Data collection activities having
a response rate of under 50 percent should be terminated."
Response rates lower than 75 percent put the statistical
validity of the results at risk and may, ultimately, provide misleading
results because the nonresponders may be unlike the sampled group
that responded.
We reviewed the reports that were written for both
studies. We were unable to review documentation that supported the
statistical sampling methods used during the review because information
was not maintained by OPIR. However, the reports contained enough
information for us to conclude that minimal OMB standards were not
followed.
Both studies used a preselection method for identifying
the survey sample. In the field office study, the field offices distributed
a form to each visitor asking whether the visitor would or would
not be willing to participate in the survey. A similar method was
used in the teleservice center study. The OPIR staff selected their
sample from the individuals who agreed to participate. We
concluded that this method of selection is not based on probability
sampling because each item in the population does not have a known
and specified (nonzero) probability of being selected. We contacted
officials from OMB and confirmed that this method, referred to as
a "self-selected" sampling method, would not meet OMB’s
statistical sampling requirements. Most experts believe that results
of customer surveys cannot be interpreted with confidence unless
probability sampling is used.
Both of the studies resulted in response rates lower
than OMB minimal requirements. The FO Referral study had a reported
response rate of 28 percent, which is well below the 50 percent
termination threshold. OPIR issued this report to the Agency and
provided estimates on a national basis with a qualifying statement
in the report methodology section stating that "Because only
28 percent of the sampled field office (FO) visitors either agreed
to participate in this survey or were able to be contacted at a later
date, we cannot be certain that the data in this report, that is
based on 992 respondents, is representative of all FO-referred
individuals." The FO Referral study reported an 81 percent referral
accuracy rate to the Commissioner of Social Security. OPIR also reported
that due to the small sample size (45 cases out of 992 respondents)
of cases requiring a referral, it was unable to isolate any specific
corrective actions.
The TSC Referral report was written in draft, but not
issued outside of OPIR due to its poor response rate and internal
disagreement over the validity of the statistical methods used during
the study. We reviewed the draft report and calculated a 34 percent
response rate, also well below OMB’s minimum requirements.
Response rates below 50 percent do not produce valid
results and ultimately waste valuable resources because the conclusions
drawn from the information are not useful to the Agency. In line
with OMB requirements, both of these studies should have been terminated
earlier in the study process. We were unable to determine the amount
of resources used to conduct these two studies, because this information
was not maintained by OPIR. From the study reports, we estimate that
at least 1,200 individuals were interviewed by OPIR staff during
the course of these two studies.
We found that OPIR management has not implemented adequate
statistical policies for addressing sampling design, selection, estimation,
and reporting procedures. To assure that proper statistical methods
are used, management needs to implement stringent policies and procedures,
thereby ensuring the quality of OPIR studies.
Some time after these two studies were completed, we
found that OPIR management was made aware of the improper statistical
methods being used. As a result, on August 9, 1995, OPIR
management issued a memorandum on reengineering OPIR’s statistical
practices. In the memorandum, management states that OPIR’s
current and future studies must be based on a probability sample
and must have a high response rate. It also states that "OPIR
must strive to achieve OMB’s minimal requirement or response
rates." This memorandum was drafted in consultation with OMB
officials. Despite management’s recognition of the statistical
deficiencies, OPIR management has not developed specific procedures
or supervisory controls, or followed up to ensure that OPIR’s
studies use appropriate statistical methods.
Due to the lack of management controls, we were unable
to ascertain either the cost-effectiveness or efficiency of OPIR’s
special studies and their usefulness to the Agency. We found that
OPIR’s management controls are not sufficient for management
to adequately assess its performance and be accountable for its special
study operations. We also found that standards for conducting studies
and for performing statistical sampling have not been established.
Management needs to make a commitment to implementing appropriate
management controls and procedures to ensure that studies are: (1) conducted
in accordance with management’s expectations; (2) cost-effective
and efficient; (3) useful to the Agency; and (4) performed in
accordance with standards.
To improve OPIR’s management controls to ensure
that it is accountable for its special study operations, we recommend
that OPIR:
1. develop and maintain a workload tracking system
which includes monitoring prior and current assignments and resources
expended;
2. develop and maintain a system for tracking the
current status of OPIR recommendations incorporating any prior
recommendations that have not been implemented; and
3. establish OPIR as a management control area in
SSA’s FMFIA program.
To ensure that OPIR studies are carried out in accordance
with management’s directives and to ensure that OPIR’s
objectives are achieved, we recommend that OPIR:
4. develop and implement policies and procedures
for conducting OPIR studies, including requirements for maintaining
adequate study documentation for examination.
To assure that the statistical sampling OPIR performs
when conducting studies is valid and defensible, we recommend that
OPIR:
5. implement statistical policies
and procedures, such as those published in the GAO’s Using
Statistical Sampling publication (GAO/PEMD-10.1.6) and modify
reporting formats to include detailed statistical information,
such as response
rates and confidence intervals, in all reports where statistical
sampling is used.
AGENCY COMMENT
SSA decided not to comment on the draft report. The
Agency has engaged a consultant to review a number of issues related
to OPIR’s operations. The Agency believes that it would be beneficial
for the consultant to consider our findings and recommendations before
providing specific comments. The full text of SSA’s response
is included in Appendix B.
Gary Kramer, Director, Program Audits
Carl Markowitz, Team Leader
Stephanie Palmer, Team Member/Auditor-In-Charge
Mary Ann Braycich, Team Member
George Richardson, Team Member
Tom Tennant, Team Member
www.socialsecurity.gov
