Skip to content
Social Security Online
Office of the Inspector General
OIG Seal image
Blank Spacer Image

Audit Report - A-13-96-11052


Office of Audit

Review of the Back-up and Recovery Procedures at the National Computer Center - A-13-96-11052 - 6/19/97

TABLE OF CONTENTS

EXECUTIVE SUMMARY

BACKGROUND

SCOPE

RESULTS OF REVIEW

BRP Only Addresses Short Term Requirements

Processing Death Notices Is Not Considered a Critical Workload

No Clear Policy for FOs to Follow for Walk-In Clients

Cabinets Containing Back-up Tapes for OSSF Were Observed Unlocked

Entrance to the Back-up Tape Vault at OSSF Did Not Have a Lock to Prevent Unauthorized Access

Requirements in the Contract for Transporting Back-up Tapes Are Not Being Verified

CONCLUSION AND RECOMMENDATIONS  

EXECUTIVE SUMMARY

Each year, the Social Security Administration (SSA) processes over 220 million earnings records, pays monthly benefits to about 45 million individuals, and issues new or replacement Social Security cards to over 16 million people. These activities are supported by SSA`s automated systems at the National Computer Center (NCC) in Baltimore, Maryland. Due to the critical role of the NCC in performing these functions, it is essential that SSA provide for continuing operations in the event of a disruption to functions performed at the NCC.

SSA is required by the Office of Management and Budget (OMB) Circular A-130 to have in place a disaster recovery plan for its automated systems. Specifically, OMB Circular A-130, "Security of Federal Automated Information Systems" requires that agencies maintain disaster recovery and continuity of operations plans for all information technology installations should events occur that prevent normal operations at the installation. Plans should be fully documented and periodically tested. The objective of this review was to determine if SSA is in compliance with OMB Circular A-130 and provisions of the Privacy Act of 1974 which apply to security and confidentiality of records used in back-up and recovery procedures.

SSA has made significant improvements in its back-up and recovery planning since the Office of the Inspector General (OIG) last reported in March 1984. SSA has chosen a recovery strategy to process only the critical workloads at a shared commercial back-up facility, a strategy we believe is the most cost-effective for SSA. SSA`s Back-up and Recovery Plan (BRP) is documented and updated annually. Once a year, SSA tests its BRP by bringing up the system at a commercial back-up facility. The network is tested by having several Program Service Centers (PSC) and field offices (FO) submit on-line transactions directly to the back-up facility.

Generally, we believe SSA is in compliance with OMB Circular A-130 and provisions of the Privacy Act of 1974 relating to the security and confidentiality of records used in back-up and recovery procedures. However, some improvements should be made in the BRP and other areas. Specifically, with respect to information contained in SSA`s January 31, 1996 BRP document, we found that:

  • The BRP only addresses short term (42 days) recovery requirements. For a long-term outage, there has been no planning by senior management for setting goals for the level of data processing service to be provided, nor for when SSA becomes fully operational. The level of service to be provided will determine the computer hardware requirements. Without the setting of service level goals by senior management, adequate planning cannot take place for the acquisition, installation, and operation of computer equipment necessary to meet management`s objectives.
  • The BRP identifies critical workloads SSA would process in the event of a disaster. However, the processing of death notices to quickly remove beneficiaries from payment status is not considered a critical workload and is given the lowest priority to process. Based on current data provided by SSA, on average over 156,000 benefit payments are terminated monthly due to the death of the beneficiary. Thus, if payments are not terminated, SSA would be issuing over $105 million monthly to deceased beneficiaries. Once full data processing services have been restored, SSA would then have to generate recovery notices. The recovery notices would create an enormous follow-up workload for SSA; and in some cases, erroneous payments would not be recovered.
  • The goal stated in the BRP is to restore on-line services to the FOs within 72 hours of a declared disaster. However, there is no policy in the BRP on what information, if any, is to be obtained from walk-in clients during the first 72 hours. Not having a clearly stated policy for the FOs to follow will result in confusion and inconsistency in the level of service provided to the client.

With respect to observations made at the Metro West (MW) building, the off-site storage facility (OSSF), and information contained in the contract for transporting the back-up tapes, we found that:

  • On May 1, 1996 one of the two cabinets at the MW building for transporting the back-up tapes to OSSF was unlocked. An unlocked tape cabinet permits unauthorized disclosure to the casual or curious observer and, therefore, is not in full compliance with the security and confidentiality provisions in the Privacy Act of 1974.
  • The entrance to the back-up tape vault at the OSSF on June 11, 1996 did not have a cipher lock to prevent unauthorized access by the Office of Central Records Operations (OCRO) personnel. About 25 people from OCRO have unauthorized access to the OSSF tape vault. As a result, personal records on the back-up tapes are not secured as required by physical safeguard provisions in the Privacy Act of 1974.
  • Compliance with requirements in the back-up tape transportation contract are not being verified by SSA. For example, during shipping, the temperature and relative humidity in the tape cargo area should be recorded daily by the vendor and should be verified monthly by SSA to the specifications in the contract. We found that the shipping process was exposing the back-up tapes to critical environmental changes in temperature and relative humidity. Changes, especially in temperature, could have damaged the tapes during shipping and caused the data to be unusable.

To improve its Back-up and Recovery Planning process, we are recommending SSA:

  • Begin planning for a long-term outage. The plan should include a time table for increasing the level of data processing service. It should also have a stated goal of when SSA would be fully operational again after a long-term outage.
  • Perform a cost/benefit analysis to determine the feasibility of processing death notices as a critical workload.
  • Establish a clear policy for treating walk-in clients while the "system" is being brought up at the back-up facility.
  • Reinforce established procedures which call for tape cabinets to be locked before leaving the NCC and have supervisors verify that the cabinets are locked.
  • Secure the entrance to the OSSF tape vault permitting access to only authorized OCRO personnel.
  • Verify contractor`s compliance with requirements in the back-up tape transportation contract.

Back to top

BACKGROUND

OMB Circular A-130 requires that Federal agencies develop a disaster recovery plan. The objective of the plan should be to provide reasonable continuity of data processing support should events occur that prevent normal operations at the installation. The plan should be fully documented and operationally tested periodically, at a frequency commensurate with the risk and magnitude of loss or harm that could result from the disruption of data processing support. In addition, the Privacy Act of 1974 requires each agency that maintains a system of records to "establish appropriate administrative, technical and physical safeguards to ensure the security and confidentiality of records and to protect against any anticipated threats or hazards to their security or integrity."

In the mid 1980’s, SSA became aware of its BRP inadequacy. This awareness was a result of SSA’s reacting to: (1) regulatory requirements, (2) an OIG Contingency Planning report, and (3) changes that were taking place in its processing environment. During the last decade, the Agency made two significant changes. First, SSA greatly increased the use of on-line processing to support its field operations. Second, SSA instituted toll-free telephone communications, the effectiveness of which rests almost entirely on the NCC computers and on-line terminals. However, SSA realized that its BRP could not even provide limited on-line support, beginning SSA`s effort to redefine its disaster recovery strategy.

In 1989, SSA sought advice and guidance from the National Academy of Sciences (NAS) concerning long-range planning for systems modernization. NAS concluded that, "SSA should limit its disaster recovery strategy to a chosen set of critical functions rather than planning to back-up all of its processing functions, because full back-up is impractical." Acting on NAS`s advice, SSA, in 1990, began to evaluate normal workloads for determining those which would be considered critical. Based on this evaluation, SSA determined it would need approximately 20 percent of the computer terminals used in FOs, PSCs, and teleservice centers support the critical workloads.

Next, SSA evaluated the cost and benefits associated with several back-up and recovery alternatives. There were 13 alternatives evaluated, each of which involved the use of a commercial or a Government-owned facility. SSA considered two to be the most viable. The first alternative was to acquire a shared commercial service. The second was to modify an existing SSA facility and move a portion of the NCC computer resources to provide a back-up capability.

The alternative back-up and recovery strategy approved by the Commissioner on June 3, 1991 was to acquire a shared commercial service. Concurrently, the Commissioner approved the designation of the critical workloads that would be processed at the back-up facility. On June 29, 1993, SSA contracted with COMDISCO in North Bergen, New Jersey to provide SSA`s back-up support. The contract has a 1-year base with a 3-year option for a total contract life of 4 years from June 29, 1993 to June 28, 1997.

Back to top

SCOPE

Our review was performed in accordance with generally accepted government auditing standards. Field work was performed at SSA headquarters in Baltimore, Maryland; Metro West building in Baltimore City, Maryland; OSSF in Boyers, Pennsylvania; and at the back-up facility (COMDISCO) in North Bergen, New Jersey between February 1996 and August 1996. The objective of this review was to determine if SSA is in compliance with OMB Circular A-130 and the Provisions of the Privacy Act of 1974 relating to the security and confidentiality of records used in back-up and recovery procedures.

To achieve our objective for this review, we:

  • reviewed OMB Circular A-130 and the Privacy Act of 1974 which respectively requires SSA to develop a disaster recovery plan and to also protect the security and confidentiality of records;
  • reviewed previous studies done by OIG and others in this area;
  • reviewed SSA`s January 31, 1996 Back-up and Recovery Plan document;
  • reviewed SSA`s recovery test results documents for December 1993, August 1994, and January 1996 conducted at the back-up facility;
  • reviewed SSA`s contract with the back-up facility vendor, the lease agreement for OSSF, and the contract for transporting the back-up tapes to OSSF;
  • interviewed SSA personnel responsible for the back-up and recovery process;
  • performed an analysis to determine if all the terminals designated as back-up devices were actually identified using the network software by regional personnel; and
  • made a site visit to the MW building, in Baltimore City, Maryland; to OSSF in Boyers, Pennsylvania; and to the back-up facility in North Bergen, New Jersey.

Back to top

RESULTS OF REVIEW

Generally, we found that SSA was in compliance with OMB Circular A-130 and the provisions of the Privacy Act of 1974 relating to security and confidentiality of records used for back-up and recovery procedures. However, further improvements are needed to strengthen SSA`s overall back-up and recovery planning process. BRP only addresses SSA`s short term outage (42 days) requirements. SSA has not planned for a long-term outage, nor set goals for the level of data processing service they want to provide. This information is important for determining the hardware requirements and their availability. Also, to minimize erroneous payments and improve efficiency, SSA should reconsider processing death notices as a critical workload. Furthermore, we observed the cabinets for transporting the back-up tapes to OSSF were not always locked and a lock has not been installed on the door of the tape vault at OSSF to prevent unauthorized access. Finally, SSA has not been verifying the contractor`s compliance with requirements in its tape transportation contract.

BRP Only Addresses Short Term Requirements

BRP document only addresses a short term solution to SSA`s back-up and recovery needs. The short term solution is to process only SSA`s critical workloads at a commercial back-up facility. The critical workloads represent about 20 percent of SSA`s total workloads and SSA has contracted with COMDISCO, to provide the back-up services. The contract permits SSA to use COMDISCO`s computer equipment for up to 42 days. After that, COMDISCO would provide a room for up to 180 days with a raised floor, power, and other supplies necessary for installing computer equipment supplied by SSA. This arrangement is referred to as a "shell site."

However, in the event of a long-term outage, which we have defined as greater than 42 days, we found no evidence of long-term planning by SSA`s senior management for what level of data processing service they expect to provide and a goal for when SSA should be fully operational again. The expected level of data processing service will drive the computer hardware requirements needed for the "shell site." Without the setting of service level goals by senior management, adequate planning cannot take place for the acquisition, installation, and operation of computer equipment necessary to meet management`s objectives.

SSA should have a plan for phasing in more service and have a stated goal for when senior management would like to have data processing services fully restored. A work group should then be established to determine if hardware could be acquired, installed and made operational in time to meet the service level goal. This information should be documented in the BRP.

Processing Death Notices Is Not Considered A Critical Workload

Because there will only be a limited number of terminals available (20 percent of existing terminals) in the event of a disaster, SSA, through its BRP, has identified the critical workloads it would process. SSA made a decision to process only those events that are favorable to the beneficiary. Examples of these events include placing an individual in pay status, changing address information, or increasing a benefit amount. However, SSA did not consider the costs and benefits (such as trust fund savings and work load savings) of considering death terminations a critical event. As a result, the processing of death notices, which would remove beneficiaries from payment status, would not be processed. We believe SSA should reconsider processing death notices as a critical workload because of the negative impact it would have on future SSA workloads and risk of wasting program finances if death notices were not processed in a timely manner.

Currently, over 156,000 beneficiaries are terminated monthly because of death. In a disaster situation, if death notices were not processed timely, SSA would be issuing over $105 million monthly to ineligible beneficiaries. Once full data processing services have been restored, SSA would then have to generate recovery notices. The recovery notices would create an enormous workload for follow-up and in some cases, the erroneous payments would not be recovered.

In making another comparison, it is currently costing SSA $29,500 a month for the right to use COMDISCO`s computers to process all of SSA`s critical workloads. The addition of one more workload item, death notices, to the critical workload list should not significantly increase the total cost of the back-up contract. We believe, this additional cost is a modest amount when compared to the potential loss to the trust funds of $105 million a month, and the additional operating expense SSA would incur for processing a large recovery workload, if death notices were not processed timely.

While we generally agree with SSA`s policy for identifying critical workloads, we also believe that SSA should reevaluate its decision of not processing death notices as a critical workload. A cost/benefit analysis should be performed to determine the feasibility of processing death notices as a critical workload. This analysis should weigh the possible additional cost to SSA, if any, against the benefit of preventing uncollectible losses to the trust funds and eliminating large recovery workloads.

No Clear Policy For FOs To Follow For Walk-In Clients

BRP does not contain a clear policy on how the field/district offices are to handle walk-in clients while the "system" is being brought up at the back-up facility. The goal for SSA is to be operational within 72 hours of the Commissioner’s declaring a disaster. For the first 72 hours or so, the FOs will not be able to get on-line to help walk-in clients. BRP does not specifically state how the FOs are expected to treat these walk-in clients.

With SSA having over 1,300 FOs and not having a stated policy, there may be an inconsistency in the level of service provided to walk-in clients during the first 72 hours. Several scenarios may occur. Some FOs may try to take all the information on paper necessary for processing a claim at a later time for when the processing capability is restored. Other FOs may take certain client information such as name, address, Social Security number, telephone number, and reason for visit, then recontact the client when processing capability is restored. Other FOs may not take any information and tell the walk-in client to recontact the office in a few days.

SSA should incorporate within BRP a clear policy on what information the FOs are to take from walk-in clients while processing capability is being restored at the back-up facility. A clear policy will help eliminate the confusion and inconsistency in the level of service provided to the client.

Cabinets Containing The Back-Up Tapes For OSSF Were Observed Unlocked

NCC ships daily to the MW building (OCRO) the back-up tapes from the previous day`s updates. MW serves as an interim storage site, where twice weekly the tapes are shipped from MW to the permanent OSSF located in Boyers, Pennsylvania.

On May 1, 1996 we reviewed the tape receiving and handling procedures at the MW building. We found that on several occasions, unlocked tape cabinets had been shipped to the MW building from Office of Systems (OS) personnel at NCC. We determined that the unlocked cabinets were caused by the failure of OS personnel to follow established procedures and by supervisors not verifying procedures were followed. An unlocked tape cabinet permits unauthorized disclosure to the casual or curious observer. Therefore, SSA is not in full compliance with provisions of the Privacy Act of 1974 which apply to security and confidentiality of records used in back-up and recovery procedures. The Privacy Act of 1974 requires SSA to "establish appropriate administrative, technical, and physical safeguards to insure the security and confidentiality of records and to protect against any anticipated threats or hazards to the security or integrity which could result in substantial harm, embarrassment, inconvenience, or unfairness to any individual on whom information is maintained."

Management should remind OS personnel of the importance of locking the cabinets before transporting the back-up tapes to OSSF. Supervisors at NCC should verify that the cabinets are locked before transporting the back-up tapes.

Entrance To The Back-Up Tape Vault At OSSF Did Not Have A Lock To Prevent Unauthorized Access

Physical security over the back-up tape vault at OSSF is not effectively maintained because the entrance to the back-up tape vault does not have a lock to prevent unauthorized access. SSA has about 78,500 square feet of storage space at OSSF, including about 6,250 square feet for the back-up tape vault. Currently, 35 people from OCRO permanently work at OSSF, including 10 people with authorized access to the tape vault room. The remaining OCRO personnel handle requests for information to be retrieved or work the SS-5 process which is a processing request for Social Security cards that come directly by mail from district offices.

The back-up tapes are delivered by the transportation contractor to OSSF late (10:00 p.m. to 12:00 p.m.) on Mondays and Thursdays. The truck stays in the secured OSSF truck coral until the next morning when the back-up tapes are delivered to OCRO`s back door. Once received, OCRO personnel verify that the shipment of tapes has the proper sequence number and the cargo seal has not been broken. OCRO personnel use a fork lift to remove the tape cabinets since the truck does not have a gate lift and OCRO does not have a loading dock. The tape cabinets are placed in a staging area inside OCRO`s secured space but outside the tape vault room. The cabinets are unlocked (same key for all cabinets) and loaded on to smaller carts of about 50 tapes in order to get through the air lock at the entrance into the tape vault room. The cabinets will not fit through the air lock.

The entrance of the air lock did not have a cipher lock to prevent unauthorized access by OCRO personnel into the tape vault area. The lack of a physical security device permits easy access for the back-up tapes to be stolen or destroyed. If any of the tapes were stolen or destroyed and a disaster were declared at the NCC, it could result in a permanent loss of critical beneficiary data to SSA.

We were told that only 10 people--the office manager, the 6 technicians who work in the tape vault, the janitor, and two mechanical maintenance people--are allowed in the tape vault. However, we found there was nothing to prevent the other 25 OCRO personnel in the immediate area from entering the tape vault. In 1990, major improvements were made in the tape vault room to reduce air dust that could cause tape damage. The OSSF vendor installed a suspended metal ceiling, a vinyl tile floor and an air lock entrance, but did not replace the lock. We believe that it was an oversight that a cipher lock was not installed on the new air lock entrance.

SSA should install a lock on the tape vault door at OSSF to prevent unauthorized access by OCRO personnel and to comply with the confidentiality provisions of the Privacy Act of 1974.

Requirements In The Contract For Transporting Back-Up Tapes Are Not Being Verified

SSA has contracted with National Underground Storage (NUS) to transport its back-up tapes from the MW building in Baltimore, Maryland to the OSSF in Boyers, Pennsylvania. We found that SSA is not verifying all the requirements in the contract with NUS and consequently, is not in compliance with provisions of the Privacy Act of 1974 which apply to security and confidentiality of records. We categorized the requirements into four task groups:

In the task one group, we identified those tasks which require NUS to provide physical security over the tapes. Examples of tasks would include: the truck must have a working alarm system, the transport area must have a device to securely hold the carts/boxes in place during transportation, and the truck must be telephone equipped. Through our observations and interviews, we were satisfied that these requirements were being met.

In the task two group, we identified those tasks which require NUS to provide environmental security over the tapes. The transportation contract requires NUS to maintain in the cargo area, at all times, a temperature of 40 - 85 degrees Fahrenheit and humidity of 20 - 70 percent while transporting SSA`s back-up tapes. To determine compliance with this requirement, we observed one of NUS`s tape deliveries to the NCC. The NUS truck did not have a climate control unit (air conditioning and heating unit), dedicated to controlling, monitoring and recording the temperature and humidity inside the cargo area. Instead, NUS modified the cab of the truck by cutting a 3 and one-half inch hole through the cab back into the cargo box and attached a blower in the cab to push cab air back into the cargo area.

Modifying the truck this way does not meet the temperature and humidity control requirement in the transportation contract for the following reasons. First, the opening in the cargo area is positioned so that, when cargo (a tape cabinet) is pushed up against it, the opening is blocked and no air is able to circulate in the cargo area. For example, the day we observed the truck there was a tape cabinet secured up against the cargo opening and it was impossible for any heat to circulate inside the cargo area from the cab. We found the cabinets to be ice cold to our touch because heat had not been circulating in the cargo area. We estimated the temperature in the cargo area to have been between 28 and 32 degrees during transit from Boyers, Pennsylvania to the NCC. These temperatures are well below the minimum contract temperature of 40 degrees and could result in the tapes freezing up.

Second, we were informed that the driver must stop in Breezewood, Pennsylvania to rest for 8 hours after being on the road for 10 hours, as required by the U.S. Department of Transportation. During this 8 hour rest period, the truck engine is turned off; consequently, no air is circulating in the cargo area during this time. The driver arrives at the rest stop around noon and the tape cabinets sit in the afternoon sun during the hottest part of the day. This is a problem in the summer when temperatures typically exceed 90 degrees. Temperature in the cargo area would also be exceeding 90 degrees, well over the maximum allowable contract temperature of 85 degrees.

Finally, NUS is only taking the temperature and humidity in the cargo area when the truck is leaving Boyers, Pennsylvania. The contract calls for a specific temperature and humidity range to be maintained at all times during transport. In order for NUS to meet this requirement, they would have to be continually monitoring the temperature and humidity in the cargo area during transport. Our observations found no equipment on the NUS truck to monitor the temperature and humidity during transport. The round trip takes approximately 20 hours and the cargo area temperature and humidity could dramatically change in that time period. Based on these facts, we conclude that SSA has no assurance the cargo area has been environmentally safe when transporting SSA`s tapes to OSSF in Boyers, Pennsylvania.

In the task three group, we identified those tasks which require NUS to provide qualified and bonded drivers. The contract authorizes SSA to review driving records for the last 3 years and requires that all drivers be bonded for at least $150,000. Through interviews we found that SSA has never requested to review driving records or verified that the drivers are bonded for $150,000 each. We were able to verify for ourselves, however, that the drivers are currently bonded for $1 million each.

In the task four group, we identified the remaining tasks not identified above. These tasks include providing timely pickup and delivery of back-up tapes and personnel to ensure safe/secure loading and unloading of back-up tapes at SSA loading docks. Through interviews and reviewing time logs, we were satisfied that these requirements were being met.

Verification of all contract requirements for compliance is important to the overall integrity and security of the back-up tapes. The back-up tape shipping process exposes critical media to environmental changes in temperature and relative humidity. Changes, especially in temperature, could damage the tapes causing the data to be unusable in a disaster recovery situation. Also, to help ensure that only qualified drivers are transporting SSA`s back-up tapes, SSA should be reviewing driving records and verifying that each driver is bonded for the amount stated in the contract.

Back to top

CONCLUSION AND RECOMMENDATIONS

SSA has made significant improvements in its back-up and recovery planning since we last reported in March 1984. At that time, back-up and recovery planning at SSA only included batch systems. Today both on-line and batch systems are included in back-up and recovery planning. BRP is well-documented and is periodically updated and tested by bringing the "system" up at a commercial back-up facility. Generally, we believe SSA is in compliance with OMB Circular A-130 and the Privacy Act of 1974 relating to security and confidentiality of records used for back-up and recovery procedures.

However, improvements could be made in the back-up and recovery planning process. Specifically, we are recommending that SSA:

1. Begin planning for a long-term outage. The plan should include a time table for increasing the level of data processing service and have a stated goal for when SSA would like to be fully operational again after a declared disaster. The plan should also include a hardware study to determine if equipment can be acquired, installed, and made operational in time to meet the service-level goal of senior management. All this information should be documented in the BRP. 

2. Perform a cost/benefit analysis to determine the feasibility of processing death notices as a critical workload and add processing death notices to the BRP if this cost/benefit analysis demonstrates cost worthiness.

3. Establish a clear policy in the BRP for treating walk-in clients while the "system" is being brought up at the back-up facility.

4. Reinforce established procedures which call for tape cabinets to be locked before leaving NCC and have supervisors verify that the cabinets are locked.

5. Secure the entrance to the tape vault permitting access to only authorized OCRO personnel.

6. Ensure the contractor comply immediately with the environmental requirements in the contract. Also, verify on an ongoing basis, the contractor`s compliance with all the requirements in the back-up tape transportation contract.

  Link to FirstGov.gov: U.S. Government portal Privacy Policy | Website Policies & Other Important Information | Site Map
Need Larger Text?
  Last reviewed or modified