How do I repair a failed source vSnap in an IBM Spectrum Protect Plus environment?

Steps

1. Log in to the target vSnap server console with the ID serveradmin by using Secure Shell (SSH) protocol.
   Enter the following command: $ ssh serveradmin@MGMT_ADDRESS

   For example, $ ssh serveradmin@10.10.10.2

2. Obtain the ID of the failed source vSnap server by opening a command prompt and entering the following command:

   $ vsnap partner show

   The output is similar to the following example:

   ID:  12345678901234567890123456789012 
   PARTNER TYPE: vsnap 
   MGMT ADDRESS: 10.10.10.1 
   API PORT: 8900 
   SSH PORT: 22

3. Verify that the MGMT ADDRESS is the address of the failed source vSnap server. Take note of the failed source vSnap server's ID number.
4. In the environment with the source vSnap server, install a new vSnap server of the same type and version, and with the same storage allocation, as the failed source vSnap server.
   For instructions about installing a vSnap server, see Installing a physical vSnap server.

   Important: Do not register the new vSnap server with IBM Spectrum Protect Plus. Do not use the Add Disk Storage wizard.
   1. You will first need to initialize the vSnap server with the following command:

      $ vsnap system init ----skip_pool id partner_id

      For example: $ vsnap system init --skip_pool –-id 12345678901234567890123456789012 using the failed source vSnap partner ID. A message indicates when the initialization is completed.

      Note: This command is different to the vSnap initialization command listed in the IBM Knowledge Center and in the Blueprints.
5. Complete the vSnap server and pool creation process as outlined in Chapter 5: vSnap Server Installation and Setup in the Blueprints.
6. Place the new source vSnap server into maintenance mode by entering the following command:

   $ vsnap system maintenance begin

   Placing the vSnap server into maintenance mode suspends operations such as snapshot creation, data restore jobs, and replication operations.
7. Initialize the new source vSnap server with the failed source vSnap server’s partner ID. Enter the following command:

   $ vsnap system init --id partner_id

   The following command is an example: $ vsnap system init --id 12345678901234567890123456789012

8. On the new source vSnap server, add the partner vSnap servers. Each partner must be added separately. To add a partner, enter the following command:

   $ vsnap partner add --remote_addr remote_ip_address --local_addr local_ip_address

   where, remote_ip_address specifies the IP address of the source vSnap server, and local_ip_address specifies the IP address of the new source vSnap server.

   The following command is an example:

   $ vsnap partner add --remote_addr 10.10.10.2 --local_addr 10.10.10.1

9. When prompted, enter the user ID and password for the target vSnap server.
   Informational messages indicate when the partners are created and updated successfully.
10. Create a repair task on the new source vSnap server by entering the following command:

    $ vsnap repair create –-async

    The output of this command is similar to the following example:

    ID: 12345678901234567890123456789012 
    PARTNER TYPE: vsnap 
    PARTNER ID: abcdef7890abcdef7890abcdef7890ab 
    TOTAL VOLUMES: N/A 
    SNAPSHOTS RESTORED: N/A 
    RETRY: No 
    CREATED: 2019-11-01 15:49:31 UTC 
    UPDATED: 2019-11-01 15:49:31 UTC 
    ENDED: N/A 
    STATUS: PENDING 
    MESSAGE: The repair has been scheduled

11. Monitor the number of volumes that are involved in the repair operation by entering the following command:

    $ vsnap repair show

    The output of this command is similar to the following example:

    ID: 12345678901234567890123456789012 
    PARTNER TYPE: vsnap 
    PARTNER ID: abcdef7890abcdef7890abcdef7890ab 
    TOTAL VOLUMES: 3 
    SNAPSHOTS RESTORED: N/A 
    RETRY: No 
    CREATED: 2019-11-01 15:49:31 UTC 
    UPDATED: 2019-11-01 15:49:31 UTC 
    ENDED: N/A 
    STATUS: ACTIVE 
    MESSAGE: Created 0 volumes. There are 3 primary volumes that have recoverable snapshots, the latest snapshot of each will be restored. Restoring 3 snapshots: 3 active, 0 pending, 0 completed, and 0 failed 

    The number of volumes that are involved in the repair operation is indicated in the TOTAL VOLUMES field.

12. Monitor the status of the repair task by viewing the repair.log file on the new source vSnap server, in the following directory /opt/vsnap/log/repair.log. Alternatively, you can enter the following command:

    $ vsnap repair show

    The output of this command is similar to the previous example. The following status messages can be displayed during the repair process:

    • STATUS: PENDING indicates that the repair job is about to run.
    • STATUS: ACTIVE indicates that the repair job is active.
    • STATUS: COMPLETED indicates that the repair job is completed.
    • STATUS: FAILED indicates that the repair job failed and must be resubmitted.
13. During the repair operation, run the vSnap repair show command to verify when the status is COMPLETED.
    $ vsnap repair session show

    The output of this command is similar to the following example:

    ID: 1 RELATIONSHIP: 72b19f6a9116a46aae6c642566906b31 
    PARTNER TYPE: vsnap 
    LOCAL SNAP: 1313 
    REMOTE SNAP: 311 
    STATUS: ACTIVE 
    SENT: 102.15GB 
    STARTED: 2019-11-01 15:51:18 UTC 
    ENDED: N/A 
    Created 0 volumes. 
    There are 3 replica volumes whose snapshots will be restored on next replication.

    A session for each volume involved in the repair operation is displayed.
    Periodically issue the $ vsnap repair session show command to ensure that the amount of data being sent for each volume is increasing in increments. As the sessions finish you will see the status change to COMPLETED. When all the sessions finish, issue the $ vsnap repair session show command to verify that the overall status is COMPLETED. A final message indicating the number of volumes for which snapshots were restored is displayed. The message output is similar to the following example:

    Created 0 volumes. 
    There are 3 primary volumes that have recoverable snapshots, the latest snapshot of each will be restored. 
    Restored 3 snapshots.

14. For any snapshots that are not restored and that indicate a FAILED status, resubmit the repair process by entering the following command:

    $ vsnap repair create --async –-retry 

15. When the repair process reports a COMPLETED status, you can resume normal operations for the vSnap server by moving it out of maintenance mode. To resume normal processing, enter the following command:

    $ vsnap system maintenance complete

16. Remove saved SSH host keys from the repaired source vSnap server and the target vSnap servers.

    Run the following commands on both the source and target vSnap servers:

    $ sudo rm -f /home/vsnap/.ssh/known_hosts

    $ sudo rm -f /root/.ssh/known_hosts

    Removing the SSH keys ensures that subsequent replication transfers do not produce errors that result from the changed host key of the repaired vSnap server.

17. Restart the vSnap service on the replaced server by entering the following command:

    $ sudo systemctl restart vsnap

18. Click System Configuration > Backup Storage > Disk to verify that the new vSnap server is correctly registered, as follows:
    • If the new vSnap server is using the same host name or IP address for registration, no change is required.
    • If the new vSnap server is using a different host name or IP address for registration, you must update the registration by selecting the pencil icon.
19. To remove recovery points that are no longer available on the source vSnap server, start a maintenance job from the IBM Spectrum Protect Plus user interface.
    For instructions, see Creating jobs and job schedules.

    Tip: You might see informational messages that are similar to the following example:

    CTGGA1843 storage snapshot spp_1004_2102_2_16de41fcbc3 not found on live Storage2101 Snapshot Type vsnap

20. To resume jobs that failed after the vSnap server became unavailable, run a storage server inventory job. For instructions, see Creating jobs and job schedules.