Storage Servers in NetBackup

From World History Wiki
Jump to: navigation, search
World History Wiki is Brought to you by:
S.J.'s Adventures


Contents

These documents are for entertainment purposes only. Use at your own risk. This site is in no way affiliated with Veritas or NetBackup, and any trademarks are those of their respective companies.

If you need assistance with anything NetBackup and/or Disaster Recovery related, You can find me on linkedIn.

For configuring MSDP see Media Server De-Duplication Pools

Some information on this page may be out of date.

OST On De-Dup Systems

Also see: OST Configuration Standards

For all Storage Server/De-duplication/OST target devices, follow the hardware installation and setup instructions before starting these steps specific to NetBackup Open Storage configurations.

DataDomain Setup

Also see: DDR Setup and Configuration

nfs status
filesys status
user add ostusr password ostUsrPW priv admin
ost set user-name ostusr
ost enable
ost lsu create

Data Domain LSUs

NOTE: The "Storage Server" is the DNS name of the Data Domain's primary network interface used for backups, and does not need to be configured on the device. If multiple DNS names are associated with different NICs, the Data Domain can be associated with multiple "Storage Servers". The OST Plug-in has the ability to handle load balancing between different network interfaces so no bonding is necessary.

  • LSUs on the Data Domain system correspond to disk pools on the NetBackup media server.
  • You can create or delete LSUs only through the Data Domain command line interface (CLI).
  • Each LSU is a top-level subdirectory of the /backup/ost directory.
  • There is no hierarchy among LSUs.
  • Important For this release, there must be one and only one LSU per Data Domain system.
ost status
ost lsu create [LSU-NAME]

License Key

license add AAAA-BBBB-CCCC-DDDD

Quantum Setup

  1. Login to the web-console and select the OST section under "Configuration".
  2. Create a storage server and LSU as needed.
    The "Storage Server" on Quantum is just a virtual name with no relationship to an actual IP or DNS name.
    Quantum relies on IP address settings to determine the data path.
  3. Create an OST user account and password
  • LSUs on Quantum systems are setup with a virtual disk size, or can be setup to use all available space.
  • They are created and deleted via the Web GUI

Sepaton Setup

  1. Login to the web-console and click on the "System" tab.
  2. Highlight and Expand the "OST" folder on the left.
  3. Click on the "Create Storage Server" link on the "Tasks:" bar.
    1. You'll be asked to enter the "Server Name", "User Name" of the OST User, "Password" for the OST User, and optionally if you want to enable "OST over Fiber channel".
      NOTE: The "Server Name" should match the DNS name associated with the IP address to be used for sending backup data. Data replication/Duplication should also be a consideration of the overall design.
    2. Click "Next Step"
    3. Now you get to choose which "SRE Node"s you want to use with OST.
    4. After making your selections, Click on "Create OST Storage Server"
    5. You be shown a summary, and be able to click on "Add Disk Volume(s)"
  4. You can access the "Add Disk Volume" in the "Tasks" bar from the OST folder under "System" by highlighting the "Storage Server"
    1. Enter the "Name" of the "Disk Volume" and select the options required for this volume.
    2. Click on "Add Disk Volume"
    3. When finished Click on "Done", or you can also click on "Add Another Disk Volume"

Also see: Example Sepaton with OST Configuration.

Replication Target

If you plan to use SLP's and/or A.I.R. to replicate the data, you'll need to repeat these steps on the target system. Depending on the device, additional authorizations or configurations may be required.

Sepaton also requires setting up Disk Volume replication pairs.

Plug-in install On Media Server

The Plug-in Install Must be done as root.

Library Installation

NetBackup Remote Manager and Monitor Service (nbrmms)

  • Discovers and monitors disk storage on NetBackup media servers.
  • Discovers, monitors, and manages Fibre Transport (FT) connections on media servers and clients for the NetBackup SAN Client option.
  • Runs on NetBackup media servers.

DataDomain requires you stop the nbrmms daemon. Sepaton and Quantum makes no mention of this daemon, but instead has you reboot the Media Server if Solaris, or re-start NetBackup on Linux (Linux also work with just terminating nbrmms, but Solaris has other configurations that are done that require a reboot). Testing has show that simply re-starting the nbrmms daemon, after the OST install is done, works for both installations.

/usr/openv/netbackup/bin/nbrmms -terminate



DataDomain Plug-in Install

Find the appropriate OST plug-in version for the OS you are to install it on (note: NetBackup 6.5.x requires 32-bit and 7.x requires 64-bit).

Example:

cd /XXXXXXX/OST/DD/OST_1.1.4.2-93601_OST_solaris_sparc-w
install.sh [ -d directorypath ] [-h]
 Def path: /usr/openv/lib/ost-plugins
 Libs: libstspiDataDomain.so and libstspiDataDomainMT.so



Quantum Plug-in Install

You can download the OST plugin from within the DXi interface from under Configuration, OST, OST Client Plug-In, Client Plug-in Download. You will be redirected to another page and you need to select the appropriate Plugin binaries. Once downloaded copy to the server and follow the following steps. These steps can also be downloaded from same location on the DXi.

Steps to install the plug-in:

  1. Copy the OST plug-in's tarball into /usr/openv/lib/ost-plugins/
    (QuantumOSTPluginsolaris_sparc.32.tar for Solaris SPARC 32-bit, QuantumOSTPluginsolaris_x86.64.tar for Solaris x86 64-bit, or QuantumOSTPluginlinuxR_x86.32.tar for RedHat Linux x86 32bit).
  2. cd /usr/openv/lib/ost-plugins/
  3. Untar the files using '/bin/tar xvf QuantumOSTPluginsolaris_<arch>.<bit>.tar'
    (i.e. '/bin/tar xvf QuantumOSTPluginsolaris_sparc.32.tar' for Solaris SPARC 32-bit)
  4. Move configuration file using /bin/mkdir -p /usr/Quantum; /bin/mv QuantumPlugin.conf /usr/Quantum
  5. For Solaris Only - Create a script in /etc/init.d called tcp_options and put the following lines to it:
#!/bin/sh

# adjust the TCP buffer size so 'setsockopt(...)' from the plugin works properly

/usr/sbin/ndd -set /dev/tcp tcp_max_buf 6291456
 
# These are the values that we used for Solaris 10 tcp options to get maximum performance
# Your environment might need different tcp options in order to obtain maximum performance
# Please consult your Solaris 10 administrator manual about how to set these options for your environment
/usr/sbin/ndd -set /dev/tcp tcp_xmit_hiwat 4194304
/usr/sbin/ndd -set /dev/tcp tcp_recv_hiwat 4194304
/usr/sbin/ndd -set /dev/tcp tcp_wscale_always 1
/usr/sbin/ndd -set /dev/tcp tcp_tstamp_if_wscale 1
Make sure your script is executable:
chmod a+x /etc/init.d/tcp_options
Link the startup script from the appropriate directory for your runlevel so the settings are persistent across reboots.
In our case we used:
ln -s /etc/init.d/tcp_options /etc/rc3.d/S20tcp_options
Reboot the server.
6. For RedHat Only, Quantum recommends you tune OS memory usage to improve performance
vi /etc/sysctl.conf
Add the following lines:
# Quantum DXi recommendations for OST
net.ipv4.tcp_rmem = 8388608 8388608 8388608
net.ipv4.tcp_wmem = 8388608 8388608 8388608
net.ipv4.tcp_mem = 10485760 10485760 10485760
net.core.netdev_max_backlog = 30000
Run following command to make changes active
/sbin/sysctl -p
Restart NetBackup to make sure all processes take the new settings.



Sepaton Plug-in Install

  1. The OST plug-in can be downloaded from the web-GUI of the Sepaton system found under "OST
    1. click on the "System" tab.
    2. Expand the directory tree on the left.
    3. Expand the "OST" folder
    4. Click on "OST Plug-in Downloads"
    5. Click on the appropriate OST Plug-in for the target Operating System.
    6. Save the plug-in to a temporary folder on the target system to be installed on.
  2. Log-in to the target system as "root" or as an Administrator.
  3. From the temporary folder run:
    Linux:   sh -i SEPATON-OST-7.1.0-nnnnn.x86_64.rpm.bin
    Windows: Click on the OST Plug-in install file.
    Both will have you accept a licensing agreement, and verify the install location.



Start nbrmms

If nbrmms was stopped for DataDomain (or Quantum on Linux), it now needs to be re-started.

Quantum suggests a complete re-start of NetBackup.

/usr/openv/netbackup/bin/nbrmms

If you rebooted, or did a full stop and start of the services/daemons verify that all NetBackup processes started normally.

verify nbrmms

bpps -a|grep nbrmms

verify that the plug-in is detected by NetBackup

sudo bpstsinfo -plugininfo

Authorizing NetBackup Media Servers

Register Storage Server


Everything from this point on can be done using the Storage Server wizard in the Admin Console; however this is the old-school command-line way of doing it:


For an example of doing the configuration via the Admin console see: Example Sepaton with OST Configuration


Run this command on the NetBackup Master/Media server you want to authorize: (only needed for the first media server added - for additional media servers skip to "Add Credentials")

Data Domain Recommends Running:

sudo nbdevconfig -creatests -storage_server [OST-Server] -stype DataDomain -media_server [NBU-Server]

Quantum Recommends Running:

sudo nbdevconfig -creatests -storage_server [OST-Server]_IP.Of.System -stype Quantum -media_server [NBU-Server] -st 9

Sepaton Recomends Running:

Sepaton refers you to NetBackup documentation for all NetBackup related configuration tasks.
Using the Data Domain example above should work.

[OST_Server] = For DataDomain this is the DNS name of the OST server's network interface you wish to send backup data to.
For Quantum this is configured within their GUI.

[NBU-Server] = the NetBackup Media server name you want to authorize to use the OST Server

-st 9 is for Storage type of Disk that is Network attached (1 + 8).

If the command fails look for details by running:

vxlogview -n 1 -i 222 -t 00:10:00 | grep [STYPE]

NOTE: if the command fails for an already existing OST server, it may get removed from NetBackup's configuration.

Add Credentials

DataDomain OST user and password

On DataDomain the OST user is created during setup

Quantum OST user and password

Quantum also has an OST user that is setup on the DXi system itself, accessible via the GUI.

Sepaton OST user and password

Sepaton also has an OST user that is setup on the Sepaton system when the Storage Server is created.

tpconfig command

Run the following on each media server that will be using the Storage Server:

sudo tpconfig -add -storage_server [OST-Server] -stype [PLUG-IN] -sts_user_id [OST-USER] -password [OST-USER-PASSWORD]

The "[OST-Server]" Must match the same name as used previously with the "nbdevconfig" command.
For Quantum this means using "[OST-Server]_IP.Of.System"

In case of Failure

Look for authentication messages in the ddfs.info file on the Data Domain system.

You can also call DataDomain/EMC Support or go to:
https://my.datadomain.com

For Quantum we don't have any information yet on how to troubleshoot

You can also call Quantum Support

For Sepaton You'll want to make sure of their support portal:

https://deltaview.sepaton.com
You can also call them.

Restart the nbrmms process

Restarting the nbrmms command may be helpful to re-set the OST plugins if needed.

sudo bpps -a|grep nbrmms
sudo /usr/openv/netbackup/bin/nbrmms -terminate
sudo bpps -a|grep nbrmms
sudo /usr/openv/netbackup/bin/nbrmms

Storage Server Supported Features

To list the capabilities of a particular Storage Server run this command:

nbdevquery -liststs -U -stype [PLUG-IN] -storage_server [OST_Server]

If additional licensed features are added after the Storage Server is configured, you'll need make NetBackup aware of the new features with this command:

nbdevconfig -updatests -storage_server [OST_Server] -stype [PLUG-IN]

Existing Disk Pools will also need to be updated. First obtain the Disk Pool Name:

nbdevconfig -listdp -stype [PLUG-IN]

Then Update each Disk Pool:

nbdevconfig -changedp -dp [DISKPOOL] -stype [PLUG-IN] -setattribute [FEATURE]

Were FEATURE is something like "OptimizedImage", or what ever new "Flag" (feature) shows up when running the first command to list the capabilities of the Storage Server.

Create Disk Pools


The following steps can also be done via the GUI's Disk Pool Configuration Wizard

See: Media_Server_De-Duplication_Pools#Create_Disk_Pool

  • Disk pools are collections of disk volumes that NetBackup administers as single entities. The Disk volumes can be control by a Storage Server via OST.
  • In OST, disk pools/volumes correspond to Logical Storage Units (LSUs) that are configured on their respective Storage Server.
  • These steps only need to be done once even if multiple Media Servers will use the same Disk Pool

Obtain identity of the LSU

sudo nbdevconfig -previewdv -storage_server [OST_Server] -stype [PLUG-IN] > /tmp/dvlist

Remeamber the "[OST-Server]" is the same name as used previously with the "nbdevconfig" and "tpconfig" commands.
For Quantum this means using "[OST-Server]_IP.Of.System"

In this example we see the DataDomain Lab12 LSU with roughly 16 TB of space:

cat /tmp/dvlist
V6.5 DiskVolume < "lablsu1" "lablsu1" 17389719650304 16664858984448 0 0 0 >

Create Disk Pool

sudo nbdevconfig -createdp -dp [DISK_POOL_NAME] -stype [PLUG-IN] -storage_servers [OST_Server] -dvlist /tmp/dvlist

NOTE: The "DISK_POOL_NAME" should be a combination of the storage server name and LSU name, such that it is easy to identify which system(s) data written to the Disk Pool physically and logically resides on as well as the network path taken to reach the storage server

You should see a response similar to this:

Disk pool nas1bkup-lsu01 has been successfully created with 1 volumes

Create Storage Units

This step can also be done via the GUI.

sudo bpstuadd -label [STU-NAME] -dp [DISK-POOL] -host [MediaServer] -M [MasterServer]

Be sure to follow standards on creating/naming Storage Units and OST standards.

Tuning OST for NetBackup

Setting the Number and Size of Buffers

For more information, see Tuning_and_Testing_for_NetBackup

For SIZE_DATA_BUFFERS_DISK file:

The maximum value supported by the Data Domain plug-in is 1 MB.

Quantum recommends using 1 MB for best overall performance.

Sepaton has no recommendations.

For NUMBER_DATA_BUFFERS_DISK

This setting depends on how much memory is available on the server. Typical settings are either 32, 64, or 128. Only extensive testing can prove what setting(s) works best.

Storage LifeCycle Policy Settings

See Understanding NetBackup Storage Lifecycle Policies for more information.

Storage Lifecycles enable NetBackup to manage the replication between De-duplication appliances, to enable a number of different functionaries. This includes things such as:

  • AIR (Automatic Image Replication)
  • Maintaining different retentions on different replication targets
  • Controlling tape creation at a remote site.
  • etc.

Timeouts

On heavily loaded environments, a Storage Server may be marked down if it does not respond to queries fast enough.

To extend timeout periods see Media_Server_De-Duplication_Pools#Timeouts

Adding additional Media Servers

'If OST is already setup and working on other Media/Master Server(s) in the environment, you can follow this simplified set of steps to configured additional Media Servers:

  1. Install the Plug-in on the new Media Server.
    Run/Copy from /admin/images/... or scp to /admin/nbu/tmp/.
  2. Validate the Plug-in was installed correctly.
  3. Add Credentials for the new Media Server to access the Storage Server.
    1. Validate the new sever shows up under the Storage Server and the correct Disk Pool (may require a refresh of the java console).
    2. Run a nbdevconfig -previewdv ... command to validate communication is working.
  4. Create new Storage Unit(s) as needed.
    Be sure to follow standards on creating/naming Storage Units

Storage Server Maintenance

DataDomain

LSU Related Commands

  • Creating an LSU
 ost lsu create lsu-name
  • Deleting an LSU
 ost lsu delete lsu66
  • Deleting all LSUs
 ost destroy
  • Displaying LSU Names
 ost lsu show [compression] [lsu-name]
 ost lsu show compression
 ost lsu show LSU_NBU1
  • OST Statistics Commands
 ost show stats
 ost show stats interval 1
  • Viewing OST Operation Latencies
 ost show histogram
  • Clearing OST Statistics
 ost reset stats
  • Displaying Statistics for Active Optimized Duplication
 ost show image-duplication active
  • Viewing OST Connections
 ost show connections
  • List of Credentials
 sudo tpconfig -dsh -all_hosts -stype DataDomain

Quantum

From within the web-interface, under "Configuration" select the OST configuration. The interface is fairly intuitive.

Sepaton

Login to the web-console and click on the "System" tab. Highlight and Expand the "OST" folder on the left. In this area you'll find a few different sub-folders each with it's own "Tasks" bar for updating or adding various configurations.

Change OST User Passwords

Data Domain

  • For DataDomain the OST user and associated password is setup on the system with these commands:
user add ostusr password [PASSWD] priv admin
ost set user-name ostusr
sysadmin@nas1# user change password ostusr
        Enter new password: 
        Re-enter new password: 
Passwords matched.
Password changed for user "ostusr".

Quantum

From within the web-interface, under "Configuration" select the OST configuration, then "Manage Users" to add, delete or change OST users.

Sepaton

Login to the web-console and click on the "System" tab. Highlight and Expand the "OST" folder on the left. In this area you'll find a few different sub-folders each with it's own "Tasks" bar for updating or adding various configurations.

Update Media Servers

Any time the OST user password is updated, you must update the Media Servers as well:

sudo tpconfig -update -storage_server [OST_Server] -stype [PLUG-IN] -sts_user_id [OST-USER] -password [OST-USER-PASSWORD]
  • If you change the OST user name or password on the OST Storage Server you must run this command to update the credentials

General

  • Keep a close eye on disk usage
  • Make sure replication stays up to date
  • Treat it like a slow moving Barge carrying lots of precious cargo that requires many parallel jobs to be loaded or unloaded quickly, and needs a wide turning radius.

Delete OST Storage Server from NetBackup

You may also want to see how the command nbdecommission handles OST related images on a media server being decommissioned.

Also see: Media_Server_De-Duplication_Pools#Deleting_a_MSDP

If an OST Storage Server is not decommissioned correctly, it will become very difficult to remove it from the NetBackup configuration. Here are the steps that need to be followed:

Expire all backup images stored on the Storage Server

It would be a good idea to make sure you have a copy of the images in a different location if needed.

Search the catalog for the specific image copy number and expire only that copy of the image.

Remove all associated Storage Units from within NetBackup

This can be done from the GUI or with this command:

bpstudel -label storage_unit_name

Remove the authorizations to the Storage Server

tpconfig -delete -storage_server <StorageServerName> -stype <PluginName> -sts_user_id ostusr

Remove the associated Disk Pool

This can be done from the GUI or with this command:

nbdevconfig -deletedp -stype <PluginName> -dp <DiskPoolName>

Remove the Storage Server

nbdevconfig -deletests -storage_server <StorageServerName> -stype <PluginName>

Errors Removing OST Devices

If something wasn't done correctly you might run into one of these issues. Keep in mind it's always best to decommission the Storage Server using the appropriate steps in the proper order; which can require weeks or even months of planning and processing. Take careful detail oriented steps when dealing with the errors bellow to insure you do not cause Data Loss.

Error removing Storage Server

DSM has encountered the following busy resource: [StorageServer]

This indicate that NetBackup still thinks it has a Disk Pool associated with the Storage Server. Correcting this problem my require re-authorizing the Storage Server and following the removal steps. If the system is no longer available, You might try tricking DNS to think it's pointing at the same server when it's actually pointing to a different Storage Server.

Error removing Disk Pool

DSM has found that one or more volumes in the disk pool [DiskPool](PlugInName)@[NBUserver] has image fragments

This occurs if not all backups have been expired AND the image cleanup process hasn't yet deleted all data from disk. Search the "Images on Disk" report to see if you can find any images needing to first be expired, and then run a Image Cleanup process. If you cannot find any images on disk run this command to look for fragments:

nbstlutil list [-U]

For each image fragment found to reside on in the Disk Pool run (this may need to be run from the NBU server having the issue):

/usr/openv/netbackup/bin/admincmd/bpexpdate -backupid <image fragment name> -d 0 -force
/usr/openv/netbackup/bin/admincmd/bpexpdate -backupid <image fragment name> -d 0 -force -nodelete
/usr/openv/netbackup/bin/admincmd/nbdelete -allvolumes -force
/usr/openv/netbackup/bin/admincmd/nbdelete -backup_id <image fragment name> -force
/usr/openv/netbackup/bin/admincmd/bpimage -cleanup -allclients

Check Activity Monitor to make sure the Image Cleanup finishes without any errors

Be Careful when running the above commands that you do not delete any important data

After removing all image fragments, the storage pool deletion should now work. The ability to remove the fragments may require the Storage Server be re-authorized using the "tpconfig" command if it was previously de-authorized, and the LSU recreated if it was deleted.

It is also important to note that Storage Life Cycle Policies have been known to create problems in this area that may make removing the Disk Pool difficult.

These additional commands may also help with removing stuck SLP image relationships:

nbstlutil remove_all -mediaid @aaaaW -force (may only work with 7.1 and earlier)
nbstlutil cancel -backupid <Backup_ID> (get ID's from nbstlutil list)
bpexpdate -stype <OST-Storage-Type> -dp <DiskPool>



Back to NetBackup