Media Server De-Duplication Pools
- 1 Setup Disk
- 2 Configure MSDP storage server
- 3 Create Disk Pool
- 4 Create Storage Unit
- 5 MSDP Maintenance
- 6 MSDP Replication
- 7 Deleting an MSDP
This document is for entertainment purposes only. Use at your own risk. This site is in no way affiliated with Symantec or NetBackup.
For more details see:
- In this example we will be using SAN disk.
- This configuration is to create a NetBackup Media Server De-duplication Pool (a standalone Pure Disk implementation will not be used).
- For 3rd party Storage Servers see: Storage Servers in NetBackup.
- Advanced Disk is very similar, except that sharing between Media servers must be done differently.
For details see: http://www.symantec.com/docs/TECH68582
The first step is to setup the disk, present it to the NetBackup server, and format it with a filesystem, and give it a mount point or drive letter. Make sure it is accessible to the OS.
You can also use high speed disk (i.e. SSD) for the database, and less extensive disk (i.e. Fiber or SATA) for the data.
You shouldn't need more then 1TB of disk for 32-64 TB of disk, depending on your data-set.
Configure MSDP storage server
For more details about MSDP Storage Servers see the NetBackup Deduplication Administration documentation.
- Start the NetBackup Administration Console
- In the left-hand column highlight Media and Device Management'.
- On the right side, click on Configure Disk Storage Servers.
- Click Next on the "Welcome" screen.
- For the Type of disk storage to configure select Media Server De-duplication Pool.
- For the Media Server, I would suggest setting it to be the same as the Storage Server Name unless you are doing something fancy.
You'll need to provide a Username and Password to be used for this storage server. Depending on your security requirements, it may be easiest to use credentials that are consistent throughout the environment.
Once all fields are completed, click on Next.
- Select the path to the filesystem for the disk to be used.
Here you can also select a different path for the Database (higher speed, smaller size filesystem), and to force it to use a specific network interface. If you have a complex network configuration or tight security requirements make sure you put the interface in here that data will have to be sent over.
If you leave it blank, it will use the interface that matches the name of the server according to the OS.
Once all selections are made click on Next.
- Select any Media Servers that should assist in helping with the workload. What you choose will depend on your architectural designs and expected load. It's not a bad idea to select all available Media servers so that if any of them are idle they can help out, but this can also complicate the environment.
- Review the Summary, and if everything looks correct click on Next. If any corrections need to be made click on Back.
- Monitor the Storage Server Creation for any errors, and correct if needed.
Click Next When completed.
- Click Next again to continue on the the Disk Pool creation, or Close if you want to do it later.
NOTE: if you get an error while trying to create a storage server on a media server that previously had a storage server configured for it you may not have properly removed all the leftover ost-plugin config files
Create Disk Pool
- Disk Pool creation is very similar to what you'll find in the Creating Disk Pools section of the Storage Servers page.
- Within the NetBackup Administration Console
- In the left hand column highlight Media and Device Management'.
- On the right side, click on Configure Disk Pool.
- Click Next on the "Welcome" screen.
- For the Type of disk select PureDisk.
- Select the Storage Server the Disk Pool is to reside on.
- Select the appropriate Volume Name. Usually this will default to PureDiskVolume.
- Provide a Disk Pool name following pre-established Naming Standards.
Once all parameters are entered click on Next.
- Review the summary. If everything looks correct click on Next, or Backup to make corrections.
- Monitor the Disk Pool creation for any errors. Correct as needed.
Click Next when complete.
- If you would like to create a Storage Unit to use the new Disk Pool, click Next, otherwise click Close.
Expanding A Disk Pool
Sometimes you need more space. There are two ways to increase the space for an MSDP. In either case, be sure you do not compromise performance requirements.
- Grow the filesystem according to the OS best practices
- This is not recommended for anything larger than 64TB.
- Add another filesystem to the disk pool
- This is recommended not to be more than 32TB per file system, and no more than 96 TB total
- Each filesystem should have sufficient performance when all three are being accessed simultaneously
- It is also recommended to have the MSDP database on its own high-performance disk, using a sperate filesystem as well.
- On the Media Server containing the MSDP, create the new file system as needed
- Make a touch file to basically say you know what you are doing and accept any potentially negative consequences.
- touch /etc/nbapp-release
- Create data directories within the filesystem
- mkdir /path/to/filesystem2/data
- Run command to add new filesystem to the diskpool
- /usr/openv/pdee/pdcr/bin/crcontrol --dsaddpartition /path/to/filesystem2/data
Create Storage Unit
Storage Unit Configuration can be done via the command line as well.
- From within the NetBackup Administration Console, expand NetBackup Management
- Expand Storage, then highlight Straoge Units
- Right click on Storage Units and then click on New Storage Unit...
- Provide a Storage Unit Name following creating/naming Storage Units|Naming Standards.
- The Storage unit type should be Disk.
- The Disk type is PureDisk.
- Select the correct 'Disk Pool previously created.
- Select the Media Server(s) to use for the Disk Pool.
- Select a maximum number of streams/concurrent jobs to allow.
- The default Maximum fragment size is usually sufficient.
- Click on "OK"
If you have a STU that is already configured the way you need, but with a few changes, you can also right click on that STU and select Copy. This will help you to quickly create multiple STU's for the same Disk Pool, but for different media servers to use, and then you can create groups that include all of those STUs to use for load-balancing or fail-over.
Force Cleanup of used space: http://www.symantec.com/docs/TECH124914
Migrate or rebuild a failed MSDP: http://www.symantec.com/docs/HOWTO89029
The NetBackup Admin GUI provides a number of different sections that can help you monitor and maintain your MSDP's.
- Storage Servers: Device Management -> Credentials -> Storage Servers
- Disk Pools: Device Management -> Devices -> Disk Pools
Some commands are also useful:
For every storage target, the NetBackup master server will periodically request the media server to check that the target is still up.
If the Media server or target are busy and cannot respond to the master servers inquiry fast enough, it may cause the storage target to be marked as "down".
This will not cause currently running backup jobs to fail, but could cause any new jobs to either fail or failover to a secondary target. To adjust the timeouts, on each media server, and/or MSDP server create the following touch files o adjust the timeouts:
touch /usr/openv/netbackup/db/config/DPS_PROXYNOEXPIRE echo 3600 > /usr/openv/netbackup/db/config/DPS_PROXYDEFAULTSENDTMO echo 3600 > /usr/openv/netbackup/db/config/DPS_PROXYDEFAULTRECVTMO
This will change the timeout to 3600 seconds, or 1 hour. The downside to this setting, is that you may not know a backup target is down for the set period of time (one hour in this example) after it actually goes off-line.
For more details see: http://www.symantec.com/docs/HOWTO88988
Optimized duplication is the ability to replicate backup images from one de-duplicated storage server (either MSDP, OST or Pure Disk based) to another de-duplicated storage server without re-hydrating the data. Basically only sending the missing blocks of data over the network to/from the remote or secondary de-duplicated storage.
The duplication can be performed via the use of any of these methods:
- Storage Lifecycle Policies
- Using the bpduplicate command
- From within the "Catalog" section of the GUI
- The source and target must be compatible de-duplication pools (basically from the same vendor and comparable versions).
- The source and target must have at least one common media sever that can access both.
- The target/destination storage unit must use the common media server(s).
- The target/destination storage unit cannot be the same as the source.
NOTE: See the de-duplication storage vendors documentation for a full list of capabilities and limitations
- Optimized duplication jobs are only tried the configured number of times before failing (unless using SLPs).
- MSDP optimized duplication does not work to storage unit groups. If a group is used, non-optimized duplication is used instead.
- You cannot create multiple copies in parallel, each copy must be a separate duplication job.
- maximum Fragment size is based on the source Storage Unit, not the target: Image fragments are copied as is and not re-sized.
- Both Push and Pull de-duplication can be performed, depending on which Media Server controls the duplication.
- A Push will allow the source Media Server to maintain ownership of both copies of the Image.
- A Pull will allow the target Media Server to gain ownership of the newly created copy.
- The Media Server specified in the target Storage Unit determines which media server controls the duplication and hows the newly created copy.
Auto Image Replication
A.I.R. or Auto Image Replication, is the method used within NetBackup to replicate backup images from one domain (or master server + associated media severs) to another.
For more details see: http://www.symantec.com/docs/HOWTO88954
A.I.R. can be done in a one-to-one, one-to-many, or many-to-one model. It's primary function is to provide Disaster Recovery capabilities when one site goes down, and needs to be restored at another site.
- The source and target domains must be comparable version of NetBackup (I.E. post 7.1).
- Clocks on masters in both domains must be synchronized for optimal performance (time zones are not a factor).
- Backups must be performed with a properly configured Storage Lifecycle Policy (SLP).
- An SLP of the same name must exist on the source and destination domain.
- The source and destination storage must be compatible Storage Server with de-duplication pools.
- The Storage Server in the target domain must support A.I.R.
- Clients in the primary domain cannot directly restore from the target domain.
- Synthetic Backups are not supported.
- Storage Unit groups cannot be used for the source copy.
- Catalog backup images can only be done between domains of the same version.
You must first configure the MSDP as a replication target
- From the Admin GUI go to: Media and Device Management -> Credentials -> Storage Server.
- Select the storage server.
- Right click and select Change.
- Select the "Replication" tab
- Click on the "Add" button
- Enter the Target Master Server (or select the optional trusted master server)
- Enter the Target Storage Server.
- Provide the target MSDP servers credentials that were configured when the MSDP was created.
- Click on OK when complete.
- Back on the Storage Server properties, Click on the OK button, after all replication targets are added.
- Now you can Setup the SLP
Deleting an MSDP
- For OST information also see: Storage_Servers_in_NetBackup#Delete_OST_Storage_Server_from_NetBackup
- Also see: http://www.symantec.com/docs/TECH150431
- Check that you have a good recent catalog backup.
- Make sure no policies are configured to use the MSDP: check for Storage Units, SLPs, Schedule's that re-direct to a different target, etc.
- Check under NetBackup management -> Storage and Policies.
- Don't forget to check Schedule's configured to override the Policies storage target.
- Expire any Backup images still on the Disk Pool. Preferably after they have been duplicated to another location or reached the end of their retention period.
- From the Administration Console you can use the Catalog section found under NetBackup Management.
- Also check the nbstlutil for any images controlled by SLPs.
If any images are controlled by SLPs you can do the following:
- Wait for the image to get processed by the SLP (usually the best option).
- Use the nbstlutil cancel -backupid <Image_ID> to discontinue the SLP processing
- Use the bpexpdate command with the -force_not_complete option; and optionally the -notimmediate if you don't want to wait for a lot of images to get deleted.
bpexpdate -stype PureDisk -dp <DiskPoolName> -dv <DiskVolume> -force_not_complete -notimmediate
- Let NetBackup run an image cleanup, or force it with:
bpimage -cleanup -allclients
- Delete any SLPs that use the MSDP
- Check under NetBackup management -> Storage and Storage Lifecycle Policies.
Optionally you can modify the SLP, but make sure no previous versions of the SLP refer to the MSDP:
nbstl -L -all_versions
- Check under NetBackup management -> Storage and Storage Lifecycle Policies.
- Delete all associated Storage Units
- Clean up images using the nbdelete and bpimage commands:
nbdelete -allvolumes -force
bpimage -cleanup -allclients
- Set the Disk pool to a down state using nbdevconfig command from the Media Server the MSDP resides on:
nbdevconfig -changestate -stype PureDisk -dp <DiskPoolName> -dv PureDiskVolume -state DOWN
- Run nbemmcmd -listhosts and note the NDMP machinetype entry for the storage server the MSDP pool resides on.
- Delete the Disk Pool
- Locate it under Media and Device Management -> Devices -> Disk Pools
- or run nbdevconfig -deletedp -stype PureDisk -dp <disk_pool_name>.
- Delete the server credentials and the storage server NDMP machine type from the EMM Database (Run this commands from the Media Server the MSDP resides on):
tpconfig -delete -straoge_server <MSDP_Straoge_Server_Name> -stype PureDisk -sts_user_id <userid>
- Delete the Storage Server
- Locate it under Media and Device Management -> Credentials -> Storage Servers
- Or run the command nbdevconfig nbdevconfig -deletests -storage_server <Storage_Server_Name> -stype PureDisk
- Remove the reference from the EMM database:
nbemmcmd -deletehost -machinename <Stroage_Server_Name> -machinetype ndmpNote: If you get an error from this command you can try running this command first:
nbemmcmd -deletealldevices -machinename <Stroage_Server_Name> -machinetype ndmpUse Causing with this command as it will remove everything associated with this Storage Server.
- Stop NetBackup on the Server housing the MSDP pool.
- Run the PDDE_deleteConfig script on the Server in question:
- Remove any *.cfg files referring to the MSDP server under /usr/openv/lib/ost-plugins (Unix) or \NetBackup\bin\ost-plugins (Windows) on all servers that were configured to use the MSDP.
- Remove the MSDP Storage and Database directories.
- Start NetBackup on the server it was stopped on.
Back to NetBackup