Tuning and Testing for NetBackup

From World History Wiki
Jump to: navigation, search
World History Wiki is Brought to you by:
S.J.'s Adventures


These documents are for entertainment purposes only. Use at your own risk. This site is in no way affiliated with Veritas or NetBackup, and any trademarks are those of their respective companies.

If you need assistance with anything NetBackup and/or Disaster Recovery related, You can find me on linkedIn.

A properly tuned NetBackup environment will better meet your particular needs. Exact tuning and configurations cannot be known without some testing and evaluation of the size and scope of the environment, but here are some general ideas that may help determine what may work for your particular environment.

Tuning Steps

Create and Modify Volume Pools Configuration

  1. From within the NetBackup Admin Console, expand:
    1. Media and Device Management
    2. Media
    3. Highlight Volume Pools
  2. Add/Modify the Volume Pools to meet necessary requirements and naming standards. For Example:
    VolumePools.JPG
  3. If using tape encryption, be sure to configure the necessary pools starting with the “ENCR_” prefix, and follow up by configuring a key management group with the same name.
  4. The fewer pools, and retention levels you actively use, the more efficient your tape usage will be.
  5. The main tuning to be done for each Volume Pool is how many partially full tapes to allow. This will help maximize tape usage, but you should set it too low or it will compromise throughput.

Create Barcode and Media Management Rules

Also see Volume Naming and Groups for NetBackup

NOTE: For this portion to work properly, at least one Robot must already be configured

  1. Highlight Robots
  2. Right click on Robots, and select “Inventory Robot”.
    RobotInventory.jpg
  3. On the “Robot Inventory” window, click on “Update volume configuration” to make it the selected option so the “Advanced Options…” button it activated.
  4. Click on the “Advanced Options…” button.
  5. In the “Advanced Robot Inventory Options” window, click on the “Barcode Rules” tab
    BarCodeRules.jpg
  6. Add Barcode Rules as needed for the bar codes being used, and their intended purposes, and pre-defined in the Naming Standards.
    Also insure the correct Media Type (density) is configured to match the tape drives they will be used with.
  7. If using “Extended Barcodes” within the tape library configuration, you’ll also need to configure “Media ID Generation” rules for each robot so that the correct 6 characters are used for the Media ID from the barcode label.
    NOTE: this can also be done in the “/usr/openv/volmgr/vm.conf” file on the media server controlling the robot (File may need to be created) by adding a line similar to:
    MEDIA_ID_BARCODE_CHARS = 0 8 1:2:3:4:5:6
    Also add a line to the “vm.conf” file to automatically import tapes placed in the I/O (or MAP/CAP) slots:
    AUTO_UPDATE_ROBOT
    Also see Volume_Naming_and_Groups_for_NetBackup#Tape_Library_Automatic_Inventory.
  8. Once all necessary configurations are made, click on “OK”.
  9. When back on the “Robot Inventory” click “Close”
    NOTE: If needed, delete all existing scratch and other un-assigned media, and then inventory the robot to get the new rules to apply. If any already assigned media need to be modified, you can do so using the “vmchange” command.

Modify Media Server Settings.

  1. Bring up the Media server list within the Admin console found under:
    1. NetBackup Management
    2. Host Properties
    3. Media Servers
  2. Highlight all servers in the list, right click, and select “Properties”.
  3. The resulting “Media Server Properties …” window will contain a number of sections listed on the Left hand column.
  4. All default settings under these sections are either: generally acceptable; already properly configured if the Installation was done correctly; or configured within the Master Server Host Properties.
    The two that have items to pay close attention to are:
    1. Logging
      • Enable Robust Logging
      • Keep logging at a minimum unless more higher levels (more verbose) are needed for troubleshooting.
        Logging.JPG
    2. Timeouts
      • Most settings should be increased, especially in larger more complex environments.
      • Below is an example screenshot of what they may look like (be sure to follow pre-established naming and configuration standards – all settings related to clients should also match the settings being used on the clients):
        TimeOuts.JPG
  5. Click “OK” when done

Modify Master Server Settings.

  1. Modify the Master Server Host Properties within the Admin console – found under:
    1. NetBackup Management
    2. Host Properties
    3. Media Servers
  2. The sections that have important items to consider include:
    1. Global Attributes
      • Job retry delay” should be long enough for an average server in the environment to do a complete reboot.
      • Maximum Jobs per client” should be set to a reasonable number based on what you anticipate your largest single server (or oracle server) needing to do in parallel to get its optimal backup performance.
        I usually recommend setting it to around 8 - 12. (NOTE: Doing more then 2-4 streams per drive/LUN is not recommended).
      • Schedule backup attempts” depends on how long your longest running jobs take (in general no backup should be longer than 24 hours), and three re-tries is usually more then enough to insure the problem was not temporary; meaning a setting of 3 tries in 24 hours is probably the highest settings you want to use.
      • Maximum backup copies” should be set to allow the maximum number of duplications you plan on doing for any single backup image, but not too high as to allow unnecessary or unauthorized duplications.
      • If you plan to keep monthly backup images for long term retention, you should “compress catalog images” that will be kept longer than the retention of your regular weekly full and daily incremental backups. It may also be beneficial to synchronize this with your average image retention on local disk.
      • Maximum indexing jobs should be set to prevent overloading your indexing servers. You should also only consider indexing servers that are under special legal requirements to minimize the added capacity and load added by an indexing server.
      • Maximum Vault jobs should be set only as high as needed and usually only “1” at a time is needed when vault is only used to eject tapes from a single robot.

        Here’s an example screenshot of what these settings may look like (be sure to follow pre-established naming and configuration standards for the environment).
        Global.JPG
    2. Retention Periods
      • Set these to what’s necessary to fulfill pre-determined retention requirements to meet the business’ needs.
      • It’s best to keep shorter retentions at lower retention level numbers, and higher or customized retentions at the higher retention level numbers. For example:
        Retentions.JPG
    3. Media
      • To insure the best possible utilization of tapes, be sure to “Enable unrestricted media sharing for all media servers” (unless you have strict security standards that require them to not be shared).
        Media.JPG
    4. Timeouts
      • Most settings should be increased, especially in larger more complex environments.
      • Below is an example screenshot of what they may look like (be sure to follow pre-established naming and configuration standards – all related client side settings should also match these settings were possible):
        MasterTimeouts.JPG
    5. Clean-up
      • Most settings should be modified, especially in larger more complex environments.
      • Below is an example screenshot of what they may look like (be sure to follow pre-established naming and configuration standards):
        Clean-Upr.JPG
    6. Default Job Priorities
      • Most of these can remain at their defaults, but you may want to compare these to what you would like your backup jobs to be set to (within the policy configurations), and even set a default other than zero for backup jobs and Catalog Backups.
        Priorities.JPG
    7. Log-in Banner
      • For legal requirements, you must notify all users that their actions are monitored and logged (especially if you enable Authentication Services and Auditing). This is done by configuring the Login Banner.
        C3-LoginBanner.JPG
    8. Resource Limit for VMware
      • These should be set based on the capabilities and capacities of the VMware environment. Careful testing, and coordination with your VMware admin should be done.

        Here’s an example of what they may look like:
        VMwareLimites.JPG
    9. SLP Parameters
      • These parameters can be left at their defaults, but should be tuned to meet the performance requirements of the environment.
      • In particular, if this is not a destination for A.I.R. you should set to “No” the setting “Auto create A.I.R. Import SLP”.
        SLPs.JPG

Validate and Update bp.conf

  • Log in to the Master Server, become root, and edit the bp.conf file.
[sjhollist@nbumaster ~]$ sudo su –
Password:
[root@nbumaster ~]# cd /usr/openv/netbackup/
[root@nbumaster netbackup]# vi bp.conf
  • Validate that all settings are as expected, and add any additional server entries as needed. If you have a backup network, the client name setting usually need to be modified to match the appropriate DNS name for the backup NIC.
SERVER = nbumaster-bkup.domain.com
SERVER = mediaserver1-bkup.domain.com
SERVER = opscenter-bkup.domain.com
CLIENT_NAME = nbumaster-bkup.domain.com
CONNECT_OPTIONS = localhost 1 0 2
USE_VXSS = PROHIBITED
VXSS_SERVICE_TYPE = INTEGRITYANDCONFIDENTIALITY
EMMSERVER = nbumaster-bkup.domain.com
HOST_CACHE_TTL = 3600
VXDBMS_NB_DATA = /usr/openv/db/data
OPS_CENTER_SERVER_NAME = opscenter-bkup.domain.com
SERVER_CONNECT_TIMEOUT = 120
CLIENT_CONNECT_TIMEOUT = 1800
CLIENT_READ_TIMEOUT = 3600
BPSTART_TIMEOUT = 900
BPEND_TIMEOUT = 900
LIST_FILES_TIMEOUT = 450
MPX_RESTORE_DELAY = 60
VERBOSE = -1
ENABLE_ROBUST_LOGGING = YES
MAX_VAULT_JOBS = 1
MAX_INDEXING_JOBS = 5
KEEP_VAULT_SESSIONS_DAYS = 3
INCOMPLETE_BKUP_JOB_CLEAN_INTERVAL = 8
KEEP_JOBS_HOURS=288
KEEP_JOBS_SUCCESSFUL_HOURS=82
JOB_PRIORITY = 30000 60000 90000 90000 90000 90000 85000 85000 80000 80000 80000 80000 75000 75000 70000 70000 50000 50000 45000 0 0 0 0 0
TELEMETRY_UPLOAD = NO 
  • Save the bp.conf file.

Create Touch files

  • All Master and Media servers need to have the default buffers bumped up considerably as the defaults are set to a least common denominator for all possible situations. The best settings to use can only be determined through careful testing, but as a good rule of thumb:
  1. The size of the buffers should match the capabilities or buffer sizes of the target storage devices. (I.E. 256K for LTO drives).
  2. The Number of buffers should be based on the available memory and I/O throughput capabilities of the server.

Here’s an example of what may be good settings to start with:

echo 256 > /usr/openv/netbackup/db/config/NUMBER_DATA_BUFFERS
echo 128 > /usr/openv/netbackup/db/config/NUMBER_DATA_BUFFERS_DISK
echo 262144 > /usr/openv/netbackup/db/config/SIZE_DATA_BUFFERS
echo 1048576 > /usr/openv/netbackup/db/config/SIZE_DATA_BUFFERS_DISK
echo 262144 > /usr/openv/netbackup/NET_BUFFER_SZ

NOTE: the “…/db/config/” directory does not exist by default and may need to be created before the above commands are executed.

  • If restoring from legacy tapes that were created with a different buffer size, and the restore is having difficulty reading the tape, you can try also setting this temporarily until the restore is complete (the value used needs to be equal to what the setting was when the tape was created):
echo 32768 > /usr/openv/netbackup/db/SIZE_DATA_BUFFERS_RESTORE

Test and Tweek

Last but not least: Test, test, tweak, and test some more as needed. It’s also a good idea to insure all OS level recommended tuning has also been completed.

OS Tuning

Update PATH

To ease administration, add the NetBackup command paths to the global settings

vi /etc/bashrc

Add this line to the end of the “/etc/bashrc” file to make command-line administration much easier to accomplish:

PATH=$PATH:/usr/openv/netbackup/bin/:/usr/openv/netbackup/bin/goodies/:/usr/openv/netbackup/bin/admincmd/:/usr/openv/volmgr/bin/

Linux

References:

Most basic Linux OS settings work fine, but in larger environments, some OS tuning may be needed. Here are some general guidelines:

Open Files

Open files should usually be set to twice the results of this command:

lsof | wc -l

To change the setting edit the /etc/security/limits.conf leaving all defaults but adding to the end of the file:

For details see: http://www.veritas.com/docs/000081370

# NetBackup Tuning:
*               soft    core    unlimited
*               hard    core    unlimited
*               soft    nofile  15000
*               hard    nofile  63535

The "soft nofile" settings should be the amount doubled of the above commands output.

Also edit /etc/profile leaving all at defaults, but add this to the end of the file:

For details see: http://www.veritas.com/docs/000081370

ulimit -S -c unlimited > /dev/null 2>&1 ulimit -aH

Shared Memory

Increasing Shared Memory settings can enable NetBackup to take full advantage of the RAM installed on the system.

In /etc/sysctl.conf leave all at defaults, but add this to the end of the file (these settings are only examples):

For Details see: http://www.veritas.com/docs/000081370

# The following are the kernel parameters for NetBackup
kernel.sem = 300 307200 32 1024
kernel.msgmni = 256
kernel.shmmni = 4096
kernel.core_pattern = /var/log/core.%e.%p
# End of NetBackup section
# Other Tuning:
net.ipv4.ip_local_port_range = 1024 65000
fs.file-max = 131072
net.core.rmem_default = 1024000
net.core.rmem_max = 1024000
net.core.wmem_default = 1024000
net.core.wmem_max = 1024000

Tape Buffers

The default buffer size for tapes is 32K on Linux, this should be changed to match the SIZE_DATA_BUFFERS setting

Edit grub.conf to modify this line (add highlighted section to match settings in the touch files above):

kernel /vmlinuz-2.4.18-24.7.x ro root=/dev/hda2 st=buffer_kbs:256,max_buffers:128

Optionally you can rebuild the kernel (make changes to st_options.h)

also see: /usr/src/linux*/drivers/scsi/README.st, subsection BOOT TIME.

XFS

For large file systems, such as for a large MDSP, you must use xfs file system on Linux/RedHat. For optimal performance you should add the option "inode64" to the /etc/fstab line for the respective file systems.

Patching

It best to not have your backup servers getting automatically patched in the middle of the night while they are the busiest. Have them patched on daytime schedule, or manually as needed.

Other Resources

NBU Tuning

http://www.veritas.com/docs/000019342



Back to NetBackup