NDMP with NetBackup

From World History Wiki
Jump to: navigation, search
World History Wiki is Brought to you by:
S.J.'s Adventures


Source Documents:

NetBackup for NDMP guide

NetBackup NDMP Appliance Information

NetBackup Hardware Compatibility List

NetApp NDMP Tape Drive Compatibility List

NDMP.org's NetApp page

NDMP Basics

For NetBackup for NDMP terminology see pages 40 - 58 of the NetBackup for NDMP Admin Guide - For Direct Copy (Direct path to Tape), see pages 59 - 62.

See “Configuring NDMP backup to Media Manager storage units” on page 56 of the NetBackup for NDMP Administrator Guide.

NDMP Host Config

For more detail see pages 40 of the NetBackup for NDMP Admin Guide

Note: you do not install any NetBackup software on the NDMP host(s).

If using "Global NDMP credentials" be sure to first set those up in the NDMP section of the Host Properties of the Master Server.

  1. On the NetBackup server, from the Administration Console: under Media and Device Management > Credentials, click NDMP Hosts.
  2. Under Actions, select New > NDMP Host.
  3. In the NDMP host name dialog box, enter the name of the NDMP server for NetBackup to back up.
    The NDMP host name is case-sensitive. Whenever this host name is used, the name must be identical to the name entered here. (For example, when you configure tape drives and storage units for this host.)
  4. Click OK.
  5. In the New NDMP Host dialog box, specify the following:
    Use global NDMP credentials for this NDMP host or Use the following credentials for this NDMP host on all media servers or Use different credentials for this NDMP host on each media server
    Enter credentials as needed.
  6. Repeat this procedure for each NDMP host that NetBackup backs up.

For Three-Way NDMP access, the above procedures are still needed. The NDMP host regardless of if it has attached tape drive(s), will need it's credentials entered for NetBackup to be able to access it.

To verify NDMP host configuration, or probe to see what devices it has, use the following commands:

tpautoconf -verify ndmp_host_name
tpautoconf -probe ndmp_host_name

NDMP Drive Config

For more detail see pages 42 - 48 of the NetBackup for NDMP Admin Guide.

You use the NetBackup Device Configuration Wizard to discover and configure the robots and drives that are already attached and configured on an NDMP host. The wizard requires NDMP protocol versions V3 or V4. The NDMP Host must first be authorized within NetBackup before the drives can be discovered.

See the NetBackup Administrator’s Guide, Volume I, for general information on configuring NetBackup media.

Drives can be shared using the Shared Storage Option (SSO) of NetBackup. The drives can be shared as both NDMP drives and non-NDMP drives.

NOTE: If using Remote NDMP (backing up over the network), do not configure NDMP-attached devices, instead configure them as ordinary NetBackup devices (attached to the NetBackup server the data is to be sent to), not as NDMP-attached devices.

NDMP policies

For more detail see pages 49 - 52 of the NetBackup for NDMP Admin Guide'

See the NetBackup Administrator’s Guide, Volume I, for more information on NetBackup policies and the Policy utility. To configure a policy for the NAS_Snapshot method, see the NetBackup Snapshot Client Administrator’s Guide and the NetBackup Snapshot Client Configuration Guide.

After you configure NetBackup for NDMP, you can schedule backups by creating an NDMP policy in NetBackup. Creating an NDMP policy is very similar to creating other NetBackup policy types.

NOTE: The main difference is that you must set the Policy Type to be NDMP'.

NDMP policies can use either NDMP storage units (local or 3-way backup) or Media Manager storage units (remote NDMP or remote "3-way" backup).

Backup Schedules

The allowable backup types for schedules in an NDMP policy are: Full, Cumulative Incremental, or Differential Incremental. User-initiated backups and archives are not allowed because the NDMP protocol does not permit these tasks.

Be careful when specifying Override policy storage unit when the Policy contains multiple NDMP clients as this will likely cause data to follow network paths.

NDMP Clients

An NDMP policy can have one or more NetBackup clients, but each NetBackup client must be an NDMP host. The Hardware and OS settings should be NDMP NDMP.

To have multiple clients in the same policy, the Backup Selections must be setup such that it will work for all clients. For this reason it's usually necessary to setup separate policies for each client, and in large environments, it may even be necessary to have multiple policies per NDMP client.

Backup Selections

Backup Selections are were a lot of the "Magic" happens with NDMP backups, and can become quite complex. They are also somewhat specific to the model and version of the NDMP Host/NetBackup Client.

Warning: You cannot back up files where the path length is greater than 1024 characters.

The backup selections list must specify directories from the perspective of the NDMP host. The following backup selections capabilities are NOT supported for an NDMP policy:

  • Wildcards in pathnames. For example, /home/* is an invalid entry.
  • Individual file names. Only directory or volume names are allowed.
  • Exclude list (because client software is not installed on the NDMP host). You can, however, exclude files by using the SET keyword.

In NetBackup, environment variables can be set within the backup selections list by specifying one or more SET directives. The SET option lets you use NDMP environment variables to pass configuration parameters to an NDMP host with each backup, such as to exclude files from backups. The format is vendor dependent; refer to the Environment Variables section bellow for more details on which variable can be passed and in what format. These variables are vendor specific.

NDMP environment variables can be one of the following types:

  • Defined as optional by the NDMP protocol specification. You can set these variables.
  • Specific to an NDMP host vendor. You can set these variables.
  • Reserved for use by NetBackup:
FILESYSTEM
LEVEL
DIRECT
EXTRACT
ACL_START

Order is also important within the Backup Selections list. The SET directive must be the first in the list, followed by the file systems or volumes to back up. The SET directives are sent to the NDMP host in the order they are entered in the list.

In general, the syntax of a SET directive is as follows:

SET environment_variable = value

For examples see pages 51 of the NetBackup for NDMP Admin Guide'

The values that each backup uses are saved and provided to subsequent restores of the directory. The NDMP host may have some environment variables that are set internally and these are also saved for restores.

Direct Copy (PTT)

For Details on Direct Copy (Direct path to Tape), see pages 59 - 62 of the NetBackup for NDMP Admin Guide.

Direct Copy, or Direct Path to tape (PTT), requires the VTL or NAS device used for primary copies of backups, to also act as the NDMP Host. It must have SAN attached physical tape and virtual devices configured on it for NDMP to allow for duplications to occur directly from the appliance to physical tape.

With NDMP configured properly, you simply have to specify correct NDMP Storage Unit related to the appliance.

Restores

Restores of NDMP host backups can be initiated from any NetBackup media server that meets the following criteria:

  • Is within the same overall NetBackup storage domain
  • Uses the same NetBackup master server that the media server uses that performed the backup

You will need about 100 MB more space on the destination storage system than the amount of data to be restored.

The data can be restored to the NDMP host where it was backed up, or to another NDMP host. Client side restores are not possible, but must be performed from a server side console.

Warning: An NDMP restore always overwrites existing files.

The "Restore type" is still "Normal Backup", but be sure to change the "Policy type" to "NDMP".

All other aspects are the same as doing a normal restore.

For Direct Copy (Direct path to Tape), if the first copy has expired, the restore from physical tape uses the appliance (NDMP host) to read the tape for the restore.

Quantum Specifics

For more details see the section on Direct Copy.

Quantum's DXi systems allow the use of NDMP to do a direct to tape "duplication" of your backup images.

This reduces the load on your NetBackup servers by only sending meta data to the NetBackup servers while the actual backup data goes over SAN.

Configuring this is a simple matter of setting up the available SAN HBA ports on the DXi as initiator ports, zoning the tape drives to it, and configuring the DXi storage server as an NDMP device.

Then when you run your duplications, you send them to the NDMP STU related to the DXi.

NetApp Specifics

Also see the "NetApp tapebkup guide.pdf"

NetApp supports two types of backups:

SMTape backup                       Dump backup
Backs up blocks of data to tape.    Backs up files and directories.
Supports only level-0 backup.       Supports level-0, incremental, and differential backups.
Does not support single file        Supports single file restore.
restore.
Capable of backing up multiple      Capable of backing up only the base Snapshot copy.
Snapshot copies in a volume.
Preserves de-duplication while      Does not preserve de-duplication while backing up data.
backing up and restoring data.
NetBackup's NDMP backups uses the Dump Backup Method by default; which is best for most restores likely to be doing. The SMTape backups are better suited for fast, full volume, backups and Disaster Recovery.

Configuration Tips

  1. Under "Media and Device Management" -> "Credentials", select "NDMP Hosts".
  2. Right click and select "New NDMP host..."
  3. Enter the name of the NDMP Host (this name is case sensitive and must match DNS, as well as any other configurations for this Host within NetBackup).
  4. Click "OK"
  5. Select "Use the following credentials for this NDMP host on all media servers"
  6. Enter the NDMP user name as it is configured on the NetApp filer. Typically this will be "ndmp".
  7. Enter the MD5 Hash output as the password.
    To obtain this Hash run the following command on the filer:
    ndmpd password ndmp
  8. Click "OK".
    If it works the "New NDMP Host" window will disappear and the newly configured NDMP Host will show up in the list of NDMP Hosts. If it doesn't work, you will get an error message.
  9. To verify the configuration use the following command:
    tpautoconf -verify NDMP_HOSTNAME


  • See NDMP Policies Section for details on setting up backups.
    Essentially, create a new policy of type "NDMP" and added the NDMP Hosts as clients in that policy.
    For direct NDMP and 3-way backups, specify an NDMP Storage Unit. For "Remote" 3-way, specify any storage unit controlled by a media server.
NOTE: The backup runs faster if the path(s) specifies one of the following:
* A full volume
* Full qtrees
* A full volume excluding qtrees
* Divide large volumes into smaller volumes or qtrees.
For example, if you divide a 500-GB volume into three qtrees, you can back up each qtree to a separate tape drive or run separate full backups on three different nights.
* Limit the amount of data in a volume or qtree to be backed up to 200 GB.
* Schedule the backups in appropriate rotations.
* Schedule backups when the load on the storage system is moderate.
* Do not divide a backup into more than 15 qtrees.


  • To setup an NDMP storage unit, requires first having tape drives attached to an NDMP host.
    These drives can be part of a library dedicated to the NDMP Host, or SAN attached drives that are shared with other NetBackup media servers (utilizing Shared Storage Option). See the Drive Configuration Section For more details.
    This procedure has not yet been tested, as we have only used 3-way remote backups thus far.

Tape Drive configuration

Tape drive configuration is almost automatic within NetApp, so long as the proper configuration file(s) is installed on the system for the particular type of tape drive(s) that are presented to the filer.

The configuration file is known as a TCF file, or Tape Configuration File.

Once the configuration file is in place, and the tape drive(s) are zoned/presented to the filer, simply re-scan the FC bus to discover the tape drives.

Snapshots

Backing up from a snapshot, as opposed to a live file system, helps insure the integrity and consistency of the data backed up.

NetApp's Snapshots allow for rotating snapshots while keeping the snapshot name the same for the most recent snapshot. This makes it easy within NetBackup to setup a specific snapshot path once without having to update it every time a new snapshot is taken.

An example of a path may be something similar to:

/vol/share1/.snapshots/weekly.0

NetBackup can also be configured to create, index, and maintain retentions on NetApp filer snapshots. Relinquishing the control and management from the NAS admin's and instead giving it to the NetBackup application. This allows for easily searching for previous versions of a file, and maintaining an index of the snapshots within NetBackup that stays in-sync with the retention of the snapshots.

NetBackup can then also control which of the snapshots get's copied to a secondary storage, such as tape out via NDMP.

Environment Variables

Environment variables can be set within the NetBackup Policies using the "SET" directive. See Backup Selections Section for more details on how to set these variables during NDMP backups.

According to NetApp: "A backup administrator rarely specifies environment variables; however, you might want to change the value of an environment variable from that set by the backup application to characterize or work around a functional or performance problem. For example, an administrator might want to temporarily disable file history generation to determine if the backup application's processing of file history information is contributing to performance issues or functional problems."

Environment variables supported for dump backups:

Variable Values Default
Description
BASE_DATE
or DUMP_DATE
0,-1, value -1
Specifies the start date for incremental backups. NetBackup will control these variables automatically when and NDMP backup is run as an Incremental for Cumulative Incremental backup.
Cumulative Incrementals are always level 1, were differentials are "last level + 1", up to a max of 9.
These variables are an alternative to the /etc/dumpdates file for controlling incremental backups.
DEBUG Y or N N
Specifies that debugging information is printed. Used only when troubleshooting.
DMP_NAME string none
Specifies the name for a multiple subtree backup. This variable is mandatory for multiple subtree backups.
EXCLUDE pattern_string none
Specifies files or directories that are excluded when backing up data.
The exclude list is a comma-separated list of file or directory names. If a match is found, it is excluded from the backup. The following are rules for specifying names in the exclude list:
  • The exact name of the file or directory must be used.
  • An asterisk (*) is a wildcard character. The asterisk must be either the first or the last character of the string. Each string can have up to two asterisks.
  • A comma in a file or directory name must be preceded with a backslash.
  • The exclude list can contain up to 32 names.
IGNORE_CTIME Y or N N
Specifies that a file is not incrementally backed up if only its ctime value has changed since the previous incremental backup.
Some applications, such as virus scanning software, change the ctime value of a file within the inode, even though the file or its attributes have not changed. As a result, an incremental backup might back up files which have not changed. The IGNORE_CTIME variable should be specified only if incremental backups are taking an unacceptable amount of time or space because the ctime value was modified.
MULTI_SUBTREE_NAMES string none
Specifies that the backup is a multiple subtree backup. Multiple subtrees are specified in the string which is a newline-separated, null-terminated list of subtree names. Subtrees are specified by path names relative to their common root directory, which must be specified as the last element of the list. If you use this variable, you must also use the DMP_NAME variable.
NDMP_UNICODE_FH Y or N N
Specifies that a Unicode name is included in addition to the NFS name of the file in the file history information. This option is not used by most backup applications and should not be set unless the backup application is designed to receive these additional file names. The HIST variable must also be set.
HIST Y or N N
Specifies that file history information is sent to the backup application. Most commercial backup applications, including NetBackup, set the HIST variable to Y. If you want to increase the speed of a backup operation, or you want to troubleshoot a problem with the file history collection, you can set this variable to N.
NO_ACLS Y or N N
Specifies that ACLs not be copied when backing up data. Ordinarily a backup using the dump command writes out metadata related to Windows ACLs. The NO_ACLS variable stops this information from being backed up.

Administration Tips:

  • When restoring files, if the NetApp filer does not use Direct access recovery (DAR), the destination path that you specify for the restore must end with the original folder and file name. If the original backup path was /vol/vol1/mydir/myfile, the destination path for the restore must end with /mydir/myfile. Otherwise, NetBackup appends /mydir/myfile to the end of the destination path. For more details on DAR, and to determine whether DAR has been disabled in NetBackup, refer to the NetBackup for NDMP Administrator's Guide.
  • For NDMP devices to share tape drives, tape reservation must be enabled in the ONTAP software on the filer as well as in NetBackup. You can use either SCSI persistent reservation or SCSI reservation, but it must match in both NetBackup and on the NDMP Host.
  • Though a storage system can detect 64 tape drive connections, only 16 concurrent backup or restore sessions with local tapes are allowed.
  • Data ONTAP 8.0 supports a blocking factor between the range of 4 KB to 256 KB. The default blocking factor is 63 KB. If using LTO tape drives, it may be advisable to change this to 256 KB, to match the buffer size of the tape drives. Keep in mind that when doing restores, this size must be supported by the target NDMP Host.
  • You can restrict the set of backup application hosts permitted to start NDMP sessions on a storage system. You can specify the authentication method to use (text or challenge) in order to allow NDMP requests. You can enable or disable monitoring of NDMP connection requests.
  • NDMP users must have the login-ndmp capability to be able to successfully authenticate NDMP sessions. A predefined role named backup, by default, has the login-ndmp capability.
  • The NDMP service is controlled by means of the Data ONTAP administrative interface or the following commands:
    ndmpd on
    (Starts the NDMP service)
    ndmpd off
    (Stops the NDMP service)
    ndmpd status
    (Displays the status of the NDMP service including any active NDMP sessions)
    ndmpd probe session-number
    (Displays details about the specified session)
    ndmpd status [session]
    (Displays general NDMP status information)
    ndmpd kill session
    (Terminates an NDMP session)
    ndmpd debug n
    (enable NDMP debug logging - n = 0,10,30,50,70)
    ndmpd debug
    (see the NDMP debug levels currently set)
    ndmp version
    (Displays the highest version of NDMP that the storage system is currently set to use)
    stats show ndmp
    (view detailed statistics about file history performance)
  • To specify the preferred network interface to be used for NDMP data connections, enter the following command:
    options ndmpd.preferred_interface interface
  • To display an NDMP session log file from a UNIX, mount the root volume of the storage system to a UNIX client and view the contents of the NDMP session log file using the cat or more UNIX commands or a text editor.
  • By default the NDMP service is not started at boot time. To start it, add the following line to the end of the /etc/rc file on the NetApp system:
    ndmpd on
  • To determine the number of objects in a volume, enter the following:
    maxfiles
  • In ONTAP 8.0, both ONTAP 7 mode and ONTAP 10 mode are combined into a single release. You cannot run both modes on the same filer concurrently.
  • Image Backup, also called referred to as SnapMirror to Tape or smtape, is a new Data ONTAP 8.0 feature. This feature backs up a volume as a single file, but can only restore the entire volume (no individual files). For details see Page 48 of the "NetBackup for NDMP: NAS Appliance Information Guide", and the "NetApp tapebkup guide.pdf".


GX Specific:
  • Use of the tpautoconf -probe command for device discovery is not supported in local NDMP configurations with the NetApp Data ONTAP GX.
  • Directory DAR is not supported with the NetApp Data ONTAP GX.
  • All volumes on the NetApp Data ONTAP GX filer are created within Virtual Servers.
    The path specified in the NetBackup policy’s Backup Selections list must contain the Virtual Server. For example:
    /vs1/vol1

HNAS Specifics

HNAS (pronounced "H" nas) is Hitachi's Network Attached Storage solution.

For more details on this solution's NDMP capabilities, get a copy of Hitachi's H-NAS "Backup Administration Guide".

Direct to tape (SAN) and network or 3-way NDMP are both supported.

Tape Drive configuration

H-NAS must have tape devices specifically configured to be allowed for use by a particular cluster node before they will be available to NDMP. Simply presenting the device to the H-NAS filer will not allow it to be used for NDMP.

Snapshots

Backing up from a snapshot, as opposed to a live file system, helps insure the integrity and consistency of the data backed up. Also, using NDMP (as apposed to a network share protocol like NFS or CIFS) also improves performance, data integrity, and allows for protecting file and directory permissions without extensive security overrides.

There are three options for specifying which snapshot to use for backups:

1. To use a specific snapshot explicitly including the snapshot name in the path to backup (this will prevent the creation of a new snapshot when the NDMP backup is started, but may require updating the path every time a backup is performed). For example:

For CIFS:

/__VOLUME__/filer/filesystem/~snapshot/snapshot_name

For NFS:

/Path/to/volume/.snapshot/snapshot_name

This option only allow for full backups for the specified snapshot as any changes would be recorded under a different snapshot name.


2. Enable the feature that does an automatic snapshot specifically for the NDMP backup. You do not specify the snapshots directly, just the volume to be backed up. This also allows for incremental backups to be done, if the snapshot is configured to be retained at least until the following backup is completed ("Delete snapshot after next backup" or even better may be until it is obsolete).

This is required, because these snapshots are also used to determine what has changed on the filesystem so that the data backed up is only what has changed since the last backup.

This feature may cause redundant snapshots of the same filesystem if scheduled snapshots are also created).

3. See NDMP_BLUEARC_USE_SNAPSHOT_RULE in the Environment variable list for a third option on determining which snapshot to use. This option is particularly useful when timing is important. Such as when trying to backup a snapshot of a database.

Environment Variables

Environment variables can be set within the NetBackup Policies using the "SET" directive. See Backup Selections Section for more details on how to set these variables during NDMP backups.

According to H-NAS documentation: "The storage management application generates most of these variables and supports configuration of additional variables. They are invoked from the Replication Rules: Add Rules page."

Environment variables supported for dump backups:

Variable Values Default
Description
EXCLUDE Comma-separated list of files or directories NULL
Specifies which files or directories to exclude from a backup.

For a full pathname the path name must start with a forward slash (/). An asterisk (*) can be used at the end as a wildcard character. The last element in the path (file or directory) can be specified by name, and it may start or end with the wildcard character *.

LEVEL 0 - 9 0
Backup level. 0 is full. 1-9 are incremental, with increasing numbers each time to insure only changed files are backed up. NetBackup will control these variables automatically when and NDMP backup is run as an Incremental for Cumulative Incremental backup.
FUTURE_FILES y or n n
Enables backup of files created after the current backup started.
HIST y or n y
Specifies that file history information is sent to the backup application. Most commercial backup applications, including NetBackup, set the HIST variable to Y. If you want to increase the speed of a backup operation, or you want to troubleshoot a problem with the file history collection, you can set this variable to N.
NDMP_BLUEARC_AWAIT_IDLE y or n y
Prevents NDMP backups and copies of a replication destination while a replication copy is actively writing data. Since the lock is held at a volume level, it may be desirable for directory-level replication to override this action.
NDMP_BLUEARC_EMBEDDED_HARDLINKS y or n n
Enable or disable inline hard linked file support. Disabling maintains backup compatibility with older systems or releases.
NDMP_BLUEARC_EXCLUDE_MIGRATED y or n n
Will files whose data has been migrated to secondary storage be included?
NDMP_BLUEARC_EXTERNAL_LINKS ignore, recreate_link, or remigrate remigrate
How a replication operation handles a cross volume link
NDMP_BLUEARC_INCLUDE_ONLY_MIGRATED y or n n
Exclude files whose data has not been migrated to secondary storage?
NDMP_BLUEARC_USE_CHANGE_LIST y or n Set using the CLI ndmp-option change_list_incr command
Should a changed object list be used to direct the search for changed files?
NDMP_BLUEARC_USE_SNAPSHOT_RULE Snapshot rule name N/A
Used to invoke the usage of a specified snapshot rule to determine which snapshot to use for backup. If set, NDMP does not create or delete snapshots. Especially useful for backing up timed snapshots of databases.



Back to NetBackup