Managed File Transfer

The series of job steps that make up this category allow for the transfer of files using a variety of file transfer protocols but with greater flexibility, filtering and performance than if you simply employ a script. Collectively these job steps are referred to as Managed File Transfer or MFT. The first series of file transfer operations to be supported are those of FTP, SFTP and FTPS. We expect to add different file transfer protocols in future versions.

 

Each job step may be used in a stand-alone fashion. Some job steps require that a connection already be present prior to usage. For example, Download Files may be used in a stand-alone fashion. Change Remote Directory requires a prior connection since the job step isn’t useful in a stand-alone mode. As with all ActiveBatch job steps you can use any job step from any category interspersed with steps from this category.

 

Job steps used in a stand-alone fashion employ connection, security credentials and network information along with the job step. You can reuse such information by creating a User Account for Managed File Transfer Connection (which is a Best Practice).

 

Ease of Use, Performance and Reliability

 

The overall benefits for MFT is that, coupled with the use of the User Account for Managed File Transfer, you can isolate the networking aspects from the operations themselves. This greatly simplifies the use of MFT. Since MFT is composed of job steps you can insert job steps from any category between MFT operations, which is something you can’t do when using a script. Finally, when MFT is transferring multiple files, you can elect to use the Parallel Transfer capability. This allows “n” connections to be created to the remote server (based on the number of files involved) and multiple files to be transferred simultaneously. MFT will sort the files to be transferred by their file size so the largest file is begun first. This can greatly reduce the elapsed time of file transfer (in performance tests we’ve seen elapsed times reduced by over fifty (50) percent. In addition, by default, MFT file transfer operations verify that the entire file has been downloaded by comparing the file’s checksum and size (where appropriate). This processing attempts to eliminate partial files.

 

Connection, Filters, and File Specification Information are described below

 

For a complete list of supported algorithms see Supported SSH Algorithms.

 

 

Several job steps provide parameters which allow you to connect to a server. This sub-section will describe those parameters.

 

Embedded Connection Data

 

This type of connection requires all the network and security credential information to be specified.

 

 

Host    The host name, FQDN or IP address of the server.

 

Credentials    Security credentials to be used when accessing the server. Username and Password are self-explanatory.

 

Protocol    The FTP protocol to be used. Standard FTP and SSH FTP. If Standard FTP is selected with Encryption specified, an SSL or FTPS connection is performed. The above example illustrates an FTPS (SSL) type connection.

 

  • Encryption    For Standard FTP. None is the default but if SSL is to be used, Implicit or Explicit encryption may be specified.

  • Private Key    For SSH FTP. If specified, a private key file to be used as part of the SSH connection. A passphrase is usually required but is not mandatory for the private key.

 

Transfer Restart Options  If an FTP connection fails, you may specify restart options which will attempt to recover the connection and restart the operation. Two parameters may be specified: Number of Retries and Restart Delay. The Number of Retries represents the number of times the connection and operation will be retried before failing. The Restart Delay is a time period ActiveBatch will wait before re-attempting the connection and operation.

 

Advanced Parameters:

 

Port    Port number to be used for the transfer. When omitted, the default, ActiveBatch will determine the port to use based on the protocol selected.

 

Passive    This Boolean parameter indicates whether the FTP session will run in Passive mode (True and the default) or Active mode (False).

 

Timeout    This parameter controls the time period (in seconds) to make a successful connection.

 

Max Parallel Transfers   This parameter controls the number of simultaneous file transfers (based on the actual number of files to be processed). You can specify a number from 1 to 10 that will create one or more concurrent file transfers.

 

Web Tunneling    This optional set of properties controls whether the connection is to be established via proxy.

 

Managed Connection

 

This type of connection uses a User Account Managed File Transfer Connection type for the network and security credential information.

 

 

User Account – This object represents a User Account with Managed File Transfer Connection type. For example:

 

 

The advantage of this approach (as opposed to embedding the network and security credentials within the job step) is reusable and easy maintenance if the host or other core data is changed.

 

 

Several of the Managed File Transfer job steps allow for additional filtering to be performed as part of the job step.

 

 

Depending on the operation to be performed you can specify several different combinations of the file criteria filtering. For example, the filter above indicates that files newer than one (1) hour with a size greater than 0 should be considered for the file operation to be performed.

 

Currently, two (2) filters may be specified: Date/Time and Size. Date/Time may be specified in absolute or relative fashion. Size is specified as a comparison operation against a number of bytes.

 

 

The Managed File Transfer steps Delete, DownloadFiles, ListFiles and UploadFiles support specifying one or more file specifications as the source (or target as the operation dictates). Each File Specification may contain whatever is a valid file specification in the context of the local or remote machine. In addition, the filename-extension portion of the specification may contain wildcard characters (asterisk or question mark). This file specification is then used as a pattern when matching eligible files or folders in according with two (2) other parameters: Recursive and Include (ListFiles only). The following table provides an expectation as to how the pattern matching operates. Note that if Filtering is also specified, eligible files are then examined under the appropriate filters to determine final eligibility.

 

Recursive No No Yes Yes

Include (ListFiles)

Files

Files & Folders

Files

Files & Folders

*.bak

File1.bak

File1.bak

FolderC.bak

File1.bak

FolderA/FileA.bak

FolderB.bat/FileB.bak

FolderC.bak/FileC.bak

File1.bak

FolderC.bak

FolderA/FileA.bak

FolderB.bat/FileB.bak

FolderC.bak/FileC.bak

*.*

File1.bak

File1.dat

File1.bak

File1.dat

FolderB.dat

FolderC.bak

File1.bak

File1.dat

FolderA/FileA.dat

FolderB/FileB.bak

FolderB/FileB.dat

FolderC/FileC.bak

FolderC/FileC.dat

File1.bak

File1.dat

FolderA/FileA.dat

FolderB.dat/FileB

FolderB/FileB.bak

FolderB/FileB.dat

FolderC/FileC.bak

FolderC/FileC.dat

*

File1

File1.bak

File1.dat

File1

File1.bak

File1.dat

FolderA

FolderB.dat

FolderC.bak

File1

File1.bak

File1.dat

FolderA/FileA

FolderA/FileA.dat

FolderB.dat/FileB

FolderB/FileB.bak

FolderB/FileB.dat

FolderC.bak/FileC

FolderC/FileC.bak

FolderC/FileC.dat

File1

File1.bak

File1.dat

FolderA

FolderA/FileA

FolderA/FileA.dat

FolderB

FolderB.dat/FileB

FolderB/FileB.bak

FolderB/FileB.dat

FolderC

FolderC.bak/FileC

FolderC/FileC.bak

FolderC/FileC.dat

 

Note the distinction between *.* and *. The first is a wildcard which assumes a period between the filename and extension. The second wildcard makes no distinction and indicates that all files/folders are retrived regardless of whether an extension is present or not.

 

Managed File Transfer Job Steps:

 

 

This job step allows you to change the directory on the remote server.

 

 

Platforms Supported

Windows and UNIX.

 

Software Prerequisites

None.

 

Job Step Properties

 

Directory – This property represents any valid directory or path specification acceptable to the server. After successful completion of this job step, the current directory will be set to this specification.

 

Handle – This optional property is the handle of a previously successful MFT Connect job step. If omitted, the last Connect handle is used.

 

 

This job step connects to a specified server. You have several different methods you can use to connect. To close an open connection, please use the Disconnect job step.

 

 

Above image - Managed File Transfer – Connect (Embedded)

 

 

Above image - Managed File Transfer – Connect (Managed Connection)

 

Platforms Supported

Windows and UNIX.

 

Software Prerequisites

Design and Windows: None.

 

Job Step Properties

 

Connection Data – This set of parameters governs the server, security credentials and network parameters for the server connection. You can select to embed the parameters within the job step or to use a User Account object (managed connection) and store the relevant parameter data. See Section Managed File Transfer – Connection Data for more information.

 

Working Directory – This optional parameter allows you to set a starting directory/path for subsequent file transfer operations.

 

Transfer Mode – This optional parameter establishes the default transfer mode when transferring files. Two (2) selections are available: ASCII and Binary. Note: Only available for Standard FTP.

 

Return Job Step

 

Handle – This string argument represents the handle to the open remote connection. You may use this connection handle for subsequent file transfer steps.

 

 

This job step is used to create a director(ies) on the remote server.

 

 

Platforms Supported

Windows and UNIX.

 

Software Prerequisites

Design and Windows: None.

 

Job Step Properties

 

Path – This property represents the directory you wish to create. Any directory or path specification acceptable to the remote server’s file system may be specified.

 

Auto-Create Parents – This Boolean property indicates whether the upper-level (or parents) directories should be created if not already present. If true (default), upper level parent directories are created if they don’t exist. If false, the upper level parent directory hierarchy must be present and exist.

 

Connection – This property represents the Connection Data. If a previous Connect was issued, this parameter may be left empty in which case the handle of the last open Connect is used.

 

 

This job is used to delete files and/or folders on the remote server.

 

 

Platforms Supported

Windows.

 

Software Prerequisites

Design and Windows: None.

 

Job Step Properties

 

File Specification – This property can contain one or more file specification(s) or folder specification(s) (including wildcard characters). Multiple file specifications, if specified, must be comma-separated (if the file specification contains a comma, you must quote the specification).

 

  • The files to delete from the remote server. This could be a relative or absolute path and supports wildcards to include multiple files.

  • If recurse is true and the pattern is something like *.log, all files matching that pattern in the top level directory and its subfolders will be deleted. No folders will be deleted. If recurse is false, only files from the top level of the specified directory will be deleted.

  • SubFolders will only be deleted if * is specified as the pattern because we cannot delete folders that have contents.

  • To delete a top level folder, recurse must be specified because its contents will be deleted also. Example, to delete the /Temp/DeleteMe folder, specify /Temp/DeleteMe as the File Specification and set Recurse to true. The DeleteMe folder along with all its files and sub folders will be deleted.

 

Recursive – This Boolean property indicates whether operation should be performed recursively assuming a multi-level path specification has been entered. If true, the file specification is examined recursively. If false, only the file specification as entered is used.

 

Filter – If specified, you can use the date/time and/or size specifications to add additional filtering as to eligibility to your file specification.

 

Connection – If specified, you can enter a stand-alone connection specification or you can simply inherit the last connection handle.

 

 

The job step allows you to disconnect a specific connection or the last connection successfully performed.

 

 

Platforms Supported

Windows and UNIX.

 

Software Prerequisites

None.

 

Job Step Properties

 

Handle – This optional property is the handle of a previously successful MFT Connect job step. If omitted, the last Connect handle is used.

 

 

This job step allows you to download files and/or folders based on both wildcards as well as filtering criteria.

 

 

Platforms Supported

Windows and UNIX.

 

Software Prerequisites

None.

 

Job Step Properties

 

Remote File(s) – One or more valid file specification(s), each comma separated, executed in the context of the remote server system. The specification can have wildcards present only in the filename-extension portion. The wildcards are supported in the context of the remote server system.

 

Recurse – This Boolean property controls whether the source specification is executed in a recursive fashion as it related to any sub-folders/directories. If false, no recursion is performed. If true, the source specification is executed in a recursive fashion.

 

Local Path – If present, a directory path specification that is present on the local system the job is executing on. If omitted, the current working directory is used.

 

Filter – If specified, filtering is performed based on Date/Time and/or Size.

 

Advanced Parameters:

 

  • Force Overwrite – This Boolean property controls whether existing files in the destination directory are overwritten if they exist. False indicates the files are not overwritten and True indicates they are.

  • Error Action – This property indicates how the job step should proceed if it encounters an error. If Fail is set, the job step is marked as failed (and any relevant job step error action is processed). If Continue is set, the job step continues executing and downloads file(s).

  • Log Successful Transfers – This Boolean property indicates whether successful transfers are noted and logged to the job’s log file. True caiuses logging; False causes no logging to occur

 

Connection – If specified, you can enter a stand-alone connection specification or you can simply inherit the last connection handle.

 

Return Job Step

 

SucceededFiles – This collection returns those files that were successfully downloaded. Information returned includes:

SourcePath, TargetPath, FileSize, FileName

 

FailedFiles – This collection returns those files in which an error occurred while downloading. Information returned includes: SourcePath, ExceptionMessage, FileName

 

 

This job step allows you to list files from a remote server.

 

 

Platforms Supported

Windows and UNIX.

 

Software Prerequisites

None.

 

Job Step Properties

 

File Specification – One or more valid file specification(s), each comma separated, executed in the context of the remote server system. The specification can have wildcards present only in the filename-extension portion. The wildcards are supported in the context of the remote server system.

 

Recurse – This Boolean property controls whether the source specification is executed in a recursive fashion as it related to any sub-folders/directories. If false, no recursion is performed. If true, the source specification is executed in a recursive fashion.

 

Include – This property indicates how the search is to be performed in terms of scope. Either FilesOnly or FolderandFiles.

 

Filter – If specified, filtering is performed based on Date/Time and/or Size.

 

Output Logging – This dropdown property indicates the extent of the logging. None (default), Simple (log file paths only) and Detailed (all).

 

Connection – If specified, you can enter a stand-alone connection specification or you can simply inherit the last connection handle.

 

Return Job Step

 

FtpDirectoryEntry – This collection represents the files and/or folders that have been returned by this job step. Information includes: FileName, FilePath, FileSize, FileTime, IsDirectory, IsSymLink.

 

 

This job step allows you to Move files within a Server system.

 

 

Platforms Supported

Windows and UNIX.

 

Software Prerequisites

None.

 

Job Step Properties

 

Source – This property represents the source file path specification. Wildcard characters are not permitted.

 

Target – This property represents the target file path specification. Wildcard characters are not permitted.

 

Connection – If specified, you can enter a stand-alone connection specification or you can simply inherit the last connection handle.

 

 

This job step allows you to execute any raw FTP command that is otherwise not directly supported by MFT.

 

 

Platforms Supported

Windows and UNIX.

 

Software Prerequisites

None.

 

Job Step Properties

 

Command – This string property represents a single FTP command that will be sent to the remote server.

 

Connection – If specified, you can enter a stand-alone connection specification or you can simply inherit the last connection handle.

 

Return Job Step

 

CommandResponse – This property contains a Code (integer) and Output (string) that represents the status of the command and any output generated.

 

 

This job step allows you to synchronize the contents of a local to remote or remote to local set of directories.

 

 

Platforms Supported

Windows and UNIX.

 

Software Prerequisites

None.

 

Job Step Properties

 

Local Path – This property represents the local (non-FTP) system. Any valid file path specification may be specified. If omitted, the current working directory is used.

 

Remote Path – This property represents the remote system. Any valid file path specification may be specified. If omitted, the current remote working directory is used.

 

File Name Pattern – This property represents the filename.extension portion and may include any valid wildcard characters. If omitted, *.* meaning all files is used.

 

Recurse – This Boolean property controls whether recursion of any sub-directories/folders occurs. If true, recursion occurs; if false (default) recursion does not occur.

 

Direction – This dropdown property controls the direction of the synchronization. LocalToRemote indicates that local files are sync’d with the remote system; RemoteToLocal indicates that remote files are sync’d with the local system

 

Delete Missing Files – This Boolean property indicates that any files missing from the Source (depends on direction of transfer) are also deleted from the Destination system. True indicates that missing Source files are deleted from the Destination system. False (default) indicates that any missing files are not deleted from the Destination system

 

Filter – If specified, filtering is performed based on Date/Time and/or Size.

 

Advanced Parameters:

 

  • Force Overwrite – This Boolean property controls whether existing files in the destination directory are overwritten if they exist. False indicates the files are not overwritten and True indicates they are.

  • Error Action – This property indicates how the job step should proceed if it encounters an error. If Fail is set, the job step is marked as failed (and any relevant job step error action is processed). If Continue is set, the job step continues executing and downloads file(s).

  • Log Successful Transfers – This Boolean property indicates whether successful transfers are noted and logged to the job’s log file. True causes logging; False causes no logging to occur.

 

Connection – If specified, you can enter a stand-alone connection specification or you can simply inherit the last connection handle.

 

Return Job Step

 

SkippedFiles – This collection returns those files that were skipped in the synchronize process because they were already present and up-to-date on both systems. Information returned includes: SourcePath and FileName.

 

DeletedFiles – This collection returns those files that were deleted because they were missing on the Source system. Information returned includes: TargetPath and FileName.

 

SucceededFiles – This collection returns those files that were successfully synchronized. Information returned includes:

SourcePath, TargetPath, FileSize, FileName

 

FailedFiles – This collection returns those files in which an error occurred while synchronizing. Information returned includes: SourcePath, ExceptionMessage, FileName

 

 

This job step allows you to upload files and/or folders based on both wildcards as well as filtering criteria.

 

 

Platforms Supported

Windows and UNIX.

 

Software Prerequisites

None.

 

Job Step Properties

 

Local Files(s) – One or more valid file specification(s), each comma separated, executed in the context of the local server system. The specification can have wildcards present only in the filename-extension portion. The wildcards are supported in the context of the local server system.

 

Recurse – This Boolean property controls whether the source specification is executed in a recursive fashion as it related to any sub-folders/directories. If false, no recursion is performed. If true, the source specification is executed in a recursive fashion.

 

Remote Path – If present, a directory path specification that is present on the remote system the job is executing on. If omitted, the current working directory is used. Note: Only a single path specification may be specified.

 

Filter – If specified, filtering is performed based on Date/Time and/or Size.

 

Advanced Parameters:

 

  • Force Overwrite – This Boolean property controls whether existing files in the destination directory are overwritten if they exist. False indicates the files are not overwritten and True indicates they are.

  • Error Action – This property indicates how the job step should proceed if it encounters an error. If Fail is set, the job step is marked as failed (and any relevant job step error action is processed). If Continue is set, the job step continues executing and downloads file(s).

  • Log Successful Transfers – This Boolean property indicates whether successful transfers are noted and logged to the job’s log file. True caiuses logging; False causes no logging to occur

 

Connection – If specified, you can enter a stand-alone connection specification or you can simply inherit the last connection handle.

 

Return Job Step

 

SucceededFiles – This collection returns those files that were successfully uploaded. Information returned includes:

 

SourcePath, TargetPath, FileSize, FileName

 

FailedFiles – This collection returns those files in which an error occurred while uploading. Information returned includes: SourcePath, ExceptionMessage, FileName