Case Study - Managed File Transfer and ForEachItem Step

Some Jobs Library Job Steps return an object or a collection of objects.  If you would like to iterate through a collection of returned objects, you can do so by passing the collection to a ForEachItem Step. The ForEachItem Step is located in the Jobs Library Flow Control category. In this case study, we will look at iterating through the FailedFiles collection that is returned by the Managed File Transfer (MFT) DownloadFiles Step. Keep in mind, any Step that returns a collection can use the ForEachItem Step to iterate through the collection.

The ForEachItem Step consists of two properties, the Context Name and the Collection. 

 

 

The Context Name property represents an execution variable that will be used for those Step(s) that are nested within the ForEachItem Step (where the nested Steps are dropped in the “Drop job step here” portion of the Step). An execution variable is also referred to as a context variable.  The Context Name property has a default value of “Value”.  This can be changed, if desired.

The Collection property represents an execution variable that is an object or a collection of objects. The Collection property assignment does not use the standard Jobs Library execution variable syntax, which is %{}. The Collection property starts with an equal sign (=). The assignment is done this way to keep the object data types intact.  

 

In this case study, we are going to establish a workflow that does the following:

 

Downloads files from an FTP Server

• If any files fail to download, the job will not fail; it will download whatever files it can.

After the download is complete, the job will check to see if any files failed to download. 

• If any files did not download, the name of the file(s) are retrieved and stored in a variable.

• An email is sent out which includes the Job that generated the email (its full path and instance ID), and the name(s) of the files that did not download.     

 

The Workflow

 

Below is an image that provides a high-level look at the workflow.  First, the DownloadFiles Step will execute. The IfBranch is the next Step that runs.  Notice the red box drawn around the IfBranch Step.  The remaining Steps are nested within the IfBranch, and therefore they will only execute if the IfBranch Expression evaluates to true.  

 

 

Let’s look at the DownloadFiles Step, accessed in the MFT category of Job Steps. Let’s describe what this Step is doing, as per the image below. All the Remote Files (*) in the FTP User’s Home directory will be downloaded from the FTP server to the Agent’s c:\inbound folder.  The Error Action property is set to “Continue”, meaning if any files fail to download (e.g. access is denied), the job will not fail out immediately, but rather it will continue to download whatever files it can. 

 

 

Observe the Connection property in the above image. It is using a Managed Connection, which requires you specify a User Account object. This means all the FTP server information is configured in the User Account Object. This User Account object Credential Type is called the “Managed File Transfer Connection”.  This is where the FTP Server host, FTP user credentials, protocol type, and other properties are set. It is recommended you use the Managed File Transfer Connection when you have two or more FTP jobs connecting to the same FTP server with the same FTP user credentials.  This allows you to share the FTP server connection information among multiple jobs.

 

You also have the option to embed the connection data in the Step itself, if you select “Embedded Connection Data” for the Connection property (see the image below).  This is typically used when you only have one job using the FTP server and FTP user credentials specified.  When Embedded is selected, properties will appear just under this selection, allowing you to configure the FTP information. Lastly, the Connection property supports a Connection Handle option, where you pass the connection handle to the Step when using an upstream MFT Connect Step.  The Connect Step is typically used when you have multiple FTP Steps in the same job (e.g. ListFiles, then Download Files).  You only need to connect once using the Connect Step, then pass the connection handle to all the following Steps.  In this example, there is only one MFT Step, so it is fine to configure the connection information on the DownloadFiles Step, and not use a Connect Step.

 

 

Next, observe what the DownloadFiles Step returns.  When you click on the Step’s return icon at the bottom of the Step, you see the following, after expanding SucceededFiles and FailedFiles. 

 

 

The Succeeded Files is a collection that includes the SourcePath, Targetpath, FileSize and FileName elements.

 

The FailedFiles is a collection that includes the SourcePath, ExceptionMessage and FileName elements. As the collection names imply, the SucceededFiles collection provides details for one or more files that were successfully downloaded. The FailedFiles collection provides details for one or more files that were not successfully downloaded. It is optional whether or not you use one or both of these collections in downstream Step(s). By default, the job log file will capture the name(s) of any files that fail to download. That may suffice for you. In addition, you can configure whether or not you want the successful file download information to be output to the job log file (see the “Log Successful Transfers” property). Again, this may be enough information for you. However, if you would like to do more with this information, for either collection, you can pass it to a ForEachItem Step and iterate through the collection.

 

Let’s look at the IfBranch that follows the DownLoadFiles Step.  In the image below, you can see all the remaining Steps. They are nested within the IfBranch.

 

 

 

 

IfBranch: This Step has a single Expression property.  If it evaluates to true, the nested Steps will execute.  If it does not evaluate to True, the nested Steps will not execute.

 

The Expression property is as follows: DownloadFiles.ReturnValue.FailedFiles.Count>0

 

First, you do not use execution variable syntax in the IfBranch’s Expression property.  

 

This is true for any variables referenced in the Expression property of the IfBranch – Constant, Active or Execution. This is so because all unquoted strings are implicitly considered to be variables.

 

As you type DownloadFiles in the Expression property of the IfBranch, ASCI-Sense will automatically popup a list of items you can choose from, as per the image below. Double-click on DownloadFiles.

 

 

Type a period. Doing so will result in the following popup:  Double-click on ReturnValue.

 

 

Type a period.  Doing so will result in the following popup: Double-click on FailedFiles

 

 

Type a period.  From there, you must manually type Count, then type >0.

 

This completes the IfBranch’s Expression property: 

 

DownloadFiles.ReturnValue.FailedFiles.Count>0.

 

If the FailedFiles collection is greater than 0, that means that at least one file failed to download.  If there is at least one file that failed to download, the Steps nested within the IfBranch will execute. 

 

The first nested Step is the Expression Step. It is used to create an execution variable. The variable named “filenames” is initialized as blank. This will store one or more file names that failed to download. 

 

 

Next, the FailedFiles collection is passed to the ForEachItem Step.  The Context Name property is “Value” by default. The Collection property is specified as depicted below. Like the IfBranch’s Expression property, variable syntax is not used in the Collection property.

 

Type an = sign first, then start to type DownloadFiles.  As you being to type this, ASCI-Sense will pop up, the same way it did for the IfBranch Expression property. Double-click on DownloadFiles, type a period, double click on ReturnValue, type a period, then double click on FailedFiles.

 

This completes the Collection property.

 

 

Nested within the ForEachItem is one Step, Expression_2. This is the only Step that will execute with each iteration of the loop. For each item in the failed files collection, the file’s name will be added to the “filenames” variable.  Observe how the element of the FailedFiles collection is referenced: ForEachItem.Value.filename.  In addition to filename, you could also reference SourcePath and ExceptionMessage (e.g. ForEachItem.Value.SourcePath). The variable ForEachItem.Value.Filename is not referenced using execution variable syntax %{} in the Expression’s Content property. The majority of Steps, when referencing an execution variable, require it be referenced using %{} syntax.  There are few exceptions which we have described in this case study. 

 

As you type ForEachItem.Value, the ASCI-Sense popup helper will appear. After selecting ForEachItem, typing a period, then selecting Value, then typing another period, you will need to know what to manually type after that.  Look at the return information again, if necessary (depicted below).  In our example, filename was used. 

 

 

After the ForEachItem step is complete, the EMailExchange Step will execute to notify individual(s) which file(s) did not download, and what job this happened on.

 

 

As depicted in the above image, an email will go out to include the Job’s full path and the instance ID. The built-in variables @path and @label identify the job, and @ID identifies the instance ID. The message includes the one or more file names that failed to download, by referencing the variable “filenames”. Notice the %{filenames} variable is using the execution variable syntax. 

 

 

This is the Job’s General property sheet:

 

 

This is the Job Instance:

 

 

As a final note, all this information can be applied similarly to the SucceededFiles return collection.  You can pass this collection to a ForEachItem Step and iterate through it.  The Collection property would be

 

=DownloadFiles.ReturnValue.SucceededFiles.  When referencing a collection element in the loop, you would use:

 

ForEachItem.Value.SourcePath (where SourcePath could also be Targetpath, FileSize or FileName).