Working with Queues

In our previous sections we’ve focused on the various ways that you can trigger and execute plans and jobs. In this section we’re going to shift the focus to Queues. ActiveBatch supports two (2) types of Queues: Execution and Generic.

Execution Queue Properties

Execution queues represent execution machines. The key properties for an execution queue are the machine name and job limit properties.

Machine name vs. Published name

Many customers who begin using ActiveBatch set the machine name property as the hostname or IP address of the desired execution machine. ActiveBatch Best Practices suggest that you establish an Active Directory/AD LDS published name for your execution machines. This allows you to failover from one machine to another with no changes to the ActiveBatch system. For example:

The above figure shows the hostname “testsys6”. While that’s a perfectly good machine name what happens if testsys6 is down? Or better yet, what if testsys6 is replaced with a larger machine named “testsys50”? You would be forced to change the machine name (you could change the DNS and IP address but that might be even messier). On the other hand if you use the Execution Agent published name to make your connection the actual machine name is hidden from you.

For example, let’s say that “testsys6” has a published name of “QA Test Machine”. If tomorrow testsys50 published that name you wouldn’t need to change anything.

The failover aspects are even more important. While we will talk about generic queue usage and failover, if you use a published name for your execution machine you allow greater failover possibilities than tying your execution queue to a specific machine. In the above example, we’re connecting to “QA Test Machine”. We don’t care what the actual machine is.

TLS Connection

For many customers the Job Scheduler and Execution Machines are located within physically secured premises. With more and more cloud computing and hosted configurations this is rapidly not becoming the standard case. In this situation we need to be concerned about sensitive information being sent from a Job Scheduler to an Execution Agent machine as well as confirming the identity of the respective machines. ActiveBatch supports TLS (sometimes referred to as SSL) encryption for the communications between these two services. ActiveBatch Administrators should read the TLS and Certificates section in the ActiveBatch Installation and Administration Guide for configuration and certificate information. We will continue this discussion by focusing on the enabling of TLS encryption on a specific Execution Queue.

The above figure represents the Execution Queue\Properties dialog. Clicking the “Secured” checkbox inserts the “tls://” prefix. If certificate validation for identity confirmation is desired, you must specify a complete FQDN specification.

Job Limit

By default, the job limit for an execution queue is set to five (5). This value can be made higher or lower. The higher you make this value the more jobs will be able to run on this queue (and machine). The value should be made appropriate to the hardware and operating system platform you’re running. The hardware mention is obvious. You need sufficient CPU, memory and disk to be able to run many jobs. The operating system platform may not be so obvious. Microsoft Windows in particular has several limitations that it imposes based on the platform (i.e. Professional or Server). Some of the limitations have to do with desktop heap and are hard and fast limits. For example, you start running jobs and they fail with an error 128 (no more children present). This error typically means that Windows ran out of desktop heap and the process cannot be created.

Queue Priority Fence

A Queue Priority Fence allows you to indicate to the ActiveBatch Job Scheduler that only jobs with a queue priority that is higher than the fence may be eligible for further dispatch. By default, the queue priority fence is zero (0). This means that all jobs are eligible. The purpose of erecting this fence is to allow only critical jobs (or SLA sensitive jobs) to be dispatch when a time deadline is in effect. For example, setting the fence to 50 would mean that only jobs with a queue priority of 51 or greater may be dispatched. Less queue priority jobs would simply remain on queue. ActiveBatch does make use of this facility as part of the “Take Actions” functionality of the SLA facility. In this case, a Queue Priority Fence is erected with a value of 100 and SLA jobs that are in danger of breaching their deadlines are queued at 110. This means that critical jobs will be dispatched ahead of non-critical ones.

Queue and User-level Variables

Normally Active Variables are created at the Plan or Job level, however, sometimes it can be useful to have variables that assume values at the machine or user level. That is the purpose behind Queue and User Account level Variables. When a variable is defined in multiple places ActiveBatch resolves the variable in the following order of precedence: Plan, Job, User and Queue. This order can yield some important benefits. For example, let’s say we have an output directory that we would like to set (C:\OUTPUT) for most cases. However, sometimes on a specific machine or two, we would like the data output to (C:\DATA) and for some Users we would refer like the data output to (C:\Users\JohnDoe\My Documents).

At the Job level our variable would be defined as:

At the User level, the variable would be defined as:

Lastly, at the Queue level, the variable would be defined as:

Generic Queues

Generic Queues are essentially virtual machines. You associate a job with a generic queue and one of the member execution queues will eventually run the job. There are many advantages to using generic queues. The mini-case studies that follow will explore this fascinating object.

First, a generic queue consists of one or more execution queues (a generic queue can actually consist of no member execution queues). Second, an execution queue can be a member of multiple generic queues. A job can be queued to run on one machine or to be broadcast to multiple machines. For this discussion we’ll take the more common task, which is to have the job dispatched to a single machine. When a job is queued to a generic queue, several checks are performed regardless of scheduling algorithm. The main overriding concept in ActiveBatch scheduling is to get jobs started as soon as possible. First, the execution queue must be in the started state. Queues that are stopped or starting are ignored in terms of potential job dispatch. Next, each execution queue is examined in terms of executing and pending jobs. There is no point in dispatching a job to an execution queue that has reached its capacity. Next, any machine or user characteristics associated with the job are compared to the list of remaining eligible queues. Finally, the generic queue scheduling algorithm is applied. Three (3) algorithms are available: Sequential, Round Robin and Load Balancing. Sequential means that jobs are dispatched to the first execution queue member until it reaches its maximum job limit. Then the next execution queue is selected. Round Robin means that each execution queue is selected and a job is dispatched in turn. Load Balancing means that jobs are dispatched to the least loaded execution queue (machine). As you can imagine Load Balancing is the most preferred scheduling algorithm.

Our first problem is to run a job on one of three (3) possible machines. For this statement to be true, the jobs resources and requirements must be available from any of the 3 machines. First, let’s create our generic queue.

After naming the queue, we move to the Generic Queue Properties page and click the Associate button. The figure above shows three (3) execution queues. We’ll select all of them and click OK.

Then we’ll select Load Balance for the scheduling algorithm and click Save and Close to create the queue. The queue name, for this case study, is GenericQueue1.

This Generic Queue is now available for assignment. The Load Balance algorithm is one of the Just-in-Time (JIT) scheduling algorithms that ActiveBatch makes available. JIT is useful when the number of jobs for a given time period can vary and/or when the number of machines available (ie. Execution Queues) may change.

You may notice a checkbox for Heuristic Queue Allocation (HQA). We’ll explain this algorithm is greater detail in a subsequent section but this algorithm is useful when AI and Machine Learning can be deployed to determine the best machine to use ahead of time (in other words, by predictive analysis and not using JIT).

Managed Queue

A Managed Queue is a special type of Generic Queue. With virtualization and cloud computing, execution machines can be provisioned as needed. ActiveBatch is in the best position to determine when workflows will execute and to provision machines as necessary.

The figure above illustrates a cloud provisioned managed generic queue. You can specify a non-zero value for requested machines in which case that number will be immediately provisioned. Several unique aspects of our support of Amazon EC2 include Placement and Availability Zones. A managed queue can support multiple availability zones. This ensures that as machines are provisioned they are provisioned across availability zones within a region.

The screenshots below illustrate a multiple placement Managed Queue using Amazon EC2. The region is us-east-1.

The first placement is us-east-1a. Note that each placement receives a security group and IP subnet.

The second placement is us-east-1b. The machine provisioning round-robins between the different placements and availability zones. The benefit is that the loss of any single availability zone will not affect availability for the Generic Queue.

While ActiveBatch tags the instances that it creates you can also tag the instances. This helps keep track of who and why the instance was created. This can also help with your permission level security using Amazon’s IAM.

The extract above was taken from the AWS Console. Note the different availability zones.

The extract above was taken from the Tags view of thes same console. You can create your own tags as well.

Heuristic Queue Allocation

ActiveBatch provides several scheduling algorithms as it relates to Generic Queues. Sequential, Round Robin and Load Balancing are three (2) scheduling algorithms that are classified as Just-in-Time (JIT). This means that the job isn’t examined as to which Execution Queue it should be assigned to until the job is ready to be dispatched for execution. JITbased algorithms are excellent when the workload varies greatly and also for reactive scheduling when execution machines have failed and aren’t available as planned.

Heuristic Queue Allocation (HQA) is an algorithm that uses Artificial Intelligence and Machine Learning to perform Predictive Analysis as to which execution machines (and queues) would be best used to handle a series of workflows. This algorithm’s analysis will grow over time but the main aspects as to examine patterns of execution for jobs and to then in a predictive fashion assign jobs to the available members of a Generic Queue. To handle this predictive analysis, the HQA process executes at least one (1) hour prior to the start of a day. The process performs its work and assigns a preferred queue (in a manner similar to ABAT$QAFFINITY) to each eligible job. We say “preferred” because if for some the machine is not available then the JIT algorithm specified with the Generic Queue will assign a new execution queue at dispatch. The ActiveBatch Installation Manual and Reference Manuals provide more detail on this feature.

A special view is available named “Heuristic Queue Allocation” that shows you all the Jobs and their Execution Queues as assigned by the HQA process.

This view also allows you to override execution queue assignments if needed.

Characteristics

In the earlier section the job was dispatched solely based on ActiveBatch considerations. Now let’s explore influencing the scheduling decision. Suppose I had a generic queue with 100 execution queues. These 100 queues represented all the machines in my company. Now, I need to run a job on all machines running Windows Server 2003 specifically. This is where the characteristics feature of the product can be used. Characteristics refer to machine and/or user-created names and values that can be used to further refine the eligibility to possible machines that can be used for execution.

Machine Characteristics

In the above figure you can see an extract of some of the machine characteristics for this queue/machine. The characteristic operating system is OS. This particular execution queue happens to be a “Windows 10” machine. Now we need to associate that characteristic with our job. For that we move to the job’s Generic Queue Properties panel.

Remember that for you to associate characteristics and values at the job level that job must be associated with a generic queue. A typical novice mistake is to associate the job with an execution queue and then not understand why ActiveBatch prevents you from adding or modifying characteristics.

In the above figure we’re going to add the OS characteristic to our job. As the characteristic is a string type, the value we enter is a string. In our case we want to run this job only on machines that run Windows 10. When we click OK that characteristic will be added to the job.

We could have also added various other machine characteristics, for example, number of processors, memory, CPU speed, etc.

User Characteristics (Static)

While machine characteristics offer some scheduling flexibility, often a customer will want to schedule jobs to machine based on their own characteristics. ActiveBatch supports this through the User Characteristics feature. A user characteristic is simply a user-created name and value that is to represent a given execution queue. A job may then perform comparison operations to determine whether a queue (machine) can be used for job execution. For example, let’s assume we needed to run a job on all machines located in the Northeast. Now I know what you’re saying. “Why not just create another generic queue with those machines?” and you’d be right. You can always create as many generic queues as you need to refine the set of execution machines. In the earlier machine characteristics example we could have created a generic queue consisting of all Windows 10 machines. The problem with that approach is that you create more work than you solve. Machine operating systems change over time and you would have to remember to reconfigure the generic queues and their members. With the characteristic approach you change the execution queue; the generic queue and most important the job(s) don’t need to change. Continuing with our example, we could create a user characteristic named Region and assign a value to that characteristic for each execution queue. Some could be Northeast, some could be Mid Atlantic, etc.

The figure above shows the user characteristic Region with the assigned value for this queue. At the job level we would associate a user characteristic and indicate the value we needed for this particular job.

The figure above shows the user characteristic dialog at the job level. The Name field contains a dropdown selector that will display all the possible user characteristics you can select. The user characteristics are essentially a union of all the execution queue members. In this case we select Region and specify the value we are interested in matching. This job will only run on execution queues in the “Northeast”. You can combine machine and user characteristics to further refine the job dispatch scheduling. Remember that with characteristics you’re refining the list of eligible queues (machines). Once that’s done, the scheduling algorithm for the generic queue kicks in. If you selected Load Balance, a winning queue will be selected based on the lightest loaded machine.

User Characteristics (Dynamic)

In the preceding section, a static User Characteristic was used. Static characteristics have their advantages and disadvantages. In the preceding example, the machine is unlikely to move from the Northeast so the characteristic lends itself to being declared statically. However, what if we need to search for something or the presence or absence of a given resource may or may not be available. This would be much more difficult to solve statically. Now this sounds like a resource, but resources are verified once they are dispatched to a machine. What if we wanted to influence the selection of the machine itself through this “resource”? That is the purpose of the dynamic User Characteristic.

For example, I have a job that requires Microsoft SQL Server 2016 be installed across a set of Execution Queues

(machines). As long as the machine has the requisite software, that machine can be eligible in terms of selection for that job. This is an excellent example where a dynamic User Characteristic is best used.

For this example, we’ll use our GQ1 generic queue and add the following User Characteristic:

The characteristic “SQL 2016” represents a Boolean characteristic that indicates whether that software is installed and present on the member execution queues of the generic queue. This short Powershell script tests for a registry key in “HKLM:\Software\Microsoft\Microsoft SQL Server\130” (Note: 130 represents SQL Server 2016). If the key is present then True is returned; otherwise False. This characteristic will be checked every 60 seconds (in reality it could be a much longer test like every 30 or 60 minutes).

GQ1 has two (2) Execution Queues: Server1 and Zeus. Both are Windows systems.

Next, we’ll create a new job named TestChars. We’re not really interested in what this job runs. We’re more interested in demonstrating this capability. So within TestChars, click on the Generic Queue category and then click the Add button for User Characteristics:

This is the condition that we’re looking for. When this condition is met, by at least one (1) Execution Queue, the job will run. What if the job doesn’t run? That’s when you would use the “View Values” button for each Execution Queue member. For example, to look at Zeus values, click on Properties and then look for the View Values button down by User Characteristics.

You can see a reference to the Region characteristic in addition to the “SQL 2016” characteristic that was provided at the Generic Queue level.

Copy File and Script Job

Load balancing and picking a machine to run a job sometimes belies a more fundamental problem. Is the script or program located where every machine can get access to it? Using UNC paths is one approach to ensuring that every machine has access to the underlying file, however, that’s not the only approach. Two other methods are available to you: Copy File and Script job. The copy file feature can be found in the Process type job dialog (see Job Properties).

By default, ActiveBatch expects file access to be made from the perspective of the execution machine. In the above figure, the c:\activebatchtutorial\dir.dat file must be located on the execution machine (that’s because the C drive reference is local). The same would be true for a UNC specification. That specification must be accessible from the execution machine. A typical novice mistake is to specify a file located on the client machine (an easy mistake to make when the file references a local disk). However, ActiveBatch does support a feature named “Copy Script” (or file). This feature looks for the file from the client machine’s perspective and then copies the file to the Job Scheduler machine. Whenever the job needs to be dispatched to an execution machine, ActiveBatch first copies the file from the Job Scheduler machine to the execution machine and then executes the job. This feature has a great benefit of not having to worry about pre-placement of the file since ActiveBatch will always ensure the file is accessible.

The second approach is Script job. A script job is where the script information is embedded within the job. For example, instead of creating a Process type job with a separate VBS file you could create a Script job and have the VBS script be a part of the job itself.

The above figure shows a very simple VBS script. The actual script language itself could have been anything that is supported on the target execution machine; for example, it could be a C-Shell or Perl script. The major advantages to a script type job are (1) you don’t need to worry about moving the file to the target execution machine and (2) any updates to make to the script are immediately available for use.

Job Broadcast

In all the previous sections in which we’ve discussed a generic queue the idea has always been to ultimately execute a job on one execution machine. That is typically what’s required, you need to run this job on one machine and ActiveBatch selects the machine based on availability, characteristics and the specified scheduling algorithm.

The above figure shows the property Queue Selection Method (see Job\Generic Queue Properties). The value “Any Execution Queue” means any one eligible execution queue. The “All Execution Queues” setting enables the Job Broadcast feature. Job Broadcast is the complement to “one eligible execution queue”. In the Job Broadcast mode, the job will be executed on all eligible queues. Note that the Generic Queue scheduling algorithm isn’t used for job broadcast. In our earlier examples, we wanted to run a job on one of a set of machines that represented the Northeast region. Now we can run a job on all Northeast region machines. Availability of the execution machine is not considered. In other words, if the machine is down and the queue is in a starting mode, the job will be on queue and obey any job operating constraints (for example, maximum dispatch time).

This facility can be quite powerful. Let’s say you needed to update a price list on all the sales laptop computers. You would create a generic queue with all the sales computers as execution queues. You write a script that pulls the file down to the device/directory of the executing machine. Normally you would have several problems to deal with. One, what if the execution machine isn’t available? Two, how do I get my script on to that machine? ActiveBatch directly addresses both problems. If the execution machine isn’t available, the job will simply queue up and execute when it is available (unless, as previously noted, a job constraint prevents a long delay). For the script availability issue you have your choice of the two methods for ensuring the script is present on the execution machine.

On-Demand Queue

While there are two types of user created queues ActiveBatch does create a built-in queue named On-Demand. The OnDemand queue is similar to a Generic Queue in that it represents no machine. However, unlike a Generic Queue, the OnDemand queue has special properties and features.

The On-Demand queue can be a “resting” place for jobs. Since by definition no execution machine is referenced the job cannot be manually triggered without also specifying a queue (which does require the Trigger with Queue permission).
If a job or plan triggers a job that is currently on the On-Demand queue, that job inherits the predecessor’s queue.

For example, JobA on queue Server1 triggers JobB on the On-Demand queue. JobB will now run on queue Server1. This last point provides a tremendous benefit to job designers. Only the first job in a stream need be associated with an actual queue (Generic or Execution). This means that a job stream of any size need not be tied to a specific execution queue (even if the job could currently only run on that queue).

Runtime Queue

The RuntimeQueue allows a designer the ability to specify the job associated queue at run-time through a special ActiveBatch variable named #queue. For example, suppose Job1, running on Server1, performs some processing but then needs to tell Job2 (the next job in the stream) to execute on queue “Server2”. While we could use the Trigger facility the downside of that operation is that a new batch run will be created and no variable sharing can occur. Instead the best approach would be for Job1 to set a variable that would tell Job2 what queue to execute on. That’s the purpose of the RuntimeQueue and the special #queue variable. To enable #queue variable support for a job you must associate the job with the RuntimeQueue. You would then associate one or more queue objects (or even better an Object List object if you had a lot of objects to manage) with your job. ActiveBatch would verify that the variable value is one of those objects. If it wasn’t, you would be denied access.

For example, let’s say I want to run my job on either Server1 or Server2 and make that decision at run-time. The first thing I need to do is to associate those objects with my job. To do that, create a new job and after completing the usual properties (i.e. name, label), click on Use Associations, then click on the Associate button and select the two (2) Execution Queues you’re interested in; click OK. Your display should look similar to this:

Next, select Job Properties (Job Library, Process or Script) and use the Submission Queue dropdown to select RuntimeQueue.

With the queue set, we can now create the special #queue variable.

#queue will now be set to server2 (you must specify the full path to the object) and that queue will be used as the Submission Queue property.

Note: The VariableQueue was replaced by the RuntimeQueue.

Runtime User Account

Analogous to the RuntimeQueue, the RuntimeUserAccount allows you to select security credentials that are appropriate to what you might at run-time. For example, let’s say you need to FTP a file to a server but the hostname might change at run-time. In that case, the User Account object containing those credentials would be incorrect.

By specifying RuntimeUserAccount you can make use of the same facility as RuntimeQueue. For example:

This FTPS job uses variables to determine the target server (FTP_SERVER) and the file to be transferred (FTP_FILE). By selecting ‘RuntimeUserAccount’ you can select the appropriate User Account object through the variable #UserAccount. Just as with RuntimeQueue you select the group of User Account object(s) by placing them in the Associations list.

In this list two (2) objects have been associated FTPAccount and FTPAccount2.

The variable #UserAccount is then set to the full path of the selected object.

Runtime Objects in Job Steps

The preceding sections discussed how you can make changes to an Execution Queue and/or User Account object from within a job. This section will discuss how you can extend this functionality to Job Steps and in so doing reference any ActiveBatch object (not just Queues and User Account objects).

For example, let’s assume you needed to trigger an ActiveBatch job but you weren’t sure until run-time which one.

The solution is actually based on the preceding sections. First, you would place the possible ActiveBatch jobs (or plans) in the Associations list. Next, you would create your own variable and ensure its value is the full path of the object you want to trigger.

The above job step triggers an object that is referenced through the variable ‘TRIGGER_OBJECT’. This variable’s value will be the full path of the object that is to be triggered.

The Associations list contains two (2) possible choices: CaseStudy1 and CaseStudy2. The TRIGGER_OBJECT variable would then be set to the full path of one of the objects.

So far the selections of object has been small; from a couple to several, but what if the list of objects was quite a few. Hundreds in fact. Associating so many objects in a single job or many jobs would be a management nightmare. What if you periodically needed to manage so many lists?

To solve that problem ActiveBatch has an object, appropriately enough named Object List object. The Object List object stores lists of objects. You can associate the Object List object within the Associations area instead of the individual objects. By using an Object List you can effectively share the object in a manner similar to the other objects that are meant to be shared within ActiveBatch. For example, an Alert object allows you to contain many alerts and then share them in a simple, effective and easy to manage manner. The Object List object performs the same purpose.

In the above figure the list of ActiveBatch objects are contained in a single object list named TriggerObjects. Once the Object List object has been created you can associate that object (and its contained objects) through normal association.

For example:

This association of ‘TriggerObjects’ also implies the association of all the objects within the Object List.

Spotted a typo or can’t find help on a certain subject? Reach out to the Documentation team at docsupport@redwood.com.