Figure 4-1 NQS Operation
This chapter explains the batch transaction methods.
Figure 4-1 NQS Operation
| Home |
|---|
The following explains the sequence of NQS operations.
Steps 2 through 5 are repeated for each subsequent NQS activation.
This section describes the minimum essential items for activating NQS. The methods of adjusting the NQS system environment in accordance with the operation mode are explained in Chapter 5, Configuration Management.
NQS is constructed using the install tool. (See the printed SUPER-UX Installation Guide.) This section explains how to construct an NQS. Only the superuser can construct an NQS.
Example:
Example:
Example:
If the activating NQS failed, see "logfile."
| Home |
|---|
Once NQS is installed and the nqsdaemon is successfully invoked, the NQS environment must then be defined. The creation of this NQS environment involves the configuration of NQS queues and devices according to the application of various sites and their definition with the qmgr(1M) function. The definition of these queues and devices is performed by the superuser or NQS manager.
This section first explains the methods of designing the configuration of the various NQS queue devices. The methods and purpose of application of each site determine the actual configuration. This section explains general points to be noted when designing an NQS configuration and includes some examples. The actual configuration depends on the individual needs of each site.
This section describes the configuration of the following four types of NQS queues.
The queue configuration method varies with the queue type. The following sections describe each of the four queue types.
The first step in determining the batch queue configuration is to classify the queues by resource limits and maximum number of requests for simultaneous execution. If batch requests that consume a large amount of resources are executed at the same time, the efficiency of the entire system drops. For example, if batch requests consuming a large amount of memory space are executed concurrently, frequent swapping via the kernel is required, higher overhead due to greater time consumption occurs, and a decline in overall in system performance is experienced.
| Home |
|---|
| Queue Type | Priority | Resource Limit |
Maximum Number of Requests for Simultaneous Execution |
|---|---|---|---|
| Queue for small scale jobs | Low | Small | Many (20) |
| Queue for medium scale jobs | Medium | Medium | Few (5) |
| Queue for large scale jobs | High | Large | Small (2) |
In the preceding table, you might have a value of 20 for Many, 5 for Few, and 2 for Small for the Max. Number of Requests for Simultaneous Execution. Queues are classified by Small, Medium, and Large resource limits. They are also classified according to the contents of the resource limits. For example, they are classified as those consuming a lot of CPU time but little memory, and those with a large file size that consume little CPU time. A highly detailed classification results in complicated operation that may cause difficulty in deciding which queue is suitable for activation. It is better to classify generally in the beginning and add special queues later.
The second factor is the classification based on the type of users allowed to use each batch queue. For example, batch queues may be divided into those used by general users and those used by specific project users. Although anyone can submit a batch request to an NQS batch queue, the NQS manager can restrict users who can submit batch requests to that batch queue.
Table 4-2 is similar to Table 4-1 with the addition of the NQS manager's exclusive queue for emergency jobs and the queue for specific projects. The emergency job queue is used for jobs that must be executed as soon as possible. It is recommended that you prepare such a queue.
| Home |
|---|
| Queue Type | Priority | Resource Limit | User Limit | Max. Number of Requests for Simultaneous Execution |
|---|---|---|---|---|
| Queue for emergency jobs | Highest | Infinite | NQS manager | Infinite |
| Queue for specific projects | Medium | Large | Specific project | Small (2) |
| Queue for small scale jobs | Low | Small | None | Many (20) |
| Queue for medium scale jobs | Medium | Medium | None | Few (5) |
| Queue for large scale jobs | High | Large | None | Small (5) |
Example values are shown in parentheses in the right-most column of the table. The values for the actual resource limits and maximum number of requests for simultaneous execution differ according to the system configuration at each site. The values are based on the amount of main memory, the configuration of the file system, the type of batch requests that are executed more frequently, etc.
In NQS, several batch queues can be combined as one and the queue complex restricting the maximum number of batch requests for simultaneous execution can be defined. For example, in the preceding table, since the specific project queue and the general large scale job queue can each execute one large scale job at the same time, the system can execute two large scale jobs concurrently. However, in the case where the system can execute only one large scale job, define the queue complex as the specific project queue and the large scale job queue. By allowing the complex to execute only one request simultaneously, the maximum number of large scale jobs for simultaneous execution by the system can be restricted to one or less. With this restriction, once one batch request from the specific project queue and large scale job queue are being executed, no more batch requests are executed, even if that queue can still execute requests simultaneously.
A device queue processes device requests. As with batch queues, multiple device queues can be created. However, device queue classification differs from batch queue classification in that there is no classification by resource limit or maximum number of requests for simultaneous execution. Instead, the queue is configured based on the device where the requests are to be output. The queue configuration based on access restrictions is the same as that for batch queues. Table 4-3 describes device queue configuration.
| Home |
|---|
| Queue Type | Priority | Device | User Limit |
|---|---|---|---|
| Queue for emergency | Highest | Device 1 | NQS Manager |
| Queue for device 1 | Medium | Device 1 | None |
| Queue for device 1 and 2 |
Low | Device 1 Device 2 |
None |
In this configuration, a device request submitted to the emergency queue or queue for device 1 and device 2 are output to device 1 or device 2, whichever is free. Device requests submitted to the queue for device 1 are only output to device 1. If device 1 is busy, the request is not output until device 1 becomes free.
Table 4-4 lists typical examples of usage of the pipe queue.
| Pipe Queue | Transfer Destination |
|---|---|
| Queue for the automatic distribution of resources |
Batch queue for small-scale jobs Batch queue for medium-scale jobs Batch queue for large-scale jobs |
| Specific purpose queue | Specific purpose queue in host 1 |
| Load distribution queue | Automatic resource distribution queue in host 1 Automatic resource distribution queue in host 2 Automatic resource distribution queue in host 3 |
The first automatic resource distribution queue transfers requests to three batch queues to which individual resource restrictions are applied. When a request is submitted to this pipe queue, it is routed to an appropriate batch queue. This automatic queue frees the user from having to select batch queues.
The second specific purpose queue allows the user to submit a request that can be executed only in a specific host, from other hosts.
For the last load balancing queue, the pipe queue selects a suitable host from multiple destination hosts such that the processing load is distributed evenly among these hosts.
| Home |
|---|
| Priority | Target Host Name | Max. No. of Requests for Simultaneous Forwarding | |
|---|---|---|---|
| Queuefor Local Host (LOC) | HIGH | LOC | INFINITE |
| Queue for Remote Host (RMT1) | MEDIUM | RMT1 | 8 |
| Queue for Remote Host (RMT2) | MEDIUM | RMT2 | 8 |
If network queues are not prepared for each target host, network requests are submitted to the default network queue (DefaultNetQueue). DefaultNetQueue stages out stdout, stderr, and jor files to the host machine to which the parent batch requests were submitted. However, DefaultNetQueue was designed for temporary use when network queues are not yet created or when the network queues are disabled. Therefore, be sure to prepare network queues for all target hosts (including local hosts).
If you do not prepare a network queue for the host to which the stdout/stderr/JOR file is sent, the following warning message appears on the NQS log file.
NQS(WARN):Network queue toward host (mid = xxx) isn't prepared.
xxx is the machine ID of the host in which a network queue is needed.
An NQS device must be configured in accordance with the physical devices (lp, for example) prepared in your host machine. The relations with device queues and forms are also important. Use the example in Figure 4-2 as a reference.
Figure 4-2 NQS Queuing
In this example, device requests submitted to device queue 1 are output to NQS device 1 or NQS device 3, whichever is free. If form 1 is specified when submitting a device request, it is only output to NQS device 1. The same applies to device queue 2, form 2, etc.
| Home |
|---|
The NQS device processes requests (device requests), directing their output to devices such as a printer. This device is not created at all when NQS is first invoked and must be created separately. A device is configured with the following four components.
The name refers to the device name. The special file is the UNIX (/dev/lp etc) device file. Server programs process the request data (lpserver is the standard). forms are described in the next section. A device is created with the create device subcommand of qmgr(1M). However, it is essential to set the forms to be defined in the device as valid, before creating the device. Read the following explanation, set the forms, and create the device.
Example:
# qmgr
Mgr: create device dev1 forms=form1 fullname=/dev/lp \
server=(/usr/lib/nqs/lpserver)
The preceding example creates the following device.
Name dev1 Special file /dev/lp Server /usr/lib/nqs/lpserver Form form1
Device grouping is done with the forms defined creating the devices. Devices with the same defined forms are in the same group (see Figure 4-3).
Figure 4-3 Grouping NQS Devices
First you must set the forms in an effective form list. Since nothing is set in the list when NQS is first activated, forms must be set in the effective form list. Forms must be defined in the devices and no device can be created without first setting the forms. Forms can be set with the set form subcommand of qmgr(1M).
Example:
# qmgr Mgr: set forms form1 form2
In the preceding example, form1 and form2 are valid forms, so form1 and form2 can be defined in the device.
| Home |
|---|
The NQS queue is the most important element in the NQS system. Since this NQS queue does not exist at all when NQS is first activated, it must be created separately. The following queue types perform the various NQS queue functions.
A NQS queue can be created with the create subcommand of the qmgr(1M) command. The following are the create subcommands.
The following examples show how to create the various types of queues. You must provide a queue in accordance with the NQS queue configuration designed in advance.
Example:
# qmgr Mgr: create batch_queue batch1 priority=20
| Home |
|---|
Example:
# qmgr Mgr: create device_queue device1 priority=20
Example:
# qmgr Mgr: create pipe_queue pipe1 priority=20 server=(/usr/lib/nqs/pipeclient)
Example:
# qmgr Mgr: create network_queue net1 destination=[103] priority=20
| Home |
|---|
Specific attributes for each queue type can be specified in NQS queues. The following is an explanation of the unique attributes of each queue.
A batch queue has the following attributes.
Example:
# qmgr Mgr: create batch_queue batch1 priority=20 run_limit=3
| Home |
|---|
A device queue has the following attributes.
The device queue called device1, which has devices dev1 and dev2 to process requests, is created in the following example. Multiple devices can be defined. In this example, the devices are selected according to their order of definition. This means that dev1 is selected first. If dev1 cannot be used, dev2 becomes the next selected device.
Example:
# qmgr Mgr: create device_queue device1 priority=20 device=(dev1, dev2)
A pipe queue has the following attributes.
Example:
# qmgr
Mgr: create pipe_queue pipe1 priority=20 run_limit=3 \
server=(/usr/lib/nqs/pipeclient)
The pipe queue called pipe1 that routes requests to the destination batch1@HOST1 or batch2@HOST1 is created in the following example. Multiple destinations can be defined. The destination is selected according to the order of lexicography. Therefore, batch1@HOST1 is selected as the destination. When requests cannot be submitted to batch1@HOST1, batch2@HOST1 is selected. batch@HOST1 refers to the queue called batch1 on the host HOST1.
Example:
# qmgr
Mgr: create pipe_queue pipe1 priority=20 server=(/usr/lib/nqs/pipeclient) \
destination=(batch1@HOST1, batch2@HOST1)
| Home |
|---|
If these conditions are satisfied, requests are registered in the pipe queue.
This function, however, is valid only for local queues. If this function is not specified, requests are registered in the pipe queue even if they cannot be routed to the destination.
The following example shows the definition of this attribute when the queue is created.
Example:
#qmgr
Mgr: create pipe_queue pipe1 priority=20 \
server=(/usr/lib/nqs/pipeclinet) check
Specify one or more local batch queues for the destination of a pipe queue with this check function. An error occurs if remote host queues, local device queues, or pipe queues are specified.
Therefore, the request is routed to the destination using the load information when the specified time is reached.
Example:
# qmgr
Mgr: create pipe_queue pipe1 priority=20 \
server=(usr/lib/nqs/lbpipeclinet -n 3 -i30) \
destination(batch@host1, pipe1@local, pipe2@local2) \
staywait
A network queue has the following attributes:
Example:
# qmgr
Mgr: create network_queue net1 destination=[103] priority=20 \
run_limit=8 server=(/usr/lib/nqs/netclient)
This procedure creates a network queue called net1 with a target host machine ID of 103. The maximum number of requests for simultaneous execution is 8. If run_limit is not specified, the maximum number of requests for simultaneous execution becomes 1.
| Home |
|---|
A NQS queue complex is used to manage multiple NQS batch queues as a group. This complex is created when trying to restrict the maximum number of requests for simultaneous execution in several batch queues (see Figure 4-4).
Figure 4-4 Creating an NQS Queue Complex
A queue complex can be created with the create complex subcommand of the qmgr(1M) command. In the following example, the queue complex called complex1, comprising the three queues, batch1, batch2 and batch3, is created.
Example:
# qmgr Mgr: create complex=(batch1, batch2, batch3) complex1
The following example defines the maximum number of requests for simultaneous execution in a queue complex. The maximum number of requests for simultaneous execution is three and this value is defined in the queue complex called complex1.
Example:
# qmgr Mgr: set complex run_limit=(3) complex1
| Home |
|---|
A transparent pipe queue can transfer requests to a local batch queue at higher speed with lower load than the conventional pipe queue. For example, the transparent pipe queue is very useful in a load balancing environment (see Section 5.11).
The following figure shows an example of how the batch queue operates.
Figure 4-5 Batch Operation
When receiving a request, the transparent pipe queue checks its own acceptance limit. If there is no problem, the transparent pipe queue tests whether any of the destination queues can accept this request. If no destination queues can accept the request, the transparent pipe queue does not accept the request, either. When the request can be accepted, the destination batch queue directly accepts it.
| Home |
|---|
This section describes the procedure for setting the transparent pipe queue (with qmgr subcommands).
First, create a pipe queue in a normal method.
Mgr: create pipe TransPipe priority=10 server=(/usr/lib/nqs/pipeclient) \
destination=(BatchSmall, BatchMiddle, BatchLarge)
A pipe queue TransPipe has been created. The destinations are batch queues on local machines. The batch queues have a resource limit. The transparent pipe queue quickly and automatically distributes requests to two or more batch queues with different limits.
Next, assign this queue as a transparent pipe queue.
Mgr: set transparent pipe_queue TransPipe
The setting of a transparent pipe queue has been completed. This setting can be checked by qstat -x or qstatq -f.
The transparent pipe queue differs from the conventional pipe queue in these respects:
When it cannot accept a request, the transparent pipe queue returns an (internal) error code, which is different from the error code returned by the conventional pipe queue, as described below:
| Home |
|---|
The NQS network environment must first be constructed in order to use network functions in NQS. The network environment is constructed with the nmapmgr(1M) command. First, the host ID of each machine on the network must be defined. Then register the machine names and host IDs in the network data base.
In the following example, the machine whose name is HOST2 and host ID is 101 is recognized as a machine on the network.
Example:
# nmapmgr NMAPMGR>: add mid 101 HOST2
The management of NQS operations is performed by the user called NQS manager. A superuser has the privileges of an NQS manager, but ordinary users can also be given this right. The NQS operator (the user who is permitted to perform the NQS operations) can also be set. The NQS manager can use all the subcommands of the qmgr(1M) command while the NQS operator can only use certain subcommands. Generally speaking, the NQS operator cannot tamper with the NQS configuration. All these settings can be done with the set manager subcommand of the qmgr(1M).
In the following example, user1 is designated as the NQS manager while user2 is the NQS operator. An NQS manager may be distinguished from an NQS operator by the letter appended to the end of the specified user name, ":m" is assigned to an NQS manager while ":o" is assigned to an NQS operator.
The preparation for NQS application is almost complete at this point. Even if NQS were to stop, the settings thus far would be registered.
Example:
- Setting the NQS manager:
- Setting the NQS operator:
| Home |
|---|
| Contents | Previous Chapter | Next Chapter | Index |