Chapter 1

Overview and Configuration

1.1 OVERVIEW

The Network Queuing System (NQS) performs batch processing on UNIX. Batch processing is a function that receives and queues requests and processes them sequentially.

NQS is derived from the system developed by the Sterling Software Corporation as part of the Numerical Aerodynamic Simulation (NAS) plan of the National Aeronautics and Space Administration (NASA).

NQS processes commands grouped as a batch job and also processes file output requests of the various exclusive peripheral units. You can submit, end, monitor, limit, and manage these jobs. NQS can also route requests to other computer systems connected to the network.

NQS can transfer or receive requests of all hosts on the network. A local host can end, monitor, limit and manage requests that are submitted to a remote host.

1.2 CONFIGURATION

NQS processes commands grouped in batch requests as well as device requests for printing. You can submit, end, monitor, limit, and manage these batch, network, or device requests. You can also route requests to other systems in a network.

The following list shows the sequence of NQS operations.

1. Create a request.
   Create a shell script file for a batch request. Commands executed in a batch job are included in this request.

2. Confirm valid resource limits.
   When necessary, confirm valid NQS resource limits for the system performing the batch request.

3. Submit the request.
   Submit the batch request.

4. Confirm the status of the request.
   Confirm the status of the submitted batch request as well as the status of the queue to which the request is submitted.

5. Alter request attributes.
   When necessary, change the attributes of a submitted batch request.

6. Transmit messages to a request.
   When necessary, send messages to the output file of the submitted batch request.

7. Terminate a request.
   End the batch request at any time.

NQS configuration consists of requests, queues, and devices. Batch and device requests are the basic units of NQS.

1.2.1 Batch Requests

A batch request processes several commands in batch mode. A batch request can be executed by submitting a UNIX shell script to NQS. Information on NQS operations can be embedded in the first comment block of this script, eliminating cumbersome option specifications. You can specify the request execution environment, although it can also be automatically assigned by NQS. A batch request can also be altered after NQS submission. The most significant request attributes are as follows.

• Request ID
  The system assigns this native ID to a request to enable its specification. It consists of a serial .number concatenated with the host name in the host. It is made unique throughout the network.

• Request name
  When a request is submitted, a request name is assigned to it to allow the user to identify it easily.

• Resource limits
  Lists maximum resources available during request execution. Some of the representative limits are on file size, CPU time, and memory size.

• Intraqueue priority
  Determines the order of requests in a NQS queue. The priority decides the order of invocation, and is not related to priority during request execution.

• nice execution value
  Alters the execution priority of each process in the batch request.

• Execution shell
  Interprets the batch request shell script.

• Execution output file
  Contains results generated by the batch request. By default, output goes to a standard output (stdout) or standard error (stderr) output file.

• Constraining time
  NQS usually invokes a batch request immediately upon submission. However, you can specify the date and time that you want NQS to execute a request.

1.2.1.1 BATCH REQUESTS AND JOB IDS

When NQS invokes a batch request, it assigns the same job ID to all processes comprising that request. NQS manages execution requests. When sending, deleting, or signaling a request, NQS sends a message to all processes that configured that request. The job ID can be accessed via request status display commands (qstat and qstatr). To check the status of the processes constituting the request, make the job ID the key in the ps(1) command.

1.2.1.2 BATCH REQUEST RESOURCE LIMITS

As described earlier, one of the batch request attributes is its resource limits. These limits can be broken into two sets: on a per-request basis, as follows, or on a per-process basis. A request may consist of several processes. Per-request limits are as follows:

• Memory size limit
• CPU time limit
• Temporary-file capacity limit
• Tape drives limit
• Permanent-file capacity limit
• Open file limit
• Process number limit
• File system group 0 (=XMU) limit
• File system group 1 limit
• File system group 2 limit
• File system group 3 limit
• CPU Resident time limit

Per-process limits are as follows.

• Corefile size limit
• Data-segment size limit
• Permanent-file size limit
• Memory size limit
• nice execution value
• Stack segment size limit
• CPU time limit
• Working set limit
• Permanent-file capacity limit
• Open file limit
• Number of CPU limit
• CPU Resident time limit

See the qsub(1) command in this guide for a full description of these attributes.

Unsupported resources are ignored. Therefore, resource limits in request attributes are useless if they go to a host that does not support those limits. Resource limits supported by various hosts can be displayed with the qlimit(1) command.

1.2.2 Device Requests

A device request specifies a device such as a printer or plotter. This request can be executed by submitting data (the print image) for processing to the device instead of submitting a shell script, in the case of a batch request. A device request has the following attributes.

• Intra-queue priority
  The factor that determines the invocation order of requests in a queue, similar to that of a batch request.

• Form name
  Form is a collection of several devices. By appending this attribute, the devices for processing the batch request are limited to those specified in form-name.

• Copy number
  The number of copies that a device prints.

1.2.3 Network Requests

A network request initiates the staging out of NQS stdout/stderr/jor files to specified hosts. When a batch request terminates, NQS creates a network request automatically and submits it to the network queue. You cannot create a network queue by yourself or submit it directly. A network request has the following attributes.

• Intraqueue priority Determines the invocation order of requests in a queue, similar to that of a batch request.

• Event number
  A number that shows the type of file to be staged out. The following three types are currently available:

           29        JOR file
           30        STDOUT file
           31        STDERR file
  

1.2.4 Request Status

A request assumes different statuses between its entry and its elimination. The types of request statuses are as follows.

• RUNNING State
  The request is being executed.

• QUEUED State
  The request is queued and scheduled for execution. It assumes the running state when its turn comes.

• WAITING State
  The request waits until the time when the execution is scheduled to be started according to the -a option.

• HOLDING State
  The request is held by the qhold command.

• SUSPENDING State
  The execution of the request is temporarily interrupted by execution of the qspnd command.

• ROUTING State<BR> The request is being transmitted from the pipe queue to another queue.

• ARRIVING State
  The request is being received from the pipe queue.

• EXITING state
  The stdout, stderr, and JOR files of the request are being sent to the specified host.

• PRE-RUNNING state
  The master request is waiting for the beginning of all related slave requests.

• POST-RUNNING state
  The master request is waiting for the termination of all related slave requests.

PRE-RUNNING and POST-RUNNING states are available only on the cluster system using the NQS/MPI function. For more detailed information, see Section 2.24.

1.2.5 Queues

A NQS queue is a collection of executable requests. A request cannot be submitted if its attributes are not the same as those defined for the queue, or if the queue is unavailable. NQS uses four types of queues: batch, device, pipe, and network.

1.2.5.1 BATCH QUEUES

Batch queues are used solely for batch requests. A batch queue has the following attributes.

• Resource limits
  These are limits to be compared with the resource limits of the request that is submitted. If the resource limits of the request exceed these limits, the request is not queued.

• Interqueue priority
  The execution priority between queues. It determines which queue is to have its requests executed by NQS first.

  The largest value request in the queue is executed first. If the priority is the same between queues, NQS executes requests according to the order in which the request entered the queues.

• Maximum number of requests for simultaneous execution
  The number of requests that a queue can execute at one time. If the number of requests that are currently executing exceeds this number, then the requests that are to be invoked next must wait for the currently executing requests to complete before they can be activated.

• Queue access state
  When users or groups that submit requests to a queue are predefined, an undefined user or group cannot submit requests.

• pipeonly attribute
  A queue with this attribute can only receive requests through the pipe queue.

• Demand delivery function
  A batch queue for load distribution. A function of the demand delivery method is provided in Section 5.11.4. This function is available only on the cluster system.

• Master queue and Slave queue
  A batch queue for NQS/MPI function. A function of the NQS/MPI is provided in Section 2.24. This function is available only on the cluster system.

1.2.5.2 DEVICE QUEUES

Device queues are used for device requests. A device queue has the following attributes.

• Device list
  A list of devices to be used by the device queue. Device requests submitted to the queue are processed by the devices set in the device list. More than one device can be set in the device list. Devices are selected for processing requests according to the order in which they are set in the list.

• Interqueue priority
  The execution priority between queues. This factor determines which queue requests are executed by NQS first.

  The largest value request in the queue is executed first. If the priority is the same between queues, NQS executes requests according to the order in which the request entered the queues.

• Maximum number of requests for simultaneous execution
  The number of requests that a queue can execute at one time. If the number of requests that are currently executed exceeds this number, the requests to be invoked next must wait for the currently executing requests to complete before they can be activated.

• Queue access state
  The users or groups that can submit requests to a queue can be predefined. In this case, an undefined user or group cannot submit requests to the queue.

• pipeonly attribute
  A queue with this attribute can only receive requests from the pipe queue.

1.2.5.3 PIPE QUEUES

Pipe queues are used for routing requests. Requests submitted to this queue are routed to other queues. The routing destinations can be queues on the local host or those on remote hosts in a network. Requests submitted to a remote host must be done through a pipe queue. This queue has the following attributes.

• Destination queue list
  The list of destination queues. Requests submitted to a queue are routed to queues set in this destination queue list. More than one destination queue can be set in the list. Currently, a routable method is used to select the destination queue. If the remote host queue is set as the destination queue, that queue is a network pipe queue.

• Interqueue priority
  The execution priority between queues. This factor determines which queue requests are executed by NQS first.

  The largest value request in the queue is executed first. If the priority is the same between queues, NQS executes requests according to the order in which the request entered the queues.

• Maximum number of requests for simultaneous execution
  The number of requests that a queue can execute at one time. If the number of requests that are currently executed exceeds this number, the requests to be invoked next must wait for the currently executing requests to finish.

• Queue access status
  The users or groups that can submit requests to a queue can be predefined. In such a case, an undefined user or group cannot submit requests to the queue.

• pipeonly attribute
  A queue with this attribute can receive requests through the pipe queue.

• Check function
  Specifying this attribute checks the status of the destination queue before requests are registered in the pipe queue. When both the following conditions are satisfied as a result of the check, the requests are registered in the pipe queue:

  --	The requests can be submitted to and executed on the destination queue.
  --	The request resource limits are less than or equal to the resource limits of the destination queue.

• Staywait
  A request with time specification (-a option) submitted to the pipe queue is kept waiting on that queue.

• Server
  The server is a program that transfers requests to other queues. The destination selection method depends on what program is set.

• Transparent function
  A request can be transferred to a high-speed, low-load, local batch queue (see Section 4.2.8).

• Demand delivery function
  A pipe queue for load distribution. A function of the demand delivery method is provided. See Section 5.11.4.

1.2.5.4 NETWORK QUEUES

Network queues are used for staging out NQS stdout/stderr/jor files. The network queue is used only for NQS to stage out stdout/stderr/jor files to the request owner, so you cannot submit a request directly to the network queues. When the execution of batch requests terminates, NQS automatically creates network requests and submits them to the network queue for the target host. A network queue has the following attributes.

• Interqueue priority
  The execution priority between queues. This factor determines which queue has its request executed by NQS first.

• Maximum number of requests for simultaneous execution
  The number of requests that a queue can execute at one time. If the number of requests that are currently executing exceed this number, the requests to be invoked next must wait for the currently executing requests to complete before they can be activated.

• Destination
  The target host machine where NQS stdout/stderr/jor files are staged out.

1.2.5.5 QUEUE STATUS

Queue status controls whether a request can be queued or executed. The queue status is divided into two properties. The first property concerns the acceptance of requests by a queue. The second property determines whether a request can be executed.

• Property 1
  

  --	CLOSED State Although requests are acceptable by the queue, the NQS daemon is not present (NQS is stopped) and the requests are not queued.
  --	ENABLED State The queue can accept requests and the NQS daemon is present (NQS is operating). This means that requests can be queued.
  --	DISABLED State The queue cannot accept requests.

• Property 2
  

  --	STOPPED State Queued requests not already running are blocked from running, and no requests are currently executing in the queue.
  --	STOPPING State Queued requests not already running are blocked from running, but at least one request is currently executing in the queue.
  --	RUNNING State Queued requests are ready to run and one or more requests are currently running in the queue.
  --	INACTIVE State Queued requests are ready to run but no requests are currently running in the queue.
  --	SHUTDOWN State The NQS daemon is not present (NQS is stopped) and requests are not executed.

1.2.6 Devices

A NQS device processes the NQS device list. Unit and server programs configure the device list that carries out the actual processing. Further, devices may be grouped into forms. These forms restrict the device that can be used when submitting a device request.

1.2.6.1 DEVICE STATUS

The status of a device determines its availability. The device status is divided into two properties. The first property concerns the acceptance of requests by a device. The second property determines whether the device is busy.

• Property 1
  

  --	ENABLED State The device can accept requests.
  --	DISABLED State The device is not accepting requests and is in an idle state.
  --	CLOSED State The device is not accepting requests and is not in an idle state.

• Property 2
  

  --	ACTIVE State The device is busy.
  --	INACTIVE State The device is in an idle state but it can be used.
  --	FAILED State The device is in an idle state and cannot be used.

1.2.6.2 DEVICE FORMS

Devices are grouped into forms. If you specify a form when submitting a device request, the devices used are those specified in that form. Figure 1-1 shows that if you want a request processed by NQS Device 3, the request should be submitted to Device Queue 1 using the Form 2 specification. See the qpr command in Chapter 3, User Commands, for more information.

Figure 1-1 Device Forms