TimeWarp Scheduler

The Scheduler application is the heart of the TimeWarp suite.

It is executed as a subprocess from the Monitor application, and uses simple pipes to communicate with the Monitor.

It also has direct interfaces with the database via the Database module and is responsible for initialising the Archiver processes.

General Operation

Start up

Once the Scheduler application is started the following set up items take place :

  1. A database connection is established.
  2. Archiver processes are started (upto MAX_ARCHIVERS Config item)

  3. A batch Queue is created
  4. A Consumer thread is started for each Archiver Process and passed details of the Archiver Process and batch queue.
  5. The Scheduler executes a blocking read on it's stdin waiting for a "signal" from the Monitor

Main Thread

The main thread of the Scheduler executes as follows :

  1. Receive "signal" on stdin
  2. if signal is "STOP" then stop scheduler programm (this also causes the Archivers to stop).
  3. if signal is not "READY" then ignore and wait for another signal
  4. Read items off the master queue in batches (governed by SCHEDULER_BATCHSIZE and SCHEDULER_STRATEGY Config items and add them to the internal batch queue.

  5. Wait for batch queue to become empty (and read more items off the master queue.
  6. Once the master queue is empty (i.e. no Items with a status of NEW) send an "READY" signal back to the Monitor

Consumer Thread

A Consumer thread is created one per Archiver Process to manage the interface between the Scheduler and Archiver. The Consumer Thread removes one item from the batch queue and passes it to the Archiver Process using a defined Interface. It then waits for the appropriate response from the Archiver Process. On recipt of the response the consumer thread will mark that item as complete in the master queue, and look for another item on the batch queue.

Ready Signal

The Monitor and the Scheduler programms exchange READY signals as a simple handshake to indicate the presence of items on the master queue. The Scheduler generates a READY Signal when it empties the master queue. The Monitor guarantees to only generate a READY signal to the Scheduler when the following conditions are true :

  • There are items on the queue to be processed
  • The Scheduler has previously generated a READY signal, or has not generated a signal before
  • Previous READY Signals from the Monitor have been acknowledged by the Scheduler.

The intent is for the Monitor to only generate a "READY" when it knows the scheduler is waiting, and to avoid sending a READY signal for every new item.

Ready Signal Implementation

In the Monitor the ready signal will be implemented as a simple flag set to true when a READY signal is sent to the Scheduler and set to FALSE whenever the Scheduler sends a READY Signal. It will start as FALSE. The Monitor will only send a READY signal to the scheduler when the flag is set to FALSE and it will then be immediately set to TRUE. On receipt of a READY from the Scheduler, the Monitor must check the queue to see if a new READY signal should be sent immediately. Note : this check could be implemented as a simple count of items added to the queue since the last READY was sent from the Monitor

Scheduler Strategy

Backoff Strategy

TimeWarp/Advanced/Scheduler (last edited 2014-04-01 13:20:31 by anthony-flury)