The HPX runtime has seven thread scheduling policies: priority_local, local, global, abp, abp-priority, hierarchy and periodic priority. These policies can be specified from the command line using the command line option --hpx:queuing. In order to use a particular scheduling policy, the runtime system must be built with the appropriate scheduler flag turned on (e.g. cmake -DHPX_LOCAL_SCHEDULER=ON, see CMake Variables used to configure HPX for more information).

Priority Local Scheduling Policy (default policy)

The priority local scheduling policy maintains one queue per operating system (OS) thread. The OS thread pulls its work from this queue. By default the number of high priority queues is equal to the number of OS threads; the number of high priority queues can be specified on the command line using --hpx:high-priority-threads. High priority threads are executed by any of the OS threads before any other work is executed. When a queue is empty work will be taken from high priority queues first. There is one low priority queue from which threads will be scheduled only when there is no other work.

For this scheduling policy there is an option to turn on NUMA sensitivity using the command line option --hpx:numa-sensitive. When NUMA sensitivity is turned on work stealing is done from queues associated with the same NUMA domain first, only after that work is stolen from other NUMA domains.

Static Priority Scheduling Policy

The static scheduling policy maintains one queue per OS thread from which each OS thread pulls its tasks (user threads). Threads are distributed in a round robin fashion. There is no thread stealing in this policy.

Local Scheduling Policy

The local scheduling policy maintains one queue per OS thread from which each OS thread pulls its tasks (user threads).

Global Scheduling Policy

The global scheduling policy maintains one shared queue from which all OS threads pull tasks.

ABP Scheduling Policy

The ABP scheduling policy maintains a double ended lock free queue for each OS thread. Threads are pushed on the top of the queue, and during work stealing threads are taken from the bottom of the queue.

Priority ABP Scheduling Policy

Priority ABP policy maintains a double ended lock free queue for each OS thread. By default the number of high priority queues is equal to the number of OS threads; the number of high priority queues can be specified on the command line using --hpx:high-priority-threads. High priority threads are executed by the first OS threads before any other work is executed. When a queue is empty work will be taken from high priority queues first. There is one low priority queue from which threads will be scheduled only when there is no other work. For this scheduling policy there is an option to turn on NUMA sensitivity using the command line option --hpx:numa-senstive. When NUMA sensitivity is turned on work stealing is done from queues associated with the same NUMA domain first, only after that work is stolen from other NUMA domains.

Hierarchy Scheduling Policy

The hierarchy policy maintains a tree of work items. Every OS thread walks the tree to obtain new work. Arity of the thread queue tree can be specified on the command line using --hpx:hierarchy-arity (default is 2). Work stealing is done from the parent queue in that tree.

Periodic Priority Scheduling Policy

Maintains one queue of work items (user threads) for each OS thread. Maintains a number of high priority queues (specified by --hpx:high-priority-threads) and one low priority queue. High priority threads are executed by the specified number of OS threads before any other work is executed. Low priority threads are executed when no other work is available.