tbb::internal::generic_scheduler Class Reference

Work stealing task scheduler. More...

#include <scheduler.h>

Inheritance diagram for tbb::internal::generic_scheduler:

Collaboration diagram for tbb::internal::generic_scheduler:

Public Member Functions
bool	is_task_pool_published () const
bool	is_local_task_pool_quiescent () const
bool	is_quiescent_local_task_pool_empty () const
bool	is_quiescent_local_task_pool_reset () const
void	attach_mailbox (affinity_id id)
void	init_stack_info ()
	Sets up the data necessary for the stealing limiting heuristics. More...
bool	can_steal ()
	Returns true if stealing is allowed. More...
void	publish_task_pool ()
	Used by workers to enter the task pool. More...
void	leave_task_pool ()
	Leave the task pool. More...
void	reset_task_pool_and_leave ()
	Resets head and tail indices to 0, and leaves task pool. More...
task **	lock_task_pool (arena_slot *victim_arena_slot) const
	Locks victim's task pool, and returns pointer to it. The pointer can be NULL. More...
void	unlock_task_pool (arena_slot *victim_arena_slot, task **victim_task_pool) const
	Unlocks victim's task pool. More...
void	acquire_task_pool () const
	Locks the local task pool. More...
void	release_task_pool () const
	Unlocks the local task pool. More...
task *	prepare_for_spawning (task *t)
	Checks if t is affinitized to another thread, and if so, bundles it as proxy. More...
void	commit_spawned_tasks (size_t new_tail)
	Makes newly spawned tasks visible to thieves. More...
void	commit_relocated_tasks (size_t new_tail)
	Makes relocated tasks visible to thieves and releases the local task pool. More...
task *	get_task (__TBB_ISOLATION_EXPR(isolation_tag isolation))
	Get a task from the local pool. More...
task *	get_task (size_t T)
	Get a task from the local pool at specified location T. More...
task *	get_mailbox_task (__TBB_ISOLATION_EXPR(isolation_tag isolation))
	Attempt to get a task from the mailbox. More...
task *	steal_task (__TBB_ISOLATION_ARG(arena_slot &victim_arena_slot, isolation_tag isolation))
	Steal task from another scheduler's ready pool. More...
size_t	prepare_task_pool (size_t n)
	Makes sure that the task pool can accommodate at least n more elements. More...
bool	cleanup_master (bool blocking_terminate)
	Perform necessary cleanup when a master thread stops using TBB. More...
void	assert_task_pool_valid () const
void	attach_arena (arena *, size_t index, bool is_master)
void	nested_arena_entry (arena *, size_t)
void	nested_arena_exit ()
void	wait_until_empty ()
void	spawn (task &first, task *&next) __TBB_override
	For internal use only. More...
void	spawn_root_and_wait (task &first, task *&next) __TBB_override
	For internal use only. More...
void	enqueue (task &, void *reserved) __TBB_override
	For internal use only. More...
void	local_spawn (task *first, task *&next)
void	local_spawn_root_and_wait (task *first, task *&next)
virtual void	local_wait_for_all (task &parent, task *child)=0
void	free_scheduler ()
	Destroy and deallocate this scheduler object. More...
task &	allocate_task (size_t number_of_bytes, __TBB_CONTEXT_ARG(task *parent, task_group_context *context))
	Allocate task object, either from the heap or a free list. More...
template<free_task_hint h>
void	free_task (task &t)
	Put task on free list. More...
void	deallocate_task (task &t)
	Return task object to the memory allocator. More...
bool	is_worker () const
	True if running on a worker thread, false otherwise. More...
bool	outermost_level () const
	True if the scheduler is on the outermost dispatch level. More...
bool	master_outermost_level () const
	True if the scheduler is on the outermost dispatch level in a master thread. More...
bool	worker_outermost_level () const
	True if the scheduler is on the outermost dispatch level in a worker thread. More...
unsigned	max_threads_in_arena ()
	Returns the concurrency limit of the current arena. More...
virtual task *	receive_or_steal_task (__TBB_ISOLATION_ARG(__TBB_atomic reference_count &completion_ref_count, isolation_tag isolation))=0
	Try getting a task from other threads (via mailbox, stealing, FIFO queue, orphans adoption). More...
void	free_nonlocal_small_task (task &t)
	Free a small task t that that was allocated by a different scheduler. More...
Public Member Functions inherited from tbb::internal::scheduler
virtual void	wait_for_all (task &parent, task *child)=0
	For internal use only. More...
virtual	~scheduler ()=0
	Pure virtual destructor;. More...

Static Public Member Functions
static bool	is_version_3_task (task &t)
static bool	is_proxy (const task &t)
	True if t is a task_proxy. More...
static generic_scheduler *	create_master (arena *a)
	Initialize a scheduler for a master thread. More...
static generic_scheduler *	create_worker (market &m, size_t index)
	Initialize a scheduler for a worker thread. More...
static void	cleanup_worker (void *arg, bool worker)
	Perform necessary cleanup when a worker thread finishes. More...
static task *	plugged_return_list ()
	Special value used to mark my_return_list as not taking any more entries. More...

Public Attributes
uintptr_t	my_stealing_threshold
	Position in the call stack specifying its maximal filling when stealing is still allowed. More...
market *	my_market
	The market I am in. More...
FastRandom	my_random
	Random number generator used for picking a random victim from which to steal. More...
task *	my_free_list
	Free list of small tasks that can be reused. More...
task *	my_dummy_task
	Fake root task created by slave threads. More...
long	my_ref_count
	Reference count for scheduler. More...
bool	my_auto_initialized
	True if *this was created by automatic TBB initialization. More...
__TBB_atomic intptr_t	my_small_task_count
	Number of small tasks that have been allocated by this scheduler. More...
task *	my_return_list
	List of small tasks that have been returned to this scheduler by other schedulers. More...
Public Attributes inherited from tbb::internal::intrusive_list_node
intrusive_list_node *	my_prev_node
intrusive_list_node *	my_next_node
Public Attributes inherited from tbb::internal::scheduler_state
size_t	my_arena_index
	Index of the arena slot the scheduler occupies now, or occupied last time. More...
arena_slot *	my_arena_slot
	Pointer to the slot in the arena we own at the moment. More...
arena *	my_arena
	The arena that I own (if master) or am servicing at the moment (if worker) More...
task *	my_innermost_running_task
	Innermost task whose task::execute() is running. A dummy task on the outermost level. More...
mail_inbox	my_inbox
affinity_id	my_affinity_id
	The mailbox id assigned to this scheduler. More...
scheduler_properties	my_properties

Static Public Attributes
static const size_t	quick_task_size = 256-task_prefix_reservation_size
	If sizeof(task) is <=quick_task_size, it is handled on a free list instead of malloc'd. More...
static const size_t	null_arena_index = ~size_t(0)
static const size_t	min_task_pool_size = 64

Protected Member Functions
	generic_scheduler (market &)

Friends
template<typename SchedulerTraits >
class	custom_scheduler

Detailed Description

Work stealing task scheduler.

None of the fields here are ever read or written by threads other than the thread that creates the instance.

Class generic_scheduler is an abstract base class that contains most of the scheduler, except for tweaks specific to processors and tools (e.g. VTune). The derived template class custom_scheduler<SchedulerTraits> fills in the tweaks.

Definition at line 115 of file scheduler.h.

Constructor & Destructor Documentation

generic_scheduler()

tbb::internal::generic_scheduler::generic_scheduler ( market &  m )

protected

Definition at line 88 of file scheduler.cpp.

    : my_market(&m)
    , my_random(this)
    , my_ref_count(1)
    , my_small_task_count(1)   // Extra 1 is a guard reference
#if __TBB_SURVIVE_THREAD_SWITCH && TBB_USE_ASSERT
    , my_cilk_state(cs_none)
#endif /* __TBB_SURVIVE_THREAD_SWITCH && TBB_USE_ASSERT */
{
    __TBB_ASSERT( !my_arena_index, "constructor expects the memory being zero-initialized" );
    __TBB_ASSERT( governor::is_set(NULL), "scheduler is already initialized for this thread" );

    my_innermost_running_task = my_dummy_task = &allocate_task( sizeof(task), __TBB_CONTEXT_ARG(NULL, &the_dummy_context) );
    my_properties.outermost = true;
#if __TBB_TASK_PRIORITY
    my_ref_top_priority = &m.my_global_top_priority;
    my_ref_reload_epoch = &m.my_global_reload_epoch;
#endif /* __TBB_TASK_PRIORITY */
#if __TBB_TASK_GROUP_CONTEXT
    // Sync up the local cancellation state with the global one. No need for fence here.
    my_context_state_propagation_epoch = the_context_state_propagation_epoch;
    my_context_list_head.my_prev = &my_context_list_head;
    my_context_list_head.my_next = &my_context_list_head;
    ITT_SYNC_CREATE(&my_context_list_mutex, SyncType_Scheduler, SyncObj_ContextsList);
#endif /* __TBB_TASK_GROUP_CONTEXT */
    ITT_SYNC_CREATE(&my_dummy_task->prefix().ref_count, SyncType_Scheduler, SyncObj_WorkerLifeCycleMgmt);
    ITT_SYNC_CREATE(&my_return_list, SyncType_Scheduler, SyncObj_TaskReturnList);
}

References __TBB_ASSERT, __TBB_CONTEXT_ARG, allocate_task(), tbb::internal::governor::is_set(), ITT_SYNC_CREATE, tbb::internal::scheduler_state::my_arena_index, my_dummy_task, tbb::internal::scheduler_state::my_innermost_running_task, tbb::internal::scheduler_state::my_properties, my_return_list, tbb::internal::scheduler_properties::outermost, and tbb::task::prefix().

Here is the call graph for this function:

Member Function Documentation

acquire_task_pool()

void tbb::internal::generic_scheduler::acquire_task_pool ( ) const

inline

Locks the local task pool.

Garbles my_arena_slot->task_pool for the duration of the lock. Requires correctly set my_arena_slot->task_pool_ptr.

ATTENTION: This method is mostly the same as generic_scheduler::lock_task_pool(), with a little different logic of slot state checks (slot is either locked or points to our task pool). Thus if either of them is changed, consider changing the counterpart as well.

Definition at line 454 of file scheduler.cpp.

                                                       {
    if ( !is_task_pool_published() )
        return; // we are not in arena - nothing to lock
    bool sync_prepare_done = false;
    for( atomic_backoff b;;b.pause() ) {
#if TBB_USE_ASSERT
        __TBB_ASSERT( my_arena_slot == my_arena->my_slots + my_arena_index, "invalid arena slot index" );
        // Local copy of the arena slot task pool pointer is necessary for the next
        // assertion to work correctly to exclude asynchronous state transition effect.
        task** tp = my_arena_slot->task_pool;
        __TBB_ASSERT( tp == LockedTaskPool || tp == my_arena_slot->task_pool_ptr, "slot ownership corrupt?" );
#endif
        if( my_arena_slot->task_pool != LockedTaskPool &&
            as_atomic(my_arena_slot->task_pool).compare_and_swap(LockedTaskPool, my_arena_slot->task_pool_ptr ) == my_arena_slot->task_pool_ptr )
        {
            // We acquired our own slot
            ITT_NOTIFY(sync_acquired, my_arena_slot);
            break;
        }
        else if( !sync_prepare_done ) {
            // Start waiting
            ITT_NOTIFY(sync_prepare, my_arena_slot);
            sync_prepare_done = true;
        }
        // Someone else acquired a lock, so pause and do exponential backoff.
    }
    __TBB_ASSERT( my_arena_slot->task_pool == LockedTaskPool, "not really acquired task pool" );
} // generic_scheduler::acquire_task_pool

References __TBB_ASSERT, tbb::internal::as_atomic(), is_task_pool_published(), ITT_NOTIFY, LockedTaskPool, tbb::internal::scheduler_state::my_arena, tbb::internal::scheduler_state::my_arena_index, tbb::internal::scheduler_state::my_arena_slot, tbb::internal::arena::my_slots, tbb::internal::atomic_backoff::pause(), tbb::internal::arena_slot_line1::task_pool, and tbb::internal::arena_slot_line2::task_pool_ptr.

Referenced by cleanup_master(), get_task(), and prepare_task_pool().

Here is the call graph for this function:

Here is the caller graph for this function:

allocate_task()

task & tbb::internal::generic_scheduler::allocate_task	(	size_t	number_of_bytes,
		__TBB_CONTEXT_ARG(task *parent, task_group_context *context)
	)

Allocate task object, either from the heap or a free list.

Returns uninitialized task object with initialized prefix.

Definition at line 298 of file scheduler.cpp.

                                                                                                           {
    GATHER_STATISTIC(++my_counters.active_tasks);
    task *t;
    if( number_of_bytes<=quick_task_size ) {
#if __TBB_HOARD_NONLOCAL_TASKS
        if( (t = my_nonlocal_free_list) ) {
            GATHER_STATISTIC(--my_counters.free_list_length);
            __TBB_ASSERT( t->state()==task::freed, "free list of tasks is corrupted" );
            my_nonlocal_free_list = t->prefix().next;
        } else
#endif
        if( (t = my_free_list) ) {
            GATHER_STATISTIC(--my_counters.free_list_length);
            __TBB_ASSERT( t->state()==task::freed, "free list of tasks is corrupted" );
            my_free_list = t->prefix().next;
        } else if( my_return_list ) {
            // No fence required for read of my_return_list above, because __TBB_FetchAndStoreW has a fence.
            t = (task*)__TBB_FetchAndStoreW( &my_return_list, 0 ); // with acquire
            __TBB_ASSERT( t, "another thread emptied the my_return_list" );
            __TBB_ASSERT( t->prefix().origin==this, "task returned to wrong my_return_list" );
            ITT_NOTIFY( sync_acquired, &my_return_list );
            my_free_list = t->prefix().next;
        } else {
            t = (task*)((char*)NFS_Allocate( 1, task_prefix_reservation_size+quick_task_size, NULL ) + task_prefix_reservation_size );
#if __TBB_COUNT_TASK_NODES
            ++my_task_node_count;
#endif /* __TBB_COUNT_TASK_NODES */
            t->prefix().origin = this;
            t->prefix().next = 0;
            ++my_small_task_count;
        }
#if __TBB_PREFETCHING
        task *t_next = t->prefix().next;
        if( !t_next ) { // the task was last in the list
#if __TBB_HOARD_NONLOCAL_TASKS
            if( my_free_list )
                t_next = my_free_list;
            else
#endif
            if( my_return_list ) // enable prefetching, gives speedup
                t_next = my_free_list = (task*)__TBB_FetchAndStoreW( &my_return_list, 0 );
        }
        if( t_next ) { // gives speedup for both cache lines
            __TBB_cl_prefetch(t_next);
            __TBB_cl_prefetch(&t_next->prefix());
        }
#endif /* __TBB_PREFETCHING */
    } else {
        GATHER_STATISTIC(++my_counters.big_tasks);
        t = (task*)((char*)NFS_Allocate( 1, task_prefix_reservation_size+number_of_bytes, NULL ) + task_prefix_reservation_size );
#if __TBB_COUNT_TASK_NODES
        ++my_task_node_count;
#endif /* __TBB_COUNT_TASK_NODES */
        t->prefix().origin = NULL;
    }
    task_prefix& p = t->prefix();
#if __TBB_TASK_GROUP_CONTEXT
    p.context = context;
#endif /* __TBB_TASK_GROUP_CONTEXT */
    // Obsolete. But still in use, so has to be assigned correct value here.
    p.owner = this;
    p.ref_count = 0;
    // Obsolete. Assign some not outrageously out-of-place value for a while.
    p.depth = 0;
    p.parent = parent;
    // In TBB 2.1 and later, the constructor for task sets extra_state to indicate the version of the tbb/task.h header.
    // In TBB 2.0 and earlier, the constructor leaves extra_state as zero.
    p.extra_state = 0;
    p.affinity = 0;
    p.state = task::allocated;
    __TBB_ISOLATION_EXPR( p.isolation = no_isolation );
    return *t;
}

References __TBB_ASSERT, __TBB_cl_prefetch, __TBB_ISOLATION_EXPR, tbb::task::allocated, tbb::task::freed, GATHER_STATISTIC, ITT_NOTIFY, my_free_list, my_return_list, my_small_task_count, tbb::internal::NFS_Allocate(), tbb::internal::no_isolation, p, parent, tbb::task::prefix(), quick_task_size, tbb::task::state(), and tbb::internal::task_prefix_reservation_size.

Referenced by generic_scheduler(), and prepare_for_spawning().

Here is the call graph for this function:

Here is the caller graph for this function:

assert_task_pool_valid()

void tbb::internal::generic_scheduler::assert_task_pool_valid ( ) const

inline

Definition at line 298 of file scheduler.h.

{}

Referenced by local_spawn(), prepare_task_pool(), and tbb::task::self().

Here is the caller graph for this function:

attach_arena()

void tbb::internal::generic_scheduler::attach_arena	(	arena *	a,
		size_t	index,
		bool	is_master
	)

Definition at line 40 of file arena.cpp.

                                                                             {
    __TBB_ASSERT( a->my_market == my_market, NULL );
    my_arena = a;
    my_arena_index = index;
    my_arena_slot = a->my_slots + index;
    attach_mailbox( affinity_id(index+1) );
    if ( is_master && my_inbox.is_idle_state( true ) ) {
        // Master enters an arena with its own task to be executed. It means that master is not
        // going to enter stealing loop and take affinity tasks.
        my_inbox.set_is_idle( false );
    }
#if __TBB_TASK_GROUP_CONTEXT
    // Context to be used by root tasks by default (if the user has not specified one).
    if( !is_master )
        my_dummy_task->prefix().context = a->my_default_ctx;
#endif /* __TBB_TASK_GROUP_CONTEXT */
#if __TBB_TASK_PRIORITY
    // In the current implementation master threads continue processing even when
    // there are other masters with higher priority. Only TBB worker threads are
    // redistributed between arenas based on the latters' priority. Thus master
    // threads use arena's top priority as a reference point (in contrast to workers
    // that use my_market->my_global_top_priority).
    if( is_master ) {
        my_ref_top_priority = &a->my_top_priority;
        my_ref_reload_epoch = &a->my_reload_epoch;
    }
    my_local_reload_epoch = *my_ref_reload_epoch;
    __TBB_ASSERT( !my_offloaded_tasks, NULL );
#endif /* __TBB_TASK_PRIORITY */
}

References __TBB_ASSERT, attach_mailbox(), tbb::internal::mail_inbox::is_idle_state(), tbb::internal::scheduler_state::my_arena, tbb::internal::scheduler_state::my_arena_index, tbb::internal::scheduler_state::my_arena_slot, my_dummy_task, tbb::internal::scheduler_state::my_inbox, tbb::internal::arena_base::my_market, my_market, tbb::internal::arena::my_slots, tbb::task::prefix(), and tbb::internal::mail_inbox::set_is_idle().

Referenced by nested_arena_entry().

Here is the call graph for this function:

Here is the caller graph for this function:

attach_mailbox()

void tbb::internal::generic_scheduler::attach_mailbox ( affinity_id  id )

inline

Definition at line 564 of file scheduler.h.

                                                              {
    __TBB_ASSERT(id>0,NULL);
    my_inbox.attach( my_arena->mailbox(id) );
    my_affinity_id = id;
}

References __TBB_ASSERT, tbb::internal::mail_inbox::attach(), id, tbb::internal::arena::mailbox(), tbb::internal::scheduler_state::my_affinity_id, tbb::internal::scheduler_state::my_arena, and tbb::internal::scheduler_state::my_inbox.

Referenced by attach_arena().

Here is the call graph for this function:

Here is the caller graph for this function:

can_steal()

bool tbb::internal::generic_scheduler::can_steal ( )

inline

Returns true if stealing is allowed.

Definition at line 182 of file scheduler.h.

                      {
        int anchor;
        // TODO IDEA: Add performance warning?
#if __TBB_ipf
        return my_stealing_threshold < (uintptr_t)&anchor && (uintptr_t)__TBB_get_bsp() < my_rsb_stealing_threshold;
#else
        return my_stealing_threshold < (uintptr_t)&anchor;
#endif
    }

References __TBB_get_bsp(), and my_stealing_threshold.

Here is the call graph for this function:

cleanup_master()

bool tbb::internal::generic_scheduler::cleanup_master

(

bool

blocking_terminate

)

Perform necessary cleanup when a master thread stops using TBB.

Definition at line 1211 of file scheduler.cpp.

                                                                {
    arena* const a = my_arena;
    market * const m = my_market;
    __TBB_ASSERT( my_market, NULL );
    if( a && is_task_pool_published() ) {
        acquire_task_pool();
        if ( my_arena_slot->task_pool == EmptyTaskPool ||
             __TBB_load_relaxed(my_arena_slot->head) >= __TBB_load_relaxed(my_arena_slot->tail) )
        {
            // Local task pool is empty
            leave_task_pool();
        }
        else {
            // Master's local task pool may e.g. contain proxies of affinitized tasks.
            release_task_pool();
            __TBB_ASSERT ( governor::is_set(this), "TLS slot is cleared before the task pool cleanup" );
            local_wait_for_all( *my_dummy_task, NULL );
            __TBB_ASSERT( !is_task_pool_published(), NULL );
            __TBB_ASSERT ( governor::is_set(this), "Other thread reused our TLS key during the task pool cleanup" );
        }
    }
#if __TBB_ARENA_OBSERVER
    if( a )
        a->my_observers.notify_exit_observers( my_last_local_observer, /*worker=*/false );
#endif
#if __TBB_SCHEDULER_OBSERVER
    the_global_observer_list.notify_exit_observers( my_last_global_observer, /*worker=*/false );
#endif /* __TBB_SCHEDULER_OBSERVER */
#if _WIN32||_WIN64
    m->unregister_master( master_exec_resource );
#endif /* _WIN32||_WIN64 */
    if( a ) {
        __TBB_ASSERT(a->my_slots+0 == my_arena_slot, NULL);
#if __TBB_STATISTICS
        *my_arena_slot->my_counters += my_counters;
#endif /* __TBB_STATISTICS */
        __TBB_store_with_release(my_arena_slot->my_scheduler, (generic_scheduler*)NULL);
    }
#if __TBB_TASK_GROUP_CONTEXT
    else { // task_group_context ownership was not transferred to arena
        default_context()->~task_group_context();
        NFS_Free(default_context());
    }
    context_state_propagation_mutex_type::scoped_lock lock(the_context_state_propagation_mutex);
    my_market->my_masters.remove( *this );
    lock.release();
#endif /* __TBB_TASK_GROUP_CONTEXT */
    my_arena_slot = NULL; // detached from slot
    free_scheduler(); // do not use scheduler state after this point

    if( a )
        a->on_thread_leaving<arena::ref_external>();
    // If there was an associated arena, it added a public market reference
    return m->release( /*is_public*/ a != NULL, blocking_terminate );
}

References __TBB_ASSERT, tbb::internal::__TBB_load_relaxed(), tbb::internal::__TBB_store_with_release(), acquire_task_pool(), EmptyTaskPool, free_scheduler(), tbb::internal::arena_slot_line1::head, tbb::internal::governor::is_set(), is_task_pool_published(), leave_task_pool(), local_wait_for_all(), lock, tbb::internal::scheduler_state::my_arena, tbb::internal::scheduler_state::my_arena_slot, my_dummy_task, my_market, tbb::internal::arena_slot_line1::my_scheduler, tbb::internal::arena::my_slots, tbb::internal::NFS_Free(), tbb::internal::arena::on_thread_leaving(), tbb::internal::arena::ref_external, tbb::internal::market::release(), release_task_pool(), tbb::internal::arena_slot_line2::tail, and tbb::internal::arena_slot_line1::task_pool.

Here is the call graph for this function:

cleanup_worker()

void tbb::internal::generic_scheduler::cleanup_worker ( void *  arg, bool  worker  )

static

Perform necessary cleanup when a worker thread finishes.

Definition at line 1201 of file scheduler.cpp.

                                                               {
    generic_scheduler& s = *(generic_scheduler*)arg;
    __TBB_ASSERT( !s.my_arena_slot, "cleaning up attached worker" );
#if __TBB_SCHEDULER_OBSERVER
    if ( worker ) // can be called by master for worker, do not notify master twice
        the_global_observer_list.notify_exit_observers( s.my_last_global_observer, /*worker=*/true );
#endif /* __TBB_SCHEDULER_OBSERVER */
    s.free_scheduler();
}

References __TBB_ASSERT, and s.

Referenced by tbb::internal::market::cleanup().

Here is the caller graph for this function:

commit_relocated_tasks()

void tbb::internal::generic_scheduler::commit_relocated_tasks ( size_t  new_tail )

inline

Makes relocated tasks visible to thieves and releases the local task pool.

Obviously, the task pool must be locked when calling this method.

Definition at line 616 of file scheduler.h.

                                                                 {
    __TBB_ASSERT( is_local_task_pool_quiescent(),
                  "Task pool must be locked when calling commit_relocated_tasks()" );
    __TBB_store_relaxed( my_arena_slot->head, 0 );
    // Tail is updated last to minimize probability of a thread making arena
    // snapshot being misguided into thinking that this task pool is empty.
    __TBB_store_release( my_arena_slot->tail, new_tail );
    release_task_pool();
}

References __TBB_ASSERT, tbb::internal::__TBB_store_relaxed(), __TBB_store_release, tbb::internal::arena_slot_line1::head, is_local_task_pool_quiescent(), tbb::internal::scheduler_state::my_arena_slot, release_task_pool(), and tbb::internal::arena_slot_line2::tail.

Referenced by prepare_task_pool().

Here is the call graph for this function:

Here is the caller graph for this function:

commit_spawned_tasks()

void tbb::internal::generic_scheduler::commit_spawned_tasks ( size_t  new_tail )

inline

Makes newly spawned tasks visible to thieves.

Definition at line 607 of file scheduler.h.

                                                                     {
    __TBB_ASSERT ( new_tail <= my_arena_slot->my_task_pool_size, "task deque end was overwritten" );
    // emit "task was released" signal
    ITT_NOTIFY(sync_releasing, (void*)((uintptr_t)my_arena_slot+sizeof(uintptr_t)));
    // Release fence is necessary to make sure that previously stored task pointers
    // are visible to thieves.
    __TBB_store_with_release( my_arena_slot->tail, new_tail );
}

References __TBB_ASSERT, tbb::internal::__TBB_store_with_release(), ITT_NOTIFY, tbb::internal::scheduler_state::my_arena_slot, sync_releasing, and tbb::internal::arena_slot_line2::tail.

Referenced by local_spawn().

Here is the call graph for this function:

Here is the caller graph for this function:

create_master()

generic_scheduler * tbb::internal::generic_scheduler::create_master ( arena *  a )

static

Initialize a scheduler for a master thread.

Definition at line 1159 of file scheduler.cpp.

                                                              {
    // add an internal market reference; the public reference is possibly added in create_arena
    generic_scheduler* s = allocate_scheduler( market::global_market(/*is_public=*/false) );
    __TBB_ASSERT( !s->my_arena, NULL );
    __TBB_ASSERT( s->my_market, NULL );
    task& t = *s->my_dummy_task;
    s->my_properties.type = scheduler_properties::master;
    t.prefix().ref_count = 1;
#if __TBB_TASK_GROUP_CONTEXT
    t.prefix().context = new ( NFS_Allocate(1, sizeof(task_group_context), NULL) )
            task_group_context( task_group_context::isolated, task_group_context::default_traits );
#if __TBB_FP_CONTEXT
    s->default_context()->capture_fp_settings();
#endif
    // Do not call init_stack_info before the scheduler is set as master or worker.
    s->init_stack_info();
    context_state_propagation_mutex_type::scoped_lock lock(the_context_state_propagation_mutex);
    s->my_market->my_masters.push_front( *s );
    lock.release();
#endif /* __TBB_TASK_GROUP_CONTEXT */
    if( a ) {
    // Master thread always occupies the first slot
        s->attach_arena( a, /*index*/0, /*is_master*/true );
    s->my_arena_slot->my_scheduler = s;
        a->my_default_ctx = s->default_context(); // also transfers implied ownership
    }
    __TBB_ASSERT( s->my_arena_index == 0, "Master thread must occupy the first slot in its arena" );
    governor::sign_on(s);

#if _WIN32||_WIN64
    s->my_market->register_master( s->master_exec_resource );
#endif /* _WIN32||_WIN64 */
    // Process any existing observers.
#if __TBB_ARENA_OBSERVER
    __TBB_ASSERT( !a || a->my_observers.empty(), "Just created arena cannot have any observers associated with it" );
#endif
#if __TBB_SCHEDULER_OBSERVER
    the_global_observer_list.notify_entry_observers( s->my_last_global_observer, /*worker=*/false );
#endif /* __TBB_SCHEDULER_OBSERVER */
    return s;
}

References __TBB_ASSERT, tbb::internal::allocate_scheduler(), tbb::task_group_context::default_traits, tbb::internal::market::global_market(), tbb::task_group_context::isolated, lock, tbb::internal::scheduler_properties::master, tbb::internal::NFS_Allocate(), tbb::task::prefix(), s, and tbb::internal::governor::sign_on().

Referenced by tbb::internal::governor::init_scheduler(), and tbb::internal::governor::init_scheduler_weak().

Here is the call graph for this function:

Here is the caller graph for this function:

create_worker()

generic_scheduler * tbb::internal::generic_scheduler::create_worker ( market &  m, size_t  index  )

static

Initialize a scheduler for a worker thread.

Definition at line 1146 of file scheduler.cpp.

                                                                             {
    generic_scheduler* s = allocate_scheduler( m );
    __TBB_ASSERT(index, "workers should have index > 0");
    s->my_arena_index = index; // index is not a real slot in arena yet
    s->my_dummy_task->prefix().ref_count = 2;
    s->my_properties.type = scheduler_properties::worker;
    // Do not call init_stack_info before the scheduler is set as master or worker.
    s->init_stack_info();
    governor::sign_on(s);
    return s;
}

References __TBB_ASSERT, tbb::internal::allocate_scheduler(), s, tbb::internal::governor::sign_on(), and tbb::internal::scheduler_properties::worker.

Referenced by tbb::internal::market::create_one_job().

Here is the call graph for this function:

Here is the caller graph for this function:

deallocate_task()

void tbb::internal::generic_scheduler::deallocate_task ( task &  t )

inline

Return task object to the memory allocator.

Definition at line 580 of file scheduler.h.

                                                        {
#if TBB_USE_ASSERT
    task_prefix& p = t.prefix();
    p.state = 0xFF;
    p.extra_state = 0xFF;
    poison_pointer(p.next);
#endif /* TBB_USE_ASSERT */
    NFS_Free((char*)&t-task_prefix_reservation_size);
#if __TBB_COUNT_TASK_NODES
    --my_task_node_count;
#endif /* __TBB_COUNT_TASK_NODES */
}

References tbb::internal::NFS_Free(), p, tbb::internal::poison_pointer(), tbb::task::prefix(), and tbb::internal::task_prefix_reservation_size.

Referenced by free_nonlocal_small_task(), free_scheduler(), and free_task().

Here is the call graph for this function:

Here is the caller graph for this function:

enqueue()

void tbb::internal::generic_scheduler::enqueue ( task &  t, void *  reserved  )

virtual

For internal use only.

Implements tbb::internal::scheduler.

Definition at line 681 of file scheduler.cpp.

                                                                {
    generic_scheduler *s = governor::local_scheduler();
    // these redirections are due to bw-compatibility, consider reworking some day
    __TBB_ASSERT( s->my_arena, "thread is not in any arena" );
    s->my_arena->enqueue_task(t, (intptr_t)prio, s->my_random );
}

References __TBB_ASSERT, tbb::internal::governor::local_scheduler(), and s.

Here is the call graph for this function:

free_nonlocal_small_task()

void tbb::internal::generic_scheduler::free_nonlocal_small_task

(

task &

t

)

Free a small task t that that was allocated by a different scheduler.

Definition at line 373 of file scheduler.cpp.

                                                          {
    __TBB_ASSERT( t.state()==task::freed, NULL );
    generic_scheduler& s = *static_cast<generic_scheduler*>(t.prefix().origin);
    __TBB_ASSERT( &s!=this, NULL );
    for(;;) {
        task* old = s.my_return_list;
        if( old==plugged_return_list() )
            break;
        // Atomically insert t at head of s.my_return_list
        t.prefix().next = old;
        ITT_NOTIFY( sync_releasing, &s.my_return_list );
        if( as_atomic(s.my_return_list).compare_and_swap(&t, old )==old ) {
#if __TBB_PREFETCHING
            __TBB_cl_evict(&t.prefix());
            __TBB_cl_evict(&t);
#endif
            return;
        }
    }
    deallocate_task(t);
    if( __TBB_FetchAndDecrementWrelease( &s.my_small_task_count )==1 ) {
        // We freed the last task allocated by scheduler s, so it's our responsibility
        // to free the scheduler.
        NFS_Free( &s );
    }
}

References __TBB_ASSERT, __TBB_cl_evict, __TBB_FetchAndDecrementWrelease, tbb::internal::as_atomic(), deallocate_task(), tbb::task::freed, ITT_NOTIFY, tbb::internal::NFS_Free(), plugged_return_list(), tbb::task::prefix(), s, tbb::task::state(), and sync_releasing.

Referenced by free_scheduler(), and free_task().

Here is the call graph for this function:

Here is the caller graph for this function:

free_scheduler()

void tbb::internal::generic_scheduler::free_scheduler

(

)

Destroy and deallocate this scheduler object.

Definition at line 261 of file scheduler.cpp.

                                       {
    __TBB_ASSERT( !my_arena_slot, NULL );
#if __TBB_TASK_GROUP_CONTEXT
    cleanup_local_context_list();
#endif /* __TBB_TASK_GROUP_CONTEXT */
    free_task<small_local_task>( *my_dummy_task );

#if __TBB_HOARD_NONLOCAL_TASKS
    while( task* t = my_nonlocal_free_list ) {
        task_prefix& p = t->prefix();
        my_nonlocal_free_list = p.next;
        __TBB_ASSERT( p.origin && p.origin!=this, NULL );
        free_nonlocal_small_task(*t);
    }
#endif
    // k accounts for a guard reference and each task that we deallocate.
    intptr_t k = 1;
    for(;;) {
        while( task* t = my_free_list ) {
            my_free_list = t->prefix().next;
            deallocate_task(*t);
            ++k;
        }
        if( my_return_list==plugged_return_list() )
            break;
        my_free_list = (task*)__TBB_FetchAndStoreW( &my_return_list, (intptr_t)plugged_return_list() );
    }
#if __TBB_COUNT_TASK_NODES
    my_market->update_task_node_count( my_task_node_count );
#endif /* __TBB_COUNT_TASK_NODES */
    // Update my_small_task_count last.  Doing so sooner might cause another thread to free *this.
    __TBB_ASSERT( my_small_task_count>=k, "my_small_task_count corrupted" );
    governor::sign_off(this);
    if( __TBB_FetchAndAddW( &my_small_task_count, -k )==k )
        NFS_Free( this );
}

References __TBB_ASSERT, deallocate_task(), free_nonlocal_small_task(), tbb::internal::scheduler_state::my_arena_slot, my_dummy_task, my_free_list, my_market, my_return_list, my_small_task_count, tbb::internal::NFS_Free(), p, plugged_return_list(), tbb::task::prefix(), and tbb::internal::governor::sign_off().

Referenced by cleanup_master().

Here is the call graph for this function:

Here is the caller graph for this function:

free_task()

template<free_task_hint hint>

void tbb::internal::generic_scheduler::free_task

(

task &

t

)

Put task on free list.

Does not call destructor.

Definition at line 627 of file scheduler.h.

                                           {
#if __TBB_HOARD_NONLOCAL_TASKS
    static const int h = hint&(~local_task);
#else
    static const free_task_hint h = hint;
#endif
    GATHER_STATISTIC(--my_counters.active_tasks);
    task_prefix& p = t.prefix();
    // Verify that optimization hints are correct.
    __TBB_ASSERT( h!=small_local_task || p.origin==this, NULL );
    __TBB_ASSERT( !(h&small_task) || p.origin, NULL );
    __TBB_ASSERT( !(h&local_task) || (!p.origin || uintptr_t(p.origin) > uintptr_t(4096)), "local_task means allocated");
    poison_value(p.depth);
    poison_value(p.ref_count);
    poison_pointer(p.owner);
    __TBB_ASSERT( 1L<<t.state() & (1L<<task::executing|1L<<task::allocated), NULL );
    p.state = task::freed;
    if( h==small_local_task || p.origin==this ) {
        GATHER_STATISTIC(++my_counters.free_list_length);
        p.next = my_free_list;
        my_free_list = &t;
    } else if( !(h&local_task) && p.origin && uintptr_t(p.origin) < uintptr_t(4096) ) {
        // a special value reserved for future use, do nothing since
        // origin is not pointing to a scheduler instance
    } else if( !(h&local_task) && p.origin ) {
        GATHER_STATISTIC(++my_counters.free_list_length);
#if __TBB_HOARD_NONLOCAL_TASKS
        if( !(h&no_cache) ) {
            p.next = my_nonlocal_free_list;
            my_nonlocal_free_list = &t;
        } else
#endif
        free_nonlocal_small_task(t);
    } else {
        GATHER_STATISTIC(--my_counters.big_tasks);
        deallocate_task(t);
    }
}

References __TBB_ASSERT, tbb::task::allocated, deallocate_task(), tbb::task::executing, free_nonlocal_small_task(), tbb::task::freed, GATHER_STATISTIC, h, tbb::internal::local_task, my_free_list, tbb::internal::no_cache, p, tbb::internal::poison_pointer(), poison_value, tbb::task::prefix(), tbb::internal::small_local_task, tbb::internal::small_task, and tbb::task::state().

Referenced by tbb::interface5::internal::task_base::destroy(), tbb::internal::allocate_additional_child_of_proxy::free(), tbb::internal::allocate_continuation_proxy::free(), tbb::internal::allocate_child_proxy::free(), and tbb::internal::auto_empty_task::~auto_empty_task().

Here is the call graph for this function:

Here is the caller graph for this function:

get_mailbox_task()

task * tbb::internal::generic_scheduler::get_mailbox_task

(

__TBB_ISOLATION_EXPR(isolation_tag isolation)

)

Attempt to get a task from the mailbox.

Gets a task only if it has not been executed by its sender or a thief that has stolen it from the sender's task pool. Otherwise returns NULL.

This method is intended to be used only by the thread extracting the proxy from its mailbox. (In contrast to local task pool, mailbox can be read only by its owner).

Definition at line 1107 of file scheduler.cpp.

                                                                                           {
    __TBB_ASSERT( my_affinity_id>0, "not in arena" );
    while ( task_proxy* const tp = my_inbox.pop( __TBB_ISOLATION_EXPR( isolation ) ) ) {
        if ( task* result = tp->extract_task<task_proxy::mailbox_bit>() ) {
            ITT_NOTIFY( sync_acquired, my_inbox.outbox() );
            result->prefix().extra_state |= es_task_is_stolen;
            return result;
        }
        // We have exclusive access to the proxy, and can destroy it.
        free_task<no_cache_small_task>(*tp);
    }
    return NULL;
}

References __TBB_ASSERT, __TBB_ISOLATION_EXPR, tbb::internal::es_task_is_stolen, ITT_NOTIFY, tbb::internal::task_proxy::mailbox_bit, tbb::internal::scheduler_state::my_affinity_id, tbb::internal::scheduler_state::my_inbox, and tbb::internal::mail_inbox::pop().

Here is the call graph for this function:

get_task() [1/2]

task * tbb::internal::generic_scheduler::get_task ( __TBB_ISOLATION_EXPR(isolation_tag isolation)  )

inline

Get a task from the local pool.

Called only by the pool owner. Returns the pointer to the task or NULL if a suitable task is not found. Resets the pool if it is empty.

Definition at line 945 of file scheduler.cpp.

                                                                                          {
    __TBB_ASSERT( is_task_pool_published(), NULL );
    // The current task position in the task pool.
    size_t T0 = __TBB_load_relaxed( my_arena_slot->tail );
    // The bounds of available tasks in the task pool. H0 is only used when the head bound is reached.
    size_t H0 = (size_t)-1, T = T0;
    task* result = NULL;
    bool task_pool_empty = false;
    __TBB_ISOLATION_EXPR( bool tasks_omitted = false );
    do {
        __TBB_ASSERT( !result, NULL );
        __TBB_store_relaxed( my_arena_slot->tail, --T );
        atomic_fence();
        if ( (intptr_t)__TBB_load_relaxed( my_arena_slot->head ) > (intptr_t)T ) {
            acquire_task_pool();
            H0 = __TBB_load_relaxed( my_arena_slot->head );
            if ( (intptr_t)H0 > (intptr_t)T ) {
                // The thief has not backed off - nothing to grab.
                __TBB_ASSERT( H0 == __TBB_load_relaxed( my_arena_slot->head )
                    && T == __TBB_load_relaxed( my_arena_slot->tail )
                    && H0 == T + 1, "victim/thief arbitration algorithm failure" );
                reset_task_pool_and_leave();
                // No tasks in the task pool.
                task_pool_empty = true;
                break;
            } else if ( H0 == T ) {
                // There is only one task in the task pool.
                reset_task_pool_and_leave();
                task_pool_empty = true;
            } else {
                // Release task pool if there are still some tasks.
                // After the release, the tail will be less than T, thus a thief
                // will not attempt to get a task at position T.
                release_task_pool();
            }
        }
        __TBB_control_consistency_helper(); // on my_arena_slot->head
#if __TBB_TASK_ISOLATION
        result = get_task( T, isolation, tasks_omitted );
        if ( result ) {
            poison_pointer( my_arena_slot->task_pool_ptr[T] );
            break;
        } else if ( !tasks_omitted ) {
            poison_pointer( my_arena_slot->task_pool_ptr[T] );
            __TBB_ASSERT( T0 == T+1, NULL );
            T0 = T;
        }
#else
        result = get_task( T );
#endif /* __TBB_TASK_ISOLATION */
    } while ( !result && !task_pool_empty );

#if __TBB_TASK_ISOLATION
    if ( tasks_omitted ) {
        if ( task_pool_empty ) {
            // All tasks have been checked. The task pool should be  in reset state.
            // We just restore the bounds for the available tasks.
            // TODO: Does it have sense to move them to the beginning of the task pool?
            __TBB_ASSERT( is_quiescent_local_task_pool_reset(), NULL );
            if ( result ) {
                // If we have a task, it should be at H0 position.
                __TBB_ASSERT( H0 == T, NULL );
                ++H0;
            }
            __TBB_ASSERT( H0 <= T0, NULL );
            if ( H0 < T0 ) {
                // Restore the task pool if there are some tasks.
                __TBB_store_relaxed( my_arena_slot->head, H0 );
                __TBB_store_relaxed( my_arena_slot->tail, T0 );
                // The release fence is used in publish_task_pool.
                publish_task_pool();
                // Synchronize with snapshot as we published some tasks.
                my_arena->advertise_new_work<arena::wakeup>();
            }
        } else {
            // A task has been obtained. We need to make a hole in position T.
            __TBB_ASSERT( is_task_pool_published(), NULL );
            __TBB_ASSERT( result, NULL );
            my_arena_slot->task_pool_ptr[T] = NULL;
            __TBB_store_with_release( my_arena_slot->tail, T0 );
            // Synchronize with snapshot as we published some tasks.
            // TODO: consider some approach not to call wakeup for each time. E.g. check if the tail reached the head.
            my_arena->advertise_new_work<arena::wakeup>();
        }

        // Now it is safe to call note_affinity because the task pool is restored.
        if ( my_innermost_running_task == result ) {
            assert_task_valid( result );
            result->note_affinity( my_affinity_id );
        }
    }
#endif /* __TBB_TASK_ISOLATION */
    __TBB_ASSERT( (intptr_t)__TBB_load_relaxed( my_arena_slot->tail ) >= 0, NULL );
    __TBB_ASSERT( result || __TBB_ISOLATION_EXPR( tasks_omitted || ) is_quiescent_local_task_pool_reset(), NULL );
    return result;
} // generic_scheduler::get_task

References __TBB_ASSERT, __TBB_control_consistency_helper, __TBB_ISOLATION_EXPR, tbb::internal::__TBB_load_relaxed(), tbb::internal::__TBB_store_relaxed(), tbb::internal::__TBB_store_with_release(), acquire_task_pool(), tbb::internal::arena::advertise_new_work(), tbb::internal::assert_task_valid(), tbb::atomic_fence(), tbb::internal::arena_slot_line1::head, is_quiescent_local_task_pool_reset(), is_task_pool_published(), tbb::internal::scheduler_state::my_affinity_id, tbb::internal::scheduler_state::my_arena, tbb::internal::scheduler_state::my_arena_slot, tbb::internal::scheduler_state::my_innermost_running_task, tbb::task::note_affinity(), tbb::internal::poison_pointer(), publish_task_pool(), release_task_pool(), reset_task_pool_and_leave(), tbb::internal::arena_slot_line2::tail, tbb::internal::arena_slot_line2::task_pool_ptr, and tbb::internal::arena::wakeup.

Here is the call graph for this function:

get_task() [2/2]

task * tbb::internal::generic_scheduler::get_task ( size_t  T )

inline

Get a task from the local pool at specified location T.

Returns the pointer to the task or NULL if the task cannot be executed, e.g. proxy has been deallocated or isolation constraint is not met. tasks_omitted tells if some tasks have been omitted. Called only by the pool owner. The caller should guarantee that the position T is not available for a thief.

Definition at line 893 of file scheduler.cpp.

{
    __TBB_ASSERT( __TBB_load_relaxed( my_arena_slot->tail ) <= T
        || is_local_task_pool_quiescent(), "Is it safe to get a task at position T?" );

    task* result = my_arena_slot->task_pool_ptr[T];
    __TBB_ASSERT( !is_poisoned( result ), "The poisoned task is going to be processed" );
#if __TBB_TASK_ISOLATION
    if ( !result )
        return NULL;

    bool omit = isolation != no_isolation && isolation != result->prefix().isolation;
    if ( !omit && !is_proxy( *result ) )
        return result;
    else if ( omit ) {
        tasks_omitted = true;
        return NULL;
    }
#else
    poison_pointer( my_arena_slot->task_pool_ptr[T] );
    if ( !result || !is_proxy( *result ) )
        return result;
#endif /* __TBB_TASK_ISOLATION */

    task_proxy& tp = static_cast<task_proxy&>(*result);
    if ( task *t = tp.extract_task<task_proxy::pool_bit>() ) {
        GATHER_STATISTIC( ++my_counters.proxies_executed );
        // Following assertion should be true because TBB 2.0 tasks never specify affinity, and hence are not proxied.
        __TBB_ASSERT( is_version_3_task( *t ), "backwards compatibility with TBB 2.0 broken" );
        __TBB_ASSERT( my_innermost_running_task != t, NULL );
        my_innermost_running_task = t; // prepare for calling note_affinity()
#if __TBB_TASK_ISOLATION
        // Task affinity has changed. Postpone calling note_affinity because the task pool is in invalid state.
        if ( !tasks_omitted )
#endif /* __TBB_TASK_ISOLATION */
        {
            poison_pointer( my_arena_slot->task_pool_ptr[T] );
            t->note_affinity( my_affinity_id );
        }
        return t;
    }

    // Proxy was empty, so it's our responsibility to free it
    free_task<small_task>( tp );
#if __TBB_TASK_ISOLATION
    if ( tasks_omitted )
        my_arena_slot->task_pool_ptr[T] = NULL;
#endif /* __TBB_TASK_ISOLATION */
    return NULL;
}

References __TBB_ASSERT, tbb::internal::__TBB_load_relaxed(), tbb::internal::task_proxy::extract_task(), GATHER_STATISTIC, tbb::internal::no_isolation, tbb::internal::poison_pointer(), tbb::internal::task_proxy::pool_bit, and tbb::task::prefix().

Here is the call graph for this function:

init_stack_info()

void tbb::internal::generic_scheduler::init_stack_info

(

)

Sets up the data necessary for the stealing limiting heuristics.

Definition at line 147 of file scheduler.cpp.

                                         {
    // Stacks are growing top-down. Highest address is called "stack base",
    // and the lowest is "stack limit".
    __TBB_ASSERT( !my_stealing_threshold, "Stealing threshold has already been calculated" );
    size_t  stack_size = my_market->worker_stack_size();
#if USE_WINTHREAD
#if defined(_MSC_VER)&&_MSC_VER<1400 && !_WIN64
    NT_TIB  *pteb;
    __asm mov eax, fs:[0x18]
    __asm mov pteb, eax
#else
    NT_TIB  *pteb = (NT_TIB*)NtCurrentTeb();
#endif
    __TBB_ASSERT( &pteb < pteb->StackBase && &pteb > pteb->StackLimit, "invalid stack info in TEB" );
    __TBB_ASSERT( stack_size >0, "stack_size not initialized?" );
    // When a thread is created with the attribute STACK_SIZE_PARAM_IS_A_RESERVATION, stack limit
    // in the TIB points to the committed part of the stack only. This renders the expression
    // "(uintptr_t)pteb->StackBase / 2 + (uintptr_t)pteb->StackLimit / 2" virtually useless.
    // Thus for worker threads we use the explicit stack size we used while creating them.
    // And for master threads we rely on the following fact and assumption:
    // - the default stack size of a master thread on Windows is 1M;
    // - if it was explicitly set by the application it is at least as large as the size of a worker stack.
    if ( is_worker() || stack_size < MByte )
        my_stealing_threshold = (uintptr_t)pteb->StackBase - stack_size / 2;
    else
        my_stealing_threshold = (uintptr_t)pteb->StackBase - MByte / 2;
#else /* USE_PTHREAD */
    // There is no portable way to get stack base address in Posix, so we use
    // non-portable method (on all modern Linux) or the simplified approach
    // based on the common sense assumptions. The most important assumption
    // is that the main thread's stack size is not less than that of other threads.
    // See also comment 3 at the end of this file
    void    *stack_base = &stack_size;
#if __linux__ && !__bg__
#if __TBB_ipf
    void    *rsb_base = __TBB_get_bsp();
#endif
    size_t  np_stack_size = 0;
    void    *stack_limit = NULL;
    pthread_attr_t  np_attr_stack;
    if( 0 == pthread_getattr_np(pthread_self(), &np_attr_stack) ) {
        if ( 0 == pthread_attr_getstack(&np_attr_stack, &stack_limit, &np_stack_size) ) {
#if __TBB_ipf
            pthread_attr_t  attr_stack;
            if ( 0 == pthread_attr_init(&attr_stack) ) {
                if ( 0 == pthread_attr_getstacksize(&attr_stack, &stack_size) ) {
                    if ( np_stack_size < stack_size ) {
                        // We are in a secondary thread. Use reliable data.
                        // IA-64 architecture stack is split into RSE backup and memory parts
                        rsb_base = stack_limit;
                        stack_size = np_stack_size/2;
                        // Limit of the memory part of the stack
                        stack_limit = (char*)stack_limit + stack_size;
                    }
                    // We are either in the main thread or this thread stack
                    // is bigger that that of the main one. As we cannot discern
                    // these cases we fall back to the default (heuristic) values.
                }
                pthread_attr_destroy(&attr_stack);
            }
            // IA-64 architecture stack is split into RSE backup and memory parts
            my_rsb_stealing_threshold = (uintptr_t)((char*)rsb_base + stack_size/2);
#endif /* __TBB_ipf */
            // Size of the stack free part
            stack_size = size_t((char*)stack_base - (char*)stack_limit);
        }
        pthread_attr_destroy(&np_attr_stack);
    }
#endif /* __linux__ */
    __TBB_ASSERT( stack_size>0, "stack size must be positive" );
    my_stealing_threshold = (uintptr_t)((char*)stack_base - stack_size/2);
#endif /* USE_PTHREAD */
}

References __TBB_ASSERT, __TBB_get_bsp(), is_worker(), tbb::internal::MByte, my_market, my_stealing_threshold, and tbb::internal::market::worker_stack_size().

Here is the call graph for this function:

is_local_task_pool_quiescent()

bool tbb::internal::generic_scheduler::is_local_task_pool_quiescent ( ) const

inline

Definition at line 530 of file scheduler.h.

                                                                   {
    __TBB_ASSERT(my_arena_slot, 0);
    task** tp = my_arena_slot->task_pool;
    return tp == EmptyTaskPool || tp == LockedTaskPool;
}

References __TBB_ASSERT, EmptyTaskPool, LockedTaskPool, tbb::internal::scheduler_state::my_arena_slot, and tbb::internal::arena_slot_line1::task_pool.

Referenced by commit_relocated_tasks(), is_quiescent_local_task_pool_empty(), and is_quiescent_local_task_pool_reset().

Here is the caller graph for this function:

is_proxy()

static bool tbb::internal::generic_scheduler::is_proxy ( const task &  t )

inlinestatic

True if t is a task_proxy.

Definition at line 260 of file scheduler.h.

                                          {
        return t.prefix().extra_state==es_task_proxy;
    }

References tbb::internal::es_task_proxy, and tbb::task::prefix().

Referenced by steal_task().

Here is the call graph for this function:

Here is the caller graph for this function:

is_quiescent_local_task_pool_empty()

bool tbb::internal::generic_scheduler::is_quiescent_local_task_pool_empty ( ) const

inline

Definition at line 536 of file scheduler.h.

                                                                         {
    __TBB_ASSERT( is_local_task_pool_quiescent(), "Task pool is not quiescent" );
    return __TBB_load_relaxed(my_arena_slot->head) == __TBB_load_relaxed(my_arena_slot->tail);
}

References __TBB_ASSERT, tbb::internal::__TBB_load_relaxed(), tbb::internal::arena_slot_line1::head, is_local_task_pool_quiescent(), tbb::internal::scheduler_state::my_arena_slot, and tbb::internal::arena_slot_line2::tail.

Referenced by leave_task_pool().

Here is the call graph for this function:

Here is the caller graph for this function:

is_quiescent_local_task_pool_reset()

bool tbb::internal::generic_scheduler::is_quiescent_local_task_pool_reset ( ) const

inline

Definition at line 541 of file scheduler.h.

                                                                         {
    __TBB_ASSERT( is_local_task_pool_quiescent(), "Task pool is not quiescent" );
    return __TBB_load_relaxed(my_arena_slot->head) == 0 && __TBB_load_relaxed(my_arena_slot->tail) == 0;
}

References __TBB_ASSERT, tbb::internal::__TBB_load_relaxed(), tbb::internal::arena_slot_line1::head, is_local_task_pool_quiescent(), tbb::internal::scheduler_state::my_arena_slot, and tbb::internal::arena_slot_line2::tail.

Referenced by get_task(), and prepare_task_pool().

Here is the call graph for this function:

Here is the caller graph for this function:

is_task_pool_published()

bool tbb::internal::generic_scheduler::is_task_pool_published ( ) const

inline

Definition at line 525 of file scheduler.h.

                                                             {
    __TBB_ASSERT(my_arena_slot, 0);
    return my_arena_slot->task_pool != EmptyTaskPool;
}

References __TBB_ASSERT, EmptyTaskPool, tbb::internal::scheduler_state::my_arena_slot, and tbb::internal::arena_slot_line1::task_pool.

Referenced by acquire_task_pool(), cleanup_master(), get_task(), leave_task_pool(), local_spawn(), prepare_task_pool(), and release_task_pool().

Here is the caller graph for this function:

is_version_3_task()

static bool tbb::internal::generic_scheduler::is_version_3_task ( task &  t )

inlinestatic

Definition at line 124 of file scheduler.h.

                                             {
        return (t.prefix().extra_state & 0x0F)>=0x1;
    }

References tbb::task::prefix().

Referenced by prepare_for_spawning().

Here is the call graph for this function:

Here is the caller graph for this function:

is_worker()

bool tbb::internal::generic_scheduler::is_worker ( ) const

inline

True if running on a worker thread, false otherwise.

Definition at line 570 of file scheduler.h.

                                               {
    return my_properties.type == scheduler_properties::worker;
}

References tbb::internal::scheduler_state::my_properties, tbb::internal::scheduler_properties::type, and tbb::internal::scheduler_properties::worker.

Referenced by tbb::internal::market::cleanup(), init_stack_info(), master_outermost_level(), nested_arena_entry(), nested_arena_exit(), and worker_outermost_level().

Here is the caller graph for this function:

leave_task_pool()

void tbb::internal::generic_scheduler::leave_task_pool ( )

inline

Leave the task pool.

Leaving task pool automatically releases the task pool if it is locked.

Definition at line 1133 of file scheduler.cpp.

                                               {
    __TBB_ASSERT( is_task_pool_published(), "Not in arena" );
    // Do not reset my_arena_index. It will be used to (attempt to) re-acquire the slot next time
    __TBB_ASSERT( &my_arena->my_slots[my_arena_index] == my_arena_slot, "arena slot and slot index mismatch" );
    __TBB_ASSERT ( my_arena_slot->task_pool == LockedTaskPool, "Task pool must be locked when leaving arena" );
    __TBB_ASSERT ( is_quiescent_local_task_pool_empty(), "Cannot leave arena when the task pool is not empty" );
    ITT_NOTIFY(sync_releasing, &my_arena->my_slots[my_arena_index]);
    // No release fence is necessary here as this assignment precludes external
    // accesses to the local task pool when becomes visible. Thus it is harmless
    // if it gets hoisted above preceding local bookkeeping manipulations.
    __TBB_store_relaxed( my_arena_slot->task_pool, EmptyTaskPool );
}

References __TBB_ASSERT, tbb::internal::__TBB_store_relaxed(), EmptyTaskPool, is_quiescent_local_task_pool_empty(), is_task_pool_published(), ITT_NOTIFY, LockedTaskPool, tbb::internal::scheduler_state::my_arena, tbb::internal::scheduler_state::my_arena_index, tbb::internal::scheduler_state::my_arena_slot, tbb::internal::arena::my_slots, sync_releasing, and tbb::internal::arena_slot_line1::task_pool.

Referenced by cleanup_master(), and reset_task_pool_and_leave().

Here is the call graph for this function:

Here is the caller graph for this function:

local_spawn()

void tbb::internal::generic_scheduler::local_spawn	(	task *	first,
		task *&	next
	)

Conceptually, this method should be a member of class scheduler. But doing so would force us to publish class scheduler in the headers.

Definition at line 596 of file scheduler.cpp.

                                                              {
    __TBB_ASSERT( first, NULL );
    __TBB_ASSERT( governor::is_set(this), NULL );
#if __TBB_TODO
    // We need to consider capping the max task pool size and switching
    // to in-place task execution whenever it is reached.
#endif
    if ( &first->prefix().next == &next ) {
        // Single task is being spawned
#if __TBB_TODO
        // TODO:
        // In the future we need to add overloaded spawn method for a single task,
        // and a method accepting an array of task pointers (we may also want to
        // change the implementation of the task_list class). But since such changes
        // may affect the binary compatibility, we postpone them for a while.
#endif
        size_t T = prepare_task_pool( 1 );
        my_arena_slot->task_pool_ptr[T] = prepare_for_spawning( first );
        commit_spawned_tasks( T + 1 );
    }
    else {
        // Task list is being spawned
#if __TBB_TODO
        // TODO: add task_list::front() and implement&document the local execution ordering which is
        // opposite to the current implementation. The idea is to remove hackish fast_reverse_vector
        // and use push_back/push_front when accordingly LIFO and FIFO order of local execution is
        // desired. It also requires refactoring of the reload_tasks method and my_offloaded_tasks list.
        // Additional benefit may come from adding counter to the task_list so that it can reserve enough
        // space in the task pool in advance and move all the tasks directly without any intermediate
        // storages. But it requires dealing with backward compatibility issues and still supporting
        // counter-less variant (though not necessarily fast implementation).
#endif
        task *arr[min_task_pool_size];
        fast_reverse_vector<task*> tasks(arr, min_task_pool_size);
        task *t_next = NULL;
        for( task* t = first; ; t = t_next ) {
            // If t is affinitized to another thread, it may already be executed
            // and destroyed by the time prepare_for_spawning returns.
            // So milk it while it is alive.
            bool end = &t->prefix().next == &next;
            t_next = t->prefix().next;
            tasks.push_back( prepare_for_spawning(t) );
            if( end )
                break;
        }
        size_t num_tasks = tasks.size();
        size_t T = prepare_task_pool( num_tasks );
        tasks.copy_memory( my_arena_slot->task_pool_ptr + T );
        commit_spawned_tasks( T + num_tasks );
    }
    if ( !is_task_pool_published() )
        publish_task_pool();
    my_arena->advertise_new_work<arena::work_spawned>();
    assert_task_pool_valid();
}

References __TBB_ASSERT, tbb::internal::arena::advertise_new_work(), assert_task_pool_valid(), commit_spawned_tasks(), tbb::internal::fast_reverse_vector< T, max_segments >::copy_memory(), end, tbb::internal::first(), tbb::internal::governor::is_set(), is_task_pool_published(), min_task_pool_size, tbb::internal::scheduler_state::my_arena, tbb::internal::scheduler_state::my_arena_slot, tbb::task::prefix(), prepare_for_spawning(), prepare_task_pool(), publish_task_pool(), tbb::internal::fast_reverse_vector< T, max_segments >::push_back(), tbb::internal::fast_reverse_vector< T, max_segments >::size(), tbb::internal::arena_slot_line2::task_pool_ptr, and tbb::internal::arena::work_spawned.

Referenced by local_spawn_root_and_wait(), spawn(), and tbb::internal::custom_scheduler< SchedulerTraits >::tally_completion_of_predecessor().

Here is the call graph for this function:

Here is the caller graph for this function:

local_spawn_root_and_wait()

void tbb::internal::generic_scheduler::local_spawn_root_and_wait	(	task *	first,
		task *&	next
	)

Definition at line 652 of file scheduler.cpp.

                                                                            {
    __TBB_ASSERT( governor::is_set(this), NULL );
    __TBB_ASSERT( first, NULL );
    auto_empty_task dummy( __TBB_CONTEXT_ARG(this, first->prefix().context) );
    internal::reference_count n = 0;
    for( task* t=first; ; t=t->prefix().next ) {
        ++n;
        __TBB_ASSERT( !t->prefix().parent, "not a root task, or already running" );
        t->prefix().parent = &dummy;
        if( &t->prefix().next==&next ) break;
#if __TBB_TASK_GROUP_CONTEXT
        __TBB_ASSERT( t->prefix().context == t->prefix().next->prefix().context,
                    "all the root tasks in list must share the same context");
#endif /* __TBB_TASK_GROUP_CONTEXT */
    }
    dummy.prefix().ref_count = n+1;
    if( n>1 )
        local_spawn( first->prefix().next, next );
    local_wait_for_all( dummy, first );
}

References __TBB_ASSERT, __TBB_CONTEXT_ARG, tbb::internal::first(), tbb::internal::governor::is_set(), local_spawn(), local_wait_for_all(), tbb::internal::auto_empty_task::prefix(), and tbb::internal::task_prefix::ref_count.

Referenced by spawn_root_and_wait().

Here is the call graph for this function:

Here is the caller graph for this function:

local_wait_for_all()

virtual void tbb::internal::generic_scheduler::local_wait_for_all ( task &  parent, task *  child  )

pure virtual

Implemented in tbb::internal::custom_scheduler< SchedulerTraits >.

Referenced by cleanup_master(), local_spawn_root_and_wait(), and wait_until_empty().

Here is the caller graph for this function:

lock_task_pool()

task ** tbb::internal::generic_scheduler::lock_task_pool ( arena_slot *  victim_arena_slot ) const

inline

Locks victim's task pool, and returns pointer to it. The pointer can be NULL.

Garbles victim_arena_slot->task_pool for the duration of the lock.

ATTENTION: This method is mostly the same as generic_scheduler::acquire_task_pool(), with a little different logic of slot state checks (slot can be empty, locked or point to any task pool other than ours, and asynchronous transitions between all these states are possible). Thus if any of them is changed, consider changing the counterpart as well

Definition at line 498 of file scheduler.cpp.

                                                                                     {
    task** victim_task_pool;
    bool sync_prepare_done = false;
    for( atomic_backoff backoff;; /*backoff pause embedded in the loop*/) {
        victim_task_pool = victim_arena_slot->task_pool;
        // NOTE: Do not use comparison of head and tail indices to check for
        // the presence of work in the victim's task pool, as they may give
        // incorrect indication because of task pool relocations and resizes.
        if ( victim_task_pool == EmptyTaskPool ) {
            // The victim thread emptied its task pool - nothing to lock
            if( sync_prepare_done )
                ITT_NOTIFY(sync_cancel, victim_arena_slot);
            break;
        }
        if( victim_task_pool != LockedTaskPool &&
            as_atomic(victim_arena_slot->task_pool).compare_and_swap(LockedTaskPool, victim_task_pool ) == victim_task_pool )
        {
            // We've locked victim's task pool
            ITT_NOTIFY(sync_acquired, victim_arena_slot);
            break;
        }
        else if( !sync_prepare_done ) {
            // Start waiting
            ITT_NOTIFY(sync_prepare, victim_arena_slot);
            sync_prepare_done = true;
        }
        GATHER_STATISTIC( ++my_counters.thieves_conflicts );
        // Someone else acquired a lock, so pause and do exponential backoff.
#if __TBB_STEALING_ABORT_ON_CONTENTION
        if(!backoff.bounded_pause()) {
            // the 16 was acquired empirically and a theory behind it supposes
            // that number of threads becomes much bigger than number of
            // tasks which can be spawned by one thread causing excessive contention.
            // TODO: However even small arenas can benefit from the abort on contention
            //       if preemption of a thief is a problem
            if(my_arena->my_limit >= 16)
                return EmptyTaskPool;
            __TBB_Yield();
        }
#else
        backoff.pause();
#endif
    }
    __TBB_ASSERT( victim_task_pool == EmptyTaskPool ||
                  (victim_arena_slot->task_pool == LockedTaskPool && victim_task_pool != LockedTaskPool),
                  "not really locked victim's task pool?" );
    return victim_task_pool;
} // generic_scheduler::lock_task_pool

References __TBB_ASSERT, __TBB_Yield, tbb::internal::as_atomic(), EmptyTaskPool, GATHER_STATISTIC, ITT_NOTIFY, LockedTaskPool, tbb::internal::scheduler_state::my_arena, tbb::internal::arena_base::my_limit, sync_cancel, and tbb::internal::arena_slot_line1::task_pool.

Referenced by steal_task().

Here is the call graph for this function:

Here is the caller graph for this function:

master_outermost_level()

bool tbb::internal::generic_scheduler::master_outermost_level ( ) const

inline

True if the scheduler is on the outermost dispatch level in a master thread.

Returns true when this scheduler instance is associated with an application thread, and is not executing any TBB task. This includes being in a TBB dispatch loop (one of wait_for_all methods) invoked directly from that thread.

Definition at line 550 of file scheduler.h.

                                                             {
    return !is_worker() && outermost_level();
}

References is_worker(), and outermost_level().

Here is the call graph for this function:

max_threads_in_arena()

unsigned tbb::internal::generic_scheduler::max_threads_in_arena ( )

inline

Returns the concurrency limit of the current arena.

Definition at line 574 of file scheduler.h.

                                                        {
    __TBB_ASSERT(my_arena, NULL);
    return my_arena->my_num_slots;
}

References __TBB_ASSERT, tbb::internal::scheduler_state::my_arena, and tbb::internal::arena_base::my_num_slots.

Referenced by tbb::internal::get_initial_auto_partitioner_divisor(), and tbb::internal::affinity_partitioner_base_v3::resize().

Here is the caller graph for this function:

nested_arena_entry()

void tbb::internal::generic_scheduler::nested_arena_entry	(	arena *	a,
		size_t	slot_index
	)

Definition at line 631 of file arena.cpp.

                                                                      {
    __TBB_ASSERT( is_alive(a->my_guard), NULL );
    __TBB_ASSERT( a!=my_arena, NULL);

    // overwrite arena settings
#if __TBB_TASK_PRIORITY
    if ( my_offloaded_tasks )
        my_arena->orphan_offloaded_tasks( *this );
    my_offloaded_tasks = NULL;
#endif /* __TBB_TASK_PRIORITY */
    attach_arena( a, slot_index, /*is_master*/true );
    __TBB_ASSERT( my_arena == a, NULL );
    governor::assume_scheduler( this );
    // TODO? ITT_NOTIFY(sync_acquired, a->my_slots + index);
    // TODO: it requires market to have P workers (not P-1)
    // TODO: a preempted worker should be excluded from assignment to other arenas e.g. my_slack--
    if( !is_worker() && slot_index >= my_arena->my_num_reserved_slots )
        my_arena->my_market->adjust_demand(*my_arena, -1);
#if __TBB_ARENA_OBSERVER
    my_last_local_observer = 0; // TODO: try optimize number of calls
    my_arena->my_observers.notify_entry_observers( my_last_local_observer, /*worker=*/false );
#endif
}

References __TBB_ASSERT, tbb::internal::market::adjust_demand(), tbb::internal::governor::assume_scheduler(), attach_arena(), is_worker(), tbb::internal::scheduler_state::my_arena, tbb::internal::arena_base::my_market, and tbb::internal::arena_base::my_num_reserved_slots.

Here is the call graph for this function:

nested_arena_exit()

void tbb::internal::generic_scheduler::nested_arena_exit

(

)

Definition at line 655 of file arena.cpp.

                                          {
#if __TBB_ARENA_OBSERVER
    my_arena->my_observers.notify_exit_observers( my_last_local_observer, /*worker=*/false );
#endif /* __TBB_ARENA_OBSERVER */
#if __TBB_TASK_PRIORITY
    if ( my_offloaded_tasks )
        my_arena->orphan_offloaded_tasks( *this );
#endif
    if( !is_worker() && my_arena_index >= my_arena->my_num_reserved_slots )
        my_arena->my_market->adjust_demand(*my_arena, 1);
    // Free the master slot.
    __TBB_ASSERT(my_arena->my_slots[my_arena_index].my_scheduler, "A slot is already empty");
    __TBB_store_with_release(my_arena->my_slots[my_arena_index].my_scheduler, (generic_scheduler*)NULL);
    my_arena->my_exit_monitors.notify_one(); // do not relax!
}

References __TBB_ASSERT, tbb::internal::__TBB_store_with_release(), tbb::internal::market::adjust_demand(), is_worker(), tbb::internal::scheduler_state::my_arena, tbb::internal::scheduler_state::my_arena_index, tbb::internal::arena_base::my_exit_monitors, tbb::internal::arena_base::my_market, tbb::internal::arena_base::my_num_reserved_slots, tbb::internal::arena_slot_line1::my_scheduler, tbb::internal::arena::my_slots, and tbb::internal::concurrent_monitor::notify_one().

Referenced by tbb::internal::nested_arena_context::~nested_arena_context().

Here is the call graph for this function:

Here is the caller graph for this function:

outermost_level()

bool tbb::internal::generic_scheduler::outermost_level ( ) const

inline

True if the scheduler is on the outermost dispatch level.

Definition at line 546 of file scheduler.h.

                                                      {
    return my_properties.outermost;
}

References tbb::internal::scheduler_state::my_properties, and tbb::internal::scheduler_properties::outermost.

Referenced by master_outermost_level(), and worker_outermost_level().

Here is the caller graph for this function:

plugged_return_list()

static task* tbb::internal::generic_scheduler::plugged_return_list ( )

inlinestatic

Special value used to mark my_return_list as not taking any more entries.

Definition at line 355 of file scheduler.h.

{return (task*)(intptr_t)(-1);}

Referenced by free_nonlocal_small_task(), and free_scheduler().

Here is the caller graph for this function:

prepare_for_spawning()

task * tbb::internal::generic_scheduler::prepare_for_spawning ( task *  t )

inline

Checks if t is affinitized to another thread, and if so, bundles it as proxy.

Returns either t or proxy containing t.

Definition at line 556 of file scheduler.cpp.

                                                              {
    __TBB_ASSERT( t->state()==task::allocated, "attempt to spawn task that is not in 'allocated' state" );
    t->prefix().state = task::ready;
#if TBB_USE_ASSERT
    if( task* parent = t->parent() ) {
        internal::reference_count ref_count = parent->prefix().ref_count;
        __TBB_ASSERT( ref_count>=0, "attempt to spawn task whose parent has a ref_count<0" );
        __TBB_ASSERT( ref_count!=0, "attempt to spawn task whose parent has a ref_count==0 (forgot to set_ref_count?)" );
        parent->prefix().extra_state |= es_ref_count_active;
    }
#endif /* TBB_USE_ASSERT */
    affinity_id dst_thread = t->prefix().affinity;
    __TBB_ASSERT( dst_thread == 0 || is_version_3_task(*t),
                  "backwards compatibility to TBB 2.0 tasks is broken" );
#if __TBB_TASK_ISOLATION
    isolation_tag isolation = my_innermost_running_task->prefix().isolation;
    t->prefix().isolation = isolation;
#endif /* __TBB_TASK_ISOLATION */
    if( dst_thread != 0 && dst_thread != my_affinity_id ) {
        task_proxy& proxy = (task_proxy&)allocate_task( sizeof(task_proxy),
                                                      __TBB_CONTEXT_ARG(NULL, NULL) );
        // Mark as a proxy
        proxy.prefix().extra_state = es_task_proxy;
        proxy.outbox = &my_arena->mailbox(dst_thread);
        // Mark proxy as present in both locations (sender's task pool and destination mailbox)
        proxy.task_and_tag = intptr_t(t) | task_proxy::location_mask;
#if __TBB_TASK_PRIORITY
        poison_pointer( proxy.prefix().context );
#endif /* __TBB_TASK_PRIORITY */
        __TBB_ISOLATION_EXPR( proxy.prefix().isolation = isolation );
        ITT_NOTIFY( sync_releasing, proxy.outbox );
        // Mail the proxy - after this point t may be destroyed by another thread at any moment.
        proxy.outbox->push(&proxy);
        return &proxy;
    }
    return t;
}

References __TBB_ASSERT, __TBB_CONTEXT_ARG, __TBB_ISOLATION_EXPR, allocate_task(), tbb::task::allocated, tbb::task::context(), tbb::internal::es_ref_count_active, tbb::internal::es_task_proxy, is_version_3_task(), ITT_NOTIFY, tbb::internal::task_proxy::location_mask, tbb::internal::arena::mailbox(), tbb::internal::scheduler_state::my_affinity_id, tbb::internal::scheduler_state::my_arena, tbb::internal::scheduler_state::my_innermost_running_task, tbb::internal::task_proxy::outbox, parent, tbb::task::parent(), tbb::internal::poison_pointer(), tbb::task::prefix(), tbb::internal::mail_outbox::push(), tbb::task::ready, tbb::task::state(), sync_releasing, and tbb::internal::task_proxy::task_and_tag.

Referenced by local_spawn().

Here is the call graph for this function:

Here is the caller graph for this function:

prepare_task_pool()

size_t tbb::internal::generic_scheduler::prepare_task_pool ( size_t  n )

inline

Makes sure that the task pool can accommodate at least n more elements.

If necessary relocates existing task pointers or grows the ready task deque. Returns (possible updated) tail index (not accounting for n).

Definition at line 400 of file scheduler.cpp.

                                                                      {
    size_t T = __TBB_load_relaxed(my_arena_slot->tail); // mirror
    if ( T + num_tasks <= my_arena_slot->my_task_pool_size )
        return T;

    size_t new_size = num_tasks;

    if ( !my_arena_slot->my_task_pool_size ) {
        __TBB_ASSERT( !is_task_pool_published() && is_quiescent_local_task_pool_reset(), NULL );
        __TBB_ASSERT( !my_arena_slot->task_pool_ptr, NULL );
        if ( num_tasks < min_task_pool_size ) new_size = min_task_pool_size;
        my_arena_slot->allocate_task_pool( new_size );
        return 0;
    }

    acquire_task_pool();
    size_t H = __TBB_load_relaxed( my_arena_slot->head ); // mirror
    task** task_pool = my_arena_slot->task_pool_ptr;;
    __TBB_ASSERT( my_arena_slot->my_task_pool_size >= min_task_pool_size, NULL );
    // Count not skipped tasks. Consider using std::count_if.
    for ( size_t i = H; i < T; ++i )
        if ( task_pool[i] ) ++new_size;
    // If the free space at the beginning of the task pool is too short, we
    // are likely facing a pathological single-producer-multiple-consumers
    // scenario, and thus it's better to expand the task pool
    bool allocate = new_size > my_arena_slot->my_task_pool_size - min_task_pool_size/4;
    if ( allocate ) {
        // Grow task pool. As this operation is rare, and its cost is asymptotically
        // amortizable, we can tolerate new task pool allocation done under the lock.
        if ( new_size < 2 * my_arena_slot->my_task_pool_size )
            new_size = 2 * my_arena_slot->my_task_pool_size;
        my_arena_slot->allocate_task_pool( new_size ); // updates my_task_pool_size
    }
    // Filter out skipped tasks. Consider using std::copy_if.
    size_t T1 = 0;
    for ( size_t i = H; i < T; ++i )
        if ( task_pool[i] )
            my_arena_slot->task_pool_ptr[T1++] = task_pool[i];
    // Deallocate the previous task pool if a new one has been allocated.
    if ( allocate )
        NFS_Free( task_pool );
    else
        my_arena_slot->fill_with_canary_pattern( T1, my_arena_slot->tail );
    // Publish the new state.
    commit_relocated_tasks( T1 );
    assert_task_pool_valid();
    return T1;
}

References __TBB_ASSERT, tbb::internal::__TBB_load_relaxed(), acquire_task_pool(), tbb::internal::arena_slot::allocate_task_pool(), assert_task_pool_valid(), commit_relocated_tasks(), tbb::internal::arena_slot::fill_with_canary_pattern(), tbb::internal::arena_slot_line1::head, is_quiescent_local_task_pool_reset(), is_task_pool_published(), min_task_pool_size, tbb::internal::scheduler_state::my_arena_slot, tbb::internal::arena_slot_line2::my_task_pool_size, new_size, tbb::internal::NFS_Free(), tbb::internal::arena_slot_line2::tail, and tbb::internal::arena_slot_line2::task_pool_ptr.

Referenced by local_spawn().

Here is the call graph for this function:

Here is the caller graph for this function:

publish_task_pool()

void tbb::internal::generic_scheduler::publish_task_pool ( )

inline

Used by workers to enter the task pool.

Does not lock the task pool in case if arena slot has been successfully grabbed.

Definition at line 1121 of file scheduler.cpp.

                                                 {
    __TBB_ASSERT ( my_arena, "no arena: initialization not completed?" );
    __TBB_ASSERT ( my_arena_index < my_arena->my_num_slots, "arena slot index is out-of-bound" );
    __TBB_ASSERT ( my_arena_slot == &my_arena->my_slots[my_arena_index], NULL);
    __TBB_ASSERT ( my_arena_slot->task_pool == EmptyTaskPool, "someone else grabbed my arena slot?" );
    __TBB_ASSERT ( __TBB_load_relaxed(my_arena_slot->head) < __TBB_load_relaxed(my_arena_slot->tail),
                   "entering arena without tasks to share" );
    // Release signal on behalf of previously spawned tasks (when this thread was not in arena yet)
    ITT_NOTIFY(sync_releasing, my_arena_slot);
    __TBB_store_with_release( my_arena_slot->task_pool, my_arena_slot->task_pool_ptr );
}

References __TBB_ASSERT, tbb::internal::__TBB_load_relaxed(), tbb::internal::__TBB_store_with_release(), EmptyTaskPool, tbb::internal::arena_slot_line1::head, ITT_NOTIFY, tbb::internal::scheduler_state::my_arena, tbb::internal::scheduler_state::my_arena_index, tbb::internal::scheduler_state::my_arena_slot, tbb::internal::arena::my_slots, sync_releasing, tbb::internal::arena_slot_line2::tail, tbb::internal::arena_slot_line1::task_pool, and tbb::internal::arena_slot_line2::task_pool_ptr.

Referenced by get_task(), and local_spawn().

Here is the call graph for this function:

Here is the caller graph for this function:

receive_or_steal_task()

virtual task* tbb::internal::generic_scheduler::receive_or_steal_task ( __TBB_ISOLATION_ARG(__TBB_atomic reference_count &completion_ref_count, isolation_tag isolation)  )

pure virtual

Try getting a task from other threads (via mailbox, stealing, FIFO queue, orphans adoption).

Returns obtained task or NULL if all attempts fail.

Implemented in tbb::internal::custom_scheduler< SchedulerTraits >.

release_task_pool()

void tbb::internal::generic_scheduler::release_task_pool ( ) const

inline

Unlocks the local task pool.

Restores my_arena_slot->task_pool munged by acquire_task_pool. Requires correctly set my_arena_slot->task_pool_ptr.

Definition at line 483 of file scheduler.cpp.

                                                       {
    if ( !is_task_pool_published() )
        return; // we are not in arena - nothing to unlock
    __TBB_ASSERT( my_arena_slot, "we are not in arena" );
    __TBB_ASSERT( my_arena_slot->task_pool == LockedTaskPool, "arena slot is not locked" );
    ITT_NOTIFY(sync_releasing, my_arena_slot);
    __TBB_store_with_release( my_arena_slot->task_pool, my_arena_slot->task_pool_ptr );
}

References __TBB_ASSERT, tbb::internal::__TBB_store_with_release(), is_task_pool_published(), ITT_NOTIFY, LockedTaskPool, tbb::internal::scheduler_state::my_arena_slot, sync_releasing, tbb::internal::arena_slot_line1::task_pool, and tbb::internal::arena_slot_line2::task_pool_ptr.

Referenced by cleanup_master(), commit_relocated_tasks(), and get_task().

Here is the call graph for this function:

Here is the caller graph for this function:

reset_task_pool_and_leave()

void tbb::internal::generic_scheduler::reset_task_pool_and_leave ( )

inline

Resets head and tail indices to 0, and leaves task pool.

The task pool must be locked by the owner (via acquire_task_pool).

Definition at line 599 of file scheduler.h.

                                                          {
    __TBB_ASSERT( my_arena_slot->task_pool == LockedTaskPool, "Task pool must be locked when resetting task pool" );
    __TBB_store_relaxed( my_arena_slot->tail, 0 );
    __TBB_store_relaxed( my_arena_slot->head, 0 );
    leave_task_pool();
}

References __TBB_ASSERT, tbb::internal::__TBB_store_relaxed(), tbb::internal::arena_slot_line1::head, leave_task_pool(), LockedTaskPool, tbb::internal::scheduler_state::my_arena_slot, tbb::internal::arena_slot_line2::tail, and tbb::internal::arena_slot_line1::task_pool.

Referenced by get_task().

Here is the call graph for this function:

Here is the caller graph for this function:

spawn()

void tbb::internal::generic_scheduler::spawn ( task &  first, task *&  next  )

virtual

For internal use only.

Implements tbb::internal::scheduler.

Definition at line 673 of file scheduler.cpp.

                                                                   {
    governor::local_scheduler()->local_spawn( &first, next );
}

References tbb::internal::first(), tbb::internal::governor::local_scheduler(), and local_spawn().

Here is the call graph for this function:

spawn_root_and_wait()

void tbb::internal::generic_scheduler::spawn_root_and_wait ( task &  first, task *&  next  )

virtual

For internal use only.

Implements tbb::internal::scheduler.

Definition at line 677 of file scheduler.cpp.

                                                                                 {
    governor::local_scheduler()->local_spawn_root_and_wait( &first, next );
}

References tbb::internal::first(), tbb::internal::governor::local_scheduler(), and local_spawn_root_and_wait().

Here is the call graph for this function:

steal_task()

task * tbb::internal::generic_scheduler::steal_task

(

__TBB_ISOLATION_ARG(arena_slot &victim_arena_slot, isolation_tag isolation)

)

Steal task from another scheduler's ready pool.

Definition at line 1042 of file scheduler.cpp.

                                                                                                             {
    task** victim_pool = lock_task_pool( &victim_slot );
    if ( !victim_pool )
        return NULL;
    task* result = NULL;
    size_t H = __TBB_load_relaxed(victim_slot.head); // mirror
    size_t H0 = H;
    bool tasks_omitted = false;
    do {
        __TBB_store_relaxed( victim_slot.head, ++H );
        atomic_fence();
        if ( (intptr_t)H > (intptr_t)__TBB_load_relaxed( victim_slot.tail ) ) {
            // Stealing attempt failed, deque contents has not been changed by us
            GATHER_STATISTIC( ++my_counters.thief_backoffs );
            __TBB_store_relaxed( victim_slot.head, /*dead: H = */ H0 );
            __TBB_ASSERT( !result, NULL );
            goto unlock;
        }
        __TBB_control_consistency_helper(); // on victim_slot.tail
        result = victim_pool[H-1];
        __TBB_ASSERT( !is_poisoned( result ), NULL );

        if ( result ) {
            __TBB_ISOLATION_EXPR( if ( isolation == no_isolation || isolation == result->prefix().isolation ) )
            {
                if ( !is_proxy( *result ) )
                    break;
                task_proxy& tp = *static_cast<task_proxy*>(result);
                // If mailed task is likely to be grabbed by its destination thread, skip it.
                if ( !(task_proxy::is_shared( tp.task_and_tag ) && tp.outbox->recipient_is_idle()) )
                    break;
                GATHER_STATISTIC( ++my_counters.proxies_bypassed );
            }
            // The task cannot be executed either due to isolation or proxy contraints.
            result = NULL;
            tasks_omitted = true;
        } else if ( !tasks_omitted ) {
            // Cleanup the task pool from holes until a task is skipped.
            __TBB_ASSERT( H0 == H-1, NULL );
            poison_pointer( victim_pool[H0] );
            H0 = H;
        }
    } while ( !result );
    __TBB_ASSERT( result, NULL );

    // emit "task was consumed" signal
    ITT_NOTIFY( sync_acquired, (void*)((uintptr_t)&victim_slot+sizeof( uintptr_t )) );
    poison_pointer( victim_pool[H-1] );
    if ( tasks_omitted ) {
        // Some proxies in the task pool have been omitted. Set the stolen task to NULL.
        victim_pool[H-1] = NULL;
        __TBB_store_relaxed( victim_slot.head, /*dead: H = */ H0 );
    }
unlock:
    unlock_task_pool( &victim_slot, victim_pool );
#if __TBB_PREFETCHING
    __TBB_cl_evict(&victim_slot.head);
    __TBB_cl_evict(&victim_slot.tail);
#endif
    if ( tasks_omitted )
        // Synchronize with snapshot as the head and tail can be bumped which can falsely trigger EMPTY state
        my_arena->advertise_new_work<arena::wakeup>();
    return result;
}

References __TBB_ASSERT, __TBB_cl_evict, __TBB_control_consistency_helper, __TBB_ISOLATION_EXPR, tbb::internal::__TBB_load_relaxed(), tbb::internal::__TBB_store_relaxed(), tbb::internal::arena::advertise_new_work(), tbb::atomic_fence(), GATHER_STATISTIC, tbb::internal::arena_slot_line1::head, is_proxy(), tbb::internal::task_proxy::is_shared(), ITT_NOTIFY, lock_task_pool(), tbb::internal::scheduler_state::my_arena, tbb::internal::no_isolation, tbb::internal::task_proxy::outbox, tbb::internal::poison_pointer(), tbb::task::prefix(), tbb::internal::mail_outbox::recipient_is_idle(), tbb::internal::arena_slot_line2::tail, tbb::internal::task_proxy::task_and_tag, unlock_task_pool(), and tbb::internal::arena::wakeup.

Here is the call graph for this function:

unlock_task_pool()

void tbb::internal::generic_scheduler::unlock_task_pool ( arena_slot *  victim_arena_slot, task **  victim_task_pool  ) const

inline

Unlocks victim's task pool.

Restores victim_arena_slot->task_pool munged by lock_task_pool.

Definition at line 547 of file scheduler.cpp.

                                                                                {
    __TBB_ASSERT( victim_arena_slot, "empty victim arena slot pointer" );
    __TBB_ASSERT( victim_arena_slot->task_pool == LockedTaskPool, "victim arena slot is not locked" );
    ITT_NOTIFY(sync_releasing, victim_arena_slot);
    __TBB_store_with_release( victim_arena_slot->task_pool, victim_task_pool );
}

References __TBB_ASSERT, tbb::internal::__TBB_store_with_release(), ITT_NOTIFY, LockedTaskPool, sync_releasing, and tbb::internal::arena_slot_line1::task_pool.

Referenced by steal_task().

Here is the call graph for this function:

Here is the caller graph for this function:

wait_until_empty()

void tbb::internal::generic_scheduler::wait_until_empty

(

)

Definition at line 671 of file arena.cpp.

                                         {
    my_dummy_task->prefix().ref_count++; // prevents exit from local_wait_for_all when local work is done enforcing the stealing
    while( my_arena->my_pool_state != arena::SNAPSHOT_EMPTY )
        local_wait_for_all(*my_dummy_task, NULL);
    my_dummy_task->prefix().ref_count--;
}

References local_wait_for_all(), tbb::internal::scheduler_state::my_arena, my_dummy_task, tbb::internal::arena_base::my_pool_state, tbb::task::prefix(), and tbb::internal::arena::SNAPSHOT_EMPTY.

Here is the call graph for this function:

worker_outermost_level()

bool tbb::internal::generic_scheduler::worker_outermost_level ( ) const

inline

True if the scheduler is on the outermost dispatch level in a worker thread.

Definition at line 554 of file scheduler.h.

                                                             {
    return is_worker() && outermost_level();
}

References is_worker(), and outermost_level().

Here is the call graph for this function:

Friends And Related Function Documentation

custom_scheduler

template<typename SchedulerTraits >

friend class custom_scheduler

friend

Definition at line 289 of file scheduler.h.

Member Data Documentation

min_task_pool_size

const size_t tbb::internal::generic_scheduler::min_task_pool_size = 64

static

Initial size of the task deque sufficient to serve without reallocation 4 nested parallel_for calls with iteration space of 65535 grains each.

Definition at line 269 of file scheduler.h.

Referenced by local_spawn(), and prepare_task_pool().

my_auto_initialized

bool tbb::internal::generic_scheduler::my_auto_initialized

True if *this was created by automatic TBB initialization.

Definition at line 171 of file scheduler.h.

my_dummy_task

task* tbb::internal::generic_scheduler::my_dummy_task

Fake root task created by slave threads.

The task is used as the "parent" argument to method wait_for_all.

Definition at line 160 of file scheduler.h.

Referenced by attach_arena(), cleanup_master(), free_scheduler(), generic_scheduler(), tbb::internal::nested_arena_context::mimic_outermost_level(), wait_until_empty(), and tbb::internal::nested_arena_context::~nested_arena_context().

my_free_list

task* tbb::internal::generic_scheduler::my_free_list

Free list of small tasks that can be reused.

Definition at line 152 of file scheduler.h.

Referenced by allocate_task(), free_scheduler(), and free_task().

my_market

market* tbb::internal::generic_scheduler::my_market

The market I am in.

Definition at line 146 of file scheduler.h.

Referenced by attach_arena(), cleanup_master(), free_scheduler(), and init_stack_info().

my_random

FastRandom tbb::internal::generic_scheduler::my_random

Random number generator used for picking a random victim from which to steal.

Definition at line 149 of file scheduler.h.

Referenced by tbb::internal::custom_scheduler< SchedulerTraits >::tally_completion_of_predecessor().

my_ref_count

long tbb::internal::generic_scheduler::my_ref_count

Reference count for scheduler.

Number of task_scheduler_init objects that point to this scheduler

Definition at line 164 of file scheduler.h.

my_return_list

task* tbb::internal::generic_scheduler::my_return_list

List of small tasks that have been returned to this scheduler by other schedulers.

Definition at line 362 of file scheduler.h.

Referenced by allocate_task(), free_scheduler(), and generic_scheduler().

my_small_task_count

__TBB_atomic intptr_t tbb::internal::generic_scheduler::my_small_task_count

Number of small tasks that have been allocated by this scheduler.

Definition at line 358 of file scheduler.h.

Referenced by allocate_task(), and free_scheduler().

my_stealing_threshold

uintptr_t tbb::internal::generic_scheduler::my_stealing_threshold

Position in the call stack specifying its maximal filling when stealing is still allowed.

Definition at line 129 of file scheduler.h.

Referenced by can_steal(), and init_stack_info().

null_arena_index

const size_t tbb::internal::generic_scheduler::null_arena_index = ~size_t(0)

static

Definition at line 135 of file scheduler.h.

quick_task_size

const size_t tbb::internal::generic_scheduler::quick_task_size = 256-task_prefix_reservation_size

static

If sizeof(task) is <=quick_task_size, it is handled on a free list instead of malloc'd.

Definition at line 122 of file scheduler.h.

Referenced by allocate_task().

---

The documentation for this class was generated from the following files:

• scheduler.h
• arena.cpp
• scheduler.cpp