trusty (9) locking.9freebsd.gz
NAME
locking — kernel synchronization primitives
DESCRIPTION
The FreeBSD kernel is written to run across multiple CPUs and as such requires several different
synchronization primitives to allow the developers to safely access and manipulate the many data types
required.
Mutexes
Mutexes (also called "sleep mutexes") are the most commonly used synchronization primitive in the kernel.
Thread acquires (locks) a mutex before accessing data shared with other threads (including interrupt
threads), and releases (unlocks) it afterwards. If the mutex cannot be acquired, the thread requesting it
will sleep. Mutexes fully support priority propagation.
See mutex(9) for details.
Spin mutexes
Spin mutexes are variation of basic mutexes; the main difference between the two is that spin mutexes never
sleep - instead, they spin, waiting for the thread holding the lock, which runs on another CPU, to release
it. Differently from ordinary mutex, spin mutexes disable interrupts when acquired. Since disabling
interrupts is expensive, they are also generally slower. Spin mutexes should be used only when necessary,
e.g. to protect data shared with interrupt filter code (see bus_setup_intr(9) for details).
Pool mutexes
With most synchronization primitives, such as mutexes, programmer must provide a piece of allocated memory
to hold the primitive. For example, a mutex may be embedded inside the structure it protects. Pool mutex
is a variant of mutex without this requirement - to lock or unlock a pool mutex, one uses address of the
structure being protected with it, not the mutex itself. Pool mutexes are seldom used.
See mtx_pool(9) for details.
Reader/writer locks
Reader/writer locks allow shared access to protected data by multiple threads, or exclusive access by a
single thread. The threads with shared access are known as readers since they should only read the
protected data. A thread with exclusive access is known as a writer since it may modify protected data.
Reader/writer locks can be treated as mutexes (see above and mutex(9)) with shared/exclusive semantics.
More specifically, regular mutexes can be considered to be equivalent to a write-lock on an rw_lock. The
rw_lock locks have priority propagation like mutexes, but priority can be propagated only to an exclusive
holder. This limitation comes from the fact that shared owners are anonymous. Another important property
is that shared holders of rw_lock can recurse, but exclusive locks are not allowed to recurse. This
ability should not be used lightly and may go away.
See rwlock(9) for details.
Read-mostly locks
Mostly reader locks are similar to reader/writer locks but optimized for very infrequent write locking.
Read-mostly locks implement full priority propagation by tracking shared owners using a caller-supplied
tracker data structure.
See rmlock(9) for details.
Shared/exclusive locks
Shared/exclusive locks are similar to reader/writer locks; the main difference between them is that
shared/exclusive locks may be held during unbounded sleep (and may thus perform an unbounded sleep). They
are inherently less efficient than mutexes, reader/writer locks and read-mostly locks. They don't support
priority propagation. They should be considered to be closely related to sleep(9). In fact it could in
some cases be considered a conditional sleep.
See sx(9) for details.
Counting semaphores
Counting semaphores provide a mechanism for synchronizing access to a pool of resources. Unlike mutexes,
semaphores do not have the concept of an owner, so they can be useful in situations where one thread needs
to acquire a resource, and another thread needs to release it. They are largely deprecated.
See sema(9) for details.
Condition variables
Condition variables are used in conjunction with mutexes to wait for conditions to occur. A thread must
hold the mutex before calling the cv_wait*(), functions. When a thread waits on a condition, the mutex is
atomically released before the thread is blocked, then reacquired before the function call returns.
See condvar(9) for details.
Giant
Giant is an instance of a mutex, with some special characteristics:
1. It is recursive.
2. Drivers and filesystems can request that Giant be locked around them by not marking themselves MPSAFE.
Note that infrastructure to do this is slowly going away as non-MPSAFE drivers either became properly
locked or disappear.
3. Giant must be locked first before other locks.
4. It is OK to hold Giant while performing unbounded sleep; in such case, Giant will be dropped before
sleeping and picked up after wakeup.
5. There are places in the kernel that drop Giant and pick it back up again. Sleep locks will do this
before sleeping. Parts of the network or VM code may do this as well, depending on the setting of a
sysctl. This means that you cannot count on Giant keeping other code from running if your code
sleeps, even if you want it to.
Sleep/wakeup
The functions tsleep(), msleep(), msleep_spin(), pause(), wakeup(), and wakeup_one() handle event-based
thread blocking. If a thread must wait for an external event, it is put to sleep by tsleep(), msleep(),
msleep_spin(), or pause(). Threads may also wait using one of the locking primitive sleep routines
mtx_sleep(9), rw_sleep(9), or sx_sleep(9).
The parameter chan is an arbitrary address that uniquely identifies the event on which the thread is being
put to sleep. All threads sleeping on a single chan are woken up later by wakeup(), often called from
inside an interrupt routine, to indicate that the resource the thread was blocking on is available now.
Several of the sleep functions including msleep(), msleep_spin(), and the locking primitive sleep routines
specify an additional lock parameter. The lock will be released before sleeping and reacquired before the
sleep routine returns. If priority includes the PDROP flag, then the lock will not be reacquired before
returning. The lock is used to ensure that a condition can be checked atomically, and that the current
thread can be suspended without missing a change to the condition, or an associated wakeup. In addition,
all of the sleep routines will fully drop the Giant mutex (even if recursed) while the thread is suspended
and will reacquire the Giant mutex before the function returns.
See sleep(9) for details.
Lockmanager locks
Shared/exclusive locks, used mostly in VFS(9), in particular as a vnode(9) lock. They have features other
lock types don't have, such as sleep timeout, writer starvation avoidance, draining, and interlock mutex,
but this makes them complicated to implement; for this reason, they are deprecated.
See lock(9) for details.
INTERACTIONS
The primitives interact and have a number of rules regarding how they can and can not be combined. Many of
these rules are checked using the witness(4) code.
Bounded vs. unbounded sleep
The following primitives perform bounded sleep: mutexes, pool mutexes, reader/writer locks and read-mostly
locks.
The following primitives block (perform unbounded sleep): shared/exclusive locks, counting semaphores,
condition variables, sleep/wakeup and lockmanager locks.
It is an error to do any operation that could result in any kind of sleep while holding spin mutex.
As a general rule, it is an error to do any operation that could result in unbounded sleep while holding
any primitive from the 'bounded sleep' group. For example, it is an error to try to acquire
shared/exclusive lock while holding mutex, or to try to allocate memory with M_WAITOK while holding read-
write lock.
As a special case, it is possible to call sleep() or mtx_sleep() while holding a single mutex. It will
atomically drop that mutex and reacquire it as part of waking up. This is often a bad idea because it
generally relies on the programmer having good knowledge of all of the call graph above the place where
mtx_sleep() is being called and assumptions the calling code has made. Because the lock gets dropped
during sleep, one must re-test all the assumptions that were made before, all the way up the call graph to
the place where the lock was acquired.
It is an error to do any operation that could result in any kind of sleep when running inside an interrupt
filter.
It is an error to do any operation that could result in unbounded sleep when running inside an interrupt
thread.
Interaction table
The following table shows what you can and can not do while holding one of the synchronization primitives
discussed:
You have: You want: spin mtx mutex sx rwlock rmlock sleep
spin mtx ok-1 no no no no no-3
mutex ok ok-1 no ok ok no-3
sx ok ok ok-2 ok ok ok-4
rwlock ok ok no ok-2 ok no-3
rmlock ok ok no-5 ok ok-2 no-5
*1 Recursion is defined per lock. Lock order is important.
*2 Readers can recurse though writers can not. Lock order is important.
*3 There are calls that atomically release this primitive when going to sleep and reacquire it on wakeup
(e.g. mtx_sleep(), rw_sleep() and msleep_spin()).
*4 Though one can sleep holding an sx lock, one can also use sx_sleep() which will atomically release this
primitive when going to sleep and reacquire it on wakeup.
*5 Read-mostly locks can be initialized to support sleeping while holding a write lock. See rmlock(9) for
details.
Context mode table
The next table shows what can be used in different contexts. At this time this is a rather easy to
remember table.
Context: spin mtx mutex sx rwlock rmlock sleep
interrupt filter: ok no no no no no
interrupt thread: ok ok no ok ok no
callout: ok ok no ok no no
syscall: ok ok ok ok ok ok
SEE ALSO
witness(4), condvar(9), lock(9), mtx_pool(9), mutex(9), rmlock(9), rwlock(9), sema(9), sleep(9), sx(9), BUS_SETUP_INTR(9), LOCK_PROFILING(9)
HISTORY
These functions appeared in BSD/OS 4.1 through FreeBSD 7.0.
BUGS
There are too many locking primitives to choose from.