witness

options WITNESS
options WITNESS_KDB
options WITNESS_SKIPSPIN

The witness module keeps track of the locks acquired and released by each thread. It also keeps track of the order in which locks are acquired with respect to each other. Each time a lock is acquired, witness uses these two lists to verify that a lock is not being acquired in the wrong order. If a lock order violation is detected, then a message is output to the kernel console detailing the locks involved and the locations in question. Witness can also be configured to drop into the kernel debugger when an order violation occurs.

The witness code also checks various other conditions such as verifying that one does not recurse on a non-recursive lock. For sleep locks, witness verifies that a new process would not be switched to when a lock is released or a lock is blocked on during an acquire while any spin locks are held. If any of these checks fail, then the kernel will panic.

The flag that controls whether or not the kernel debugger is entered when a lock order violation is detected can be set in a variety of ways. By default, the flag is off, but if the WITNESS_KDB kernel option is specified, then the flag will default to on. It can also be set from the loader(8) via the debug.witness.kdb environment variable or after the kernel has booted via the debug.witness.kdb sysctl. If the flag is set to zero, then the debugger will not be entered. If the flag is non-zero, then the debugger will be entered.

The witness code can also be configured to skip all checks on spin mutexes. By default, this flag defaults to off, but it can be turned on by specifying the WITNESS_SKIPSPIN kernel option. The flag can also be set via the loader(8) environment variable debug.witness.skipspin. If the variable is set to a non-zero value, then spin mutexes are skipped. Once the kernel has booted, the status of this flag can be examined but not set via the read-only sysctl debug.witness.skipspin.

The sysctl debug.witness.watch specifies the level of witness involvement in the system. A value of 1 specifies that witness is enabled. A value of 0 specifies that witness is disabled, but that can be enabled again. This will maintain a small amount of overhead in the system. A value of -1 specifies that witness is disabled permanently and that cannot be enabled again. The sysctl debug.witness.watch can be set via loader(8).

The witness code also provides two extra ddb(4) commands if both witness and ddb(4) are compiled into the kernel:

show locks

Outputs the list of locks held by the current thread to the kernel console along with the filename and line number at which each lock was last acquired by this thread.

show witness

Dump the current order list to the kernel console. The code first displays the lock order tree for all of the sleep locks. Then it displays the lock order tree for all of the spin locks. Finally, it displays a list of locks that have not yet been acquired.

ddb(4), loader(8), sysctl(8), mutex(9)

The witness code first appeared in BSD/OS 5.0 and was imported from there into FreeBSD 5.0.

The witness code currently does not handle recursion of shared sx(9) locks properly.