Ubuntu Manpage: crash - Analyze Linux crash dump data or a live system

NAME

       crash - Analyze Linux crash dump data or a live system

SYNOPSIS

       crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS]    (dumpfile form)
       crash [OPTION]... [NAMELIST]                         (live system form)

DESCRIPTION

Crash is a tool for interactively analyzing the state of the Linux system while it is
running, or after a kernel crash has occurred and a core dump has been created by the
netdump, diskdump, LKCD, kdump, xendump kvmdump or VMware facilities. It is loosely based
on the SVR4 UNIX crash command, but has been significantly enhanced by completely merging
it with the gdb(1) debugger. The marriage of the two effectively combines the kernel-
specific nature of the traditional UNIX crash utility with the source code level debugging
capabilities of gdb(1).

In the dumpfile form, both a NAMELIST and a MEMORY-IMAGE argument must be entered. In the
live system form, the NAMELIST argument must be entered if the kernel's vmlinux file is
not located in a known location, such as the /usr/lib/debug/lib/modules/<kernel-version>
directory.

The crash utility has also been extended to support the analysis of dumpfiles generated by
a crash of the Xen hypervisor. In that case, the NAMELIST argument must be that of the
xen-syms binary. Live system analysis is not supported for the Xen hypervisor.

The crash utility command set consists of common kernel core analysis tools such as kernel
stack back traces of all processes, source code disassembly, formatted kernel structure
and variable displays, virtual memory data, dumps of linked-lists, etc., along with
several commands that delve deeper into specific kernel subsystems. Appropriate gdb
commands may also be entered, which in turn are passed on to the gdb module for execution.
If desired, commands may be placed in either a $HOME/.crashrc file and/or in a .crashrc
file in the current directory. During initialization, the commands in $HOME/.crashrc are
executed first, followed by those in the ./.crashrc file.

The crash utility is designed to be independent of Linux version dependencies. When new
kernel source code impacts the correct functionality of crash and its command set, the
utility will be updated to recognize new kernel code changes, while maintaining backwards
compatibility with earlier releases.

OPTIONS

NAMELIST
This is a pathname to an uncompressed kernel image (a vmlinux file), or a Xen
hypervisor image (a xen-syms file) which has been compiled with the "-g" option.
If using the dumpfile form, a vmlinux file may be compressed in either gzip or
bzip2 formats.

MEMORY-IMAGE[@ADDRESS]
A kernel core dump file created by the netdump, diskdump, LKCD kdump, xendump
kvmdump or VMware facilities.

If a MEMORY-IMAGE argument is not entered, the session will be invoked on the live
system, which typically requires root privileges because of the device file used to
access system RAM. By default, /dev/crash will be used if it exists. If it does
not exist, then /dev/mem will be used; but if the kernel has been configured with
CONFIG_STRICT_DEVMEM, then /proc/kcore will be used. It is permissible to
explicitly enter /dev/crash, /dev/mem or /proc/kcore.

An @ADDRESS value must be appended to the MEMORY-IMAGE if the dumpfile is a raw RAM
dumpfile that has no header information describing the file contents. Multiple
MEMORY-IMAGE@ADDRESS ordered pairs may be entered, with each dumpfile containing a
contiguous block of RAM, where the ADDRESS value is the physical start address of
the block expressed in hexadecimal. The physical address value(s) will be used to
create a temporary ELF header in /var/tmp, which will only exist during the crash
session. If a raw RAM dumpile represents a live memory source, such as that
specified by the QEMU mem-path argument of a memory-backend-file object, then
"live:" must be prepended to the MEMORY-IMAGE name.

As VMware facility, the crash utility is able to process VMware VM memory dump
generated by VM suspend or guest core dump. In that case, .vmss or .guest file
should be used as a MEMORY-IMAGE and .vmem file must be located in the same folder.

mapfile
If the NAMELIST file is not the same kernel that is running (live system form), or
the kernel that was running when the system crashed (dumpfile form), then the
System.map file of the original kernel should be entered on the command line.

-h [option]
--help [option]
Without an option argument, display a crash usage help message. If the option
argument is a crash command name, the help page for that command is displayed. If
it is the string "input", a page describing the various crash command line input
options is displayed. If it is the string "output", a page describing command line
output options is displayed. If it is the string "all", then all of the possible
help messages are displayed. After the help message is displayed, crash exits.

-s Silently proceed directly to the "crash>" prompt without displaying any version,
GPL, or crash initialization data during startup, and by default, runtime command
output is not passed to any scrolling command.

-i file
Execute the command(s) contained in file prior to displaying the "crash>" prompt
for interactive user input.

-d num Set the internal debug level. The higher the number, the more debugging data will
be printed when crash initializes and runs.

-S Use /boot/System.map as the mapfile.

-e vi | emacs
Set the readline(3) command line editing mode to "vi" or "emacs". The default
editing mode is "vi".

-f Force the usage of a compressed vmlinux file if its original name does not start
with "vmlinux".

-k Indicate that the NAMELIST file is an LKCD "Kerntypes" debuginfo file.

-g [namelist]
Determine if a vmlinux or xen-syms namelist file contains debugging data.

-t Display the system-crash timestamp and exit.

-L Attempt to lock all of its virtual address space into memory by calling
mlockall(MCL_CURRENT|MCL_FUTURE) during initialization. If the system call fails,
an error message will be displayed, but the session continues.

-c tty-device
Open the tty-device as the console used for debug messages.

-p page-size
If a processor's page size cannot be determined by the dumpfile, and the processor
default cannot be used, use page-size.

-o filename
Only used with the MEMORY-IMAGE@ADDRESS format for raw RAM dumpfiles, specifies a
filename of a new ELF vmcore that will be created and used as the dumpfile. It
will be saved to allow future use as a standalone vmcore, replacing the original
raw RAM dumpfile.

-m option=value
--machdep option=value
Pass an option and value pair to machine-dependent code. These architecture-
specific option/pairs should only be required in very rare circumstances:

X86_64:
phys_base=<physical-address>
irq_eframe_link=<value>
irq_stack_gap=<value>
max_physmem_bits=<value>
kernel_image_size=<value>
vm=orig (pre-2.6.11 virtual memory address ranges)
vm=2.6.11 (2.6.11 and later virtual memory address ranges)
vm=xen (Xen kernel virtual memory address ranges)
vm=xen-rhel4 (RHEL4 Xen kernel virtual address ranges)
vm=5level (5-level page tables)
page_offset=<PAGE_OFFSET-value>
PPC64:
vm=orig
vm=2.6.14 (4-level page tables)
IA64:
phys_start=<physical-address>
init_stack_size=<size>
vm=4l (4-level page tables)
ARM:
phys_base=<physical-address>
ARM64:
phys_offset=<physical-address>
kimage_voffset=<kimage_voffset-value>
max_physmem_bits=<value>
vabits_actual=<value>
X86:
page_offset=<CONFIG_PAGE_OFFSET-value>

-x Automatically load extension modules from a particular directory. If a directory
is specified in the CRASH_EXTENSIONS shell environment variable, then that
directory will be used. Otherwise /usr/lib64/crash/extensions (64-bit
architectures) or /usr/lib/crash/extensions (32-bit architectures) will be used; if
they do not exist, then the ./extensions directory will be used.

--active
Track only the active task on each cpu.

--buildinfo
Display the crash binary's build date, the user ID of the builder, the hostname of
the machine where the build was done, the target architecture, the version number,
and the compiler version.

--memory_module modname
Use the modname as an alternative kernel module to the crash.ko module that creates
the /dev/crash device.

--memory_device device
Use device as an alternative device to the /dev/crash, /dev/mem or /proc/kcore
devices.

--log dumpfile
Dump the contents of the kernel log buffer. A kernel namelist argument is not
necessary, but the dumpfile must contain the VMCOREINFO data taken from the
original /proc/vmcore ELF header.

--no_kallsyms
Do not use kallsyms-generated symbol information contained within kernel module
object files.

--no_modules
Do not access or display any kernel module related information.

--no_ikconf
Do not attempt to read configuration data that was built into kernels configured
with CONFIG_IKCONFIG.

--no_data_debug
Do not verify the validity of all structure member offsets and structure sizes that
it uses.

--no_kmem_cache
Do not initialize the kernel's slab cache infrastructure, and commands that use
kmem_cache-related data will not work.

--no_elf_notes
Do not use the registers from the ELF NT_PRSTATUS notes saved in a compressed kdump
header for backtraces.

--kmem_cache_delay
Delay the initialization of the kernel's slab cache infrastructure until it is
required by a run-time command.

--readnow
Pass this flag to the embedded gdb module, which will override its two-stage
strategy that it uses for reading symbol tables from the NAMELIST.

--smp Specify that the system being analyzed is an SMP kernel.

-v
--version
Display the version of the crash utility, the version of the embedded gdb module,
GPL information, and copyright notices.

--cpus number
Specify the number of cpus in the SMP system being analyzed.

--osrelease dumpfile
Display the OSRELEASE vmcoreinfo string from a kdump dumpfile header.

--hyper
Force the session to be that of a Xen hypervisor.

--p2m_mfn pfn
When a Xen Hypervisor or its dom0 kernel crashes, the dumpfile is typically
analyzed with either the Xen hypervisor or the dom0 kernel. It is also possible to
analyze any of the guest domU kernels if the pfn_to_mfn_list_list pfn value of the
guest kernel is passed on the command line along with its NAMELIST and the
dumpfile.

--xen_phys_start physical-address
Supply the base physical address of the Xen hypervisor's text and static data for
older xendump dumpfiles that did not pass that information in the dumpfile header.

--zero_excluded
If the makedumpfile(8) facility has filtered a compressed kdump dumpfile to exclude
various types of non-essential pages, or has marked a compressed or ELF kdump
dumpfile as incomplete due to an ENOSPC or other error during its creation, any
attempt to read missing pages will fail. With this flag, reads from any of those
pages will return zero-filled memory.

--no_panic
Do not attempt to find the task that was running when the kernel crashed. Set the
initial context to that of the "swapper" task on cpu 0.

--more Use /bin/more as the command output scroller, overriding the default of
/usr/bin/less and any settings in either ./.crashrc or $HOME/.crashrc.

--less Use /usr/bin/less as the command output scroller, overriding any settings in either
./.crashrc or $HOME/.crashrc.

--hex Set the default command output radix to 16, overriding the default radix of 10, and
any radix settings in either ./.crashrc or $HOME/.crashrc.

--dec Set the default command output radix to 10, overriding any radix settings in either
./.crashrc or $HOME/.crashrc. This is the default radix setting.

--CRASHPAGER
Use the output paging command defined in the CRASHPAGER shell environment variable,
overriding any settings in either ./.crashrc or $HOME/.crashrc.

--no_scroll
Do not pass run-time command output to any scrolling command.

--no_strip
Do not strip cloned kernel text symbol names.

--no_crashrc
Do not execute the commands in either $HOME/.crashrc or ./.crashrc.

--mod directory
When loading the debuginfo data of kernel modules with the mod -S command, search
for their object files in directory instead of in the standard location.

--src directory
Search for the kernel source code in directory instead of in the standard location
that is compiled into the debuginfo data.

--kaslr offset|auto
If an x86_64 kernel was configured with CONFIG_RANDOMIZE_BASE, the offset value is
equal to the difference between the symbol values compiled into the vmlinux file
and their relocated KASLR values. If set to auto, the KASLR offset value will be
automatically calculated.

--reloc size
When analyzing live x86 kernels that were configured with a CONFIG_PHYSICAL_START
value that is larger than its CONFIG_PHYSICAL_ALIGN value, then it will be
necessary to enter a relocation size equal to the difference between the two
values.

--hash count
Set the number of internal hash queue heads used for list gathering and
verification. The default count is 32768.

--minimal
Bring up a session that is restricted to the log, dis, rd, sym, eval, set and exit
commands. This option may provide a way to extract some minimal/quick information
from a corrupted or truncated dumpfile, or in situations where one of the several
kernel subsystem initialization routines would abort the crash session.

--kvmhost [32|64]
When examining an x86 KVM guest dumpfile, this option specifies that the KVM host
that created the dumpfile was an x86 (32-bit) or an x86_64 (64-bit) machine,
overriding the automatically determined value.

--kvmio <size>
override the automatically-calculated KVM guest I/O hole size.

--offline [show|hide]
Show or hide command output that is related to offline cpus. The default setting
is show.

COMMANDS

Each crash command generally falls into one of the following categories:

Symbolic display
Displays of kernel text/data, which take full advantage of the power of gdb to
format and display data structures symbolically.

System state
The majority of crash commands consist of a set of "kernel-aware" commands, which
delve into various kernel subsystems on a system-wide or per-task basis.

Utility functions
A set of useful helper commands serving various purposes, some simple, others quite
powerful.

Session control
Commands that control the crash session itself.

The following alphabetical list consists of a very simple overview of each crash command.
However, since individual commands often have several options resulting in significantly
different output, it is suggested that the full description of each command be viewed by
executing crash -h <command>, or during a crash session by simply entering help command.

* "pointer to" is shorthand for either the struct or union commands. It displays the
contents of a kernel structure or union.

alias creates a single-word alias for a command.

ascii displays an ascii chart or translates a numeric value into its ascii components.

bpf provides information on currently-loaded eBPF programs and maps.

bt displays a task's kernel-stack backtrace. If it is given the -a option, it
displays the stack traces of the active tasks on all CPUs. It is often used with
the foreach command to display the backtraces of all tasks with one command.

btop translates a byte value (physical offset) to its page number.

dev displays data concerning the character and block device assignments, I/O port
usage, I/O memory usage, and PCI device data.

dis disassembles memory, either entire kernel functions, from a location for a
specified number of instructions, or from the start of a function up to a specified
memory location.

eval evaluates an expression or numeric type and displays the result in hexadecimal,
decimal, octal and binary.

exit causes crash to exit.

extend dynamically loads or unloads crash shared object extension modules.

files displays information about open files in a context.

foreach
repeats a specified command for the specified (or all) tasks in the system.

fuser displays the tasks using the specified file or socket.

gdb passes its argument to the embedded gdb module. It is useful for executing gdb
commands that have the same name as crash commands.

help alone displays the command menu; if followed by a command name, a full description
of a command, its options, and examples are displayed. Its output is far more
complete and useful than this man page.

ipcs displays data about the System V IPC facilities.

irq displays data concerning interrupt request numbers and bottom-half interrupt
handling.

kmem displays information about the use of kernel memory.

list displays the contents of a linked list.

log displays the kernel log_buf contents in chronological order.

mach displays data specific to the machine type.

mod displays information about the currently installed kernel modules, or adds or
deletes symbolic or debugging information about specified kernel modules.

mount displays information about the currently-mounted filesystems.

net display various network related data.

p passes its arguments to the gdb "print" command for evaluation and display.

ps displays process status for specified, or all, processes in the system.

pte translates the hexadecimal contents of a PTE into its physical page address and
page bit settings.

ptob translates a page frame number to its byte value.

ptov translates a hexadecimal physical address into a kernel virtual address.

q is an alias for the "exit" command.

rd displays the contents of memory, with the output formatted in several different
manners.

repeat repeats a command indefinitely, optionally delaying a given number of seconds
between each command execution.

runq displays the tasks on the run queue.

sbitmapq
dumps the contents of the sbitmap_queue structure and the used bits in the bitmap.
Also, it shows the dump of a structure array associated with the sbitmap_queue.

search searches a range of user or kernel memory space for given value.

set either sets a new context, or gets the current context for display.

sig displays signal-handling data of one or more tasks.

struct displays either a structure definition or the contents of a kernel structure at a
specified address.

swap displays information about each configured swap device.

sym translates a symbol to its virtual address, or a static kernel virtual address to
its symbol -- or to a symbol-plus-offset value, if appropriate.

sys displays system-specific data.

task displays the contents of a task_struct.

tree displays the contents of a red-black tree or a radix tree.

timer displays the timer queue entries, both old- and new-style, in chronological order.

union is similar to the struct command, except that it works on kernel unions.

vm displays basic virtual memory information of a context.

vtop translates a user or kernel virtual address to its physical address.

waitq walks the wait queue list displaying the tasks which are blocked on the specified
wait queue.

whatis displays the definition of structures, unions, typedefs or text/data symbols.

wr modifies the contents of memory on a live system. It can only be used if /dev/mem
is the device file being used to access system RAM, and should obviously be used
with great care.

When crash is invoked with a Xen hypervisor binary as the NAMELIST, the command set is
slightly modified. The *, alias, ascii, bt, dis, eval, exit, extend, gdb, help, list,
log, p, pte, rd, repeat, search, set, struct, sym, sys, union, whatis, wr and q commands
are the same as above. The following commands are specific to the Xen hypervisor:

domain displays the contents of the domain structure for selected, or all, domains.

doms displays domain status for selected, or all, domains.

dumpinfo
displays Xen dump information for selected, or all, cpus.

pcpus displays physical cpu information for selected, or all, cpus.

vcpus displays vcpu status for selected, or all, vcpus.

FILES

       .crashrc
              Initialization  commands.   The  file  can  be located in the user's HOME directory
              and/or the current directory.  Commands found in the  .crashrc  file  in  the  HOME
              directory are executed before those in the current directory's .crashrc file.

ENVIRONMENT

       EDITOR Command  input  is  read  using  readline(3).  If EDITOR is set to emacs or vi then
              suitable keybindings are used.  If EDITOR is not set, then vi is used.  This can be
              overridden  by  set  vi  or  set  emacs  commands located in a .crashrc file, or by
              entering -e emacs on the crash command line.

       CRASHPAGER
              If CRASHPAGER is set, its value is used as the name of the program to which command
              output will be sent.  If not, then command output is sent to /usr/bin/less -E -X by
              default.

       CRASH_MODULE_PATH
              Specifies an alternative directory tree to search for kernel module object files.

       CRASH_EXTENSIONS
              Specifies  a  directory  containing  extension  modules   that   will   be   loaded
              automatically if the -x command line option is used.

NOTES

       If  crash does not work, look for a newer version: kernel evolution frequently makes crash
       updates necessary.

       The command set scroll off will cause output to be sent directly to  the  terminal  rather
       than through a paging program.  This is useful, for example, if you are running crash in a
       window of emacs.

AUTHOR

       Dave Anderson <anderson@redhat.com> wrote crash.

       Jay Fenlason <fenlason@redhat.com> and Dave Anderson <anderson@redhat.com> wrote this  man
       page.