Ubuntu Manpage: libhugetlbfs - preload library to back text, data, malloc() or shared memory with

NAME

       libhugetlbfs  -  preload  library  to  back  text,  data,  malloc()  or shared memory with
       hugepages

SYNOPSIS

       export [environment options]
       [LD_PRELOAD=libhugetlbfs.so] target_application

DESCRIPTION

libhugetlbfs is a library that can back application text, data, malloc() and shared memory
with hugepages. This is of benefit to applications that use large amounts of address space
and suffer a performance hit due to TLB misses. Wall-clock time or oprofile can be used to
determine if there is a performance benefit from using libhugetlbfs or not. In all cases
but shared memory, a hugetlbfs mount must exist and a hugepage pool defined for hugepages
to be used.

Some limited functionality is available for unmodified dynamically linked applications. By
preloading the library, the library can back malloc() and shared memory, and text and data
segments can be partially backed if they are large enough.

For the effective backing of text and data with huge pages, the application must be linked
to the library and the ELF segments correctly aligned using the ld helpers. Once linked,
malloc or shared memory can still be backed but no pre-loading is required. See
/usr/share/doc/libhugetlbfs/HOWTO and ld.hugetlbfs(1) for detailed instructions on
relinking applications.

For applications that are hugepage-aware and linked to the library get_huge_pages() can be
used for the direct allocation of hugepage-backed regions.

Unless otherwise specified, libhugetlbfs will use the default hugepage size to back memory
regions. The default size is the value of Hugepagesize displayed in /proc/meminfo. The
size can be specified in bytes or in kilobytes, megabytes, or gigabytes by appending K, M,
or G respectively. It is an error to specify a invalid, unsupported, or otherwise
unconfigured huge page size. Kernel 2.6.27 or later is required to specify any pagesize
other than the default.

See /usr/share/docs/libhugetlbfs/HOWTO for detailed instructions on how the library should
be used, particularly when relinking the application. This manual page provides a brief
synopsis of the environment variables as a quick reference.

The following variables affect what memory regions are backed by hugepages. In all cases,
the environment being unset implies the feature should remain disabled.

HUGETLB_DEFAULT_PAGE_SIZE=<pagesize>
This sets the default hugepage size to be used by libhugetlbfs. If not set,
libhugetlbfs will use the kernel's default hugepage size.

HUGETLB_MORECORE=[yes|<pagesize>]
This enables the hugepage malloc() feature, instructing libhugetlbfs to override
glibc's normal morecore() function with a hugepage version and use it for malloc().
All application malloc() memory should come from hugepage memory until it runs out,
it will then fallback to base pages. Note that applications that use custom
allocators may not be able to back their heaps using hugepages and this environment
variable. It may be necessary to modify the custom allocator to use
get_huge_pages().

HUGETLB_SHM=yes
When this environment variable is set, the SHM_HUGETLB flag is added to the
shmget() call and the size parameter is aligned to back the shared memory segment
with hugepages. In the event hugepages cannot be used, base pages will be used
instead and a warning will be printed to explain the failure. The pagesize cannot
be specified with this parameter. To change the kernels default hugepage size, use
the pagesize= kernel boot parameter (2.6.26 or later required).

HUGETLB_ELFMAP=[no|[R[<=pagesize>]:[W[<=pagesize>]]]
If the application has been relinked (see the HOWTO for instructions), this
environment variable determines whether read-only, read-write, both or no segments
are backed by hugepages and what pagesize should be used. If the recommended
relinking method has been used, then hugeedit can be used to automatically back the
text or data by default.

HUGETLB_FORCE_ELFMAP=yes
Force the use of hugepages for text and data segments even if the application has
not been relinked to align the ELF segments on a hugepage boundary. Partial
segment remapping is not guaranteed to work and the segments must be large enough
to contain at least one hugepage for the remapping to occur.

The following options affect how libhugetlbfs behaves.

HUGETLB_RESTRICT_EXE=e1:e2:...:eN
By default, libhugetlbfs will act on any program that it is loaded with, either via
LD_PRELOAD or by explicitly linking with -lhugetlbfs.

There are situations in which it is desirable to restrict libhugetlbfs' actions to
specific programs. For example, some ISV applications are wrapped in a series of
scripts that invoke bash, python, and/or perl. It is more convenient to set the
environment variables related to libhugetlbfs before invoking the wrapper scripts,
yet this has the unintended and undesirable consequence of causing the script
interpreters to use and consume hugepages. There is no obvious benefit to causing
the script interpreters to use hugepages, and there is a clear disadvantage: fewer
hugepages are available to the actual application.

To address this scenario, set HUGETLB_RESTRICT_EXE to a colon-separated list of
programs to which the other libhugetlbfs environment variables should apply. (If
not set, libhugetlbfs will attempt to apply the requested actions to all programs.)
For example,

HUGETLB_RESTRICT_EXE=hpcc:long_hpcc

will restrict libhugetlbfs' actions to programs named /home/fred/hpcc and
/bench/long_hpcc but not /bin/hpcc_no.

HUGETLB_MORECORE_SHRINK=yes
By default, the hugepage heap does not shrink. Shrinking is enabled by setting this
environment variable. It is disabled by default as glibc occasionally exhibits
strange behaviour if it mistakes the heap returned by libhugetlbfs as a foreign
brk().

HUGETLB_NO_PREFAULT
By default libhugetlbfs will prefault regions it creates to ensure they can be
referenced without receiving a SIGKILL. On kernels older than 2.6.27, this was
necessary as the system did not guarantee that future faults would succeed on
regions mapped MAP_PRIVATE. Prefaulting impacts the performance of malloc() and
can result in poor placement on NUMA systems. If it is known the hugepage pool is
large enough to run the application or the kernel is 2.6.27 or later, this
environment variable should be set.

HUGETLB_NO_RESERVE=yes

By default, the kernel will reserve huge pages at mmap() time to ensure that future
faults will succeed. This avoids unexpected application failure at fault time but
some applications depend on memory overcommit to create large sparse mappings. For
this type of application, setting this environment variable will create huge page
backed mappings without a reservation. Use this option with extreme care as in the
event huge pages are not available when the mapping is used, the application will
be killed. On older kernels, the use of this feature can trigger the OOM killer.
Hence, even with this variable set, reservations may still be used for safety.

HUGETLB_MORECORE_HEAPBASE=address
libhugetlbfs normally picks an address to use as the base of the heap for malloc()
automatically. This environment variable fixes which address is used.

HUGETLB_PATH=<path>
The path to the hugetlbfs mount is automatically determined at run-time. In the
event there are multiple mounts and the wrong one is being selected, use this
option to select the correct one. This may be the case if an application-specific
mount with a fixed quota has been created for example.

HUGETLB_SHARE=1
By default, libhugetlbfs uses unlinked hugetlbfs files to store remapped program
segment data. If the same program is started multiple times using hugepage
segments, multiple hugepages will be used to store the same program data. The
reduce this wastage, setting this environment variable will share read-only
segments between multiple invocations of a program at the cost of the memory being
used whether the applications are running or not. It is also possible that a
malicious application inferfere with other applications executable code. See the
HOWTO for more detailed information on this topic.

The following options control the verbosity of libhugetlbfs.

HUGETLB_VERBOSE=<level>
The default value for this is 1 and the range of the value is from 0 to 99. The
higher the value, the more verbose the output is. 0 is quiet and 3 will output much
debugging information.

HUGETLB_DEBUG
Once set, this will give very detailed output on what is happening in the library
and run extra diagnostics.

FILES

       [DESTDIR|/usr/share]/doc/libhugetlbfs/HOWTO

AUTHORS

       libhugetlbfs was written by various people on the libhugetlbfs-devel mailing list.

                                        September 27, 2008                        LIBHUGETLBFS(7)

NAME

SYNOPSIS

DESCRIPTION

FILES

SEE ALSO

AUTHORS