NAME

       libhugetlbfs  -  preload  library  to  back  text,  data,  malloc()  or shared memory with
       hugepages

SYNOPSIS

       export [environment options]
       [LD_PRELOAD=libhugetlbfs.so] target_application

DESCRIPTION

       libhugetlbfs is a library that can back application text, data, malloc() and shared memory
       with hugepages. This is of benefit to applications that use large amounts of address space
       and suffer a performance hit due to TLB misses. Wall-clock time or oprofile can be used to
       determine  if there is a performance benefit from using libhugetlbfs or not.  In all cases
       but shared memory, a hugetlbfs mount must exist and a hugepage pool defined for  hugepages
       to be used.

       Some limited functionality is available for unmodified dynamically linked applications. By
       preloading the library, the library can back malloc() and shared memory, and text and data
       segments can be partially backed if they are large enough.

       For the effective backing of text and data with huge pages, the application must be linked
       to the library and the ELF segments correctly aligned using the ld helpers.  Once  linked,
       malloc  or  shared  memory  can  still  be  backed  but  no  pre-loading  is required. See
       /usr/share/doc/libhugetlbfs/HOWTO  and  ld.hugetlbfs(1)  for  detailed   instructions   on
       relinking applications.

       For applications that are hugepage-aware and linked to the library get_huge_pages() can be
       used for the direct allocation of hugepage-backed regions.

       Unless otherwise specified, libhugetlbfs will use the default hugepage size to back memory
       regions.  The  default  size  is the value of Hugepagesize displayed in /proc/meminfo. The
       size can be specified in bytes or in kilobytes, megabytes, or gigabytes by appending K, M,
       or  G  respectively.  It  is  an  error  to  specify  a invalid, unsupported, or otherwise
       unconfigured huge page size. Kernel 2.6.27 or later is required to  specify  any  pagesize
       other than the default.

       See /usr/share/docs/libhugetlbfs/HOWTO for detailed instructions on how the library should
       be used, particularly when relinking the application.  This manual page provides  a  brief
       synopsis of the environment variables as a quick reference.

       The  following variables affect what memory regions are backed by hugepages. In all cases,
       the environment being unset implies the feature should remain disabled.

       HUGETLB_DEFAULT_PAGE_SIZE=<pagesize>
              This sets the default hugepage size to  be  used  by  libhugetlbfs.   If  not  set,
              libhugetlbfs will use the kernel's default hugepage size.

       HUGETLB_MORECORE=[yes|<pagesize>]
              This  enables  the  hugepage malloc() feature, instructing libhugetlbfs to override
              glibc's normal morecore() function with a hugepage version and use it for malloc().
              All application malloc() memory should come from hugepage memory until it runs out,
              it will then fallback to base  pages.   Note  that  applications  that  use  custom
              allocators may not be able to back their heaps using hugepages and this environment
              variable.  It  may  be  necessary  to  modify   the   custom   allocator   to   use
              get_huge_pages().

       HUGETLB_SHM=yes
              When  this  environment  variable  is  set,  the  SHM_HUGETLB  flag is added to the
              shmget() call and the size parameter is aligned to back the shared  memory  segment
              with  hugepages.  In  the  event  hugepages cannot be used, base pages will be used
              instead and a warning will be printed to explain the failure. The  pagesize  cannot
              be  specified with this parameter. To change the kernels default hugepage size, use
              the pagesize= kernel boot parameter (2.6.26 or later required).

       HUGETLB_ELFMAP=[no|[R[<=pagesize>]:[W[<=pagesize>]]]
              If the application has  been  relinked  (see  the  HOWTO  for  instructions),  this
              environment  variable determines whether read-only, read-write, both or no segments
              are backed by hugepages and what  pagesize  should  be  used.  If  the  recommended
              relinking method has been used, then hugeedit can be used to automatically back the
              text or data by default.

       HUGETLB_FORCE_ELFMAP=yes
              Force the use of hugepages for text and data segments even if the  application  has
              not  been  relinked  to  align  the  ELF  segments on a hugepage boundary.  Partial
              segment remapping is not guaranteed to work and the segments must be  large  enough
              to contain at least one hugepage for the remapping to occur.

       The following options affect how libhugetlbfs behaves.

       HUGETLB_RESTRICT_EXE=e1:e2:...:eN
              By default, libhugetlbfs will act on any program that it is loaded with, either via
              LD_PRELOAD or by explicitly linking with -lhugetlbfs.

              There are situations in which it is desirable to restrict libhugetlbfs' actions  to
              specific  programs.   For example, some ISV applications are wrapped in a series of
              scripts that invoke bash, python, and/or perl.  It is more convenient  to  set  the
              environment  variables related to libhugetlbfs before invoking the wrapper scripts,
              yet this has the unintended and  undesirable  consequence  of  causing  the  script
              interpreters  to use and consume hugepages.  There is no obvious benefit to causing
              the script interpreters to use hugepages, and there is a clear disadvantage:  fewer
              hugepages are available to the actual application.

              To  address  this  scenario,  set HUGETLB_RESTRICT_EXE to a colon-separated list of
              programs to which the other libhugetlbfs environment variables should  apply.   (If
              not set, libhugetlbfs will attempt to apply the requested actions to all programs.)
              For example,

                  HUGETLB_RESTRICT_EXE=hpcc:long_hpcc

              will  restrict  libhugetlbfs'  actions  to  programs  named   /home/fred/hpcc   and
              /bench/long_hpcc but not /bin/hpcc_no.

       HUGETLB_MORECORE_SHRINK=yes
              By default, the hugepage heap does not shrink. Shrinking is enabled by setting this
              environment variable. It is disabled by  default  as  glibc  occasionally  exhibits
              strange  behaviour  if  it  mistakes the heap returned by libhugetlbfs as a foreign
              brk().

       HUGETLB_NO_PREFAULT
              By default libhugetlbfs will prefault regions it creates  to  ensure  they  can  be
              referenced  without  receiving  a  SIGKILL.  On kernels older than 2.6.27, this was
              necessary as the system did not guarantee  that  future  faults  would  succeed  on
              regions  mapped  MAP_PRIVATE.   Prefaulting impacts the performance of malloc() and
              can result in poor placement on NUMA systems. If it is known the hugepage  pool  is
              large  enough  to  run  the  application  or  the  kernel  is 2.6.27 or later, this
              environment variable should be set.

       HUGETLB_NO_RESERVE=yes

              By default, the kernel will reserve huge pages at mmap() time to ensure that future
              faults  will  succeed. This avoids unexpected application failure at fault time but
              some applications depend on memory overcommit to create large sparse mappings.  For
              this  type  of application, setting this environment variable will create huge page
              backed mappings without a reservation. Use this option with extreme care as in  the
              event  huge  pages are not available when the mapping is used, the application will
              be killed. On older kernels, the use of this feature can trigger  the  OOM  killer.
              Hence, even with this variable set, reservations may still be used for safety.

       HUGETLB_MORECORE_HEAPBASE=address
              libhugetlbfs  normally picks an address to use as the base of the heap for malloc()
              automatically. This environment variable fixes which address is used.

       HUGETLB_PATH=<path>
              The path to the hugetlbfs mount is automatically determined  at  run-time.  In  the
              event  there  are  multiple  mounts  and  the wrong one is being selected, use this
              option to select the correct one. This may be the case if  an  application-specific
              mount with a fixed quota has been created for example.

       HUGETLB_SHARE=1
              By  default,  libhugetlbfs  uses unlinked hugetlbfs files to store remapped program
              segment data. If  the  same  program  is  started  multiple  times  using  hugepage
              segments,  multiple  hugepages  will  be  used  to store the same program data. The
              reduce this  wastage,  setting  this  environment  variable  will  share  read-only
              segments  between multiple invocations of a program at the cost of the memory being
              used whether the applications are running or  not.  It  is  also  possible  that  a
              malicious  application  inferfere  with other applications executable code. See the
              HOWTO for more detailed information on this topic.

       The following options control the verbosity of libhugetlbfs.

       HUGETLB_VERBOSE=<level>
              The default value for this is 1 and the range of the value is from  0  to  99.  The
              higher the value, the more verbose the output is. 0 is quiet and 3 will output much
              debugging information.

       HUGETLB_DEBUG
              Once set, this will give very detailed output on what is happening in  the  library
              and run extra diagnostics.

FILES

       [DESTDIR|/usr/share]/doc/libhugetlbfs/HOWTO

SEE ALSO

       oprofile(1),     ld.hugetlbfs(1),     hugectl(8),     hugeedit(8),     gethugepagesize(3),
       gethugepagesizes(3),  getpagesizes(3),   hugetlbfs_test_path(3),   hugetlbfs_find_path(3),
       hugetlbfs_find_path_for_size(3),  hugetlbfs_test_path(3), hugetlbfs_test_path_for_size(3),
       hugetlbfs_unlinked_fd(3),      hugetlbfs_unlinked_fd_for_size(3),       get_huge_pages(3),
       free_huge_pages(3)

AUTHORS

       libhugetlbfs was written by various people on the libhugetlbfs-devel mailing list.

                                        September 27, 2008                        LIBHUGETLBFS(7)