NAME

       libhugetlbfs - preload library to back text, data, malloc() or shared memory with hugepages

SYNOPSIS

       export [environment options]
       [LD_PRELOAD=libhugetlbfs.so] target_application

DESCRIPTION

       libhugetlbfs  is  a  library  that  can  back  application  text,  data,  malloc() and shared memory with
       hugepages. This is of benefit to applications that use large  amounts  of  address  space  and  suffer  a
       performance  hit  due  to  TLB misses. Wall-clock time or oprofile can be used to determine if there is a
       performance benefit from using libhugetlbfs or not.  In all cases but shared memory,  a  hugetlbfs  mount
       must exist and a hugepage pool defined for hugepages to be used.

       Some limited functionality is available for unmodified dynamically linked applications. By preloading the
       library,  the  library  can  back malloc() and shared memory, and text and data segments can be partially
       backed if they are large enough.

       For the effective backing of text and data with huge pages, the application must be linked to the library
       and the ELF segments correctly aligned using the ld helpers. Once linked, malloc  or  shared  memory  can
       still be backed but no pre-loading is required. See /usr/share/doc/libhugetlbfs/HOWTO and ld.hugetlbfs(1)
       for detailed instructions on relinking applications.

       For  applications  that are hugepage-aware and linked to the library get_huge_pages() can be used for the
       direct allocation of hugepage-backed regions.

       Unless otherwise specified, libhugetlbfs will use the default hugepage size to back memory  regions.  The
       default  size is the value of Hugepagesize displayed in /proc/meminfo. The size can be specified in bytes
       or in kilobytes, megabytes, or gigabytes by appending K, M, or G respectively. It is an error to  specify
       a  invalid,  unsupported, or otherwise unconfigured huge page size. Kernel 2.6.27 or later is required to
       specify any pagesize other than the default.

       See /usr/share/docs/libhugetlbfs/HOWTO for detailed instructions on  how  the  library  should  be  used,
       particularly  when  relinking  the  application.   This  manual  page  provides  a  brief synopsis of the
       environment variables as a quick reference.

       The following variables affect what memory regions are backed by hugepages. In all cases, the environment
       being unset implies the feature should remain disabled.

       HUGETLB_DEFAULT_PAGE_SIZE=<pagesize>
              This sets the default hugepage size to be used by libhugetlbfs.  If not set, libhugetlbfs will use
              the kernel's default hugepage size.

       HUGETLB_MORECORE=[yes|<pagesize>]
              This enables the hugepage malloc() feature, instructing libhugetlbfs to  override  glibc's  normal
              morecore()  function  with  a  hugepage version and use it for malloc().  All application malloc()
              memory should come from hugepage memory until it runs out, it will then fallback  to  base  pages.
              Note  that  applications  that  use  custom  allocators  may not be able to back their heaps using
              hugepages and this environment variable. It may be necessary to modify the custom allocator to use
              get_huge_pages().

       HUGETLB_SHM=yes
              When this environment variable is set, the SHM_HUGETLB flag is added to the shmget() call and  the
              size parameter is aligned to back the shared memory segment with hugepages. In the event hugepages
              cannot  be  used,  base  pages  will  be used instead and a warning will be printed to explain the
              failure. The pagesize cannot be specified with this  parameter.  To  change  the  kernels  default
              hugepage size, use the pagesize= kernel boot parameter (2.6.26 or later required).

       HUGETLB_ELFMAP=[no|[R[<=pagesize>]:[W[<=pagesize>]]]
              If  the  application has been relinked (see the HOWTO for instructions), this environment variable
              determines whether read-only, read-write, both or no segments are backed  by  hugepages  and  what
              pagesize  should  be used. If the recommended relinking method has been used, then hugeedit can be
              used to automatically back the text or data by default.

       HUGETLB_FORCE_ELFMAP=yes
              Force the use of hugepages for text and data  segments  even  if  the  application  has  not  been
              relinked  to  align  the  ELF  segments  on a hugepage boundary.  Partial segment remapping is not
              guaranteed to work and the segments must be large enough to contain at least one hugepage for  the
              remapping to occur.

       The following options affect how libhugetlbfs behaves.

       HUGETLB_RESTRICT_EXE=e1:e2:...:eN
              By  default, libhugetlbfs will act on any program that it is loaded with, either via LD_PRELOAD or
              by explicitly linking with -lhugetlbfs.

              There are situations in which it is  desirable  to  restrict  libhugetlbfs'  actions  to  specific
              programs.  For example, some ISV applications are wrapped in a series of scripts that invoke bash,
              python,  and/or  perl.   It  is  more  convenient  to  set  the  environment  variables related to
              libhugetlbfs before invoking the wrapper scripts, yet this  has  the  unintended  and  undesirable
              consequence  of causing the script interpreters to use and consume hugepages.  There is no obvious
              benefit to causing the script interpreters to use hugepages, and there is  a  clear  disadvantage:
              fewer hugepages are available to the actual application.

              To  address this scenario, set HUGETLB_RESTRICT_EXE to a colon-separated list of programs to which
              the other libhugetlbfs environment variables should apply.  (If not set, libhugetlbfs will attempt
              to apply the requested actions to all programs.)  For example,

                  HUGETLB_RESTRICT_EXE=hpcc:long_hpcc

              will restrict libhugetlbfs' actions to programs named /home/fred/hpcc and /bench/long_hpcc but not
              /bin/hpcc_no.

       HUGETLB_MORECORE_SHRINK=yes
              By default, the hugepage heap does not shrink. Shrinking is enabled by  setting  this  environment
              variable.  It  is  disabled  by  default  as  glibc  occasionally exhibits strange behaviour if it
              mistakes the heap returned by libhugetlbfs as a foreign brk().

       HUGETLB_NO_PREFAULT
              By default libhugetlbfs will prefault regions it creates to ensure they can be referenced  without
              receiving  a  SIGKILL.  On  kernels  older  than  2.6.27, this was necessary as the system did not
              guarantee that future faults would succeed on regions mapped MAP_PRIVATE.  Prefaulting impacts the
              performance of malloc() and can result in poor placement on NUMA  systems.  If  it  is  known  the
              hugepage  pool  is  large  enough  to  run  the application or the kernel is 2.6.27 or later, this
              environment variable should be set.

       HUGETLB_NO_RESERVE=yes

              By default, the kernel will reserve huge pages at mmap() time to ensure that  future  faults  will
              succeed.  This avoids unexpected application failure at fault time but some applications depend on
              memory overcommit to create large sparse mappings. For this  type  of  application,  setting  this
              environment  variable will create huge page backed mappings without a reservation. Use this option
              with extreme care as in the event huge pages are not available  when  the  mapping  is  used,  the
              application  will be killed. On older kernels, the use of this feature can trigger the OOM killer.
              Hence, even with this variable set, reservations may still be used for safety.

       HUGETLB_MORECORE_HEAPBASE=address
              libhugetlbfs normally picks an address to use as the base of the heap for malloc()  automatically.
              This environment variable fixes which address is used.

       HUGETLB_PATH=<path>
              The  path  to  the hugetlbfs mount is automatically determined at run-time. In the event there are
              multiple mounts and the wrong one is being selected, use this option to select  the  correct  one.
              This  may  be  the  case  if an application-specific mount with a fixed quota has been created for
              example.

       HUGETLB_SHARE=1
              By default, libhugetlbfs uses unlinked hugetlbfs files to store remapped program segment data.  If
              the  same  program  is  started multiple times using hugepage segments, multiple hugepages will be
              used to store the same program data. The reduce this wastage, setting  this  environment  variable
              will  share read-only segments between multiple invocations of a program at the cost of the memory
              being used whether the applications are running or not. It  is  also  possible  that  a  malicious
              application  inferfere  with  other  applications executable code. See the HOWTO for more detailed
              information on this topic.

       The following options control the verbosity of libhugetlbfs.

       HUGETLB_VERBOSE=<level>
              The default value for this is 1 and the range of the value is from 0 to 99. The higher the  value,
              the more verbose the output is. 0 is quiet and 3 will output much debugging information.

       HUGETLB_DEBUG
              Once  set,  this  will give very detailed output on what is happening in the library and run extra
              diagnostics.

FILES

       [DESTDIR|/usr/share]/doc/libhugetlbfs/HOWTO

SEE ALSO

       oprofile(1),   ld.hugetlbfs(1),   hugectl(8),   hugeedit(8),   gethugepagesize(3),   gethugepagesizes(3),
       getpagesizes(3),    hugetlbfs_test_path(3),    hugetlbfs_find_path(3),   hugetlbfs_find_path_for_size(3),
       hugetlbfs_test_path(3),            hugetlbfs_test_path_for_size(3),             hugetlbfs_unlinked_fd(3),
       hugetlbfs_unlinked_fd_for_size(3), get_huge_pages(3), free_huge_pages(3)

AUTHORS

       libhugetlbfs was written by various people on the libhugetlbfs-devel mailing list.

                                               September 27, 2008                                LIBHUGETLBFS(7)