NAME

       xen-tscmode - Xen TSC (time stamp counter) and timekeeping discussion

OVERVIEW

       As of Xen 4.0, a new config option called tsc_mode may be specified for each domain.  The
       default for tsc_mode handles the vast majority of hardware and software environments.
       This document is targeted for Xen users and administrators that may need to select a non-
       default tsc_mode.

       Proper selection of tsc_mode depends on an understanding not only of the guest operating
       system (OS), but also of the application set that will ever run on this guest OS.  This is
       because tsc_mode applies equally to both the OS and ALL apps that are running on this
       domain, now or in the future.

       Key questions to be answered for the OS and/or each application are:

       •   Does the OS/app use the rdtsc instruction at all?  (We will explain below how to
           determine this.)

       •   At what frequency is the rdtsc instruction executed by either the OS or any running
           apps?  If the sum exceeds about 10,000 rdtsc instructions per second per processor, we
           call this a "high-TSC-frequency" OS/app/environment.  (This is relatively rare, and
           developers of OS's and apps that are high-TSC-frequency are usually aware of it.)

       •   If the OS/app does use rdtsc, will it behave incorrectly if "time goes backwards" or
           if the frequency of the TSC suddenly changes?  If so, we call this a "TSC-sensitive"
           app or OS; otherwise it is "TSC-resilient".

       This last is the US$64,000 question as it may be very difficult (or, for legacy apps, even
       impossible) to predict all possible failure cases.  As a result, unless proven otherwise,
       any app that uses rdtsc must be assumed to be TSC-sensitive and, as we will see, this is
       the default starting in Xen 4.0.

       Xen's new tsc_mode parameter determines the circumstances under which the family of rdtsc
       instructions are executed "natively" vs emulated.  Roughly speaking, native means rdtsc is
       fast but TSC-sensitive apps may, under unpredictable circumstances, run incorrectly;
       emulated means there is some performance degradation (unobservable in most cases), but
       TSC-sensitive apps will always run correctly.  Prior to Xen 4.0, all rdtsc instructions
       were native: "fast but potentially incorrect."  Starting at Xen 4.0, the default is that
       all rdtsc instructions are "correct but potentially slow".  The tsc_mode parameter in 4.0
       provides an intelligent default but allows system administrator's to adjust how rdtsc
       instructions are executed differently for different domains.

       The non-default choices for tsc_mode are:

       •   tsc_mode=1 (always emulate).

           All rdtsc instructions are emulated; this is the best choice when TSC-sensitive apps
           are running and it is necessary to understand worst-case performance degradation for a
           specific hardware environment.

       •   tsc_mode=2 (never emulate).

           This is the same as prior to Xen 4.0 and is the best choice if it is certain that all
           apps running in this VM are TSC-resilient and highest performance is required.

       •   tsc_mode=3 (PVRDTSCP).

           This mode has been removed.

       If tsc_mode is left unspecified (or set to tsc_mode=0), a hybrid algorithm is utilized to
       ensure correctness while providing the best performance possible given:

       •   the requirement of correctness,

       •   the underlying hardware, and

       •   whether or not the VM has been saved/restored/migrated

       To understand this in more detail, the rest of this document must be read.

DETERMINING RDTSC FREQUENCY

       To determine the frequency of rdtsc instructions that are emulated, an "xl" command can be
       used by a privileged user of domain0.  The command:

           # xl debug-key s; xl dmesg | tail

       provides information about TSC usage in each domain where TSC emulation is currently
       enabled.

TSC HISTORY

       To understand tsc_mode completely, some background on TSC is required:

       The x86 "timestamp counter", or TSC, is a 64-bit register on each processor that increases
       monotonically.  Historically, TSC incremented every processor cycle, but on recent
       processors, it increases at a constant rate even if the processor changes frequency (for
       example, to reduce processor power usage).  TSC is known by x86 programmers as the
       fastest, highest-precision measurement of the passage of time so it is often used as a
       foundation for performance monitoring.  And since it is guaranteed to be monotonically
       increasing and, at 64 bits, is guaranteed to not wraparound within 10 years, it is
       sometimes used as a random number or a unique sequence identifier, such as to stamp
       transactions so they can be replayed in a specific order.

       On most older SMP and early multi-core machines, TSC was not synchronized between
       processors.  Thus if an application were to read the TSC on one processor, then was moved
       by the OS to another processor, then read TSC again, it might appear that "time went
       backwards".  This loss of monotonicity resulted in many obscure application bugs when TSC-
       sensitive apps were ported from a uniprocessor to an SMP environment; as a result, many
       applications -- especially in the Windows world -- removed their dependency on TSC and
       replaced their timestamp needs with OS-specific functions, losing both performance and
       precision. On some more recent generations of multi-core machines, especially multi-socket
       multi-core machines, the TSC was synchronized but if one processor were to enter certain
       low-power states, its TSC would stop, destroying the synchrony and again causing obscure
       bugs.  This reinforced decisions to avoid use of TSC altogether.  On the most recent
       generations of multi-core machines, however, synchronization is provided across all
       processors in all power states, even on multi-socket machines, and provide a flag that
       indicates that TSC is synchronized and "invariant".  Thus TSC is once again useful for
       applications, and even newer operating systems are using and depending upon TSC for
       critical timekeeping tasks when running on these recent machines.

       We will refer to hardware that ensures TSC is both synchronized and invariant as "TSC-
       safe" and any hardware on which TSC is not (or may not remain) synchronized as "TSC-
       unsafe".

       As a result of TSC's sordid history, two classes of applications use TSC: old applications
       designed for single processors, and the most recent enterprise applications which require
       high-frequency high-precision timestamping.

       We will refer to apps that might break if running on a TSC-unsafe machine as "TSC-
       sensitive"; apps that don't use TSC, or do use TSC but use it in a way that monotonicity
       and frequency invariance are unimportant as "TSC-resilient".

       The emergence of virtualization once again complicates the usage of TSC.  When features
       such as save/restore or live migration are employed, a guest OS and all its currently
       running applications may be invisibly transported to an entirely different physical
       machine.  While TSC may be "safe" on one machine, it is essentially impossible to
       precisely synchronize TSC across a data center or even a pool of machines.  As a result,
       when run in a virtualized environment, rare and obscure "time going backwards" problems
       might once again occur for those TSC-sensitive applications.  Worse, if a guest OS moves
       from, for example, a 3GHz machine to a 1.5GHz machine, attempts by an OS/app to measure
       time intervals with TSC may without notice be incorrect by a factor of two.

       The rdtsc (read timestamp counter) instruction is used to read the TSC register.  The
       rdtscp instruction is a variant of rdtsc on recent processors.  We refer to these together
       as the rdtsc family of instructions, or just "rdtsc".  Instructions in the rdtsc family
       are non-privileged, but privileged software may set a cpuid bit to cause all rdtsc family
       instructions to trap.  This trap can be detected by Xen, which can then transparently
       "emulate" the results of the rdtsc instruction and return control to the code following
       the rdtsc instruction.

       To provide a "safe" TSC, i.e. to ensure both TSC monotonicity and a fixed rate, Xen
       provides rdtsc emulation whenever necessary or when explicitly specified by a per-VM
       configuration option.  TSC emulation is relatively slow -- roughly 15-20 times slower than
       the rdtsc instruction when executed natively.  However, except when an OS or application
       uses the rdtsc instruction at a high frequency (e.g. more than about 10,000 times per
       second per processor), this performance degradation is not noticeable (i.e. <0.3%).  And,
       TSC emulation is nearly always faster than OS-provided alternatives (e.g. Linux's
       gettimeofday).  For environments where it is certain that all apps are TSC-resilient (e.g.
       "TSC-safeness" is not necessary) and highest performance is a requirement, TSC emulation
       may be entirely disabled (tsc_mode==2).

       The default mode (tsc_mode==0) checks TSC-safeness of the underlying hardware on which the
       virtual machine is launched.  If it is TSC-safe, rdtsc will execute at hardware speed; if
       it is not, rdtsc will be emulated.  Once a virtual machine is save/restored or migrated,
       however, there are two possibilities: TSC remains native IF the source physical machine
       and target physical machine have the same TSC frequency (or, for HVM/PVH guests, if TSC
       scaling support is available); else TSC is emulated.  Note that, though emulated, the
       "apparent" TSC frequency will be the TSC frequency of the initial physical machine, even
       after migration.

       Finally, tsc_mode==1 always enables TSC emulation, regardless of the underlying physical
       hardware. The "apparent" TSC frequency will be the TSC frequency of the initial physical
       machine, even after migration.  This mode is useful to measure any performance degradation
       that might be encountered by a tsc_mode==0 domain after migration occurs, or a tsc_mode==3
       domain when it is running on TSC-unsafe hardware.

       Note that while Xen ensures that an emulated TSC is "safe" across migration, it does not
       ensure that it continues to tick at the same rate during the actual migration.  As an
       oversimplified example, if TSC is ticking once per second in a guest, and the guest is
       saved when the TSC is 1000, then restored 30 seconds later, TSC is only guaranteed to be
       greater than or equal to 1001, not precisely 1030.  This has some OS implications as will
       be seen in the next section.

TSC INVARIANT BIT and NO_MIGRATE

       Related to TSC emulation, the "TSC Invariant" bit is architecturally defined in a cpuid
       bit on the most recent x86 processors.  If set, TSC invariance ensures that the TSC is
       "safe", that is it will increment at a constant rate regardless of power events, will be
       synchronized across all processors, and was properly initialized to zero on all processors
       at boot-time by system hardware/BIOS.  As long as system software never writes to TSC, TSC
       will be safe and continuously incremented at a fixed rate and thus can be used as a system
       "clocksource".

       This bit is used by some OS's, and specifically by Linux starting with version 2.6.30(?),
       to select TSC as a system clocksource.  Once selected, TSC remains the Linux system
       clocksource unless manually overridden.  In a virtualized environment, since it is not
       possible to synchronize TSC across all the machines in a pool or data center, a migration
       may "break" TSC as a usable clocksource; while time will not go backwards, it may not
       track wallclock time well enough to avoid certain time-sensitive consequences.  As a
       result, Xen can only expose the TSC Invariant bit to a guest OS if it is certain that the
       domain will never migrate.  As of Xen 4.0, the "no_migrate=1" VM configuration option may
       be specified to disable migration.  If no_migrate is selected and the VM is running on a
       physical machine with "TSC Invariant", Linux 2.6.30+ will safely use TSC as the system
       clocksource.  But, attempts to migrate or, once saved, restore this domain will fail.

       There is another cpuid-related complication: The x86 cpuid instruction is non-privileged.
       HVM domains are configured to always trap this instruction to Xen, where Xen can "filter"
       the result.  In a PV OS, all cpuid instructions have been replaced by a paravirtualized
       equivalent of the cpuid instruction ("pvcpuid") and also trap to Xen.  But apps in a PV
       guest that use a cpuid instruction execute it directly, without a trap to Xen.  As a
       result, an app may directly examine the physical TSC Invariant cpuid bit and make
       decisions based on that bit.

HARDWARE TSC SCALING

       Intel VMX TSC scaling and AMD SVM TSC ratio allow the guest TSC read by guest rdtsc/p
       increasing in a different frequency than the host TSC frequency.

       If a HVM container in default TSC mode (tsc_mode=0) is created on a host that provides
       constant TSC, its guest TSC frequency will be the same as the host. If it is later
       migrated to another host that provides constant TSC and supports Intel VMX TSC scaling/AMD
       SVM TSC ratio, its guest TSC frequency will be the same before and after migration.

       For above HVM container in default TSC mode (tsc_mode=0), if above hosts support rdtscp,
       both guest rdtsc and rdtscp instructions will be executed natively before and after
       migration.

AUTHORS

       Dan Magenheimer <dan.magenheimer@oracle.com>