Ubuntu Manpage: pypy3 - fast, compliant alternative implementation of the Python 3 language

NAME

       pypy3 - fast, compliant alternative implementation of the Python 3 language

SYNOPSIS

       pypy3 [options] [-c cmd|-m mod|file.py|-] [arg...]

OPTIONS

       -i     Inspect interactively after running script.

       -O     Skip assert statements.

       -OO    Remove docstrings when importing modules in addition to -O.

       -c CMD Program passed in as CMD (terminates option list).

       -S     Do not import site on initialization.

       -s     Don't add the user site directory to sys.path.

       -u     Unbuffered binary stdout and stderr.

       -h, --help
              Show a help message and exit.

       -m MOD Library module to be run as a script (terminates option list).

       -W ARG Warning control (arg is action:message:category:module:lineno).

       -E     Ignore environment variables (such as PYTHONPATH).

       -B     Disable writing bytecode (.pyc) files.

       -X track-resources
              Produce  a  ResourceWarning  whenever  a  file  or  socket is closed by the garbage
              collector.

       --version
              Print the PyPy version.

       --info Print translation information about this PyPy executable.

       --jit ARG
              Low level JIT parameters.  Mostly internal.  Run --jit help for more information.

ENVIRONMENT

PYTHONPATH
Add directories to pypy3's module search path. The format is the same as shell's
PATH.

PYTHONSTARTUP
A script referenced by this variable will be executed before the first prompt is
displayed, in interactive mode.

PYTHONDONTWRITEBYTECODE
If set to a non-empty value, equivalent to the -B option. Disable writing .pyc
files.

PYTHONINSPECT
If set to a non-empty value, equivalent to the -i option. Inspect interactively
after running the specified script.

PYTHONIOENCODING
If this is set, it overrides the encoding used for stdin/stdout/stderr. The syntax
is encodingname:errorhandler The errorhandler part is optional and has the same
meaning as in str.encode.

PYTHONNOUSERSITE
If set to a non-empty value, equivalent to the -s option. Don't add the user site
directory to sys.path.

PYTHONWARNINGS
If set, equivalent to the -W option (warning control). The value should be a
comma-separated list of -W parameters.

PYPYLOG
If set to a non-empty value, enable logging, the format is:

fname or +fname
logging for profiling: includes all debug_start/debug_stop but not any
nested debug_print. fname can be - to log to stderr. The +fname form can
be used if there is a : in fname

:fname Full logging, including debug_print.

prefix:fname
Conditional logging. Multiple prefixes can be specified, comma-separated.
Only sections whose name match the prefix will be logged.

PYPYLOG=jit-log-opt,jit-backend:logfile will generate a log suitable for jitviewer,
a tool for debugging performance issues under PyPy.

PYPY_IRC_TOPIC
If set to a non-empty value, print a random #pypy IRC topic at startup of
interactive mode.

PyPy's default garbage collector is called incminimark - it's an incremental, generational
moving collector. Here we hope to explain a bit how it works and how it can be tuned to
suit the workload.

Incminimark first allocates objects in so called nursery - place for young objects, where
allocation is very cheap, being just a pointer bump. The nursery size is a very crucial
variable - depending on your workload (one or many processes) and cache sizes you might
want to experiment with it via PYPY_GC_NURSERY environment variable. When the nursery is
full, a minor collection is performed. Freed objects are no longer referencable and just
die, just by not being referenced any more; on the other hand, objects found to still be
alive must survive and are copied from the nursery to the old generation. Either to
arenas, which are collections of objects of the same size, or directly allocated with
malloc if they're larger. (A third category, the very large objects, are initially
allocated outside the nursery and never move.)

Since Incminimark is an incremental GC, the major collection is incremental: the goal is
not to have any pause longer than 1ms, but in practice it depends on the size and
characteristics of the heap: occasionally, there can be pauses between 10-100ms.

Semi-manual GC management
If there are parts of the program where it is important to have a low latency, you might
want to control precisely when the GC runs, to avoid unexpected pauses. Note that this
has effect only on major collections, while minor collections continue to work as usual.

As explained above, a full major collection consists of N steps, where N depends on the
size of the heap; generally speaking, it is not possible to predict how many steps will be
needed to complete a collection.

gc.enable() and gc.disable() control whether the GC runs collection steps automatically.
When the GC is disabled the memory usage will grow indefinitely, unless you manually call
gc.collect() and gc.collect_step().

gc.collect() runs a full major collection.

gc.collect_step() runs a single collection step. It returns an object of type
GcCollectStepStats, the same which is passed to the corresponding GC Hooks. The following
code is roughly equivalent to a gc.collect():

while True:
if gc.collect_step().major_is_done:
break

For a real-world example of usage of this API, you can look at the 3rd-party module
pypytools.gc.custom, which also provides a with customgc.nogc() context manager to mark
sections where the GC is forbidden.

Fragmentation
Before we discuss issues of "fragmentation", we need a bit of precision. There are two
kinds of related but distinct issues:

• If the program allocates a lot of memory, and then frees it all by dropping all
references to it, then we might expect to see the RSS to drop. (RSS = Resident Set Size
on Linux, as seen by "top"; it is an approximation of the actual memory usage from the
OS's point of view.) This might not occur: the RSS may remain at its highest value.
This issue is more precisely caused by the process not returning "free" memory to the
OS. We call this case "unreturned memory".

• After doing the above, if the RSS didn't go down, then at least future allocations
should not cause the RSS to grow more. That is, the process should reuse unreturned
memory as long as it has got some left. If this does not occur, the RSS grows even
larger and we have real fragmentation issues.

gc.get_stats
There is a special function in the gc module called get_stats(memory_pressure=False).

memory_pressure controls whether or not to report memory pressure from objects allocated
outside of the GC, which requires walking the entire heap, so it's disabled by default due
to its cost. Enable it when debugging mysterious memory disappearance.

Example call looks like that:

>>> gc.get_stats(True)
Total memory consumed:
GC used: 4.2MB (peak: 4.2MB)
in arenas: 763.7kB
rawmalloced: 383.1kB
nursery: 3.1MB
raw assembler used: 0.0kB
memory pressure: 0.0kB
-----------------------------
Total: 4.2MB

Total memory allocated:
GC allocated: 4.5MB (peak: 4.5MB)
in arenas: 763.7kB
rawmalloced: 383.1kB
nursery: 3.1MB
raw assembler allocated: 0.0kB
memory pressure: 0.0kB
-----------------------------
Total: 4.5MB

In this particular case, which is just at startup, GC consumes relatively little memory
and there is even less unused, but allocated memory. In case there is a lot of unreturned
memory or actual fragmentation, the "allocated" can be much higher than "used". Generally
speaking, "peak" will more closely resemble the actual memory consumed as reported by RSS.
Indeed, returning memory to the OS is a hard and not solved problem. In PyPy, it occurs
only if an arena is entirely free---a contiguous block of 64 pages of 4 or 8 KB each. It
is also rare for the "rawmalloced" category, at least for common system implementations of
malloc().

The details of various fields:

• GC in arenas - small old objects held in arenas. If the amount "allocated" is much
higher than the amount "used", we have unreturned memory. It is possible but unlikely
that we have internal fragmentation here. However, this unreturned memory cannot be
reused for any malloc(), including the memory from the "rawmalloced" section.

• GC rawmalloced - large objects allocated with malloc. This is gives the current (first
block of text) and peak (second block of text) memory allocated with malloc(). The
amount of unreturned memory or fragmentation caused by malloc() cannot easily be
reported. Usually you can guess there is some if the RSS is much larger than the total
memory reported for "GC allocated", but do keep in mind that this total does not include
malloc'ed memory not known to PyPy's GC at all. If you guess there is some, consider
using jemalloc as opposed to system malloc.

• nursery - amount of memory allocated for nursery, fixed at startup, controlled via an
environment variable

• raw assembler allocated - amount of assembler memory that JIT feels responsible for

• memory pressure, if asked for - amount of memory we think got allocated via external
malloc (eg loading cert store in SSL contexts) that is kept alive by GC objects, but not
accounted in the GC

GC Hooks
GC hooks are user-defined functions which are called whenever a specific GC event occur,
and can be used to monitor GC activity and pauses. You can install the hooks by setting
the following attributes:

gc.hooks.on_gc_minor
Called whenever a minor collection occurs. It corresponds to gc-minor sections
inside PYPYLOG.

gc.hooks.on_gc_collect_step
Called whenever an incremental step of a major collection occurs. It corresponds to
gc-collect-step sections inside PYPYLOG.

gc.hooks.on_gc_collect
Called after the last incremental step, when a major collection is fully done. It
corresponds to gc-collect-done sections inside PYPYLOG.

To uninstall a hook, simply set the corresponding attribute to None. To install all hooks
at once, you can call gc.hooks.set(obj), which will look for methods on_gc_* on obj. To
uninstall all the hooks at once, you can call gc.hooks.reset().

The functions called by the hooks receive a single stats argument, which contains various
statistics about the event.

Note that PyPy cannot call the hooks immediately after a GC event, but it has to wait
until it reaches a point in which the interpreter is in a known state and calling
user-defined code is harmless. It might happen that multiple events occur before the hook
is invoked: in this case, you can inspect the value stats.count to know how many times the
event occurred since the last time the hook was called. Similarly, stats.duration
contains the total time spent by the GC for this specific event since the last time the
hook was called.

On the other hand, all the other fields of the stats object are relative only to the last
event of the series.

The attributes for GcMinorStats in the on_gc_minor hook are:

count The number of minor collections occurred since the last hook call.

duration
The total time spent inside minor collections since the last hook call, in seconds.

duration_min
The duration of the fastest minor collection since the last hook call.

duration_max
The duration of the slowest minor collection since the last hook call.

total_memory_used
The amount of memory used at the end of the minor collection, in bytes. This
include the memory used in arenas (for GC-managed memory) and raw-malloced
memory (e.g., the content of numpy arrays).

pinned_objects
the number of pinned objects.

The attributes for GcCollectStepStats in the on_gc_collect_step hook are:

count, duration, duration_min, duration_max
See above.

oldstate, newstate
Integers which indicate the state of the GC before and after the step.

major_is_done
Boolean which indicate whether this was the last step of the major collection

The value of oldstate and newstate is one of these constants, defined inside
gc.GcCollectStepStats: STATE_SCANNING, STATE_MARKING, STATE_SWEEPING, STATE_FINALIZING,
STATE_USERDEL. It is possible to get a string representation of it by indexing the
GC_STATES tuple.

The attributes for GcCollectStats in the on_gc_collect hook are:

count See above.

num_major_collects
The total number of major collections which have been done since the start.
Contrarily to count, this is an always-growing counter and it's not reset between
invocations.

arenas_count_before, arenas_count_after
Number of arenas used before and after the major collection.

arenas_bytes
Total number of bytes used by GC-managed objects.

rawmalloc_bytes_before, rawmalloc_bytes_after
Total number of bytes used by raw-malloced objects, before and after the major
collection.

pinned_objects
the number of pinned objects.

Note that GcCollectStats does not have a duration field. This is because all the GC work
is done inside gc-collect-step: gc-collect-done is used only to give additional stats, but
doesn't do any actual work.

Here is an example of GC hooks in use:

import sys
import gc

class MyHooks(object):
done = False

def on_gc_minor(self, stats):
print 'gc-minor: count = %02d, duration = %d' % (stats.count,
stats.duration)

def on_gc_collect_step(self, stats):
old = gc.GcCollectStepStats.GC_STATES[stats.oldstate]
new = gc.GcCollectStepStats.GC_STATES[stats.newstate]
print 'gc-collect-step: %s --> %s' % (old, new)
print ' count = %02d, duration = %d' % (stats.count,
stats.duration)

def on_gc_collect(self, stats):
print 'gc-collect-done: count = %02d' % stats.count
self.done = True

hooks = MyHooks()
gc.hooks.set(hooks)

# simulate some GC activity
lst = []
while not hooks.done:
lst = [lst, 1, 2, 3]

Environment variables
PyPy's default incminimark garbage collector is configurable through several environment
variables:

PYPY_GC_NURSERY
The nursery size. Defaults to 1/2 of your last-level cache, or 4M if unknown.
Small values (like 1 or 1KB) are useful for debugging.

PYPY_GC_NURSERY_DEBUG
If set to non-zero, will fill nursery with garbage, to help debugging.

PYPY_GC_INCREMENT_STEP
The size of memory marked during the marking step. Default is size of nursery
times 2. If you mark it too high your GC is not incremental at all. The minimum is
set to size that survives minor collection times 1.5 so we reclaim anything all the
time.

PYPY_GC_MAJOR_COLLECT
Major collection memory factor. Default is 1.82, which means trigger a major
collection when the memory consumed equals 1.82 times the memory really used at the
end of the previous major collection.

PYPY_GC_GROWTH
Major collection threshold's max growth rate. Default is 1.4. Useful to collect
more often than normally on sudden memory growth, e.g. when there is a temporary
peak in memory usage.

PYPY_GC_MAX
The max heap size. If coming near this limit, it will first collect more often,
then raise an RPython MemoryError, and if that is not enough, crash the program
with a fatal error. Try values like 1.6GB.

PYPY_GC_MAX_DELTA
The major collection threshold will never be set to more than PYPY_GC_MAX_DELTA the
amount really used after a collection. Defaults to 1/8th of the total RAM size
(which is constrained to be at most 2/3/4GB on 32-bit systems). Try values like
200MB.

PYPY_GC_MIN
Don't collect while the memory size is below this limit. Useful to avoid spending
all the time in the GC in very small programs. Defaults to 8 times the nursery.

PYPY_GC_DEBUG
Enable extra checks around collections that are too slow for normal use. Values
are 0 (off), 1 (on major collections) or 2 (also on minor collections).

PYPY_GC_MAX_PINNED
The maximal number of pinned objects at any point in time. Defaults to a
conservative value depending on nursery size and maximum object size inside the
nursery. Useful for debugging by setting it to 0.

AUTHOR

       The PyPy Project

COPYRIGHT