The search in these cases should include all finally handlers that are
after the current ip. If a handler starts at exactly ip then it is
considered "after" the ip. This can happen when END_FINALLY is followed
immediately by a finally handler (from a different finally).
Consider the function:
def f():
try:
return 0
finally:
print(1)
The current bytecode emitter generates the following code:
00 SETUP_FINALLY 5
02 LOAD_CONST_SMALL_INT 0
03 RETURN_VALUE
04 LOAD_CONST_NONE ****
05 LOAD_GLOBAL print
07 LOAD_CONST_SMALL_INT 1
08 CALL_FUNCTION n=1 nkw=0
10 POP_TOP
11 END_FINALLY
12 LOAD_CONST_NONE
13 RETURN_VALUE
The LOAD_CONST_NONE marked with **** is dead code because it follows a
RETURN_VALUE, and nothing jumps to this LOAD_CONST_NONE. If the emitter
could remove this this dead code it would produce:
00 SETUP_FINALLY 4
02 LOAD_CONST_SMALL_INT 0
03 RETURN_VALUE
04 LOAD_GLOBAL print
06 LOAD_CONST_SMALL_INT 1
07 CALL_FUNCTION n=1 nkw=0
09 POP_TOP
10 END_FINALLY
11 LOAD_CONST_NONE
12 RETURN_VALUE
In this case the finally block (which starts at offset 4) immediately
follows the RETURN_VALUE. When RETURN_VALUE executes ip will point to
offset 4 in the bytecode (because the dispatch of the opcode does *ip++)
and so the finally handler will only be found if a >= comparison is used.
It's a similar story for break/continue:
while True:
try:
break
finally:
print(1)
Although technically in this case the > comparison still works because the
extra byte from the UNWIND_JUMP (encoding the number of exception handlers
to unwind) doesn't have a *ip++ (just a *ip) so ip remains pointing within
the UNWIND_JUMP opcode, and not at the start of the following finally
handler. Nevertheless, the change is made to use >= for consistency with
the RETURN_VALUE change.
Signed-off-by: Damien George <damien@micropython.org>
Catch calls to legacy:
MP_REGISTER_MODULE(name, module, enable)
Emit a friendly error suggesting they be rewritten to:
MP_REGISTER_MODULE(name, module).
Signed-off-by: Phil Howard <phil@pimoroni.com>
This file is not executable so shouldn't have the shebang line. This line
can cause issues when building on Windows msvc when the PyPython variable
is set to something other than "python", because it reverts back to using
the shebang line.
The top comment is also changed to """ style which matches all other
preprocessing scripts in the py/ directory.
Signed-off-by: Damien George <damien@micropython.org>
Without this, newer versions of gcc (eg 11.2.0) used with -O2 can warn
about `q_ptr` being maybe uninitialized, because it doesn't know that there
is at least one qstr being written in to this (alloca'd) memory.
As part of this, change the type of `n` to `size_t` so the compiler knows
it's unsigned and can generate better code.
Code size change for this commit:
bare-arm: -28 -0.049%
minimal x86: -4 -0.002%
unix x64: +0 +0.000%
unix nanbox: -16 -0.003%
stm32: -24 -0.006% PYBV10
cc3200: -32 -0.017%
esp8266: +8 +0.001% GENERIC
esp32: -52 -0.003% GENERIC
nrf: -24 -0.013% pca10040
rp2: -32 -0.006% PICO
samd: -28 -0.020% ADAFRUIT_ITSYBITSY_M4_EXPRESS
Signed-off-by: Damien George <damien@micropython.org>
This was made redundant by f2040bfc7e, which
also did not update this function for the change to qstr-opcode encoding,
so it does not work correctly anyway.
Signed-off-by: Damien George <damien@micropython.org>
Support for architecture-specific qstr linking was removed in
d4d53e9e11, where native code was changed to
access qstr values via qstr_table. The only remaining use for the special
qstr link table in persistentcode.c is to support native module written in
C, linked via mpy_ld.py. But native modules can also use the standard
module-level qstr_table (and obj_table) which was introduced in the .mpy
file reworking in f2040bfc7e.
This commit removes the remaining native qstr liking support in
persistentcode.c's load_raw_code function, and adds two new relocation
options for constants.qstr_table and constants.obj_table. mpy_ld.py is
updated to use these relocations options instead of the native qstr link
table.
Signed-off-by: Damien George <damien@micropython.org>
It's no longer needed because this macro is now processed after
preprocessing the source code via cpp (in the qstr extraction stage), which
means unused MP_REGISTER_MODULE's are filtered out by the preprocessor.
Signed-off-by: Damien George <damien@micropython.org>
This cleans up the parsing of MP_REGISTER_MODULE() and generation of
genhdr/moduledefs.h so that it uses the same process as compressed error
string messages, using the output of qstr extraction.
This makes sure all MP_REGISTER_MODULE()'s that are part of the build are
correctly picked up. Previously the extraction would miss some (eg if you
had a mod.c file in the board directory for an stm32 board).
Build speed is more or less unchanged.
Thanks to @stinos for the ports/windows/msvc/genhdr.targets changes.
Signed-off-by: Damien George <damien@micropython.org>
This allows mpy-cross to dynamically select whether ARMv7-M float
instructions are supported in @micropython.asm_thumb functions.
Signed-off-by: Damien George <damien@micropython.org>
This follows on from a5324a1074 and allows
mpy-cross to dynamically select whether ARMv7-M instructions are supported
in @micropython.asm_thumb functions.
The config option MICROPY_EMIT_INLINE_THUMB_ARMV7M is no longer needed, it
is now controlled by MICROPY_EMIT_THUMB_ARMV7M.
Signed-off-by: Damien George <damien@micropython.org>
The following changes are made:
- If MICROPY_VFS is enabled then mp_vfs_import_stat and mp_vfs_open are
automatically used for mp_import_stat and mp_builtin_open respectively.
- If MICROPY_PY_IO is enabled then "open" is automatically included in the
set of builtins, and points to mp_builtin_open_obj.
This helps to clean up and simplify the most common port configuration.
Signed-off-by: Damien George <damien@micropython.org>
The examples/natmod features0 and features1 examples now build and run on
ARMv6-M platforms. More complicated examples are not yet supported because
the compiler emits references to built-in functions like __aeabi_uidiv.
Signed-off-by: Damien George <damien@micropython.org>
If __thumb2__ is defined by the compiler then .mpy files marked as ARMV6M
and above (up to ARMV7EMDP) are supported. If it's not defined then only
ARMV6M .mpy files are supported. This makes sure that on CPUs like
Cortex-M0+ (where __thumb2__ is not defined) only .mpy files marked as
ARMV6M can be imported.
Signed-off-by: Damien George <damien@micropython.org>
This commit adjusts the asm_thumb_xxx functions so they can be dynamically
configured to use ARMv7-M instructions or not. This is available when
MICROPY_DYNAMIC_COMPILER is enabled, and then controlled by the value of
mp_dynamic_compiler.native_arch.
If MICROPY_DYNAMIC_COMPILER is disabled the previous behaviour is retained:
the functions emit ARMv7-M instructions only if MICROPY_EMIT_THUMB_ARMV7M
is enabled.
Signed-off-by: Damien George <damien@micropython.org>
This eliminates the need to save and restore the exception unwind handler
pointer when calling nlr_push.
Signed-off-by: Damien George <damien@micropython.org>
For example, ussl can come from axtls or mbedtls. If neither are enabled
then don't try and set an empty definition twice, and only include it
once in MICROPY_REGISTERED_MODULES.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Now that constant tuples are supported in the parser, eg (1, True, "str"),
it's a small step to allow anything that is a constant to be used with the
pattern:
from micropython import const
X = const(obj)
This commit makes the required changes to allow the following types of
constants:
from micropython import const
_INT = const(123)
_FLOAT = const(1.2)
_COMPLEX = const(3.4j)
_STR = const("str")
_BYTES = const(b"bytes")
_TUPLE = const((_INT, _STR, _BYTES))
_TUPLE2 = const((None, False, True, ..., (), _TUPLE))
Prior to this, only integers could be used in const(...).
Signed-off-by: Damien George <damien@micropython.org>
The recent rework of bytecode made all constants global with respect to the
module (previously, each function had its own constant table). That means
the constant table for a module is shared among all functions/methods/etc
within the module.
This commit add support to the compiler to de-duplicate constants in this
module constant table. So if a constant is used more than once -- eg 1.0
or (None, None) -- then the same object is reused for all instances.
For example, if there is code like `print(1.0, 1.0)` then the parser will
create two independent constants 1.0 and 1.0. The compiler will then (with
this commit) notice they are the same and only put one of them in the
constant table. The bytecode will then reuse that constant twice in the
print expression. That allows the second 1.0 to be reclaimed by the GC,
also means the constant table has one less entry so saves a word.
Signed-off-by: Damien George <damien@micropython.org>
Prior to this commit, all qstrs were required to be allocated (by calling
mp_emit_common_use_qstr) in the MP_PASS_SCOPE pass (the first one). But
this is an unnecessary restriction, which is lifted by this commit.
Lifting the restriction simplifies the compiler because it can allocate
qstrs in later passes.
This also generates better code, because in some cases (eg when a variable
is closed over) the scope of an identifier is not known until a bit later
and then the identifier no longer needs its qstr allocated in the global
table.
Code size is reduced for all ports with this commit.
Signed-off-by: Damien George <damien@micropython.org>
Some architectures (like esp32 xtensa) cannot read byte-wise from
executable memory. This means the prelude for native functions -- which is
usually located after the machine code for the native function -- must be
placed in separate memory that can be read byte-wise. Prior to this commit
this was achieved by enabling N_PRELUDE_AS_BYTES_OBJ for the emitter and
MICROPY_EMIT_NATIVE_PRELUDE_AS_BYTES_OBJ for the runtime. The prelude was
then placed in a bytes object, pointed to by the module's constant table.
This behaviour is changed by this commit so that a pointer to the prelude
is stored either in mp_obj_fun_bc_t.child_table, or in
mp_obj_fun_bc_t.child_table[num_children] if num_children > 0. The reasons
for doing this are:
1. It decouples the native emitter from runtime requirements, the emitted
code no longer needs to know if the system it runs on can/can't read
byte-wise from executable memory.
2. It makes all ports have the same emitter behaviour, there is no longer
the N_PRELUDE_AS_BYTES_OBJ option.
3. The module's constant table is now used only for actual constants in the
Python code. This allows further optimisations to be done with the
constants (eg constant deduplication).
Code size change for those ports that enable the native emitter:
unix x64: +80 +0.015%
stm32: +24 +0.004% PYBV10
esp8266: +88 +0.013% GENERIC
esp32: -20 -0.002% GENERIC[incl -112(data)]
rp2: +32 +0.005% PICO
Signed-off-by: Damien George <damien@micropython.org>
mpy-cross will now generate native code based on the size of
mp_code_state_native_t, and the runtime will use this struct to calculate
the offset of the .state field. This makes native code generation and
execution (which rely on this struct) independent to the settings
MICROPY_STACKLESS and MICROPY_PY_SYS_SETTRACE, both of which change the
size of the mp_code_state_t struct.
Fixes issue #5059.
Signed-off-by: Damien George <damien@micropython.org>
Prior to this commit, even with unicode disabled .py and .mpy files could
contain unicode characters, eg by entering them directly in a string as
utf-8 encoded.
The only thing the compiler disallowed (with unicode disabled) was using
\uxxxx and \Uxxxxxxxx notation to specify a character within a string with
value >= 0x100; that would give a SyntaxError.
With this change mpy-cross will now accept \u and \U notation to insert a
character with value >= 0x100 into a string (because the -mno-unicode
option is now gone, there's no way to forbid this). The runtime will
happily work with strings with such characters, just like it already works
with strings with characters that were utf-8 encoded directly.
This change simplifies things because there are no longer any feature
flags in .mpy files, and any bytecode .mpy will now run on any target.
Signed-off-by: Damien George <damien@micropython.org>
This replaces occurences of
foo_t *foo = m_new_obj(foo_t);
foo->base.type = &foo_type;
with
foo_t *foo = mp_obj_malloc(foo_t, &foo_type);
Excludes any places where base is a sub-field or when new0/memset is used.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This is to replace the following:
mp_foo_obj_t *self = m_new_obj(mp_foo_obj_t);
self->base.type = &mp_type_foo;
with:
mp_foo_obj_t *self = mp_obj_malloc(mp_foo_obj_t, &mp_type_foo);
Calling the function is less code than inlining setting the type
everywhere, adds up to ~100 bytes on PYBV11.
It also helps to avoid an easy mistake of forgetting to set the type.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
When in a class body or at the module level don't implicitly close over
variables that have been assigned to.
Fixes issue #8603.
Signed-off-by: Damien George <damien@micropython.org>
This contains a string useful for identifying the underlying machine. This
string is kept consistent with the second part of the REPL banner via the
new config option MICROPY_BANNER_MACHINE.
This makes os.uname() more or less redundant, as all the information in
os.uname() is now available in the sys module.
Signed-off-by: Damien George <damien@micropython.org>
This commit adds the git hash and build date to sys.version. This is
allowed according to CPython docs, and is what PyPy does. The docs state:
A string containing the version number of the Python interpreter plus
additional information on the build number and compiler used.
Eg on CPython:
Python 3.10.4 (main, Mar 23 2022, 23:05:40) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.version
'3.10.4 (main, Mar 23 2022, 23:05:40) [GCC 11.2.0]'
and PyPy:
Python 2.7.12 (5.6.0+dfsg-4, Nov 20 2016, 10:43:30)
[PyPy 5.6.0 with GCC 6.2.0 20161109] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>> import sys
>>>> sys.version
'2.7.12 (5.6.0+dfsg-4, Nov 20 2016, 10:43:30)\n[PyPy 5.6.0 with GCC ...
With this commit on MicroPython we now have:
MicroPython v1.18-371-g9d08eb024 on 2022-04-28; linux [GCC 11.2.0] v...
Use Ctrl-D to exit, Ctrl-E for paste mode
>>> import sys
>>> sys.version
'3.4.0; MicroPython v1.18-371-g9d08eb024 on 2022-04-28'
Note that the start of the banner is the same as the end of sys.version.
This helps to keep code size under control because the string can be reused
by the compiler.
Signed-off-by: Damien George <damien@micropython.org>
Entering tab at the REPL will now make it insert an indent (4 spaces) in
the following cases:
- after any whitespace on a line
- at the start of a line that is not the first line
This changes the existing behaviour where a tab would insert an indent only
if there were no matches in the auto-complete search, and it was the start
of the line. This means, if there were any symbols in the global
namespace, tab could never be used to indent.
Note that entering tab at the start of the first line will still do
auto-completion, but will now do nothing if there are no symbols in the
global namespace, which is more consistent than before.
Signed-off-by: Damien George <damien@micropython.org>
This commit adds support to the parser so that tuples which contain only
constant elements (bool, int, str, bytes, etc) are immediately converted to
a tuple object. This makes it more efficient to use tuples containing
constant data because they no longer need to be created at runtime by the
bytecode (or native code).
Furthermore, with this improvement constant tuples that are part of frozen
code are now able to be stored fully in ROM (this will be implemented in
later commits).
Code size is increased by about 400 bytes on Cortex-M4 platforms.
See related issue #722.
Signed-off-by: Damien George <damien@micropython.org>
To keep the separate parts of the code that use these values in sync. And
make it easier to add new object types.
Signed-off-by: Damien George <damien@micropython.org>
This reverts commit 7e8222ae06.
The prelude data must exist somewhere in the native code so load_raw_code
and mpy-tool.py can access and parse it.
Signed-off-by: Damien George <damien@micropython.org>
If MICROPY_SCHEDULER_STATIC_NODES is enabled then C code can declare a
static mp_sched_node_t and schedule a callback using
mp_sched_schedule_node(). In contrast to using mp_sched_schedule(), the
node version will have at most one pending callback outstanding, and will
always be able to schedule if there is nothing already scheduled on this
node. This guarantees that the the callback will be called exactly once
after it is scheduled.
Signed-off-by: Damien George <damien@micropython.org>
The Thumb instruction set has special 16 bit encodings for PUSH involving
LR and POP involving PC, which are commonly used in nested functions.
Using this encoding is particularly important for ARMv6-M, where the more
general 32 bit encoding of PUSH and POP is unavailable.
- The classification of source files in makeqstrdefs.py has been moved into
functions to consolidate the logic for that classification into a single
place.
- Classification of source files (into C or C++ or "other" files) is based
on the filename extension.
- For C++ there are many more common filename extensions than just ".cpp";
see "Options Controlling the Kind of Output" in man gcc for example. All
common extensions for C++ source files which need preprocessing have been
added.
The values are always real objects, only the key can be MP_OBJ_NULL to
indicate a **kwargs entry.
Signed-off-by: Damien George <damien@micropython.org>
There were two issues with the existing code:
1. "1 << i" is computed as a 32-bit number so would overflow when
executed on 64-bit machines (when mp_uint_t is 64-bit). This meant that
*args beyond 32 positions would not be handled correctly.
2. star_args must fit as a positive small int so that it is encoded
correctly in the emitted code. MP_SMALL_INT_BITS is too big because it
overflows a small int by 1 bit. MP_SMALL_INT_BITS - 1 does not work
because it produces a signed small int which is then sign extended when
extracted (even by mp_obj_get_int_truncated), and this sign extension
means that any position arg after *args is also treated as a star-arg.
So the maximum bit position is MP_SMALL_INT_BITS - 2. This means that
MP_OBJ_SMALL_INT_VALUE() can be used instead of
mp_obj_get_int_truncated() to get the value of star_args.
These issues are fixed by this commit, and a test added.
Signed-off-by: Damien George <damien@micropython.org>
This replaces instances of uint with size_t and int with ssize_t in
the mp_call_prepare_args_n_kw_var() function since all of the variables
are used as array offsets.
Also sort headers while we are touching this.
Signed-off-by: David Lechner <david@pybricks.com>
To reach this check, n_kw has to be >= 1 and therefore args2_alloc has
to be >= 2. Therefore new_alloc will always be >= 4. So this check will
never be true and can be removed.
Signed-off-by: David Lechner <david@pybricks.com>
This fixes overallocating an extra mp_obj_t when the length of *args and
**args is known. Previously we were allocating 1 mp_obj_t for each
n_args and n_kw plus the length of each *arg and **arg (if they are
known). Since n_args includes *args and n_kw includes **args, this was
allocating an extra mp_obj_t in addition to the length of these args
when unpacked.
To fix this, we just subtract 1 from the length to account for the 1
already implicitly allocated by n_args and n_kw.
Signed-off-by: David Lechner <david@pybricks.com>
This is a partial implementation of PEP 448 to allow unpacking multiple
star args in a function or method call.
This is implemented by changing the emitted bytecodes so that both
positional args and star args are stored as positional args. A bitmap is
added to indicate if an argument at a given position is a positional
argument or a star arg.
In the generated code, this new bitmap takes the place of the old star arg.
It is stored as a small int, so this means only the first N arguments can
be star args where N is the number of bits in a small int.
The runtime is modified to interpret this new bytecode format while still
trying to perform as few memory reallocations as possible.
Signed-off-by: David Lechner <david@pybricks.com>
This is a partial implementation of PEP 448 to allow multiple ** unpackings
when calling a function or method.
The compiler is modified to encode the argument as a None: obj key-value
pair (similar to how regular keyword arguments are encoded as str: obj
pairs). The extra object that was pushed on the stack to hold a single **
unpacking object is no longer used and is removed.
The runtime is modified to decode this new format.
Signed-off-by: David Lechner <david@pybricks.com>
This warning can happen on clang 13.0.1 building mpy-cross:
../py/vm.c:748:25: error: array index -3 refers past the last possible
element for an array in 64-bit address space containing 64-bit (8-byte)
elements (max possible 2305843009213693952 elements)
[-Werror,-Warray-bounds]
sp[-MP_OBJ_ITER_BUF_NSLOTS + 1] = MP_OBJ_NULL;
^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~
Using pointer access instead of array access works around this warning.
Fixes issue #8467.
Signed-off-by: Damien George <damien@micropython.org>
This commit adds optimised l32i/s32i functions that select the best load/
store encoding based on the size of the offset, and uses the function when
necessary in code generation.
Without this, ASM_LOAD_REG_REG_OFFSET() could overflow the word offset
(using a narrow encoding), for example when loading the prelude from the
constant table when there are many (>16) constants.
Fixes issue #8458.
Signed-off-by: Damien George <damien@micropython.org>
The sys module should always be available (if it's compiled in), eg to
change sys.path for importing. So provide an explicit alias from "sys" to
"usys" so that "import sys" can always work.
Signed-off-by: Damien George <damien@micropython.org>
These jumps are always forwards, and it's more efficient in the VM to
decode an unsigned argument. These opcodes are already optimised versions
of the sequence "dup-top pop-jump-if-x pop" so it doesn't hurt generality
to optimise them further.
Signed-off-by: Damien George <damien@micropython.org>
This commit introduces changes:
- All jump opcodes are changed to have variable length arguments, of either
1 or 2 bytes (previously they were fixed at 2 bytes). In most cases only
1 byte is needed to encode the short jump offset, saving bytecode size.
- The bytecode emitter now selects 1 byte jump arguments when the jump
offset is guaranteed to fit in 1 byte. This is achieved by checking if
the code size changed during the last pass and, if it did (if it shrank),
then requesting that the compiler make another pass to get the correct
offsets of the now-smaller code. This can continue multiple times until
the code stabilises. The code can only ever shrink so this iteration is
guaranteed to complete. In most cases no extra passes are needed, the
original 4 passes are enough to get it right by the 4th pass (because the
2nd pass computes roughly the correct labels and the 3rd pass computes
the correct size for the jump argument).
This change to the jump opcode encoding reduces .mpy files and RAM usage
(when bytecode is in RAM) by about 2% on average.
The performance of the VM is not impacted, at least within measurment of
the performance benchmark suite.
Code size is reduced for builds that include a decent amount of frozen
bytecode. ARM Cortex-M builds without any frozen code increase by about
350 bytes.
Signed-off-by: Damien George <damien@micropython.org>
Some compilers will warn about unused variables like scope_flags. So use
MP_BC_PRELUDE_SIG_DECODE() which will silence these warnings.
Signed-off-by: Damien George <damien@micropython.org>
This adds a new MP_SMALL_INT_BITS macro that is a compile-time constant
that contains the number of bits available in an MP_SMALL_INT.
We can use this in place of the runtime function mp_small_int_bits().
Signed-off-by: David Lechner <david@pybricks.com>
The bytecode state variables mp_showbc_code_start and mp_showbc_constants
have been removed and made local variables passed into the various
functions.
As part of this, the DECODE_PTR macro is fixed so it extracts the relevant
pointer from the child_table (a regression introduced in
f2040bfc7e).
Signed-off-by: Damien George <damien@micropython.org>
This means that all constants for EMIT_ARG(load_const_obj, obj) are created
in the parser (rather than some in the compiler).
Signed-off-by: Damien George <damien@micropython.org>
This commit adds generic support for mutable module attributes on built in
modules, by adding support for an optional hook function for module
attribute lookup. If a module wants to support additional attribute load/
store/delete (beyond what is in the constant, globals dict) then it should
add at the very end of its globals dict MP_MODULE_ATTR_DELEGATION_ENTRY().
This should point to a custom function which will handle any additional
attributes.
The mp_module_generic_attr() function is provided as a helper function for
additional attributes: it requires an array of qstrs (terminated in
MP_QSTRnull) and a corresponding array of objects (with a 1-1 mapping
between qstrs and objects). If the qstr is found in the array then the
corresponding object is loaded/stored/deleted.
Signed-off-by: Damien George <damien@micropython.org>
Background: .mpy files are precompiled .py files, built using mpy-cross,
that contain compiled bytecode functions (and can also contain machine
code). The benefit of using an .mpy file over a .py file is that they are
faster to import and take less memory when importing. They are also
smaller on disk.
But the real benefit of .mpy files comes when they are frozen into the
firmware. This is done by loading the .mpy file during compilation of the
firmware and turning it into a set of big C data structures (the job of
mpy-tool.py), which are then compiled and downloaded into the ROM of a
device. These C data structures can be executed in-place, ie directly from
ROM. This makes importing even faster because there is very little to do,
and also means such frozen modules take up much less RAM (because their
bytecode stays in ROM).
The downside of frozen code is that it requires recompiling and reflashing
the entire firmware. This can be a big barrier to entry, slows down
development time, and makes it harder to do OTA updates of frozen code
(because the whole firmware must be updated).
This commit attempts to solve this problem by providing a solution that
sits between loading .mpy files into RAM and freezing them into the
firmware. The .mpy file format has been reworked so that it consists of
data and bytecode which is mostly static and ready to run in-place. If
these new .mpy files are located in flash/ROM which is memory addressable,
the .mpy file can be executed (mostly) in-place.
With this approach there is still a small amount of unpacking and linking
of the .mpy file that needs to be done when it's imported, but it's still
much better than loading an .mpy from disk into RAM (although not as good
as freezing .mpy files into the firmware).
The main trick to make static .mpy files is to adjust the bytecode so any
qstrs that it references now go through a lookup table to convert from
local qstr number in the module to global qstr number in the firmware.
That means the bytecode does not need linking/rewriting of qstrs when it's
loaded. Instead only a small qstr table needs to be built (and put in RAM)
at import time. This means the bytecode itself is static/constant and can
be used directly if it's in addressable memory. Also the qstr string data
in the .mpy file, and some constant object data, can be used directly.
Note that the qstr table is global to the module (ie not per function).
In more detail, in the VM what used to be (schematically):
qst = DECODE_QSTR_VALUE;
is now (schematically):
idx = DECODE_QSTR_INDEX;
qst = qstr_table[idx];
That allows the bytecode to be fixed at compile time and not need
relinking/rewriting of the qstr values. Only qstr_table needs to be linked
when the .mpy is loaded.
Incidentally, this helps to reduce the size of bytecode because what used
to be 2-byte qstr values in the bytecode are now (mostly) 1-byte indices.
If the module uses the same qstr more than two times then the bytecode is
smaller than before.
The following changes are measured for this commit compared to the
previous (the baseline):
- average 7%-9% reduction in size of .mpy files
- frozen code size is reduced by about 5%-7%
- importing .py files uses about 5% less RAM in total
- importing .mpy files uses about 4% less RAM in total
- importing .py and .mpy files takes about the same time as before
The qstr indirection in the bytecode has only a small impact on VM
performance. For stm32 on PYBv1.0 the performance change of this commit
is:
diff of scores (higher is better)
N=100 M=100 baseline -> this-commit diff diff% (error%)
bm_chaos.py 371.07 -> 357.39 : -13.68 = -3.687% (+/-0.02%)
bm_fannkuch.py 78.72 -> 77.49 : -1.23 = -1.563% (+/-0.01%)
bm_fft.py 2591.73 -> 2539.28 : -52.45 = -2.024% (+/-0.00%)
bm_float.py 6034.93 -> 5908.30 : -126.63 = -2.098% (+/-0.01%)
bm_hexiom.py 48.96 -> 47.93 : -1.03 = -2.104% (+/-0.00%)
bm_nqueens.py 4510.63 -> 4459.94 : -50.69 = -1.124% (+/-0.00%)
bm_pidigits.py 650.28 -> 644.96 : -5.32 = -0.818% (+/-0.23%)
core_import_mpy_multi.py 564.77 -> 581.49 : +16.72 = +2.960% (+/-0.01%)
core_import_mpy_single.py 68.67 -> 67.16 : -1.51 = -2.199% (+/-0.01%)
core_qstr.py 64.16 -> 64.12 : -0.04 = -0.062% (+/-0.00%)
core_yield_from.py 362.58 -> 354.50 : -8.08 = -2.228% (+/-0.00%)
misc_aes.py 429.69 -> 405.59 : -24.10 = -5.609% (+/-0.01%)
misc_mandel.py 3485.13 -> 3416.51 : -68.62 = -1.969% (+/-0.00%)
misc_pystone.py 2496.53 -> 2405.56 : -90.97 = -3.644% (+/-0.01%)
misc_raytrace.py 381.47 -> 374.01 : -7.46 = -1.956% (+/-0.01%)
viper_call0.py 576.73 -> 572.49 : -4.24 = -0.735% (+/-0.04%)
viper_call1a.py 550.37 -> 546.21 : -4.16 = -0.756% (+/-0.09%)
viper_call1b.py 438.23 -> 435.68 : -2.55 = -0.582% (+/-0.06%)
viper_call1c.py 442.84 -> 440.04 : -2.80 = -0.632% (+/-0.08%)
viper_call2a.py 536.31 -> 532.35 : -3.96 = -0.738% (+/-0.06%)
viper_call2b.py 382.34 -> 377.07 : -5.27 = -1.378% (+/-0.03%)
And for unix on x64:
diff of scores (higher is better)
N=2000 M=2000 baseline -> this-commit diff diff% (error%)
bm_chaos.py 13594.20 -> 13073.84 : -520.36 = -3.828% (+/-5.44%)
bm_fannkuch.py 60.63 -> 59.58 : -1.05 = -1.732% (+/-3.01%)
bm_fft.py 112009.15 -> 111603.32 : -405.83 = -0.362% (+/-4.03%)
bm_float.py 246202.55 -> 247923.81 : +1721.26 = +0.699% (+/-2.79%)
bm_hexiom.py 615.65 -> 617.21 : +1.56 = +0.253% (+/-1.64%)
bm_nqueens.py 215807.95 -> 215600.96 : -206.99 = -0.096% (+/-3.52%)
bm_pidigits.py 8246.74 -> 8422.82 : +176.08 = +2.135% (+/-3.64%)
misc_aes.py 16133.00 -> 16452.74 : +319.74 = +1.982% (+/-1.50%)
misc_mandel.py 128146.69 -> 130796.43 : +2649.74 = +2.068% (+/-3.18%)
misc_pystone.py 83811.49 -> 83124.85 : -686.64 = -0.819% (+/-1.03%)
misc_raytrace.py 21688.02 -> 21385.10 : -302.92 = -1.397% (+/-3.20%)
The code size change is (firmware with a lot of frozen code benefits the
most):
bare-arm: +396 +0.697%
minimal x86: +1595 +0.979% [incl +32(data)]
unix x64: +2408 +0.470% [incl +800(data)]
unix nanbox: +1396 +0.309% [incl -96(data)]
stm32: -1256 -0.318% PYBV10
cc3200: +288 +0.157%
esp8266: -260 -0.037% GENERIC
esp32: -216 -0.014% GENERIC[incl -1072(data)]
nrf: +116 +0.067% pca10040
rp2: -664 -0.135% PICO
samd: +844 +0.607% ADAFRUIT_ITSYBITSY_M4_EXPRESS
As part of this change the .mpy file format version is bumped to version 6.
And mpy-tool.py has been improved to provide a good visualisation of the
contents of .mpy files.
In summary: this commit changes the bytecode to use qstr indirection, and
reworks the .mpy file format to be simpler and allow .mpy files to be
executed in-place. Performance is not impacted too much. Eventually it
will be possible to store such .mpy files in a linear, read-only, memory-
mappable filesystem so they can be executed from flash/ROM. This will
essentially be able to replace frozen code for most applications.
Signed-off-by: Damien George <damien@micropython.org>
This allows the compiler to merge strings: e.g. "update",
"difference_update" and "symmetric_difference_update" will all point to the
same memory.
No functional change.
The size reduction depends on the number of qstrs in the build. The change
this commit brings is:
bare-arm: -4 -0.007%
minimal x86: +150 +0.092% [incl +48(data)]
unix x64: -608 -0.118%
unix nanbox: -572 -0.126% [incl +32(data)]
stm32: -1392 -0.352% PYBV10
cc3200: -448 -0.244%
esp8266: -1208 -0.173% GENERIC
esp32: -1028 -0.068% GENERIC[incl -1020(data)]
nrf: -440 -0.252% pca10040
rp2: -1072 -0.217% PICO
samd: -368 -0.264% ADAFRUIT_ITSYBITSY_M4_EXPRESS
Performance is also improved (on bare metal at least) for the
core_import_mpy_multi.py, core_import_mpy_single.py and core_qstr.py
performance benchmarks.
Originally at adafruit#4583
Signed-off-by: Artyom Skrobov <tyomitch@gmail.com>
The inclusion of `umachine` in the list of built-in modules is now done
centrally in py/objmodule.c. Enabling MICROPY_PY_MACHINE will include this
module.
As part of this, all ports now have `umachine` as the core module name
(previously some had only `machine` as the name).
Signed-off-by: Damien George <damien@micropython.org>
Damien George