micropython/tests
Jim Mussared b326edf68c all: Remove MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE.
This commit removes all parts of code associated with the existing
MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE optimisation option, including the
-mcache-lookup-bc option to mpy-cross.

This feature originally provided a significant performance boost for Unix,
but wasn't able to be enabled for MCU targets (due to frozen bytecode), and
added significant extra complexity to generating and distributing .mpy
files.

The equivalent performance gain is now provided by the combination of
MICROPY_OPT_LOAD_ATTR_FAST_PATH and MICROPY_OPT_MAP_LOOKUP_CACHE (which has
been enabled on the unix port in the previous commit).

It's hard to provide precise performance numbers, but tests have been run
on a wide variety of architectures (x86-64, ARM Cortex, Aarch64, RISC-V,
xtensa) and they all generally agree on the qualitative improvements seen
by the combination of MICROPY_OPT_LOAD_ATTR_FAST_PATH and
MICROPY_OPT_MAP_LOOKUP_CACHE.

For example, on a "quiet" Linux x64 environment (i3-5010U @ 2.10GHz) the
change from CACHE_MAP_LOOKUP_IN_BYTECODE, to LOAD_ATTR_FAST_PATH combined
with MAP_LOOKUP_CACHE is:

diff of scores (higher is better)
N=2000 M=2000       bccache -> attrmapcache      diff      diff% (error%)
bm_chaos.py        13742.56 ->   13905.67 :   +163.11 =  +1.187% (+/-3.75%)
bm_fannkuch.py        60.13 ->      61.34 :     +1.21 =  +2.012% (+/-2.11%)
bm_fft.py         113083.20 ->  114793.68 :  +1710.48 =  +1.513% (+/-1.57%)
bm_float.py       256552.80 ->  243908.29 : -12644.51 =  -4.929% (+/-1.90%)
bm_hexiom.py         521.93 ->     625.41 :   +103.48 = +19.826% (+/-0.40%)
bm_nqueens.py     197544.25 ->  217713.12 : +20168.87 = +10.210% (+/-3.01%)
bm_pidigits.py      8072.98 ->    8198.75 :   +125.77 =  +1.558% (+/-3.22%)
misc_aes.py        17283.45 ->   16480.52 :   -802.93 =  -4.646% (+/-0.82%)
misc_mandel.py     99083.99 ->  128939.84 : +29855.85 = +30.132% (+/-5.88%)
misc_pystone.py    83860.10 ->   82592.56 :  -1267.54 =  -1.511% (+/-2.27%)
misc_raytrace.py   21490.40 ->   22227.23 :   +736.83 =  +3.429% (+/-1.88%)

This shows that the new optimisations are at least as good as the existing
inline-bytecode-caching, and are sometimes much better (because the new
ones apply caching to a wider variety of map lookups).

The new optimisations can also benefit code generated by the native
emitter, because they apply to the runtime rather than the generated code.
The improvement for the native emitter when LOAD_ATTR_FAST_PATH and
MAP_LOOKUP_CACHE are enabled is (same Linux environment as above):

diff of scores (higher is better)
N=2000 M=2000        native -> nat-attrmapcache  diff      diff% (error%)
bm_chaos.py        14130.62 ->   15464.68 :  +1334.06 =  +9.441% (+/-7.11%)
bm_fannkuch.py        74.96 ->      76.16 :     +1.20 =  +1.601% (+/-1.80%)
bm_fft.py         166682.99 ->  168221.86 :  +1538.87 =  +0.923% (+/-4.20%)
bm_float.py       233415.23 ->  265524.90 : +32109.67 = +13.756% (+/-2.57%)
bm_hexiom.py         628.59 ->     734.17 :   +105.58 = +16.796% (+/-1.39%)
bm_nqueens.py     225418.44 ->  232926.45 :  +7508.01 =  +3.331% (+/-3.10%)
bm_pidigits.py      6322.00 ->    6379.52 :    +57.52 =  +0.910% (+/-5.62%)
misc_aes.py        20670.10 ->   27223.18 :  +6553.08 = +31.703% (+/-1.56%)
misc_mandel.py    138221.11 ->  152014.01 : +13792.90 =  +9.979% (+/-2.46%)
misc_pystone.py    85032.14 ->  105681.44 : +20649.30 = +24.284% (+/-2.25%)
misc_raytrace.py   19800.01 ->   23350.73 :  +3550.72 = +17.933% (+/-2.79%)

In summary, compared to MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE, the new
MICROPY_OPT_LOAD_ATTR_FAST_PATH and MICROPY_OPT_MAP_LOOKUP_CACHE options:
- are simpler;
- take less code size;
- are faster (generally);
- work with code generated by the native emitter;
- can be used on embedded targets with a small and constant RAM overhead;
- allow the same .mpy bytecode to run on all targets.

See #7680 for further discussion.  And see also #7653 for a discussion
about simplifying mpy-cross options.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-09-16 16:04:03 +10:00
..
basics tests/basics: Split f-string debug printing to separate file with .exp. 2021-08-26 23:56:02 +10:00
cmdline all: Remove MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE. 2021-09-16 16:04:03 +10:00
cpydiff py: Implement partial PEP-498 (f-string) support. 2021-08-14 16:58:40 +10:00
esp32 tests: Rename run-tests to run-tests.py for consistency. 2021-03-12 19:56:09 +11:00
extmod extmod/modframebuf: Enable blit between different formats via a palette. 2021-08-25 15:31:23 +10:00
feature_check py: Implement partial PEP-498 (f-string) support. 2021-08-14 16:58:40 +10:00
float tests/float: Make bytes/bytearray construct tests work with obj repr C. 2021-06-18 14:16:07 +10:00
import py/builtinimport: Change relative import's ValueError to ImportError. 2021-05-30 19:35:03 +10:00
inlineasm tests: Format all Python code with black, except tests in basics subdir. 2020-03-30 13:21:58 +11:00
internal_bench tests: Format all Python code with black, except tests in basics subdir. 2020-03-30 13:21:58 +11:00
io extmod/vfs_posix_file: Allow closing an already closed file. 2021-02-11 22:54:41 +11:00
jni tests: Rename run-tests to run-tests.py for consistency. 2021-03-12 19:56:09 +11:00
micropython all: Remove MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE. 2021-09-16 16:04:03 +10:00
misc tests/misc/sys_settrace_features.py: Fix running with non-dflt encoding. 2020-12-18 13:57:17 +11:00
multi_bluetooth tests/multi_bluetooth/ble_subscribe.py: Add test for subscription. 2021-08-14 22:44:47 +10:00
multi_net extmod/modlwip: Fix close and clean up of UDP and raw sockets. 2021-08-13 23:46:11 +10:00
net_hosted extmod/uasyncio: Get addr and bind server socket before creating task. 2021-06-26 22:30:22 +10:00
net_inet tests: Use .errno instead of .args[0] for OSError exceptions. 2021-04-23 22:03:46 +10:00
perf_bench tests/perf_bench: Use math.log instead of math.log2. 2021-09-13 18:27:39 +10:00
pyb stm32: Replace master/slave with controller/peripheral in I2C and SPI. 2021-07-18 11:23:41 +10:00
pybnative tests/pybnative: Make while.py test run on boards without pyb.delay. 2021-09-01 00:43:41 +10:00
qemu-arm qemu-arm: Add testing of frozen native modules. 2019-08-20 15:14:08 +10:00
stress tests: Format all Python code with black, except tests in basics subdir. 2020-03-30 13:21:58 +11:00
thread tests/thread: Make exc1,exit1,exit2,stacksize1,start1 tests run on rp2. 2021-05-10 13:07:16 +10:00
unicode tests: Format all Python code with black, except tests in basics subdir. 2020-03-30 13:21:58 +11:00
unix tests/unix: Add ffi test for integer types. 2021-06-06 22:52:25 +10:00
wipy tests: Format all Python code with black, except tests in basics subdir. 2020-03-30 13:21:58 +11:00
README tests: Rename run-tests to run-tests.py for consistency. 2021-03-12 19:56:09 +11:00
run-internalbench.py tests: Rename run-tests to run-tests.py for consistency. 2021-03-12 19:56:09 +11:00
run-multitests.py tests/run-multitests.py: Add broadcast and wait facility. 2021-08-13 23:26:34 +10:00
run-natmodtests.py all: Rename "sys" module to "usys". 2020-09-04 00:10:24 +10:00
run-perfbench.py tests/run-perfbench.py: Fix native feature check. 2021-05-11 23:45:36 +10:00
run-tests-exp.py tests: Rename run-tests to run-tests.py for consistency. 2021-03-12 19:56:09 +11:00
run-tests-exp.sh tests: Rename run-tests to run-tests.py for consistency. 2021-03-12 19:56:09 +11:00
run-tests.py all: Remove MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE. 2021-09-16 16:04:03 +10:00

README

This directory contains tests for various functionality areas of MicroPython.
To run all stable tests, run "run-tests.py" script in this directory.

Tests of capabilities not supported on all platforms should be written
to check for the capability being present. If it is not, the test
should merely output 'SKIP' followed by the line terminator, and call
sys.exit() to raise SystemExit, instead of attempting to test the
missing capability. The testing framework (run-tests.py in this
directory, test_main.c in qemu_arm) recognizes this as a skipped test.

There are a few features for which this mechanism cannot be used to
condition a test. The run-tests.py script uses small scripts in the
feature_check directory to check whether each such feature is present,
and skips the relevant tests if not.

Tests are generally verified by running the test both in MicroPython and
in CPython and comparing the outputs. If the output differs the test fails
and the outputs are saved in a .out and a .exp file respectively.
For tests that cannot be run in CPython, for example because they use
the machine module, a .exp file can be provided next to the test's .py
file. A convenient way to generate that is to run the test, let it fail
(because CPython cannot run it) and then copy the .out file (but not
before checking it manually!)

When creating new tests, anything that relies on float support should go in the
float/ subdirectory.  Anything that relies on import x, where x is not a built-in
module, should go in the import/ subdirectory.