Updates all README.md and docs, and manifests to `require("mip")`.
Also extend and improve the documentation on freezing and packaging.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This uses the frozentest.mpy that is also used by ports/minimal.
Also fixes two bugs that these new tests picked up:
- File extension matching in manifestfile.py.
- Handling of freeze_mpy results in makemanifest.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
All variants (except minimal) enable text compression and fat/lfs, so move
them to the common mpconfigport.mk.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This is a no-op for coverage and minimal.
The standard and dev variants have been merged and enable the same feature
set as a typical bare-metal board. And remove the CI for the dev build.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
The freedos variant is untested by CI and is difficult to maintain. The
fast variant is not a good name for what it does.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Changes in this commit:
- Manifest include's now use the directory path where possible (no longer
necessary to include the manifest.py file explicitly).
- Add manifest.py for all drivers and components that are referenced by
port/board manifests.
- Replace all uses of freeze() with package()/module(), except for port and
board modules.
- Use opt=3 everywhere, for consistency and to reduce code size.
- Use require() instead of include() for all micropython-lib references.
- Remove support for optional board-level manifest.py in mimxrt port, to
make it behave the same as other ports (the board must set
FROZEN_MANIFEST to a custom manifest.py, which can optionally include the
default, port-level manifest).
- Also reinstates modules that were accidentally removed from the esp8266
512k build in fbe9417b90.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
Since commit e65d1e69e8 there is no longer an
io.FileIO class, so this option is no longer needed.
This option also controlled whether or not files supported being opened in
binary mode (eg 'rb'), and could, if disabled, lead to confusion as to why
opening a file in binary mode silently did the wrong thing (it would just
open in text mode if MICROPY_PY_IO_FILEIO was disabled).
The various VFS implementations (POSIX, FAT, LFS) were the only places
where enabling this option made a difference, and in almost all cases where
one of these filesystems were enabled, MICROPY_PY_IO_FILEIO was also
enabled. So it makes sense to just unconditionally enable this feature
(ability to open a file in binary mode) in all cases, and so just remove
this config option altogether. That makes configuration simpler and means
binary file support always exists (and opening a file in binary mode is
arguably more fundamental than opening in text mode, so if anything should
be configurable then it should be the ability to open in text mode).
Signed-off-by: Damien George <damien@micropython.org>
The executable now lives in the build directory, and since the build
directory already contains the variant name there is no need to also add
it to the executable.
Signed-off-by: Damien George <damien@micropython.org>
With a new option to evenly split the GC heap over multiple areas. This
adds code coverage for gc_add() and code associated with
MICROPY_GC_SPLIT_HEAP.
All in-tree uses of MICROPY_PORT_ROOT_POINTERS have been replaced with
MP_REGISTER_ROOT_POINTER(), so now we can remove both
MICROPY_PORT_ROOT_POINTERS and MICROPY_BOARD_ROOT_POINTERS from the code
and remaining config files.
Signed-off-by: David Lechner <david@pybricks.com>
This ROM level is not yet fully defined, but it at least enables
MICROPY_PY_SYS_TRACEBACKLIMIT. The coverage build should have everything
enabled, so it makes sense to use this ROM level for it.
Signed-off-by: Damien George <damien@micropython.org>
So that the default configuration for the dev and coverage variants
includes all options set by MICROPY_CONFIG_ROM_LEVEL_EXTRA_FEATURES.
Note that enabling MICROPY_PY_SYS_STDIO_BUFFER on unix doesn't do anything
because unix doesn't use shared/runtime/sys_stdio_mphal.c.
Signed-off-by: Damien George <damien@micropython.org>
The default is the same as before: MICROPY_PY_USELECT=0 and
MICROPY_PY_USELECT_POSIX=1. But now this can be easily overridden at the
make command-line using, eg:
make VARIANT=dev CFLAGS_EXTRA=-DMICROPY_PY_USELECT=1
Signed-off-by: Damien George <damien@micropython.org>
For ports with MICROPY_VFS and MICROPY_PY_IO enabled their configuration
can now be simplified to use the defaults for mp_import_stat and
mp_builtin_open.
This commit makes no functional change, except for the following minor
points:
- the built-in "open" is removed from the minimal port (it previously did
nothing)
- the duplicate built-in "input" is removed from the esp32 port
- qemu-arm now delegates to VFS import/open
Signed-off-by: Damien George <damien@micropython.org>
These files that are reformatted only now fall under the list of files to
apply uncrustify/black formatting to.
Signed-off-by: Damien George <damien@micropython.org>
All variants now use extmod/moduos.c as their uos module implementation.
In particular this means they all have MICROPY_VFS enabled and use VfsPosix
for their filesystem.
As part of this, the available functions in uos become more consistent with
other ports:
- coverage variant gets uos.urandom
- minimal and standard variant get: unlink, chdir, getcwd, listdir
Signed-off-by: Damien George <damien@micropython.org>
In commit 86ce442607 the '.frozen' entry was
added at the start of sys.path, to allow control over when frozen modules
are searched during import, and retain existing behaviour whereby frozen
was searched before the filesystem.
But Python semantics of sys.path require sys.path[0] to be the directory of
the currently executing script, or ''.
This commit moves the '.frozen' entry to second place in sys.path, so
sys.path[0] retains its correct value (described above).
Signed-off-by: Damien George <damien@micropython.org>
Frozen modules will be searched preferentially, but gives the user the
ability to override this behavior.
This matches the previous behavior where "" was implicitly the frozen
search path, but the frozen list was checked before the filesystem.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This feature is not enabled on any port, it's not in CPython's io module,
and functionality is better suited to the micropython-lib implementation of
pkg_resources.
This commit removes all parts of code associated with the existing
MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE optimisation option, including the
-mcache-lookup-bc option to mpy-cross.
This feature originally provided a significant performance boost for Unix,
but wasn't able to be enabled for MCU targets (due to frozen bytecode), and
added significant extra complexity to generating and distributing .mpy
files.
The equivalent performance gain is now provided by the combination of
MICROPY_OPT_LOAD_ATTR_FAST_PATH and MICROPY_OPT_MAP_LOOKUP_CACHE (which has
been enabled on the unix port in the previous commit).
It's hard to provide precise performance numbers, but tests have been run
on a wide variety of architectures (x86-64, ARM Cortex, Aarch64, RISC-V,
xtensa) and they all generally agree on the qualitative improvements seen
by the combination of MICROPY_OPT_LOAD_ATTR_FAST_PATH and
MICROPY_OPT_MAP_LOOKUP_CACHE.
For example, on a "quiet" Linux x64 environment (i3-5010U @ 2.10GHz) the
change from CACHE_MAP_LOOKUP_IN_BYTECODE, to LOAD_ATTR_FAST_PATH combined
with MAP_LOOKUP_CACHE is:
diff of scores (higher is better)
N=2000 M=2000 bccache -> attrmapcache diff diff% (error%)
bm_chaos.py 13742.56 -> 13905.67 : +163.11 = +1.187% (+/-3.75%)
bm_fannkuch.py 60.13 -> 61.34 : +1.21 = +2.012% (+/-2.11%)
bm_fft.py 113083.20 -> 114793.68 : +1710.48 = +1.513% (+/-1.57%)
bm_float.py 256552.80 -> 243908.29 : -12644.51 = -4.929% (+/-1.90%)
bm_hexiom.py 521.93 -> 625.41 : +103.48 = +19.826% (+/-0.40%)
bm_nqueens.py 197544.25 -> 217713.12 : +20168.87 = +10.210% (+/-3.01%)
bm_pidigits.py 8072.98 -> 8198.75 : +125.77 = +1.558% (+/-3.22%)
misc_aes.py 17283.45 -> 16480.52 : -802.93 = -4.646% (+/-0.82%)
misc_mandel.py 99083.99 -> 128939.84 : +29855.85 = +30.132% (+/-5.88%)
misc_pystone.py 83860.10 -> 82592.56 : -1267.54 = -1.511% (+/-2.27%)
misc_raytrace.py 21490.40 -> 22227.23 : +736.83 = +3.429% (+/-1.88%)
This shows that the new optimisations are at least as good as the existing
inline-bytecode-caching, and are sometimes much better (because the new
ones apply caching to a wider variety of map lookups).
The new optimisations can also benefit code generated by the native
emitter, because they apply to the runtime rather than the generated code.
The improvement for the native emitter when LOAD_ATTR_FAST_PATH and
MAP_LOOKUP_CACHE are enabled is (same Linux environment as above):
diff of scores (higher is better)
N=2000 M=2000 native -> nat-attrmapcache diff diff% (error%)
bm_chaos.py 14130.62 -> 15464.68 : +1334.06 = +9.441% (+/-7.11%)
bm_fannkuch.py 74.96 -> 76.16 : +1.20 = +1.601% (+/-1.80%)
bm_fft.py 166682.99 -> 168221.86 : +1538.87 = +0.923% (+/-4.20%)
bm_float.py 233415.23 -> 265524.90 : +32109.67 = +13.756% (+/-2.57%)
bm_hexiom.py 628.59 -> 734.17 : +105.58 = +16.796% (+/-1.39%)
bm_nqueens.py 225418.44 -> 232926.45 : +7508.01 = +3.331% (+/-3.10%)
bm_pidigits.py 6322.00 -> 6379.52 : +57.52 = +0.910% (+/-5.62%)
misc_aes.py 20670.10 -> 27223.18 : +6553.08 = +31.703% (+/-1.56%)
misc_mandel.py 138221.11 -> 152014.01 : +13792.90 = +9.979% (+/-2.46%)
misc_pystone.py 85032.14 -> 105681.44 : +20649.30 = +24.284% (+/-2.25%)
misc_raytrace.py 19800.01 -> 23350.73 : +3550.72 = +17.933% (+/-2.79%)
In summary, compared to MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE, the new
MICROPY_OPT_LOAD_ATTR_FAST_PATH and MICROPY_OPT_MAP_LOOKUP_CACHE options:
- are simpler;
- take less code size;
- are faster (generally);
- work with code generated by the native emitter;
- can be used on embedded targets with a small and constant RAM overhead;
- allow the same .mpy bytecode to run on all targets.
See #7680 for further discussion. And see also #7653 for a discussion
about simplifying mpy-cross options.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
It's a bit of a pitfall with user C modules that including them in the
build does not automatically enable them. This commit changes the docs and
examples for user C modules to encourage writers of user C modules to
enable them unconditionally. This makes things simpler and covers most use
cases.
See discussion in issue #6960, and also #7086.
Signed-off-by: Damien George <damien@micropython.org>
Add working example code to provide a starting point for users with files
that they can just copy, and include the modules in the coverage test to
verify the complete user C module build functionality. The cexample module
uses the code originally found in cmodules.rst, which has been updated to
reflect this and partially rewritten with more complete information.
And enable this feature on unix, the coverage variant. The .exp test file
is needed so the test can run on CPython versions prior to "@=" operator
support.
Signed-off-by: Damien George <damien@micropython.org>
This commit adds support for using Bluetooth on the unix port via a H4
serial interface (distinct from a USB dongle), with both BTstack and NimBLE
Bluetooth stacks.
Note that MICROPY_PY_BLUETOOTH is now disabled for the coverage variant.
Prior to this commit Bluetooth was anyway not being built on Travis because
libusb was not detected. But now that bluetooth works in H4 mode it will
be built, and will lead to a large decrease in coverage because Bluetooth
tests cannot be run on Travis.
So that micropython-dev can be used to test VFS code, and inspect and build
filesystem images that are compatible with bare-metal systems.
Signed-off-by: Damien George <damien@micropython.org>
Changes are:
- The default manifest.py is moved to the variants directory (it's in
"boards" in other ports).
- The coverage variant now uses a custom manifest in its variant directory
to add frzmpy/frzstr.
- The frzmpy/frzstr tests are moved to variants/coverage/.
No functionality change is intended with this commit, it just consolidates
the separate implementations of GC helper code to the lib/utils/ directory
as a general set of helper functions useful for any port. This reduces
duplication of code, and makes it easier for future ports or embedders to
get the GC implementation correct.
Ports should now link against gchelper_native.c and either gchelper_m0.s or
gchelper_m3.s (currently only Cortex-M is supported but other architectures
can follow), or use the fallback gchelper_generic.c which will work on
x86/x64/ARM.
The gc_helper_get_sp function from gchelper_m3.s is not really GC related
and was only used by cc3200, so it has been moved to that port and renamed
to cortex_m3_get_sp.
Add -Wdouble-promotion and -Wfloat-conversion for most ports to ban out
implicit floating point conversions, and add extra Travis builds using
MICROPY_FLOAT_IMPL_FLOAT to uncover warnings which weren't found
previously. For the unix port -Wsign-comparison is added as well but only
there since only clang supports this but gcc doesn't.
Based on eg 1e6fd9f2b4, it's understood that
the intention for unix builds is that regular builds disable assert, but
the coverage build should set -O0 and enable asserts.
It looks like this didn't work (even before variants were introduced, eg at
v1.11) -- coverage always built with -Os and -DNDEBUG.
This commit makes it possible for variants to have finer-grained control
over COPT flags, and enables assert() and -O0 on coverage builds.
Other variants already match the defaults so they have been updated.
Jim Mussared