Changes in this commit:
- Change MICROPY_HW_BOARD_NAME definition to match the product name.
- Rename board folder's name to match the product name style.
- Change related files like Makefile, document descriptions, test cases, CI
and tools.
Signed-off-by: Takeo Takahashi <takeo.takahashi.xv@renesas.com>
- Use HCI_TRACE macro consistently.
- Use the same colour formatting.
- Add a tool to convert to .pcap for Wireshark.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
The mpremote REPL can now be closed with either ctrl+] or ctrl+x, which
gives users a choice, useful if the ']' key is difficult to access.
Fixes issue #11197.
Signed-off-by: Jonas Scharpf <jonas@brainelectronics.de>
This adds a new MODE_PYPROJECT, which gives basic support to allow
packaging a small subset of micropython-lib packages to PyPI.
This change allows a package in micropython-lib to:
- Add a "pypi" name to its metadata indicating that it's based on a PyPI
package.
- Add "stdlib" to its metadata indicating that it's a micropython version
of a stdlib package.
- Add a "pypi_publish" name to its metadata to indicate that it can be
published to PyPI (this can be different to the package name, e.g. "foo"
might want to be published as "micropython-foo").
When a package requires() another one, if it's in MODE_PYPROJECT then if
the package is from pypi then it will record that as a pypi dependency
instead (or no dependency at all if it's from stdlib).
Also allows require() to explicitly specify the pypi name.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This is a follow up to d263438a6e, which
solved one problem (reset on disconnect) but introduced a second one (hang
in bootloader).
To solve both probles, False/False is needed for DTR/RTS for ESPxx, but
that would then block stm32 and others. Any unconditional combination of
DTR/RTS ends up blocking normal operation on some type of board or another.
A simple overview (for windows only):
DTR CTS ESP8266/ESP32 STM32/SAMD51/RP2040
unspecified unspecified Reset on disconnect OK
True False Hang in bootloader OK
False False OK No Repl
True True Reset on disconnect No Repl
False True Reset on disconnect No Repl
serial.manufacturer: wch.cn/Silicon Labs Microsoft
serial.description: USB-SERIAL CH340 / USB Serial Device
CP210x USB to UART
Bridge
The updated logic will only set the DTR/RTS signals for boards that do not
use standard Microsoft drivers (based on the manufacturer). It would also
be possible to check against a list of known driver manufactures (like
wch.cn or Silicon Labs) but this would require a list of known drivers for
all ports.
Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
Currently, certain mpremote filesystem operations can fail on Windows due
to a mixing of '/' and '\' for path separators. Eg if filesystem_command()
is called with a destination that ends in / then dest.endswith(os.path.sep)
will return False, which gives the wrong behaviour (it does end in a path
separator).
For similar reasons to 7e9a15966a, it's best
to use '/' everywhere in pyboard.py and mpremote, because the target device
understands only '/'. mpremote already does this, so the remaining place
to fix it is in pyboard.y, to convert all incoming paths to use '/' instead
of '\'.
This effectively reverts 57fd66b80f which
tried to fix the problem in a different way.
See also related 1f84440538.
Signed-off-by: Damien George <damien@micropython.org>
This allows the entire configuration to be defined in a single file,
including the logic for including pyboard.py and automatically versioning
based on the git tag.
Building the package works both via `python -m build` as well as
`hatch build`. `python -m build ` has the advantage of automatically
fetching all dependencies, you don't need to manually install any hatch
packages.
In order to make the versioning work, and also keep things simpler for end
users, mpremote releases will now be the same as MicroPython releases and
use the same tag. The version strings for mpremote will look like:
- X.Y.Z -- clean build at the tag
- X.Y.Z.postN+gHASH -- clean build, N revisions from the most recent tag
- X.Y.Z.postN+gHASH.dYYYYMMDD -- dirty build, N revisions from out
This commit extends on the idea from #8404.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Now that the code-size-check CI action gives a nice report (as a comment)
on the code size difference, it's possible to have a few more ports
reported there. In this commit, unix, stm32 and rp2 are added. Unix
represents non-MCU builds, and stm32 and rp2 represent ARM-based builds,
for ports that have lots of features enabled.
Signed-off-by: Damien George <damien@micropython.org>
These are for working with the filesystem when using pyboard.py as a
library, rather than at the command line.
- fs_listdir returns a list of tuples, in the same format as os.ilistdir().
- fs_readfile returns the contents of a file as a bytes object.
- fs_writefile allows writing a bytes object to a file.
- fs_stat returns an os.statresult.
All raise FileNotFoundError (or OSError(ENOENT) on Python 2) if the file is
not found (or PyboardError on other errors).
Updated fs_cp and fs_get to use fs_stat to compute file size.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This is useful when using pyboard.py as a library rather than at the
command line.
pyb.eval("1+1") --> b"2"
pyb.eval("{'a': '\x00'}") --> b"{'a': '\\x00'}"
Now you can also do
pyb.eval("1+1", parse=True) --> 2
pyb.eval("{'a': '\x00'}", parse=True) --> {'a': '\x00'}
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
On MacOS and Windows there are a few default serial devices that are
returned by `serial.tools.list_ports.comports()`. For example on MacOS:
```
{'description': 'n/a',
'device': '/dev/cu.Bluetooth-Incoming-Port',
'hwid': 'n/a',
'interface': None,
'location': None,
'manufacturer': None,
'name': 'cu.Bluetooth-Incoming-Port',
'pid': None,
'product': None,
'serial_number': None,
'vid': None}
{'description': 'n/a',
'device': '/dev/cu.wlan-debug',
'hwid': 'n/a',
'interface': None,
'location': None,
'manufacturer': None,
'name': 'cu.wlan-debug',
'pid': None,
'product': None,
'serial_number': None,
'vid': None}
```
Users of mpremote most likely do not want to connect to these ports. It
would be desirable if mpremote did not select this ports when using the
auto connect behavior. These serial ports do not have USB VID or PID
values and serial ports for Micropython boards with FTDI/serial-to-USB
adapter or native USB CDC/ACM support do.
Check for the presence of a USB VID / PID int value when selecting a
serial port to auto connect to. All serial ports will still be listed by
the `list` command and can still be selected by name when connecting.
Signed-off-by: Michael Mogenson <michael.mogenson@gmail.com>
The zephyr CI takes the most time out of all CI jobs, so remove the
standard qemu_x86 build to speed it up. The remaining builds should still
cover enough cases to catch errors.
Signed-off-by: Damien George <damien@micropython.org>
This used to be used to generate .rst docs from inline comments in the C
code (specifically for APIs) but is now unused.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
App the mp_ prefix to usbd_ symbols and files which are defined here and
not in TinyUSB.
rp2 only for now. This includes some groundwork for dynamic USB devices
(defined in Python).
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
So that filesystems mounted with "mpremote mount" can have their files
iterated over, making them consistent with other files.
Signed-off-by: Damien George <damien@micropython.org>
The except handler for OSError didn't include the line that actually calls
os.listdir, so an invalid path wasn't handled correctly.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
In order for v1.19.1 to load a .mpy, the formerly-feature-flags which are
now used for the sub-version must be zero.
The sub-version is only used to indicate a native version change, so it
should be zero when emitting bytecode-only .mpy files.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This applies to nimble, btstack, axtls, mbedtls, lwip.
Rather than having the ports individually manage GIT_SUBMODULES for these
components, make extmod.mk append them when the relevant feature is
enabled.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This check used to just show the regular expression that failed to match,
but the rules are pretty subtle and hard to interpret from the regular
expression alone.
Add some basic checks for the main things that go wrong:
- Missing capitalisation.
- Missing full-stop.
- Missing path.
- Single-word subject.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
The "signed-off" check assumes that the Signed-off-by: line is the last,
but there may me many lines of comments after this.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Fixes in this commit are:
- Make --follow the default for "run" (accidentally changed in 68d094358).
- Add help strings for "repl": --capture --inject-file --inject-code
- Update help strings for "run".
- Fix encoding for --inject-code (accidentally broken in 68d094358).
- Remove ability to --no-follow for "eval". It was there previously because
it shared the same code path with "exec" and "run", but makes no sense
for "eval", so might as well remove.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Tweak the existing codeformat.py and verifygitlog.py to allow them to be
easily called by pre-commit.
(This turned out to be easier than using any existing pre-commit hooks,
without making subtle changes in the formatting.)
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
Uncrustify versions are not mutually compatible:
1. Version 0.73 or newer produce slightly different formatting. It may be
possible to tweak these by adding more config items, but this will cause
older versions to error out with 'Unknown option'.
2. Version 0.75 prints a range of deprecation warnings due to config file
changes, and returns a non-zero exit code. These are actually fixable
as most are the default value, and pp_indent has changed from 'true' to '1'
which is backwards compatible. However issue 1 remains, so probably better
to have it fail explicitly.
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
This supports the same package sources as the new `mip` tool.
- micropython-lib (by name)
- http(s) & github packages with json description
- directly downloading a .py/.mpy file
The version is specified with an optional `@version` on the end of the
package name. The target dir, index, and mpy/no-mpy can be set through
command line args.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
No functional change other than to allow slightly more flexibility in how
--foo arguments are specified.
This removes all custom handling for --foo args in all commands and
replaces it with per-command argparse configs.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
No functional change.
This makes each built-in command defined by just a handler method and
simplifies a lot of the logic around tracking the board state.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Updates all README.md and docs, and manifests to `require("mip")`.
Also extend and improve the documentation on freezing and packaging.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This uses the frozentest.mpy that is also used by ports/minimal.
Also fixes two bugs that these new tests picked up:
- File extension matching in manifestfile.py.
- Handling of freeze_mpy results in makemanifest.
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
frozentest.mpy was previously duplicated in ports/minimal and
ports/powerpc.
This needs to be re-generated on every .mpy version increase, so might as
well just have a single copy of it.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
The intent is to allow us to make breaking changes to the native ABI (e.g.
changes to dynruntime.h) without needing the bytecode version to increment.
With this commit the two bits previously used for the feature flags (but
now unused as of .mpy version 6) encode a sub-version. A bytecode-only
.mpy file can be loaded as long as MPY_VERSION matches, but a native .mpy
(i.e. one with an arch set) must also match MPY_SUB_VERSION. This allows 3
additional updates to the native ABI per bytecode revision.
The sub-version is set to 1 because the previous commits that changed the
layout of mp_obj_type_t have changed the native ABI.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
This is a no-op for coverage and minimal.
The standard and dev variants have been merged and enable the same feature
set as a typical bare-metal board. And remove the CI for the dev build.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
When iterating over filesystem/folders with os.iterdir(), an open file
(directory) handle is used internally. Currently this file handle is only
closed once the iterator is completely drained, eg. once all entries have
been looped over / converted into list etc.
If a program opens an iterdir but does not loop over it, or starts to loop
over the iterator but breaks out of the loop, then the handle never gets
closed. In this state, when the iter object is cleaned up by the garbage
collector this open handle can cause corruption of the filesystem.
Fixes issues #6568 and #8506.
In case the version from pypi is installed or some other version is
available in sys.path, prepend `$(TOP)/mpy-cross` to sys.path instead.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Previous the build directory just used the board name, now make it use the
variant name too.
This shouldn't have any change because the existing directory should not
exist (all builds run by these scripts remove their build directory after
completion), but it makes debugging easier.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
The metadata can be version, description, and license.
After executing a manifest, the top-level metadata can be queried, and also
each file output from the manifest will have the metadata of the
containing manifest.
Use the version metadata to "tag" files before freezing such that they have
__version__ available.
By default, don't include micropython-lib/unix-ffi in the search.
If unix_ffi=True is passed to require(), then include unix-ffi and make it
take precedence over the other locations (e.g. python-stdlib).
This does two things:
- Prevents non-unix builds from using unix-only packages.
- Allows the unix build to optionally use a more full-featured (e.g. ffi)
based package, even with the same name as one from e.g. stdlib.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
If an include path is a directory, then it implicitly grabs the manifest.py
file inside that directory. This simplifies most manifest.py files.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
This splits the manifest file loading logic from makemanifest.py and
updates makemanifest.py to use it.
This will allow non-freezing uses of manifests, such as defining packages
and dependencies in micropython-lib.
Also adds additional methods to the manifest "API":
- require() - to get a package from micropython-lib.
- module() - to define a single-file module
- package() - to define a multi-file package
module() and package() should replace most uses of freeze() and can also
be also used in non-freezing scenarios.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Having two separate manifests is confusing. It's simpler to have the daily
builds use the same configuration as the stable, release builds.
Signed-off-by: Damien George <damien@micropython.org>
It has been about 8 years since support for this chip was added. Reasons
to remove it are:
- It is no longer easy to obtain this part.
- There are now many other options for WiFi.
- It's not a good use of developer time to maintain it.
Signed-off-by: Damien George <damien@micropython.org>
Rather than having the autobuild know about the particular variants, have
the mpconfigboard.mk describe them and make autobuild discover them
automatically.
Adds a "query-variants" target to stm32/Makefile to allow the set of
possible variants to be queried.
Removes pybv3 from the autobuild as this isn't use by the downloads page.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
- Add lib/wiznet5k into the 'make submodules' step.
- Split the stm32 builds for wiznet5k and cc3k.
- Run 'make .... clean' after making the wiznet5k build.
This allows a remote file to be edited locally by copying it over, running
the local editor, then copying it back.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
The executable now lives in the build directory, and since the build
directory already contains the variant name there is no need to also add
it to the executable.
Signed-off-by: Damien George <damien@micropython.org>
Binaries built using the Make build system now no longer appear in the
working directory of the build, but rather in the build directory. Thus
some paths had to be adjusted.
Formerly, py/formatfloat would print whole numbers inaccurately with
nonzero digits beyond the decimal place. This resulted from its strategy
of successive scaling of the argument by 0.1 which cannot be exactly
represented in floating point. The change in this commit avoids scaling
until the value is smaller than 1, so all whole numbers print with zero
fractional part.
Fixes issue #4212.
Signed-off-by: Dan Ellis dan.ellis@gmail.com
The CI scripts were using a PPA to get a backported version of uncrustify
on Ubuntu 20.04. However, this causes CI to intermittently fail due to
connection issues to launchpad.net or the key server.
Ubuntu 22.04 has a newer version of uncrustify removing the need for the
PPA. Ubuntu 22.04 is now in beta on GitHub actions, so it can be used.
Signed-off-by: David Lechner <david@pybricks.com>
Updates the Zephyr port build instructions and CI to use the latest Zephyr
release tag.
Tested on frdm_k64f.
Signed-off-by: Maureen Helm <maureen.helm@intel.com>
This tests the build when -O2 is used, which can lead to additional
compiler analysis and warnings.
Signed-off-by: Damien George <damien@micropython.org>
Frozen identifiers now include their full name hierarchy, eg their class
name. This makes it easier to understand the generated code.
Signed-off-by: Damien George <damien@micropython.org>
Now that the native qstr link table is gone, merging a native .mpy file
with a bytecode .mpy file is not as simple as concatenating the .mpy data.
The qstr_table and obj_table tables from all merged .mpy files must now be
joined together, because they are global to the .mpy file (and hence global
to the merged .mpy file). This means the bytecode needs to be be decoded,
qstr_table and obj_table indices updated to point to the correct entries in
the new tables, and then the bytecode re-encoded.
This commit makes this change to the merging feature in mpy-tool.py. This
can now merge an arbitrary number of bytecode .mpy files, and up to one
native .mpy file.
Signed-off-by: Damien George <damien@micropython.org>
This was made redundant by f2040bfc7e, which
also did not update this function for the change to qstr-opcode encoding,
so it does not work correctly anyway.
Signed-off-by: Damien George <damien@micropython.org>
Support for architecture-specific qstr linking was removed in
d4d53e9e11, where native code was changed to
access qstr values via qstr_table. The only remaining use for the special
qstr link table in persistentcode.c is to support native module written in
C, linked via mpy_ld.py. But native modules can also use the standard
module-level qstr_table (and obj_table) which was introduced in the .mpy
file reworking in f2040bfc7e.
This commit removes the remaining native qstr liking support in
persistentcode.c's load_raw_code function, and adds two new relocation
options for constants.qstr_table and constants.obj_table. mpy_ld.py is
updated to use these relocations options instead of the native qstr link
table.
Signed-off-by: Damien George <damien@micropython.org>
This enables the new `-X realtime` runtime option when running tests on
macOS. This causes MicroPython to configure all threads to be high
priority so that they are allowed to use high precision timers. This
makes tests that depend on the passage of time more likely to succeed.
CI tests that were disabled because of this are now enabled again.
Signed-off-by: David Lechner <david@pybricks.com>
The examples/natmod features0 and features1 examples now build and run on
ARMv6-M platforms. More complicated examples are not yet supported because
the compiler emits references to built-in functions like __aeabi_uidiv.
Signed-off-by: Damien George <damien@micropython.org>
Some architectures (like esp32 xtensa) cannot read byte-wise from
executable memory. This means the prelude for native functions -- which is
usually located after the machine code for the native function -- must be
placed in separate memory that can be read byte-wise. Prior to this commit
this was achieved by enabling N_PRELUDE_AS_BYTES_OBJ for the emitter and
MICROPY_EMIT_NATIVE_PRELUDE_AS_BYTES_OBJ for the runtime. The prelude was
then placed in a bytes object, pointed to by the module's constant table.
This behaviour is changed by this commit so that a pointer to the prelude
is stored either in mp_obj_fun_bc_t.child_table, or in
mp_obj_fun_bc_t.child_table[num_children] if num_children > 0. The reasons
for doing this are:
1. It decouples the native emitter from runtime requirements, the emitted
code no longer needs to know if the system it runs on can/can't read
byte-wise from executable memory.
2. It makes all ports have the same emitter behaviour, there is no longer
the N_PRELUDE_AS_BYTES_OBJ option.
3. The module's constant table is now used only for actual constants in the
Python code. This allows further optimisations to be done with the
constants (eg constant deduplication).
Code size change for those ports that enable the native emitter:
unix x64: +80 +0.015%
stm32: +24 +0.004% PYBV10
esp8266: +88 +0.013% GENERIC
esp32: -20 -0.002% GENERIC[incl -112(data)]
rp2: +32 +0.005% PICO
Signed-off-by: Damien George <damien@micropython.org>
Prior to this commit, even with unicode disabled .py and .mpy files could
contain unicode characters, eg by entering them directly in a string as
utf-8 encoded.
The only thing the compiler disallowed (with unicode disabled) was using
\uxxxx and \Uxxxxxxxx notation to specify a character within a string with
value >= 0x100; that would give a SyntaxError.
With this change mpy-cross will now accept \u and \U notation to insert a
character with value >= 0x100 into a string (because the -mno-unicode
option is now gone, there's no way to forbid this). The runtime will
happily work with strings with such characters, just like it already works
with strings with characters that were utf-8 encoded directly.
This change simplifies things because there are no longer any feature
flags in .mpy files, and any bytecode .mpy will now run on any target.
Signed-off-by: Damien George <damien@micropython.org>
This will add a space after a comma if it doesn't have one, but will allow
more than one space if the spaces are already there.
Signed-off-by: Damien George <damien@micropython.org>
This adds a rule to cover all ports/**/*.[ch] file to the code formatting
list. Explicit exclusions are also added for code in ports/ which is third
party, or which requires a lot of reformatting.
Signed-off-by: Damien George <damien@micropython.org>
To keep the separate parts of the code that use these values in sync. And
make it easier to add new object types.
Signed-off-by: Damien George <damien@micropython.org>
Specifying the option `--unsafe-links` (or `-l`) to `mpremote mount` will
allow symlinks to be followed in the local directory that point outside of
the base directory path.
For the unsafe case the `path_check()` method of `PyboardCommand` still
checks for a common path but without expanding symlinks. While this check
is currently redundant, it makes the purpose of the method clearer for
possible future uses or extensions.
With the existing code problems can occur with remounting, the "if t -
t_last_activity > QUIET_TIMEOUT:" check can be triggered early before the
REPL string comes through, meaning that the remount doesn't happen.
On certain boards the "MPY: soft reboot" line comes through immediately
(getting the routine past initial timeout) but then there's a slightly
longer delay while the board restarts before it prints out the startup
header and the REPL prompt.
This commit adds some extra pattern monitoring during the timeout loop to
track the state if a soft restart is actually started.
These jumps are always forwards, and it's more efficient in the VM to
decode an unsigned argument. These opcodes are already optimised versions
of the sequence "dup-top pop-jump-if-x pop" so it doesn't hurt generality
to optimise them further.
Signed-off-by: Damien George <damien@micropython.org>
This commit introduces changes:
- All jump opcodes are changed to have variable length arguments, of either
1 or 2 bytes (previously they were fixed at 2 bytes). In most cases only
1 byte is needed to encode the short jump offset, saving bytecode size.
- The bytecode emitter now selects 1 byte jump arguments when the jump
offset is guaranteed to fit in 1 byte. This is achieved by checking if
the code size changed during the last pass and, if it did (if it shrank),
then requesting that the compiler make another pass to get the correct
offsets of the now-smaller code. This can continue multiple times until
the code stabilises. The code can only ever shrink so this iteration is
guaranteed to complete. In most cases no extra passes are needed, the
original 4 passes are enough to get it right by the 4th pass (because the
2nd pass computes roughly the correct labels and the 3rd pass computes
the correct size for the jump argument).
This change to the jump opcode encoding reduces .mpy files and RAM usage
(when bytecode is in RAM) by about 2% on average.
The performance of the VM is not impacted, at least within measurment of
the performance benchmark suite.
Code size is reduced for builds that include a decent amount of frozen
bytecode. ARM Cortex-M builds without any frozen code increase by about
350 bytes.
Signed-off-by: Damien George <damien@micropython.org>
Since cpydiff is code used as documentation, there are cases where we may
want to use Black's `fmt: on/off/skip` comments to avoid automatic
formatting. However, we don't want these comments to be distracting in the
generated documentation.
This rewrites the code to omit these comments when generating the docs.
Signed-off-by: David Lechner <david@pybricks.com>
Tested on PYBV10 and PYBD_SF6, with MBOOT_FSLOAD enabled and programming
new firmware from a .dfu.gz file stored on the SD card.
Signed-off-by: Damien George <damien@micropython.org>
That caused the compile of frozen_content.c to fail if characters like
backslash were in a short string. Thanks to @hippy for identifying the
spot to change.
This makes the auto soft-reset behaviour of mpremote more logical, and now
configurable via these new commands.
Signed-off-by: Damien George <damien@micropython.org>
Background: .mpy files are precompiled .py files, built using mpy-cross,
that contain compiled bytecode functions (and can also contain machine
code). The benefit of using an .mpy file over a .py file is that they are
faster to import and take less memory when importing. They are also
smaller on disk.
But the real benefit of .mpy files comes when they are frozen into the
firmware. This is done by loading the .mpy file during compilation of the
firmware and turning it into a set of big C data structures (the job of
mpy-tool.py), which are then compiled and downloaded into the ROM of a
device. These C data structures can be executed in-place, ie directly from
ROM. This makes importing even faster because there is very little to do,
and also means such frozen modules take up much less RAM (because their
bytecode stays in ROM).
The downside of frozen code is that it requires recompiling and reflashing
the entire firmware. This can be a big barrier to entry, slows down
development time, and makes it harder to do OTA updates of frozen code
(because the whole firmware must be updated).
This commit attempts to solve this problem by providing a solution that
sits between loading .mpy files into RAM and freezing them into the
firmware. The .mpy file format has been reworked so that it consists of
data and bytecode which is mostly static and ready to run in-place. If
these new .mpy files are located in flash/ROM which is memory addressable,
the .mpy file can be executed (mostly) in-place.
With this approach there is still a small amount of unpacking and linking
of the .mpy file that needs to be done when it's imported, but it's still
much better than loading an .mpy from disk into RAM (although not as good
as freezing .mpy files into the firmware).
The main trick to make static .mpy files is to adjust the bytecode so any
qstrs that it references now go through a lookup table to convert from
local qstr number in the module to global qstr number in the firmware.
That means the bytecode does not need linking/rewriting of qstrs when it's
loaded. Instead only a small qstr table needs to be built (and put in RAM)
at import time. This means the bytecode itself is static/constant and can
be used directly if it's in addressable memory. Also the qstr string data
in the .mpy file, and some constant object data, can be used directly.
Note that the qstr table is global to the module (ie not per function).
In more detail, in the VM what used to be (schematically):
qst = DECODE_QSTR_VALUE;
is now (schematically):
idx = DECODE_QSTR_INDEX;
qst = qstr_table[idx];
That allows the bytecode to be fixed at compile time and not need
relinking/rewriting of the qstr values. Only qstr_table needs to be linked
when the .mpy is loaded.
Incidentally, this helps to reduce the size of bytecode because what used
to be 2-byte qstr values in the bytecode are now (mostly) 1-byte indices.
If the module uses the same qstr more than two times then the bytecode is
smaller than before.
The following changes are measured for this commit compared to the
previous (the baseline):
- average 7%-9% reduction in size of .mpy files
- frozen code size is reduced by about 5%-7%
- importing .py files uses about 5% less RAM in total
- importing .mpy files uses about 4% less RAM in total
- importing .py and .mpy files takes about the same time as before
The qstr indirection in the bytecode has only a small impact on VM
performance. For stm32 on PYBv1.0 the performance change of this commit
is:
diff of scores (higher is better)
N=100 M=100 baseline -> this-commit diff diff% (error%)
bm_chaos.py 371.07 -> 357.39 : -13.68 = -3.687% (+/-0.02%)
bm_fannkuch.py 78.72 -> 77.49 : -1.23 = -1.563% (+/-0.01%)
bm_fft.py 2591.73 -> 2539.28 : -52.45 = -2.024% (+/-0.00%)
bm_float.py 6034.93 -> 5908.30 : -126.63 = -2.098% (+/-0.01%)
bm_hexiom.py 48.96 -> 47.93 : -1.03 = -2.104% (+/-0.00%)
bm_nqueens.py 4510.63 -> 4459.94 : -50.69 = -1.124% (+/-0.00%)
bm_pidigits.py 650.28 -> 644.96 : -5.32 = -0.818% (+/-0.23%)
core_import_mpy_multi.py 564.77 -> 581.49 : +16.72 = +2.960% (+/-0.01%)
core_import_mpy_single.py 68.67 -> 67.16 : -1.51 = -2.199% (+/-0.01%)
core_qstr.py 64.16 -> 64.12 : -0.04 = -0.062% (+/-0.00%)
core_yield_from.py 362.58 -> 354.50 : -8.08 = -2.228% (+/-0.00%)
misc_aes.py 429.69 -> 405.59 : -24.10 = -5.609% (+/-0.01%)
misc_mandel.py 3485.13 -> 3416.51 : -68.62 = -1.969% (+/-0.00%)
misc_pystone.py 2496.53 -> 2405.56 : -90.97 = -3.644% (+/-0.01%)
misc_raytrace.py 381.47 -> 374.01 : -7.46 = -1.956% (+/-0.01%)
viper_call0.py 576.73 -> 572.49 : -4.24 = -0.735% (+/-0.04%)
viper_call1a.py 550.37 -> 546.21 : -4.16 = -0.756% (+/-0.09%)
viper_call1b.py 438.23 -> 435.68 : -2.55 = -0.582% (+/-0.06%)
viper_call1c.py 442.84 -> 440.04 : -2.80 = -0.632% (+/-0.08%)
viper_call2a.py 536.31 -> 532.35 : -3.96 = -0.738% (+/-0.06%)
viper_call2b.py 382.34 -> 377.07 : -5.27 = -1.378% (+/-0.03%)
And for unix on x64:
diff of scores (higher is better)
N=2000 M=2000 baseline -> this-commit diff diff% (error%)
bm_chaos.py 13594.20 -> 13073.84 : -520.36 = -3.828% (+/-5.44%)
bm_fannkuch.py 60.63 -> 59.58 : -1.05 = -1.732% (+/-3.01%)
bm_fft.py 112009.15 -> 111603.32 : -405.83 = -0.362% (+/-4.03%)
bm_float.py 246202.55 -> 247923.81 : +1721.26 = +0.699% (+/-2.79%)
bm_hexiom.py 615.65 -> 617.21 : +1.56 = +0.253% (+/-1.64%)
bm_nqueens.py 215807.95 -> 215600.96 : -206.99 = -0.096% (+/-3.52%)
bm_pidigits.py 8246.74 -> 8422.82 : +176.08 = +2.135% (+/-3.64%)
misc_aes.py 16133.00 -> 16452.74 : +319.74 = +1.982% (+/-1.50%)
misc_mandel.py 128146.69 -> 130796.43 : +2649.74 = +2.068% (+/-3.18%)
misc_pystone.py 83811.49 -> 83124.85 : -686.64 = -0.819% (+/-1.03%)
misc_raytrace.py 21688.02 -> 21385.10 : -302.92 = -1.397% (+/-3.20%)
The code size change is (firmware with a lot of frozen code benefits the
most):
bare-arm: +396 +0.697%
minimal x86: +1595 +0.979% [incl +32(data)]
unix x64: +2408 +0.470% [incl +800(data)]
unix nanbox: +1396 +0.309% [incl -96(data)]
stm32: -1256 -0.318% PYBV10
cc3200: +288 +0.157%
esp8266: -260 -0.037% GENERIC
esp32: -216 -0.014% GENERIC[incl -1072(data)]
nrf: +116 +0.067% pca10040
rp2: -664 -0.135% PICO
samd: +844 +0.607% ADAFRUIT_ITSYBITSY_M4_EXPRESS
As part of this change the .mpy file format version is bumped to version 6.
And mpy-tool.py has been improved to provide a good visualisation of the
contents of .mpy files.
In summary: this commit changes the bytecode to use qstr indirection, and
reworks the .mpy file format to be simpler and allow .mpy files to be
executed in-place. Performance is not impacted too much. Eventually it
will be possible to store such .mpy files in a linear, read-only, memory-
mappable filesystem so they can be executed from flash/ROM. This will
essentially be able to replace frozen code for most applications.
Signed-off-by: Damien George <damien@micropython.org>
This changes the git commit message line length check to ignore lines that
contain URLs, since these cannot be wrapped without breaking tools that
detect URLs and create a link.
Signed-off-by: David Lechner <david@pybricks.com>
This allows the compiler to merge strings: e.g. "update",
"difference_update" and "symmetric_difference_update" will all point to the
same memory.
No functional change.
The size reduction depends on the number of qstrs in the build. The change
this commit brings is:
bare-arm: -4 -0.007%
minimal x86: +150 +0.092% [incl +48(data)]
unix x64: -608 -0.118%
unix nanbox: -572 -0.126% [incl +32(data)]
stm32: -1392 -0.352% PYBV10
cc3200: -448 -0.244%
esp8266: -1208 -0.173% GENERIC
esp32: -1028 -0.068% GENERIC[incl -1020(data)]
nrf: -440 -0.252% pca10040
rp2: -1072 -0.217% PICO
samd: -368 -0.264% ADAFRUIT_ITSYBITSY_M4_EXPRESS
Performance is also improved (on bare metal at least) for the
core_import_mpy_multi.py, core_import_mpy_single.py and core_qstr.py
performance benchmarks.
Originally at adafruit#4583
Signed-off-by: Artyom Skrobov <tyomitch@gmail.com>
Changes are:
- decision to remount local filesystem on remote device is made only if
"MPY: soft reboot" is seen in the output after sending a ctrl-D
- a nice message is printed to the user when the remount occurs
- soft reset during raw REPL is now handled correctly
Fixes issue #7731.
Signed-off-by: Damien George <damien@micropython.org>
Damien George