Wykres commitów

668 Commity (master)

Autor SHA1 Wiadomość Data
Damien George bdbc869f9e py/persistentcode: Bump .mpy sub-version to 6.3.
This is required because the .mpy native ABI was changed by the
introduction of `mp_proto_fun_t`, see commits:
- 416465d81e
- 5e3006f117
- e2ff00e811

And three `mp_binary` functions were added to `mp_fun_table` in
commit d2276f0d41.

Signed-off-by: Damien George <damien@micropython.org>
2024-03-28 16:18:26 +11:00
Damien George d2276f0d41 py/dynruntime: Add mp_binary_get_size/get_val_array/set_val_array.
These are needed to read/write array.array objects, which is useful in
native code to provide fast extensions that work with big arrays of data.

Signed-off-by: Damien George <damien@micropython.org>
2024-03-28 16:18:09 +11:00
Christian Walther be89d4376b nrf/modules/machine: Enable code formatting.
It destroys a few manual alignments, but these seem minor compared to
the benefit of automated code style consistency.

Signed-off-by: Christian Walther <cwalther@gmx.ch>
2024-03-26 12:49:09 +11:00
Damien George 51d05c442a tools/manifestfile.py: Fix freeze() when script is an empty iterable.
The documentation for `freeze()` says that:
- If `script` is `None`, all files in `path` will be frozen.
- If `script` is an iterable then `freeze()` is called on all items of the
  iterable.

This commit makes sure this behaviour is followed when an empty tuple/list
is passed in for `script` (previously an empty tuple/list froze all files).

Fixes issue #14125.

Signed-off-by: Damien George <damien@micropython.org>
2024-03-25 13:41:37 +11:00
Damien George badc0106bd tools/ci.sh: Update webassembly CI tests.
Signed-off-by: Damien George <damien@micropython.org>
2024-03-22 14:31:25 +11:00
Damien George bf18ddd989 tools/mpy-tool.py: Fix merging of more than 128 mpy files.
The argument to MP_BC_MAKE_FUNCTION (raw code index) was being encoded as a
byte instead of a variable unsigned int.  That meant that if there were
more than 128 merged mpy files the encoding would be invalid.

Fix that by using `mp_encode_uint(idx)` to encode the raw code index.  And
also use `Opcode` constants for the opcode values to make it easier to
understand the code.

Signed-off-by: Damien George <damien@micropython.org>
2024-03-19 10:08:05 +11:00
iabdalkader 486ca3a688 tools/ci.sh: Add Arduino GIGA to stm32 CI build.
Signed-off-by: iabdalkader <i.abdalkader@gmail.com>
2024-03-15 18:11:28 +11:00
Phil Howard dda9b9c6da all: Prune trailing whitespace.
Prune trailing whitespace across the whole project (almost), done
automatically with:

    grep -IUrl --color "[[:blank:]]$" --exclude-dir=.git --exclude=*.exp |\
        xargs sed -i 's/[[:space:]]*$//'

Exceptions:
- Skip third-party code in lib/ and drivers/cc3100/
- Skip generated code in bluetooth_init_cc2564C_1.5.c
- Preserve command output whitespace in docs, eg:
  docs/esp8266/tutorial/repl.rst

Signed-off-by: Phil Howard <phil@gadgetoid.com>
2024-03-07 16:25:17 +11:00
Angus Gratton decf8e6a8b all: Remove the "STATIC" macro and just use "static" instead.
The STATIC macro was introduced a very long time ago in commit
d5df6cd44a.  The original reason for this was
to have the option to define it to nothing so that all static functions
become global functions and therefore visible to certain debug tools, so
one could do function size comparison and other things.

This STATIC feature is rarely (if ever) used.  And with the use of LTO and
heavy inline optimisation, analysing the size of individual functions when
they are not static is not a good representation of the size of code when
fully optimised.

So the macro does not have much use and it's simpler to just remove it.
Then you know exactly what it's doing.  For example, newcomers don't have
to learn what the STATIC macro is and why it exists.  Reading the code is
also less "loud" with a lowercase static.

One other minor point in favour of removing it, is that it stops bugs with
`STATIC inline`, which should always be `static inline`.

Methodology for this commit was:

1) git ls-files | egrep '\.[ch]$' | \
   xargs sed -Ei "s/(^| )STATIC($| )/\1static\2/"

2) Do some manual cleanup in the diff by searching for the word STATIC in
   comments and changing those back.

3) "git-grep STATIC docs/", manually fixed those cases.

4) "rg -t python STATIC", manually fixed codegen lines that used STATIC.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2024-03-07 14:20:42 +11:00
Angus Gratton 1ef2944b98 tools/manifestfile.py: Add --unix-ffi option.
Follow up to 35dd959133, allows explicitly
adding the unix-ffi library path from the command line.

This option is needed when building unix-ffi manifests in micropython-lib
CI.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2024-02-19 13:32:23 +11:00
Damien George def6ad4742 py/emitglue: Include fun_data_len in mp_raw_code_t only when saving.
Reduces the size of mp_raw_code_t in the case when MICROPY_DEBUG_PRINTERS
is enabled.

Signed-off-by: Damien George <damien@micropython.org>
2024-02-16 14:17:01 +11:00
Damien George a3a73b64a3 tools/mpy-tool.py: Skip generating frozen mp_raw_code_t when possible.
This reduces frozen code size by using the bytecode directly as the
`mp_proto_fun_t`.

Signed-off-by: Damien George <damien@micropython.org>
2024-02-16 14:17:01 +11:00
Damien George e2ff00e811 py/emitglue: Introduce mp_proto_fun_t as a more general mp_raw_code_t.
Allows bytecode itself to be used instead of an mp_raw_code_t in the simple
and common cases of a bytecode function without any children.

This can be used to further reduce frozen code size, and has the potential
to optimise other areas like importing.

Signed-off-by: Damien George <damien@micropython.org>
2024-02-16 14:17:01 +11:00
Damien George 5e3006f117 py/emitglue: Simplify mp_raw_code_t's kind and scope_flags members.
To simplify their access and reduce code size.

The `scope_flags` member is only ever used to determine if a function is a
generator or not, so make it reflect that fact as a bool type.

Signed-off-by: Damien George <damien@micropython.org>
2024-02-16 12:48:02 +11:00
Damien George 416465d81e py/emitglue: Provide a truncated mp_raw_code_t for non-asm code.
The `asm_n_pos_args` and `asm_type_sig` members of `mp_raw_code_t` are only
used for raw codes of type MP_CODE_NATIVE_ASM, which are rare, for example
in frozen code.  So using a truncated `mp_raw_code_t` in these cases helps
to reduce frozen code size on targets that have MICROPY_EMIT_INLINE_ASM
enabled.

With this, change in firmware size of RPI_PICO builds is -648.

Signed-off-by: Damien George <damien@micropython.org>
2024-02-16 12:48:02 +11:00
Damien George 39bf055d23 py/emitglue: Reorder and resize members of mp_raw_code_t.
The mp_raw_code_t struct has been reordered and some members resized.  The
`n_pos_args` member is renamed to `asm_n_pos_args`, and `type_sig` renamed
to `asm_type_sig` to indicate that these are used only for the inline-asm
emitters.  These two members are also grouped together in the struct.

The justifications for resizing the members are:
- `fun_data_len` can be 32-bits without issue
- `n_children` is already limited to 16-bits by
  `mp_emit_common_t::ct_cur_child`
- `scope_flags` is already limited to 16-bits by `scope_t::scope_flags`
- `prelude_offset` is already limited to 16-bits by the argument to
  `mp_emit_glue_assign_native()`
- it's reasonable to limit the maximim number of inline-asm arguments to 12
  (24 bits for `asm_type_sig` divided by 2)

This change helps to reduce frozen code size (and in some cases RAM usage)
in the following cases:
- 64-bit targets
- builds with MICROPY_PY_SYS_SETTRACE enabled
- builds with MICROPY_EMIT_MACHINE_CODE enabled but MICROPY_EMIT_INLINE_ASM
  disabled

With this change, unix 64-bit builds are -4080 bytes in size.  Bare-metal
ports like rp2 are unchanged (because mp_raw_code_t is still 32 bytes on
those 32-bit targets).

Signed-off-by: Damien George <damien@micropython.org>
2024-02-16 12:48:02 +11:00
Damien George 0432f73206 tools/mpy-tool.py: Fix static qstrs when freezing without qstr header.
It's rare to freeze .mpy files without specifying a qstr header from a
firmware build, but it can be useful for testing, eg
`mpy-tool.py -f test.mpy`.  Fix this case so static qstrs are properly
excluded from the frozen qstr list.

Signed-off-by: Damien George <damien@micropython.org>
2024-02-12 11:56:24 +11:00
Damien George 35dd959133 tools/manifestfile.py: Change library search to use a list of paths.
This commit changes how library packages are searched for when a manifest
file is loaded: there is now simply a list of library paths that is
searched in order for the given package.  This list defaults to the
main directories in micropython-lib, but can be added to -- either appended
or prepended -- by using `add_library()`.

In particular the way unix-ffi library packages are searched has changed,
because the `unix_ffi` argument to `require()` is now removed.  Instead, if
a build wants to include packages from micropython-lib/unix-ffi, then it
must explicitly add this to the list of paths to search using:

    add_library("unix-ffi", "$(MPY_LIB_DIR)/unix-ffi")

Work done in collaboration with Jim Mussared.

Signed-off-by: Damien George <damien@micropython.org>
2024-02-08 17:16:10 +11:00
Angus Gratton 84e90639fa github/workflows: Move codespell to a GitHub workflow, version it.
Similar to ruff.yaml, it's simpler to run the codespell command directly
from a workflow file.  And developers can run codespell directly from the
command line without the need for options, or just use pre-commit.

This commit also applies a specific version to codespell, same as
pre-commit (introduced in a166d805f4).

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2024-02-07 15:19:38 +11:00
Jim Mussared 7ea503929a py/qstr: Add support for MICROPY_QSTR_BYTES_IN_HASH=0.
This disables using qstr hashes altogether, which saves RAM and flash
(two bytes per interned string on a typical build) as well as code size.
On PYBV11 this is worth over 3k flash.

qstr comparison will now be done just by length then data. This affects
qstr_find_strn although this has a negligible performance impact as, for a
given comparison, the length and first character will ~usually be
different anyway.

String hashing (e.g. builtin `hash()` and map.c) now need to compute the
hash dynamically, and for the map case this does come at a performance
cost.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2024-01-25 16:38:17 +11:00
Angus Gratton 307ecc5707 docs: Add note about position-only arguments in CPython vs MicroPython.
Required modifying the gen-cpydiff.py code to allow a "preamble" section to
be inserted at the top of any of the generated files.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2024-01-25 11:56:38 +11:00
Damien George 7bbcee3cf0 tests: Move port-specific test directories into tests/ports/ directory.
To keep them all together, mirroring the top-level directory structure.

Signed-off-by: Damien George <damien@micropython.org>
2024-01-22 11:48:27 +11:00
Jochen Sprickerhof 16c6bc47cf tools/mpremote: Reduce dependency on importlib_metadata.
No longer require importlib_metadata on new Python versions as it is
included in the standard distribution.

Signed-off-by: Jochen Sprickerhof <git@jochen.sprickerhof.de>
2024-01-17 11:27:47 +11:00
Damien George 7a794d0d8e tools/gen-changelog.sh: Exclude "-preview" tags from generated log.
Signed-off-by: Damien George <damien@micropython.org>
2024-01-05 15:06:14 +11:00
Jim Mussared 5d28bb4adb tools/manifestfile.py: Add support for external libraries.
This adds a `add_library(name, path)` method for use in manifest.py that
allows registering an external path (e.g. to another repo) by name.

This name can then be passed to `require("package", library="name")` to
reference packages in that repo/library rather than micropython-lib.

Within the external library, `require()` continues to work as normal
(referencing micropython-lib) by default, but they can also specify the
library name to require another package from that repo/library.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-12-21 15:28:32 +11:00
Jim Mussared 74fd7b3d32 tools/ci.sh: Set `ulimit -n` for unix CI.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-12-21 11:05:31 +11:00
IhorNehrutsa 4365edb810 esp32: Change minimum supported IDF version to v5.0.4.
Also, IDF v5.1.2 is now supported, just not used by default.

IDF v5.0.2 still builds but we cannot guarantee continued support for this
version moving forward.

Signed-off-by: IhorNehrutsa <IhorNehrutsa@gmail.com>
2023-12-11 13:00:09 +11:00
Damien George d3595fed2b tools/ci.sh: Build ESP32_GENERIC-SPIRAM as part of esp32 CI.
Signed-off-by: Damien George <damien@micropython.org>
2023-12-06 12:37:25 +11:00
Angus Gratton 527c4b0497 github/workflows: Enable ccache for esp32 build.
CCaches are scoped per-job.

Uses https://github.com/hendrikmuhs/ccache-action to get desired behaviour
(updating the cache on each run).

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2023-12-06 12:15:33 +11:00
Angus Gratton b6df8f8452 github/workflows: Use build matrix for esp32 port.
Allows splitting the esp32 job into multiple parts without too much
boilerplate.  The matrix is parameterised using the name of the function to
call in tools/ci.sh, to minimise the dependency on GitHub Actions.

This can get esp32 build times down around 3m if IDF is cached already.

If the cache is cold, the cache preparation step on each job can double up
against each other.  However, restructuring the workflow to not do this
seems either complex or requires copy-pasting the entire cache step.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2023-12-06 12:13:56 +11:00
Angus Gratton 9f620d2819 github/workflows: Cache ESP-IDF checkout and installation.
Cache is keyed on the ESP-IDF version used in CI, so there shouldn't be any
cache invalidation issues when ESP-IDF version changes.

Restoring from cache takes approx 15s, compared to 2-3m to perform these
steps (ESP-IDF tools install, ESP-IDF clone, ESP-IDF submodule clone) the
first time.

Cache size is approx 1.6GB, the git clone is tweaked as much as possible to
keep the size down.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2023-12-06 12:12:52 +11:00
Andrew Leech bbc5a18d09 tools/mpremote: Add ioctl to specify large read buffer size.
Speeds up importing files from mounted filesystem.

Also fix the return code for invalid / unsupported ioctl requests.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
2023-11-09 11:30:11 +11:00
Alessandro Gatti bea6ff82fa tools/tinytest-codegen.py: Externalise tests list.
Remove port-specific test directories and excluded tests from
tinytest-codegen, and let it read said information from an external file.
This way tinytest-codegen is not limited to always generate tests for the
`qemu-arm` target.

This allows having port-specific test directory and excluded tests for more
than one QEMU bare-metal target.

The `qemu-arm` port Makefile was modified to work with the generator
changes and a tests profile file was added to said port.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2023-11-07 15:53:27 +11:00
Jim Mussared 724ebb9916 stm32/boards/make-pins.py: Update to use tools/boardgen.py.
Requires additions to tools/boardgen.py for stm32 pin generation.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-11-03 14:16:49 +11:00
Jim Mussared 4bd6ec9ae4 tools/boardgen.py: Add initial implementation of a common make-pins.py.
For now, this implements the functionality required for esp32 and rp2,
including support for numeric pins, rp2 alternate functions, and rp2
extended pins.

This also updates the rp2 port to use the same structure for pins.h and
pins.csv as for esp32, and moves the pin definitions directly into the
table (rather than having a table of pointers), which is a small code size
improvement.

Support for "hidden" pins in pins.csv is added (matching the stm32
implementation).

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-11-03 14:06:27 +11:00
Jim Mussared bbd8760bd9 all: Update Python formatting to ruff-format.
This updates a small number of files that change with ruff-format's (vs
black's) rules.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-11-03 13:30:41 +11:00
Jim Mussared 303ccca7c6 all: Replace "black" with "ruff format".
- Add config for [tool.ruff.format] to pyproject.toml.
- Update pre-commit to run both ruff and ruff-format (and only check C
  files when running codeformat.py)
- Update CI.
- Simplify codeformat.py to remove all the Python-specific logic (just run
  "ruff format" directly).

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-11-03 13:30:38 +11:00
Jim Mussared e579ebb11b teensy: Remove the teensy port.
This port is largely unmaintained, has limited features (the only hardware
support is for GPIO and timer, and no machine module), only supports a
small number of Teensy boards, and can be confused with the mimxrt support
for Teensy 4.x.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
2023-10-31 13:04:41 +11:00
Jim Mussared 64c79a5423 py/qstr: Add support for sorted qstr pools.
This provides a significant performance boost for qstr_find_strn, which is
called a lot during parsing and loading of .mpy files, as well as interning
of string objects (which happens in most string methods that return new
strings).

Also adds comments to explain the "static" qstrs.  These are part of the
.mpy ABI and avoid needing to duplicate string data for QSTRs known to
already be in the firmware.  The static pool isn't currently sorted, but in
the future we could either split the static pool into the sorted regions,
or in the next .mpy version just sort them.

Based on initial work done by @amirgon in #6896.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-10-30 11:10:02 +11:00
Jim Mussared 3bf70f16e9 py/mkrules.mk: Add MICROPY_PREVIEW_VERSION_2.
This provides a way to enable features and changes slated for MicroPython
2.x, by running `make MICROPY_PREVIEW_VERSION_2=1`. Also supported for
the cmake ports (except Zephyr).

This is an alternative to having a 2.x development branch (or equivalently,
keeping a 1.x release branch). Any feature or change that needs to be
"hidden" until 2.x can use this flag (either in the Makefile or the
preprocessor).

A good example is changing function arguments or other public API features,
in particular to aid in improving consistency between ports.

When `MICROPY_PREVIEW_VERSION_2` is enabled, the REPL banner is amended to
say "MicroPython (with v2.0 preview) vX.Y.Z", and sys.implementation gets a
new field `_v2` set to `True`.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-10-27 15:28:46 +11:00
Damien George 6967ff3c58 py/persistentcode: Bump .mpy sub-version.
This is required because the previous commit changed the .mpy native ABI.

Signed-off-by: Damien George <damien@micropython.org>
2023-10-16 11:25:31 +11:00
Andrew Leech d2a9d70c09 tools/ci.sh: Ensure enough commits are fetched for a common ancestor.
This commit updates the ci script to automatically fetch all upstream if
the common commit hasn't been found; this should preserve the speed of CI
checks for most PR's, and use a reliable but slow fetch if needed for older
ones.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
2023-10-11 14:05:14 +11:00
Jim Mussared 69e34b6b6b all: Switch to new preview build versioning scheme.
See https://github.com/micropython/micropython/issues/12127 for details.

Previously at the point when a release is made, we update mpconfig.h
and set a git tag. i.e. the version increments at the release.

Now the version increments immediately after the release. The workflow is:
1. Final commit in the cycle updates mpconfig.h to set (X, Y, 0, 0) (i.e.
   clear the pre-release state).
2. This commit is tagged "vX.Y.0".
3. First commit for the new cycle updates mpconfig.h to set (X, Y+1, 0, 1)
   (i.e. increment the minor version, set the pre-release state).
4. This commit is tagged "vX.Y+1.0-preview".

The idea is that a nightly build is actually a "preview" of the _next_
release. i.e. any documentation describing the current release may not
actually match the nightly build. So we use "preview" as our semver
pre-release identifier.

Changes in this commit:
 - Add MICROPY_VERSION_PRERELEASE to mpconfig.h to allow indicating that
   this is not a release version.
 - Remove unused MICROPY_VERSION integer.
 - Append "-preview" to MICROPY_VERSION_STRING when the pre-release state
   is set.
 - Update py/makeversionhdr.py to no longer generate MICROPY_GIT_HASH.
 - Remove the one place MICROPY_GIT_HASH was used (it can use
   MICROPY_GIT_TAG instead).
 - Update py/makeversionhdr.py to also understand
   MICROPY_VERSION_PRERELEASE in mpconfig.h.
 - Update py/makeversionhdr.py to convert the git-describe output into
   semver-compatible "X.Y.Z-preview.N.gHASH".
 - Update autobuild.sh to generate filenames using the new scheme.
 - Update remove_old_firmware.py to match new scheme.
 - Update mpremote's pyproject.toml to handle the "-preview" suffix in the
   tag. setuptools_scm maps to this "rc0" to match PEP440.
 - Fix docs heading where it incorrectly said "vvX.Y.Z" for release docs.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-10-06 12:10:14 +11:00
Damien George 040a96d00e docs,tools: Change remaining "urequests" references to "requests".
Signed-off-by: Damien George <damien@micropython.org>
2023-10-05 14:04:45 +11:00
Damien George 92717a95c0 tools/metrics.py: Fix esp32 and esp8266 board names after renaming.
Signed-off-by: Damien George <damien@micropython.org>
2023-10-04 15:44:25 +11:00
Damien George cf490a7091 all: Fix various spelling mistakes found by codespell 2.2.6.
Signed-off-by: Damien George <damien@micropython.org>
2023-10-03 11:24:50 +11:00
Jos Verlinde 58c2c503a9 tools/mpremote: Add support for rfc2217, serial over TCP.
Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
2023-09-29 14:40:17 +10:00
Jim Mussared 88ecc78eb3 tools/autobuild/build-downloads.py: Verify standard features.
Defines the list of standard features and ensures that each board.json
only uses those ones. This list can be extended, but needs to be a
deliberate decision.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-09-29 14:18:33 +10:00
Damien George b0e03b3e07 tools/autobuild: Include .bin firmware in renesas-ra build output.
Signed-off-by: Damien George <damien@micropython.org>
2023-09-18 11:05:14 +10:00
iabdalkader 174bb28d8e tools/ci.sh: Add ARDUINO_PORTENTA_C33 to RA CI build.
Signed-off-by: iabdalkader <i.abdalkader@gmail.com>
2023-09-15 00:10:52 +10:00