Wykres commitów

396 Commity (master)

Autor SHA1 Wiadomość Data
Damien George 9955553001 extmod/modutime: Provide a generic time module.
Based on extmod/utime_mphal.c, with:
- a globals dict added
- time.localtime wrapper added
- time.time wrapper added
- time.time_ns function added

New configuration options are added for this module:
- MICROPY_PY_UTIME (enabled at basic features level)
- MICROPY_PY_UTIME_GMTIME_LOCALTIME_MKTIME
- MICROPY_PY_UTIME_TIME_TIME_NS

Signed-off-by: Damien George <damien@micropython.org>
2023-04-27 15:09:56 +10:00
Jim Mussared a42a8022d8 extmod/modbluetooth: Make all HCI transports trace in the same format.
- Use HCI_TRACE macro consistently.
 - Use the same colour formatting.
 - Add a tool to convert to .pcap for Wireshark.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-04-26 11:37:20 +10:00
Jim Mussared 256f47e2f8 extmod/btstack: Fix indicate/notify queuing.
This adds a mechanism to track a pending notify/indicate operation that
is deferred due to the send buffer being full. This uses a tracked alloc
that is passed as the content arg to the callback.

This replaces the previous mechanism that did this via the global pending
op queue, shared with client read/write ops.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-04-26 11:37:20 +10:00
Damien George 9e6885ad82 extmod/btstack: Switch to use hci_dump_init instead of hci_dump_open.
The latter is no longer available in the version of BTstack now in use by
this repository.

Signed-off-by: Damien George <damien@micropython.org>
2023-04-22 00:39:31 +10:00
Damien George b3c8ab37ec py/gc: Make gc_dump_info/gc_dump_alloc_table take a printer as argument.
So that callers can redirect the output if needed.

Signed-off-by: Damien George <damien@micropython.org>
2023-03-10 10:58:10 +11:00
Jim Mussared 69b93527d5 ports: Make BOARD default from BOARD_DIR in Makefile's.
This allows:

    $ make BOARD_DIR=path/to/board

to infer BOARD=board, rather than the previous behavior that required
additionally setting BOARD explicitly.

Also makes the same change for VARIANT_DIR -> VARIANT on Unix.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
2023-02-23 09:47:08 +11:00
小权一句两句 94ee1b629a unix/main: Free pathbuf when there's an error opening a file. 2023-01-20 19:38:14 +11:00
Juan Francisco Cantero Hurtado 720f2cfba9 unix/Makefile: Disable building the shared library of libffi.
MicroPython doesn't need the shared library and disabling it in the
configure could ease the build of a static version of MicroPython.
2023-01-19 15:24:43 +11:00
David Lechner 958f748e53 unix/moduos: Implement 2-arg version of os.getenv().
This adds the `default` argument of `os.getenv(key, default=None)`.

Signed-off-by: David Lechner <david@pybricks.com>
2022-12-14 13:38:39 +11:00
Jeff Epler d75ff42297 unix/coverage: Add extra GC coverage test for ATB gap byte.
The assertion that is added here (to gc.c) fails when running this new test
if ALLOC_TABLE_GAP_BYTE is set to 0.

Signed-off-by: Jeff Epler <jepler@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
2022-12-08 14:35:08 +11:00
iabdalkader 71881116e6 extmod/extmod.mk: Set default mbedtls config file in extmod Makefile. 2022-11-09 18:43:13 +01:00
Damien George 3bb371c5f2 unix/variants: Change mip package to mip-cmdline.
mip-cmdline adds command-line support to mip, useful for the unix port, via
micropython -m mip ...

Signed-off-by: Damien George <damien@micropython.org>
2022-11-09 14:58:54 +11:00
Damien George 9347545f9e extmod/mbedtls: Enable MBEDTLS_TLS_DEFAULT_ALLOW_SHA1_IN_KEY_EXCHANGE.
This was already enabled on all ports except mimxrt.  Now it's enabled on
all of them.

Signed-off-by: Damien George <damien@micropython.org>
2022-10-22 19:06:21 +11:00
Damien George b337678964 extmod/mbedtls: Add common configuration file, and use it in all ports.
This is a no-op change.

Signed-off-by: Damien George <damien@micropython.org>
2022-10-22 19:06:21 +11:00
Damien George 67f98ba10c extmod/btstack: Update BTstack bindings to work with latest BTstack.
The following multi-tests pass (eg with PYBD_SF6+LEGO_HUB_NO6):

    ble_gap_advertise.py
    ble_gap_connect.py
    ble_gap_device_name.py
    ble_gattc_discover_services.py
    ble_gatt_data_transfer.py
    perf_gatt_char_write.py
    perf_gatt_notify.py
    stress_log_filesystem.py

These are the same tests that passed prior to this BTstack update.

Also tested on the unix port using H4 transport.

Signed-off-by: Damien George <damien@micropython.org>
2022-10-22 14:28:25 +11:00
Damien George 89b3207376 unix/modffi: Move header includes inside MICROPY_PY_FFI guard.
So ffi.h is not needed if this module is disabled.

Signed-off-by: Damien George <damien@micropython.org>
2022-10-14 14:36:17 +11:00
Jim Mussared d6d8722558 extmod: Make extmod.mk self-contained.
This makes it so that all a port needs to do is set the relevant variables
and "include extmod.mk" and doesn't need to worry about adding anything to
OBJ, CFLAGS, SRC_QSTR, etc.

Make all extmod variables (src, flags, etc) private to extmod.mk.

Also move common/shared, extmod-related fragments (e.g. wiznet, cyw43,
bluetooth) into extmod.mk.

Now that SRC_MOD, CFLAGS_MOD, CXXFLAGS_MOD are unused by both extmod.mk
(and user-C-modules in a previous commit), remove all uses of them from
port makefiles.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-10-11 23:31:49 +11:00
Jim Mussared 87011f6353 extmod/extmod.mk: Make extmod.mk handle GIT_SUBMODULES.
This applies to nimble, btstack, axtls, mbedtls, lwip.

Rather than having the ports individually manage GIT_SUBMODULES for these
components, make extmod.mk append them when the relevant feature is
enabled.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-10-11 23:29:09 +11:00
Jim Mussared 065af04a4e unix/Makefile: Don't use _MOD variable names.
This conflicts with the triple-usage of these variables for user-C-modules
and extmod source.

For CFLAGS_MOD, just use CFLAGS directly. For SRC, use SRC_C directly as
the relevant files are all guarded by the preprocessor anyway.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-10-11 23:27:01 +11:00
Jim Mussared 8e912a501a unix: Enable sys.executable.
Gives the absolute path to the unix micropython binary.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
2022-10-11 18:10:30 +11:00
Jim Mussared 924a3e03ec top: Replace upip with mip everywhere.
Updates all README.md and docs, and manifests to `require("mip")`.

Also extend and improve the documentation on freezing and packaging.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-30 17:34:03 +10:00
Jim Mussared 920da9c5e3 unix/variants/coverage: Add test for manifest freeze_mpy().
This uses the frozentest.mpy that is also used by ports/minimal.

Also fixes two bugs that these new tests picked up:
 - File extension matching in manifestfile.py.
 - Handling of freeze_mpy results in makemanifest.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-19 23:51:10 +10:00
Jim Mussared 94beeabd2e py/obj: Convert make_new into a mp_obj_type_t slot.
Instead of being an explicit field, it's now a slot like all the other
methods.

This is a marginal code size improvement because most types have a make_new
(100/138 on PYBV11), however it improves consistency in how types are
declared, removing the special case for make_new.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-19 19:06:15 +10:00
Jim Mussared 6da41b5900 py/obj: Merge getiter and iternext mp_obj_type_t slots.
The goal here is to remove a slot (making way to turn make_new into a slot)
as well as reduce code size by the ~40 references to mp_identity_getiter
and mp_stream_unbuffered_iter.

This introduces two new type flags:
- MP_TYPE_FLAG_ITER_IS_ITERNEXT: This means that the "iter" slot in the
  type is "iternext", and should use the identity getiter.
- MP_TYPE_FLAG_ITER_IS_CUSTOM: This means that the "iter" slot is a pointer
  to a mp_getiter_iternext_custom_t instance, which then defines both
  getiter and iternext.

And a third flag that is the OR of both, MP_TYPE_FLAG_ITER_IS_STREAM: This
means that the type should use the identity getiter, and
mp_stream_unbuffered_iter as iternext.

Finally, MP_TYPE_FLAG_ITER_IS_GETITER is defined as a no-op flag to give
the default case where "iter" is "getiter".

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-19 19:06:13 +10:00
Jim Mussared a52cd5b07d py/obj: Add accessors for type slots and use everywhere.
This is a no-op, but sets the stage for changing the mp_obj_type_t
representation.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-19 19:06:07 +10:00
Jim Mussared 9dce82776d all: Remove unnecessary locals_dict cast.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-19 19:06:01 +10:00
Jim Mussared 662b9761b3 all: Make all mp_obj_type_t defs use MP_DEFINE_CONST_OBJ_TYPE.
In preparation for upcoming rework of mp_obj_type_t layout.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-19 19:06:01 +10:00
Jim Mussared fb2a57800a all: Simplify buffer protocol to just a "get buffer" callback.
The buffer protocol type only has a single member, and this existing layout
creates problems for the upcoming split/slot-index mp_obj_type_t layout
optimisations.

If we need to make the buffer protocol more sophisticated in the future
either we can rely on the mp_obj_type_t optimisations to just add
additional slots to mp_obj_type_t or re-visit the buffer protocol then.

This change is a no-op in terms of generated code.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-19 18:40:39 +10:00
Jim Mussared 51b054dd66 unix: Refactor mpconfigvariant.mk.
All variants (except minimal) enable text compression and fat/lfs, so move
them to the common mpconfigport.mk.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-13 17:40:55 +10:00
Jim Mussared 3e5b1be8ca py/mpconfig: Add "everything" features from unix coverage.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-13 17:40:33 +10:00
Jim Mussared c1530a0ce8 unix: Refactor mpconfigport.h and mpconfigvariant.h.
This is a no-op for coverage and minimal.

The standard and dev variants have been merged and enable the same feature
set as a typical bare-metal board.  And remove the CI for the dev build.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-13 17:39:03 +10:00
Jim Mussared f4fed02537 unix/variants: Remove freedos and fast variants.
The freedos variant is untested by CI and is difficult to maintain.  The
fast variant is not a good name for what it does.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-13 17:37:55 +10:00
Andrew Leech 86676a43aa unix/mpconfigport: Enable MICROPY_ENABLE_FINALISER when VFS is used. 2022-09-13 13:06:56 +10:00
Jim Mussared 203dae41fb all: Update all manifest.py files to use new features.
Changes in this commit:
- Manifest include's now use the directory path where possible (no longer
  necessary to include the manifest.py file explicitly).
- Add manifest.py for all drivers and components that are referenced by
  port/board manifests.
- Replace all uses of freeze() with package()/module(), except for port and
  board modules.
- Use opt=3 everywhere, for consistency and to reduce code size.
- Use require() instead of include() for all micropython-lib references.
- Remove support for optional board-level manifest.py in mimxrt port, to
  make it behave the same as other ports (the board must set
  FROZEN_MANIFEST to a custom manifest.py, which can optionally include the
  default, port-level manifest).
- Also reinstates modules that were accidentally removed from the esp8266
  512k build in fbe9417b90.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
2022-09-05 18:43:18 +10:00
Jim Mussared 64c62f8cf1 README: Simplify and update, and move unix section to separate file.
Changes are:
- Remove unix- and stm32-specific sections (move unix to its own
  README.md), stm32 was duplicated.
- Add links to GitHub Discussions and Discord.
- Update information about the project.
- Add a getting started section.
- Explain `make submodules`.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
2022-08-30 13:11:33 +10:00
Jim Mussared 3a910b1565 py/objstr: Optimise mp_obj_new_str_from_vstr for known-safe strings.
The new `mp_obj_new_str_from_utf8_vstr` can be used when you know you
already have a unicode-safe string.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-26 16:44:35 +10:00
Jim Mussared 8a0ee5a5c0 py/objstr: Split mp_obj_str_from_vstr into bytes/str versions.
Previously the desired output type was specified.  Now make the type part
of the function name.  Because this function is used in a few places this
saves code size due to smaller call-site.

This makes `mp_obj_new_str_type_from_vstr` a private function of objstr.c
(which is almost the only place where the output type isn't a compile-time
constant).

This saves ~140 bytes on PYBV11.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-26 16:43:55 +10:00
Chris Waggoner b1efc1340a mpy-cross,unix: Include alloca.h for NetBSD. 2022-08-23 13:39:18 +10:00
Damien George 8f4c108025 all: Remove MICROPY_PY_IO_FILEIO config option.
Since commit e65d1e69e8 there is no longer an
io.FileIO class, so this option is no longer needed.

This option also controlled whether or not files supported being opened in
binary mode (eg 'rb'), and could, if disabled, lead to confusion as to why
opening a file in binary mode silently did the wrong thing (it would just
open in text mode if MICROPY_PY_IO_FILEIO was disabled).

The various VFS implementations (POSIX, FAT, LFS) were the only places
where enabling this option made a difference, and in almost all cases where
one of these filesystems were enabled, MICROPY_PY_IO_FILEIO was also
enabled.  So it makes sense to just unconditionally enable this feature
(ability to open a file in binary mode) in all cases, and so just remove
this config option altogether.  That makes configuration simpler and means
binary file support always exists (and opening a file in binary mode is
arguably more fundamental than opening in text mode, so if anything should
be configurable then it should be the ability to open in text mode).

Signed-off-by: Damien George <damien@micropython.org>
2022-08-18 11:54:17 +10:00
Efi Weiss 98bd7e33b3 unix/modusocket: Support proto and flags arguments to getaddrinfo.
Signed-off-by: Efi Weiss <efiwiss@gmail.com>
2022-08-17 12:02:42 +10:00
Damien George ec24cd1d25 mpy-cross,unix: Remove .gitignore file.
Now that all build artefacts are placed in a build/ directory the gitignore
is no longer needed.

Signed-off-by: Damien George <damien@micropython.org>
2022-08-12 16:38:04 +10:00
Jim Mussared 28aaab9590 py/objstr: Add hex/fromhex to bytes/memoryview/bytearray.
These were added in Python 3.5.

Enabled via MICROPY_PY_BUILTINS_BYTES_HEX, and enabled by default for all
ports that currently have ubinascii.

Rework ubinascii to use the implementation of these methods.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-08-12 12:44:30 +10:00
David Yang b6651a7a89 unix/modjni: Add missing const qualifier. 2022-08-11 14:09:16 +10:00
Damien George d53c3b6ade unix/variants: Remove variant suffix from executable filename.
The executable now lives in the build directory, and since the build
directory already contains the variant name there is no need to also add
it to the executable.

Signed-off-by: Damien George <damien@micropython.org>
2022-08-11 13:34:34 +10:00
Daniel Jour 47c84286e8 all: Fix paths to mpy-cross and micropython binaries.
Binaries built using the Make build system now no longer appear in the
working directory of the build, but rather in the build directory.  Thus
some paths had to be adjusted.
2022-08-11 13:31:13 +10:00
David Lechner 3c32ca6e77 unix/unix_mphal: Allow overriding hal time functions.
This adds #ifdefs around each of the mp_hal_* time functions for the unix
port.  This allows variants to override individual functions as needed.

Signed-off-by: David Lechner <david@pybricks.com>
2022-07-29 11:06:12 +10:00
Jim Mussared 092784da19 ports: Remove unused mp_type_{fileio/textio} macros in mpconfigport.h.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-07-26 18:07:22 +10:00
Rob Knegjens d2e4cf00cc unix: Enable MICROPY_GC_SPLIT_HEAP on coverage build.
With a new option to evenly split the GC heap over multiple areas.  This
adds code coverage for gc_add() and code associated with
MICROPY_GC_SPLIT_HEAP.
2022-07-23 00:43:08 +10:00
David Lechner 03fb671833 unix/mpconfigport: Allow overriding MICROPY_EVENT_POLL_HOOK.
This allows variants to supply their own `MICROPY_EVENT_POLL_HOOK`.

Signed-off-by: David Lechner <david@pybricks.com>
2022-07-19 11:24:42 +10:00
David Lechner c947c25294 unix/Makefile: Only checkout libffi submodule when used.
This moves the libffi submodule variable modifier inside of the if
statement where it is actually used so that the submodule will only be
checked out if it is actually being used.

A new DEPLIBS variable is also introduced to prevent building the libffi
submodule when not needed.

Signed-off-by: David Lechner <david@pybricks.com>
2022-07-19 11:17:16 +10:00
Damien George b89422ceaa extmod: Always use custom mbedtls error message code.
All ports that use mbedtls use the custom error messages in
mp_mbedtls_errors.c.  This commit simplifies the build so that ports don't
need to explicitly add this file, it's now used by default when mbedtls is
enabled.

Signed-off-by: Damien George <damien@micropython.org>
2022-07-18 22:55:22 +10:00
Damien George b87f796ef5 unix/Makefile: Only include axtls in GIT_SUBMODULES list when enabled.
Signed-off-by: Damien George <damien@micropython.org>
2022-07-18 22:45:38 +10:00
Damien George 13c78ab6fe unix/mpconfigport: Switch ussl from axtls to mbedtls.
The reasons to make this switch are:
- The axtls library is not being maintained/updated, mbedtls is.
- So CI and tests can run against mbedtls, which is now the main TLS
  library used by the ports (eg stm32, rp2, mimxrt, esp32). Only esp8266
  uses axtls.

Increases unix standard build on x86-64 by about 89000 bytes.

Signed-off-by: Damien George <damien@micropython.org>
2022-07-18 22:43:01 +10:00
Damien George 26dec1fafb unix/mbedtls: Add custom config for mbedtls.
Based on existing bare-metal config from stm32.  Also uses shorter error
messages from lib/mbedtls_errors.

Signed-off-by: Damien George <damien@micropython.org>
2022-07-18 22:42:51 +10:00
Damien George e30e7ced6f unix/mpconfigport: Make MICROPY_EVENT_POLL_HOOK call usleep directly.
So that MICROPY_EVENT_POLL_HOOK can be used without including py/mphal.h.

Signed-off-by: Damien George <damien@micropython.org>
2022-07-18 18:12:39 +10:00
David Lechner 7e4b205cb0 py/mpstate: Drop MICROPY_PORT_ROOT_POINTERS from mp_state_vm_t.
All in-tree uses of MICROPY_PORT_ROOT_POINTERS have been replaced with
MP_REGISTER_ROOT_POINTER(), so now we can remove both
MICROPY_PORT_ROOT_POINTERS and MICROPY_BOARD_ROOT_POINTERS from the code
and remaining config files.

Signed-off-by: David Lechner <david@pybricks.com>
2022-07-18 13:51:16 +10:00
David Lechner e531b72b56 extmod/nimble: Use MP_REGISTER_ROOT_POINTER().
This uses MP_REGISTER_ROOT_POINTER() to register bluetooth_nimble_memory
and bluetooth_nimble_root_pointers and removes the same from all
mpconfigport.h.

Signed-off-by: David Lechner <david@pybricks.com>
2022-07-18 13:50:14 +10:00
David Lechner 8fa6191f95 extmod/btstack: Use MP_REGISTER_ROOT_POINTER().
This uses MP_REGISTER_ROOT_POINTER() to register
bluetooth_btstack_root_pointers and removes the same from all
mpconfigport.h.

Signed-off-by: David Lechner <david@pybricks.com>
2022-07-18 13:49:51 +10:00
David Lechner e5631d9595 unix/alloc: Use MP_REGISTER_ROOT_POINTER() for mmap_region_head.
This uses MP_REGISTER_ROOT_POINTER() to register mmap_region_head and
removes the same from mpconfigport.h.

Signed-off-by: David Lechner <david@pybricks.com>
2022-07-18 13:49:09 +10:00
David Lechner 81dbea1ce3 shared/readline: Use MP_REGISTER_ROOT_POINTER().
This uses MP_REGISTER_ROOT_POINTER() to register the readline_history root
pointer array used by shared/readline.c and removes the registration from
all mpconfigport.h files.

This also required adding a new MICROPY_READLINE_HISTORY_SIZE config option
since not all ports used the same sized array.

Signed-off-by: David Lechner <david@pybricks.com>
2022-07-18 13:48:49 +10:00
David Lechner 093c4b6b26 unix/main: Restore tty settings on nlr_jump_fail().
Since nlr_jump_fail() exits the process, it can leave the terminal in raw
mode which means characters are not echoed.  Fix this by restoring the
original terminal mode.

Signed-off-by: David Lechner <david@pybricks.com>
2022-07-12 17:20:37 +10:00
David Lechner 5ab7dfe6db unix/moduos: Include errno.h.
The file `ports/unix/moduos.c` uses `errno` so it needs to include
`errno.h`, otherwise a compiler error can occur.

Signed-off-by: David Lechner <david@pybricks.com>
2022-07-05 23:44:07 +10:00
Damien George afa4d0a4b7 unix/variants/dev: Allow all config options to be overridden.
They can be overridden on the make command line, for example.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-29 12:48:33 +10:00
Damien George 510bea9383 unix/variants/coverage: Change config to use ROM level everything.
This ROM level is not yet fully defined, but it at least enables
MICROPY_PY_SYS_TRACEBACKLIMIT.  The coverage build should have everything
enabled, so it makes sense to use this ROM level for it.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-29 12:48:33 +10:00
Damien George fad978ae5f unix/variants: Enable remaining options on dev,coverage from ROM extra.
So that the default configuration for the dev and coverage variants
includes all options set by MICROPY_CONFIG_ROM_LEVEL_EXTRA_FEATURES.

Note that enabling MICROPY_PY_SYS_STDIO_BUFFER on unix doesn't do anything
because unix doesn't use shared/runtime/sys_stdio_mphal.c.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-29 12:23:11 +10:00
Damien George fe55d3e016 unix/variants: Move setting of MICROPY_PY_USELECT to port config file.
The default is the same as before: MICROPY_PY_USELECT=0 and
MICROPY_PY_USELECT_POSIX=1.  But now this can be easily overridden at the
make command-line using, eg:

    make VARIANT=dev CFLAGS_EXTRA=-DMICROPY_PY_USELECT=1

Signed-off-by: Damien George <damien@micropython.org>
2022-06-29 12:22:56 +10:00
David Lechner c118b5d0e4 extmod/extmod.mk: Separate out extmod file list from py.mk to extmod.mk.
This separates extmod source files from `py.mk`.  Previously, `py.mk`
assumed that every consumer of the py/ directory also wanted to include
extmod/.  However, this is not the case.  For example, building mpy-cross
uses py/ but doesn't need extmod/.

This commit moves all extmod-specific items from `py.mk` to `extmod.mk` and
explicitly includes `extmod.mk` in ports that use it.

Signed-off-by: David Lechner <david@pybricks.com>
2022-06-21 00:14:34 +10:00
Damien George ac3fb974bc unix/variants: Enable seeding random module on import in dev, coverage.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-07 23:41:49 +10:00
Damien George d2f018bff1 unix,windows: Factor out code that generates random bytes to a new func.
Signed-off-by: Damien George <damien@micropython.org>
2022-06-07 23:41:30 +10:00
stijn 5bb2a85d74 windows: Use BCryptGenRandom to implement os.urandom.
Fix urandom not working on windows (there's no /dev/urandom) by using
a proper cryptographic random function (same one as CPython >= 3.11).
2022-06-07 23:32:30 +10:00
Damien George efe23aca71 all: Remove third argument to MP_REGISTER_MODULE.
It's no longer needed because this macro is now processed after
preprocessing the source code via cpp (in the qstr extraction stage), which
means unused MP_REGISTER_MODULE's are filtered out by the preprocessor.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-02 16:31:37 +10:00
Damien George 6e71cde6aa ports: Use default VFS config for import_stat and builtin_open.
For ports with MICROPY_VFS and MICROPY_PY_IO enabled their configuration
can now be simplified to use the defaults for mp_import_stat and
mp_builtin_open.

This commit makes no functional change, except for the following minor
points:
- the built-in "open" is removed from the minimal port (it previously did
  nothing)
- the duplicate built-in "input" is removed from the esp32 port
- qemu-arm now delegates to VFS import/open

Signed-off-by: Damien George <damien@micropython.org>
2022-05-25 13:04:45 +10:00
David Lechner c012318d74 unix: Implement `-X realtime` command-line option on macOS.
This adds a new command line option to the unix port `-X realtime` to
enable realtime priority on threads.  This enables high precision timers
for applications that need more accurate timers.

Related docs:
https://developer.apple.com/library/archive/technotes/tn2169/_index.html

Fixes issue #8621.

Signed-off-by: David Lechner <david@pybricks.com>
2022-05-24 00:51:47 +10:00
David Lechner be5657b64f ports: Rename thread_t to mp_thread_t.
This adds the `mp_` prefix to the `thread_t` type.  The name `thread_t`
conflicts with the same in `mach/mach_types.h` on macOS.

Signed-off-by: David Lechner <david@lechnology.com>
2022-05-24 00:51:23 +10:00
Jim Mussared 63431a5536 unix: Make port-specific modules use MP_REGISTER_MODULE.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-05-18 20:57:09 +10:00
Jim Mussared 4eab44a1ec extmod: Make extmod modules use MP_REGISTER_MODULE.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-05-18 20:49:12 +10:00
Damien George 5b700b0af9 all: Reformat remaining C code that doesn't have a space after a comma.
Signed-off-by: Damien George <damien@micropython.org>
2022-05-05 13:30:40 +10:00
Damien George fca5701f74 py/malloc: Introduce m_tracked_calloc, m_tracked_free functions.
Enabled by MICROPY_TRACKED_ALLOC.

Signed-off-by: Damien George <damien@micropython.org>
2022-05-05 10:31:50 +10:00
Jim Mussared 0e7bfc88c6 all: Use mp_obj_malloc everywhere it's applicable.
This replaces occurences of

    foo_t *foo = m_new_obj(foo_t);
    foo->base.type = &foo_type;

with

    foo_t *foo = mp_obj_malloc(foo_t, &foo_type);

Excludes any places where base is a sub-field or when new0/memset is used.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-05-03 22:28:14 +10:00
Damien George 402df833fe py/modsys: Introduce sys.implementation._machine constant.
This contains a string useful for identifying the underlying machine.  This
string is kept consistent with the second part of the REPL banner via the
new config option MICROPY_BANNER_MACHINE.

This makes os.uname() more or less redundant, as all the information in
os.uname() is now available in the sys module.

Signed-off-by: Damien George <damien@micropython.org>
2022-04-28 17:23:03 +10:00
Damien George 40047823bc py/modsys: Append MicroPython git version and build date to sys.version.
This commit adds the git hash and build date to sys.version.  This is
allowed according to CPython docs, and is what PyPy does.  The docs state:

    A string containing the version number of the Python interpreter plus
    additional information on the build number and compiler used.

Eg on CPython:

    Python 3.10.4 (main, Mar 23 2022, 23:05:40) [GCC 11.2.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import sys
    >>> sys.version
    '3.10.4 (main, Mar 23 2022, 23:05:40) [GCC 11.2.0]'

and PyPy:

    Python 2.7.12 (5.6.0+dfsg-4, Nov 20 2016, 10:43:30)
    [PyPy 5.6.0 with GCC 6.2.0 20161109] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>>> import sys
    >>>> sys.version
    '2.7.12 (5.6.0+dfsg-4, Nov 20 2016, 10:43:30)\n[PyPy 5.6.0 with GCC ...

With this commit on MicroPython we now have:

    MicroPython v1.18-371-g9d08eb024 on 2022-04-28; linux [GCC 11.2.0] v...
    Use Ctrl-D to exit, Ctrl-E for paste mode
    >>> import sys
    >>> sys.version
    '3.4.0; MicroPython v1.18-371-g9d08eb024 on 2022-04-28'

Note that the start of the banner is the same as the end of sys.version.
This helps to keep code size under control because the string can be reused
by the compiler.

Signed-off-by: Damien George <damien@micropython.org>
2022-04-28 15:23:17 +10:00
Damien George 5c32111fa0 ports: Reformat more C and Python source code.
These files that are reformatted only now fall under the list of files to
apply uncrustify/black formatting to.

Signed-off-by: Damien George <damien@micropython.org>
2022-04-20 16:17:35 +10:00
Damien George aab005c75b extmod/modusocket: Provide config macro for socket.listen backlog deflt.
To make it possible to change this value for any given port or board.

Signed-off-by: Damien George <damien@micropython.org>
2022-04-11 15:28:56 +10:00
Jon Bjarni Bjarnason 919f696ad2 extmod/modusocket: Implement optional socket.listen backlog argument.
This follows the CPython change: https://bugs.python.org/issue21455

Socket listen backlog defaults to 2 if not given, based on most bare metal
targets not having many resources for a large backlog.  On UNIX it defaults
to SOMAXCONN or 128, whichever is less.
2022-04-11 15:26:47 +10:00
Damien George eec07332b1 unix/variants: Enable a few optimisations and features on dev, coverage.
Signed-off-by: Damien George <damien@micropython.org>
2022-03-10 13:34:43 +11:00
Damien George 3c20ddb41a unix/variants: Use rom feature config for standard, dev, coverage.
This change is a no-op in terms of functionality.

Signed-off-by: Damien George <damien@micropython.org>
2022-03-10 13:32:25 +11:00
Damien George 0ac3191d8c unix/mpconfigport.h: Collect together config options from extra level.
This change is a no-op in terms of functionality.

Signed-off-by: Damien George <damien@micropython.org>
2022-03-10 13:29:13 +11:00
Damien George ac2293161e py/modsys: Add optional mutable attributes sys.ps1/ps2 and use them.
This allows customising the REPL prompt strings.

Signed-off-by: Damien George <damien@micropython.org>
2022-03-10 10:58:33 +11:00
Damien George cac939ddc3 py/modsys: Add optional sys.tracebacklimit attribute.
With behaviour as per CPython.

Signed-off-by: Damien George <damien@micropython.org>
2022-03-10 10:43:21 +11:00
Damien George 2b409ef8a4 unix/moduos: Convert module to use extmod version.
All variants now use extmod/moduos.c as their uos module implementation.
In particular this means they all have MICROPY_VFS enabled and use VfsPosix
for their filesystem.

As part of this, the available functions in uos become more consistent with
other ports:
- coverage variant gets uos.urandom
- minimal and standard variant get: unlink, chdir, getcwd, listdir

Signed-off-by: Damien George <damien@micropython.org>
2022-03-09 21:13:57 +11:00
Damien George 3c2aa5ff93 unix/.gitignore: Remove *.py from ignore list.
.py files are valid source files and shouldn't be ignored.  This line was
from the early days when .py files in the unix directory were used for
testing.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-28 19:02:58 +11:00
Damien George c4b8dae438 tests/unix: Add coverage test for freezing various objects.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-28 19:02:58 +11:00
Damien George f2040bfc7e py: Rework bytecode and .mpy file format to be mostly static data.
Background: .mpy files are precompiled .py files, built using mpy-cross,
that contain compiled bytecode functions (and can also contain machine
code). The benefit of using an .mpy file over a .py file is that they are
faster to import and take less memory when importing.  They are also
smaller on disk.

But the real benefit of .mpy files comes when they are frozen into the
firmware.  This is done by loading the .mpy file during compilation of the
firmware and turning it into a set of big C data structures (the job of
mpy-tool.py), which are then compiled and downloaded into the ROM of a
device.  These C data structures can be executed in-place, ie directly from
ROM.  This makes importing even faster because there is very little to do,
and also means such frozen modules take up much less RAM (because their
bytecode stays in ROM).

The downside of frozen code is that it requires recompiling and reflashing
the entire firmware.  This can be a big barrier to entry, slows down
development time, and makes it harder to do OTA updates of frozen code
(because the whole firmware must be updated).

This commit attempts to solve this problem by providing a solution that
sits between loading .mpy files into RAM and freezing them into the
firmware.  The .mpy file format has been reworked so that it consists of
data and bytecode which is mostly static and ready to run in-place.  If
these new .mpy files are located in flash/ROM which is memory addressable,
the .mpy file can be executed (mostly) in-place.

With this approach there is still a small amount of unpacking and linking
of the .mpy file that needs to be done when it's imported, but it's still
much better than loading an .mpy from disk into RAM (although not as good
as freezing .mpy files into the firmware).

The main trick to make static .mpy files is to adjust the bytecode so any
qstrs that it references now go through a lookup table to convert from
local qstr number in the module to global qstr number in the firmware.
That means the bytecode does not need linking/rewriting of qstrs when it's
loaded.  Instead only a small qstr table needs to be built (and put in RAM)
at import time.  This means the bytecode itself is static/constant and can
be used directly if it's in addressable memory.  Also the qstr string data
in the .mpy file, and some constant object data, can be used directly.
Note that the qstr table is global to the module (ie not per function).

In more detail, in the VM what used to be (schematically):

    qst = DECODE_QSTR_VALUE;

is now (schematically):

    idx = DECODE_QSTR_INDEX;
    qst = qstr_table[idx];

That allows the bytecode to be fixed at compile time and not need
relinking/rewriting of the qstr values.  Only qstr_table needs to be linked
when the .mpy is loaded.

Incidentally, this helps to reduce the size of bytecode because what used
to be 2-byte qstr values in the bytecode are now (mostly) 1-byte indices.
If the module uses the same qstr more than two times then the bytecode is
smaller than before.

The following changes are measured for this commit compared to the
previous (the baseline):
- average 7%-9% reduction in size of .mpy files
- frozen code size is reduced by about 5%-7%
- importing .py files uses about 5% less RAM in total
- importing .mpy files uses about 4% less RAM in total
- importing .py and .mpy files takes about the same time as before

The qstr indirection in the bytecode has only a small impact on VM
performance.  For stm32 on PYBv1.0 the performance change of this commit
is:

diff of scores (higher is better)
N=100 M=100             baseline -> this-commit  diff      diff% (error%)
bm_chaos.py               371.07 ->  357.39 :  -13.68 =  -3.687% (+/-0.02%)
bm_fannkuch.py             78.72 ->   77.49 :   -1.23 =  -1.563% (+/-0.01%)
bm_fft.py                2591.73 -> 2539.28 :  -52.45 =  -2.024% (+/-0.00%)
bm_float.py              6034.93 -> 5908.30 : -126.63 =  -2.098% (+/-0.01%)
bm_hexiom.py               48.96 ->   47.93 :   -1.03 =  -2.104% (+/-0.00%)
bm_nqueens.py            4510.63 -> 4459.94 :  -50.69 =  -1.124% (+/-0.00%)
bm_pidigits.py            650.28 ->  644.96 :   -5.32 =  -0.818% (+/-0.23%)
core_import_mpy_multi.py  564.77 ->  581.49 :  +16.72 =  +2.960% (+/-0.01%)
core_import_mpy_single.py  68.67 ->   67.16 :   -1.51 =  -2.199% (+/-0.01%)
core_qstr.py               64.16 ->   64.12 :   -0.04 =  -0.062% (+/-0.00%)
core_yield_from.py        362.58 ->  354.50 :   -8.08 =  -2.228% (+/-0.00%)
misc_aes.py               429.69 ->  405.59 :  -24.10 =  -5.609% (+/-0.01%)
misc_mandel.py           3485.13 -> 3416.51 :  -68.62 =  -1.969% (+/-0.00%)
misc_pystone.py          2496.53 -> 2405.56 :  -90.97 =  -3.644% (+/-0.01%)
misc_raytrace.py          381.47 ->  374.01 :   -7.46 =  -1.956% (+/-0.01%)
viper_call0.py            576.73 ->  572.49 :   -4.24 =  -0.735% (+/-0.04%)
viper_call1a.py           550.37 ->  546.21 :   -4.16 =  -0.756% (+/-0.09%)
viper_call1b.py           438.23 ->  435.68 :   -2.55 =  -0.582% (+/-0.06%)
viper_call1c.py           442.84 ->  440.04 :   -2.80 =  -0.632% (+/-0.08%)
viper_call2a.py           536.31 ->  532.35 :   -3.96 =  -0.738% (+/-0.06%)
viper_call2b.py           382.34 ->  377.07 :   -5.27 =  -1.378% (+/-0.03%)

And for unix on x64:

diff of scores (higher is better)
N=2000 M=2000        baseline -> this-commit     diff      diff% (error%)
bm_chaos.py          13594.20 ->  13073.84 :  -520.36 =  -3.828% (+/-5.44%)
bm_fannkuch.py          60.63 ->     59.58 :    -1.05 =  -1.732% (+/-3.01%)
bm_fft.py           112009.15 -> 111603.32 :  -405.83 =  -0.362% (+/-4.03%)
bm_float.py         246202.55 -> 247923.81 : +1721.26 =  +0.699% (+/-2.79%)
bm_hexiom.py           615.65 ->    617.21 :    +1.56 =  +0.253% (+/-1.64%)
bm_nqueens.py       215807.95 -> 215600.96 :  -206.99 =  -0.096% (+/-3.52%)
bm_pidigits.py        8246.74 ->   8422.82 :  +176.08 =  +2.135% (+/-3.64%)
misc_aes.py          16133.00 ->  16452.74 :  +319.74 =  +1.982% (+/-1.50%)
misc_mandel.py      128146.69 -> 130796.43 : +2649.74 =  +2.068% (+/-3.18%)
misc_pystone.py      83811.49 ->  83124.85 :  -686.64 =  -0.819% (+/-1.03%)
misc_raytrace.py     21688.02 ->  21385.10 :  -302.92 =  -1.397% (+/-3.20%)

The code size change is (firmware with a lot of frozen code benefits the
most):

       bare-arm:  +396 +0.697%
    minimal x86: +1595 +0.979% [incl +32(data)]
       unix x64: +2408 +0.470% [incl +800(data)]
    unix nanbox: +1396 +0.309% [incl -96(data)]
          stm32: -1256 -0.318% PYBV10
         cc3200:  +288 +0.157%
        esp8266:  -260 -0.037% GENERIC
          esp32:  -216 -0.014% GENERIC[incl -1072(data)]
            nrf:  +116 +0.067% pca10040
            rp2:  -664 -0.135% PICO
           samd:  +844 +0.607% ADAFRUIT_ITSYBITSY_M4_EXPRESS

As part of this change the .mpy file format version is bumped to version 6.
And mpy-tool.py has been improved to provide a good visualisation of the
contents of .mpy files.

In summary: this commit changes the bytecode to use qstr indirection, and
reworks the .mpy file format to be simpler and allow .mpy files to be
executed in-place.  Performance is not impacted too much.  Eventually it
will be possible to store such .mpy files in a linear, read-only, memory-
mappable filesystem so they can be executed from flash/ROM.  This will
essentially be able to replace frozen code for most applications.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-24 18:08:43 +11:00
stijn 5f50f4a130 unix: Show compiler info in REPL banner.
The unix port's main.c gets used by unix and windows ports, and with a
variety of compilers, so it's convenient to see which version is actually
being used immediately when starting micropython.  This is similar to what
CPython does.
2022-02-22 00:59:31 +11:00
stijn 8bb50c6301 unix/Makefile: Remove explicit addition of -std=c++ flag.
This was added merely for building the C++ user module example, so it's a
better fit to add it in the corresponding micropython.mk.
2022-02-18 14:40:16 +11:00
Damien George fbd47fc46c ports: Consolidate inclusion of umachine module in built-ins.
The inclusion of `umachine` in the list of built-in modules is now done
centrally in py/objmodule.c.  Enabling MICROPY_PY_MACHINE will include this
module.

As part of this, all ports now have `umachine` as the core module name
(previously some had only `machine` as the name).

Signed-off-by: Damien George <damien@micropython.org>
2022-02-03 10:08:54 +11:00
stijn dd6967202a py/modmath: Add math.tau, math.nan and math.inf constants.
Configurable by the new MICROPY_PY_MATH_CONSTANTS option.
2022-01-23 09:28:33 +11:00
Damien George aac5a97d08 ports: Move '.frozen' to second entry in sys.path.
In commit 86ce442607 the '.frozen' entry was
added at the start of sys.path, to allow control over when frozen modules
are searched during import, and retain existing behaviour whereby frozen
was searched before the filesystem.

But Python semantics of sys.path require sys.path[0] to be the directory of
the currently executing script, or ''.

This commit moves the '.frozen' entry to second place in sys.path, so
sys.path[0] retains its correct value (described above).

Signed-off-by: Damien George <damien@micropython.org>
2021-12-29 23:55:36 +11:00
Damien George de43b500bd py/runtime: Allow initialising sys.path/argv with defaults.
If MICROPY_PY_SYS_PATH_ARGV_DEFAULTS is enabled (which it is by default)
then sys.path and sys.argv will be initialised and populated with default
values.  This keeps all bare-metal ports aligned.

Signed-off-by: Damien George <damien@micropython.org>
2021-12-18 00:08:07 +11:00
Jim Mussared 86ce442607 ports: Add '.frozen' as the first entry in sys.path.
Frozen modules will be searched preferentially, but gives the user the
ability to override this behavior.

This matches the previous behavior where "" was implicitly the frozen
search path, but the frozen list was checked before the filesystem.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-12-18 00:08:07 +11:00