mirror
/
micropython
kopia lustrzana https://github.com/micropython/micropython
1
0
Forkuj 0
Kod Zgłoszenia Projekty Wydania Wiki Aktywność
micropython/py/mpstate.h

305 wiersze
9.2 KiB
C
Czysty Zwykły widok Historia

py: Put all global state together in state structures. This patch consolidates all global variables in py/ core into one place, in a global structure. Root pointers are all located together to make GC tracing easier and more efficient.
2015-01-01 23:30:53 +00:00
/*
all: Use the name MicroPython consistently in comments There were several different spellings of MicroPython present in comments, when there should be only one.
2017-06-30 07:22:17 +00:00
* This file is part of the MicroPython project, http://micropython.org/
py: Put all global state together in state structures. This patch consolidates all global variables in py/ core into one place, in a global structure. Root pointers are all located together to make GC tracing easier and more efficient.
2015-01-01 23:30:53 +00:00
*
* The MIT License (MIT)
*
* Copyright (c) 2014 Damien P. George
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to deal
* in the Software without restriction, including without limitation the rights
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*/
all: Unify header guard usage. The code conventions suggest using header guards, but do not define how those should look like and instead point to existing files. However, not all existing files follow the same scheme, sometimes omitting header guards altogether, sometimes using non-standard names, making it easy to accidentally pick a "wrong" example. This commit ensures that all header files of the MicroPython project (that were not simply copied from somewhere else) follow the same pattern, that was already present in the majority of files, especially in the py folder. The rules are as follows. Naming convention: * start with the words MICROPY_INCLUDED * contain the full path to the file * replace special characters with _ In addition, there are no empty lines before #ifndef, between #ifndef and one empty line before #endif. #endif is followed by a comment containing the name of the guard macro. py/grammar.h cannot use header guards by design, since it has to be included multiple times in a single C file. Several other files also do not need header guards as they are only used internally and guaranteed to be included only once: * MICROPY_MPHALPORT_H * mpconfigboard.h * mpconfigport.h * mpthreadport.h * pin_defs_*.h * qstrdefs*.h
2017-06-29 21:14:58 +00:00
#ifndef MICROPY_INCLUDED_PY_MPSTATE_H
#define MICROPY_INCLUDED_PY_MPSTATE_H
py: Put all global state together in state structures. This patch consolidates all global variables in py/ core into one place, in a global structure. Root pointers are all located together to make GC tracing easier and more efficient.
2015-01-01 23:30:53 +00:00
#include <stdint.h>
#include "py/mpconfig.h"
py/gc: Make memory manager and garbage collector thread safe. By using a single, global mutex, all memory-related functions (alloc, free, realloc, collect, etc) are made thread safe. This means that only one thread can be in such a function at any one time.
2016-04-25 15:28:57 +00:00
#include "py/mpthread.h"
py: Put all global state together in state structures. This patch consolidates all global variables in py/ core into one place, in a global structure. Root pointers are all located together to make GC tracing easier and more efficient.
2015-01-01 23:30:53 +00:00
#include "py/misc.h"
#include "py/nlr.h"
#include "py/obj.h"
py: Put mp_sys_path, mp_sys_argv and gc_collected in mp_state_ctx_t. Without mp_sys_path and mp_sys_argv in the root pointer section of the state, their memory was being incorrectly collected by GC.
2015-02-07 17:24:10 +00:00
#include "py/objlist.h"
py: Put all global state together in state structures. This patch consolidates all global variables in py/ core into one place, in a global structure. Root pointers are all located together to make GC tracing easier and more efficient.
2015-01-01 23:30:53 +00:00
#include "py/objexcept.h"
all: Use the name MicroPython consistently in comments There were several different spellings of MicroPython present in comments, when there should be only one.
2017-06-30 07:22:17 +00:00
// This file contains structures defining the state of the MicroPython
py: Put all global state together in state structures. This patch consolidates all global variables in py/ core into one place, in a global structure. Root pointers are all located together to make GC tracing easier and more efficient.
2015-01-01 23:30:53 +00:00
// memory system, runtime and virtual machine. The state is a global
// variable, but in the future it is hoped that the state can become local.
py: Add MICROPY_DYNAMIC_COMPILER option to config compiler at runtime. This new compile-time option allows to make the bytecode compiler configurable at runtime by setting the fields in the mp_dynamic_compiler structure. By using this feature, the compiler can generate bytecode that targets any MicroPython runtime/VM, regardless of the host and target compile-time settings. Options so far that fall under this dynamic setting are: - maximum number of bits that a small int can hold; - whether caching of lookups is used in the bytecode; - whether to use unicode strings or not (lexer behaviour differs, and therefore generated string constants differ).
2016-02-11 22:30:53 +00:00
// This structure contains dynamic configuration for the compiler.
#if MICROPY_DYNAMIC_COMPILER
typedef struct mp_dynamic_compiler_t {
uint8_t small_int_bits; // must be <= host small_int_bits
bool py_builtins_str_unicode;
py/compile: Add support to select the native emitter at runtime.
2019-03-08 23:59:25 +00:00
uint8_t native_arch;
mpy-cross: Set number of registers in nlr_buf_t based on native arch. Fixes #5059. Done in collaboration with Jim Mussared.
2019-09-18 03:45:20 +00:00
uint8_t nlr_buf_num_regs;
py: Add MICROPY_DYNAMIC_COMPILER option to config compiler at runtime. This new compile-time option allows to make the bytecode compiler configurable at runtime by setting the fields in the mp_dynamic_compiler structure. By using this feature, the compiler can generate bytecode that targets any MicroPython runtime/VM, regardless of the host and target compile-time settings. Options so far that fall under this dynamic setting are: - maximum number of bits that a small int can hold; - whether caching of lookups is used in the bytecode; - whether to use unicode strings or not (lexer behaviour differs, and therefore generated string constants differ).
2016-02-11 22:30:53 +00:00
} mp_dynamic_compiler_t;
extern mp_dynamic_compiler_t mp_dynamic_compiler;
#endif
py: Add micropython.schedule() function and associated runtime code.
2017-02-16 07:05:06 +00:00
// These are the values for sched_state
#define MP_SCHED_IDLE (1)
#define MP_SCHED_LOCKED (-1)
#define MP_SCHED_PENDING (0) // 0 so it's a quick check in the VM
typedef struct _mp_sched_item_t {
mp_obj_t func;
mp_obj_t arg;
} mp_sched_item_t;
py: Put all global state together in state structures. This patch consolidates all global variables in py/ core into one place, in a global structure. Root pointers are all located together to make GC tracing easier and more efficient.
2015-01-01 23:30:53 +00:00
// This structure hold information about the memory allocation system.
typedef struct _mp_state_mem_t {
#if MICROPY_MEM_STATS
size_t total_bytes_allocated;
size_t current_bytes_allocated;
size_t peak_bytes_allocated;
#endif
byte *gc_alloc_table_start;
py/gc: Use size_t instead of mp_uint_t to count things related to heap. size_t is the correct type to use to count things related to the size of the address space. Using size_t (instead of mp_uint_t) is important for the efficiency of ports that configure mp_uint_t to larger than the machine word size.
2015-12-17 01:09:11 +00:00
size_t gc_alloc_table_byte_len;
py: Put all global state together in state structures. This patch consolidates all global variables in py/ core into one place, in a global structure. Root pointers are all located together to make GC tracing easier and more efficient.
2015-01-01 23:30:53 +00:00
#if MICROPY_ENABLE_FINALISER
byte *gc_finaliser_table_start;
#endif
py/gc: Move away from using mp_uint_t, instead use uintptr_t and size_t. The GC works with concrete pointers and so the types should reflect this.
2015-11-27 13:07:48 +00:00
byte *gc_pool_start;
byte *gc_pool_end;
py: Put all global state together in state structures. This patch consolidates all global variables in py/ core into one place, in a global structure. Root pointers are all located together to make GC tracing easier and more efficient.
2015-01-01 23:30:53 +00:00
int gc_stack_overflow;
py: Add option to reduce GC stack integer size to save RAM. A new option MICROPY_GC_STACK_ENTRY_TYPE is added to select a custom type instead of size_t for the gc_stack array items. This can be beneficial for small devices, especially those that are low on memory anyway. If a device has 1MB or less of heap (and 16-byte GC blocks) then this type can be uint16_t, saving 128 bytes of RAM.
2018-02-23 17:59:31 +00:00
MICROPY_GC_STACK_ENTRY_TYPE gc_stack[MICROPY_ALLOC_GC_STACK_SIZE];
py: Put all global state together in state structures. This patch consolidates all global variables in py/ core into one place, in a global structure. Root pointers are all located together to make GC tracing easier and more efficient.
2015-01-01 23:30:53 +00:00
// This variable controls auto garbage collection. If set to 0 then the
// GC won't automatically run when gc_alloc can't find enough blocks. But
// you can still allocate/free memory and also explicitly call gc_collect.
uint16_t gc_auto_collect_enabled;
py/gc: Implement GC running by allocation threshold. Currently, MicroPython runs GC when it could not allocate a block of memory, which happens when heap is exhausted. However, that policy can't work well with "inifinity" heaps, e.g. backed by a virtual memory - there will be a lot of swap thrashing long before VM will be exhausted. Instead, in such cases "allocation threshold" policy is used: a GC is run after some number of allocations have been made. Details vary, for example, number or total amount of allocations can be used, threshold may be self-adjusting based on GC outcome, etc. This change implements a simple variant of such policy for MicroPython. Amount of allocated memory so far is used for threshold, to make it useful to typical finite-size, and small, heaps as used with MicroPython ports. And such GC policy is indeed useful for such types of heaps too, as it allows to better control fragmentation. For example, if a threshold is set to half size of heap, then for an application which usually makes big number of small allocations, that will (try to) keep half of heap memory in a nice defragmented state for an occasional large allocation. For an application which doesn't exhibit such behavior, there won't be any visible effects, except for GC running more frequently, which however may affect performance. To address this, the GC threshold is configurable, and by default is off so far. It's configured with gc.threshold(amount_in_bytes) call (can be queries without an argument).
2016-07-20 21:37:30 +00:00
#if MICROPY_GC_ALLOC_THRESHOLD
size_t gc_alloc_amount;
size_t gc_alloc_threshold;
#endif
py/gc: Use size_t instead of mp_uint_t to count things related to heap. size_t is the correct type to use to count things related to the size of the address space. Using size_t (instead of mp_uint_t) is important for the efficiency of ports that configure mp_uint_t to larger than the machine word size.
2015-12-17 01:09:11 +00:00
size_t gc_last_free_atb_index;
py: Put mp_sys_path, mp_sys_argv and gc_collected in mp_state_ctx_t. Without mp_sys_path and mp_sys_argv in the root pointer section of the state, their memory was being incorrectly collected by GC.
2015-02-07 17:24:10 +00:00
#if MICROPY_PY_GC_COLLECT_RETVAL
py/gc: Use size_t instead of mp_uint_t to count things related to heap. size_t is the correct type to use to count things related to the size of the address space. Using size_t (instead of mp_uint_t) is important for the efficiency of ports that configure mp_uint_t to larger than the machine word size.
2015-12-17 01:09:11 +00:00
size_t gc_collected;
py: Put mp_sys_path, mp_sys_argv and gc_collected in mp_state_ctx_t. Without mp_sys_path and mp_sys_argv in the root pointer section of the state, their memory was being incorrectly collected by GC.
2015-02-07 17:24:10 +00:00
#endif
py/gc: Make memory manager and garbage collector thread safe. By using a single, global mutex, all memory-related functions (alloc, free, realloc, collect, etc) are made thread safe. This means that only one thread can be in such a function at any one time.
2016-04-25 15:28:57 +00:00
py/gc: Don't include or init gc_mutex when GIL is enabled. When threads and the GIL are enabled, then the GC mutex is not needed. The gc_mutex field is never used in this case because of: #if MICROPY_PY_THREAD && !MICROPY_PY_THREAD_GIL #define GC_ENTER() mp_thread_mutex_lock(&MP_STATE_MEM(gc_mutex), 1) #define GC_EXIT() mp_thread_mutex_unlock(&MP_STATE_MEM(gc_mutex)) #else #define GC_ENTER() #define GC_EXIT() #endif So, we can completely remove gc_mutex everywhere when MICROPY_PY_THREAD && !MICROPY_PY_THREAD_GIL.
2020-01-22 17:19:37 +00:00
#if MICROPY_PY_THREAD && !MICROPY_PY_THREAD_GIL
py/gc: Make memory manager and garbage collector thread safe. By using a single, global mutex, all memory-related functions (alloc, free, realloc, collect, etc) are made thread safe. This means that only one thread can be in such a function at any one time.
2016-04-25 15:28:57 +00:00
// This is a global mutex used to make the GC thread-safe.
mp_thread_mutex_t gc_mutex;
#endif
py: Put all global state together in state structures. This patch consolidates all global variables in py/ core into one place, in a global structure. Root pointers are all located together to make GC tracing easier and more efficient.
2015-01-01 23:30:53 +00:00
} mp_state_mem_t;
// This structure hold runtime and VM information. It includes a section
// which contains root pointers that must be scanned by the GC.
typedef struct _mp_state_vm_t {
py/mpstate.h: Adjust start of root pointer section to exclude non-ptrs. This patch moves the start of the root pointer section in mp_state_ctx_t so that it skips entries that are not pointers and don't need scanning. Previously, the start of the root pointer section was at the very beginning of the mp_state_ctx_t struct (which is the beginning of mp_state_thread_t). This was the original assembler version of the NLR code was hard-coded to have the nlr_top pointer at the start of this state structure. But now that the NLR code is partially written in C there is no longer this restriction on the location of nlr_top (and a comment to this effect has been removed in this patch). So now the root pointer section starts part way through the mp_state_thread_t structure, after the entries which are not root pointers. This patch also moves the non-pointer entries for MICROPY_ENABLE_SCHEDULER outside the root pointer section. Moving non-pointer entries out of the root pointer section helps to make the GC more precise and should help to prevent some cases of collectable garbage being kept. This patch also has a measurable improvement in performance of the pystone.py benchmark: on unix x86-64 and stm32 there was an improvement of roughly 0.6% (tested with both gcc 7.3 and gcc 8.1).
2018-05-12 12:09:34 +00:00
//
// CONTINUE ROOT POINTER SECTION
// This must start at the start of this structure and follows
// the state in the mp_state_thread_t structure, continuing
// the root pointer section from there.
py: Put all global state together in state structures. This patch consolidates all global variables in py/ core into one place, in a global structure. Root pointers are all located together to make GC tracing easier and more efficient.
2015-01-01 23:30:53 +00:00
//
qstr_pool_t *last_pool;
// non-heap memory for creating an exception if we can't allocate RAM
mp_obj_exception_t mp_emergency_exception_obj;
// memory for exception arguments if we can't allocate RAM
#if MICROPY_ENABLE_EMERGENCY_EXCEPTION_BUF
#if MICROPY_EMERGENCY_EXCEPTION_BUF_SIZE > 0
py: Make sure that static emg-exc-buffer is aligned to size of mp_obj_t. This buffer is used to allocate objects temporarily, and such objects require that their underlying memory be correctly aligned for their data type. Aligning for mp_obj_t should be sufficient for emergency exceptions, but in general the memory buffer should aligned to the maximum alignment of the machine (eg on a 32-bit machine with mp_obj_t being 4 bytes, a double may not be correctly aligned). This patch fixes a bug for certain nan-boxing builds, where mp_obj_t is 8 bytes and must be aligned to 8 bytes (even though the machine is 32 bit).
2017-04-10 06:02:56 +00:00
// statically allocated buf (needs to be aligned to mp_obj_t)
mp_obj_t mp_emergency_exception_buf[MICROPY_EMERGENCY_EXCEPTION_BUF_SIZE / sizeof(mp_obj_t)];
py: Put all global state together in state structures. This patch consolidates all global variables in py/ core into one place, in a global structure. Root pointers are all located together to make GC tracing easier and more efficient.
2015-01-01 23:30:53 +00:00
#else
// dynamically allocated buf
byte *mp_emergency_exception_buf;
#endif
#endif
py: Add MICROPY_KBD_EXCEPTION config option to provide mp_kbd_exception. Defining and initialising mp_kbd_exception is boiler-plate code and so the core runtime can provide it, instead of each port needing to do it themselves. The exception object is placed in the VM state rather than on the heap.
2016-12-15 02:00:19 +00:00
#if MICROPY_KBD_EXCEPTION
// exception object of type KeyboardInterrupt
mp_obj_exception_t mp_kbd_exception;
#endif
py/modsys: Implement sys.modules. This for example will allow people to reload modules which didn't load successfully (e.g. due to syntax error).
2015-12-04 22:09:10 +00:00
// dictionary with loaded modules (may be exposed as sys.modules)
mp_obj_dict_t mp_loaded_modules_dict;
py: Put all global state together in state structures. This patch consolidates all global variables in py/ core into one place, in a global structure. Root pointers are all located together to make GC tracing easier and more efficient.
2015-01-01 23:30:53 +00:00
py: Add micropython.schedule() function and associated runtime code.
2017-02-16 07:05:06 +00:00
#if MICROPY_ENABLE_SCHEDULER
py/scheduler: Rename sched_stack to sched_queue. Behaviour was changed from stack to queue in 8977c7eb581f5d06500edb1ea29aea5cbda04f28, and this updates variable names to match. Also updates other references (docs, error messages).
2019-07-11 01:55:31 +00:00
mp_sched_item_t sched_queue[MICROPY_SCHEDULER_DEPTH];
py: Add micropython.schedule() function and associated runtime code.
2017-02-16 07:05:06 +00:00
#endif
modsys: Add basic sys.exc_info() implementation. The implementation is very basic and non-compliant and provided solely for CPython compatibility. The function itself is bad Python2 heritage, its usage is discouraged.
2015-04-25 00:17:41 +00:00
// current exception being handled, for sys.exc_info()
#if MICROPY_PY_SYS_EXC_INFO
py: Wrap all obj-ptr conversions in MP_OBJ_TO_PTR/MP_OBJ_FROM_PTR. This allows the mp_obj_t type to be configured to something other than a pointer-sized primitive type. This patch also includes additional changes to allow the code to compile when sizeof(mp_uint_t) != sizeof(void*), such as using size_t instead of mp_uint_t, and various casts.
2015-11-27 17:01:44 +00:00
mp_obj_base_t *cur_exception;
modsys: Add basic sys.exc_info() implementation. The implementation is very basic and non-compliant and provided solely for CPython compatibility. The function itself is bad Python2 heritage, its usage is discouraged.
2015-04-25 00:17:41 +00:00
#endif
py: Implement new sys.atexit feature. This patch implements a new sys.atexit function which registers a function that is later executed when the main script ends. It is configurable via MICROPY_PY_SYS_ATEXIT, disabled by default. This is not compliant with CPython, rather it can be used to implement a CPython compatible "atexit" module if desired (similar to how sys.print_exception can be used to implement functionality of the "traceback" module).
2019-08-05 13:06:41 +00:00
#if MICROPY_PY_SYS_ATEXIT
// exposed through sys.atexit function
mp_obj_t sys_exitfunc;
#endif
py: Put all global state together in state structures. This patch consolidates all global variables in py/ core into one place, in a global structure. Root pointers are all located together to make GC tracing easier and more efficient.
2015-01-01 23:30:53 +00:00
// dictionary for the __main__ module
mp_obj_dict_t dict_main;
py: Put mp_sys_path, mp_sys_argv and gc_collected in mp_state_ctx_t. Without mp_sys_path and mp_sys_argv in the root pointer section of the state, their memory was being incorrectly collected by GC.
2015-02-07 17:24:10 +00:00
// these two lists must be initialised per port, after the call to mp_init
mp_obj_list_t mp_sys_path_obj;
mp_obj_list_t mp_sys_argv_obj;
py: Put all global state together in state structures. This patch consolidates all global variables in py/ core into one place, in a global structure. Root pointers are all located together to make GC tracing easier and more efficient.
2015-01-01 23:30:53 +00:00
// dictionary for overridden builtins
#if MICROPY_CAN_OVERRIDE_BUILTINS
mp_obj_dict_t *mp_module_builtins_override_dict;
#endif
py/persistentcode: Maintain root ptr list of imported native .mpy code. On ports where normal heap memory can contain executable code (eg ARM-based ports such as stm32), native code loaded from an .mpy file may be reclaimed by the GC because there's no reference to the very start of the native machine code block that is reachable from root pointers (only pointers to internal parts of the machine code block are reachable, but that doesn't help the GC find the memory). This commit fixes this issue by maintaining an explicit list of root pointers pointing to native code that is loaded from an .mpy file. This is not needed for all ports so is selectable by the new configuration option MICROPY_PERSISTENT_CODE_TRACK_RELOC_CODE. It's enabled by default if a port does not specify any special functions to allocate or commit executable memory. A test is included to test that native code loaded from an .mpy file does not get reclaimed by the GC. Fixes #6045. Signed-off-by: Damien George <damien@micropython.org>
2020-07-27 13:52:38 +00:00
#if MICROPY_PERSISTENT_CODE_TRACK_RELOC_CODE
// An mp_obj_list_t that tracks relocated native code to prevent the GC from reclaiming them.
mp_obj_t track_reloc_code_list;
#endif
py: Put all global state together in state structures. This patch consolidates all global variables in py/ core into one place, in a global structure. Root pointers are all located together to make GC tracing easier and more efficient.
2015-01-01 23:30:53 +00:00
// include any root pointers defined by a port
MICROPY_PORT_ROOT_POINTERS
extmod/modlwip: slip: Use stream protocol and be port-independent. Based on the original patch by Galen Hazelwood: https://github.com/micropython/micropython/pull/1517 .
2015-10-26 21:04:33 +00:00
// root pointers for extmod
py: Clean up instantiation of dupterm object. To comply with already established scheme for extmod's.
2016-01-01 12:19:26 +00:00
py/mpstate.h: Add repl_line state for MICROPY_REPL_EVENT_DRIVEN.
2018-02-26 05:08:58 +00:00
#if MICROPY_REPL_EVENT_DRIVEN
vstr_t *repl_line;
#endif
py: Clean up instantiation of dupterm object. To comply with already established scheme for extmod's.
2016-01-01 12:19:26 +00:00
#if MICROPY_PY_OS_DUPTERM
extmod/uos_dupterm: Update uos.dupterm() and helper funcs to have index. The uos.dupterm() signature and behaviour is updated to reflect the latest enhancements in the docs. It has minor backwards incompatibility in that it no longer accepts zero arguments. The dupterm_rx helper function is moved from esp8266 to extmod and generalised to support multiple dupterm slots. A port can specify multiple slots by defining the MICROPY_PY_OS_DUPTERM config macro to an integer, being the number of slots it wants to have; 0 means to disable the dupterm feature altogether. The unix and esp8266 ports are updated to work with the new interface and are otherwise unchanged with respect to functionality.
2017-10-13 09:01:57 +00:00
mp_obj_t dupterm_objs[MICROPY_PY_OS_DUPTERM];
py: Clean up instantiation of dupterm object. To comply with already established scheme for extmod's.
2016-01-01 12:19:26 +00:00
#endif
extmod/modlwip: slip: Use stream protocol and be port-independent. Based on the original patch by Galen Hazelwood: https://github.com/micropython/micropython/pull/1517 .
2015-10-26 21:04:33 +00:00
#if MICROPY_PY_LWIP_SLIP
mp_obj_t lwip_slip_stream;
#endif
extmod: Add generic VFS sub-system. This provides mp_vfs_XXX functions (eg mount, open, listdir) which are agnostic to the underlying filesystem type, and just require an object with the relevant filesystem-like methods (eg .mount, .open, .listidr) which can then be mounted. These mp_vfs_XXX functions would typically be used by a port to implement the "uos" module, and mp_vfs_open would be the builtin open function. This feature is controlled by MICROPY_VFS, disabled by default.
2017-01-27 04:10:09 +00:00
#if MICROPY_VFS
extmod/vfs: Expose mp_vfs_mount_t type. It should only be used for low-level things and with caution, for example putting mounted VFS data in ROM or the static data section.
2017-01-27 11:40:15 +00:00
struct _mp_vfs_mount_t *vfs_cur;
struct _mp_vfs_mount_t *vfs_mount_table;
extmod: Add generic VFS sub-system. This provides mp_vfs_XXX functions (eg mount, open, listdir) which are agnostic to the underlying filesystem type, and just require an object with the relevant filesystem-like methods (eg .mount, .open, .listidr) which can then be mounted. These mp_vfs_XXX functions would typically be used by a port to implement the "uos" module, and mp_vfs_open would be the builtin open function. This feature is controlled by MICROPY_VFS, disabled by default.
2017-01-27 04:10:09 +00:00
#endif
extmod/modbluetooth: Add low-level Python BLE API.
2019-07-23 14:49:06 +00:00
#if MICROPY_PY_BLUETOOTH
mp_obj_t bluetooth;
#endif
py: Put all global state together in state structures. This patch consolidates all global variables in py/ core into one place, in a global structure. Root pointers are all located together to make GC tracing easier and more efficient.
2015-01-01 23:30:53 +00:00
//
// END ROOT POINTER SECTION
////////////////////////////////////////////////////////////
py: Improve allocation policy of qstr data. Previous to this patch all interned strings lived in their own malloc'd chunk. On average this wastes N/2 bytes per interned string, where N is the number-of-bytes for a quanta of the memory allocator (16 bytes on 32 bit archs). With this patch interned strings are concatenated into the same malloc'd chunk when possible. Such chunks are enlarged inplace when possible, and shrunk to fit when a new chunk is needed. RAM savings with this patch are highly varied, but should always show an improvement (unless only 3 or 4 strings are interned). New version typically uses about 70% of previous memory for the qstr data, and can lead to savings of around 10% of total memory footprint of a running script. Costs about 120 bytes code size on Thumb2 archs (depends on how many calls to gc_realloc are made).
2015-06-13 20:53:22 +00:00
// pointer and sizes to store interned string data
// (qstr_last_chunk can be root pointer but is also stored in qstr pool)
byte *qstr_last_chunk;
py/qstr: Use size_t instead of mp_uint_t when counting allocated bytes.
2015-12-17 12:41:40 +00:00
size_t qstr_last_alloc;
size_t qstr_last_used;
py: Improve allocation policy of qstr data. Previous to this patch all interned strings lived in their own malloc'd chunk. On average this wastes N/2 bytes per interned string, where N is the number-of-bytes for a quanta of the memory allocator (16 bytes on 32 bit archs). With this patch interned strings are concatenated into the same malloc'd chunk when possible. Such chunks are enlarged inplace when possible, and shrunk to fit when a new chunk is needed. RAM savings with this patch are highly varied, but should always show an improvement (unless only 3 or 4 strings are interned). New version typically uses about 70% of previous memory for the qstr data, and can lead to savings of around 10% of total memory footprint of a running script. Costs about 120 bytes code size on Thumb2 archs (depends on how many calls to gc_realloc are made).
2015-06-13 20:53:22 +00:00
py/qstr: Don't include or init qstr_mutex when GIL is enabled. When threads and the GIL are enabled, then the qstr mutex is not needed. The qstr_mutex field is never used in this case because of: #if MICROPY_PY_THREAD && !MICROPY_PY_THREAD_GIL #define QSTR_ENTER() mp_thread_mutex_lock(&MP_STATE_VM(qstr_mutex), 1) #define QSTR_EXIT() mp_thread_mutex_unlock(&MP_STATE_VM(qstr_mutex)) #else #define QSTR_ENTER() #define QSTR_EXIT() #endif So, we can completely remove qstr_mutex everywhere when MICROPY_PY_THREAD && !MICROPY_PY_THREAD_GIL.
2020-01-22 17:19:37 +00:00
#if MICROPY_PY_THREAD && !MICROPY_PY_THREAD_GIL
py: Make interning of qstrs thread safe.
2016-05-26 09:06:46 +00:00
// This is a global mutex used to make qstr interning thread-safe.
mp_thread_mutex_t qstr_mutex;
#endif
py: Don't include mp_optimise_value or opt_level() if compiler disabled. Without the compiler enabled the mp_optimise_value is unused, and the micropython.opt_level() function is not useful, so exclude these from the build to save RAM and code size.
2018-04-04 04:24:03 +00:00
#if MICROPY_ENABLE_COMPILER
py: Put all global state together in state structures. This patch consolidates all global variables in py/ core into one place, in a global structure. Root pointers are all located together to make GC tracing easier and more efficient.
2015-01-01 23:30:53 +00:00
mp_uint_t mp_optimise_value;
py: Add global default_emit_opt variable to make emit kind persistent. mp_compile no longer takes an emit_opt argument, rather this setting is now provided by the global default_emit_opt variable. Now, when -X emit=native is passed as a command-line option, the emitter will be set for all compiled modules (included imports), not just the top-level script. In the future there could be a way to also set this variable from a script. Fixes issue #4267.
2019-08-23 01:20:50 +00:00
#if MICROPY_EMIT_NATIVE
uint8_t default_emit_opt; // one of MP_EMIT_OPT_xxx
#endif
py: Don't include mp_optimise_value or opt_level() if compiler disabled. Without the compiler enabled the mp_optimise_value is unused, and the micropython.opt_level() function is not useful, so exclude these from the build to save RAM and code size.
2018-04-04 04:24:03 +00:00
#endif
py: Put all global state together in state structures. This patch consolidates all global variables in py/ core into one place, in a global structure. Root pointers are all located together to make GC tracing easier and more efficient.
2015-01-01 23:30:53 +00:00
// size of the emergency exception buf, if it's dynamically allocated
#if MICROPY_ENABLE_EMERGENCY_EXCEPTION_BUF && MICROPY_EMERGENCY_EXCEPTION_BUF_SIZE == 0
mp_int_t mp_emergency_exception_buf_size;
#endif
py: Implement a simple global interpreter lock. This makes the VM/runtime thread safe, at the cost of not being able to run code in parallel.
2016-05-26 10:42:53 +00:00
py/mpstate.h: Adjust start of root pointer section to exclude non-ptrs. This patch moves the start of the root pointer section in mp_state_ctx_t so that it skips entries that are not pointers and don't need scanning. Previously, the start of the root pointer section was at the very beginning of the mp_state_ctx_t struct (which is the beginning of mp_state_thread_t). This was the original assembler version of the NLR code was hard-coded to have the nlr_top pointer at the start of this state structure. But now that the NLR code is partially written in C there is no longer this restriction on the location of nlr_top (and a comment to this effect has been removed in this patch). So now the root pointer section starts part way through the mp_state_thread_t structure, after the entries which are not root pointers. This patch also moves the non-pointer entries for MICROPY_ENABLE_SCHEDULER outside the root pointer section. Moving non-pointer entries out of the root pointer section helps to make the GC more precise and should help to prevent some cases of collectable garbage being kept. This patch also has a measurable improvement in performance of the pystone.py benchmark: on unix x86-64 and stm32 there was an improvement of roughly 0.6% (tested with both gcc 7.3 and gcc 8.1).
2018-05-12 12:09:34 +00:00
#if MICROPY_ENABLE_SCHEDULER
volatile int16_t sched_state;
py/scheduler: Convert micropythyon.schedule() to a circular buffer. This means the schedule operates on a first-in, first-executed manner rather than the current last-in, first executed.
2019-03-21 00:52:10 +00:00
uint8_t sched_len;
uint8_t sched_idx;
py/mpstate.h: Adjust start of root pointer section to exclude non-ptrs. This patch moves the start of the root pointer section in mp_state_ctx_t so that it skips entries that are not pointers and don't need scanning. Previously, the start of the root pointer section was at the very beginning of the mp_state_ctx_t struct (which is the beginning of mp_state_thread_t). This was the original assembler version of the NLR code was hard-coded to have the nlr_top pointer at the start of this state structure. But now that the NLR code is partially written in C there is no longer this restriction on the location of nlr_top (and a comment to this effect has been removed in this patch). So now the root pointer section starts part way through the mp_state_thread_t structure, after the entries which are not root pointers. This patch also moves the non-pointer entries for MICROPY_ENABLE_SCHEDULER outside the root pointer section. Moving non-pointer entries out of the root pointer section helps to make the GC more precise and should help to prevent some cases of collectable garbage being kept. This patch also has a measurable improvement in performance of the pystone.py benchmark: on unix x86-64 and stm32 there was an improvement of roughly 0.6% (tested with both gcc 7.3 and gcc 8.1).
2018-05-12 12:09:34 +00:00
#endif
py: Implement a simple global interpreter lock. This makes the VM/runtime thread safe, at the cost of not being able to run code in parallel.
2016-05-26 10:42:53 +00:00
#if MICROPY_PY_THREAD_GIL
// This is a global mutex used to make the VM/runtime thread-safe.
mp_thread_mutex_t gil_mutex;
#endif
py/map: Add an optional cache of (map+index) to speed up map lookups. The existing inline bytecode caching optimisation, selected by MICROPY_OPT_CACHE_MAP_LOOKUP_IN_BYTECODE, reserves an extra byte in the bytecode after certain opcodes, which at runtime stores a map index of the likely location of this field when looking up the qstr. This scheme is incompatible with bytecode-in-ROM, and doesn't work with native generated code. It also stores bytecode in .mpy files which is of a different format to when the feature is disabled, making generation of .mpy files more complex. This commit provides an alternative optimisation via an approach that adds a global cache for map offsets, then all mp_map_lookup operations use it. It's less precise than bytecode caching, but allows the cache to be independent and external to the bytecode that is executing. It also works for the native emitter and adds a similar performance boost on top of the gain already provided by the native emitter. Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-08-18 04:52:48 +00:00
#if MICROPY_OPT_MAP_LOOKUP_CACHE
// See mp_map_lookup.
uint8_t map_lookup_cache[MICROPY_OPT_MAP_LOOKUP_CACHE_SIZE];
#endif
py: Put all global state together in state structures. This patch consolidates all global variables in py/ core into one place, in a global structure. Root pointers are all located together to make GC tracing easier and more efficient.
2015-01-01 23:30:53 +00:00
} mp_state_vm_t;
py: Add MP_STATE_THREAD to hold state specific to a given thread.
2016-04-22 22:44:56 +00:00
// This structure holds state that is specific to a given thread.
// Everything in this structure is scanned for root pointers.
typedef struct _mp_state_thread_t {
// Stack top at the start of program
char *stack_top;
#if MICROPY_STACK_CHECK
size_t stack_limit;
#endif
py: Introduce a Python stack for scoped allocation. This patch introduces the MICROPY_ENABLE_PYSTACK option (disabled by default) which enables a "Python stack" that allows to allocate and free memory in a scoped, or Last-In-First-Out (LIFO) way, similar to alloca(). A new memory allocation API is introduced along with this Py-stack. It includes both "local" and "nonlocal" LIFO allocation. Local allocation is intended to be equivalent to using alloca(), whereby the same function must free the memory. Nonlocal allocation is where another function may free the memory, so long as it's still LIFO. Follow-up patches will convert all uses of alloca() and VLA to the new scoped allocation API. The old behaviour (using alloca()) will still be available, but when MICROPY_ENABLE_PYSTACK is enabled then alloca() is no longer required or used. The benefits of enabling this option are (or will be once subsequent patches are made to convert alloca()/VLA): - Toolchains without alloca() can use this feature to obtain correct and efficient scoped memory allocation (compared to using the heap instead of alloca(), which is slower). - Even if alloca() is available, enabling the Py-stack gives slightly more efficient use of stack space when calling nested Python functions, due to the way that compilers implement alloca(). - Enabling the Py-stack with the stackless mode allows for even more efficient stack usage, as well as retaining high performance (because the heap is no longer used to build and destroy stackless code states). - With Py-stack and stackless enabled, Python-calling-Python is no longer recursive in the C mp_execute_bytecode function. The micropython.pystack_use() function is included to measure usage of the Python stack.
2017-11-26 12:28:40 +00:00
#if MICROPY_ENABLE_PYSTACK
uint8_t *pystack_start;
uint8_t *pystack_end;
uint8_t *pystack_cur;
#endif
py/mpstate.h: Adjust start of root pointer section to exclude non-ptrs. This patch moves the start of the root pointer section in mp_state_ctx_t so that it skips entries that are not pointers and don't need scanning. Previously, the start of the root pointer section was at the very beginning of the mp_state_ctx_t struct (which is the beginning of mp_state_thread_t). This was the original assembler version of the NLR code was hard-coded to have the nlr_top pointer at the start of this state structure. But now that the NLR code is partially written in C there is no longer this restriction on the location of nlr_top (and a comment to this effect has been removed in this patch). So now the root pointer section starts part way through the mp_state_thread_t structure, after the entries which are not root pointers. This patch also moves the non-pointer entries for MICROPY_ENABLE_SCHEDULER outside the root pointer section. Moving non-pointer entries out of the root pointer section helps to make the GC more precise and should help to prevent some cases of collectable garbage being kept. This patch also has a measurable improvement in performance of the pystone.py benchmark: on unix x86-64 and stm32 there was an improvement of roughly 0.6% (tested with both gcc 7.3 and gcc 8.1).
2018-05-12 12:09:34 +00:00
py/gc: Make gc_lock_depth have a count per thread. This commit makes gc_lock_depth have one counter per thread, instead of one global counter. This makes threads properly independent with respect to the GC, in particular threads can now independently lock the GC for themselves without locking it for other threads. It also means a given thread can run a hard IRQ without temporarily locking the GC for all other threads and potentially making them have MemoryError exceptions at random locations (this really only occurs on MCUs with multiple cores and no GIL, eg on the rp2 port). The commit also removes protection of the GC lock/unlock functions, which is no longer needed when the counter is per thread (and this also fixes the cas where a hard IRQ calling gc_lock() may stall waiting for the mutex). It also puts the check for `gc_lock_depth > 0` outside the GC mutex in gc_alloc, gc_realloc and gc_free, to potentially prevent a hard IRQ from waiting on a mutex if it does attempt to allocate heap memory (and putting the check outside the GC mutex is now safe now that there is a gc_lock_depth per thread). Signed-off-by: Damien George <damien@micropython.org>
2021-05-04 13:56:43 +00:00
// Locking of the GC is done per thread.
uint16_t gc_lock_depth;
py/mpstate.h: Adjust start of root pointer section to exclude non-ptrs. This patch moves the start of the root pointer section in mp_state_ctx_t so that it skips entries that are not pointers and don't need scanning. Previously, the start of the root pointer section was at the very beginning of the mp_state_ctx_t struct (which is the beginning of mp_state_thread_t). This was the original assembler version of the NLR code was hard-coded to have the nlr_top pointer at the start of this state structure. But now that the NLR code is partially written in C there is no longer this restriction on the location of nlr_top (and a comment to this effect has been removed in this patch). So now the root pointer section starts part way through the mp_state_thread_t structure, after the entries which are not root pointers. This patch also moves the non-pointer entries for MICROPY_ENABLE_SCHEDULER outside the root pointer section. Moving non-pointer entries out of the root pointer section helps to make the GC more precise and should help to prevent some cases of collectable garbage being kept. This patch also has a measurable improvement in performance of the pystone.py benchmark: on unix x86-64 and stm32 there was an improvement of roughly 0.6% (tested with both gcc 7.3 and gcc 8.1).
2018-05-12 12:09:34 +00:00
////////////////////////////////////////////////////////////
// START ROOT POINTER SECTION
// Everything that needs GC scanning must start here, and
// is followed by state in the mp_state_vm_t structure.
//
mp_obj_dict_t *dict_locals;
mp_obj_dict_t *dict_globals;
nlr_buf_t *nlr_top;
py: Integrate sys.settrace feature into the VM and runtime. This commit adds support for sys.settrace, allowing to install Python handlers to trace execution of Python code. The interface follows CPython as closely as possible. The feature is disabled by default and can be enabled via MICROPY_PY_SYS_SETTRACE.
2019-08-14 14:09:36 +00:00
py/mpstate: Make exceptions thread-local. This moves mp_pending_exception from mp_state_vm_t to mp_state_thread_t. This allows exceptions to be scheduled on a specific thread. Signed-off-by: David Lechner <david@pybricks.com>
2021-05-11 02:53:22 +00:00
// pending exception object (MP_OBJ_NULL if not pending)
volatile mp_obj_t mp_pending_exception;
py: Support single argument to optimised MP_OBJ_STOP_ITERATION. The MP_OBJ_STOP_ITERATION optimisation is a shortcut for creating a StopIteration() exception object, and means that heap memory does not need to be allocated for the exception (in cases where it can be used). This commit allows this optimised object to take an optional argument (before, it could only have no argument). The commit also adds some new tests to cover corner cases with StopIteration and generators that previously did not work. Signed-off-by: Damien George <damien@micropython.org>
2021-06-29 07:34:34 +00:00
// If MP_OBJ_STOP_ITERATION is propagated then this holds its argument.
mp_obj_t stop_iteration_arg;
py: Integrate sys.settrace feature into the VM and runtime. This commit adds support for sys.settrace, allowing to install Python handlers to trace execution of Python code. The interface follows CPython as closely as possible. The feature is disabled by default and can be enabled via MICROPY_PY_SYS_SETTRACE.
2019-08-14 14:09:36 +00:00
#if MICROPY_PY_SYS_SETTRACE
mp_obj_t prof_trace_callback;
bool prof_callback_is_executing;
struct _mp_code_state_t *current_code_state;
#endif
py: Add MP_STATE_THREAD to hold state specific to a given thread.
2016-04-22 22:44:56 +00:00
} mp_state_thread_t;
py: Move locals/globals dicts to the thread-specific state. Each threads needs to have its own private references to its current locals/globals dicts, otherwise functions running within different contexts (eg imported from different files) can behave very strangely.
2017-02-27 12:56:46 +00:00
// This structure combines the above 3 structures.
// The order of the entries are important for root pointer scanning in the GC to work.
py: Put all global state together in state structures. This patch consolidates all global variables in py/ core into one place, in a global structure. Root pointers are all located together to make GC tracing easier and more efficient.
2015-01-01 23:30:53 +00:00
typedef struct _mp_state_ctx_t {
py: Add MP_STATE_THREAD to hold state specific to a given thread.
2016-04-22 22:44:56 +00:00
mp_state_thread_t thread;
py: Put all global state together in state structures. This patch consolidates all global variables in py/ core into one place, in a global structure. Root pointers are all located together to make GC tracing easier and more efficient.
2015-01-01 23:30:53 +00:00
mp_state_vm_t vm;
mp_state_mem_t mem;
} mp_state_ctx_t;
extern mp_state_ctx_t mp_state_ctx;
#define MP_STATE_VM(x) (mp_state_ctx.vm.x)
#define MP_STATE_MEM(x) (mp_state_ctx.mem.x)
py/mpstate: Schedule KeyboardInterrupt on main thread. This introduces a new macro to get the main thread and uses it to ensure that asynchronous exceptions such as KeyboardInterrupt (CTRL+C) are only scheduled on the main thread. This is more deterministic than being scheduled on a random thread and is more in line with CPython that only allow signal handlers to run on the main thread. Fixes issue #7026. Signed-off-by: David Lechner <david@pybricks.com>
2021-05-11 03:18:17 +00:00
#define MP_STATE_MAIN_THREAD(x) (mp_state_ctx.thread.x)
py: Put all global state together in state structures. This patch consolidates all global variables in py/ core into one place, in a global structure. Root pointers are all located together to make GC tracing easier and more efficient.
2015-01-01 23:30:53 +00:00
py: Add basic _thread module, with ability to start a new thread.
2016-04-22 22:52:33 +00:00
#if MICROPY_PY_THREAD
extern mp_state_thread_t *mp_thread_get_state(void);
#define MP_STATE_THREAD(x) (mp_thread_get_state()->x)
#else
py/mpstate: Schedule KeyboardInterrupt on main thread. This introduces a new macro to get the main thread and uses it to ensure that asynchronous exceptions such as KeyboardInterrupt (CTRL+C) are only scheduled on the main thread. This is more deterministic than being scheduled on a random thread and is more in line with CPython that only allow signal handlers to run on the main thread. Fixes issue #7026. Signed-off-by: David Lechner <david@pybricks.com>
2021-05-11 03:18:17 +00:00
#define MP_STATE_THREAD(x) MP_STATE_MAIN_THREAD(x)
py: Add basic _thread module, with ability to start a new thread.
2016-04-22 22:52:33 +00:00
#endif
py: Add MP_STATE_THREAD to hold state specific to a given thread.
2016-04-22 22:44:56 +00:00
all: Unify header guard usage. The code conventions suggest using header guards, but do not define how those should look like and instead point to existing files. However, not all existing files follow the same scheme, sometimes omitting header guards altogether, sometimes using non-standard names, making it easy to accidentally pick a "wrong" example. This commit ensures that all header files of the MicroPython project (that were not simply copied from somewhere else) follow the same pattern, that was already present in the majority of files, especially in the py folder. The rules are as follows. Naming convention: * start with the words MICROPY_INCLUDED * contain the full path to the file * replace special characters with _ In addition, there are no empty lines before #ifndef, between #ifndef and one empty line before #endif. #endif is followed by a comment containing the name of the guard macro. py/grammar.h cannot use header guards by design, since it has to be included multiple times in a single C file. Several other files also do not need header guards as they are only used internally and guaranteed to be included only once: * MICROPY_MPHALPORT_H * mpconfigboard.h * mpconfigport.h * mpthreadport.h * pin_defs_*.h * qstrdefs*.h
2017-06-29 21:14:58 +00:00
#endif // MICROPY_INCLUDED_PY_MPSTATE_H