diff --git a/docs/develop/gettingstarted.rst b/docs/develop/gettingstarted.rst index 000b7d6139..c2d3816d42 100644 --- a/docs/develop/gettingstarted.rst +++ b/docs/develop/gettingstarted.rst @@ -322,7 +322,8 @@ tests tools - Contains helper tools including the ``upip`` and the ``pyboard.py`` module. + Contains scripts used by the build and CI process, as well as user tools such + as ``pyboard.py`` and ``mpremote``. examples diff --git a/docs/develop/optimizations.rst b/docs/develop/optimizations.rst index d972cde666..7f2c8cbe72 100644 --- a/docs/develop/optimizations.rst +++ b/docs/develop/optimizations.rst @@ -25,6 +25,8 @@ into the firmware image as part of the main firmware compilation process, which the bytecode will be executed from ROM. This can lead to a significant memory saving, and reduce heap fragmentation. +See :ref:`manifest` for more information. + Variables --------- diff --git a/docs/esp8266/tutorial/intro.rst b/docs/esp8266/tutorial/intro.rst index ac46e68b5a..75739bd6f9 100644 --- a/docs/esp8266/tutorial/intro.rst +++ b/docs/esp8266/tutorial/intro.rst @@ -23,7 +23,7 @@ convertor to make the UART available to your PC. The minimum requirement for flash size is 1Mbyte. There is also a special build for boards with 512KB, but it is highly limited comparing to the normal build: there is no support for filesystem, and thus features which -depend on it won't work (WebREPL, upip, etc.). As such, 512KB build will +depend on it won't work (WebREPL, mip, etc.). As such, 512KB build will be more interesting for users who build from source and fine-tune parameters for their particular application. diff --git a/docs/pyboard/tutorial/lcd160cr_skin.rst b/docs/pyboard/tutorial/lcd160cr_skin.rst index fa0debcb1c..a0fe88a2e6 100644 --- a/docs/pyboard/tutorial/lcd160cr_skin.rst +++ b/docs/pyboard/tutorial/lcd160cr_skin.rst @@ -42,7 +42,7 @@ There is a test program which you can use to test the features of the display, and which also serves as a basis to start creating your own code that uses the LCD. This test program is available on GitHub `here `__. -Copy it to the board over USB mass storage, or by using `mpremote`. +Copy it to the board over USB mass storage, or by using :ref:`mpremote`. To run the test from the MicroPython prompt do:: diff --git a/docs/reference/glossary.rst b/docs/reference/glossary.rst index da951189e2..4c66f70319 100644 --- a/docs/reference/glossary.rst +++ b/docs/reference/glossary.rst @@ -52,7 +52,7 @@ Glossary cross-compiler Also known as ``mpy-cross``. This tool runs on your PC and converts a :term:`.py file` containing MicroPython code into a :term:`.mpy file` - containing MicroPython bytecode. This means it loads faster (the board + containing MicroPython :term:`bytecode`. This means it loads faster (the board doesn't have to compile the code), and uses less space on flash (the bytecode is more space efficient). @@ -128,7 +128,7 @@ Glossary Unlike the :term:`CPython` stdlib, micropython-lib modules are intended to be installed individually - either using manual copying or - using :term:`upip`. + using :term:`mip`. MicroPython port MicroPython supports different :term:`boards `, RTOSes, and @@ -151,16 +151,26 @@ Glossary machine-independent features. It can also function in a similar way to :term:`CPython`'s ``python`` executable. + mip + A package installer for MicroPython (mip - "mip installs packages"). It + installs MicroPython packages either from :term:`micropython-lib`, + GitHub, or arbitrary URLs. mip can be used on-device on + network-capable boards, and internally by tools such + as :term:`mpremote`. + + mpremote + A tool for interacting with a MicroPython device. See :ref:`mpremote`. + .mpy file The output of the :term:`cross-compiler`. A compiled form of a - :term:`.py file` that contains MicroPython bytecode instead of Python - source code. + :term:`.py file` that contains MicroPython :term:`bytecode` instead of + Python source code. native Usually refers to "native code", i.e. machine code for the target microcontroller (such as ARM Thumb, Xtensa, x86/x64). The ``@native`` decorator can be applied to a MicroPython function to generate native - code instead of bytecode for that function, which will likely be + code instead of :term:`bytecode` for that function, which will likely be faster but use more RAM. port @@ -193,8 +203,10 @@ Glossary as a serial port over USB. upip - (Literally, "micro pip"). A package manager for MicroPython, inspired + A now-obsolete package manager for MicroPython, inspired by :term:`CPython`'s pip, but much smaller and with reduced - functionality. - upip runs both on the :term:`Unix port ` and on - :term:`baremetal` ports which offer filesystem and networking support. + functionality. See its replacement, :term:`mip`. + + webrepl + A way of connecting to the REPL (and transferring files) on a device + over the internet from a browser. See https://micropython.org/webrepl diff --git a/docs/reference/manifest.rst b/docs/reference/manifest.rst index b756de47ed..9bcafd5839 100644 --- a/docs/reference/manifest.rst +++ b/docs/reference/manifest.rst @@ -1,35 +1,177 @@ +.. _manifest: + MicroPython manifest files ========================== -When building firmware for a device the following components are included in -the compilation process: +Summary +------- -- the core MicroPython virtual machine and runtime -- port-specific system code and drivers to interface with the - microcontroller/device that the firmware is targeting -- standard built-in modules, like ``sys`` -- extended built-in modules, like ``json`` and ``machine`` -- extra modules written in C/C++ -- extra modules written in Python +MicroPython has a feature that allows Python code to be "frozen" into the +firmware, as an alternative to loading code from the filesystem. -All the modules included in the firmware are available via ``import`` from -Python code. The extra modules written in Python that are included in a build -(the last point above) are called *frozen modules*, and are specified by a -``manifest.py`` file. Changing this manifest requires rebuilding the firmware. +This has the following benefits: -It's also possible to add additional modules to the filesystem of the device -once it is up and running. Adding and removing modules to/from the filesystem -does not require rebuilding the firmware so is a simpler process than rebuilding -firmware. The benefit of using a manifest is that frozen modules are more -efficient: they are faster to import and take up less RAM once imported. +- the code is pre-compiled to bytecode, avoiding the need for the Python + source to be compiled at load-time. +- the bytecode can be executed directly from ROM (i.e. flash memory) rather than + being copied into RAM. Similarly any constant objects (strings, tuples, etc) + are loaded from ROM also. This can lead to significantly more memory being + available for your application. +- on devices that do not have a filesystem, this is the only way to + load Python code. -MicroPython manifest files are Python files and can contain arbitrary Python -code. There are also a set of commands (predefined functions) which are used -to specify the Python source files to include. These commands are described -below. +During development, freezing is generally not recommended as it will +significantly slow down your development cycle, as each update will require +re-flashing the entire firmware. However, it can still be useful to +selectively freeze some rarely-changing dependencies (such as third-party +libraries). -Freezing source code --------------------- +The way to list the Python files to be be frozen into the firmware is via +a "manifest", which is a Python file that will be interpreted by the build +process. Typically you would write a manifest file as part of a board +definition, but you can also write a stand-alone manifest file and use it with +an existing board definition. + +Manifest files can define dependencies on libraries from :term:`micropython-lib` +as well as Python files on the filesystem, and also on other manifest files. + +Writing manifest files +---------------------- + +A manifest file is a Python file containing a series of function calls. See the +available functions defined below. + +Any paths used in manifest files can include the following variables. These all +resolve to absolute paths. + +- ``$(MPY_DIR)`` -- path to the micropython repo. +- ``$(MPY_LIB_DIR)`` -- path to the micropython-lib submodule. Prefer to use + ``require()``. +- ``$(PORT_DIR)`` -- path to the current port (e.g. ``ports/stm32``) +- ``$(BOARD_DIR)`` -- path to the current board + (e.g. ``ports/stm32/boards/PYBV11``) + +Custom manifest files should not live in the main MicroPython repository. You +should keep them in version control with the rest of your project. + +Typically a manifest used for compiling firmware will need to include the port +manifest, which might include frozen modules that are required for the board to +function. If you just want to add additional modules to an existing board, then +include the board manifest (which will in turn include the port manifest). + +Building with a custom manifest +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Your manifest can be specified on the ``make`` command line with: + +.. code-block:: bash + + $ make BOARD=MYBOARD FROZEN_MANIFEST=/path/to/my/project/manifest.py + +This applies to all ports, including CMake-based ones (e.g. esp32, rp2), as the +Makefile wrapper that will pass this into the CMake build. + +Adding a manifest to a board definition +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you have a custom board definition, you can make it include your custom +manifest automatically. On make-based ports (most ports), in your +``mpconfigboard.mk`` set the ``FROZEN_MANIFEST`` variable. + +.. code-block:: makefile + + FROZEN_MANIFEST ?= $(BOARD_DIR)/manifest.py + +On CMake-based ports (e.g. esp32, rp2), instead use ``mpconfigboard.cmake`` + +.. code-block:: cmake + + set(MICROPY_FROZEN_MANIFEST ${MICROPY_BOARD_DIR}/manifest.py) + +High-level functions +~~~~~~~~~~~~~~~~~~~~ + +Note: The ``opt`` keyword argument can be set on the various functions, this controls +the optimisation level used by the cross-compiler. +See :func:`micropython.opt_level`. + +.. function:: package(package_path, files=None, base_path=".", opt=None) + + This is equivalent to copying the "package_path" directory to the device + (except as frozen code). + + In the simplest case, to freeze a package "foo" in the current directory: + + .. code-block:: python3 + + package("foo") + + will recursively include all .py files in foo, and will be frozen as + ``foo/**/*.py``. + + If the package isn't in the same directory as the manifest file, use ``base_path``: + + .. code-block:: python3 + + package("foo", base_path="path/to/libraries") + + You can use the variables above, such as ``$(PORT_DIR)`` in ``base_path``. + + To restrict to certain files in the package use ``files`` (note: paths + should be relative to the package): ``package("foo", files=["bar/baz.py"])``. + +.. function:: module(module_path, base_path=".", opt=None) + + Include a single Python file as a module. + + If the file is in the current directory: + + .. code-block:: python3 + + module("foo.py") + + Otherwise use base_path to locate the file: + + .. code-block:: python3 + + module("foo.py", base_path="src/drivers") + + You can use the variables above, such as ``$(PORT_DIR)`` in ``base_path``. + +.. function:: require(name, unix_ffi=False) + + Require a package by name (and its dependencies) from :term:`micropython-lib`. + + Optionally specify unix_ffi=True to use a module from the unix-ffi directory. + +.. function:: include(manifest_path) + + Include another manifest. + + Typically a manifest used for compiling firmware will need to include the + port manifest, which might include frozen modules that are required for + the board to function. + + The *manifest* argument can be a string (filename) or an iterable of + strings. + + Relative paths are resolved with respect to the current manifest file. + + If the path is to a directory, then it implicitly includes the + manifest.py file inside that directory. + + You can use the variables above, such as ``$(PORT_DIR)`` in ``manifest_path``. + +.. function:: metadata(description=None, version=None, license=None, author=None) + + Define metadata for this manifest file. This is useful for manifests for + micropython-lib packages. + +Low-level functions +~~~~~~~~~~~~~~~~~~~ + +These functions are documented for completeness, but with the exception of +``freeze_as_str`` all functionality can be accessed via the high-level functions. .. function:: freeze(path, script=None, opt=0) @@ -42,9 +184,7 @@ Freezing source code module will start after *path*, i.e. *path* is excluded from the module name. - If *path* is relative, it is resolved to the current ``manifest.py``. Use - ``$(MPY_DIR)``, ``$(MPY_LIB_DIR)``, ``$(PORT_DIR)``, ``$(BOARD_DIR)`` if you - need to access specific paths. + If *path* is relative, it is resolved to the current ``manifest.py``. If *script* is None, all files in *path* will be frozen. @@ -75,71 +215,48 @@ Freezing source code Freeze the input, which must be ``.mpy`` files that are frozen directly. See ``freeze()`` for further details on the arguments. - -Including other manifest files ------------------------------- - -.. function:: include(manifest, **kwargs) - - Include another manifest. - - The *manifest* argument can be a string (filename) or an iterable of - strings. - - Relative paths are resolved with respect to the current manifest file. - - Optional *kwargs* can be provided which will be available to the included - script via the *options* variable. - - For example: - - .. code-block:: python3 - - include("path.py", extra_features=True) - - then in path.py: - - .. code-block:: python3 - - options.defaults(standard_features=True) - # freeze minimal modules. - if options.standard_features: - # freeze standard modules. - if options.extra_features: - # freeze extra modules. - - Examples -------- -To freeze a single file which is available as ``import mydriver``, use: +To freeze a single file from the current directory which will be available as +``import mydriver``, use: .. code-block:: python3 - freeze(".", "mydriver.py") + module("mydriver.py") -To freeze a set of files which are available as ``import test1`` and -``import test2``, and which are compiled with optimisation level 3, use: +To freeze a directory of files in a subdirectory "mydriver" of the current +directory which will be available as ``import mydriver``, use: .. code-block:: python3 - freeze("/path/to/tests", ("test1.py", "test2.py"), opt=3) + package("mydriver") -To freeze a module which can be imported as ``import mymodule``, use: +To freeze the "hmac" library from :term:`micropython-lib`, use: .. code-block:: python3 - freeze( - "../relative/path", - ( - "mymodule/__init__.py", - "mymodule/core.py", - "mymodule/extra.py", - ), - ) + require("hmac") -To include a manifest from the MicroPython repository, use: +A more complete example of a custom ``manifest.py`` file for the ``PYBD_SF2`` +board is: .. code-block:: python3 - include("$(MPY_DIR)/extmod/uasyncio/manifest.py") + # Include the board's default manifest. + include("$(BOARD_DIR)/manifest.py") + # Add a custom driver + module("mydriver.py") + # Add aiorepl from micropython-lib + require("aiorepl") + +Then the board can be compiled with + +.. code-block:: bash + + $ cd ports/stm32 + $ make BOARD=PYBD_SF2 FROZEN_MANIFEST=~/src/myproject/manifest.py + +Note that most boards do not have their own ``manifest.py``, rather they use the +port one directly, in which case your manifest should just +``include("$(PORT_DIR)/boards/manifest.py")`` instead. diff --git a/docs/reference/packages.rst b/docs/reference/packages.rst index eb44992ed6..0c049d1fb2 100644 --- a/docs/reference/packages.rst +++ b/docs/reference/packages.rst @@ -1,314 +1,153 @@ .. _packages: -Distribution packages, package management, and deploying applications -===================================================================== - -Just as the "big" Python, MicroPython supports creation of "third party" -packages, distributing them, and easily installing them in each user's -environment. This chapter discusses how these actions are achieved. -Some familiarity with Python packaging is recommended. - -Overview --------- - -Steps below represent a high-level workflow when creating and consuming -packages: - -1. Python modules and packages are turned into distribution package - archives, and published at the Python Package Index (PyPI). -2. :term:`upip` package manager can be used to install a distribution package - on a :term:`MicroPython port` with networking capabilities (for example, - on the Unix port). -3. For ports without networking capabilities, an "installation image" - can be prepared on the Unix port, and transferred to a device by - suitable means. -4. For low-memory ports, the installation image can be frozen as the - bytecode into MicroPython executable, thus minimizing the memory - storage overheads. - -The sections below describe this process in details. - -Distribution packages ---------------------- - -Python modules and packages can be packaged into archives suitable for -transfer between systems, storing at the well-known location (PyPI), -and downloading on demand for deployment. These archives are known as -*distribution packages* (to differentiate them from Python packages -(means to organize Python source code)). - -The MicroPython distribution package format is a well-known tar.gz -format, with some adaptations however. The Gzip compressor, used as -an external wrapper for TAR archives, by default uses 32KB dictionary -size, which means that to uncompress a compressed stream, 32KB of -contiguous memory needs to be allocated. This requirement may be not -satisfiable on low-memory devices, which may have total memory available -less than that amount, and even if not, a contiguous block like that -may be hard to allocate due to memory fragmentation. To accommodate -these constraints, MicroPython distribution packages use Gzip compression -with the dictionary size of 4K, which should be a suitable compromise -with still achieving some compression while being able to uncompressed -even by the smallest devices. - -Besides the small compression dictionary size, MicroPython distribution -packages also have other optimizations, like removing any files from -the archive which aren't used by the installation process. In particular, -:term:`upip` package manager doesn't execute ``setup.py`` during installation -(see below), and thus that file is not included in the archive. - -At the same time, these optimizations make MicroPython distribution -packages not compatible with :term:`CPython`'s package manager, ``pip``. -This isn't considered a big problem, because: - -1. Packages can be installed with :term:`upip`, and then can be used with - CPython (if they are compatible with it). -2. In the other direction, majority of CPython packages would be - incompatible with MicroPython by various reasons, first of all, - the reliance on features not implemented by MicroPython. - -Summing up, the MicroPython distribution package archives are highly -optimized for MicroPython's target environments, which are highly -resource constrained devices. - - -``upip`` package manager ------------------------- - -MicroPython distribution packages are intended to be installed using -the :term:`upip` package manager. :term:`upip` is a Python application which is -usually distributed (as frozen bytecode) with network-enabled -:term:`MicroPython ports `. At the very least, -:term:`upip` is available in the :term:`MicroPython Unix port`. - -On any :term:`MicroPython port` providing :term:`upip`, it can be accessed as -following:: - - import upip - upip.help() - upip.install(package_or_package_list, [path]) - -Where *package_or_package_list* is the name of a distribution -package to install, or a list of such names to install multiple -packages. Optional *path* parameter specifies filesystem -location to install under and defaults to the standard library -location (see below). - -An example of installing a specific package and then using it:: - - >>> import upip - >>> upip.install("micropython-pystone_lowmem") - [...] - >>> import pystone_lowmem - >>> pystone_lowmem.main() - -Note that the name of Python package and the name of distribution -package for it in general don't have to match, and oftentimes they -don't. This is because PyPI provides a central package repository -for all different Python implementations and versions, and thus -distribution package names may need to be namespaced for a particular -implementation. For example, all packages from `micropython-lib` -follow this naming convention: for a Python module or package named -``foo``, the distribution package name is ``micropython-foo``. - -For the ports which run MicroPython executable from the OS command -prompts (like the Unix port), `upip` can be (and indeed, usually is) -run from the command line instead of MicroPython's own REPL. The -commands which corresponds to the example above are:: - - micropython -m upip -h - micropython -m upip install [-p ] ... - micropython -m upip install micropython-pystone_lowmem - -[TODO: Describe installation path.] - - -Cross-installing packages -------------------------- - -For :term:`MicroPython ports ` without native networking -capabilities, the recommend process is "cross-installing" them into a -"directory image" using the :term:`MicroPython Unix port`, and then -transferring this image to a device by suitable means. - -Installing to a directory image involves using ``-p`` switch to :term:`upip`:: - - micropython -m upip install -p install_dir micropython-pystone_lowmem - -After this command, the package content (and contents of every dependency -packages) will be available in the ``install_dir/`` subdirectory. You -would need to transfer contents of this directory (without the -``install_dir/`` prefix) to the device, at the suitable location, where -it can be found by the Python ``import`` statement (see discussion of -the :term:`upip` installation path above). - - -Cross-installing packages with freezing ---------------------------------------- - -For the low-memory :term:`MicroPython ports `, the process -described in the previous section does not provide the most efficient -resource usage,because the packages are installed in the source form, -so need to be compiled to the bytecome on each import. This compilation -requires RAM, and the resulting bytecode is also stored in RAM, reducing -its amount available for storing application data. Moreover, the process -above requires presence of the filesystem on a device, and the most -resource-constrained devices may not even have it. - -The bytecode freezing is a process which resolves all the issues -mentioned above: - -* The source code is pre-compiled into bytecode and store as such. -* The bytecode is stored in ROM, not RAM. -* Filesystem is not required for frozen packages. - -Using frozen bytecode requires building the executable (firmware) -for a given :term:`MicroPython port` from the C source code. Consequently, -the process is: - -1. Follow the instructions for a particular port on setting up a - toolchain and building the port. For example, for ESP8266 port, - study instructions in ``ports/esp8266/README.md`` and follow them. - Make sure you can build the port and deploy the resulting - executable/firmware successfully before proceeding to the next steps. -2. Build :term:`MicroPython Unix port` and make sure it is in your PATH and - you can execute ``micropython``. -3. Change to port's directory (e.g. ``ports/esp8266/`` for ESP8266). -4. Run ``make clean-frozen``. This step cleans up any previous - modules which were installed for freezing (consequently, you need - to skip this step to add additional modules, instead of starting - from scratch). -5. Run ``micropython -m upip install -p modules ...`` to - install packages you want to freeze. -6. Run ``make clean``. -7. Run ``make``. - -After this, you should have the executable/firmware with modules as -the bytecode inside, which you can deploy the usual way. - -Few notes: - -1. Step 5 in the sequence above assumes that the distribution package - is available from PyPI. If that is not the case, you would need - to copy Python source files manually to ``modules/`` subdirectory - of the port directory. (Note that upip does not support - installing from e.g. version control repositories). -2. The firmware for baremetal devices usually has size restrictions, - so adding too many frozen modules may overflow it. Usually, you - would get a linking error if this happens. However, in some cases, - an image may be produced, which is not runnable on a device. Such - cases are in general bugs, and should be reported and further - investigated. If you face such a situation, as an initial step, - you may want to decrease the amount of frozen modules included. - - -Creating distribution packages ------------------------------- - -Distribution packages for MicroPython are created in the same manner -as for CPython or any other Python implementation, see references at -the end of chapter. Setuptools (instead of distutils) should be used, -because distutils do not support dependencies and other features. "Source -distribution" (``sdist``) format is used for packaging. The post-processing -discussed above, (and pre-processing discussed in the following section) -is achieved by using custom ``sdist`` command for setuptools. Thus, packaging -steps remain the same as for the standard setuptools, the user just -needs to override ``sdist`` command implementation by passing the -appropriate argument to ``setup()`` call:: - - from setuptools import setup - import sdist_upip - - setup( - ..., - cmdclass={'sdist': sdist_upip.sdist} - ) - -The sdist_upip.py module as referenced above can be found in -`micropython-lib`: -https://github.com/micropython/micropython-lib/blob/master/sdist_upip.py - - -Application resources ---------------------- - -A complete application, besides the source code, oftentimes also consists -of data files, e.g. web page templates, game images, etc. It's clear how -to deal with those when application is installed manually - you just put -those data files in the filesystem at some location and use the normal -file access functions. - -The situation is different when deploying applications from packages - this -is more advanced, streamlined and flexible way, but also requires more -advanced approach to accessing data files. This approach is treating -the data files as "resources", and abstracting away access to them. - -Python supports resource access using its "setuptools" library, using -``pkg_resources`` module. MicroPython, following its usual approach, -implements subset of the functionality of that module, specifically -``pkg_resources.resource_stream(package, resource)`` function. -The idea is that an application calls this function, passing a -resource identifier, which is a relative path to data file within -the specified package (usually top-level application package). It -returns a stream object which can be used to access resource contents. -Thus, the ``resource_stream()`` emulates interface of the standard -`open()` function. - -Implementation-wise, ``resource_stream()`` uses file operations -underlyingly, if distribution package is install in the filesystem. -However, it also supports functioning without the underlying filesystem, -e.g. if the package is frozen as the bytecode. This however requires -an extra intermediate step when packaging application - creation of -"Python resource module". - -The idea of this module is to convert binary data to a Python bytes -object, and put it into the dictionary, indexed by the resource name. -This conversion is done automatically using overridden ``sdist`` command -described in the previous section. - -Let's trace the complete process using the following example. Suppose -your application has the following structure:: - - my_app/ - __main__.py - utils.py - data/ - page.html - image.png - -``__main__.py`` and ``utils.py`` should access resources using the -following calls:: - - import pkg_resources - - pkg_resources.resource_stream(__name__, "data/page.html") - pkg_resources.resource_stream(__name__, "data/image.png") - -You can develop and debug using the :term:`MicroPython Unix port` as usual. -When time comes to make a distribution package out of it, just use -overridden "sdist" command from sdist_upip.py module as described in -the previous section. - -This will create a Python resource module named ``R.py``, based on the -files declared in ``MANIFEST`` or ``MANIFEST.in`` files (any non-``.py`` -file will be considered a resource and added to ``R.py``) - before -proceeding with the normal packaging steps. - -Prepared like this, your application will work both when deployed to -filesystem and as frozen bytecode. - -If you would like to debug ``R.py`` creation, you can run:: - - python3 setup.py sdist --manifest-only - -Alternatively, you can use tools/mpy_bin2res.py script from the -MicroPython distribution, in which can you will need to pass paths -to all resource files:: - - mpy_bin2res.py data/page.html data/image.png - -References ----------- - -* Python Packaging User Guide: https://packaging.python.org/ -* Setuptools documentation: https://setuptools.readthedocs.io/ -* Distutils documentation: https://docs.python.org/3/library/distutils.html +Package management +================== + +Installing packages with ``mip`` +-------------------------------- + +Network-capable boards include the ``mip`` module, which can install packages +from :term:`micropython-lib` and from third-party sites (including GitHub). + +``mip`` ("mip installs packages") is similar in concept to Python's ``pip`` tool, +however it does not use the PyPI index, rather it uses :term:`micropython-lib` +as its index by default. ``mip`` will automatically fetch compiled +:term:`.mpy file` when downloading from micropython-lib. + +The most common way to use ``mip`` is from the REPL:: + + >>> import mip + >>> mip.install("pkgname") # Installs the latest version of "pkgname" (and dependencies) + >>> mip.install("pkgname", version="x.y") # Installs version x.y of "pkgname" + >>> mip.install("pkgname", mpy=False) # Installs the source version (i.e. .py rather than .mpy files) + +``mip`` will detect an appropriate location on the filesystem by searching +``sys.path`` for the first entry ending in ``/lib``. You can override the +destination using ``target``, but note that this path must be in ``sys.path`` to be +able to subsequently import it.:: + + >>> mip.install("pkgname", target="third-party") + >>> sys.path.append("third-party") + +As well as downloading packages from the micropython-lib index, ``mip`` can also +install third-party libraries. The simplest way is to download a file directly:: + + >>> mip.install("http://example.com/x/y/foo.py") + >>> mip.install("http://example.com/x/y/foo.mpy") + +When installing a file directly, the ``target`` argument is still supported to set +the destination path, but ``mpy`` and ``version`` are ignored. + +The URL can also start with ``github:`` as a simple way of pointing to content +hosted on GitHub:: + + >>> mip.install("github:org/repo/path/foo.py") # Uses default branch + >>> mip.install("github:org/repo/path/foo.py", version="branch-or-tag") # Optionally specify the branch or tag + +More sophisticated packages (i.e. with more than one file, or with dependencies) +can be downloaded by specifying the path to their ``package.json``. + + >>> mip.install("http://example.com/x/package.json") + >>> mip.install("github:org/user/path/package.json") + +If no json file is specified, then "package.json" is implicitly added:: + + >>> mip.install("http://example.com/x/") + >>> mip.install("github:org/repo") + >>> mip.install("github:org/repo", version="branch-or-tag") + + +Using ``mip`` on the Unix port +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +On the Unix port, ``mip`` can be used at the REPL as above, and also by using ``-m``:: + + $ ./micropython -m mip install pkgname-or-url + $ ./micropython -m mip install pkgname-or-url@version + +The ``--target=path``, ``--no-mpy``, and ``--index`` arguments can be set:: + + $ ./micropython -m mip install --target=third-party pkgname + $ ./micropython -m mip install --no-mpy pkgname + $ ./micropython -m mip install --index https://host/pi pkgname + +Installing packages with ``mpremote`` +------------------------------------- + +The :term:`mpremote` tool also includes the same functionality as ``mip`` and +can be used from a host PC to install packages to a locally connected device +(e.g. via USB or UART):: + + $ mpremote install pkgname + $ mpremote install pkgname@x.y + $ mpremote install http://example.com/x/y/foo.py + $ mpremote install github:org/repo + $ mpremote install github:org/repo@branch-or-tag + +The ``--target=path``, ``--no-mpy``, and ``--index`` arguments can be set:: + + $ mpremote install --target=/flash/third-party pkgname + $ mpremote install --no-mpy pkgname + $ mpremote install --index https://host/pi pkgname + +Installing packages manually +---------------------------- + +Packages can also be installed (in either .py or .mpy form) by manually copying +the files to the device. Depending on the board this might be via USB Mass Storage, +the :term:`mpremote` tool (e.g. ``mpremote fs cp path/to/package.py :package.py``), +:term:`webrepl`, etc. + +Writing & publishing packages +----------------------------- + +Publishing to :term:`micropython-lib` is the easiest way to make your package +broadly accessible to MicroPython users, and automatically available via +``mip`` and ``mpremote`` and compiled to bytecode. See +https://github.com/micropython/micropython-lib for more information. + +To write a "self-hosted" package that can be downloaded by ``mip`` or +``mpremote``, you need a static webserver (or GitHub) to host either a +single .py file, or a package.json file alongside your .py files. + +A typical package.json for an example ``mlx90640`` library looks like:: + + { + "urls": [ + ["mlx90640/__init__.py", "github:org/micropython-mlx90640/mlx90640/__init__.py"], + ["mlx90640/utils.py", "github:org/micropython-mlx90640/mlx90640/utils.py"] + ], + "deps": [ + ["collections-defaultdict", "latest"], + ["os-path", "latest"] + ], + "version": "0.2" + } + +This includes two files, hosted at a GitHub repo named +``org/micropython-mlx90640``, which install into the ``mlx90640`` directory on +the device. It depends on ``collections-defaultdict`` and ``os-path`` which will +be installed automatically. + +Freezing packages +----------------- + +When a Python module or package is imported from the device filesystem, it is +compiled into :term:`bytecode` in RAM, ready to be executed by the VM. For +a :term:`.mpy file`, this conversion has been done already, but the bytecode +still ends up in RAM. + +For low-memory devices, or for large applications, it can be advantageous to +instead run the bytecode from ROM (i.e. flash memory). This can be done +by "freezing" the bytecode into the MicroPython firmware, which is then flashed +to the device. The runtime performance is the same (although importing is +faster), but it can free up significant amounts of RAM for your program to +use. + +The downside of this approach is that it's much slower to develop, because you +have to flash the firmware each time, but it can be still useful to freeze +dependencies that don't change often. + +Freezing is done by writing a manifest file and using it in the build, often as +part of a custom board definition. See the :ref:`manifest` guide for more +information. diff --git a/examples/hwapi/README.md b/examples/hwapi/README.md index 1992eb6609..df16b4c86b 100644 --- a/examples/hwapi/README.md +++ b/examples/hwapi/README.md @@ -116,7 +116,7 @@ For example, one may invent a "configuration manager" helper module which will try to detect current board (among well-known ones), and load appropriate `hwconfig_*.py` - this assumes that a user would lazily deploy them all (or that application will be automatically installed, e.g. using MicroPython's -`upip` package manager). The key point in this case remains the same as +`mip` package manager). The key point in this case remains the same as elaborated above - always assume there can, and will be a custom configuration, and it should be well supported. So, any automatic detection should be overridable by a user, and instructions how to do so are among the most diff --git a/ports/esp32/boards/manifest.py b/ports/esp32/boards/manifest.py index fcc48d7218..3f6c8cfde5 100644 --- a/ports/esp32/boards/manifest.py +++ b/ports/esp32/boards/manifest.py @@ -1,11 +1,10 @@ freeze("$(PORT_DIR)/modules") -module("upip.py", base_path="$(MPY_DIR)/tools", opt=3) -module("upip_utarfile.py", base_path="$(MPY_DIR)/tools", opt=3) include("$(MPY_DIR)/extmod/uasyncio") # Require some micropython-lib modules. require("dht") require("ds18x20") +require("mip") require("neopixel") require("ntptime") require("onewire") diff --git a/ports/esp8266/README.md b/ports/esp8266/README.md index dd50fc1af9..b54d8958d7 100644 --- a/ports/esp8266/README.md +++ b/ports/esp8266/README.md @@ -200,20 +200,22 @@ Python prompt over WiFi, connecting through a browser. - GitHub repository https://github.com/micropython/webrepl. Please follow the instructions there. -__upip__ +__mip__ -The ESP8266 port comes with builtin `upip` package manager, which can -be used to install additional modules (see the main README for more -information): +The ESP8266 port comes with the built-in `mip` package manager, which can +be used to install additional modules: ``` ->>> import upip ->>> upip.install("micropython-pystone_lowmem") +>>> import mip +>>> mip.install("hmac") [...] ->>> import pystone_lowmem ->>> pystone_lowmem.main() +>>> import hmac +>>> hmac.new(b"1234567890", msg="hello world").hexdigest() ``` +See [Package management](https://docs.micropython.org/en/latest/reference/packages.html) for more +information about `mip`. + Downloading and installing packages may requite a lot of free memory, if you get an error, retry immediately after the hard reset. diff --git a/ports/esp8266/boards/manifest.py b/ports/esp8266/boards/manifest.py index e7defd0bb7..53975f6a6b 100644 --- a/ports/esp8266/boards/manifest.py +++ b/ports/esp8266/boards/manifest.py @@ -1,9 +1,8 @@ freeze("$(PORT_DIR)/modules") -module("upip.py", base_path="$(MPY_DIR)/tools", opt=3) -module("upip_utarfile.py", base_path="$(MPY_DIR)/tools", opt=3) -require("ntptime") require("dht") -require("onewire") require("ds18x20") +require("mip") require("neopixel") +require("ntptime") +require("onewire") require("webrepl") diff --git a/ports/rp2/boards/PICO_W/manifest.py b/ports/rp2/boards/PICO_W/manifest.py index 4d9eb87f20..8a74006c64 100644 --- a/ports/rp2/boards/PICO_W/manifest.py +++ b/ports/rp2/boards/PICO_W/manifest.py @@ -1,7 +1,5 @@ include("../manifest.py") -module("upip.py", base_path="$(MPY_DIR)/tools", opt=3) -module("upip_utarfile.py", base_path="$(MPY_DIR)/tools", opt=3) - +require("mip") require("ntptime") require("urequests") diff --git a/ports/unix/README.md b/ports/unix/README.md index efc68b2455..a3a0dba75a 100644 --- a/ports/unix/README.md +++ b/ports/unix/README.md @@ -24,21 +24,26 @@ Use `CTRL-D` (i.e. EOF) to exit the shell. Learn about command-line options (in particular, how to increase heap size which may be needed for larger applications): - $ ./micropython -h + $ ./build-standard/micropython -h To run the complete testsuite, use: $ make test -The Unix port comes with a builtin package manager called upip, e.g.: +The Unix port comes with a built-in package manager called `mip`, e.g.: - $ ./micropython -m upip install micropython-pystone - $ ./micropython -m pystone + $ ./build-standard/micropython -m mip install hmac -Browse available modules on -[PyPI](https://pypi.python.org/pypi?%3Aaction=search&term=micropython). -Standard library modules come from the -[micropython-lib](https://github.com/micropython/micropython-lib) project. +or + + $ ./build-standard/micropython + >>> import mip + >>> mip.install("hmac") + +Browse available modules at [micropython-lib] +(https://github.com/micropython/micropython-lib). See +[Package management](https://docs.micropython.org/en/latest/reference/packages.html) +for more information about `mip`. External dependencies --------------------- @@ -65,6 +70,5 @@ or not). If you intend to build MicroPython with additional options (like cross-compiling), the same set of options should be passed to `make deplibs`. To actually enable/disable use of dependencies, edit the `ports/unix/mpconfigport.mk` file, which has inline descriptions of the -options. For example, to build SSL module (required for the `upip` tool -described above, and so enabled by default), `MICROPY_PY_USSL` should be set -to 1. +options. For example, to build the SSL module, `MICROPY_PY_USSL` should be +set to 1. diff --git a/ports/unix/variants/manifest.py b/ports/unix/variants/manifest.py index bf7ce992ad..e7fe747cb7 100644 --- a/ports/unix/variants/manifest.py +++ b/ports/unix/variants/manifest.py @@ -1,2 +1 @@ -module("upip.py", base_path="$(MPY_DIR)/tools", opt=3) -module("upip_utarfile.py", base_path="$(MPY_DIR)/tools", opt=3) +require("mip") diff --git a/tools/upip.py b/tools/upip.py deleted file mode 100644 index 2932cca50c..0000000000 --- a/tools/upip.py +++ /dev/null @@ -1,351 +0,0 @@ -# -# upip - Package manager for MicroPython -# -# Copyright (c) 2015-2018 Paul Sokolovsky -# -# Licensed under the MIT license. -# -import sys -import gc -import uos as os -import uerrno as errno -import ujson as json -import uzlib -import upip_utarfile as tarfile - -gc.collect() - - -debug = False -index_urls = ["https://micropython.org/pi", "https://pypi.org/pypi"] -install_path = None -cleanup_files = [] -gzdict_sz = 16 + 15 - -file_buf = bytearray(512) - - -class NotFoundError(Exception): - pass - - -def op_split(path): - if path == "": - return ("", "") - r = path.rsplit("/", 1) - if len(r) == 1: - return ("", path) - head = r[0] - if not head: - head = "/" - return (head, r[1]) - - -# Expects *file* name -def _makedirs(name, mode=0o777): - ret = False - s = "" - comps = name.rstrip("/").split("/")[:-1] - if comps[0] == "": - s = "/" - for c in comps: - if s and s[-1] != "/": - s += "/" - s += c - try: - os.mkdir(s) - ret = True - except OSError as e: - if e.errno != errno.EEXIST and e.errno != errno.EISDIR: - raise e - ret = False - return ret - - -def save_file(fname, subf): - global file_buf - with open(fname, "wb") as outf: - while True: - sz = subf.readinto(file_buf) - if not sz: - break - outf.write(file_buf, sz) - - -def install_tar(f, prefix): - meta = {} - for info in f: - # print(info) - fname = info.name - try: - fname = fname[fname.index("/") + 1 :] - except ValueError: - fname = "" - - save = True - for p in ("setup.", "PKG-INFO", "README"): - # print(fname, p) - if fname.startswith(p) or ".egg-info" in fname: - if fname.endswith("/requires.txt"): - meta["deps"] = f.extractfile(info).read() - save = False - if debug: - print("Skipping", fname) - break - - if save: - outfname = prefix + fname - if info.type != tarfile.DIRTYPE: - if debug: - print("Extracting " + outfname) - _makedirs(outfname) - subf = f.extractfile(info) - save_file(outfname, subf) - return meta - - -def expandhome(s): - if "~/" in s: - h = os.getenv("HOME") - s = s.replace("~/", h + "/") - return s - - -import ussl -import usocket - -warn_ussl = True - - -def url_open(url): - global warn_ussl - - if debug: - print(url) - - proto, _, host, urlpath = url.split("/", 3) - try: - port = 443 - if ":" in host: - host, port = host.split(":") - port = int(port) - ai = usocket.getaddrinfo(host, port, 0, usocket.SOCK_STREAM) - except OSError as e: - fatal("Unable to resolve %s (no Internet?)" % host, e) - # print("Address infos:", ai) - ai = ai[0] - - s = usocket.socket(ai[0], ai[1], ai[2]) - try: - # print("Connect address:", addr) - s.connect(ai[-1]) - - if proto == "https:": - s = ussl.wrap_socket(s, server_hostname=host) - if warn_ussl: - print("Warning: %s SSL certificate is not validated" % host) - warn_ussl = False - - # MicroPython rawsocket module supports file interface directly - s.write("GET /%s HTTP/1.0\r\nHost: %s:%s\r\n\r\n" % (urlpath, host, port)) - l = s.readline() - protover, status, msg = l.split(None, 2) - if status != b"200": - if status == b"404" or status == b"301": - raise NotFoundError("Package not found") - raise ValueError(status) - while 1: - l = s.readline() - if not l: - raise ValueError("Unexpected EOF in HTTP headers") - if l == b"\r\n": - break - except Exception as e: - s.close() - raise e - - return s - - -def get_pkg_metadata(name): - for url in index_urls: - try: - f = url_open("%s/%s/json" % (url, name)) - except NotFoundError: - continue - try: - return json.load(f) - finally: - f.close() - raise NotFoundError("Package not found") - - -def fatal(msg, exc=None): - print("Error:", msg) - if exc and debug: - raise exc - sys.exit(1) - - -def install_pkg(pkg_spec, install_path): - package = pkg_spec.split("==") - data = get_pkg_metadata(package[0]) - - if len(package) == 1: - latest_ver = data["info"]["version"] - else: - latest_ver = package[1] - packages = data["releases"][latest_ver] - del data - gc.collect() - assert len(packages) == 1 - package_url = packages[0]["url"] - print("Installing %s %s from %s" % (pkg_spec, latest_ver, package_url)) - f1 = url_open(package_url) - try: - f2 = uzlib.DecompIO(f1, gzdict_sz) - f3 = tarfile.TarFile(fileobj=f2) - meta = install_tar(f3, install_path) - finally: - f1.close() - del f3 - del f2 - gc.collect() - return meta - - -def install(to_install, install_path=None): - # Calculate gzip dictionary size to use - global gzdict_sz - sz = gc.mem_free() + gc.mem_alloc() - if sz <= 65536: - gzdict_sz = 16 + 12 - - if install_path is None: - install_path = get_install_path() - if install_path[-1] != "/": - install_path += "/" - if not isinstance(to_install, list): - to_install = [to_install] - print("Installing to: " + install_path) - # sets would be perfect here, but don't depend on them - installed = [] - try: - while to_install: - if debug: - print("Queue:", to_install) - pkg_spec = to_install.pop(0) - if pkg_spec in installed: - continue - meta = install_pkg(pkg_spec, install_path) - installed.append(pkg_spec) - if debug: - print(meta) - deps = meta.get("deps", "").rstrip() - if deps: - deps = deps.decode("utf-8").split("\n") - to_install.extend(deps) - except Exception as e: - print( - "Error installing '{}': {}, packages may be partially installed".format(pkg_spec, e), - file=sys.stderr, - ) - - -def get_install_path(): - global install_path - if install_path is None: - # sys.path[0] is current module's path - install_path = sys.path[1] - if install_path == ".frozen": - install_path = sys.path[2] - install_path = expandhome(install_path) - return install_path - - -def cleanup(): - for fname in cleanup_files: - try: - os.unlink(fname) - except OSError: - print("Warning: Cannot delete " + fname) - - -def help(): - print( - """\ -upip - Simple PyPI package manager for MicroPython -Usage: micropython -m upip install [-p ] ... | -r -import upip; upip.install(package_or_list, []) - -If isn't given, packages will be installed to sys.path[1], or -sys.path[2] if the former is .frozen (path can be set from MICROPYPATH -environment variable if supported).""" - ) - print("Default install path:", get_install_path()) - print( - """\ - -Note: only MicroPython packages (usually, named micropython-*) are supported -for installation, upip does not support arbitrary code in setup.py. -""" - ) - - -def main(): - global debug - global index_urls - global install_path - install_path = None - - if len(sys.argv) < 2 or sys.argv[1] == "-h" or sys.argv[1] == "--help": - help() - return - - if sys.argv[1] != "install": - fatal("Only 'install' command supported") - - to_install = [] - - i = 2 - while i < len(sys.argv) and sys.argv[i][0] == "-": - opt = sys.argv[i] - i += 1 - if opt == "-h" or opt == "--help": - help() - return - elif opt == "-p": - install_path = sys.argv[i] - i += 1 - elif opt == "-r": - list_file = sys.argv[i] - i += 1 - with open(list_file) as f: - while True: - l = f.readline() - if not l: - break - if l[0] == "#": - continue - to_install.append(l.rstrip()) - elif opt == "-i": - index_urls = [sys.argv[i]] - i += 1 - elif opt == "--debug": - debug = True - else: - fatal("Unknown/unsupported option: " + opt) - - to_install.extend(sys.argv[i:]) - if not to_install: - help() - return - - install(to_install) - - if not debug: - cleanup() - - -if __name__ == "__main__": - main() diff --git a/tools/upip_utarfile.py b/tools/upip_utarfile.py deleted file mode 100644 index 21b899f020..0000000000 --- a/tools/upip_utarfile.py +++ /dev/null @@ -1,95 +0,0 @@ -import uctypes - -# http://www.gnu.org/software/tar/manual/html_node/Standard.html -TAR_HEADER = { - "name": (uctypes.ARRAY | 0, uctypes.UINT8 | 100), - "size": (uctypes.ARRAY | 124, uctypes.UINT8 | 11), -} - -DIRTYPE = "dir" -REGTYPE = "file" - - -def roundup(val, align): - return (val + align - 1) & ~(align - 1) - - -class FileSection: - def __init__(self, f, content_len, aligned_len): - self.f = f - self.content_len = content_len - self.align = aligned_len - content_len - - def read(self, sz=65536): - if self.content_len == 0: - return b"" - if sz > self.content_len: - sz = self.content_len - data = self.f.read(sz) - sz = len(data) - self.content_len -= sz - return data - - def readinto(self, buf): - if self.content_len == 0: - return 0 - if len(buf) > self.content_len: - buf = memoryview(buf)[: self.content_len] - sz = self.f.readinto(buf) - self.content_len -= sz - return sz - - def skip(self): - sz = self.content_len + self.align - if sz: - buf = bytearray(16) - while sz: - s = min(sz, 16) - self.f.readinto(buf, s) - sz -= s - - -class TarInfo: - def __str__(self): - return "TarInfo(%r, %s, %d)" % (self.name, self.type, self.size) - - -class TarFile: - def __init__(self, name=None, fileobj=None): - if fileobj: - self.f = fileobj - else: - self.f = open(name, "rb") - self.subf = None - - def next(self): - if self.subf: - self.subf.skip() - buf = self.f.read(512) - if not buf: - return None - - h = uctypes.struct(uctypes.addressof(buf), TAR_HEADER, uctypes.LITTLE_ENDIAN) - - # Empty block means end of archive - if h.name[0] == 0: - return None - - d = TarInfo() - d.name = str(h.name, "utf-8").rstrip("\0") - d.size = int(bytes(h.size), 8) - d.type = [REGTYPE, DIRTYPE][d.name[-1] == "/"] - self.subf = d.subf = FileSection(self.f, d.size, roundup(d.size, 512)) - return d - - def __iter__(self): - return self - - def __next__(self): - v = self.next() - if v is None: - raise StopIteration - return v - - def extractfile(self, tarinfo): - return tarinfo.subf