Merge pull request #341 from chr15m/proxy-scp-command

Proxy scp so piku user can scp, and add 'download' local cmd.

.github/workflows/core-tests.yml

@@ -4,13 +4,13 @@ on: [push, pull_request]
 
 jobs:
   build:
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-latest
     strategy:
       max-parallel: 3
       matrix:
-        # This is currently overkill since we are targeting 3.5
-        # but affords us visibility onto syntax changes in newer Pythons
-        python-version: [3.6, 3.7, 3.8, 3.9]
+        # This range ensures coding standard does not conflict with 
+        #  changes in newer Pythons
+        python-version: [3.7, 3.8, 3.9, "3.10"]
 
     steps:
     - uses: actions/checkout@v3

.github/workflows/jammy/Dockerfile

@@ -1,4 +1,4 @@
-FROM ubuntu:bionic
+FROM ubuntu:jammy
 
 ENV DEBIAN_FRONTEND=noninteractive
 

.github/workflows/os-packages.yml

@@ -8,7 +8,7 @@ jobs:
     strategy:
       max-parallel: 4
       matrix:
-        distro: [ buster, bionic, focal ]
+        distro: [ buster, jammy, focal, bullseye ]
 
     steps:
     - uses: actions/checkout@v3

CONTRIBUTING.md

@@ -1,10 +1,10 @@
-# Contributing to `piku`
+# Contributing
 
 `piku` is a stable project, but we welcome contributions that:
 
 * Help us move beyond Python 3.8+ (which is the current target due to Linux LTS distribution alignment)
 * Help us do better automated testing
-* Improve documentation (some docs are a bit old by now) 
+* Improve documentation (some docs are a bit old by now)
 * Help us deploy `piku` in various Linux distributions and environments (check the sister repositories in the project)
 * Provide sample deployments of common applications (again, check the sister repositories in the project)
 * Allow us to better support more language runtimes
@@ -12,10 +12,12 @@
 
 ## Code Size / Style
 
-By its very nature, `piku` is a very small program. By today's standards of all-encompassing solutions this may seem strange, 
-but it would benefit from being kept that way.
+By its very nature, `piku` is a very small program. By today's standards of all-encompassing solutions this may seem strange, but it would benefit from being kept that way.
+
+- Small and focused PRs. Please don't include changes that don't address the subject of your PR.
+- Follow the style of importing functions directly e.g. `from os.path import abspath`
+- Follow `PEP8`.
 
 So please keep that in mind when contributing.
 
-For instance, if your runtime or framework needs additional setup, it might be better to contribute an utility script to run 
-in a `release` entry in the `Procfile` rather than patching `piku.py` -- but do hack at it if that is the best way to achieve it.
+For instance, if your runtime or framework needs additional setup, it might be better to contribute an utility script to run in a `release` entry in the `Procfile` rather than patching `piku.py`--but do hack at it if that is the best way to achieve it.

README.md

@@ -1,6 +1,6 @@
 ![piku logo](./img/logo.png)
 
-`piku`, inspired by [dokku][dokku], allows you do `git push` deployments to your own servers, no matter how small they are.
+`piku`, inspired by `dokku`, allows you do `git push` deployments to your own servers, no matter how small they are.
 
 ## Demo
 
@@ -8,40 +8,31 @@
 
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
-### Documentation: [Using](#using-piku) | [Install](#install) | [Procfile](docs/DESIGN.md#procfile-format) | [ENV](./docs/ENV.md) | [Examples](./examples/README.md) | [Roadmap](https://github.com/piku/piku/projects/2) | [Contributing](./docs/CONTRIBUTING.md) | [LinuxConf Talk](https://www.youtube.com/watch?v=ec-GoDukHWk) | [Fast Web App Tutorial](https://github.com/piku/webapp-tutorial)
+### Documentation: [Install](#install) | [Using](#workflow) | [Procfile](https://piku.github.io/configuration/procfile.html) | [ENV](https://piku.github.io/configuration/env.html) | [Examples](https://piku.github.io/community/examples.html) | [Roadmap](https://github.com/orgs/piku/projects/2) | [Contributing](https://piku.github.io/community/contributing.html) | [LinuxConf Talk](https://www.youtube.com/watch?v=ec-GoDukHWk) | [Fast Web App Tutorial](https://github.com/piku/webapp-tutorial) | [Discussion Forum](https://github.com/piku/piku/discussions)
+
+## [Install](https://piku.github.io/install)
+
+TL;DR:
+
+```bash
+curl https://piku.github.io/get | sh
+```
+
+There are also [other installation methods](https://piku.github.io/install) available, including [`cloud-init`](https://github.com/piku/cloud-init) and [manual installation](https://piku.github.io/install).
 
 ## Project Activity
 
 **`piku` is considered STABLE**. It is actively maintained, but "actively" here means the feature set is pretty much done, so it is only updated when new language runtimes are added or reproducible bugs crop up.
 
-It is currently being refactored to require Python 3.7 or above, since even though 3.8+ is now the baseline Python 3 version in Ubuntu LTS 20.04 and Debian 11 has already moved on to 3.9, there are no substantial differences between those versions.
+It currently requires Python 3.7 or above, since even though 3.8+ is now the baseline Python 3 version in Ubuntu LTS 20.04 and Debian 11 has already moved on to 3.9, there are no substantial differences between those versions.
 
-## Deprecation Notices
+## Motivation
 
-Since most of its users run it on LTS distributions, there is no rush to introduce disruption. The current plan is to throw up a warning for older runtimes and do regression testing for 3.7, 3.8, 3.9 and 3.10 (replacing the current bracket of tests from 3.5 to 3.8), and make sure we also cover Ubuntu 22.04, Debian 11 and Fedora 37+.
-
-## Goals and Motivation
-
-I kept finding myself wanting an Heroku/CloudFoundry-like way to deploy stuff on a few ARM boards and [my Raspberry Pi cluster][raspi-cluster], but since [dokku][dokku] didn't work on ARM at the time and even `docker` can be overkill sometimes, I decided to roll my own.
+We wanted an Heroku/CloudFoundry-like way to deploy stuff on a few `ARM` boards, but since `dokku` didn't work on `ARM` at the time and even `docker` can be overkill sometimes, a simpler solution was needed.
 
 `piku` is currently able to deploy, manage and independently scale multiple applications per host on both ARM and Intel architectures, and works on any cloud provider (as well as bare metal) that can run Python, `nginx` and `uwsgi`.
 
-### Core values
-
- * Must run on low end devices.
- * Accessible to hobbyists and K-12 schools.
- * ~1500 lines readable code.
- * Functional code style.
- * Few (single?) dependencies
- * [12 factor app](https://12factor.net).
- * Simplify user experience.
- * Cover 80% of common use cases.
- * Sensible defaults for all features.
- * Leverage distro packages in Raspbian/Debian/Ubuntu (Alpine and RHEL support is WIP)
- * Leverage standard tooling (`git`, `ssh`, `uwsgi`, `nginx`).
- * Preserve backwards compatibility where possible
-
-## Using `piku`
+## Workflow
 
 `piku` supports a Heroku-like workflow:
 
@@ -53,12 +44,12 @@ I kept finding myself wanting an Heroku/CloudFoundry-like way to deploy stuff on
    * For Go, it defines a separate `GOPATH` for each app.
    * For Node, it installs whatever is in `package.json` into `node_modules`.
    * For Java, it builds your app depending on either `pom.xml` or `build.gradle` file.
+   * For Clojure, it can use either `leiningen` or the Clojure CLI and a `deps.edn` file.
    * For Ruby, it does `bundle install` of your gems in an isolated folder.
-* It then looks at a [`Procfile`](docs/DESIGN.md#procfile-format) which is [documented here](docs/DESIGN.md#procfile-format) and starts the relevant workers using [uWSGI][uwsgi] as a generic process manager.
+* It then looks at a [`Procfile`](https://piku.github.io/configuration/procfile.html) and starts the relevant workers using `uwsgi` as a generic process manager.
 * You can optionally also specify a `release` worker which is run once when the app is deployed.
 * You can then remotely change application settings (`config:set`) or scale up/down worker processes (`ps:scale`).
-* You can also bake application and `nginx` settings into an [`ENV`](./docs/ENV.md) file which is [documented here](./docs/ENV.md).
-
+* You can also bake application and `nginx` settings into an [`ENV`](https://piku.github.io/configuration/env.html) file.
 You can also deploy a `gh-pages` style static site using a `static` worker type, with the root path as the argument, and run a `release` task to do some processing on the server after `git push`.
 
 ### Virtual Hosts and SSL
@@ -73,11 +64,11 @@ If you are on a LAN and are accessing `piku` from macOS/iOS/Linux clients, you c
 
 Besides static sites, `piku` also supports directly mapping specific URL prefixes to filesystem paths (to serve static assets) or caching back-end responses (to remove load from applications).
 
-These features are configured by setting appropriate values in the [`ENV`](./docs/ENV.md) file.
+These features are configured by setting appropriate values in the [`ENV`](https://piku.github.io/configuration/env.html) file.
 
 ### Supported Platforms
 
-`piku` is intended to work in any POSIX-like environment where you have Python, `nginx`, [`uWSGI`][uwsgi] and SSH: it has been deployed on Linux, FreeBSD, [Cygwin][cygwin] and the [Windows Subsystem for Linux][wsl].
+`piku` is intended to work in any POSIX-like environment where you have Python, `nginx`, `uwsgi` and SSH: it has been deployed on Linux, FreeBSD, [Cygwin][cygwin] and the [Windows Subsystem for Linux][wsl].
 
 As a baseline, it began its development on an original 256MB Rasbperry Pi Model B, and still runs reliably on it.
 
@@ -89,61 +80,21 @@ But its main use is as a micro-PaaS to run applications on cloud servers with bo
 
 But as a general rule, if it can be invoked from a shell, it can be run inside `piku`.
 
-## Install
+### Core values
 
-`piku` can manage multiple apps on a single machine, and all you need is a VPS, Raspberry Pi, or other server.
+ * Run on low end devices.
+ * Accessible to hobbyists and K-12 schools.
+ * ~1500 lines readable code.
+ * Functional code style.
+ * Few (single?) dependencies
+ * [12 factor app](https://12factor.net).
+ * Simplify user experience.
+ * Cover 80% of common use cases.
+ * Sensible defaults for all features.
+ * Leverage distro packages in Raspbian/Debian/Ubuntu (Alpine and RHEL support is WIP)
+ * Leverage standard tooling (`git`, `ssh`, `uwsgi`, `nginx`).
+ * Preserve backwards compatibility where possible
 
-There are two main ways of deploying `piku` onto a new server:
-
-* Use [`piku-bootstrap`](https://github.com/piku/piku-bootstrap) to reconfigure a new or existing Ubuntu virtual machine.
-* Use `cloud-init` when creating a new virtual machine or barebones automated deployment (check [this repository](https://github.com/piku/cloud-init) for examples).
-
-## Manage - via the `piku` helper
-
-To make life easier you can also install the [piku](./piku) helper into your path (e.g. `~/bin`).
-
-```shell
-curl https://raw.githubusercontent.com/piku/piku/master/piku > ~/bin/piku && chmod 755 ~/bin/piku
-```
-
-This shell script simplifies working with multiple `piku` remotes and applications:
-
-* If you `cd` into a project folder that has a `git` remote called `piku` the helper will infer the remote server and app name and use them automatically:
-
-```shell
-$ piku logs
-$ piku config:set MYVAR=12
-$ piku stop
-$ piku deploy
-$ piku destroy
-$ piku # <- show available remote and local commands
-```
-
-* If you are starting a new project, `piku init` will download example `Procfile` and `ENV` files into the current folder:
-
-```shell
-$ piku init
-Wrote ./ENV file.
-Wrote ./Procfile.
-```
-
-* The `piku` helper also lets you pass settings to the underlying SSH command: `-t` to run interactive commands remotely, and `-A` to proxy authentication credentials in order to do remote `git` pulls.
-
-For instance, here's how to use the `-t` flag to obtain a `bash` shell in the app directory of one of your `piku` apps:
-
-```shell
-$ piku -t run bash
-Piku remote operator.
-Server: piku@cloud.mccormickit.com
-App: dashboard
-
-piku@piku:~/.piku/apps/dashboard$ ls
-data  ENV  index.html  package.json  package-lock.json  Procfile  server.wisp
-```
-
-[dokku]: https://github.com/dokku/dokku
 [raspi-cluster]: https://github.com/rcarmo/raspi-cluster
 [cygwin]: http://www.cygwin.com
-[uwsgi]: https://github.com/unbit/uwsgi
 [wsl]: https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux
-

docs/CONTRIBUTING.md

@@ -1,9 +0,0 @@
-We welcome contributions!
-
-### Guidelines
-
- * Small and focused PRs. Please don't include changes that don't address the subject of your PR.
- * Follow the style of importing functions directly e.g. `from os.path import abspath`.
- * Check out the [core values of Piku](../README.md#core-values).
- * PEP8.
-

docs/DESIGN.md

@@ -1,100 +0,0 @@
-# Design Notes
-
-The idea behind `piku` is that it provides the simplest possible way to deploy web apps or services. Simplicity comes at the expense of features, of course, and this document tries to capture the trade-offs.
-
-## Why uWSGI
-
-Using [uWSGI][uwsgi] in [emperor mode][emperor] gives us the following features for free:
-
-* Painless Python WSGI and `virtualenv` integration
-* Process monitoring, restarting, basic resource limiting, etc.
-* Basic security scaffolding, beginning with the ability to define `uid`/`gid` on a per-app basis (if necessary)
-
-## Application packaging
-
-An app is simply a `git` repository with some additional files on the top level, the most important of which is the `Procfile`.
-
-### `Procfile` format
-
-`piku` recognizes five kinds of process declarations in the `Procfile`:
-
-* `wsgi` workers, in the format `dotted.module:entry_point` (Python-only)
-* `web` workers, which can be anything that honors the `PORT` environment variable
-* `static` workers, which simply mount the first argument as the root static path
-* `cron` workers, which require a simplified `cron` expression preceding the command to be run (e.g. `cron: */5 * * * * python batch.py` to run a batch every 5 minutes)
-* `worker` processes, which are standalone workers and can have arbitrary names
-
-So a Python application could have a `Procfile` like such:
-
-```bash
-wsgi: module.submodule:app
-worker: python long_running_script.py 
-fetcher: python fetcher.py
-# Simple cron expression: minute [0-59], hour [0-23], day [0-31], month [1-12], weekday [1-7] (starting Monday, no ranges allowed on any field)
-cron: 0 0 * * * python midnight_cleanup.py
-```
-
-...whereas a generic app would be:
-
-```bash
-web: embedded_server --port $PORT
-worker: background_worker
-```
-
-Any worker will be automatically respawned upon failure ([uWSGI][uwsgi] will automatically shun/throttle crashy workers).
-
-## `ENV` settings
-
-Since `piku` is targeted at [12 Factor apps][12f], it allows you to set environment variables in a number of ways, the simplest of which is by adding an `ENV` file to your repository:
-
-```bash
-SETTING1=foo
-# piku supports comments and variable expansion
-SETTING2=${SETTING1}/bar
-# if this isn't defined, piku will assign a random TCP port
-PORT=9080
-```
-
-See [ENV.md](./ENV.md) for a full list of Piku variables which can also be set.
-
-Environment variables can be changed after deployment using `config:set`.
-
-## Runtime detection
-
-`piku` follows a very simple set of rules to determine what kind of runtime is required:
-
-1. If there's a `requirements.txt` file at the top level, then the app is assumed to require Python.
-2. _TODO: Go_
-3. _TODO: Node_
-4. _TODO: Java_
-2. For all the rest, a `Procfile` is required to determine application entry points. 
-
-
-## Application isolation
-
-Application isolation can be tackled at several levels, the most relevant of which being:
-
-* OS/process isolation
-* Runtime/library isolation
-
-For 1.0, all applications run under the same `uid`, under separate branches of the same filesystem, and without any resource limiting.
-
-Ways to improve upon that (short of full containerisation) typically entail the use of a `chroot` jail environment (which is available under most POSIX systems in one form or another) or Linux kernel namespaces - both of which are supported by [uWSGI][uwsgi] (which can also handle resource limiting to a degree).
-
-As to runtime isolation, `piku` only provides `virtualenv` support until 1.0. Python apps can run under Python 2 or 3 depending on the setting of `PYTHON_VERSION`, but will always use pre-installed interpreters (Go, Node and Java support will share these limitations in each major version).
-
-## Internals
-
-`piku` uses two `git` repositories for each app: a bare repository for client push, and a clone for deployment (which is efficient in terms of storage since `git` tries to use hardlinks on local clones whenever possible).
-
-This separation makes it easier to cope with long/large deployments and restore apps to a pristine condition, since the app will only go live after the deployment clone is reset (via `git checkout -f`).
-
-## Components
-
-This diagram (available as a `dot` file in the `img` folder) outlines how its components interact:
-
-![](../img/piku.png)
-
-[uwsgi]: https://github.com/unbit/uwsgi
-[emperor]: http://uwsgi-docs.readthedocs.org/en/latest/Emperor.html
-[12f]: http://12factor.net

docs/FAQ.md

@@ -2,23 +2,21 @@
 
 **Q:** Why `piku`?
 
-**A:** Partly because it's supposed to run on a [Pi][pi], because it's Japanese onomatopeia for 'twitch' or 'jolt', and because I know the name will annoy some of my friends.
+**A:** Partly because it's started out on the [Raspberry Pi][pi], because it's Japanese onomatopeia for 'twitch' or 'jolt', and because we knew the name would be cute and amusing.
 
 **Q:** Why Python/why not Go?
 
-**A:** I actually thought about doing this in Go right off the bat, but [click][click] is so cool and I needed to have [uWSGI][uwsgi] running anyway, so I caved in. But I'm very likely to take something like [suture](https://github.com/thejerf/suture) and port this across, doing away with [uWSGI][uwsgi] altogether.
+**A:** We actually thought about doing this in Go right off the bat, but [click][click] is so cool and we needed to have `uwsgi` running anyway, so we caved in. But possible future directions are likely to take something like [suture](https://github.com/thejerf/suture) and port this across (or just use [Caddy](http://caddyserver.com)), doing away with `uwsgi` altogether.
 
-Go also (at the time) did not have a way to vendor dependencies that I was comfortable with, and that is also why Go support fell behind. Hopefully that will change soon.
+Go also (at the time) did not have a way to vendor dependencies that we were comfortable with, and that is also why Go support fell behind. Hopefully that will change soon.
 
 **Q:** Does it run under Python 3?
 
-**A:** Right now, it _only_ runs on Python 3, even though it can deploy apps written in both major versions. It began its development using 2.7 and using`click` for abstracting the simpler stuff, and I eventually switched over to 3.5 once it was supported in Debian Stretch and Raspbian since I wanted to make installing it on the Raspberry Pi as simple as possible.
+**A:** Right now, it _only_ runs on Python 3, even though it can deploy apps written in both major versions. It began its development using 2.7 and using`click` for abstracting the simpler stuff, and we eventually switched over to 3.5 once it was supported in Debian Stretch and Raspbian since we wanted to make installing it on the Raspberry Pi as simple as possible.
 
 **Q:** Why not just use `dokku`?
 
-**A:** I used `dokku` daily for most of my personal stuff for a good while. But it relied on a number of `x64` containers that needed to be completely rebuilt for ARM, and when I decided I needed something like this (March 2016) that was barely possible - `docker` itself was not fully baked for ARM yet, and people were at the time trying to get `herokuish` and `buildstep` to build on ARM.
-
-
+**A:** We used `dokku` daily for many projects. But it relied on a number of `x64` containers that needed to be completely rebuilt for `ARM`, and when we decided we needed something like this (March 2016) that was barely possible - `docker` itself was not fully baked for `ARM` yet, and people were at the time just starting to get `herokuish` and `buildstep` to build on `ARM`.
 
 [click]: http://click.pocoo.org
 [pi]: http://www.raspberrypi.org

docs/INSTALL.md

@@ -1,72 +1,48 @@
 # Installation
 
-As of June 2018, these documents are being split off into distribution-specific files in the same folder. If you are running `piku` on specific Linux versions, feel free to contribute your own instructions.
+## TL;DR:
 
-> Please use distro-specific instructions whenever possible, since there have been recent improvements in `uwsgi` packaging that greatly simplify installation. 
+To install it on your server, `ssh` in as `root` and run this:
 
-Also, `piku` now **requires Python 3** and a Linux distribution that runs `systemd`, such as Raspbian Jessie/Debian 8+/Ubuntu
+```bash
+curl https://piku.github.io/get | sh
+```
 
-These generic installation notes should cover most Debian Linux variants (on any architecture). Very minor changes should be required to deploy on RHEL variants like CentOS, and there is specific emphasis on Raspbian because that's the typical deployment target.
+## Installation Methods
 
-You can, however, run `piku` on any POSIX-like environment where you have Python, [uWSGI][uwsgi] and SSH.
+`piku` requires `Python 3`, [uWSGI][uwsgi], `ssh`, and a Linux distribution that runs `systemd`, such as Raspbian Jessie/Debian 8+/Ubuntu/Fedora/CentOS.
 
-For installation, you only require `root`/`sudo` access and the following files:
+There are 3 main ways to install `piku` on a server:
 
-* `piku.py`
-* `uwsgi-piku.service`
-* `piku-nginx.path`
-* `piku-nginx.service`
-* `nginx-default.dist`
+1. Use [piku-bootstrap](https://github.com/piku/piku-bootstrap) to do it if your server is already provisioned (that is what the TL;DR command does)
 
-Copy them across to the machine you'll be using as a server before you get started with the rest.
+2. Use `cloud-init` to do it automatically at VPS build time (see the [`cloud-init`](https://github.com/piku/cloud-init) repository, which has examples for most common cloud providers)
 
-## Setting up the `piku` user (Debian Linux, any architecture)
+3. Manually: Follow the guide below or one of the platform-specfic guides. 
+
+There is also an [Ansible playbook](https://github.com/piku/ansible-setup).
+
+!!! Contributing
+    If you are running `piku` on specific Linux versions, feel free to contribute your own instructions.
+
+## Generic Installation Steps
+
+### Set up the `piku` user
 
 `piku` requires a separate user account to run. To create a new user with the right group membership (we're using the built-in `www-data` group because it's generally thought of as a less-privileged group), enter the following command:
 
 ```bash
-sudo adduser --disabled-password --gecos 'PaaS access' --ingroup www-data piku
+# pick a username
+export PAAS_USERNAME=piku
+# create it
+sudo adduser --disabled-password --gecos 'PaaS access' --ingroup www-data $PAAS_USERNAME
+# copy & setup piku.py
+sudo su - $PAAS_USERNAME -c "wget https://raw.githubusercontent.com/piku/piku/master/piku.py && python3 ~/piku.py setup"
 ```
 
-This user _is not supposed to login to your system_. Instead, you'll interact with `piku` via SSH, and set things up by using `su`:
+The `setup` output should be something like this:
 
-```bash
-sudo su - piku
-mkdir ~/.ssh
-chmod 700 ~/.ssh
-# now copy the piku script to this user account
-cp /tmp/piku.py ~/piku.py
 ```
-
-## Dependencies
-
-Before running `piku` for the first time, you need to install the following Python packages at the system level:
-
-### Raspbian Jessie, Debian 8, Ubuntu 16.04
-
-```bash
-sudo apt-get install git python3-virtualenv python3-pip
-sudo pip3 install -U click
-```
-
-### Raspbian Wheezy
-
-```bash
-sudo apt-get install git python3
-sudo easy_install3 -U pip3
-sudo pip3 install -U click virtualenv
-```
-
-These may or may not be installed already (`click` usually isn't). For Raspbian Wheezy this is the preferred approach, since current `apt` packages are fairly outdated.
-
-## Initialization
-
-To set everything up, type `python piku.py setup`:
-
-```bash
-sudo su - piku
-python3 piku.py setup
-
 Creating '/home/piku/.piku/apps'.
 Creating '/home/piku/.piku/repos'.
 Creating '/home/piku/.piku/envs'.
@@ -77,9 +53,9 @@ Creating '/home/piku/.piku/logs'.
 Setting '/home/piku/piku.py' as executable.
 ```
 
-## Setting up SSH access
+### Set up `ssh` access
 
-If you don't have an SSH public key (or never used one before), you need to create one. The following instructions assume you're running some form of UNIX on your own machine (Windows users should check the documentation for their SSH client, unless you have [Cygwin][cygwin] installed).
+If you don't have an `ssh` public key (or never used one before), you need to create one. The following instructions assume you're running some form of UNIX on your own machine (Windows users should check the documentation for their `ssh` client, unless you have [Cygwin][cygwin] installed).
 
 **On your own machine**, issue the `ssh-keygen` command and follow the prompts:
 
@@ -101,8 +77,6 @@ The key's randomart image is:
 +-----------------+
 ```
 
-## Adding the key to `piku`
-
 Copy the resulting `id_rsa.pub` (or equivalent, just make sure it's the _public_ file) to your `piku` server and do the following:
 
 ```bash
@@ -121,9 +95,9 @@ cat .ssh/authorized_keys
 command="FINGERPRINT=85:29:07:cb:de:ad:be:ef:42:65:00:c8:d2:6b:9e:ff NAME=default /home/piku/piku.py $SSH_ORIGINAL_COMMAND",no-agent-forwarding,no-user-rc,no-X11-forwarding,no-port-forwarding ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDhTYZi/qeJBKgU3naI8FNjQgeMYnMsEtqrOmUc4lJoPNH2qBUTNkzwThGqsBm2HNLPURWMiEifBqF+kRixMud67Co7Zs9ys7pwFXkJB9bbZasd2JCGfVZ4UYXHnvgejSWkLAV/4bObhsbP2vWOmbbm91Cwn+PGJgoiW08yrd45lsDmgv9cUAJS3e8LkgVELvIDg49yM5ArB88oxwMEoUgWU2OniHmH0o1zw5I8WXHRhHOjb8cGsdTYfXEizRKKRTM2Mu6dKRt1GNL0UbWi8iS3uJHGD3AcQ4ApdMl5X0gTixKHponStOrSMy19/ltuIy8Sjr7KKPxz07ikMYr7Vpcp youruser@yourlaptop.lan
 ```
 
-This line is what enables you to SSH (and perform `git` over SSH operations) to the `piku` user without a password, verifying your identity via your public key, restricting what can be done remotely and passing on to `piku` itself the commands you'll be issuing.
+This line is what enables you to `ssh` (and perform `git` over `ssh` operations) to the `piku` user without a password, verifying your identity via your public key, restricting what can be done remotely and passing on to `piku` itself the commands you'll be issuing.
 
-## Testing
+### Test
 
 From your machine, do:
 
@@ -156,193 +130,5 @@ Commands:
 Connection to pi.lan closed.
 ```
 
-And that's it, you're set. Now to configure [uWSGI][uwsgi], which is what `piku` relies upon to manage your apps at runtime.
-
-## uWSGI Installation (Debian Linux variants, any architecture)
-
-[uWSGI][uwsgi] can be installed in a variety of fashions. These instructions cover both pre-packaged and source installs depending on your system.
-
-### Raspbian Jessie, Debian 8
-
-> **Warning**
-> 
-> These OS releases are no longer supported and these instructions are kept for reference purposes only.
-
-In Raspbian Jessie, Debian 8 and other `systemd` distributions where [uWSGI][uwsgi] is already available pre-compiled (but split into a number of plugins), do the following:
-
-```bash
-# At the time of this writing, this installs uwsgi 2.0.7 on Raspbian Jessie.
-# You can also install uwsgi-plugins-all if you want to get runtime support for other languages
-sudo apt-get install uwsgi uwsgi-plugin-python3
-# refer to our executable using a link, in case there are more versions installed
-sudo ln -s `which uwsgi` /usr/local/bin/uwsgi-piku
-# disable the standard uwsgi startup script
-sudo systemctl disable uwsgi
-
-# add our own startup script
-sudo cp /tmp/uwsgi-piku.service /etc/systemd/system/
-sudo systemctl enable uwsgi-piku
-sudo systemctl start uwsgi-piku
-
-# check it's running
-sudo systemctl status uwsgi-piku.service
-```
-**Important Note:** Make sure you run `piku.py setup` as outlined above before starting the service.
-
-Also, please note that `uwsgi-piku.service`, as provided, creates a `/run/uwsgi-piku` directory for it to place socket files and sundry. This is not actually used at the moment, since the `uwsgi` socket file is placed inside the `piku` user directory for consistency across OS distributions. This will be cleaned up in a later release.
-
-### Raspbian Wheezy
-
-> **Warning**
-> 
-> This OS release is no longer supported and these instructions are kept for reference purposes only.
-
-Since Raspbian Wheezy is a fairly old distribution by now, its `uwsgi-*` packages are completely outdated (and depend on Python 2.6), so we have to compile and install our own version, as well as using an old-style `init` script to have it start automatically upon boot.
-
-```bash
-sudo apt-get install build-essential python-dev libpcre3-dev
-# At the time of this writing, this installs 2.0.12
-sudo pip install uwsgi
-# refer to our executable using a link, in case there are more versions installed
-sudo ln -s `which uwsgi` /usr/local/bin/uwsgi-piku
-
-# set up our init script
-sudo cp /tmp/uwsgi-piku.dist /etc/init.d/uwsgi-piku
-sudo chmod +x /etc/init.d/uwsgi-piku
-sudo update-rc.d uwsgi-piku defaults
-sudo service uwsgi-piku start
-```
-**Important Note:** Make sure you run `python3 piku.py setup` as outlined above before starting the service.
-
-### Ubuntu 14.04 LTS
-
-> **Warning**
-> 
-> This OS release is no longer supported and these instructions are kept for reference purposes only.
-
-This is a mix of both the above, and should change soon when we get 16.04. If you have trouble, install [uWSGI][uwsgi] via `pip` instead.
-
-```bash
-# At the time of this writing, this installs uwsgi 1.9.17 on Ubuntu 14.04 LTS.
-# You can also install uwsgi-plugins-all if you want to get runtime support for other languages
-sudo apt-get install uwsgi uwsgi-plugin-python3
-# refer to our executable using a link, in case there are more versions installed
-sudo ln -s `which uwsgi` /usr/local/bin/uwsgi-piku
-
-# set up our init script
-sudo cp /tmp/uwsgi-piku.dist /etc/init.d/uwsgi-piku
-sudo chmod +x /etc/init.d/uwsgi-piku
-sudo update-rc.d uwsgi-piku defaults
-sudo service uwsgi-piku start
-```
-
-## nginx Installation (Raspbian 8, Ubuntu 16.04)
-
-> **Warning**
-> 
-> These OS releases are no longer supported and these instructions are kept for reference purposes only.
-
-*PLEASE NOTE:* There is a bug in `nginx` 1.6.2 under Raspbian 8 that causes it to try to allocate around a gigabyte of RAM when using SSL with SPDY. I seriously recommend using Ubuntu instead, if you can, or disabling SSL altogether.
-
-```bash
-sudo apt-get install nginx
-# Set up nginx to pick up our config files
-sudo cp /tmp/nginx.default.dist /etc/nginx/sites-available/default
-# Set up systemd.path to reload nginx upon config changes
-sudo cp ./piku-nginx.{path, service} /etc/systemd/system/
-sudo systemctl enable piku-nginx.{path,service}
-sudo systemctl start piku-nginx.path
-# Check the status of piku-nginx.service
-systemctl status piku-nginx.path # should return `Active: active (waiting)`
-# Restart NGINX
-sudo systemctl restart nginx
-```
-
-## Java 8 Installation (All Debian Linux variants, on Raspberry Pi)
-
-> **Warning**
-> 
-> OpenJDK 8 is no longer shipping with most distributions and these instructions are kept for reference purposes only.
-
-To be able to deploy Java apps, we're going to need to install Java (and, since we're going to be doing so on ARM, it's best to use Oracle's runtime). To do that, we're going to use the `webupd8team` PPA, which has a (cross-platform) Java installer.
-
-First, get rid of OpenJDK and import the PPA key:
-
-```bash
-sudo apt-get remove openjdk*
-sudo apt-key adv --recv-key --keyserver keyserver.ubuntu.com EEA14886
-```
-
-### Raspbian Jessie
-
-> **Warning**
-> 
-> This OS release is no longer supported and these instructions are kept for reference purposes only.
-
-For Jessie, we're going to use the `trusty` version of the installer:
-
-```bash
-sudo tee /etc/apt/sources.list.d/webupd8team.list
-deb http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main 
-deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main
-^D
-```
-
-### Ubuntu 16.04 for ARM
-
-> **Warning**
-> 
-> This OS release is no longer supported and these instructions are kept for reference purposes only.
-
-For Xenial, we're going to use its own version:
-
-```bash
-sudo tee /etc/apt/sources.list.d/webupd8team.list
-deb http://ppa.launchpad.net/webupd8team/java/ubuntu xenial main 
-deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu xenial main
-^D
-```
-
-Now perform the actual install:
-
-```bash
-sudo apt-get update
-sudo apt-get install oracle-java8-installer oracle-java8-set-default
-```
-
-## Go Installation (All Debian Linux variants, on Raspberry Pi)
-
-> This is **EXPERIMENTAL** and may not work at all.
-
-### Raspbian Wheezy/Jessie
-
-> **Warning**
-> 
-> Wheezy and Jessie are no longer supported and these instructions are kept for reference purposes only.
-
-Since Raspbian's Go compiler is version 1.0.2, we need something more up-to-date.
-
-1. Get an [ARM 6 binary tarball][goarm]
-2. Unpack it under the `piku` user like such:
-
-```bash
-sudo su - piku
-tar -zxvf /tmp/go1.5.3.linux-arm.tar.gz
-# remove unnecessary files
-rm -rf go/api go/blog go/doc go/misc go/test
-```
-
-3. Give it a temporary `GOPATH` and install `godep`:
-
-```bash
-sudo su - piku
-GOROOT=$HOME/go GOPATH=$HOME/gopath PATH=$PATH:$HOME/go/bin go get github.com/tools/godep
-# temporary workaround until this is fixed in godep or Go 1.7(?)
-GOROOT=$HOME/go GOPATH=$HOME/gopath PATH=$PATH:$HOME/go/bin go get golang.org/x/sys/unix
-```
-
-_TODO: complete this._
-
-[goarm]: http://dave.cheney.net/unofficial-arm-tarballs
 [uwsgi]: https://github.com/unbit/uwsgi
-[cygwin]: http://www.cygwin.com
+[cygwin]: http://www.cygwin.com

docs/PLUGINS.md

@@ -1,41 +0,0 @@
-Thanks to [jsenin](https://github.com/jsenin), `piku` currently has experimental support for external plugins via [#129](https://github.com/piku/piku/pull/129). 
-
-Plugins are inserted into the commands listing and can perform arbitrary actions. At this moment there are no official plugins, but here is an example file that should be placed at `~/.piku/plugins/postgres/__init__.py` that could contain the commands to manage a Postgres database:
-
-```python
-import click
-
-@click.group()
-def postgres():
-    """Postgres command plugin"""
-    pass
-
-@postgres.command("postgres:create")
-@click.argument('name')
-@click.argument('user')
-@click.argument('password')
-def postgres_create():
-    """Postgres create a database"""
-    pass
-
-@postgres.command("postgres:drop")
-@click.argument('name')
-def postgres_drop():
-    """Postgres drops a database"""
-    pass
-
-@postgres.command("postgres:import")
-@click.argument('name')
-def postgres_drop():
-    """Postgres import a database"""
-    pass
-
-@postgres.command("postgres:dump")
-@click.argument('name')
-def postgres_drop():
-    """Postgres dumps a database SQL"""
-    pass
-
-def cli_commands():
-    return postgres
-```

docs/README.md

@@ -0,0 +1,3 @@
+# Documentation
+
+There is now a [documentation site](https://piku.github.io) available, which is built from the `piku.github.io` repository. This directory holds only a small subset of the documentation.

docs/old/RASPBERRY_PI_QUICKSTART.md

@@ -1,155 +0,0 @@
-# Setting up a Raspberry Pi Piku Server from Scratch
-
-## DISCLAIMER!
-
-### These instructions are correct as of April 1st 2016
-
-Start by flashing a SD card with [the latest Raspbian Jessie Lite image](https://www.raspberrypi.org/downloads/raspbian/).
-
-# Do this in your Raspberry Pi as 'pi' user
-
-Boot it, launch *raspi-config* to perform (at least) the following configuration:
-
-```bash
-# as 'pi' user
-sudo raspi-config
-```
-
-* 1) expand filesystem 
-* 2) change default user password
-* 3) set up memory split as you wish (for a headless server, 16MB for GPU)
-
-Optionally:
-
-* 4) set up over-clocking.
-
-# Secure your install
-
-Delete the existing SSH keys and recreate them (why? [read this](https://www.raspberrypi.org/forums/viewtopic.php?t=126892)).
-
-```bash
-# as 'pi' user
-sudo rm -v /etc/ssh/ssh_host_*
-sudo dpkg-reconfigure openssh-server
-sudo reboot
-```
-
-This will recreate the server keys. Next, update your system:
-
-```bash
-# as 'pi' user
-sudo apt update
-sudo apt upgrade
-```
-
-# Install required packages
-
-As of April 2016, the shipping versions with Raspbian are recent enough to run `piku`:
-
-```bash
-# as 'pi' user
-sudo apt install -y python-virtualenv python-pip git uwsgi uwsgi-plugin-python nginx
-sudo pip install -U click
-sudo reboot
-```
-
-# Meanwhile, go get the goodies while Raspberry Pi is rebooting
-
-(We assume you know about ssh keys and have one "at hand", you'll need to copy it)
-
-Clone the [piku repo](https://github.com/piku/piku) somewhere and copy files to your Raspberry Pi
-
-```bash
-# as yourself in your desktop/laptop computer
-scp piku.py uwsgi-piku.service nginx.default.dist pi@your_machine:/tmp
-scp your_public_ssh_key.pub pi@your_machine:/tmp
-```
-
-# Back to the Pi
-
-Prepare uWSGI (part one):
-```bash
-# as 'pi' user
-sudo ln -s `which uwsgi` /usr/local/bin/uwsgi-piku
-sudo systemctl disable uwsgi
-sudo cp /tmp/uwsgi-piku.service /etc/systemd/system/
-sudo systemctl daemon-reload
-sudo systemctl enable uwsgi-piku
-```
-
-Prepare nginx:
-
-```bash
-sudo apt-get install nginx
-# Set up nginx to pick up our config files
-sudo cp /tmp/nginx.default.dist /etc/nginx/sites-available/default
-# Set up systemd.path to reload nginx upon config changes
-sudo cp ./piku-nginx.{path, service} /etc/systemd/system/
-sudo systemctl enable piku-nginx.{path,service}
-sudo systemctl start piku-nginx.path
-# Check the status of piku-nginx.service
-systemctl status piku-nginx.path # should return `Active: active (waiting)`
-# Restart NGINX
-sudo systemctl restart nginx
-```
-
-Create 'piku' user and set it up
-
-```bash
-# as 'pi' user
-sudo adduser --disabled-password --gecos 'PaaS access' --ingroup www-data piku
-sudo su - piku
-# this is now done as 'piku' user
-mkdir ~/.ssh
-chmod 700 ~/.ssh
-cp /tmp/piku.py ~/piku.py
-python piku.py setup
-python piku.py setup:ssh /tmp/id_rsa.pub
-# return to 'pi' user
-exit
-```
-
-Prepare uWSGI (part two):
-
-```bash
-# as 'pi' user
-sudo systemctl start uwsgi-piku
-sudo systemctl status uwsgi-piku.service
-```
-
-
-# If you're still here, odds are your Pi is ready for work
-
-Go back to your machine and try these commands:
-
-```bash
-# as yourself in your desktop/laptop computer
-ssh piku@your_machine
-
-Usage: piku.py [OPTIONS] COMMAND [ARGS]...
-
-  The smallest PaaS you've ever seen
-
-Options:
-  --help  Show this message and exit.
-
-Commands:
-  apps              List applications
-  config            Show application configuration
-  config:get        Retrieve a configuration setting
-  config:live       Show live configuration settings
-  config:set        Set a configuration setting
-  deploy            Deploy an application
-  destroy           Destroy an application
-  disable           Disable an application
-  enable            Enable an application
-  logs              Tail an application log
-  ps                Show application worker count
-  ps:scale          Show application configuration
-  restart           Restart an application
-  setup             Initialize paths
-  setup:ssh         Set up a new SSH key
-Connection to your_machine closed.
-```
-
-If you find any bugs with this quickstart guide, please let [Luis Correia](http://twitter.com/luisfcorreia) know ;)

docs/old/ROADMAP.md

@@ -1,58 +0,0 @@
-# Roadmap
-
-This was the original roadmap, filed here in 2019-11-21 for future reference:
-
-- [ ] Prebuilt Raspbian image with everything baked in
-- [ ] `chroot`/namespace isolation (tentative)
-- [ ] Relay commands to other nodes
-- [ ] Proxy deployments to other nodes (build on one box, deploy to many)
-- [ ] Support Clojure/Java deployments through `boot` or `lein`
-- [ ] Sample Go app
-- [ ] Support Go deployments (in progress)
-- [ ] nginx SSL optimization/cypher suites, own certificates
-- [ ] Review deployment messages
-- [ ] WIP: Review docs/CLI command documentation (short descriptions done, need `help <cmd>` and better descriptions)
-- [ ] Lua/WSAPI support
-- [x] Support for Java Apps with maven/gradle (in progress through jwsgi, by @matrixjnr)
-- [x] Django and Wisp examples (by @chr15m)
-- [x] Project logo (by @chr15m)
-- [x] Various release/deployment improvements (by @chr15m)
-- [x] Support Node deployments (by @chr15m)
-- [x] Let's Encrypt support (by @chr15m)
-- [x] Allow setting `nginx` IP bindings in `ENV` file (`NGINX_IPV4_ADDRESS` and `NGINX_IPV6_ADDRESS`)
-- [x] Cleanups to remove 2.7 syntax internally
-- [x] Change to Python 3 runtime as default, with `PYTHON_VERSION = 2` as fallback
-- [x] Run in Python 3 only
-- [x] (experimental) REPL in `feature/repl`
-- [x] Python 3 support through `PYTHON_VERSION = 3`
-- [x] static URL mapping to arbitrary paths (hat tip to @carlosefr for `nginx` tuning)
-- [x] remote CLI (requires `ssh -t`)
-- [x] saner uWSGI logging
-- [x] `gevent` activated when `UWSGI_GEVENT = <integer>`
-- [x] enable CloudFlare ACL when `NGINX_CLOUDFLARE_ACL = True`
-- [x] Autodetect SPDY/HTTPv2 support and activate it
-- [x] Basic nginx SSL config with self-signed certificates and UNIX domain socket connection
-- [x] nginx support - creates an nginx config file if `NGINX_SERVER_NAME` is defined
-- [x] Testing with pre-packaged [uWSGI][uwsgi] versions on Debian Jessie (yes, it was painful)
-- [x] Support barebones binary deployments
-- [x] Complete installation instructions (see `INSTALL.md`, which also has a draft of Go installation steps)
-- [x] Installation helper/SSH key setup
-- [x] Worker scaling
-- [x] Remote CLI commands for changing/viewing applied/live settings
-- [x] Remote tailing of all logfiles for a single application
-- [x] HTTP port selection (and per-app environment variables)
-- [x] Sample Python app
-- [X] `Procfile` support (`wsgi` and `worker` processes for now, `web` processes being tested)
-- [x] Basic CLI commands to manage apps
-- [x] `virtualenv` isolation
-- [x] Support Python deployments
-- [x] Repo creation upon first push
-- [x] Basic understanding of [how `dokku` works](http://off-the-stack.moorman.nu/2013-11-23-how-dokku-works.html)
-
-[click]: http://click.pocoo.org
-[pi]: http://www.raspberrypi.org
-[dokku]: https://github.com/dokku/dokku
-[raspi-cluster]: https://github.com/rcarmo/raspi-cluster
-[cygwin]: http://www.cygwin.com
-[uwsgi]: https://github.com/unbit/uwsgi
-[wsl]: https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux

docs/old/SCRIPTS.md

@@ -1,43 +0,0 @@
-# Ubuntu 16.04
-
-Full installation sequence on a blank Ubuntu 16.04 machine:
-
-```bash
-sudo apt-get update
-sudo apt-get -y dist-upgrade
-sudo apt-get -y autoremove
-sudo apt-get install -y tmux vim htop fail2ban uwsgi uwsgi-plugin-python uwsgi-plugin-python3 uwsgi-plugin-asyncio-python3 uwsgi-plugin-gevent-python uwsgi-plugin-tornado-python nginx libxml2-dev libxslt1-dev python-dev zlib1g-dev build-essential git python3-virtualenv python3-pip python3-click
-sudo pip3 install -U click pip
-sudo adduser --disabled-password --gecos 'PaaS access' --ingroup www-data piku
-
-# move to /tmp and grab our distribution files
-cd /tmp
-wget https://raw.githubusercontent.com/piku/piku/master/piku.py
-wget https://raw.githubusercontent.com/piku/piku/master/piku-nginx.path
-wget https://raw.githubusercontent.com/piku/piku/master/piku-nginx.service
-wget https://raw.githubusercontent.com/piku/piku/master/nginx.default.dist
-wget https://raw.githubusercontent.com/piku/piku/master/uwsgi-piku.service
-# Set up nginx to pick up our config files
-sudo cp /tmp/nginx.default.dist /etc/nginx/sites-available/default
-# Set up systemd.path to reload nginx upon config changes
-sudo cp ./piku-nginx.{path, service} /etc/systemd/system/
-sudo systemctl enable piku-nginx.{path,service}
-sudo systemctl start piku-nginx.path
-# Restart NGINX
-sudo systemctl restart nginx
-sudo cp /tmp/uwsgi-piku.service /etc/systemd/system/
-# refer to our executable using a link, in case there are more versions installed
-sudo ln -s `which uwsgi` /usr/local/bin/uwsgi-piku
-# disable the standard uwsgi startup script
-sudo systemctl disable uwsgi
-sudo systemctl enable uwsgi-piku
-sudo su - piku
-mkdir ~/.ssh
-chmod 700 ~/.ssh
-# now copy the piku script to this user account
-cp /tmp/piku.py ~/piku.py
-python3 piku.py setup
-# Now import your SSH key using setup:ssh
-
-sudo systemctl start uwsgi-piku
-```

docs/raspbian-9.4-stretch-10-buster.md

@@ -1,122 +0,0 @@
-# Installation on Raspbian Stretch or Buster
-
-> This is a standalone, distribution-specific version of `INSTALL.md`. You do not need to read or follow the original file, but can refer to it for generic steps like setting up SSH keys (which are assumed to be common knowledge here)
-
-`piku` setup is simplified in modern Debian versions, since it can take advantage of some packaging improvements in [uWSGI][uwsgi] and does not require a custom `systemd` service. However, Stretch still ships with Python 3.5, which means it's not an ideal environment for new deployments on both Intel and ARM devices (Buster, in turn, ships with Python 3.7).
-
-## Setting up your Raspberry Pi
-
-Download and install [Raspbian](https://www.raspberrypi.org/downloads/raspbian/) onto an SD card.
-
-After you install it is recommended that you do the following to update your installation to the latest available software.
-
-```bash
-# update apt-get
-sudo apt-get update
-
-# upgrade all software
-sudo apt-get upgrade
-```
-
-Configure your installation.  It is recommended that `Change Password` from the default and setup `Locale Options` (Locale and Timezone) and `EXPAND FILESYSTEM`.  You will also want to `Enable SSH`.
-```bash
-# configure your installation
-sudo raspi-config
-```
-
-At this point it is a good idea to `sudo shutdown -h now` and make a backup image of the card.
-
-## Dependencies
-
-Before installing `piku`, you need to install the following packages:
-
-```bash
-sudo apt-get install -y build-essential certbot git \
-    libjpeg-dev libxml2-dev libxslt1-dev zlib1g-dev nginx \
-    python-certbot-nginx python-dev python-pip python-virtualenv \
-    python3-dev python3-pip python3-click python3-virtualenv \
-    uwsgi uwsgi-plugin-asyncio-python3 uwsgi-plugin-gevent-python \
-    uwsgi-plugin-python uwsgi-plugin-python3 uwsgi-plugin-tornado-python \
-    uwsgi-plugin-lua5.1 uwsgi-plugin-lua5.2 uwsgi-plugin-luajit
-```
-## Setting up the `piku` user
-
-`piku` requires a separate user account to run. To create a new user with the right group membership (we're using the built-in `www-data` group because it's generally thought of as a less-privileged group).  This user (`piku`) _is not supposed to login to your system_. Instead, you'll interact with `piku` via SSH, and set things up by using `su`.
-
-To create the `piku` user account, enter the following commands:
-
-```bash
-# pick a username
-export PAAS_USERNAME=piku
-# create it
-sudo adduser --disabled-password --gecos 'PaaS access' --ingroup www-data $PAAS_USERNAME
-```
-
-You will have to copy your ssh public key to the raspberry pi from your main system.  If you already have `ssh`d into your newly installed pi you can do the following:
-
-```bash
-# copy your public key to /tmp (I'm assuming it's the first entry in authorized_keys)
-head -1 ~/.ssh/authorized_keys > /tmp/pubkey
-# install piku and have it set up SSH keys and default files
-sudo su - $PAAS_USERNAME -c "wget https://raw.githubusercontent.com/piku/piku/master/piku.py && python3 ~/piku.py setup && python3 ~/piku.py setup:ssh /tmp/pubkey"
-rm /tmp/pubkey
-```
-
-The `setup` output should be something like this:
-
-```
-Creating '/home/piku/.piku/apps'.
-Creating '/home/piku/.piku/repos'.
-Creating '/home/piku/.piku/envs'.
-Creating '/home/piku/.piku/uwsgi'.
-Creating '/home/piku/.piku/uwsgi-available'.
-Creating '/home/piku/.piku/uwsgi-enabled'.
-Creating '/home/piku/.piku/logs'.
-Setting '/home/piku/piku.py' as executable.
-```
-
-## uWSGI Configuration
-
-[uWSGI][uwsgi] in Stretch and Buster requires very little configuration, since it is already properly packaged. All you need to do is create a symlink to the `piku` configuration file in `/etc/uwsgi/apps-enabled`:
-
-```bash
-sudo ln /home/$PAAS_USERNAME/.piku/uwsgi/uwsgi.ini /etc/uwsgi/apps-enabled/piku.ini
-sudo systemctl restart uwsgi
-```
-
-## `nginx` Configuration
-
-`piku` requires you to edit `/etc/nginx/sites-available/default` to the following, so it can inject new site configurations into `nginx`:
-
-```
-server {
-    listen 80 default_server;
-    listen [::]:80 default_server;
-    root /var/www/html;
-    index index.html index.htm;
-    server_name _;
-    location / {
-        try_files $uri $uri/ =404;
-    }
-}
-# replace `PAAS_USERNAME` with the username you created.
-include /home/PAAS_USERNAME/.piku/nginx/*.conf;
-```
-
-## Set up systemd.path to reload nginx upon config changes
-
-```bash
-# Set up systemd.path to reload nginx upon config changes
-sudo cp ./piku-nginx.{path, service} /etc/systemd/system/
-sudo systemctl enable piku-nginx.{path,service}
-sudo systemctl start piku-nginx.path
-# Check the status of piku-nginx.service
-systemctl status piku-nginx.path # should return `Active: active (waiting)`
-# Restart NGINX
-sudo systemctl restart nginx
-```
-## Notes
-
-> This file was last updated on June 2019
-
-[uwsgi]: https://github.com/unbit/uwsgi

docs/ubuntu-18.04-bionic.md

@@ -1,99 +0,0 @@
-# Installation on Ubuntu 18.04 LTS (Bionic)
-
-> This is a standalone, distribution-specific version of `INSTALL.md`. You do not need to read or follow the original file, but can refer to it for generic steps like setting up SSH keys (which are assumed to be common knowledge here)
-
-`piku` setup is simplified in Bionic, since it can take advantage of some packaging improvements in [uWSGI][uwsgi] and does not require a custom `systemd` service. Since Bionic also ships with Python 3.6, this is an ideal environment for new deployments on both Intel and ARM devices.
-
-## Dependencies
-
-Before installing `piku`, you need to install the following packages:
-
-```bash
-sudo apt-get update
-sudo apt-get install -y build-essential certbot git \
-    libjpeg-dev libxml2-dev libxslt1-dev zlib1g-dev nginx \
-    python-certbot-nginx python-dev python-pip python-virtualenv \
-    python3-dev python3-pip python3-click python3-virtualenv \
-    uwsgi uwsgi-plugin-asyncio-python3 uwsgi-plugin-gevent-python \
-    uwsgi-plugin-python uwsgi-plugin-python3 uwsgi-plugin-tornado-python
-```
-## Setting up the `piku` user
-
-`piku` requires a separate user account to run. To create a new user with the right group membership (we're using the built-in `www-data` group because it's generally thought of as a less-privileged group), enter the following commands:
-
-```bash
-# pick a username
-export PAAS_USERNAME=piku
-# create it
-sudo adduser --disabled-password --gecos 'PaaS access' --ingroup www-data $PAAS_USERNAME
-```
-
-This user _is not supposed to login to your system_. Instead, you'll interact with `piku` via SSH, and set things up by using `su`:
-
-```bash
-# copy your public key to /tmp (I'm assuming it's the first entry in authorized_keys)
-head -1 ~/.ssh/authorized_keys > /tmp/pubkey
-# install piku and have it set up SSH keys and default files
-sudo su - $PAAS_USERNAME -c "wget https://raw.githubusercontent.com/piku/piku/master/piku.py && python3 ~/piku.py setup && python3 ~/piku.py setup:ssh /tmp/pubkey"
-rm /tmp/pubkey
-```
-
-The `setup` output should be something like this:
-
-```
-Creating '/home/piku/.piku/apps'.
-Creating '/home/piku/.piku/repos'.
-Creating '/home/piku/.piku/envs'.
-Creating '/home/piku/.piku/uwsgi'.
-Creating '/home/piku/.piku/uwsgi-available'.
-Creating '/home/piku/.piku/uwsgi-enabled'.
-Creating '/home/piku/.piku/logs'.
-Setting '/home/piku/piku.py' as executable.
-```
-
-## uWSGI Configuration
-
-[uWSGI][uwsgi] in Bionic requires very little configuration, since it is already properly packaged. All you need to do is place a link to the `piku` configuration file in `/etc/uwsgi/apps-enabled`:
-
-```bash
-sudo ln /home/$PAAS_USERNAME/.piku/uwsgi/uwsgi.ini /etc/uwsgi/apps-enabled/piku.ini
-sudo systemctl restart uwsgi
-```
-
-## `nginx` Configuration
-
-`piku` requires you to edit `/etc/nginx/sites-available/default` to the following, so it can inject new site configurations into `nginx`:
-
-```
-server {
-    listen 80 default_server;
-    listen [::]:80 default_server;
-    root /var/www/html;
-    index index.html index.htm;
-    server_name _;
-    location / {
-        try_files $uri $uri/ =404;
-    }
-}
-# replace `PAAS_USERNAME` with the username you created.
-include /home/PAAS_USERNAME/.piku/nginx/*.conf;
-```
-
-## Set up systemd.path to reload nginx upon config changes
-
-```bash
-# Set up systemd.path to reload nginx upon config changes
-sudo cp ./piku-nginx.{path, service} /etc/systemd/system/
-sudo systemctl enable piku-nginx.{path,service}
-sudo systemctl start piku-nginx.path
-# Check the status of piku-nginx.service
-systemctl status piku-nginx.path # should return `Active: active (waiting)`
-# Restart NGINX
-sudo systemctl restart nginx
-```
-
-## Notes
-
-> This file was last updated on November 2018
-
-[uwsgi]: https://github.com/unbit/uwsgi

examples/README.md

@@ -1,16 +1,3 @@
 # Piku Examples
 
-The following table lists the available examples, including some third-party applications you can deploy with `piku`:
-
-| Stack     | Simple | Framework | Utility | Third-Party |
-|-----------|:------:|:---------:|:-------:|:-----------:|
-| C | [Baremetal C](https://github.com/piku/sample-bare-metal-c) |
-| Clojure | [Sample app](https://github.com/piku/sample-clojure-app) |
-| ClojureScript | | | [Piku Monitoring](https://github.com/piku/piku-monitoring) |
-| Elixir | | [Phoenix App](https://github.com/piku/sample-phoenix-app) |
-| Go | [Sample app](./golang) | | | [Gitea (Static Binary)](https://github.com/piku/deploy-gitea) |
-| Java | [Sample app](https://github.com/piku/sample-java-app) |
-| NodeJS | [Sample app](https://github.com/piku/sample-nodejs-app) | [Express with Wisp](./nodejs-wisp) | | [Node-RED](https://github.com/piku/deploy-nodered),<br/>[BitTorrent Tracker](https://github.com/piku/deploy-bittorrent-tracker) |
-| Python | [Background workers](https://github.com/piku/sample-python-app),<br/>[Postgres database](./python-postgres) | [Django app](https://github.com/piku/sample-django-app),<br/>[Flask Web app Tutorial](https://github.com/piku/webapp-tutorial) | [WebDAV Server](https://github.com/piku/deploy-webdav) |
-| Ruby | [Sample app](https://github.com/piku/sample-ruby-app) | [Rails app](https://github.com/piku/sample-rails-app) |
-| SPA | | | | [Snapdrop](https://github.com/piku/deploy-snapdrop) |
+Besides the examples in this folder, there is also [a set of community-supplied example applications](https://piku.github.io/community/examples.html)

examples/python-postgres/requirements.txt

@@ -1,4 +1,4 @@
 Django>=2.2.9
 pytz==2019.1
-sqlparse==0.3.0
+sqlparse==0.4.4
 psycopg2-binary==2.8.3

piku

@@ -52,6 +52,8 @@ else
       ssh -o LogLevel=QUIET ${sshflags} "$server" "$@" | grep -v "INTERNAL"
       echo "  shell             Local command to start an SSH session in the remote."
       echo "  init              Local command to download an example ENV and Procfile."
+      echo "  download          Local command to scp down a remote file. args: REMOTE-FILE(s) LOCAL-PATH"
+      echo "                    Remote file path is relative to the app folder."
       ;;
     apps|setup|setup:ssh|update)
       ssh ${sshflags} "$server" "$@"
@@ -59,6 +61,9 @@ else
     shell)
       ssh -t "$server" run "$app" bash
       ;;
+    download)
+      scp "$server:~/.piku/apps/${app}/${2}" ${3:-'.'}
+      ;;
     *)
       shift # remove cmd arg
       ssh ${sshflags} "$server" "$cmd" "$app" "$@"

piku.py

@@ -4,9 +4,9 @@
 
 try:
     from sys import version_info
-    assert version_info >= (3, 5)
+    assert version_info >= (3, 7)
 except AssertionError:
-    exit("Piku requires Python 3.5 or above")
+    exit("Piku requires Python 3.7 or above")
 
 from importlib import import_module
 from collections import defaultdict, deque
@@ -245,6 +245,12 @@ def get_free_port(address=""):
     return port
 
 
+def get_boolean(value):
+    """Convert a boolean-ish string to a boolean."""
+
+    return value.lower() in ['1', 'on', 'true', 'enabled', 'yes', 'y']
+
+
 def write_config(filename, bag, separator='='):
     """Helper for writing out config files"""
 
@@ -379,6 +385,13 @@ def do_deploy(app, deltas={}, newrev=None):
         workers = parse_procfile(procfile)
         if workers and len(workers) > 0:
             settings = {}
+            if "preflight" in workers:
+                echo("-----> Running preflight.", fg='green')
+                retval = call(workers["preflight"], cwd=app_path, env=settings, shell=True)
+                if retval:
+                    echo("-----> Exiting due to preflight command error value: {}".format(retval))
+                    exit(retval)
+                workers.pop("preflight", None)
             if exists(join(app_path, 'requirements.txt')) and found_app("Python"):
                 settings.update(deploy_python(app, deltas))
             elif exists(join(app_path, 'Gemfile')) and found_app("Ruby Application") and check_requirements(['ruby', 'gem', 'bundle']):
@@ -392,8 +405,10 @@ def do_deploy(app, deltas={}, newrev=None):
                 settings.update(deploy_java_gradle(app, deltas))
             elif (exists(join(app_path, 'Godeps')) or len(glob(join(app_path, '*.go')))) and found_app("Go") and check_requirements(['go']):
                 settings.update(deploy_go(app, deltas))
+            elif exists(join(app_path, 'deps.edn')) and found_app("Clojure CLI") and check_requirements(['java', 'clojure']):
+                settings.update(deploy_clojure_cli(app, deltas))
             elif exists(join(app_path, 'project.clj')) and found_app("Clojure Lein") and check_requirements(['java', 'lein']):
-                settings.update(deploy_clojure(app, deltas))
+                settings.update(deploy_clojure_leiningen(app, deltas))
             elif 'release' in workers and 'web' in workers:
                 echo("-----> Generic app detected.", fg='green')
                 settings.update(deploy_identity(app, deltas))
@@ -476,7 +491,29 @@ def deploy_java_maven(app, deltas={}):
     return spawn_app(app, deltas)
 
 
-def deploy_clojure(app, deltas={}):
+def deploy_clojure_cli(app, deltas={}):
+    """Deploy a Clojure Application"""
+
+    virtual = join(ENV_ROOT, app)
+    target_path = join(APP_ROOT, app, 'target')
+    env_file = join(APP_ROOT, app, 'ENV')
+
+    if not exists(target_path):
+        makedirs(virtual)
+    env = {
+        'VIRTUAL_ENV': virtual,
+        "PATH": ':'.join([join(virtual, "bin"), join(app, ".bin"), environ['PATH']]),
+        "CLJ_CONFIG": environ.get('CLJ_CONFIG', join(environ['HOME'], '.clojure')),
+    }
+    if exists(env_file):
+        env.update(parse_settings(env_file, env))
+    echo("-----> Building Clojure Application")
+    call('clojure -T:build release', cwd=join(APP_ROOT, app), env=env, shell=True)
+
+    return spawn_app(app, deltas)
+
+
+def deploy_clojure_leiningen(app, deltas={}):
     """Deploy a Clojure Application"""
 
     virtual = join(ENV_ROOT, app)
@@ -657,6 +694,7 @@ def spawn_app(app, deltas={}):
     app_path = join(APP_ROOT, app)
     procfile = join(app_path, 'Procfile')
     workers = parse_procfile(procfile)
+    workers.pop("preflight", None)
     workers.pop("release", None)
     ordinals = defaultdict(lambda: 1)
     worker_count = {k: 1 for k in workers.keys()}
@@ -709,7 +747,7 @@ def spawn_app(app, deltas={}):
             env['PORT'] = str(get_free_port())
             echo("-----> picking free port {PORT}".format(**env))
 
-        if env.get('DISABLE_IPV6', 'false').lower() == 'true':
+        if get_boolean(env.get('DISABLE_IPV6', 'false')):
             safe_defaults.pop('NGINX_IPV6_ADDRESS', None)
             echo("-----> nginx will NOT use IPv6".format(**locals()))
 
@@ -721,6 +759,10 @@ def spawn_app(app, deltas={}):
 
         # Set up nginx if we have NGINX_SERVER_NAME set
         if 'NGINX_SERVER_NAME' in env:
+            # Hack to get around ClickCommand
+            env['NGINX_SERVER_NAME'] = env['NGINX_SERVER_NAME'].split(',')
+            env['NGINX_SERVER_NAME'] = ' '.join(env['NGINX_SERVER_NAME'])
+
             nginx = command_output("nginx -V")
             nginx_ssl = "443 ssl"
             if "--with-http_v2_module" in nginx:
@@ -786,13 +828,13 @@ def spawn_app(app, deltas={}):
 
             # restrict access to server from CloudFlare IP addresses
             acl = []
-            if env.get('NGINX_CLOUDFLARE_ACL', 'false').lower() == 'true':
+            if get_boolean(env.get('NGINX_CLOUDFLARE_ACL', 'false')):
                 try:
                     cf = loads(urlopen('https://api.cloudflare.com/client/v4/ips').read().decode("utf-8"))
                     if cf['success'] is True:
                         for i in cf['result']['ipv4_cidrs']:
                             acl.append("allow {};".format(i))
-                        if env.get('DISABLE_IPV6', 'false').lower() == 'false':
+                        if get_boolean(env.get('DISABLE_IPV6', 'false')):
                             for i in cf['result']['ipv6_cidrs']:
                                 acl.append("allow {};".format(i))
                         # allow access from controlling machine
@@ -911,21 +953,21 @@ def spawn_app(app, deltas={}):
             env['PIKU_INTERNAL_NGINX_COMMON'] = expandvars(NGINX_COMMON_FRAGMENT, env)
 
             echo("-----> nginx will map app '{}' to hostname(s) '{}'".format(app, env['NGINX_SERVER_NAME']))
-            if env.get('NGINX_HTTPS_ONLY', 'false').lower() == 'true':
+            if get_boolean(env.get('NGINX_HTTPS_ONLY', 'false')):
                 buffer = expandvars(NGINX_HTTPS_ONLY_TEMPLATE, env)
                 echo("-----> nginx will redirect all requests to hostname(s) '{}' to HTTPS".format(env['NGINX_SERVER_NAME']))
             else:
                 buffer = expandvars(NGINX_TEMPLATE, env)
 
             # remove all references to IPv6 listeners (for enviroments where it's disabled)
-            if env.get('DISABLE_IPV6', 'false').lower() == 'true':
+            if get_boolean(env.get('DISABLE_IPV6', 'false')):
                 buffer = '\n'.join([line for line in buffer.split('\n') if 'NGINX_IPV6' not in line])
             # change any unecessary uWSGI specific directives to standard proxy ones
             if 'wsgi' not in workers and 'jwsgi' not in workers:
                 buffer = buffer.replace("uwsgi_", "proxy_")
 
             # map Cloudflare connecting IP to REMOTE_ADDR
-            if env.get('NGINX_CLOUDFLARE_ACL', 'false').lower() == 'true':
+            if get_boolean(env.get('NGINX_CLOUDFLARE_ACL', 'false')):
                 buffer = buffer.replace("REMOTE_ADDR $remote_addr", "REMOTE_ADDR $http_cf_connecting_ip")
 
             with open(nginx_conf, "w") as h:
@@ -963,7 +1005,7 @@ def spawn_app(app, deltas={}):
     write_config(live, env)
     write_config(scaling, worker_count, ':')
 
-    if env.get('PIKU_AUTO_RESTART', 'true').lower() not in ['0', 'false']:
+    if get_boolean(env.get('PIKU_AUTO_RESTART', 'true')):
         config = glob(join(UWSGI_ENABLED, '{}*.ini'.format(app)))
         if len(config):
             echo("-----> Removing uwsgi configs to trigger auto-restart.")
@@ -1150,18 +1192,23 @@ def spawn_worker(app, kind, command, env, ordinal=1):
         copyfile(available, enabled)
 
 
-def do_restart(app):
-    """Restarts a deployed app"""
-
+def do_stop(app):
     config = glob(join(UWSGI_ENABLED, '{}*.ini'.format(app)))
 
     if len(config) > 0:
-        echo("Restarting app '{}'...".format(app), fg='yellow')
+        echo("Stopping app '{}'...".format(app), fg='yellow')
         for c in config:
             remove(c)
-        spawn_app(app)
     else:
-        echo("Error: app '{}' not deployed!".format(app), fg='red')
+        echo("Error: app '{}' not deployed!".format(app), fg='red')  # TODO app could be already stopped. Need to able to tell the difference.
+
+
+def do_restart(app):
+    """Restarts a deployed app"""
+    # This must work even if the app is stopped when called. At the end, the app should be running.
+    echo("restarting app '{}'...".format(app), fg='yellow')
+    do_stop(app)
+    spawn_app(app)
 
 
 def multi_tail(app, filenames, catch_up=20):
@@ -1537,16 +1584,8 @@ def cmd_setup_ssh(public_key_file):
 @argument('app')
 def cmd_stop(app):
     """Stop an app, e.g: piku stop <app>"""
-
     app = exit_if_invalid(app)
-    config = glob(join(UWSGI_ENABLED, '{}*.ini'.format(app)))
-
-    if len(config) > 0:
-        echo("Stopping app '{}'...".format(app), fg='yellow')
-        for c in config:
-            remove(c)
-    else:
-        echo("Error: app '{}' not deployed!".format(app), fg='red')
+    do_stop(app)
 
 
 # --- Internal commands ---
@@ -1559,6 +1598,7 @@ def cmd_git_hook(app):
     app = sanitize_app_name(app)
     repo_path = join(GIT_ROOT, app)
     app_path = join(APP_ROOT, app)
+    data_path = join(DATA_ROOT, app)
 
     for line in stdin:
         # pylint: disable=unused-variable
@@ -1567,7 +1607,10 @@ def cmd_git_hook(app):
         if not exists(app_path):
             echo("-----> Creating app '{}'".format(app), fg='green')
             makedirs(app_path)
-            call('git clone --quiet {} {}'.format(repo_path, app), cwd=APP_ROOT, shell=True)
+            # The data directory may already exist, since this may be a full redeployment (we never delete data since it may be expensive to recreate)
+            if not exists(data_path):
+                makedirs(data_path)
+            call("git clone --quiet {} {}".format(repo_path, app), cwd=APP_ROOT, shell=True)
         do_deploy(app, newrev=newrev)
 
 
@@ -1606,6 +1649,13 @@ def cmd_git_upload_pack(app):
     call('git-shell -c "{}" '.format(argv[1] + " '{}'".format(app)), cwd=GIT_ROOT, shell=True)
 
 
+@piku.command("scp", context_settings=dict(ignore_unknown_options=True, allow_extra_args=True))
+@pass_context
+def cmd_scp(ctx):
+    """Simple wrapper to allow scp to work."""
+    call(" ".join(["scp"] + ctx.args), cwd=GIT_ROOT, shell=True)
+
+
 def _get_plugin_commands(path):
     sys_path.append(abspath(path))