Added restart command, diagram, README updates

README.md

@@ -20,6 +20,7 @@ A free, user-friendly, extendable application and [API](http://docs.webodm.org)
  * [Getting Help](#getting-help)
  * [Support the Project](#support-the-project)
  * [Become a Contributor](#become-a-contributor)
+ * [Architecture Overview](#architecture-overview)
  * [Run the docker version as a Linux Service](#run-the-docker-version-as-a-linux-service)
  * [Run it natively](#run-it-natively)
  
@@ -85,7 +86,7 @@ You **will not be able to distribute a single job across multiple processing nod
 If you want to run WebODM in production, make sure to pass the `--no-debug` flag while starting WebODM:
 
 ```bash
-./webodm.sh down && ./webodm.sh start --no-debug
+./webodm.sh restart --no-debug
 ```
 
 This will disable the `DEBUG` flag from `webodm/settings.py` within the docker container. This is [really important](https://docs.djangoproject.com/en/1.11/ref/settings/#std:setting-DEBUG).
@@ -99,7 +100,7 @@ WebODM has the ability to automatically request and install a SSL certificate vi
  - Run the following:
 
 ```bash
-./webodm.sh down && ./webodm.sh start --ssl --hostname webodm.myorg.com
+./webodm.sh restart --ssl --hostname webodm.myorg.com
 ```
 
 That's it! The certificate will automatically renew when needed.
@@ -111,7 +112,7 @@ If you want to specify your own key/certificate pair, simply pass the `--ssl-key
 When using Docker, all processing results are stored in a docker volume and are not available on the host filesystem. If you want to store your files on the host filesystem instead of a docker volume, you need to pass a path via the `--media-dir` option:
 
 ```bash
-./webodm.sh down && ./webodm.sh start --media-dir /home/user/webodm_data
+./webodm.sh restart --media-dir /home/user/webodm_data
 ```
 
 Note that existing task results will not be available after the change. Refer to the [Migrate Data Volumes](https://docs.docker.com/engine/tutorials/dockervolumes/#backup-restore-or-migrate-data-volumes) section of the Docker documentation for information on migrating existing task results.
@@ -122,7 +123,7 @@ Sympthoms | Possible Solutions
 --------- | ------------------
 While starting WebODM you get: `from six.moves import _thread as thread ImportError: cannot import name _thread` | Try running: `sudo pip install --ignore-installed six`
 While starting WebODM you get: `'WaitNamedPipe','The system cannot find the file specified.'` | 1. Make sure you have enabled VT-x virtualization in the BIOS.<br/>2. Try to downgrade your version of Python to 2.7
-While Accessing the WebODM interface you get: `OperationalError at / could not translate host name “db” to address: Name or service not known` or `ProgrammingError at / relation “auth_user” does not exist` | Try restarting your computer, then type: `./webodm.sh down && ./webodm.sh start`
+While Accessing the WebODM interface you get: `OperationalError at / could not translate host name “db” to address: Name or service not known` or `ProgrammingError at / relation “auth_user” does not exist` | Try restarting your computer, then type: `./webodm.sh restart`
 Task output or console shows one of the following:<ul><li>`MemoryError`</li><li>`Killed`</li></ul> |  Make sure that your Docker environment has enough RAM allocated: [MacOS Instructions](http://stackoverflow.com/a/39720010), [Windows Instructions](https://docs.docker.com/docker-for-windows/#advanced)
 After an update, you get: `django.contrib.auth.models.DoesNotExist: Permission matching query does not exist.` | Try to remove your WebODM folder and start from a fresh git clone
 Task fails with `Process exited with code null`, no task console output | If the computer running node-opendronemap is using an old or 32bit CPU, you need to compile [OpenDroneMap](https://github.com/OpenDroneMap/OpenDroneMap) from sources and setup node-opendronemap natively. You cannot use docker. Docker images work with CPUs with 64-bit extensions, MMX, SSE, SSE2, SSE3 and SSSE3 instruction set support or higher.
@@ -168,7 +169,7 @@ Developer, I'm looking to build an app that will stay behind a firewall and just
 - [ ] Volumetric Measurements
 - [X] Cluster management and setup.
 - [ ] Mission Planner
-- [ ] Plugins/Webhooks System
+- [X] Plugins/Webhooks System
 - [X] API
 - [X] Documentation
 - [ ] Android Mobile App
@@ -204,6 +205,17 @@ There are many ways to contribute back to the project:
 
 If you know Python, web technologies (JS, HTML, CSS, etc.) or both, it's easy to make a change to WebODM! Make a fork, clone the repository and run `./devenv.sh start`. That's it! See the [Development Quickstart](http://docs.webodm.org/#development-quickstart) and [Contributing](/CONTRIBUTING.md) documents for more information. All ideas are considered and people of all skill levels are welcome to contribute.
 
+## Architecture Overview
+
+WebODM is built with scalability and performance in mind. While the default setup places all databases and applications on the same machine, users can separate its components for increased performance (ex. place a Celery worker on a separate machine for running background tasks).
+
+![Architecture](https://user-images.githubusercontent.com/1951843/36916884-3a269a7a-1e23-11e8-997a-a57cd6ca7950.png)
+
+A few things to note:
+ * We use Celery workers to do background tasks such as resizing images and processing task results, but we use an ad-hoc scheduling mechanism to communicate with node-OpenDroneMap (which processes the orthophotos, 3D models, etc.). The choice to use two separate systems for task scheduling is due to the flexibility that an ad-hoc mechanism gives us for certain operations (capture task output, persistent data and ability to restart tasks mid-way, communication via REST calls, etc.).
+ * If loaded on multiple machines, Celery workers should all share their `app/media` directory with the Django application (via network shares). You can manage workers via `./worker.sh`
+
+
 ## Run the docker version as a Linux Service
 
 If you wish to run the docker version with auto start/monitoring/stop, etc, as a systemd style Linux Service, a systemd unit file is included in the service folder of the repo.
@@ -224,30 +236,24 @@ If all pre-requisites have been met, and repository is checked out to /opt/WebOD
 
 First, to install the service, and enable the service to run at startup from now on:
 ```bash
-sudo systemctl enable /opt/WebODM/service/webodm.service
+sudo systemctl enable /opt/WebODM/service/webodm-gunicorn.service
 ```
 
 To manually stop the service:
 ```bash
-sudo systemctl stop webodm
+sudo systemctl stop webodm-gunicorn
 ```
 
 To manually start the service:
 ```bash
-sudo systemctl start webodm
+sudo systemctl start webodm-gunicorn
 ```
 
 To manually check service status:
 ```bash
-sudo systemctl status webodm
+sudo systemctl status webodm-gunicorn
 ```
 
-The service runs within a screen session, so as the odm user you can easily jump into the screen session by using:
-```bash
-screen -r webodm
-```
-(if you wish to exit the screen session, don't use ctrl+c, that will kill webodm, use `CTRL+A` then hit the `D` key)
-
 ## Run it natively
 
 WebODM can run natively on Windows, MacOS and Linux.
@@ -261,6 +267,7 @@ To run WebODM, you will need to install:
  * GDAL (>= 2.1)
  * Node.js (>= 6.0)
  * Nginx (Linux/MacOS) - OR - Apache + mod_wsgi (Windows)
+ * Redis (>= 2.6)
 
 On Linux, make sure you have:
 
@@ -302,6 +309,12 @@ ALTER SYSTEM SET postgis.enable_outdb_rasters TO True;
 ALTER SYSTEM SET postgis.gdal_enabled_drivers TO 'GTiff';
 ```
 
+Start the redis broker:
+
+```bash
+redis-server
+```
+
 Then:
 
 ```bash
@@ -313,6 +326,12 @@ python manage.py collectstatic --noinput
 chmod +x start.sh && ./start.sh --no-gunicorn
 ```
 
+Finally, start at least one celery worker:
+
+```bash
+./worker.sh start
+```
+
 The `start.sh` script will use Django's built-in server if you pass the `--no-gunicorn` parameter. This is good for testing, but bad for production. 
 
 In production, if you have nginx installed, modify the configuration file in `nginx/nginx.conf` to match your system's configuration and just run `start.sh` without parameters. 
@@ -345,5 +364,6 @@ python --version
 pip --version
 npm --version
 gdalinfo --version
+redis-server --version
 ```
 Should all work without errors.

slate/diagrams/README.md

@@ -0,0 +1,3 @@
+## How To Open Diagrams
+
+Import the .xml files in this directory to https://www.draw.io

webodm.sh

@@ -25,6 +25,7 @@ DEFAULT_HOST="$WO_HOST"
 DEFAULT_MEDIA_DIR="$WO_MEDIA_DIR"
 DEFAULT_SSL="$WO_SSL"
 DEFAULT_SSL_INSECURE_PORT_REDIRECT="$WO_SSL_INSECURE_PORT_REDIRECT"
+DEFAULT_BROKER="$WO_BROKER"
 
 # Parse args for overrides
 POSITIONAL=()
@@ -70,6 +71,11 @@ case $key in
     --no-debug)
     export WO_DEBUG=NO
     shift # past argument
+    ;;
+	--broker)
+    export WO_BROKER="$2"
+    shift # past argument
+    shift # past value
     ;;
     *)    # unknown option
     POSITIONAL+=("$1") # save it in an array for later
@@ -103,6 +109,7 @@ usage(){
   echo "	--ssl-cert	<path>	Manually specify a path to the certificate file (.pem) to use with nginx to enable SSL (default: None)"
   echo "	--ssl-insecure-port-redirect	<port>	Insecure port number to redirect from when SSL is enabled (default: $DEFAULT_SSL_INSECURE_PORT_REDIRECT)"
   echo "	--no-debug	Disable debug for production environments (default: disabled)"
+  echo "	--broker	Set the URL used to connect to the celery broker (default: $DEFAULT_BROKER)"
   exit
 }
 
@@ -146,6 +153,22 @@ run(){
 }
 
 start(){
+	echo "Starting WebODM..."
+	echo ""
+	echo "Using the following environment:"
+	echo "================================"
+	echo "Host: $WO_HOST"
+	echo "Port: $WO_PORT"
+	echo "Media directory: $WO_MEDIA_DIR"
+	echo "SSL: $WO_SSL"
+	echo "SSL key: $WO_SSL_KEY"
+	echo "SSL certificate: $WO_SSL_CERT"
+	echo "SSL insecure port redirect: $WO_SSL_INSECURE_PORT_REDIRECT"
+	echo "Celery Broker: $WO_BROKER"
+	echo "================================"
+	echo "Make sure to issue a $0 down if you decide to change the environment."
+	echo ""
+
 	command="docker-compose -f docker-compose.yml -f docker-compose.nodeodm.yml"
 	
 	if [ "$WO_SSL" = "YES" ]; then
@@ -189,6 +212,10 @@ start(){
 	run "$command start || $command up"
 }
 
+down(){
+	run "docker-compose -f docker-compose.yml -f docker-compose.nodeodm.yml down"
+}
+
 rebuild(){
 	run "docker-compose down --remove-orphans"
 	run "rm -fr node_modules/ || sudo rm -fr node_modules/"
@@ -232,30 +259,20 @@ resetpassword(){
 
 if [[ $1 = "start" ]]; then
 	environment_check
-	echo "Starting WebODM..."
-	echo ""
-	echo "Using the following environment:"
-	echo "================================"
-	echo "Host: $WO_HOST"
-	echo "Port: $WO_PORT"
-	echo "Media directory: $WO_MEDIA_DIR"
-	echo "SSL: $WO_SSL"
-	echo "SSL key: $WO_SSL_KEY"
-	echo "SSL certificate: $WO_SSL_CERT"
-	echo "SSL insecure port redirect: $WO_SSL_INSECURE_PORT_REDIRECT"
-	echo "================================"
-	echo "Make sure to issue a $0 down if you decide to change the environment."
-	echo ""
-
 	start
 elif [[ $1 = "stop" ]]; then
 	environment_check
 	echo "Stopping WebODM..."
 	run "docker-compose -f docker-compose.yml -f docker-compose.nodeodm.yml stop"
+elif [[ $1 = "restart" ]]; then
+	environment_check
+	echo "Restarting WebODM..."
+	down
+	start
 elif [[ $1 = "down" ]]; then
 	environment_check
 	echo "Tearing down WebODM..."
-	run "docker-compose -f docker-compose.yml -f docker-compose.nodeodm.yml down"
+	down
 elif [[ $1 = "rebuild" ]]; then
 	environment_check
 	echo  "Rebuilding WebODM..."