Enterprise-Onion-Toolkit/docs.d/HOW-TO-INSTALL.md

# Per-platform Installation

All installations are currently being built from fresh sources for
reasons of development and keeping up to date with core, evolving
features.  As such, installations make take extensive time to build,
especially on slower machines.

## Raspbian

* `sudo apt-get install -y git`
* `git clone https://github.com/alecmuffett/eotk.git`
* `cd eotk`
* `./opt.d/build-raspbian-stretch.sh`

## Ubuntu 20.04LTS

Install a `ubuntu-20.04.2-live-server-amd64.iso` server instance; and then:

* `git clone https://github.com/alecmuffett/eotk.git`
* `cd eotk`
* `./opt.d/build-ubuntu-20.04.sh`

## Ubuntu 18.04LTS

Install a `ubuntu-18.04.2-live-server-amd64.iso` server instance; and then:

* `git clone https://github.com/alecmuffett/eotk.git`
* `cd eotk`
* `./opt.d/build-ubuntu-18.04.sh`

## CentOS 8.2.2004

Install a minimal server and then:

* `sudo yum -y install git`
* `git clone https://github.com/alecmuffett/eotk.git`
* `cd eotk`
* `./opt.d/build-centos-8.2.2004.sh`

## macOS Mojave

Install [Homebrew](https://brew.sh); and then:

* `git clone https://github.com/alecmuffett/eotk.git`
* `cd eotk`
* `./opt.d/build-macos-mojave.sh`

## FreeBSD 12.1

Install a base server and then:

* `pkg install git`
* `git clone https://github.com/alecmuffett/eotk.git`
* `cd eotk`
* `./opt.d/build-freebsd-12.1.sh`

## Other Platform Requirements

EOTK requires recent `tor` and also recent `openresty` with the
`ngx_http_substitutions_filter_module` NGINX module enabled.  They
simply need to be accessible in $PATH.  The above "build" scripts
simply supply that.

# Demonstration And Testing

After installation, you can do:

* `./eotk config demo.d/wikipedia.tconf`
  * or: `./eotk config demo.d/wikipedia-v3.tconf`
* `./eotk start wikipedia`
* `./eotk maps -a` # and connect to one of the onions you've created

Be aware that you will suffer from HTTPS certificate errors 
until you buy a HTTPS certificate.

# Creating HTTPS Certificates for Testing & Development

When connecting to the resulting onions over HTTP/SSL, you will be
using wildcard self-signed SSL certificates - you *will* encounter
many "broken links" which are due to the SSL certificate not being
valid.

This is *expected* and *proper* behaviour; there are currently three
ways to address this.

## Install `mkcert`

The *best* solution for development purposes is to [install `mkcert`
onto the machine which will be running
EOTK](https://github.com/FiloSottile/mkcert#installation) and
configure your own personal Certificate Authority for the certificates
that you will need.

You can then add `set ssl_mkcert 1` to configurations, and your
`mkcert` root certificate will be used to sign the resulting onion
certificates. You can [install that certificate into your local copy
of Tor Browser](/docs.d/ADDING-A-ROOT-CERTIFICATE-TO-TOR-BROWSER.md);
of course it will not work for anyone else.

## Visit `/hello-onion/` URLs

The old solution was/is much more manual: EOTK will use OpenSSL to
create certificates, and then, for any onion - eg:
www.a2s3c4d5e6f7g8h9.onion - EOTK provides a fixed url:

* `https://www.a2s3c4d5e6f7g8h9.onion/hello-onion/`

...which (`/hello-onion/`) is internally served by the NGINX proxy and
provides a stable, fixed URL for SSL certificate acceptance; inside
TorBrowser another effective solution is to open all the broken links,
images and resources "in a new Tab" and accept the certificate there.

In production, of course, one would expect to use an SSL EV
certificate to provide identity and assurance to an onion site,
rendering these issues moot.

## Buy a SSL Certificate from HARICA or Digicert

See below.

# Buying a HTTPS Certificate from a Certificate Authority

If you choose to buy an Onion HTTPS certificate from (e.g.) HARICA, 
what will happen, and what will you need to do?

## You will need to create a CSR (Certificate Signing Request)

I chose to buy:

* a Server Certificate 
* with Domain-Level (DV) Trust
* with a reasonable duration
* using the in-browser generated CSR 
* using the ECDSA algorithm at 256 bits
* **important:** remember the password! 
* **important:** download the private key!

The HARICA website provides an in-browser method of generating a CSR,
if you are not already familiar. Make sure to generate a good
passphrase, and remember it, because you will need it soon.
Also: make sure to download the `privateKey.pem` file that
is offered, and keep it in a safe place.

If you manually create the CSR on the server, you'll use the 'onionaddress.key' file generated by openssl.
## You will need to prove ownership of the site, to the CA

For example: HARICA will tell you that you need to post 
a secret key at a particular URL on your onion site; 
the message will read something like:

> Place the file FILENAME to http://ONIONADDRESS.onion/.well-known/pki-validation/

...and they will offer you a file to download.

Download this file, and open it with a text editor;
the content will be a long secret string,
like THISISAREALLYLONGHEXADECIMALSECRET

Add a line to your EOTK configuration, substituting the values where necessary:

```
set ssl_proof_csv /.well-known/pki-validation/FILENAME,THISISAREALLYLONGHEXADECIMALSECRET
```

Then do something like:

```
eotk config projectname.conf && eotk nxreload projectname
```

...to install the URL handlers.

HARICA has a process of validation which involves generating an onion-csr. You won't need to add anything to your configuration.

### Optional: what if you have multiple Onion addresses?

You can put multiple `path,value` strings into `ssl_proof_csv`, space-separated; 
use trailing backslashes to put entries onto separate lines:

```
set ssl_proof_csv \
    /.well-known/pki-validation/key1,value1 \
    /.well-known/pki-validation/key2,value2 \
    /.well-known/pki-validation/key3,value3    
```

### Optional: what if your multiple "proof" URLs all have the SAME pathname?

The `ssl_proof_csv` hack works if all the proof URLs are
different; but if Digicert (or whomever) were to give you the 
same pathname (e.g. `/.well-known/pki-validation/fileauth.txt`) 
for _all_ of the onions, what do you do?

Answer: you use "splicing".  If you have onion addresses named
`xxxxxxxxxxxxxxxx` and `yyyyyyyyyyyyyyyy`, then you can create files:

* `templates.d/nginx-site-xxxxxxxxxxxxxxxx.onion.conf`
* `templates.d/nginx-site-yyyyyyyyyyyyyyyy.onion.conf`

...and into each put something similar to the following incantation;
customise as necessary:

```
    location = "/.well-known/pki-validation/fileauth.txt" {
      return 200 "RESPECTIVE-PROOF-STRING-GOES-HERE";
    }
```

...then when you next `eotk config` and `eotk nxreload`, that code
should be spliced into the correct configuration for each onion.

## You will need to install the certificates for your project

For each certificate, HARICA will offer you several files to download; 
download the "PEM Bundle" file and copy it to your EOTK server. 
Also: copy the `privateKey.pem` file (mentioned above) to the EOTK server.

Next, change Directory into `~/eotk/projects.d/PROJECTNAME.d/ssl.d`; 
you should see your development certificates, which will look like:

```
$ ls
ONIONADDRESS.onion.cert
ONIONADDRESS.onion.pem
```

There are two steps to installation:

Step 1: copy the PEM Bundle file from HARICA, on top of `ONIONADDRESS.onion.cert`

Step 2: unlock and extract (or rename) the private key, by doing:

`openssl ec -in privateKey.pem -out ONIONADDRESS.onion.pem`

...and typing in the password that you chose during the CSR setup, earlier; 
if you chose to use RSA as the algorithm, you will need to use 
`openssl rsa ...` instead.

If you manually created the CSR, then rename the 'onionaddress.key' file
the CSR generated to 'onionaddress.onion.pem'.

Then: change directory back to the EOTK directory, 
and do `eotk nxreload projectname`, and test it.

# Configuring Start-On-Boot, And Logfile Compression

Once you have installed EOTK (below) and configured and tested it
for your project, run:

* `./eotk make-scripts`

This will create two files:

* `./eotk-init.sh` - for installing on your system as a startup script
* `./eotk-housekeeping.sh` - for cronjob log rotation and other cleanup work

Please read the individual files for installation instructions; local
setup is intended to be pretty simple, let me know if anything is
confusing.

# Using EOTK

## I Want To Create My Own Project!

Okay, there are two ways to create your own project:

### Easy With Automatic Onions

Create a config file with a `.tconf` suffix - we'll pretend it's
`foo.tconf` - and use this kind of syntax:

```
set project myproject
hardmap %NEW_V3_ONION% foo.com
hardmap %NEW_V3_ONION% foo.co.uk
hardmap %NEW_V3_ONION% foo.de
```

...and then run

`eotk config foo.tconf`

...which will create the onions for you and will populate a `foo.conf`
for you, and it will then configure EOTK from *that*.  You should
probably *delete* `foo.tconf` afterwards, since forcibly reusing it
would trash your existing onions.

### Slightly Harder With Manually-Mined Onions

* Do `eotk genkey` - it will print the name of the onion it generates
  * Do this as many times as you wish/need.
  * Alternately get a tool like `scallion`, `shallot`, or `eschalot` and use that to "mine" a desirable onion address.
    * https://github.com/katmagic/Shallot - in C, for CPUs
      * Seems okay on Linux, not sure about other platforms
    * https://github.com/lachesis/scallion - in C#, for CPUs & GPUs (GPU == very fast)
      * Advertised as working on Windows, Linux; works well on OSX under "Mono"
    * https://github.com/ReclaimYourPrivacy/eschalot - in C, for CPUs
      * Works well under Linux, and BSD systems. Uses multiple threads. Can use a wordlist to generate onion addresses
    * Be sure to store your mined private keys in `secrets.d` with a
      filename like `a2s3c4d5e6f7g8h9.key` where `a2s3c4d5e6f7g8h9` is
      the corresponding onion address.
* Create a config file with a `.conf` suffix - we'll pretend it's
  `foo.conf` - and use this kind of syntax, substituting
  `a2s3c4d5e6f7g8h9` for the onion address that you generated.

```
set project myproject
hardmap secrets.d/a2s3c4d5e6f7g8h9.key foo.com
```

...and then (IMPORTANT) run:

`eotk config foo.conf`

...which will configure EOTK.

### Then Start Your Project

Like this:

`eotk start myproject`

## What If I Have Subdomains?

When you are setting up the mappings in a config file, you may have to
accomodate "subdomains"; the general form of a internet hostname is
like this:

* `hostname.domain.tld` # like: www.facebook.com or www.gov.uk
  * or: `hostname.domain.sld.tld` # like: www.amazon.co.uk
* `hostname.subdom.domain.tld` # like: www.prod.facebook.com
* `hostname.subsubdom.subdom.domain.tld` # cdn.lhr.eu.foo.net
* `hostname.subsubsubdom.subsubdom.subdom.domain.tld` # ...

...and so on, where:

* tld = [top level domain](https://en.wikipedia.org/wiki/Top-level_domain)
  * sld = [second level domain](https://en.wikipedia.org/wiki/.uk#Second-level_domains)
* domain = *generally the name of the organisation you are interested in*
* subdomain = *some kind of internal structure*
* hostname = *actual computer, or equivalent*

When you are setting up mappings, generally the rules are:

* you will **map one domain per onion**
* you will **ignore all hostnames**
* you will **append all possible subdomain stems**

So if your browser tells you that you are fetching content from
`cdn7.dublin.ireland.europe.foo.co.jp`, you should add a line like:

```
hardmap %NEW_V3_ONION% foo.co.jp europe ireland.europe dublin.ireland.europe
```

...and EOTK should do the rest. All this is necessary purely for
correctness of the self-signed SSL-Certificates - which are going to
be weird, anyway - and the rest of the HTML-rewriting code in EOTK
will be blind to subdomains.

### Subdomain Summary

Subdomains are supported like this, for `dev` as an example:

```
set project myproject
hardmap secrets.d/a2s3c4d5e6f7g8h9.key foo.com dev
```

...and if you have multiple subdomains:

```
hardmap secrets.d/a2s3c4d5e6f7g8h9.key foo.com dev blogs dev.blogs [...]
```

## My Company Has A Lot Of Sites/Domains!

Example:

* `www.foo.com.au`
* `www.syd.foo.com.au`
* `www.per.foo.com.au`,
* `www.cdn.foo.net`
* `www.foo.aws.amazon.com`
* ...

Put them all in the same project as separate mappings, remembering to
avoid the actual "hostnames" as described above:

```
set project fooproj
hardmap %NEW_V3_ONION% foo.com.au syd per
hardmap %NEW_V3_ONION% foo.net cdn
hardmap %NEW_V3_ONION% foo.aws.amazon.com
```

Onion mapping/translations will be applied for all sites in the same project.

## Troubleshooting

The logs for any given project will reside in
`projects.d/<PROJECTNAME>.d/logs.d/`

If something is problematic, first try:

* `git pull` and...
* `eotk config <filename>.conf` again, and then...
* `eotk bounce -a`

### Lots Of Broken Images, Missing Images, Missing CSS

This is probably an SSL/HTTPS thing.

Because of the nature of SSL self-signed certificates, you have to
manually accept the certificate of each and every site for which a
certificate has been created. See the second of the YouTube videos for
some mention of this.

In short: this is normal and expected behaviour.  You can temporarily
fix this by:

* right-clicking on the image for `Open In New Tab`, and accepting the
  certificate
* or using `Inspect Element > Network` to find broken resources, and
  doing the same
* or - if you know the list of domains in advance - visiting the
  `/hello-onion/` URL for each of them, in advance, to accept
  certificates.

If you get an
[official SSL certificate for your onion site](https://blog.digicert.com/ordering-a-onion-certificate-from-digicert/)
then the problem will vanish. Until then, I am afraid that you will be
stuck playing certificate "whack-a-mole".

### NGINX: Bad Gateway

Generally this means that NGINX cannot connect to the remote website,
which usually happens because:

* the site name in the config file, is wrong
* the nginx daemon tries to do a DNS resolution, which fails

Check the NGINX logfiles in the directory cited above, for
confirmation.

If DNS resolution is failing, *PROBABLY* the cause is probably lack
of access to Google DNS / 8.8.8.8; therefore in your config file
you should add a line like this - to use `localhost` as an example:

```
set nginx_resolver 127.0.0.1
```

...and then do:

```
eotk stop -a
eotk config filename.conf
eotk start -a
```

If you need a local DNS resolver, I recommend `dnsmasq`.

### I Can't Connect, It's Just Hanging!

If your onion project has just started, it can take up to a few
minutes to connect for the first time; also sometimes TorBrowser
caches stale descriptors for older onions.  Try restarting TorBrowser
(or use the `New Identity` menu item) and have a cup of tea.  If it
persists, check the logfiles.

### OnionBalance Runs For A Few Days, And Then Just Stops Responding!

Is the clock/time of day correct on all your machines?  Are you
running NTP?  We are not sure but having an incorrect clock may be a
contributory factor to this issue.

##  OnionBalance And Load-Balancing

*NEW FOR 2019:* 
OnionBalance as-of June 2019 is an flaky piece of software which is
hard to run on modern Linux because an stale python crypto library;
more than 90% of Onion sites will not practically need it - or, not
initially, anyway - so I am *deprecating OnionBalance in EOTK* until
it is majorly overhauled and also supports v3 onion addressing.

So, I recommend people to avoid OnionBalance and use hardmap (local)
for the moment, until Tor fix it.  Consider OB if and only if your
system is under sustained high bandwidth and strongly demonstrates
extended choking on throughput.

## Video Demonstrations

**These videos are instructive, but dated.**

Commands have changed - but not very much - but please check the
documentation rather than trust the videos; consider the videos to be
advisory rather than correct.

* [Basic Introduction to EOTK](https://www.youtube.com/watch?v=ti_VkVmE3J4)
* [Rough Edges: SSL Certificates & Strange Behaviour](https://www.youtube.com/watch?v=UieLTllLPlQ)
* [Using OnionBalance](https://www.youtube.com/watch?v=HNJaMNVCb-U)

[![Basic Introduction to EOTK](http://img.youtube.com/vi/ti_VkVmE3J4/0.jpg)](http://www.youtube.com/watch?v=ti_VkVmE3J4)
[![Rough Edges: SSL Certificates & Strange Behaviour](http://img.youtube.com/vi/UieLTllLPlQ/0.jpg)](http://www.youtube.com/watch?v=UieLTllLPlQ)
[![Using OnionBalance](http://img.youtube.com/vi/HNJaMNVCb-U/0.jpg)](http://www.youtube.com/watch?v=HNJaMNVCb-U)