Skip to content
Snippets Groups Projects
Unverified Commit 93c5b063 authored by Achilleas Pipinellis's avatar Achilleas Pipinellis
Browse files

Cleanup and cross-link autoscale docs

parent 07e82d5b
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
docs/configuration/autoscale.md
+ 7
− 64
View file @ 93c5b063
......@@ -279,6 +279,9 @@ in [GitLab Runner - Advanced Configuration - The `[runners.machine]` section][ru
## Distributed runners caching
NOTE: **Note:**
Read how to [install your own cache server](../install/registry_and_cache_servers.md#install-your-own-cache-server).
To speed up your jobs, GitLab Runner provides a [cache mechanism][cache]
where selected directories and/or files are saved and shared between subsequent
jobs.
......@@ -321,40 +324,11 @@ That will remove the runner token from the S3 URL (`runner/<runner-id>`) and
all configured Runners will share the same cache. Remember that you can also
set `Path` to separate caches between Runners when cache sharing is enabled.
### Install your own cache server
If you don't want to use a SaaS S3 server, you can install your own
S3-compatible caching server:
1. Login to a dedicated machine where the cache server will be running.
1. Make sure that Docker Engine is installed on that machine.
1. Start [minio], a simple S3-compatible server written in Go:
```bash
docker run -it --restart always -p 9005:9000 \
-v /.minio:/root/.minio -v /export:/export \
--name minio \
minio/minio:latest server /export
```
You can modify the port `9005` to expose the cache server on different port.
1. Check the IP address of the server:
```bash
hostname --ip-address
```
1. Your cache server will be available at `MY_CACHE_IP:9005`.
1. Read the Access and Secret Key of minio with: `sudo cat /.minio/config.json`.
1. Create a bucket that will be used by the Runner: `sudo mkdir /export/runner`.
`runner` is the name of the bucket in that case. If you choose a different
bucket then it will be different.
1. All caches will be stored in the `/export` directory.
1. Configure `config.toml` as [described above](#distributed-runners-caching).
## Distributed container registry mirroring
NOTE: **Note:**
Read how to [install a container registry](../install/registry_and_cache_servers.md#install-a-proxy-container-registry).
To speed up jobs executed inside of Docker containers, you can use the [Docker
registry mirroring service][registry]. This will provide a proxy between your
Docker machines and all used registries. Images will be downloaded once by the
......@@ -383,37 +357,6 @@ Where `10.11.12.13:12345` is the IP address and port where your registry mirror
is listening for connections from the Docker service. It must be accessible for
each host created by Docker Machine.
### Install a proxy container registry
1. Login to a dedicated machine where the container registry proxy will be running.
1. Make sure that Docker Engine is installed on this machine.
1. Create a new container registry:
```bash
docker run -d -p 6000:5000 \
-e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io \
--restart always \
--name registry registry:2
```
You can modify the port number (`6000`) to expose Docker registry on a
different port.
1. Check the IP address of the server:
```bash
hostname --ip-address
```
You should preferably choose the private networking IP address. The private
networking is usually the fastest solution for internal communication
between machines of a single provider (DigitalOcean, AWS, Azure, etc,)
Usually the private networking is also not accounted to your monthly
bandwidth limit.
1. The container registry will be accessible under `YOUR_REGISTRY_IP:6000`.
1. Configure `config.toml` as [described above](#distributed-container-registry-mirroring).
## A complete example of `config.toml`
The `config.toml` below uses the [`digitalocean` Docker Machine driver](https://docs.docker.com/machine/drivers/digital-ocean/):
......@@ -469,4 +412,4 @@ and one option for Docker Machine itself (`engine-registry-mirror`).
[docker-machine-installation]: https://docs.docker.com/machine/install-machine/
[runners-cache]: advanced-configuration.md#the-runners-cache-section
[runners-machine]: advanced-configuration.md#the-runners-machine-section
[registry]: https://docs.docker.com/docker-trusted-registry/overview/
[registry]: https://docs.docker.com/registry/
docs/executors/docker_machine.md
+ 17
− 32
View file @ 93c5b063
......@@ -2,7 +2,7 @@
> The auto scale feature was introduced in GitLab Runner 1.1.0.
For an overview of the auto-scale architecture, take a look at the
For an overview of the autoscale architecture, take a look at the
[comprehensive documentation on autoscaling](../configuration/autoscale.md).
## Preparing the environment
......@@ -12,32 +12,27 @@ installed in the same machine:
1. Log in to a new Linux-based machine that will serve as a bastion server
where Docker will spawn new machines from
1. Install GitLab Runner following the
[GitLab Runner installation documentation](../install/index.md)
1. Install Docker Machine following the
[Docker Machine installation documentation][docker-machine-installation]
NOTE: **Note:**
Optionally but recommended, prepare a
[proxy container registry](../configuration/autoscale.md#install-a-proxy-container-registry)
and a [cache server](../configuration/autoscale.md#install-your-own-cache-server)
to be used with the autoscaled Runners.
1. [Install GitLab Runner](../install/index.md)
1. [Install Docker Machine](https://docs.docker.com/machine/install-machine/)
1. Optionally but recommended, prepare a
[proxy container registry and a cache server](../install/registry_and_cache_servers.md)
to be used with the autoscaled Runners
If you need to use any virtualization/cloud providers that aren't handled by
Docker's Machine internal drivers, the appropriate driver plugin must be
installed. The Docker Machine driver plugin installation and configuration is
out of the scope of this documentation. For more details please read the
[Docker Machine documentation][docker-machine-docs].
[Docker Machine documentation](https://docs.docker.com/machine/)
## Configuring GitLab Runner
1. [Register a GitLab Runner](../register/index.md#gnu-linux) and select the
`docker+machine` executor when asked
1. Edit [`config.toml`][toml] and configure the Runner to use Docker machine.
Visit the dedicated page covering detailed information about
[GitLab Runner Autoscaling](../configuration/autoscale.md)
`docker+machine` executor when asked.
1. Edit [`config.toml`](../commands/README.md#configuration-file) and configure
the Runner to use Docker machine. Visit the dedicated page covering detailed
information about [GitLab Runner Autoscaling](../configuration/autoscale.md).
1. Try to build your project. In a few seconds, if you run `docker-machine ls`
you should see a new machine being created
you should see a new machine being created.
## Upgrading the Runner
......@@ -50,9 +45,9 @@ out of the scope of this documentation. For more details please read the
killall -SIGQUIT gitlab-runner
```
Sending the [`SIGQUIT` signal][signals] will make the Runner to stop
gracefully. It will stop accepting new jobs, and will exit as soon as the
current builds are finished.
Sending the [`SIGQUIT` signal](../commands/README.md#signals) will make the
Runner to stop gracefully. It will stop accepting new jobs, and will exit
as soon as the current jobs are finished.
1. Wait until the Runner exits. You can check its status with `gitlab-runner status`
or await a graceful shutdown for up to 30 minutes with:
......@@ -64,7 +59,7 @@ out of the scope of this documentation. For more details please read the
done
```
1. You can now safely upgrade the Runner without interrupting any builds
1. You can now safely upgrade the Runner without interrupting any jobs
## Managing the Docker Machines
......@@ -88,14 +83,4 @@ out of the scope of this documentation. For more details please read the
```
1. You can now manage (upgrade or remove) any Docker Machines with the
[`docker-machine` command][docker-machine-command]
[docker-machine-installation]: https://docs.docker.com/machine/install-machine/
[s3]: https://aws.amazon.com/s3/
[minio]: https://www.minio.io/
[caching]: ../configuration/autoscale.md#distributed-runners-caching
[registry]: ../configuration/autoscale.md#distributed-docker-registry-mirroring
[toml]: ../commands/README.md#configuration-file
[signals]: ../commands/README.md#signals
[docker-machine-command]: https://docs.docker.com/machine/reference/
[docker-machine-docs]: https://docs.docker.com/machine/
[`docker-machine` command](https://docs.docker.com/machine/reference/)
docs/install/index.md
+ 19
− 7
View file @ 93c5b063
......@@ -6,15 +6,27 @@ comments: false
GitLab Runner can be installed and used on GNU/Linux, macOS, FreeBSD, and Windows.
There are three ways to install it. Use Docker, download a binary manually, or
use a repository for rpm/deb packages. Below you can find
information on the different installation methods:
use a repository for rpm/deb packages. Below you can find information on the
different installation methods.
- [Install using GitLab's repository for Debian/Ubuntu/CentOS/RedHat](linux-repository.md) (preferred)
- [Install on GNU/Linux manually](linux-manually.md)
## Repositories
- [Install using GitLab's repository for Debian/Ubuntu/CentOS/RedHat](linux-repository.md)
## Binaries
- [Install on GNU/Linux](linux-manually.md)
- [Install on macOS](osx.md)
- [Install on Windows](windows.md)
- [Install as a Docker service](docker.md)
- [Install in Auto-scaling mode using Docker machine](autoscaling.md)
- [Install on FreeBSD](freebsd.md)
- [Install nightly builds](bleeding-edge.md)
## Docker service
- [Install as a Docker service](docker.md)
- [Install on Kubernetes](kubernetes.md)
- [Install the nightly binary manually](bleeding-edge.md)
## Autoscale
- [Install in autoscaling mode using Docker machine](../executors/docker_machine.md)
- [Install the registry and cache servers](registry_and_cache_servers.md)
docs/install/docker-registry-and-cache-server.md → docs/install/registry_and_cache_servers.md
+ 84
− 0
View file @ 93c5b063
# Prepare the Docker Registry and Cache Server
# Install your own container registry and cache server
To speedup the builds we advise to setup a personal Docker registry server
working in proxy mode. A cache server is also recommended.
When using the Runner in [autoscale mode](../configuration/autoscale.md), it
is advised to set up a personal container registry and a cache server.
## Install Docker Registry
## Install a proxy container registry
>**Note:**
Read more in [Distributed registry mirroring][registry].
1. Login to a dedicated machine where Docker registry proxy will be running
2. Make sure that Docker Engine is installed on this machine
3. Create a new Docker registry:
1. Login to a dedicated machine where the container registry proxy will be running
1. Make sure that [Docker Engine](https://docs.docker.com/install/) is installed
on that machine
1. Create a new container registry:
```bash
docker run -d -p 6000:5000 \
......@@ -19,10 +17,10 @@ Read more in [Distributed registry mirroring][registry].
--name registry registry:2
```
You can modify the port number (`6000`) to expose Docker registry on a
different port.
You can modify the port number (`6000`) to expose the registry on a
different port
4. Check the IP address of the server:
1. Check the IP address of the server:
```bash
hostname --ip-address
......@@ -34,17 +32,21 @@ Read more in [Distributed registry mirroring][registry].
Usually the private networking is also not accounted to your monthly
bandwidth limit.
5. Docker registry will be accessible under `MY_REGISTRY_IP:6000`.
5. Docker registry will be accessible under `MY_REGISTRY_IP:6000`
You can now proceed and
[configure `config.toml`](../configuration/autoscale.md#distributed-container-registry-mirroring)
to use the new registry server.
## Install the cache server
## Install your own cache server
>**Note:**
You can use any other S3-compatible server, including [Amazon S3][S3]. Read
more in [Distributed runners caching][caching].
If you don't want to use a SaaS S3 server, you can install your own
S3-compatible caching server:
1. Login to a dedicated machine where the cache server will be running
1. Make sure that Docker Engine is installed on that machine
1. Start [minio], a simple S3-compatible server written in Go:
1. Make sure that [Docker Engine](https://docs.docker.com/install/) is installed
on that machine
1. Start [minio](https://minio.io/), a simple S3-compatible server written in Go:
```bash
docker run -it --restart always -p 9005:9000 \
......@@ -53,7 +55,7 @@ more in [Distributed runners caching][caching].
minio/minio:latest server /export
```
You can modify the port `9005` to expose the cache server on different port.
You can modify the port `9005` to expose the cache server on different port
1. Check the IP address of the server:
......@@ -62,11 +64,21 @@ more in [Distributed runners caching][caching].
```
1. Your cache server will be available at `MY_CACHE_IP:9005`
1. Read the Access and Secret Key of minio with: `sudo cat /.minio/config.json`
1. Create a bucket that will be used by the Runner: `sudo mkdir /export/runner`.
`runner` is the name of the bucket in that case. If you choose a different
bucket then it will be different
1. All caches will be stored in the `/export` directory
[caching]: ../configuration/autoscale.md#distributed-runners-caching
[registry]: ../configuration/autoscale.md#distributed-docker-registry-mirroring
1. Create a bucket that will be used by the Runner:
```
sudo mkdir /export/runner
```
`runner` is the name of the bucket in that case. If you choose a different
bucket, then it will be different. All caches will be stored in the
`/export` directory.
1. Read the Access and Secret Key of minio and use it to configure the Runner:
```
sudo cat /.minio/config.json
```
You can now proceed and
[configure `config.toml`](../configuration/autoscale.md#distributed-runners-caching)
to use the new cache server.
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment