## Docker workshop
---
## About containers
----
### A brief history about containers
* During the unix development of Unix V7 in 1979, the chroot system call was introduced, changing the root directory of a process and its children to a new location in the filesystem.
* Chroot was added to BSD in 1982.
* 2000: FreeBSD Jails
* 2001: Linux VServer
* 2004: Solaris Containers
----
* 2005: Open VZ (Open Virtuzzo)
* 2006: Process Containers
* 2008: LXC
* 2013: Docker
* 2016: The Importance of Container Security Is Revealed
* 2017: Container Tools Become Mature
* Kubernetes Grows Up
----
### How they work?
With containers, instead of virtualizing the underlying computer like a virtual machine (VM), just the OS is virtualized.
---
## About images
----
> A Docker image is a file, comprised of multiple layers, used to execute code in a Docker container. An image is essentially built from the instructions for a complete and executable version of an application, which relies on the host OS kernel. When the Docker user runs an image, it becomes one or multiple instances of that container.
---
## The plan
We'll explore most of the basic docker concepts creating a real microservice project consistent on:
----
* Rust program talking to kafka
* CI compiling docker process using gitlab-ci
* Kafka cluster using docker-compose
* Containers monitored with telegraf and grafana
* Load balancer with traefik
At the end of the exercice you will have a working environment with those services.
---
## Requirements
----
Edit your hosts file with exercice name `docker.local`. This will be used to access services from load-balancer using Host header.
```ini
127.0.0.1 docker.local
```
Having docker and docker-compose installed and working in the system.
---
## Creating a rust application with docker
Just create a rust project of hello world which will be the base of our rust kafka-client
```sh
cargo new hello_world --bin
```
----
Add a `Dockerfile` in the root folder of the application, this file is used to create the docker image of our hello-world.
----
```Dockerfile
FROM rust
WORKDIR /app
COPY . .
RUN cargo install --path .
CMD ["hello_world"]
```
----
Test if the `Dockerfile` compiles properly our app
```sh
docker build .
```
----
If your image compiles properly; Congratulations!! you have your first docker image available.
At this point you can use this image in your localhost or push it to some Docker registry to make it available to other docker hosts.
But we're going to apply some CD here to automatize this process and be aligned with Continuous Delivery from the first step.
----
Just add the `gitlab-ci.yml` which is provided for the gitlab itself.
----
```yaml
# This file is a template, and might need editing before it works on your project.
build-master:
# Official docker image.
image: docker:latest
stage: build
services:
- docker:dind
before_script:
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
script:
- docker build --pull -t "$CI_REGISTRY_IMAGE" .
- docker push "$CI_REGISTRY_IMAGE"
only:
- master
build:
# Official docker image.
image: docker:latest
stage: build
services:
- docker:dind
before_script:
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
script:
- docker build --pull -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG" .
- docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG"
except:
- master
```
----
This file work pretty similar to travis-ci or drone CI systems.
It tells gitlab to compile your package and push it to the gitlab docker registry.
----
Commit and push this file (or use gitlab-ci add files template) and will be able to
see the process of your building in gitlab-ci. After successfull build you will be able to pull your created image
from any docker host machine.
----
You can test your new application by using `docker run`
```sh
$ docker run registry.gitlab.com/gtrias/rust-example
Hello, world!
```
This will download our previous built image (if it isn't available in localhost already) and run it as a container.
----
If you want to update local docker image with latest version you can run `docker pull`
```sh
docker pull registry.gitlab.com/gtrias/rust-example
```
---
## Creating kafka cluster using docker-compose
----
Docker compose is a layer over docker API which allows you to manage collections of docker containers using a definition file. Since we don't want to remember all the time our docker commands we'll going to define our kafka cluster using a yml definition file.
----
We are going to base the cluster from already done job [here](https://github.com/wurstmeister/kafka-docker).
So let's clone this project to follow.
----
The base docker-compose has this shape:
```yaml
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
build: .
ports:
- "9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: 172.17.0.1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
volumes:
- /var/run/docker.sock:/var/run/docker.sock
```
----
This specify two services, one zookeper and kafka itself.
----
You can pull, create containers and run them running:
```sh
docker-compose up -d
```
----
As `docker run` command this will download needed images, create containers
and run them. Note the `-d` param refers to detached, it won't close the containers if you
close the current shell. Additionally note the `build .` in kafka service, this tells docker-compose
to build this image instead of looking one already created one.
----
After this process finishes you will have your running containers ready to work :)
You can see the status running
```sh
[genar@thinktravel kafka-docker]$ docker-compose ps
Name Command State Ports
----------------------------------------------------------------------------------------------------------------------
kafka-docker_kafka_1 start-kafka.sh Up 0.0.0.0:32768->9092/tcp
kafka-docker_zookeeper_1 /bin/sh -c /usr/sbin/sshd ... Up 0.0.0.0:2181->2181/tcp, 22/tcp, 2888/tcp, 3888/tcp
```
----
Also you can chech their logs to ensure they are working as expected:
```sh
genar@thinktravel kafka-docker]$ docker-compose logs -f --tail 20
Attaching to kafka-docker_zookeeper_1, kafka-docker_kafka_1
zookeeper_1 | 2019-06-24 19:38:29,608 [myid:] - INFO [main:Environment@100] - Server environment:os.name=Linux
zookeeper_1 | 2019-06-24 19:38:29,608 [myid:] - INFO [main:Environment@100] - Server environment:os.arch=amd64
zookeeper_1 | 2019-06-24 19:38:29,608 [myid:] - INFO [main:Environment@100] - Server environment:os.version=5.1.5-arch1-2-ARCH
zookeeper_1 | 2019-06-24 19:38:29,608 [myid:] - INFO [main:Environment@100] - Server environment:user.name=root
zookeeper_1 | 2019-06-24 19:38:29,609 [myid:] - INFO [main:Environment@100] - Server environment:user.home=/root
zookeeper_1 | 2019-06-24 19:38:29,609 [myid:] - INFO [main:Environment@100] - Server environment:user.dir=/opt/zookeeper-3.4.13
zookeeper_1 | 2019-06-24 19:38:29,612 [myid:] - INFO [main:ZooKeeperServer@836] - tickTime set to 2000
```
----
To stop all containers just type `docker-compose stop` or if you want to stop and also remove them run `docker-compose down` (if containers are destroyed all data not stored in volumes will be destroyed too)
----
Now you can scale kafka to 2 brokers running:
```sh
docker-compose scale kafka=2
```
---
## Make our rust application talk to kafka
----
The proper implementation to listen kafka has been implemented [in this repository]( https://gitlab.com/gtrias/rust-example)
---
Add our rust app to the kafka docker-compose.yml file:
```yaml
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
build: .
ports:
- "9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: 172.17.0.1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
volumes:
- /var/run/docker.sock:/var/run/docker.sock
rust-client:
image: registry.gitlab.com/gtrias/rust-example
```
---
## Add kafka-manager
```yaml
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
build: .
ports:
- "9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: 172.17.0.1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
volumes:
- /var/run/docker.sock:/var/run/docker.sock
kafka-manager:
image: sheepkiller/kafka-manager
ports:
- 9000:9000
environment:
APPLICATION_SECRET: letmein
ZK_HOSTS: zookeeper:2181
rust-client:
image: registry.gitlab.com/gtrias/rust-example
```
---
### Generate some data to kafka to see if is received by our rust app
You can use kafka-manager to generate the topics to be consumed by rust app.
See examples here: https://wurstmeister.github.io/kafka-docker/
---
## Docker API
----
Docker has a full rich API to manage containers, here I'm going to talk about the most basic and used (by me) to manage host containers:
----
### List of images images in the docker host
```sh
[genar@thinktravel orgmode]$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
factorial-backend_backend latest 06cf665f0d95 7 weeks ago 1.78GB
factorial-backend_sidekiq latest 06cf665f0d95 7 weeks ago 1.78GB
memcached latest 1941054e2bdf 8 weeks ago 62.2MB
elixir latest c4eee7f74185 2 months ago 1.08GB
ruby 2.5 e86557c9a8ab 2 months ago 873MB
redis latest 0f55cf3661e9 4 months ago 95MB
mysql 5.7 e47e309f72c8 4 months ago 372MB
node 8 4f01e5319662 4 months ago 893MB
node 8-alpine e8ae960eaa9e 4 months ago 66.3MB
jwilder/nginx-proxy latest 60f01f8052f5 4 months ago 148MB
phpmyadmin/phpmyadmin latest c6ba363e7c9b 4 months ago 166MB
ruby 2.4.1 e7ca4a0b5b6d 21 months ago 684MB
```
----
### Get an image
```sh
docker pull traefik
```
----
### Run a new container from an image
```sh
docker run -d -p 8080:8080 -p 80:80 -v $PWD/traefik.toml:/etc/traefik/traefik.toml traefik
9e4a65ae2982832902c47e9429a8fbc81aee56de5aea414ab33d133da22d513e
```
This will get the image for you if its not already on the system.
----
### Show running container logs
```sh
docker logs -f --tail 20 e9c118b35981
```
----
### Stop and remove a running container
```sh
docker rm -f e9c118b35981
```
----
### Run a command inside a running docker container
```sh
docker exec -ti e9c118b35981 bin/rails console
```
Also you can enter to the container shell (if it has any installed inside which is the most habitual)
```sh
docker exec -ti e9c118b35981 bash
```
----
### Pull images, create containers and run them
```sh
docker-compose up -d
```
Note: -d is to run them detached, if not the process will be binded in your shell session and be lost if you close it.
----
### Stop all containers
```sh
docker-compose stop
```
----
### See all containers logs
```sh
docker-compose logs -f --tail 20
```
----
### See one container logs
```sh
docker-compose logs -f --tail 20 traefik
```
----
### Stop and remove all containers (information non stored on volume will be lost)
```sh
docker-compose down
```
---
## Add traefik for easy access to our services
Usually when you want to have a running container you also want to publish it to internet. A very nice way to achieve this is using a load-balancer able to reconfigure itself when some of the containers change. [traefik](https://traefik.io) does exact this job in a very neat way.
----
A picture is worth a thousand words
----
Add it to your project:
```yaml
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
build: .
ports:
- 9092
environment:
KAFKA_ADVERTISED_HOST_NAME: 172.17.0.1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
volumes:
- /var/run/docker.sock:/var/run/docker.sock
kafka-manager:
image: sheepkiller/kafka-manager
ports:
- 9000:9000
environment:
APPLICATION_SECRET: letmein
ZK_HOSTS: zookeeper:2181
labels:
- "traefik.port=9000"
- "traefik.frontend.rule=Host:kafka-manager.docker.local"
rust-client:
image: registry.gitlab.com/gtrias/rust-example
command: hello_world --topics my-topic --brokers kafka:9092
traefik:
image: traefik
command: --web --docker --docker.domain=docker.local --logLevel=DEBUG
ports:
# access this with the correct Host header to access the respective container
- "80:80"
# management UI
- "8080:8080"
volumes:
# traefik does its magic by reading information about running containers from the docker socket
- /var/run/docker.sock:/var/run/docker.sock
- /dev/null:/traefik.toml
```
---
## Links
* [What’s the Diff: VMs vs Containers](https://www.backblaze.com/blog/vm-vs-containers/)
* [A Brief History of Containers: From the 1970s to 2017](https://blog.aquasec.com/a-brief-history-of-containers-from-1970s-chroot-to-docker-2016)
* [Digging into Docker layers](https://medium.com/@jessgreb01/digging-into-docker-layers-c22f948ed612)
* https://github.com/wurstmeister/kafka-docker
* https://github.com/spicavigo/kafka-rust
* [traefik](https://traefik.io)