## Docker workshop ![](https://i.imgur.com/y3iifqT.png) --- ## About containers ![](https://i.imgur.com/ZnSQqlx.png) ---- ### A brief history about containers * During the unix development of Unix V7 in 1979, the chroot system call was introduced, changing the root directory of a process and its children to a new location in the filesystem. * Chroot was added to BSD in 1982. * 2000: FreeBSD Jails * 2001: Linux VServer * 2004: Solaris Containers ---- * 2005: Open VZ (Open Virtuzzo) * 2006: Process Containers * 2008: LXC * 2013: Docker * 2016: The Importance of Container Security Is Revealed * 2017: Container Tools Become Mature * Kubernetes Grows Up ---- ### How they work? With containers, instead of virtualizing the underlying computer like a virtual machine (VM), just the OS is virtualized. ![](https://i.imgur.com/N3ABScV.png) --- ## About images ---- > A Docker image is a file, comprised of multiple layers, used to execute code in a Docker container. An image is essentially built from the instructions for a complete and executable version of an application, which relies on the host OS kernel. When the Docker user runs an image, it becomes one or multiple instances of that container. --- ## The plan We'll explore most of the basic docker concepts creating a real microservice project consistent on: ---- * Rust program talking to kafka * CI compiling docker process using gitlab-ci * Kafka cluster using docker-compose * Containers monitored with telegraf and grafana * Load balancer with traefik At the end of the exercice you will have a working environment with those services. --- ## Requirements ---- Edit your hosts file with exercice name `docker.local`. This will be used to access services from load-balancer using Host header. ```ini 127.0.0.1 docker.local ``` Having docker and docker-compose installed and working in the system. --- ## Creating a rust application with docker Just create a rust project of hello world which will be the base of our rust kafka-client ```sh cargo new hello_world --bin ``` ---- Add a `Dockerfile` in the root folder of the application, this file is used to create the docker image of our hello-world. ---- ```Dockerfile FROM rust WORKDIR /app COPY . . RUN cargo install --path . CMD ["hello_world"] ``` ---- Test if the `Dockerfile` compiles properly our app ```sh docker build . ``` ---- If your image compiles properly; Congratulations!! you have your first docker image available. At this point you can use this image in your localhost or push it to some Docker registry to make it available to other docker hosts. But we're going to apply some CD here to automatize this process and be aligned with Continuous Delivery from the first step. ---- Just add the `gitlab-ci.yml` which is provided for the gitlab itself. ---- ```yaml # This file is a template, and might need editing before it works on your project. build-master: # Official docker image. image: docker:latest stage: build services: - docker:dind before_script: - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY script: - docker build --pull -t "$CI_REGISTRY_IMAGE" . - docker push "$CI_REGISTRY_IMAGE" only: - master build: # Official docker image. image: docker:latest stage: build services: - docker:dind before_script: - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY script: - docker build --pull -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG" . - docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG" except: - master ``` ---- This file work pretty similar to travis-ci or drone CI systems. It tells gitlab to compile your package and push it to the gitlab docker registry. ---- Commit and push this file (or use gitlab-ci add files template) and will be able to see the process of your building in gitlab-ci. After successfull build you will be able to pull your created image from any docker host machine. ---- You can test your new application by using `docker run` ```sh $ docker run registry.gitlab.com/gtrias/rust-example Hello, world! ``` This will download our previous built image (if it isn't available in localhost already) and run it as a container. ---- If you want to update local docker image with latest version you can run `docker pull` ```sh docker pull registry.gitlab.com/gtrias/rust-example ``` --- ## Creating kafka cluster using docker-compose ![](https://media.giphy.com/media/6AFldi5xJQYIo/giphy.gif) ---- Docker compose is a layer over docker API which allows you to manage collections of docker containers using a definition file. Since we don't want to remember all the time our docker commands we'll going to define our kafka cluster using a yml definition file. ---- We are going to base the cluster from already done job [here](https://github.com/wurstmeister/kafka-docker). So let's clone this project to follow. ---- The base docker-compose has this shape: ```yaml version: '2' services: zookeeper: image: wurstmeister/zookeeper ports: - "2181:2181" kafka: build: . ports: - "9092" environment: KAFKA_ADVERTISED_HOST_NAME: 172.17.0.1 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 volumes: - /var/run/docker.sock:/var/run/docker.sock ``` ---- This specify two services, one zookeper and kafka itself. ---- You can pull, create containers and run them running: ```sh docker-compose up -d ``` ---- As `docker run` command this will download needed images, create containers and run them. Note the `-d` param refers to detached, it won't close the containers if you close the current shell. Additionally note the `build .` in kafka service, this tells docker-compose to build this image instead of looking one already created one. ---- After this process finishes you will have your running containers ready to work :) You can see the status running ```sh [genar@thinktravel kafka-docker]$ docker-compose ps Name Command State Ports ---------------------------------------------------------------------------------------------------------------------- kafka-docker_kafka_1 start-kafka.sh Up 0.0.0.0:32768->9092/tcp kafka-docker_zookeeper_1 /bin/sh -c /usr/sbin/sshd ... Up 0.0.0.0:2181->2181/tcp, 22/tcp, 2888/tcp, 3888/tcp ``` ---- Also you can chech their logs to ensure they are working as expected: ```sh genar@thinktravel kafka-docker]$ docker-compose logs -f --tail 20 Attaching to kafka-docker_zookeeper_1, kafka-docker_kafka_1 zookeeper_1 | 2019-06-24 19:38:29,608 [myid:] - INFO [main:Environment@100] - Server environment:os.name=Linux zookeeper_1 | 2019-06-24 19:38:29,608 [myid:] - INFO [main:Environment@100] - Server environment:os.arch=amd64 zookeeper_1 | 2019-06-24 19:38:29,608 [myid:] - INFO [main:Environment@100] - Server environment:os.version=5.1.5-arch1-2-ARCH zookeeper_1 | 2019-06-24 19:38:29,608 [myid:] - INFO [main:Environment@100] - Server environment:user.name=root zookeeper_1 | 2019-06-24 19:38:29,609 [myid:] - INFO [main:Environment@100] - Server environment:user.home=/root zookeeper_1 | 2019-06-24 19:38:29,609 [myid:] - INFO [main:Environment@100] - Server environment:user.dir=/opt/zookeeper-3.4.13 zookeeper_1 | 2019-06-24 19:38:29,612 [myid:] - INFO [main:ZooKeeperServer@836] - tickTime set to 2000 ``` ---- To stop all containers just type `docker-compose stop` or if you want to stop and also remove them run `docker-compose down` (if containers are destroyed all data not stored in volumes will be destroyed too) ---- Now you can scale kafka to 2 brokers running: ```sh docker-compose scale kafka=2 ``` --- ## Make our rust application talk to kafka ---- The proper implementation to listen kafka has been implemented [in this repository]( https://gitlab.com/gtrias/rust-example) --- Add our rust app to the kafka docker-compose.yml file: ```yaml version: '2' services: zookeeper: image: wurstmeister/zookeeper ports: - "2181:2181" kafka: build: . ports: - "9092" environment: KAFKA_ADVERTISED_HOST_NAME: 172.17.0.1 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 volumes: - /var/run/docker.sock:/var/run/docker.sock rust-client: image: registry.gitlab.com/gtrias/rust-example ``` --- ## Add kafka-manager ```yaml version: '2' services: zookeeper: image: wurstmeister/zookeeper ports: - "2181:2181" kafka: build: . ports: - "9092" environment: KAFKA_ADVERTISED_HOST_NAME: 172.17.0.1 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 volumes: - /var/run/docker.sock:/var/run/docker.sock kafka-manager: image: sheepkiller/kafka-manager ports: - 9000:9000 environment: APPLICATION_SECRET: letmein ZK_HOSTS: zookeeper:2181 rust-client: image: registry.gitlab.com/gtrias/rust-example ``` --- ### Generate some data to kafka to see if is received by our rust app You can use kafka-manager to generate the topics to be consumed by rust app. See examples here: https://wurstmeister.github.io/kafka-docker/ --- ## Docker API ---- Docker has a full rich API to manage containers, here I'm going to talk about the most basic and used (by me) to manage host containers: ---- ### List of images images in the docker host ```sh [genar@thinktravel orgmode]$ docker images REPOSITORY TAG IMAGE ID CREATED SIZE factorial-backend_backend latest 06cf665f0d95 7 weeks ago 1.78GB factorial-backend_sidekiq latest 06cf665f0d95 7 weeks ago 1.78GB memcached latest 1941054e2bdf 8 weeks ago 62.2MB elixir latest c4eee7f74185 2 months ago 1.08GB ruby 2.5 e86557c9a8ab 2 months ago 873MB redis latest 0f55cf3661e9 4 months ago 95MB mysql 5.7 e47e309f72c8 4 months ago 372MB node 8 4f01e5319662 4 months ago 893MB node 8-alpine e8ae960eaa9e 4 months ago 66.3MB jwilder/nginx-proxy latest 60f01f8052f5 4 months ago 148MB phpmyadmin/phpmyadmin latest c6ba363e7c9b 4 months ago 166MB ruby 2.4.1 e7ca4a0b5b6d 21 months ago 684MB ``` ---- ### Get an image ```sh docker pull traefik ``` ---- ### Run a new container from an image ```sh docker run -d -p 8080:8080 -p 80:80 -v $PWD/traefik.toml:/etc/traefik/traefik.toml traefik 9e4a65ae2982832902c47e9429a8fbc81aee56de5aea414ab33d133da22d513e ``` This will get the image for you if its not already on the system. ---- ### Show running container logs ```sh docker logs -f --tail 20 e9c118b35981 ``` ---- ### Stop and remove a running container ```sh docker rm -f e9c118b35981 ``` ---- ### Run a command inside a running docker container ```sh docker exec -ti e9c118b35981 bin/rails console ``` Also you can enter to the container shell (if it has any installed inside which is the most habitual) ```sh docker exec -ti e9c118b35981 bash ``` ---- ### Pull images, create containers and run them ```sh docker-compose up -d ``` Note: -d is to run them detached, if not the process will be binded in your shell session and be lost if you close it. ---- ### Stop all containers ```sh docker-compose stop ``` ---- ### See all containers logs ```sh docker-compose logs -f --tail 20 ``` ---- ### See one container logs ```sh docker-compose logs -f --tail 20 traefik ``` ---- ### Stop and remove all containers (information non stored on volume will be lost) ```sh docker-compose down ``` --- ## Add traefik for easy access to our services Usually when you want to have a running container you also want to publish it to internet. A very nice way to achieve this is using a load-balancer able to reconfigure itself when some of the containers change. [traefik](https://traefik.io) does exact this job in a very neat way. ---- A picture is worth a thousand words ![](https://d33wubrfki0l68.cloudfront.net/7c5fd7d38c371e23cdff059e6cebb10292cd441c/7d420/assets/img/traefik-architecture.svg) ---- Add it to your project: ```yaml version: '2' services: zookeeper: image: wurstmeister/zookeeper ports: - "2181:2181" kafka: build: . ports: - 9092 environment: KAFKA_ADVERTISED_HOST_NAME: 172.17.0.1 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 volumes: - /var/run/docker.sock:/var/run/docker.sock kafka-manager: image: sheepkiller/kafka-manager ports: - 9000:9000 environment: APPLICATION_SECRET: letmein ZK_HOSTS: zookeeper:2181 labels: - "traefik.port=9000" - "traefik.frontend.rule=Host:kafka-manager.docker.local" rust-client: image: registry.gitlab.com/gtrias/rust-example command: hello_world --topics my-topic --brokers kafka:9092 traefik: image: traefik command: --web --docker --docker.domain=docker.local --logLevel=DEBUG ports: # access this with the correct Host header to access the respective container - "80:80" # management UI - "8080:8080" volumes: # traefik does its magic by reading information about running containers from the docker socket - /var/run/docker.sock:/var/run/docker.sock - /dev/null:/traefik.toml ``` --- ## Links * [What’s the Diff: VMs vs Containers](https://www.backblaze.com/blog/vm-vs-containers/) * [A Brief History of Containers: From the 1970s to 2017](https://blog.aquasec.com/a-brief-history-of-containers-from-1970s-chroot-to-docker-2016) * [Digging into Docker layers](https://medium.com/@jessgreb01/digging-into-docker-layers-c22f948ed612) * https://github.com/wurstmeister/kafka-docker * https://github.com/spicavigo/kafka-rust * [traefik](https://traefik.io)
{"title":"Docker workshop","slideOptions":{"theme":"night"}}
    256 views