W1 - 3. SNA_Containers PDF

System & Network Administration Containers Le Hai Duong, PhD. ([email protected]) Problem A typical web application developed in any modern language or framework. At a minimum, the following ingredients are needed to install and run the app: ❏ The code for the application and its corre...

System & Network Administration Containers Le Hai Duong, PhD. ([email protected]) Problem A typical web application developed in any modern language or framework. At a minimum, the following ingredients are needed to install and run the app: ❏ The code for the application and its correct configuration ❏ Libraries and other dependencies, each pinned to a specific version that is known to be compatible ❏ An interpreter (e.g., Python or Ruby) or run time (JRE) to execute the code, also version pinned ❏ Localizations such as user accounts, environment settings, and services provided by the operating system A typical site runs dozens or hundreds of such applications. → Incompatible dependencies required by separate applications lead to systems that are underutilized because they cannot be shared Containers ❏ A container image simplifies matters by packaging an application and its prerequisites into a standard, portable file ❏ e.g., Docker, rkt, systemd-nspawn ❏ Tens or hundreds of containers can run simultaneously without conflicts ❏ Containers are a fusion of numerous existing kernel features, filesystem tricks, and networking hacks → a container engine is the management software that pulls it all together ❏ A container is an isolated group of processes that are restricted to a private root filesystem and process namespace ❏ cannot access files or system resources outside their container ❏ applications that run within a container are not aware of their containerized state and do not require modification  Kernel support The container engine uses several kernel features that are essential for isolating processes: ❏ Namespaces isolate ❏ Control groups (cgroup) → prevent runaway containers from consuming all available CPU and memory ❏ Capabilities allow processes to execute certain sensitive kernel operations and system calls ❏ Secure computing mode (seccomp) → more fine-grained control than do capabilities  Images ❏ Template for a container ❏ Container images are union filesystems that are organized to resemble the root filesystem of a typical Linux distribution ❏ The directory layout and the locations of binaries, libraries, and supporting files conform to standard Linux filesystem hierarchy specifications ❏ To create a container, Docker points to the read-only union filesystem (https://en.wikipedia.org/wiki/UnionFS) of an image and adds a read/write layer that the container can update ❏ containerized processes modify the filesystem, their changes are transparently saved within the read/write layer ❏ Many containers can share the same immutable base layers, thus improving storage efficiency and reducing startup times  Networking ❏ The default way to connect containers to the network is to use a network namespace and a bridge within the host ❏ containers have private IP addresses that aren’t reachable from outside the host ❏ host acts as a poor man’s IP router and proxies traffic between the outside world and the containers Docker The open source container engine ❏ docker is an executable command that handles all management tasks for the Docker system ❏ dockerd is the persistent daemon process that implements container and image operations A container relies on the image template as a basis for execution. When dockerd runs a container, it creates a writable filesystem layer that is separate from the source image. The container can read any of the files and other metadata stored within the image, but any writes are confined to the container’s own read/write layer. centralized collection of images The container experience To download images from the Docker Hub, use docker pull. name of image template $ docker pull ubuntu:latest latest: Pulling from library/ubuntu 6e3729cf69e0: Pull complete Digest: sha256:27cb6e6ccef575a4698b66f5de06c7ecd61589132d5a91d098 f7f3f9285415a9 Status: Downloaded newer image for ubuntu:latest docker.io/library/ubuntu:latest The container experience (conti.) Examine the locally available images with docker images $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE ubuntu latest 6b7dfa7e8fdb 10 days ago 77.8MB debian latest 291bf168077c 13 days ago 124MB mysql 8.0.22 d4c3cafb11d 23 months ago 545MB Test run: $ docker run debian /bin/echo "Hello World" Hello World Docker created a container from the Debian base image and ran the command /bin/echo "Hello World" inside it. The container experience (conti.) Start an interactive shell (bash)within the container and connects the “outer” shell’s I/O channels to it: $ docker run --hostname debian -it debian /bin/bash root@debian:/# ls bin boot dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var root@debian:/# uname -r 5.4.0-54-generic root@debian:/# ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 4024 3348 pts/0 Ss 00:58 0:00 /bin/bash root 342 0.0 0.0 6752 3004 pts/0 R+ 00:59 0:00 ps aux root@debian:/# exit $ uname -r 5.4.0-54-generic The container experience (conti.) - Long-lived containers that run in the background and accept connections over the network; background (-d) a container named “nginx” - tunnel port 8080 from the host into the port 80 within the container $ docker run -p 8080:80 --hostname nginx --name nginx -d nginx Unable to find image 'nginx:latest' locally latest: Pulling from library/nginx 025c56f98b67: Pull complete ec0f5d052824: Pull complete cc9fb8360807: Pull complete defc9ba04d7c: Pull complete 885556963dad: Pull complete f12443e5c9f7: Pull complete Digest: sha256:75263be7e5846fc69cb6c42553ff9c93d653d769b94917dbda71d42d3f3c00d 3 Status: Downloaded newer image for nginx:latest 56dd3a7703fa3975741716c3040436cd930e2e2d70296a49794d29bd5391d247 $ curl localhost:8080 Welcome to nginx!... The container experience (conti.) docker ps shows a brief summary of running containers: $ docker ps IMAGE COMMAND STATUS PORTS nginx "/docker-entrypoint...." Up 4 minutes 0.0.0.0:8080->80/tcp debian "/bin/bash" Up 14 minutes The container experience (conti.) docker exec creates a new process in an existing container. For example, to debug or troubleshoot, we could start an interactive shell in a container: $ docker exec -ti nginx bash root@nginx:/# apt-get update && apt-get -y install procps root@nginx:/# ps ax PID TTY STAT TIME COMMAND 1 ? Ss 0:00 nginx: master process nginx -g daemon off; 28 ? S 0:00 nginx: worker process 29 pts/0 Ss 0:00 bash 370 pts/0 R+ 0:00 ps ax The container experience (conti.) We can stop and start the container: $ docker stop nginx nginx $ docker ps --format "{{.ID}}\t{{.Image}}:{{.Status}}" 6dae120a4546 debian: Up About an hour $ docker start nginx nginx $ docker ps --format "{{.ID}}\t{{.Image}}:{{.Status}}" 1fa2edd2d6f2 nginx: Up 3 seconds 6dae120a4546 debian: Up About an hour $ docker ps -a --format "{{.ID}}\t{{.Image}}:\t{{.Status}}" 1fa2edd2d6f2 nginx: Up 9 minutes 6dae120a4546 debian: Up 2 hours 1168bf7fe827 debian: Exited (0) 2 hours ago 399c77434b36 debian: Exited (0) 2 hours ago The container experience (conti.) When we finish with the container, we can stop and remove it: $ docker stop nginx && docker rm nginx nginx nginx $ docker container ls CONTAINER ID IMAGE COMMAND CREATED STATUS … 6dae120a4546 debian "/bin/bash" 2 hours ago Up 2 hours …  The filesystem layers for most containers consist of static application code, libraries, and other supporting or OS files. The read/write filesystem layer allows containers to make local Volumes modifications to these layers. However, heavy reliance on the overlay filesystem isn’t the best storage solution for data-intensive applications such as databases. For those kinds of apps, Docker has the notion of volumes.  Volumes ❏ Independent, writable directory within a container that’s maintained separately from the union filesystem ❏ If the container is removed, the data in the volume persists and can be accessed from the host ❏ Can also be shared among multiple containers  Volumes (conti.) We add a volume to a container with docker’s -v argument: $ docker run -v /data --rm --hostname web --name web -d nginx 89775399208ba42c9d9961ca38c0448cfc07059293d72782732841d06c08a8ea $ docker inspect -f '{{ json.Mounts }}' web [{"Type":"volume","Name":"d3baabb9b0f7bd744b97e39d7c66b303a557f896919 ccf67e41878484d6934ef","Source":"/var/lib/docker/volumes/d3baabb9b0f7 bd744b97e39d7c66b303a557f896919ccf67e41878484d6934ef/_data","Destinat ion":"/data","Driver":"local","Mode":"","RW":true,"Propagation":""}] $ sudo touch /var/lib/docker/volumes/d3baabb9b0f7bd744b97e39d7c66b303a557f896919cc f67e41878484d6934ef/_data/test $ docker exec -it web bash root@web:/# ls /data test - If /data already exists within the container, any files found there are copied to the volume - If the container terminates or needs to be removed, we can find the data volume at the Source directory on the host  Volumes (conti.) For a higher-level overview of volumes on the system, we run $ docker volume ls DRIVER VOLUME NAME local d3baabb9b0f7bd744b97e39d7c66b303a557f896919ccf67e41878484d6934ef “bind mounts” mount volumes on the host and in containers simultaneously $ docker run -v /mnt/data:/data --rm --name web -d nginx ba8aa08e14ec5112e032c523260ac26d1508cae80bba26f59b233abdd8de934 $ sudo touch /mnt/data/test $ docker exec -it web bash root@ba8aa08e14ec:/# ls /data test For bind-mounted volumes, Docker does not copy existing files from the container’s mount directory to the volume. As with a traditional filesystem mount, the volume’s contents supersede the original contents of the container’s mount directory.  Docker networks Docker creates three default networking options. List them: $ docker network ls NETWORK ID NAME DRIVER SCOPE d80ce7b6c852 bridge bridge local b3581338a28d host host local 77acecccbe26 none null local ❏ “bridge” connects the host’s network to the container namespace; ❏ containers reside on a private namespaced network within the host ❏ Docker creates iptables rules that route traffic from the host’s public interface to the container’s interface on the bridge network ❏ With “host”” networking, no separate network namespace is used ❏ container shares the network stack with the host, including all its interfaces ❏ “None”” networking indicates that Docker shouldn’t take any steps whatsoever to configure networking. ❏ for advanced use cases that have custom net- working requirements\ ❏ Pass the --net argument to docker run to select a container’s network.  Namespaces and the bridge network ❏ A bridge is a Linux kernel feature that connects two network segments ❏ Docker creates a bridge called docker0 on the host ❏ Each container is given a namespaced virtual network interface that has an IP address within the bridged network range ❖ container’s eth0 is paired with vethXXX ❖ vethXXX is linked to docker0 same as a machine linked to switch $ docker inspect -f '{{ json.NetworkSettings.Networks.bridge }}' ubu_br { "IPAMConfig":null, "Links":null, "Aliases":null, "NetworkID":"d80ce7b6c8526c009790e54bb212c5d06ccd8668a021de3325 a316daa729d0de", "EndpointID":"caf906d0cacf10ae46dee80736c83973ef0fbd5b20fdf1025514 bd6429cf2e2b", "Gateway":"172.17.0.1", "IPAddress":"172.17.0.2", "IPPrefixLen":16, "IPv6Gateway":"", "GlobalIPv6Address":"", "GlobalIPv6PrefixLen":0, "MacAddress":"02:42:ac:11:00:02", "DriverOpts":null }  Image Containerize your own building applications by building images that include your application code. Image building 1. The build process begins with a base image. 2. Add your application by committing any changes as new layers and saving the image to the local image database. 3. Then a. can create containers from the image b. can also push your image to a registry to make it accessible to other systems running Docker  Building from a Dockerfile ❏ A Dockerfile is a recipe for building an image. ❏ Contains a series of instructions and shell commands. ❏ docker build command reads the Dockerfile, runs its instructions in sequence, and commits the result as an image Example of Using Dockerfile (conti.) Dockerfile that builds the official NGINX image for Debian: Example of Using Dockerfile Build a derived NGINX image that adds a custom index.html, replacing the default from the official image. Example of Using Dockerfile Run the image and see our customized index.html: