Introduction
Connecting applications or machines to a VPN (Virtual Private Network) is a commonly used method to make secure communications between two networks. Note that, described solution is not part of site to site VPN solutions. Also, the solution is directly deployed on Bare Metal Kubernetes with OpenVPN Client. While we redirect all of our Kubernetes’ Pods network traffic through a specific OpenVPN Server, we protect our Kubernetes Networking.
First of all, every pod in your cluster can connect different VPN Serves. The most important side of this solution is the flexibility to manage your resources without managing persistent OpenVPN Client hosts and distribute them from Kubernetes. Also, after connecting a pod to OpenVPN Server, your pod still will be communicating with your Kubernetes Services and Resources.
It can provide
- a secure communication between your pod and your cloud service(s) and instance(s).
- a secure communication between your resource(s) located in headquarter(s) and your resource(s) from subsidiary office(s).
- a secure jump host solution for managing resource(s) in different location(s) and region(s). (For docker solution visit Eren Manış’s Solution).
- a secure and untraceable connection to the internet.
Following image describes the general architecture of this use cases.
Due to Kubernetes Networking, it provides container-to-container communication like they are in the same host. Every container can reach any other container from localhost. For more information about networking in Kubernetes please read following documentation of Kubernetes. Networking Document on Github.
It provides a simple networking architecture inside a pod.
- Container inside the pod connects to a OpenVPN Server
- All the containers connect to the same OpenVPN Server directly.
OpenVPN Server deployment is not included because there are a lot of articles about this deployment process. If you want to reach cloud or bare metal deployment of OpenVPN Server, pleases visit the following OpenVPN Documentation. You can easily find the solution that fits your use cases such as cloud environments with BYOL (Bring Your Own License) licencing method.
Kubernetes
There are some points before we start the deployment.
- Version Compatibility
- ConfigMap Read Only (Permission Denied to execute script)
- OpenVPN Routing
Version Compatibility
Following Deployment YAML can be used after Kubernetes v1.16. Because of the changes in the Kubernetes API, we have to migrate our old Deployments to the new ones. It affects API versions with apps/v1 and we have to define selector in our deployment specs.
After Kubernetes v1.16, Deployment in the extensions/v1beta1, apps/v1beta1, and apps/v1beta2 API versions is no longer served.
Please read the Vallery Lancey (Lyft)’s blog about API deprecation and converting old API objects to new API objects.
ConfigMap Read Only
ConfigMaps in Kubernetes are mounted as Read Only mode by default and container cannot execute scripts even though the default user is root (privileged: true). We have a workaround with InitContainer config by mounting empty volume. It is done due to security reasons. For more information about it, please visit joelsmith’s PR.
OpenVPN Routing
OpenVPN Client oppresses all of the default routing on Kubernetes Pod. It prevents the communications between client side (Kubernetes Services and Hosts in Local Area) and client itself. It only matters when your OpenVPN Server or OpenVPN Client configuration redirects all traffic through OpenVPN Server. I would like to thank Eren Manış for a pair debugging session.
Deployment
Let’s explain the OpenVPN Client deployment on Bare Metal Kubernetes.
Ingredients:
- dperson/openvpn-client image
- ConfigMap (The Script to override the Routing)
- 2 x Secret (*.ovpn file, auth.txt file)
- Deployment
First we create our Secret. The *.ovpn file that contains OpenVPN Client side configurations and certificate and the auth.txt file should contain username line and password line that is defined to OpenVPN Server.
Sample auth.txt:
username password
Creating Secrets:
kubectl create secret generic vpn-config --from-file=client.ovpn kubectl create secret generic vpn-auth --from-file=auth.txt
ConfigMap 10routeconfig.yaml:
kind: ConfigMap
metadata:
name: route-script
apiVersion: v1
data:
route-override.sh: |-
#!/bin/sh
VPN_GATEWAY=$(route -n | awk 'NR==3' | awk '{ print $2 }')
ip route del 0.0.0.0/1 via $VPN_GATEWAY
echo "Route Updated"
Apply ConfigMap:
kubectl apply -f 10routeconfig.yaml
Deployment 20deployment.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: openvpn-client
spec:
selector:
matchLabels:
app: openvpn-client
vpn: vpn-id
replicas: 1
template:
metadata:
labels:
app: openvpn-client
vpn: vpn-id
spec:
volumes:
- name: vpn-config
secret:
secretName: vpn-config
items:
- key: client.ovpn
path: client.ovpn
- name: vpn-auth
secret:
secretName: vpn-auth
items:
- key: auth.txt
path: auth.txt
- name: route-script
configMap:
name: route-script
items:
- key: route-override.sh
path: route-override.sh
- name: tmp
emptyDir: {}
initContainers:
- name: vpn-route-init
image: busybox
command: ['/bin/sh', '-c', 'cp /vpn/route-override.sh /tmp/route/route-override.sh; chown root:root /tmp/route/route-override.sh; chmod o+x /tmp/route/route-override.sh;']
volumeMounts:
- name: tmp
mountPath: /tmp/route
- name: route-script
mountPath: /vpn/route-override.sh
subPath: route-override.sh
containers:
- name: vpn
image: dperson/openvpn-client
command: ["/bin/sh","-c"]
args: ["openvpn --config 'vpn/client.ovpn' --auth-user-pass 'vpn/auth.txt' --script-security 3 --route-up /tmp/route/route-override.sh;"]
stdin: true
tty: true
securityContext:
privileged: true
capabilities:
add:
- NET_ADMIN
env:
- name: TZ
value: "Turkey"
volumeMounts:
- name: vpn-config
mountPath: /vpn/client.ovpn
subPath: client.ovpn
- name: vpn-auth
mountPath: /vpn/auth.txt
subPath: auth.txt
- name: tmp
mountPath: /tmp/route
- name: app1
image: python:3.6-stretch
command:
- sleep
- "100000"
tty: true
dnsConfig:
nameservers:
- 8.8.8.8
- 8.8.4.4
OpenVPN Command has some extended parameters. For the explanation of these parameters please visit Reference Manual for OpenVPN 2.4.
Conclusion
Secure connection over some infrastructures like VPN is very useful in certain use cases. These are generally about secure communications between our service(s) in different region(s).
To sum it up, the solution covers some of the use cases in this area. We learn how to deploy OpenVPN Client on Bare Metal Kubernetes while we persist our Kubernetes Networking.
For further enhancement, you can reduce your Kubernetes YAMLs by creating your own docker image that includes routing-script.
Buğra Öztürk
DataBoss Project Technical Lead