k8s安装负载均衡器:Metallb
By admin
- 3 minutes read - 637 words在使用kubenetes的过程中,如何将服务开放到集群外部访问是一个重要的问题。当使用云平台(阿里云、腾讯云、AWS等)的容器服务时,我们可以通过配置 service 为 LoadBalancer 模式来绑定云平台的负载均衡器,从而实现外网的访问。但是,如果对于自建的 kubernetes裸机集群,这个问题则要麻烦的多。
祼机集群不支持负载均衡的方式,可用的不外乎NodePort、HostNetwork、ExternalIPs等方式来实现外部访问。但这些方式并不完美,他们或多或少都存在的一些缺点,这使得裸机集群成为Kubernetes生态系统中的二等公民。
MetalLB 旨在通过提供与标准网络设备集成的Network LB实施来解决这个痛点,从而使裸机群集上的外部服务也尽可能“正常运行”,减少运维上的管理成本。它是一种纯软件的解决方案,参考 https://kubernetes.github.io/ingress-nginx/deploy/baremetal/。
从 v0.13.0 版本开始,官方对解决方案进行了部分调整,操作步骤简洁一些,建议使用最新版本,参考官方教程 https://metallb.universe.tf/installation/
部署
- 创建namespace
$ kubectl create namespace metallb-system
- 新建 secret
$ kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
参数 from 前面是两个-
- 部署
root@sxf-virtual-machine:~# kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.12.1/manifests/metallb.yaml
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/controller created
podsecuritypolicy.policy/speaker created
serviceaccount/controller created
serviceaccount/speaker created
clusterrole.rbac.authorization.k8s.io/metallb-system:controller created
clusterrole.rbac.authorization.k8s.io/metallb-system:speaker created
role.rbac.authorization.k8s.io/config-watcher created
role.rbac.authorization.k8s.io/pod-lister created
role.rbac.authorization.k8s.io/controller created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:controller created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:speaker created
rolebinding.rbac.authorization.k8s.io/config-watcher created
rolebinding.rbac.authorization.k8s.io/pod-lister created
rolebinding.rbac.authorization.k8s.io/controller created
daemonset.apps/speaker created
deployment.apps/controller created
查看部署结果
root@sxf-virtual-machine:~# kubectl get pod -n metallb-system
NAME READY STATUS RESTARTS AGE
controller-6cc57c4567-cn766 1/1 Running 0 29m
speaker-4fv86 1/1 Running 0 32m
root@sxf-virtual-machine:~# kubectl get sa -n metallb-system
NAME SECRETS AGE
controller 1 33m
default 1 33m
speaker 1 33m
- 配置 MetalLB
配置 MetalLB 为 Layer 2模式 (使用 yaml 文件部署), 文件 MetalLB-Layer2-Configmap.yaml
kind: ConfigMap
apiVersion: v1
metadata:
name: config
namespace: metallb-system
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- 192.168.3.251-192.168.3.254
这里配置一个由 MetalLB 二层模式控制的 service 外部 IP 段为 192.168.3.251 - 192.168.3.254
。
还可以指定多个网段
addresses:
- 192.168.12.0/24
- 192.168.144.0/20
除了自动分配IP外,Metallb 还支持在定义服务的时候,通过 spec.loadBalancerIP
指定一个静态IP 。
$ kubectl apply -f MetalLB-Layer2-Configmap.yaml
使用kubectl log -f [matellb-contoller-pod]能看到配置更新过程
查看结果
root@sxf-virtual-machine:~# kubectl get cm -n metallb-system
NAME DATA AGE
config 1 22s
kube-root-ca.crt 1 43m
从版本 v0.13.0 版本开始使用 IPAddressPool 代替,参考 https://metallb.universe.tf/configuration/_advanced_ipaddresspool_config/
测试
创建一个Nginx服务,服务类型为LoadBalancer:
deploy.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.23-alpine
ports:
- name: http
containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: nginx
spec:
#loadBalancerIP: x.y.z.a # 指定公网IP
ports:
- name: http
port: 80
protocol: TCP
targetPort: 80
selector:
app: nginx
type: LoadBalancer
服务创建,并查看服务信息:
$ kubectl apply -f deploy.yaml
查看pod
root@sxf-virtual-machine:~# kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx-8c9df995d-pvmkl 1/1 Running 0 2m10s
查看服务
root@sxf-virtual-machine:~# kubectl get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default details ClusterIP 10.110.60.221 <none> 9080/TCP 3d1h
default hostnames ClusterIP 10.111.190.216 <none> 80/TCP 3d1h
default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3d23h
default nginx LoadBalancer 10.108.230.26 192.168.3.253 80:30829/TCP 65s
default productpage ClusterIP 10.98.222.204 <none> 9080/TCP 3d1h
default ratings ClusterIP 10.104.86.58 <none> 9080/TCP 3d1h
default reviews ClusterIP 10.101.246.133 <none> 9080/TCP 3d1h
istio-system istio-eastwestgateway LoadBalancer 10.99.130.139 192.168.3.252 15021:31544/TCP,15443:32575/TCP,15012:30747/TCP,15017:30099/TCP 3d
istio-system istio-ingressgateway LoadBalancer 10.96.196.70 192.168.3.251 15021:30035/TCP,80:31245/TCP,443:31639/TCP 3d22h
istio-system istiod ClusterIP 10.107.246.136 <none> 15010/TCP,15012/TCP,443/TCP,15014/TCP 3d23h
kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 3d23h
这里就发现一些 LoadBalancer 类型的服务,被分配到了固定的 IP地址信息。
服务自动生成的 yaml 文件
root@sxf-virtual-machine:~# kubectl get svc nginx -o yaml
apiVersion: v1
kind: Service
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"nginx","namespace":"default"},"spec":{"ports":[{"name":"http","port":80,"protocol":"TCP","targetPort":80}],"selector":{"app":"nginx"},"type":"LoadBalancer"}}
creationTimestamp: "2021-09-29T11:26:35Z"
name: nginx
namespace: default
resourceVersion: "1274438"
uid: f2fb62dc-e277-4432-8496-2b5fb5984a55
spec:
clusterIP: 10.108.230.26
clusterIPs:
- 10.108.230.26
externalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: http
nodePort: 30355
port: 80
protocol: TCP
targetPort: 80
selector:
app: nginx
sessionAffinity: None
type: LoadBalancer
status:
loadBalancer:
ingress:
- ip: 192.168.3.253
这时如果你 ping
这个 ip 的话,会发现无法 ping 通
root@sxf-virtual-machine:~# ping 192.168.3.253
PING 192.168.3.253 (192.168.3.253): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3
这个是正常的,因为它是一个虚拟IP地址,所以根本无法ping 通(此虚拟IP与物理网卡共用同一个 MAC
地址,那么这个IP是如何工作的呢?又是如何收到流量请求的呢?很值得思考)。
那如何测试这个虚拟IP是否能正常提供服务呢,其实只需要使用 telnet 命令就可以了,如
root@sxf-virtual-machine:~# telnet 192.168.3.253 80
这里我们提供的是一个 web
服务,暴露的监听端口是 80
,对应容器的端口为 30829
。
root@sxf-virtual-machine:~# telnet 192.168.3.253 80
Trying 192.168.3.253...
Connected to 192.168.3.253.
Escape character is '^]'.
看到这个信息则说明此虚拟IP可以正常提供服务了。
这里我们用 curl 192.168.3.253
再次验证一下nginx服务,就会发现返回了 Nginx 的欢迎信息。
root@sxf-virtual-machine:~# curl 192.168.3.253
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
可以看到可以正常访问pod的内容。
剩下的就是我们如何使用了,经常部署方案是为集群服务创建一个 Ingress 的服务,参考 https://stackoverflow.com/questions/65789968/microk8s-metallb-ingress
最后我们清理pod
root@sxf-virtual-machine:~# kubectl delete -f deploy.yaml
deployment.apps "nginx" deleted
service "nginx" deleted
其它
除了本文介绍的 Metallb 另,还有另一种解决方案为 openELB ,它是由国内互联网 青云 公司开源的一种方法,目前在 CNCF 沙盒 在托管,已在国内不少企业如 本来生活、苏州电视台、视源股份、云智天下、Jollychic、QingCloud、百旺、Rocketbyte 等海内外多家企业采用。
参考
- https://github.com/metallb/metallb
- https://kind.sigs.k8s.io/docs/user/loadbalancer/
- https://stackoverflow.com/questions/65789968/microk8s-metallb-ingress
- https://kubernetes.io/zh/docs/concepts/services-networking/ingress/
- https://kubernetes.github.io/ingress-nginx/deploy/baremetal/#a-pure-software-solution-metallb