资源限制

LimitRanger

LimitRanger 概述

https://k8s-docs.netlify.app/en/docs/concepts/policy/limit-range

LimitRanger，名称空间级资源限制
LimitRanger 可以为 Pod、container、PVC等在没有设定默认资源限制时为其设定默认值，也可以设定其资源使用的范围区间，以避免应用占用过多的系统资源。

LimitRanger Explain

max 和 min 指的是创建的对象所使用的资源必须在此范围之间
default 和 defaultRequest 只有在未对创建的对象所使用资源加以限制时才会生效

apiVersion: v1
kind: LimitRange
metadata:
  name: core-resource-limits
  namespace: dev # 对dev名称空间内的资源使用加以限制
spec:
  limits <[]Object> -required- # 对象列表格式，可以设定多个限制的资源对象
    type	<string> -required- # 针对哪些资源类型设定限制，可以为Pod、Container、PersistentVolumeClaim
    default <map[string]string> # 创建资源时，如未对使用资源加以限制，则使用此默认值（最大使用量）
    defaultRequest <map[string]string> # 创建资源时，如未对使用资源加以限制，则使用此默认值（初始请求量）
    max <map[string]string> # 创建资源时，最大使用的资源范围
    min <map[string]string> # 创建资源时，最小使用的资源范围
    maxLimitRequestRatio <map[string]string> # max和min两者定义的值 不能大于相差的比例

Pod

apiVersion: v1
kind: LimitRange
metadata:
  name: core-resource-limits
  namespace: dev # 对dev名称空间内的资源使用加以限制
spec:
  limits:
    - type: Pod # 对Pod加以限制
      max: # 创建Pod时，最大使用的cpu和内存资源范围
        cpu: "4" 
        memory: "4Gi" 
      min: # 创建Pod时，最小使用的cpu和内存资源范围
        cpu: "500m" 
        memory: "16Mi"

验证

# kubectl describe limitranges -n dev
Name:       core-resource-limits
Namespace:  dev
Type        Resource  Min   Max  Default Request  Default Limit  Max Limit/Request Ratio
----        --------  ---   ---  ---------------  -------------  -----------------------
Pod         cpu       500m  4    -                -              -
Pod         memory    16Mi  4Gi  -                -              -

---------------------------------

apiVersion: v1
kind: Pod
metadata:
  name: demoapp
  namespace: dev
spec:
  containers:
  - name: demo
    image: ikubernetes/demoapp:v1.0
    imagePullPolicy: IfNotPresent

# 因未cpu和memory最低资源需求，所以报错
# kubectl apply -f demoapp.yaml 
Error from server (Forbidden): error when creating "demoapp.yaml": pods "demoapp" is forbidden: [minimum memory usage per Pod is 16Mi.  No request is specified, minimum cpu usage per Pod is 500m.  No request is specified, maximum memory usage per Pod is 4Gi.  No limit is specified, maximum cpu usage per Pod is 4.  No limit is specified]


---------------------------------

apiVersion: v1
kind: Pod
metadata:
  name: demoapp
  namespace: dev
spec:
  containers:
  - name: demo
    image: ikubernetes/demoapp:v1.0
    imagePullPolicy: IfNotPresent
    resources:
      requests:
        cpu: "600m"
        memory: "64Mi"
      limits:
        cpu: "5"
        memory: "5Gi"

# 虽然满足了cpu和memory最低资源需求，但超出了max限制，因此报错
# kubectl apply -f demoapp.yaml 
Error from server (Forbidden): error when creating "demoapp.yaml": pods "demoapp" is forbidden: [maximum cpu usage per Pod is 4, but limit is 5, maximum memory usage per Pod is 4Gi, but limit is 5368709120]



---------------------------------


apiVersion: v1
kind: Pod
metadata:
  name: demoapp
  namespace: dev
spec:
  containers:
  - name: demo
    image: ikubernetes/demoapp:v1.0
    imagePullPolicy: IfNotPresent
    resources:
      requests:
        cpu: "1"
        memory: "1Gi"
      limits:
        cpu: "2"
        memory: "2Gi"

# 合理范围内不会报错
# kubectl apply -f demoapp.yaml 
pod/demoapp created

Container

apiVersion: v1
kind: LimitRange
metadata:
  name: core-resource-limits
  namespace: dev # 对dev名称空间内的资源使用加以限制
spec:
  limits:
    - type: Container # 对Container加以限制
      max:
        cpu: "4" 
        memory: "1Gi" 
      min:
        cpu: "100m" 
        memory: "4Mi" 
      default: # 创建Container时，如未对使用资源加以限制，则使用此默认值（最大使用量，resources.limits）
        cpu: "2" 
        memory: "512Mi" 
      defaultRequest: # 创建Container时，如未对使用资源加以限制，则使用此默认值（初始请求量，resources.requests）
        cpu: "500m" 
        memory: "64Mi" 
      maxLimitRequestRatio: # max和min两者定义的值 不能大于相差的比例
        cpu: "4" # 表示max和min两者定义相差不能大于4倍，假设requests定义了cpu: "500m"，则limits中定义的cpu不能大于2000m

验证

# kubectl describe limitranges -n dev
Name:       core-resource-limits
Namespace:  dev
Type        Resource  Min   Max  Default Request  Default Limit  Max Limit/Request Ratio
----        --------  ---   ---  ---------------  -------------  -----------------------
Container   cpu       100m  4    500m             2              4
Container   memory    4Mi   1Gi  64Mi             512Mi          -


---------------------------------

# 未定义资源限制
piVersion: v1
kind: Pod
metadata:
  name: demoapp
  namespace: dev
spec:
  containers:
  - name: demo
    image: ikubernetes/demoapp:v1.0
    imagePullPolicy: IfNotPresent

# kubectl apply -f demoapp.yaml 
pod/demoapp created

# kubectl describe pod -n dev demoapp 
Name:         demoapp
Namespace:    dev
...
    # 由LimitRanger生成的默认值：
    Limits:
      cpu:     2
      memory:  512Mi
    Requests:
      cpu:        500m
      memory:     64Mi
...


---------------------------------

apiVersion: v1
kind: Pod
metadata:
  name: demoapp
  namespace: dev
spec:
  containers:
  - name: demo
    image: ikubernetes/demoapp:v1.0
    imagePullPolicy: IfNotPresent
    resources:
      requests:
        cpu: "0.2" # 最低需求0.2核
        memory: "256Mi"
      limits:
        cpu: "1" # 最大限制1核，高出requests 5倍，超出4倍的限制
        memory: "512Mi"

# 因此报错
# kubectl apply -f demoapp.yaml 
Error from server (Forbidden): error when creating "demoapp.yaml": pods "demoapp" is forbidden: cpu max limit to request ratio per Container is 4, but provided ratio is 5.000000


apiVersion: v1
kind: Pod
metadata:
  name: demoapp
  namespace: dev
spec:
  containers:
  - name: demo
    image: ikubernetes/demoapp:v1.0
    imagePullPolicy: IfNotPresent
    resources:
      requests:
        cpu: "0.2" # 最低需求0.2核
        memory: "256Mi"
      limits:
        cpu: "0.5" # 最大限制0.5核，高出requests 2.5倍，未超出4倍的限制
        memory: "512Mi"

# 因此不会报错
# kubectl apply -f demoapp.yaml 
pod/demoapp created


# kubectl describe -n dev pod demoapp 
Name:         demoapp
Namespace:    dev
...
    Limits:
      cpu:     500m
      memory:  512Mi
    Requests:
      cpu:        200m
      memory:     256Mi
...

PersistentVolumeClaim

apiVersion: v1
kind: LimitRange
metadata:
  name: core-resource-limits
  namespace: dev # 对dev名称空间内的资源使用加以限制
spec:
  limits:
    - type: PersistentVolumeClaim # 对PVC加以限制
      max: # 不能申请超过10G的PV
        storage: "10Gi"
      min: # 不能申请低于1G的PV
        storage: "1Gi"
      default: # 默认不能申请超过5G的PV
        storage: "5Gi"
      defaultRequest: # 默认申请1G的PV
        storage: "1Gi"
      maxLimitRequestRatio: # 请求和限制不能相差5倍
        storage: "5"

LimitRanger Example-1

apiVersion: v1
kind: LimitRange
metadata:
  name: core-resource-limits
  namespace: dev # 对dev名称空间内的资源使用加以限制
spec:
  limits:
    - type: Pod # 对Pod加以限制
      max: # 创建Pod时，最大使用的cpu和内存资源范围
        cpu: "4" 
        memory: "4Gi" 
      min: # 创建Pod时，最小使用的cpu和内存资源范围
        cpu: "500m" 
        memory: "16Mi" 
    - type: Container # 对Container加以限制
      max:
        cpu: "4" 
        memory: "1Gi" 
      min:
        cpu: "100m" 
        memory: "4Mi" 
      default: # 创建Container时，如未对使用资源加以限制，则使用此默认值（最大使用量）
        cpu: "2" 
        memory: "512Mi" 
      defaultRequest: # 创建Container时，如未对使用资源加以限制，则使用此默认值（初始请求量）
        cpu: "500m" 
        memory: "64Mi" 
      maxLimitRequestRatio: # max和min两者定义的值 不能大于相差的比例
        cpu: "4" # 表示max和min两者定义相差不能大于4倍，假设requests定义了cpu: "500m"，则limits中定义的cpu不能大于2000m
    - type: PersistentVolumeClaim # 对PVC加以限制
      max:
        storage: "10Gi"
      min:
        storage: "1Gi"
      default:
        storage: "5Gi"
      defaultRequest:
        storage: "1Gi"
      maxLimitRequestRatio:
        storage: "5"

ResourceQuota

ResourceQuota 概述

ResourceQuota，名称空间级资源限制，可限制名称空间中处于非终止状态的所有Pod对象的计算资源需求及计算资源限制总量。
ResourceQuota 资源还支持为本地名称空间中的PVC存储资源的需求总量和限制总量提供限额，它能够分别从名称空间中的全部PVC、隶属于特定存储类的PVC以及基于本地临时存储的PVC三个类别分别进行定义。
- 本地临时存储的PVC就是local存储卷

PS：

ResourceQuota 主要是用于限制每个命名空间中的资源使用情况，包括 CPU、内存、存储等。它**并不直接限制节点上 Pod 的数量。**而限制的是名称空间的Pod数量

ResourceQuota Explain

总量限额：名称空间内所有pod、PVC等资源加在一起总共的资源限制

apiVersion: v1
kind: ResourceQuota
metadata:
  name: resourcequota-demo
  namespace: dev # 限制的名称空间
spec:
  hard: # 定义硬限制，硬限制表示必须在一定范围内，与其对应的有软限制，软限制表示允许在一定宽限期内超出范围 超过宽限期后即被删除
    pods: "5" # 最多创建5个pod
    # 自v1.9版本起开始支持以count/<resource>.<group>的格式支持对所有资源类型对象的计数配额，例如count/deployments.apps、count/deployments.extensions和 count/services等。
    count/services: "5" # 最多创建5个service，核心v1群组无需指定后缀
    count/configmaps: "5"
    count/secrets: "5"
    count/deployments.apps: "2" # 最多创建2个deployments，特定群组需指定后缀.apps
    count/statefulsets.apps: "2"
    count/cronjobs.batch: "2"
    # requests和limits的总量限额：
    requests.cpu: "2" # CPU资源相关请求的总量限额；
    requests.memory: "4Gi" # 内存资源相关请求的总量限额；
    limits.cpu: "4" # CPU资源相关限制的总量限额；
    limits.memory: "8Gi" # 内存资源相关限制的总量限额；
    # PVC的总量限额：
    persistentvolumeclaims: "6" # 可以创建的PVC总数限额；数量限制；
    requests.storage: "20Gi" # 所有PVC存储需求的总量限额；空间限制；
    # <storage-class-name>.storageclass.storage.k8s.io/requests.storage：特定的存储类上可使用的所有PVC存储需求的总量限额：
    longhorn.storageclass.storage.k8s.io/requests.storage: "20Gi"
    # <storage-class-name>.storageclass.storage.k8s.io/persistentvolumeclaims：特定的存储类上可使用的PVC总数限额；
    longhorn.storageclass.storage.k8s.io/persistentvolumeclaims: "6"

    #requests.ephemeral-storage：所有Pod可以使用的本地临时存储资源的requets总量；
    #limits.ephemeral-storage：所有Pod可用的本地临时存储资源的limits总量。

注意事项

注意：只要定义了限额就必须在Pod中定义对应的限制，否则会报错
例如：定义了 requests.cpu 后在创建Pod时也必须指定 requests.cpu

# vim demoapp.yaml
apiVersion: v1
kind: Pod
metadata:
  name: demoapp
  namespace: dev
spec:
  containers:
  - name: demo
    image: ikubernetes/demoapp:v1.0
    imagePullPolicy: IfNotPresent

# kubectl apply -f demoapp.yaml 
Error from server (Forbidden): error when creating "demoapp.yaml": pods "demoapp" is forbidden: failed quota: resourcequota-demo: must specify requests.cpu

ResourceQuota Example

pods

yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: resourcequota-demo
  namespace: dev
spec:
  hard:
    pods: "5" # 最多创建5个pod

验证

# kubectl get resourcequotas -n dev
NAME                 AGE   REQUEST     LIMIT
resourcequota-demo   10s   pods: 0/5   
# kubectl describe resourcequotas -n dev
Name:       resourcequota-demo
Namespace:  dev
Resource    Used  Hard
--------    ----  ----
pods        0     5



# kubectl run demoapp01 --image=nginx:alpine -n dev
pod/demoapp01 created
# kubectl describe resourcequotas -n dev
Name:       resourcequota-demo
Namespace:  dev
Resource    Used  Hard
--------    ----  ----
pods        1     5


# 超出限制的数量后Pod将无法创建
# for i in `echo {2..6}` ; do kubectl run "demoapp0${i}" --image=nginx:alpine -n dev ; done
pod/demoapp02 created
pod/demoapp03 created
pod/demoapp04 created
pod/demoapp05 created
Error from server (Forbidden): pods "demoapp06" is forbidden: exceeded quota: resourcequota-demo, requested: pods=1, used: pods=5, limited: pods=5


# kubectl describe resourcequotas -n dev
Name:       resourcequota-demo
Namespace:  dev
Resource    Used  Hard
--------    ----  ----
pods        5     5

services

yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: resourcequota-demo
  namespace: dev
spec:
  hard:
    count/services: "5" # 最多创建5个service，核心v1群组无需指定后缀

验证

# kubectl apply -f resourcequota-demo.yaml 
resourcequota/resourcequota-demo configured

# 在之前定义的基础上指定会以当前配置为准（之前定义了限制创建5个pod，但此次yaml清单中只限制了services数量，则之前限制的pod数量会被删除）
# kubectl describe resourcequotas -n dev
Name:           resourcequota-demo
Namespace:      dev
Resource        Used  Hard
--------        ----  ----
count/services  0     5


# kubectl create service clusterip demosvc01 --tcp=80:80 -n dev
service/demosvc01 created

# kubectl describe resourcequotas -n dev
Name:           resourcequota-demo
Namespace:      dev
Resource        Used  Hard
--------        ----  ----
count/services  1     5


# 超出限制的数量后services将无法创建
# for i in `echo {2..6}` ; do kubectl create service clusterip "demosvc0${i}" --tcp=80:80  -n dev ; done
service/demosvc02 created
service/demosvc03 created
service/demosvc04 created
service/demosvc05 created
error: failed to create ClusterIP service: services "demosvc06" is forbidden: exceeded quota: resourcequota-demo, requested: count/services=1, used: count/services=5, limited: count/services=5



# kubectl describe resourcequotas -n dev
Name:           resourcequota-demo
Namespace:      dev
Resource        Used  Hard
--------        ----  ----
count/services  5     5

deployments

yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: resourcequota-demo
  namespace: dev
spec:
  hard:
    count/deployments.apps: "2" # 最多创建2个deployments，特定群组需指定后缀.apps

验证

# kubectl apply -f resourcequota-demo.yaml 
resourcequota/resourcequota-demo configured


# kubectl describe resourcequotas -n dev
Name:                   resourcequota-demo
Namespace:              dev
Resource                Used  Hard
--------                ----  ----
count/deployments.apps  0     2



# kubectl create deployment deploy01 --image=nginx:apline -n dev
deployment.apps/deploy01 created
# kubectl describe resourcequotas -n dev
Name:                   resourcequota-demo
Namespace:              dev
Resource                Used  Hard
--------                ----  ----
count/deployments.apps  1     2


# kubectl create deployment deploy02 --image=nginx:apline -n dev
deployment.apps/deploy02 created
# kubectl describe resourcequotas -n dev
Name:                   resourcequota-demo
Namespace:              dev
Resource                Used  Hard
--------                ----  ----
count/deployments.apps  2     2


# 超出限制的数量后deployment将无法创建
# kubectl create deployment deploy03 --image=nginx:apline -n dev
error: failed to create deployment: deployments.apps "deploy03" is forbidden: exceeded quota: resourcequota-demo, requested: count/deployments.apps=1, used: count/deployments.apps=2, limited: count/deployments.apps=2

requests.cpu

yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: resourcequota-demo
  namespace: dev
spec:
  hard:
    requests.cpu: "2" # CPU资源相关请求的总量限额；

验证

# kubectl describe resourcequotas -n dev
Name:         resourcequota-demo
Namespace:    dev
Resource      Used  Hard
--------      ----  ----
requests.cpu  0     2


# vim demoapp01.yaml
apiVersion: v1
kind: Pod
metadata:
  name: demoapp01
  namespace: dev
spec:
  containers:
  - name: demo
    image: ikubernetes/demoapp:v1.0
    imagePullPolicy: IfNotPresent
    resources:
      requests:
        cpu: "0.5"
# kubectl apply -f demoapp01.yaml 
pod/demoapp01 created
# kubectl describe resourcequotas -n dev
Name:         resourcequota-demo
Namespace:    dev
Resource      Used  Hard
--------      ----  ----
requests.cpu  500m  2


# vim demoapp02.yaml
apiVersion: v1
kind: Pod
metadata:
  name: demoapp02
  namespace: dev
spec:
  containers:
  - name: demo
    image: ikubernetes/demoapp:v1.0
    imagePullPolicy: IfNotPresent
    resources:
      requests:
        cpu: "1"
# kubectl apply -f demoapp02.yaml 
pod/demoapp02 created
# kubectl describe resourcequotas -n dev
Name:         resourcequota-demo
Namespace:    dev
Resource      Used   Hard
--------      ----   ----
requests.cpu  1500m  2 # 限制配额会累加
# kubectl get pod -n dev
NAME        READY   STATUS    RESTARTS   AGE
demoapp01   1/1     Running   0          5m39s
demoapp02   1/1     Running   0          3m56s



# vim demoapp03.yaml
apiVersion: v1
kind: Pod
metadata:
  name: demoapp03
  namespace: dev
spec:
  containers:
  - name: demo
    image: ikubernetes/demoapp:v1.0
    imagePullPolicy: IfNotPresent
    resources:
      requests:
        cpu: "1" # 上面已经使用了1.5个cpu，而限制是最多使用2个cpu，1.5+1=2.5 显然已经超出范围，因此会报以下错误：
# kubectl apply -f demoapp03.yaml 
Error from server (Forbidden): error when creating "demoapp.yaml": pods "demoapp03" is forbidden: exceeded quota: resourcequota-demo, requested: requests.cpu=1, used: requests.cpu=1500m, limited: requests.cpu=2

—

限制每个节点运行的Pod数量

要限制每个节点运行的 Pod 数量，可以使用 Kubernetes 的调度器策略。