Monitoring GPU workloads on GKE with NVIDIA Data Center GPU Manager (DCGM)

apiVersion: v1 <br /> kind: ConfigMap <br /> metadata: <br /> name: nvidia-dcgm-exporter-metrics <br /> … <br /> data: <br /> counters.csv: | <br /> # Utilization (the sample period varies depending on the product),, <br /> DCGM_FI_DEV_GPU_UTIL, gauge, GPU utilization (in %). <br /> DCGM_FI_DEV_MEM_COPY_UTIL, gauge, Memory utilization (in %). <br /> <br /> # Utilization of IP blocks,, <br /> DCGM_FI_PROF_SM_ACTIVE, gauge, <br /> DCGM_FI_PROF_SM_OCCUPANCY, gauge, <br /> DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, gauge, <br /> DCGM_FI_PROF_PIPE_FP64_ACTIVE, gauge, <br /> DCGM_FI_PROF_PIPE_FP32_ACTIVE, gauge, <br /> DCGM_FI_PROF_PIPE_FP16_ACTIVE, gauge, <br /> <br /> # Memory usage,, <br /> DCGM_FI_DEV_FB_FREE, gauge, <br /> DCGM_FI_DEV_FB_USED, gauge, <br /> DCGM_FI_DEV_FB_TOTAL, gauge, <br /> <br /> # PCIE,, <br /> DCGM_FI_PROF_PCIE_TX_BYTES, gauge, <br /> DCGM_FI_PROF_PCIE_RX_BYTES, gauge, <br /> <br /> # NVLink,, <br /> DCGM_FI_PROF_NVLINK_TX_BYTES, gauge, <br /> DCGM_FI_PROF_NVLINK_RX_BYTES, gauge,