前言篇幅较长纯手打如有错误请留言一、Prometheus介绍官网https://prometheus.io/docs/introduction/overview/1.什么是PrometheusPrometheus是一个开源监控系统它前身是SoundCloud的警告工具包。从2012年开始许多公司和组织开始使用Prometheus。该项目的开发人员和用户社区非常活跃越来越多的开发人员和用户参与到该项目中。目前它是一个独立的开源项目且不依赖与任何公司。 为了强调这点和明确该项目治理结构Prometheus在2016年继Kurberntes之后加入了Cloud Native Computing Foundation。2.Prometheus的主要特征多维度数据模型灵活的查询语言不依赖分布式存储单个服务器节点是自主的以HTTP方式通过pull模型拉去时间序列数据也通过中间网关支持push模型通过服务发现或者静态配置来发现目标服务对象支持多种多样的图表和界面展示grafana也支持它3.组件Prometheus生态包括了很多组件它们中的一些是可选的主服务Prometheus Server负责抓取和存储时间序列数据客户库负责检测应用程序代码支持短生命周期的PUSH网关基于Rails/SQL仪表盘构建器的GUI多种导出工具可以支持Prometheus存储数据转化为HAProxy、StatsD、Graphite等工具所需要的数据存储格式警告管理器命令行查询工具其他各种支撑工具多数Prometheus组件是Go语言写的这使得这些组件很容易编译和部署。4.架构Prometheus服务可以直接通过目标拉取数据或者间接地通过中间网关拉取数据。它在本地存储抓取的所有数据并通过一定规则进行清理和整理数据并把得到的结果存储到新的时间序列中PromQL和其他API可视化地展示收集的数据。二、安装及配置由于环境限制Prometheus需要通过nginx代理才能访问所以在Prometheus启动命令上加了参数还需要配置nginx反向代理详情如下1.安装Prometheus[rootmaster supp_app]# wget https://github.com/prometheus/prometheus/releases/tag/v3.4.1/prometheus-3.4.1.linux-amd64.tar.gz[rootmaster supp_app]# tar zxf prometheus-3.4.1.linux-amd64.tar.gz[rootmaster supp_app]# cd prometheus-3.4.1.linux-amd64/2.文件介绍[rootmaster prometheus-3.4.1.linux-amd64]# ls -lrttotal294972-rwxr-xr-x11001docker154802412May3118:46 prometheus## Prometheus启动文件-rwxr-xr-x11001docker146211128May3118:46 promtool## Prometheus工具-rw-r--r--11001docker3773May3118:58 NOTICE## 声明Prometheus用了哪些依赖-rw-r--r--11001docker11357May3118:58 LICENSE## 许可证-rw-r--r--11001docker1877Aug2217:47 prometheus.yml## Prometheus配置文件drwxr-xr-x28root root4096Aug2715:00 data## Prometheus自带数据库TSDB的数据目录3.配置解析[rootmaster prometheus-3.4.1.linux-amd64]# cat prometheus.yml# my global configglobal: scrape_interval: 15s# Set the scrape interval to every 15 seconds. Default is every 1 minute.evaluation_interval: 15s# Evaluate rules every 15 seconds. The default is every 1 minute.# scrape_timeout is set to the global default (10s).# Alertmanager configurationalerting: alertmanagers: - static_configs: - targets:# - alertmanager:9093# Load rules once and periodically evaluate them according to the global evaluation_interval.rule_files:# - first_rules.yml# - second_rules.yml# A scrape configuration containing exactly one endpoint to scrape:# Here its Prometheus itself.scrape_configs:# The job name is added as a label jobjob_name to any timeseries scraped from this config.- job_name:prometheus# metrics_path defaults to /metrics# scheme defaults to http.static_configs: - targets:[localhost:9090]# The label name is added as a label label_namelabel_value to any timeseries scraped from this config.labels: app:prometheus更详细的配置参考https://prometheus.io/docs/prometheus/latest/configuration/configuration/4.自编启动脚本由于Prometheus没有启动脚本每次启停都要输入一长串命令简单写个脚本方便后期操作[rootmaster prometheus-3.4.1.linux-amd64]# vim start.sh#!/bin/bash# Author:# Prometheus start scriptPRO_HOME$(cd$(dirname$0);pwd)usage(){echoUsage: /bin/bash start.sh OPTIONechoOPTIONS:start|stop|restart}log_time(){localSTRING${1:-No message provided}localTIME$(date%Y-%m-%d %H:%M:%S).$((date %N/1000000))printf%s--%s\n$TIME$STRING}start(){if[[!-e$PRO_HOME/pro.pid]];thentouch$PRO_HOME/pro.pidfiPIDcat$PRO_HOME/pro.pidif[[-n$PID]];thenlog_timePrometheus service has been start!exit1fi## 添加了--web.external-urlprometheus参数因为机器只能开放80端口用nginx做了反向代理具体的参数解释看下面解释nohup$PRO_HOME/prometheus --web.external-urlprometheus--config.file$PRO_HOME/prometheus.ymlpro.log21sleep3ps-ef|grep$PRO_HOME|grep-vgrep|awk{print $2}pro.pid log_timePrometheus service started!PID1cat$PRO_HOME/pro.pidlog_timePrometheus service pid:$PID1}stop(){PIDcat$PRO_HOME/pro.pidif[[-z$PID]];thenlog_timePrometheus service dont start!elsekill$PIDecho-n$PRO_HOME/pro.pid log_timePrometheus service stoped!fi}case$1instart)start;;stop)stop;;restart)stop start;;*)usage;;esac–web.external-url The URL under which Prometheus is externally reachable (for example, if Prometheus is served via a reverse proxy). Used for generating relative and absolute links back to Prometheus itself. If the URL has a path portion, it will be used to prefix all HTTP endpoints served by Prometheus. If omitted, relevant URL components will be derived automatically.5.启动服务[rootmaster prometheus-3.4.1.linux-amd64]# sh start.sh start2025-08-2716:30:42.942--Prometheusservicestarted!2025-08-2716:30:42.946--Prometheus process pid:3585681[rootmaster prometheus-3.4.1.linux-amd64]# ps -ef | grep 3585681root35856811116:30 pts/1 00:00:00 /opt/supp_app/prometheus-3.4.1.linux-amd64/prometheus --web.external-urlprometheus--config.file/opt/supp_app/prometheus-3.4.1.linux-amd64/prometheus.yml root35858503444503016:31 pts/1 00:00:00grep--colorauto35856816.配置nginx[rootmaster prometheus-3.4.1.linux-amd64]# egrep -v #|^$ /opt/web_app/nginx-1.16.1/conf/nginx.confworker_processes2;events{worker_connections60000;}http{include mime.types;default_type application/octet-stream;sendfile on;keepalive_timeout65;server{listen80;server_name127.0.0.1;## 添加下面这段locationlocation /prometheus/{proxy_pass http://localhost:9090/prometheus/;}location /jenkins{proxy_pass http://localhost:8080;proxy_set_header Host$host;proxy_set_header X-Real-IP$remote_addr;proxy_set_header X-Forwarded-For$proxy_add_x_forwarded_for;proxy_set_header X-Forwarded-Proto$scheme;}location /status{stub_status;allow127.0.0.1;deny all;}error_page500502503504/50x.html;location/50x.html{root html;}}}重启nginx[rootmaster prometheus-3.4.1.linux-amd64]# /opt/web_app/nginx-1.16.1/sbin/nginx -tnginx: the configurationfile/opt/web_app/nginx-1.16.1/conf/nginx.conf syntax is ok nginx: configurationfile/opt/web_app/nginx-1.16.1/conf/nginx.conftestis successful[rootmaster prometheus-3.4.1.linux-amd64]# /opt/web_app/nginx-1.16.1/sbin/nginx -s reload7.浏览器打开Prometheus三、安装exporter本次需要安装redis_exporter、node_exporter、nginx_exporter这几个exporter都可以在Prometheus官网找到。官网链接https://prometheus.io/docs/instrumenting/exporters/1.下载tar包[rootmaster supp_app]# wget https://github.com/prometheus/node_exporter/releases/download/v1.9.1/node_exporter-1.9.1.linux-amd64.tar.gz[rootmaster supp_app]# wget https://github.com/oliver006/redis_exporter/releases/download/v1.69.0/redis_exporter-v1.69.0.linux-amd64.tar.gz[rootmaster supp_app]# wget https://github.com/nginx/nginx-prometheus-exporter/releases/download/v1.3.0/nginx-prometheus-exporter_1.3.0_linux_amd64.tar.gz[rootmaster supp_app]# tar zxf nginx-prometheus-exporter_1.3.0_linux_amd64.tar.gz[rootmaster supp_app]# tar zxf node_exporter-1.9.1.linux-amd64.tar.gz[rootmaster supp_app]# tar zxf redis_exporter-v1.69.0.linux-amd64.tar.gz2.启动exporter启动nginx_exporter[rootmaster nginx_exporter]# nohup ./nginx-prometheus-exporter --nginx.scrape-uri http://127.0.0.1/status [rootmaster nginx_exporter]# ps -ef | grep nginx-proroot48141110Jul08 ? 00:07:07 /opt/supp_app/nginx_exporter/nginx-prometheus-exporter -nginx.scrape-uri http://127.0.0.1/status root36050133444503017:23 pts/1 00:00:00grep--colorauto nginx-pro启动redis_exporter[rootmaster redis_exporter-v1.69.0.linux-amd64]# nohup ./redis_exporter --web.listen-addresslocalhost:9121 --exclude-latency-histogram-metrics [rootmaster redis_exporter-v1.69.0.linux-amd64]# ps -ef | grep redis_exporroot308102010Aug22 ? 00:13:33 ./redis_exporter --web.listen-addresslocalhost:9121 --exclude-latency-histogram-metrics root36144783444503017:49 pts/1 00:00:00grep--colorauto redis_expor启动node_exporter[rootmaster node_exporter-1.9.1.linux-amd64]# nohup /opt/supp_app/node_exporter-1.9.1.linux-amd64/node_exporter node_exporter.log 21 [rootmaster node_exporter-1.9.1.linux-amd64]# ps -ef | grep node_exporterroot322962110Jun19 ? 01:15:01 /opt/supp_app/node_exporter-1.9.1.linux-amd64/node_exporter root36160413444503017:53 pts/1 00:00:00grep--colorauto node_exporter四、配置Prometheus采集数据1.修改配置在prometheus.yml文件里面加入如下配置[rootmaster prometheus-3.4.1.linux-amd64]# cat prometheus.yml# my global configglobal: scrape_interval: 15s# Set the scrape interval to every 15 seconds. Default is every 1 minute.evaluation_interval: 15s# Evaluate rules every 15 seconds. The default is every 1 minute.# scrape_timeout is set to the global default (10s).# Alertmanager configurationalerting: alertmanagers: - static_configs: - targets:# - alertmanager:9093# Load rules once and periodically evaluate them according to the global evaluation_interval.rule_files:# - first_rules.yml# - second_rules.yml# A scrape configuration containing exactly one endpoint to scrape:# Here its Prometheus itself.scrape_configs:# The job name is added as a label jobjob_name to any timeseries scraped from this config.- job_name:prometheus# metrics_path defaults to /metrics# scheme defaults to http.metrics_path:/prometheus/metricsstatic_configs: - targets:[localhost:9090]# The label name is added as a label label_namelabel_value to any timeseries scraped from this config.labels: app:prometheus- job_name:nodestatic_configs: - targets:[localhost:9100]- job_name:nginx_exporterstatic_configs: - targets:[localhost:9113]- job_name:redis_exporter_targetsfile_sd_configs: - files: - targets-redis-instances.json## redis是6节点的集群模式这个文件里面都是redis的访问地址metrics_path: /scrape relabel_configs: - source_labels:[__address__]target_label: __param_target - source_labels:[__param_target]target_label: instance - target_label: __address__ replacement:localhost:9121- job_name: redis_exporter static_configs: - targets:[localhost:9121]查看targets-redis-instances.json文件[rootmaster prometheus-3.4.1.linux-amd64]# lsdata NOTICE prometheus promtool standalone-redis-instances.json targets-redis-instances.json LICENSE pro.log prometheus.yml pro.pid start.sh[rootmaster prometheus-3.4.1.linux-amd64]# cat targets-redis-instances.json[{targets:[redis://localhost:6379,redis://localhost:6380,redis://localhost:6381,redis://localhost:6382,redis://localhost:6383,redis://localhost:6384],labels:{business:core platform}}]需要注意的是本次是测试环境所以6个redis节点都在同一台机器上生产环境最起码要部署在两台机器上面保证redis集群的可用性2.重启Prometheus这里我们用刚才写好的启动脚本直接重启[rootmaster prometheus-3.4.1.linux-amd64]# sh start.sh restart2025-08-2717:57:45.830--Prometheusservicestoped!2025-08-2717:57:48.860--Prometheusservicestarted!2025-08-2717:57:48.865--Prometheus process pid:36175923.通过浏览器查看exporter状态state全是up就没有问题4.查看监控指标可以使用curl命令在服务器上查看各exporter的endpoint[rootmaster web_app]# curl http://localhost:9100/metrics nginx-metric% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed10067496067496005992k0--:--:-- --:--:-- --:--:-- 5992k[rootmaster web_app]# head -20 nginx-metric# HELP go_gc_duration_seconds A summary of the wall-time pause (stop-the-world) duration in garbage collection cycles.# TYPE go_gc_duration_seconds summarygo_gc_duration_seconds{quantile0}3.0135e-05 go_gc_duration_seconds{quantile0.25}3.8797e-05 go_gc_duration_seconds{quantile0.5}4.7997e-05 go_gc_duration_seconds{quantile0.75}5.0852e-05 go_gc_duration_seconds{quantile1}8.6899e-05 go_gc_duration_seconds_sum12.599998827go_gc_duration_seconds_count284197# HELP go_gc_gogc_percent Heap size target percentage configured by the user, otherwise 100. This value is set by the GOGC environment variable, and the runtime/debug.SetGCPercent function. Sourced from /gc/gogc:percent# TYPE go_gc_gogc_percent gaugego_gc_gogc_percent100# HELP go_gc_gomemlimit_bytes Go runtime memory limit configured by the user, otherwise math.MaxInt64. This value is set by the GOMEMLIMIT environment variable, and the runtime/debug.SetMemoryLimit function. Sourced from /gc/gomemlimit:bytes# TYPE go_gc_gomemlimit_bytes gaugego_gc_gomemlimit_bytes9.223372036854776e18# HELP go_goroutines Number of goroutines that currently exist.# TYPE go_goroutines gaugego_goroutines8# HELP go_info Information about the Go environment.# TYPE go_info gauge也可以在浏览器选择一个metric查看数据五、Grafana介绍官网https://grafana.com/docs/grafana/latest/introduction/grafana-enterprise/1.什么是grafanaGrafana是一款开源的‌数据可视化与监控分析平台‌主要用于连接多种数据源如Prometheus、InfluxDB等通过交互式仪表盘实时展示和分析时序数据广泛应用于IT运维、云服务监控等领域。2.核心功能多数据源支持‌支持Prometheus、InfluxDB、Elasticsearch、MySQL等30数据源无需迁移数据即可统一分析。‌‌通过插件扩展兼容性例如集成AWS、腾讯云等云服务监控数据。‌‌动态仪表盘‌提供拖拽式编辑器可自定义折线图、热力图等10图表类型并支持时间范围筛选、阈值告警等交互功能。‌支持团队协作共享仪表盘设置细粒度权限控制。‌‌告警与自动化‌可配置条件触发告警通过邮件、Slack等渠道通知并与PagerDuty等运维工具联动。‌‌3.主要用途系统监控‌结合Prometheus监控服务器、容器集群状态实时展示CPU、内存等指标。‌‌业务分析‌对接MySQL或日志服务可视化用户增长、交易数据等业务指标。‌‌云原生集成‌作为Kubernetes、OpenTelemetry等云原生技术的标准观测工具。‌‌六、安装Grafana1.下载[rootmaster grafana]# wget https://dl.grafana.com/enterprise/release/grafana-enterprise-12.0.2.linux-amd64.tar.gz[rootmaster grafana]# tar zxf grafana-enterprise-12.0.2.linux-amd64.tar.gz2.修改配置文件因为环境限制需要nginx做反向代理所以修改grafana.ini大概在58行的位置[roottest supp_app]# cd grafana-v12.0.2/[roottest grafana-v12.0.2]# lsbin conf Dockerfile docs LICENSE NOTICE.md npm-artifacts packaging plugins-bundled public README.md storybook tools VERSION[roottest grafana-v12.0.2]# cd conf/[roottest conf]# lsdefaults.ini ldap_multiple.toml ldap.toml provisioning sample.ini[roottest conf]# cp defaults.ini grafana.ini[roottest conf]# vim grafana.ini4950# The public facing domain name used to access grafana from a browser51domainlocalhost5253# Redirect to correct domain if host header does not match domain54# Prevents DNS rebinding attacks55enforce_domainfalse5657# The full public facing url58root_url%(protocol)s://%(domain)s:%(http_port)s/grafana/5960# Serve Grafana from subpath specified in root_url setting. By default it is set to false for compatibility reasons.61serve_from_sub_pathtrue6263# Log web requests64router_loggingfalse6566# the path relative working pathroot_url2.配置启动文件这里是把grafana注册成系统服务了[rootmaster grafana]# cp -r grafana-v12.0.2/ /usr/local/grafana/[rootmaster grafana]# vim /etc/systemd/system/grafana-server.service[Unit]DescriptionGrafana ServerAfternetwork.target[Service]TypesimpleUsergrafanaGroupusersExecStart/usr/local/grafana/bin/grafana server--config/usr/local/grafana/conf/grafana.ini--homepath/usr/local/grafanaRestarton-failure[Install]WantedBymulti-user.target3.启动grafana[roottest grafana]# useradd grafana[rootmaster grafana]# systemctl start grafana-server[rootmaster grafana]# systemctl status grafana-server● grafana-server.service - Grafana Server Loaded: loaded(/etc/systemd/system/grafana-server.service;disabled;vendor preset: disabled)Active: active(running)since Mon2025-07-0717:37:12 CST;1months20days ago Main PID:291119(grafana)Tasks:23(limit:11715)Memory:173.1M CGroup: /system.slice/grafana-server.service └─291119 /usr/local/grafana/bin/grafana server--config/usr/local/grafana/conf/grafana.ini--homepath/usr/local/grafana Aug2718:29:24 master grafana[291119]:loggercontextuserId1orgId1unameadmint2025-08-27T18:29:24.959787608:00levelinfomsgRequest Aug 27 18:29:39 master grafana[291119]: loggercontext userId1 orgId1 unameadmin t2025-08-27T18:29:39.98741551108:00 levelinfo msgRequesAug2718:29:42 master grafana[291119]:loggercontextuserId1orgId1unameadmint2025-08-27T18:29:42.9620790808:00levelinfomsgRequest Aug 27 18:29:52 master grafana[291119]: loggercontext userId1 orgId1 unameadmin t2025-08-27T18:29:52.90714969908:00 levelinfo msgRequesAug2718:30:10 master grafana[291119]:loggercontextuserId1orgId1unameadmint2025-08-27T18:30:10.9062332608:00levelinfomsgRequest Aug 27 18:30:13 master grafana[291119]: loggercontext userId1 orgId1 unameadmin t2025-08-27T18:30:13.91933848508:00 levelinfo msgRequesAug2718:30:19 master grafana[291119]:loggercontextuserId1orgId1unameadmint2025-08-27T18:30:19.9199253208:00levelinfomsgRequest Aug 27 18:30:25 master grafana[291119]: loggercontext userId1 orgId1 unameadmin t2025-08-27T18:30:25.91742490708:00 levelinfo msgReques4.修改nginx配置前面提到grafana需要用到反向代理这里用的nginx跟Prometheus是同一个nginx[rootmaster conf]# !egrepegrep-v#|^$/opt/web_app/nginx-1.16.1/conf/nginx.conf worker_processes2;events{worker_connections60000;}http{include mime.types;default_type application/octet-stream;sendfile on;keepalive_timeout65;server{listen80;server_name127.0.0.1;##添加下面这段配置location /grafana/{proxy_pass http://localhost:3000/grafana/;proxy_set_header Host$host;proxy_set_header X-Real-IP$remote_addr;proxy_set_header X-Forwarded-For$proxy_add_x_forwarded_for;proxy_set_header X-Forwarded-Proto$scheme;}location /prometheus/{proxy_pass http://localhost:9090/prometheus/;}location /jenkins{proxy_pass http://localhost:8080;proxy_set_header Host$host;proxy_set_header X-Real-IP$remote_addr;proxy_set_header X-Forwarded-For$proxy_add_x_forwarded_for;proxy_set_header X-Forwarded-Proto$scheme;}location /status{stub_status;allow127.0.0.1;deny all;}error_page500502503504/50x.html;location/50x.html{root html;}}}重启nginx[rootmaster conf]# /opt/web_app/nginx-1.16.1/sbin/nginx -tnginx: the configurationfile/opt/web_app/nginx-1.16.1/conf/nginx.conf syntax is ok nginx: configurationfile/opt/web_app/nginx-1.16.1/conf/nginx.conftestis successful[rootmaster conf]# /opt/web_app/nginx-1.16.1/sbin/nginx -s reload七、配置Grafana Dashboard1.登录grafana浏览器输入http://IP/grafana账号admin密码admin账密可以修改2.添加数据源3.添加Dashboardgrafana官网提供了一些Dashboard Template可供参考Dashboard地址https://grafana.com/grafana/dashboards/?plcmtoss-nav本次测试我们选的模板ID分别是node dashboard ID1860redis dashboard ID18345nginx dashboard ID14900导入dashboard如果网络不允许也可以在官网下载到本地通过本地导入4.查看整体效果redis和nginx添加dashboard的步骤都一样输入对应的ID即可下面分别展示各监控项的大盘node监控大盘redis监控大盘nginx监控大盘