代码之家  ›  专栏  ›  技术社区  ›  Zhao Li

Kibana没有连接到ElasticSearch

kibana kubernetes elasticsearch
  •  4
  • Zhao Li  · 技术社区  · 7 年前

    我正在尝试使用Kubernetes 1.7.12 fluentd elasticsearch插件: https://github.com/kubernetes/kubernetes/tree/v1.7.12/cluster/addons/fluentd-elasticsearch

    ElasticSearch启动并可以响应: 

    {
 "name" : "0322714ad5b7",
 "cluster_name" : "kubernetes-logging",
 "cluster_uuid" : "_na_",
 "version" : {
   "number" : "2.4.1",
   "build_hash" : "c67dc32e24162035d18d6fe1e952c4cbcbe79d16",
   "build_timestamp" : "2016-09-27T18:57:55Z",
   "build_snapshot" : false,
   "lucene_version" : "5.5.2"
 },
 "tagline" : "You Know, for Search"
}

    但Kibana仍然无法连接到它。连接错误始于: 

    {"type":"log","@timestamp":"2018-01-23T07:42:06Z","tags":["warning","elasticsearch"],"pid":6,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}
{"type":"log","@timestamp":"2018-01-23T07:42:06Z","tags":["warning","elasticsearch"],"pid":6,"message":"No living connections"}

    ElasticSearch启动后,错误变为: 

    {"type":"log","@timestamp":"2018-01-23T07:42:08Z","tags":["status","plugin:elasticsearch@1.0.0","error"],"pid":6,"state":"red","message":"Status changed from red to red - Service Unavailable","prevState":"red","prevMsg":"Unable to connect to Elasticsearch at http://elasticsearch-logging:9200."}

    因此,似乎Kibana最终能够从ElasticSearch获得响应,但仍然无法建立连接。

    这就是Kibana仪表板的外观: enter image description here

    我试图让日志输出更多信息,但对Kibana和ElasticSearch的了解还不够,无法知道下一步还能尝试什么。

    我可以使用此 docker-compose.yml : 

    version: '2'
services:
 elasticsearch-logging:
   image: gcr.io/google_containers/elasticsearch:v2.4.1-2
   ports:
     - "9200:9200"
     - "9300:9300"

 kibana-logging:
   image: gcr.io/google_containers/kibana:v4.6.1-1
   ports:
     - "5601:5601"
   depends_on:
     - elasticsearch-logging
   environment:
     - ELASTICSEARCH_URL=http://elasticsearch-logging:9200

    根据我从这个问题中可以看出的情况,似乎不应该有太多的参与: Kibana on Docker cannot connect to Elasticsearch 还有这个博客: https://gunith.github.io/docker-kibana-elasticsearch/

    但我不知道我错过了什么。

    你知道我还能尝试什么吗?

    感谢您抽出宝贵时间。:)

    更新1:

    curl 惯性导航与制导 http://elasticsearch-logging 在Kubernetes集群上产生了相同的输出: 

    {
  "name" : "elasticsearch-logging-v1-68km4",
  "cluster_name" : "kubernetes-logging",
  "cluster_uuid" : "_na_",
  "version" : {
    "number" : "2.4.1",
    "build_hash" : "c67dc32e24162035d18d6fe1e952c4cbcbe79d16",
    "build_timestamp" : "2016-09-27T18:57:55Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.2"
  },
  "tagline" : "You Know, for Search"
}

    卷曲 惯性导航与制导 http://elasticsearch-logging/_cat/indices?pretty 由于代理规则,Kubernetes群集上的超时。使用 docker编写。yml公司 和 卷曲 本地ing(例如。 curl localhost:9200/_cat/indices?pretty )结果如下: 

    {
  "error" : {
    "root_cause" : [ {
      "type" : "master_not_discovered_exception",
      "reason" : null
    } ],
    "type" : "master_not_discovered_exception",
    "reason" : null
  },
  "status" : 503
}

    这个 docker-compose 日志显示: 

    [2018-01-23 17:04:39,110][DEBUG][action.admin.cluster.state] [ac1f2a13a637] no known master node, scheduling a retry

[2018-01-23 17:05:09,112][DEBUG][action.admin.cluster.state] [ac1f2a13a637] timed out while retrying [cluster:monitor/state] after failure (timeout [30s])
[2018-01-23 17:05:09,116][WARN ][rest.suppressed          ] path: /_cat/indices, params: {pretty=}
MasterNotDiscoveredException[null]
     at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$5.onTimeout(TransportMasterNodeAction.java:234)
     at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:236)
     at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:804)
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
     at java.lang.Thread.run(Thread.java:745)

    更新2: 跑步 kubectl --namespace kube-system logs -c kubedns po/kube-dns-667321983-dt5lz --tail 50 --follow 产量: 

    I0124 16:43:33.591112       5 dns.go:264] New service: kibana-logging
I0124 16:43:33.591225       5 dns.go:264] New service: nginx
I0124 16:43:33.591251       5 dns.go:264] New service: registry
I0124 16:43:33.591274       5 dns.go:264] New service: sudoe
I0124 16:43:33.591295       5 dns.go:264] New service: default-http-backend
I0124 16:43:33.591317       5 dns.go:264] New service: kube-dns
I0124 16:43:33.591344       5 dns.go:462] Added SRV record &{Host:kube-dns.kube-system.svc.cluster.local. Port:53 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
I0124 16:43:33.591369       5 dns.go:462] Added SRV record &{Host:kube-dns.kube-system.svc.cluster.local. Port:53 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
I0124 16:43:33.591390       5 dns.go:264] New service: kubernetes
I0124 16:43:33.591409       5 dns.go:462] Added SRV record &{Host:kubernetes.default.svc.cluster.local. Port:443 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
I0124 16:43:33.591429       5 dns.go:264] New service: elasticsearch-logging

    更新3:

    我仍在努力让一切正常运转,但在其他人的帮助下,我相信这是RBAC的问题。我不完全确定,但由于权限的原因,elasticsearch节点似乎无法连接到主节点(我从来不知道它甚至是必需的)。

    以下是一些有帮助的步骤,以防帮助其他人起步:

    启用RBAC时: 

    # kubectl --kubeconfig kubeconfig.yaml --namespace kube-system logs po/elasticsearch-logging-v1-wkwcs
F0119 00:18:44.285773       9 elasticsearch_logging_discovery.go:60] kube-system namespace doesn't exist: User "system:serviceaccount:kube-system:default" cannot get namespaces in the namespace "kube-system". (get namespaces kube-system)
goroutine 1 [running]:
k8s.io/kubernetes/vendor/github.com/golang/glog.stacks(0x1f7f600, 0xc400000000, 0xee, 0x1b2)
        vendor/github.com/golang/glog/glog.go:766 +0xa5
k8s.io/kubernetes/vendor/github.com/golang/glog.(*loggingT).output(0x1f5f5c0, 0xc400000003, 0xc42006c300, 0x1ef20c8, 0x22, 0x3c, 0x0)
        vendor/github.com/golang/glog/glog.go:717 +0x337
k8s.io/kubernetes/vendor/github.com/golang/glog.(*loggingT).printf(0x1f5f5c0, 0xc400000003, 0x16949d6, 0x1e, 0xc420579ee8, 0x2, 0x2)
        vendor/github.com/golang/glog/glog.go:655 +0x14c
k8s.io/kubernetes/vendor/github.com/golang/glog.Fatalf(0x16949d6, 0x1e, 0xc420579ee8, 0x2, 0x2)
        vendor/github.com/golang/glog/glog.go:1145 +0x67
main.main()
        cluster/addons/fluentd-elasticsearch/es-image/elasticsearch_logging_discovery.go:60 +0xb53
[2018-01-19 00:18:45,273][INFO ][node                     ] [elasticsearch-logging-v1-wkwcs] version[2.4.1], pid[5], build[c67dc32/2016-09-27T18:57:55Z]
[2018-01-19 00:18:45,275][INFO ][node                     ] [elasticsearch-logging-v1-wkwcs] initializing ...

    # kubectl --kubeconfig kubeconfig.yaml --namespace kube-system exec kibana-logging-2104905774-69wgv curl elasticsearch-logging.kube-system:9200/_cat/indices?pretty

{
  "error" : {
    "root_cause" : [ {
      "type" : "master_not_discovered_exception",
      "reason" : null
    } ],
    "type" : "master_not_discovered_exception",
    "reason" : null
  },
  "status" : 503
}

    RBAC关闭时: 

    #  kubectl --kubeconfig kubeconfig.yaml --namespace kube-system log elasticsearch-logging-v1-7shgk
[2018-01-26 01:19:52,294][INFO ][node                     ] [elasticsearch-logging-v1-7shgk] version[2.4.1], pid[5], build[c67dc32/2016-09-27T18:57:55Z]
[2018-01-26 01:19:52,294][INFO ][node                     ] [elasticsearch-logging-v1-7shgk] initializing ...
[2018-01-26 01:19:53,077][INFO ][plugins                  ] [elasticsearch-logging-v1-7shgk] modules [reindex, lang-expression, lang-groovy], plugins [], sites []

    #  kubectl --kubeconfig kubeconfig.yaml --namespace kube-system exec elasticsearch-logging-v1-7shgk curl http://elasticsearch-logging:9200/_cat/indices?pretty
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    40  100    40    0     0      2      0  0:00:20  0:00:15  0:00:05    10
green open .kibana 1 1 1 0 6.2kb 3.1kb

    感谢大家的帮助:)

    2 回复  |  直到 7 年前
        1
  •  4
  •   whites11    7 年前

    一些疑难解答提示:

    1) 确保ElasticSearch正常运行。

    输入运行elasticsearch的容器并运行:

    curl localhost:9200

    您应该得到一个JSON,其中包含一些关于elasticsearch的数据。

    2) 确保可以从kibana容器访问ElasticSearch

    进入kibana容器并运行:

    curl <elasticsearch_service_name>:9200

    您应该得到与上面相同的输出。

    3) 确保ES指数良好。

    从elasticsearch容器运行以下命令:

    curl localhost:9200/_cat/indices?pretty

    您应该得到一个表,其中包含ES集群中的所有索引及其状态(如果只有一个ES副本,则应为绿色或黄色)。

    如果上述任一点失败,请检查ES容器的日志中是否有任何错误消息,并尝试解决这些问题。

        2
  •  1
  •   Pamir Erdem    7 年前

    此异常表示2个配置错误 1、Kubernetes的DNS插件工作不正常。检查dns加载项日志 2、吊舱2吊舱通信工作不正常。这与您的基础sdn addon cni法兰绒印花布有关。

    您可以通过ping一个pod与另一个pod进行检查。如果它不工作,请检查您的网络配置,尤其是kube代理组件。

    推荐文章
    khateeb  ·  如何进行Elasticsearch ESQL Distinct查询?
    1 年前
    dabljues  ·  带过滤器和出现次数的ElasticSearch查询
    1 年前
    user16971617  ·  [余弦]相似性不支持弹性搜索中幅度为零的向量
    1 年前
    omnes_flumina  ·  Elasticsearch[size]查询格式错误,查询名称后没有start_object
    1 年前
    Peter Penzov  ·  为客户端安装兼容的弹性搜索版本
    1 年前
    d3vr10  ·  PGSync错误,连接到elasticsearch时发生身份验证异常(401):“无法为REST请求[/]对用户[elastic]进行身份验证”
    1 年前
    Bardo  ·  使用logstash在ELK上获取城市名称的地理位置
    1 年前
    Rodrick Zadrozny  ·  在补全提示中使用正则表达式时不起作用
    1 年前
    I lebzdel I  ·  如何为单个字段添加char_filter
    1 年前
    I lebzdel I  ·  如果文本字段包含日期,如何添加模糊性
    1 年前
    关于   移动版
    代码之家 - 一站式码农服务社区
    沪ICP备11025650号