Nacos客户端注册报错

共饮一杯无

发布于 2022-11-24 19:44:39

2.8K0

问题描述

Nacos客户端注册报错：failed to req API__nacos_v1_ns_instance_list after all servers([xxx_8848]) tried_ ErrCode_503, ErrMsg_server is DOWN now, please try again later! 我靠，突然发现服务全都注册不到nacos，但是访问nacos的UI地址又发现nacos是启动着的，并且相关config配置也可以正常显示

日志查看

nacos客户端报错如下：大概意思就是nacos挂了你等会再试吧。。

2021-11-10 14:30:30.154 ERROR [com.alibaba.nacos.client.naming.updater] [] com.alibaba.nacos.client.naming          613 : [NA] failed to request

com.alibaba.nacos.api.exception.NacosException: server is DOWN now, please try again later!
        at com.alibaba.nacos.client.naming.net.NamingProxy.callServer(NamingProxy.java:611)
        at com.alibaba.nacos.client.naming.net.NamingProxy.reqApi(NamingProxy.java:538)
        at com.alibaba.nacos.client.naming.net.NamingProxy.reqApi(NamingProxy.java:491)
        at com.alibaba.nacos.client.naming.net.NamingProxy.reqApi(NamingProxy.java:486)
        at com.alibaba.nacos.client.naming.net.NamingProxy.queryList(NamingProxy.java:400)
        at com.alibaba.nacos.client.naming.core.HostReactor.updateService(HostReactor.java:339)
        at com.alibaba.nacos.client.naming.core.HostReactor$UpdateTask.run(HostReactor.java:420)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:819)

2021-11-10 14:30:30.154 ERROR [com.alibaba.nacos.client.naming.updater] [] com.alibaba.nacos.client.naming          548 : request: /nacos/v1/ns/instance/list failed, servers: [xxx.xxx.xxx.xxx:8848], code: 500,msg: server is DOWN now, please try again later!
2021-11-10 14:30:30.155  WARN [com.alibaba.nacos.client.naming.updater] [] com.alibaba.nacos.client.naming          444 : [NA] failed to update serviceName: test@@xxx-gateway

com.alibaba.nacos.api.exception.NacosException: failed to req API:/nacos/v1/ns/instance/list after all servers([xxx:8848]) tried: ErrCode:503, ErrMsg:server is DOWN now, please try again later!
        at com.alibaba.nacos.client.naming.net.NamingProxy.reqApi(NamingProxy.java:552)
        at com.alibaba.nacos.client.naming.net.NamingProxy.reqApi(NamingProxy.java:491)
        at com.alibaba.nacos.client.naming.net.NamingProxy.reqApi(NamingProxy.java:486)
        at com.alibaba.nacos.client.naming.net.NamingProxy.queryList(NamingProxy.java:400)
        at com.alibaba.nacos.client.naming.core.HostReactor.updateService(HostReactor.java:339)
        at com.alibaba.nacos.client.naming.core.HostReactor$UpdateTask.run(HostReactor.java:420)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:819)

这就有点费解了，我的nacos明明也是能够访问的，nacos.log日志也没发现一样，咋就整不上呢 nacos启动日志：

2021-11-10 14:12:24,931 INFO Initializing ExecutorService 'taskScheduler'

2021-11-10 14:12:24,961 INFO Exposing 2 endpoint(s) beneath base path '/actuator'

2021-11-10 14:12:25,089 INFO Tomcat started on port(s): 8848 (http) with context path '/nacos'

2021-11-10 14:12:25,093 INFO Started Nacos in 11.691 seconds (JVM running for 12.911)

2021-11-10 14:12:25,093 INFO Nacos Log files: /home/nacos/logs

2021-11-10 14:12:25,094 INFO Nacos Log files: /home/nacos/conf

2021-11-10 14:12:25,094 INFO Nacos Log files: /home/nacos/data

2021-11-10 14:12:25,094 INFO Nacos started successfully in stand alone mode. use external storage

解决过程和办法

发现这种诡异的问题，还以为nacos是好好的，测试了一波网络，网络也是通的，然后重启了一波nacos，发现nacos.log启动日志也没啥不对，正常启动的，客户端注册还是一样的报错。奇怪。。后面查看了protocol-raft.log

java.lang.IllegalArgumentException: Blank leader
        at com.alipay.sofa.jraft.util.Requires.requireTrue(Requires.java:85)
        at com.alipay.sofa.jraft.RouteTable.updateLeader(RouteTable.java:181)
        at com.alipay.sofa.jraft.RouteTable.refreshLeader(RouteTable.java:264)
        at com.alibaba.nacos.core.distributed.raft.JRaftServer.refreshRouteTable(JRaftServer.java:501)
        at com.alibaba.nacos.core.distributed.raft.JRaftServer.lambda$createMultiRaftGroup$1(JRaftServer.java:276)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2021-11-10 14:40:35,045 ERROR Fail to refresh raft metadata info for group : naming_persistent_service, error is : {}

目测应该是选举失败了，错误原因nacos1.4.0使用了jraft， jraft会记录前一次启动的集群地址，如果重启机器ip变了的话，会导致jraft记录的地址失效，从而导致选主出问题。然后想到了昨天机房有做过重启，具体干了啥未知，所以出现了这个奇怪的问题。 nacos/data/目录下的protocol文件夹放置一些缓存的ip历史记录,删除{nacos.home}/nacos/data/protocol文件夹后重新启动nacos，再重启nacos客户端，注册正常了。

PS:发现nacos的项目下面也有人提这个问题 https://github.com/alibaba/nacos/issues/4210

本文参与腾讯云自媒体同步曝光计划，分享自作者个人站点/博客。

原始发表：2021-11-10，如有侵权请联系 cloudcommunity@tencent.com 删除

tcp/ip