Redis 高可用集群

哨兵模式的虽然解决了读写分离,以及Master宕机之后重新选举出新的Master来恢复Redis服务,但是选举恢复的过程耗时还是比较旧的,在生产上是难以接受.那有没有更好的解决办法呢?
有那就是采用Cluster(集群),哨兵模式的不足是只有一个master,当master节点出现问题,整个Redis服务就陷入瘫痪状态.Cluster 集群就很好的解决了这个问题,一个Redis Cluster 部署多台Redis 主从服务.

Redis Cluster 节点通信机制

维护集群的元数据(集群节点信息,主从角色,节点数量,各节点共享的数据等)有两种方式:集中
式和gossip.

集中式

优点在于元数据的更新和读取,时效性非常好,一旦元数据出现变更立即就会更新到集中式的存储中,其他节点读取的时候立即就可以立即感知到;不足在于所有的元数据的更新压力全部集中在一个地方,可能导致元数据的存储压力.很多中间件都会借助zookeeper集中式存储元数据.

gossip 协议方式

gossip协议包含多种消息,包括ping,pong,meet,fail等等。

meet:某个节点发送meet给新加入的节点,让新节点加入集群中,然后新节点就会开始与其他节点进行通信;
ping:每个节点都会频繁给其他节点发送ping,其中包含自己的状态还有自己维护的集群元数据，互相通过 ping交换元数据(类似自己感知到的集群节点增加和移除,hash slot信息等);
pong: 对ping和meet消息的返回,包含自己的状态和其他信息,也可以用于信息广播和更新;
fail: 某个节点判断另一个节点fail之后，就发送fail给其他节点,通知其他节点,指定的节点宕机了.

gossip协议的优点在于元数据的更新比较分散,不是集中在一个地方,更新请求会陆陆续续,打到所有节点上去更新,有一定的延时,降低了压力;缺点在于元数据更新有延时可能导致集群的一些操作会有一些滞后.

gossip通信的10000端口

每个节点都有一个专门用于节点间gossip通信的端口,就是自己提供服务的端口号+10000,比如7001,那么用于节点间通信的就是17001端口.每个节点每隔一段时间都会往另外几个节点发送ping消息,同时其他几点接收到ping消息之后返回pong消息.

Redis Cluster 原理

Redis Cluster 将所有的数据划分为 16384 (2^15 -1)个Slots, 每个Master负责一部分Slots.

Redis客户端拥有Redis集群的很多信息,比如 info命令中看到的,除了这些,Redis 客户端还会将同步Slots信息缓存起来.

Slots计算

在执行存值操作的时候,默认会将key经过CRC16算法计算一个int值,再对16384 进行取模预算得到Slots信息. 这样就知道这个key 需要存储在哪台Master 节点上了.

及时出现: 客户端将key 发送到错误的Master节点上, 对应的Master节点发现,对应的Slot 不属于自己,会给客户端发送一个重定向,并提供key的Slot对应的正确Master 节点信息—— 跳转重定向.

Redis Cluster 搭建

搭建 Redis Cluster, 每一个节点(不管是主,还是从)一开始都相当于Master,在扩容的时候也是一样,需要水平扩容的节点也是Master.

搭建Redis Cluster

这里我们创建一个包含 3个一主一从的Redis 集群,3个主从服务分别位于不同的3个虚拟机上来实现分布式的效果.

配置文件修改

这里我们准备在3台机器上分别建好一个/clusterconfig的文件夹,假设服务器是 A,B,C

分别:

A 服务器: 创建两个端口 8001 8004

B 服务器: 创建两个端口 8002 8005

C 服务器: 创建两个端口 8003 8006

每台机器上启动两个Redis服务. 分别是bindip:port.

接下来我们已其中一台为例,来设置配置，其他的只需要更加虚拟机ip,以及端口来更换就好了。

bind 172.30.60.2

关闭保护模式
protected-mode no

port 8001

开启守护模式
daemonize yes

这个也可以改成和端口一致
pidfile /var/run/redis_6379.pid

文件存储路径
dir "/usr/local/redis-5.0.2/clusterconfigs/8001"

密码设置
masterauth zwh
requirepass zwh

开启aof
appendonly yes

集群配置：
开启集群模式
cluster-enabled yes
cluster-config-file nodes-8001.conf
cluster-node-timeout 5000

启动所有Redis 节点,并验证下进程

1
2
3

redis-server redis.conf

ps -ef | grep redis

1
2
3

[root@oldconan redis-5.0.2]# ps -ef |grep redis 
root       8902      1  0 15:34 ?        00:00:00 src/redis-server 172.30.60.2:8001 [cluster]
root       8907   8569  0 15:34 pts/0    00:00:00 grep --color=auto redis

可以看到，redis.conf中开启了集群模式，进程中也会显示cluster;

集群模式启动

前面我们将6个Redis 节点都已经启动好了,现在他们是独立的Master节点,下面将启动集群模式.

1 2	进入客户端 dis-cli -a zwh --cluster create --cluster-replicas 1 172.30.60.2:8001 172.30.60.2:8004 172.30.60.3:8002 172.30.60.3:8005 172.30.60.4:8003 172.30.60.4:8006

这里有必要解释下这些参数的意思

dis-cli -a password --cluster create --cluster-replicas count bindip:port ... bindip:port 

password : 密码
count    : slave的数量

Redis 自己设设定那个ip,port 是Master ,Slave ;并给每个Master 分配 Slot, Slot的范围是 0 – 16383 ( 2^15 - 1);

>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 172.30.60.3:8005 to 172.30.60.2:8001
Adding replica 172.30.60.2:8004 to 172.30.60.3:8002
Adding replica 172.30.60.4:8006 to 172.30.60.4:8003
>>> Trying to optimize slaves allocation for anti-affinity
[OK] Perfect anti-affinity obtained!
M: 4ec8e913999792fc2f3d056fcab6c8a73fc1858e 172.30.60.2:8001
   slots:[0-5460] (5461 slots) master
S: 9a8e6254af86f278d33108060ef1fac12e287bec 172.30.60.2:8004
   replicates 23bfc097bf05021ef74e942bcd388d75a1ef2694
M: ee77ad4c9b27e9b212ddf5f929c3cb895752a06a 172.30.60.3:8002
   slots:[5461-10922] (5462 slots) master
S: dfca9aff3abccba24483dce1efee76c8c8f2158f 172.30.60.3:8005
   replicates 4ec8e913999792fc2f3d056fcab6c8a73fc1858e
M: 23bfc097bf05021ef74e942bcd388d75a1ef2694 172.30.60.4:8003
   slots:[10923-16383] (5461 slots) master
S: 5f63eeb3e408b47d40862c16e39480832853daaf 172.30.60.4:8006
   replicates ee77ad4c9b27e9b212ddf5f929c3cb895752a06a
Can I set the above configuration? (type 'yes' to accept):

系统询问你是否确认 slot 分配; 我们输入 yes 确认;

Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
...
>>> Performing Cluster Check (using node 172.30.60.2:8001)
M: 4ec8e913999792fc2f3d056fcab6c8a73fc1858e 172.30.60.2:8001
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: dfca9aff3abccba24483dce1efee76c8c8f2158f 172.30.60.3:8005
   slots: (0 slots) slave
   replicates 4ec8e913999792fc2f3d056fcab6c8a73fc1858e
M: 23bfc097bf05021ef74e942bcd388d75a1ef2694 172.30.60.4:8003
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 5f63eeb3e408b47d40862c16e39480832853daaf 172.30.60.4:8006
   slots: (0 slots) slave
   replicates ee77ad4c9b27e9b212ddf5f929c3cb895752a06a
M: ee77ad4c9b27e9b212ddf5f929c3cb895752a06a 172.30.60.3:8002
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 9a8e6254af86f278d33108060ef1fac12e287bec 172.30.60.2:8004
   slots: (0 slots) slave
   replicates 23bfc097bf05021ef74e942bcd388d75a1ef2694
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@oldconan redis-5.0.2]#

到这里Redis 集群就已经创建完毕了, 我们可以进入任何一个节点,查看节点信息;

>redis-cli -a zwh -h 172.30.60.2 -p 8001

172.30.60.2:8001>CLUSTER NODES
b3e4648c23770c5f5271092ad14148eac4e7dd67 172.30.60.2:9000@19000 master - 0 1605061190568 0 connected
23bfc097bf05021ef74e942bcd388d75a1ef2694 172.30.60.4:8003@18003 master - 0 1605061191000 5 connected 10923-16383
5f63eeb3e408b47d40862c16e39480832853daaf 172.30.60.4:8006@18006 slave ee77ad4c9b27e9b212ddf5f929c3cb895752a06a 0 1605061190568 6 connected
9a8e6254af86f278d33108060ef1fac12e287bec 172.30.60.2:8004@18004 slave 23bfc097bf05021ef74e942bcd388d75a1ef2694 0 1605061190000 5 connected
ee77ad4c9b27e9b212ddf5f929c3cb895752a06a 172.30.60.3:8002@18002 myself,master - 0 1605061191000 3 connected 5461-10922
4ec8e913999792fc2f3d056fcab6c8a73fc1858e 172.30.60.2:8001@18001 master - 0 1605061190000 1 connected 0-5460
dfca9aff3abccba24483dce1efee76c8c8f2158f 172.30.60.3:8005@18005 slave 4ec8e913999792fc2f3d056fcab6c8a73fc1858e 0 1605061191776 4 connected

从CLUSTER NODES 打印出来的信息,我们可以看到有3个Master,3个Slave;

有一点值得注意的是:

Redis 没有将我们创建在同一台机器上的两个端口设置为对应的主从节点.而是错开到不同机器. 这是为了避免一台机器出现问题,导致集群一部分Slot全部访问不了.

Redis Cluster 水平扩容

这里我们只准备扩容一对Master-slave,Redis Cluster 水平扩容和前面一样,扩容的机器一开始也是Master ,实际扩容也最好将两个节点设置在不同机器上.

这里我们在172.30.60.2的这台机器上创建了2个conf,准备将9000端口的配置作为Master ,9001端口的配置作为Slave;

1:启动 1个需要扩容的Master Redis节点.

1	redis-server 9000.conf

2: 将Master Redis节点添加到Cluster中, 命令中是第一个ip是需要添加的节点ip和端口, 尾部ip和端口是集群中任意一个节点的ip和端口.

1	redis-cli -a zwh --cluster add-node 172.30.60.2:9000 172.30.60.2:8001

[root@oldconan redis-5.0.2]# src/redis-cli -a zwh --cluster add-node 172.30.60.2:9000 172.30.60.2:8001
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Adding node 172.30.60.2:9000 to cluster 172.30.60.2:8001
>>> Performing Cluster Check (using node 172.30.60.2:8001)
M: 4ec8e913999792fc2f3d056fcab6c8a73fc1858e 172.30.60.2:8001
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: dfca9aff3abccba24483dce1efee76c8c8f2158f 172.30.60.3:8005
   slots: (0 slots) slave
   replicates 4ec8e913999792fc2f3d056fcab6c8a73fc1858e
M: 23bfc097bf05021ef74e942bcd388d75a1ef2694 172.30.60.4:8003
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 5f63eeb3e408b47d40862c16e39480832853daaf 172.30.60.4:8006
   slots: (0 slots) slave
   replicates ee77ad4c9b27e9b212ddf5f929c3cb895752a06a
M: ee77ad4c9b27e9b212ddf5f929c3cb895752a06a 172.30.60.3:8002
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 9a8e6254af86f278d33108060ef1fac12e287bec 172.30.60.2:8004
   slots: (0 slots) slave
   replicates 23bfc097bf05021ef74e942bcd388d75a1ef2694
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 172.30.60.2:9000 to make it join the cluster.
[OK] New node added correctly.

3: 查看节点信息

172.30.60.3:8002> CLUSTER NODES
b3e4648c23770c5f5271092ad14148eac4e7dd67 172.30.60.2:9000@19000 master - 0 1605061190568 0 connected
23bfc097bf05021ef74e942bcd388d75a1ef2694 172.30.60.4:8003@18003 master - 0 1605061191000 5 connected 10923-16383
5f63eeb3e408b47d40862c16e39480832853daaf 172.30.60.4:8006@18006 slave ee77ad4c9b27e9b212ddf5f929c3cb895752a06a 0 1605061190568 6 connected
9a8e6254af86f278d33108060ef1fac12e287bec 172.30.60.2:8004@18004 slave 23bfc097bf05021ef74e942bcd388d75a1ef2694 0 1605061190000 5 connected
ee77ad4c9b27e9b212ddf5f929c3cb895752a06a 172.30.60.3:8002@18002 myself,master - 0 1605061191000 3 connected 5461-10922
4ec8e913999792fc2f3d056fcab6c8a73fc1858e 172.30.60.2:8001@18001 master - 0 1605061190000 1 connected 0-5460
dfca9aff3abccba24483dce1efee76c8c8f2158f 172.30.60.3:8005@18005 slave 4ec8e913999792fc2f3d056fcab6c8a73fc1858e 0 1605061191776 4 connected

4: 给新加入的Master 节点分配Slot

1	src/redis-cli -a zwh --cluster reshard 172.30.60.2:8001

[root@oldconan redis-5.0.2]# src/redis-cli -a zwh --cluster reshard  172.30.60.2:8001


Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing Cluster Check (using node 172.30.60.2:8001)
M: 4ec8e913999792fc2f3d056fcab6c8a73fc1858e 172.30.60.2:8001
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: dfca9aff3abccba24483dce1efee76c8c8f2158f 172.30.60.3:8005
   slots: (0 slots) slave
   replicates 4ec8e913999792fc2f3d056fcab6c8a73fc1858e
M: b3e4648c23770c5f5271092ad14148eac4e7dd67 172.30.60.2:9000
   slots: (0 slots) master
M: 23bfc097bf05021ef74e942bcd388d75a1ef2694 172.30.60.4:8003
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 5f63eeb3e408b47d40862c16e39480832853daaf 172.30.60.4:8006
   slots: (0 slots) slave
   replicates ee77ad4c9b27e9b212ddf5f929c3cb895752a06a
M: ee77ad4c9b27e9b212ddf5f929c3cb895752a06a 172.30.60.3:8002
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 9a8e6254af86f278d33108060ef1fac12e287bec 172.30.60.2:8004
   slots: (0 slots) slave
   replicates 23bfc097bf05021ef74e942bcd388d75a1ef2694
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 300

询问你需要给新添加的Master 分配多少Slot; 这里我们分配300

What is the receiving node ID? b3e4648c23770c5f5271092ad14148eac4e7dd67
Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node #1: all

询问你是从所有主节点瓜分 Slot 还是指定某一个id, 这里我们从所有Master节点各分配一点.

最终会输出很多分配记录. 我们可以进入某个节点,查看节点信息

5:查看节点信息

172.30.60.3:8002> CLUSTER NODES
b3e4648c23770c5f5271092ad14148eac4e7dd67 172.30.60.2:9000@19000 master - 0 1605061330000 7 connected 0-98 5461-5561 10923-11021
23bfc097bf05021ef74e942bcd388d75a1ef2694 172.30.60.4:8003@18003 master - 0 1605061331068 5 connected 11022-16383
5f63eeb3e408b47d40862c16e39480832853daaf 172.30.60.4:8006@18006 slave ee77ad4c9b27e9b212ddf5f929c3cb895752a06a 0 1605061331575 6 connected
9a8e6254af86f278d33108060ef1fac12e287bec 172.30.60.2:8004@18004 slave 23bfc097bf05021ef74e942bcd388d75a1ef2694 0 1605061330000 5 connected
ee77ad4c9b27e9b212ddf5f929c3cb895752a06a 172.30.60.3:8002@18002 myself,master - 0 1605061330000 3 connected 5562-10922
4ec8e913999792fc2f3d056fcab6c8a73fc1858e 172.30.60.2:8001@18001 master - 0 1605061330057 1 connected 99-5460
dfca9aff3abccba24483dce1efee76c8c8f2158f 172.30.60.3:8005@18005 slave 4ec8e913999792fc2f3d056fcab6c8a73fc1858e 0 1605061330000 4 connected

6: 给新加入的Master 添加 Slave

启动将作为 Slave 的Redis 9001服务;

1	redis-server 9001.conf

将9001 节点添加到Redis 集群

[root@oldconan redis-5.0.2]# src/redis-cli -a zwh --cluster add-node 172.30.60.2:9001 172.30.60.2:8001
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Adding node 172.30.60.2:9001 to cluster 172.30.60.2:8001
>>> Performing Cluster Check (using node 172.30.60.2:8001)
M: 4ec8e913999792fc2f3d056fcab6c8a73fc1858e 172.30.60.2:8001
   slots:[99-5460] (5362 slots) master
   1 additional replica(s)
S: dfca9aff3abccba24483dce1efee76c8c8f2158f 172.30.60.3:8005
   slots: (0 slots) slave
   replicates 4ec8e913999792fc2f3d056fcab6c8a73fc1858e
M: b3e4648c23770c5f5271092ad14148eac4e7dd67 172.30.60.2:9000
   slots:[0-98],[5461-5561],[10923-11021] (299 slots) master
M: 23bfc097bf05021ef74e942bcd388d75a1ef2694 172.30.60.4:8003
   slots:[11022-16383] (5362 slots) master
   1 additional replica(s)
S: 5f63eeb3e408b47d40862c16e39480832853daaf 172.30.60.4:8006
   slots: (0 slots) slave
   replicates ee77ad4c9b27e9b212ddf5f929c3cb895752a06a
M: ee77ad4c9b27e9b212ddf5f929c3cb895752a06a 172.30.60.3:8002
   slots:[5562-10922] (5361 slots) master
   1 additional replica(s)
S: 9a8e6254af86f278d33108060ef1fac12e287bec 172.30.60.2:8004
   slots: (0 slots) slave
   replicates 23bfc097bf05021ef74e942bcd388d75a1ef2694
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 172.30.60.2:9001 to make it join the cluster.
[OK] New node added correctly.

进入9000 Master 节点Redis 服务的客户端, 将9001 挂属给9000 Master.

172.30.60.2:9000> CLUSTER NODES
9a8e6254af86f278d33108060ef1fac12e287bec 172.30.60.2:8004@18004 slave 23bfc097bf05021ef74e942bcd388d75a1ef2694 0 1605062137301 5 connected
b3e4648c23770c5f5271092ad14148eac4e7dd67 172.30.60.2:9000@19000 myself,master - 0 1605062137000 7 connected 0-98 5461-5561 10923-11021
4ec8e913999792fc2f3d056fcab6c8a73fc1858e 172.30.60.2:8001@18001 master - 0 1605062137000 1 connected 99-5460
23bfc097bf05021ef74e942bcd388d75a1ef2694 172.30.60.4:8003@18003 master - 0 1605062137100 5 connected 11022-16383
5f63eeb3e408b47d40862c16e39480832853daaf 172.30.60.4:8006@18006 slave ee77ad4c9b27e9b212ddf5f929c3cb895752a06a 0 1605062137806 3 connected
dfca9aff3abccba24483dce1efee76c8c8f2158f 172.30.60.3:8005@18005 slave 4ec8e913999792fc2f3d056fcab6c8a73fc1858e 0 1605062138309 1 connected
ee77ad4c9b27e9b212ddf5f929c3cb895752a06a 172.30.60.3:8002@18002 master - 0 1605062137503 3 connected 5562-10922
6d4e86543b6ba652fe2f26bc0ca4c3de7c10cec4 172.30.60.2:9001@19001 master - 0 1605062136292 0 connected
172.30.60.2:9000> CLUSTER REPLICATE b3e4648c23770c5f5271092ad14148eac4e7dd67
(error) ERR Can't replicate myself
172.30.60.2:9000> quit
[root@xinyi redis-5.0.2]# src/redis-cli -a zwh -h 172.30.60.2 -p 9001 
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
172.30.60.2:9001> CLUSTER REPLICATE b3e4648c23770c5f5271092ad14148eac4e7dd67
OK

最后检查;

172.30.60.2:9001> CLUSTER NODES
5f63eeb3e408b47d40862c16e39480832853daaf 172.30.60.4:8006@18006 slave ee77ad4c9b27e9b212ddf5f929c3cb895752a06a 0 1605062360209 3 connected
dfca9aff3abccba24483dce1efee76c8c8f2158f 172.30.60.3:8005@18005 slave 4ec8e913999792fc2f3d056fcab6c8a73fc1858e 0 1605062359000 1 connected
4ec8e913999792fc2f3d056fcab6c8a73fc1858e 172.30.60.2:8001@18001 master - 0 1605062359502 1 connected 99-5460
23bfc097bf05021ef74e942bcd388d75a1ef2694 172.30.60.4:8003@18003 master - 0 1605062359000 5 connected 11022-16383
ee77ad4c9b27e9b212ddf5f929c3cb895752a06a 172.30.60.3:8002@18002 master - 0 1605062359705 3 connected 5562-10922
6d4e86543b6ba652fe2f26bc0ca4c3de7c10cec4 172.30.60.2:9001@19001 myself,slave b3e4648c23770c5f5271092ad14148eac4e7dd67 0 1605062358000 0 connected
b3e4648c23770c5f5271092ad14148eac4e7dd67 172.30.60.2:9000@19000 master - 0 1605062360208 7 connected 0-98 5461-5561 10923-11021
9a8e6254af86f278d33108060ef1fac12e287bec 172.30.60.2:8004@18004 slave 23bfc097bf05021ef74e942bcd388d75a1ef2694 0 1605062358594 5 connected

Redis Cluster 移除扩容

从水平扩容的流程可以推测出来,如果移除扩容,我们需要先将Slave节点移除, 再要移除Master 服务中存储的数据迁移,将Slot归还. 最后在移除Master节点.

1: 移除Slave节点

移除slave节点
[root@xinyi redis-5.0.2]# src/redis-cli -a zwh --cluster del-node 172.30.60.2:9001 6d4e86543b6ba652fe2f26bc0ca4c3de7c10cec4

检查
[root@xinyi redis-5.0.2]# src/redis-cli -a zwh -h 172.30.60.2 -p 9000
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
172.30.60.2:9000> CLUSTER NODES
9a8e6254af86f278d33108060ef1fac12e287bec 172.30.60.2:8004@18004 slave 23bfc097bf05021ef74e942bcd388d75a1ef2694 0 1605082521974 5 connected
b3e4648c23770c5f5271092ad14148eac4e7dd67 172.30.60.2:9000@19000 myself,master - 0 1605082521000 7 connected 0-98 5461-5561 10923-11021
4ec8e913999792fc2f3d056fcab6c8a73fc1858e 172.30.60.2:8001@18001 master - 0 1605082521570 1 connected 99-5460
23bfc097bf05021ef74e942bcd388d75a1ef2694 172.30.60.4:8003@18003 master - 0 1605082520964 5 connected 11022-16383
5f63eeb3e408b47d40862c16e39480832853daaf 172.30.60.4:8006@18006 slave ee77ad4c9b27e9b212ddf5f929c3cb895752a06a 0 1605082521468 3 connected
dfca9aff3abccba24483dce1efee76c8c8f2158f 172.30.60.3:8005@18005 slave 4ec8e913999792fc2f3d056fcab6c8a73fc1858e 0 1605082521000 1 connected
ee77ad4c9b27e9b212ddf5f929c3cb895752a06a 172.30.60.3:8002@18002 master - 0 1605082522477 3 connected 5562-10922
172.30.60.2:9000>

2:迁移数据

1	src/redis-cli -a zwh --cluster reshard 172.30.60.2:9001

[root@xinyi redis-5.0.2]# src/redis-cli -a zwh --cluster reshard 172.30.60.2:9000
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing Cluster Check (using node 172.30.60.2:9000)
M: b3e4648c23770c5f5271092ad14148eac4e7dd67 172.30.60.2:9000
   slots:[0-98],[5461-5561],[10923-11021] (299 slots) master
S: 9a8e6254af86f278d33108060ef1fac12e287bec 172.30.60.2:8004
   slots: (0 slots) slave
   replicates 23bfc097bf05021ef74e942bcd388d75a1ef2694
M: 4ec8e913999792fc2f3d056fcab6c8a73fc1858e 172.30.60.2:8001
   slots:[99-5460] (5362 slots) master
   1 additional replica(s)
M: 23bfc097bf05021ef74e942bcd388d75a1ef2694 172.30.60.4:8003
   slots:[11022-16383] (5362 slots) master
   1 additional replica(s)
S: 5f63eeb3e408b47d40862c16e39480832853daaf 172.30.60.4:8006
   slots: (0 slots) slave
   replicates ee77ad4c9b27e9b212ddf5f929c3cb895752a06a
S: dfca9aff3abccba24483dce1efee76c8c8f2158f 172.30.60.3:8005
   slots: (0 slots) slave
   replicates 4ec8e913999792fc2f3d056fcab6c8a73fc1858e
M: ee77ad4c9b27e9b212ddf5f929c3cb895752a06a 172.30.60.3:8002
   slots:[5562-10922] (5361 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 300
What is the receiving node ID? ee77ad4c9b27e9b212ddf5f929c3cb895752a06a
Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node #1: b3e4648c23770c5f5271092ad14148eac4e7dd67
Source node #2: done

Ready to move 300 slots.
  Source nodes:
    M: b3e4648c23770c5f5271092ad14148eac4e7dd67 172.30.60.2:9000
       slots:[0-98],[5461-5561],[10923-11021] (299 slots) master
  Destination node:
    M: ee77ad4c9b27e9b212ddf5f929c3cb895752a06a 172.30.60.3:8002
       slots:[5562-10922] (5361 slots) master
       1 additional replica(s)
  Resharding plan:
    Moving slot 0 from b3e4648c23770c5f5271092ad14148eac4e7dd67
    Moving slot 1 from b3e4648c23770c5f5271092ad14148eac4e7dd67
    Moving slot 2 from b3e4648c23770c5f5271092ad14148eac4e7dd67


    最后输入 yes 确认计划

检查Slot 信息

172.30.60.2:8001> CLUSTER NODES
dfca9aff3abccba24483dce1efee76c8c8f2158f 172.30.60.3:8005@18005 slave 4ec8e913999792fc2f3d056fcab6c8a73fc1858e 0 1605082815692 4 connected
4ec8e913999792fc2f3d056fcab6c8a73fc1858e 172.30.60.2:8001@18001 myself,master - 0 1605082815000 1 connected 99-5460
b3e4648c23770c5f5271092ad14148eac4e7dd67 172.30.60.2:9000@19000 master - 0 1605082816501 7 connected
23bfc097bf05021ef74e942bcd388d75a1ef2694 172.30.60.4:8003@18003 master - 0 1605082816704 5 connected 11022-16383
5f63eeb3e408b47d40862c16e39480832853daaf 172.30.60.4:8006@18006 slave ee77ad4c9b27e9b212ddf5f929c3cb895752a06a 0 1605082815000 8 connected
ee77ad4c9b27e9b212ddf5f929c3cb895752a06a 172.30.60.3:8002@18002 master - 0 1605082816198 8 connected 0-98 5461-11021
9a8e6254af86f278d33108060ef1fac12e287bec 172.30.60.2:8004@18004 slave 23bfc097bf05021ef74e942bcd388d75a1ef2694 0 1605082816000 5 connected
172.30.60.2:8001> quit

可以看到9000 节点后面没有Slot信息了.

3:删除Master

1	src/redis-cli -a zwh --cluster del-node 172.30.60.2:9000 b3e4648c23770c5f5271092ad14148eac4e7dd67

检查节点

[root@xinyi redis-5.0.2]# src/redis-cli -a zwh --cluster del-node 172.30.60.2:9000 b3e4648c23770c5f5271092ad14148eac4e7dd67
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Removing node b3e4648c23770c5f5271092ad14148eac4e7dd67 from cluster 172.30.60.2:9000
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
[root@xinyi redis-5.0.2]# src/redis-cli -a zwh -h 172.30.60.2 -p 8001
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
172.30.60.2:8001> CLUSTER NODES
dfca9aff3abccba24483dce1efee76c8c8f2158f 172.30.60.3:8005@18005 slave 4ec8e913999792fc2f3d056fcab6c8a73fc1858e 0 1605082892917 4 connected
4ec8e913999792fc2f3d056fcab6c8a73fc1858e 172.30.60.2:8001@18001 myself,master - 0 1605082892000 1 connected 99-5460
23bfc097bf05021ef74e942bcd388d75a1ef2694 172.30.60.4:8003@18003 master - 0 1605082892000 5 connected 11022-16383
5f63eeb3e408b47d40862c16e39480832853daaf 172.30.60.4:8006@18006 slave ee77ad4c9b27e9b212ddf5f929c3cb895752a06a 0 1605082891405 8 connected
ee77ad4c9b27e9b212ddf5f929c3cb895752a06a 172.30.60.3:8002@18002 master - 0 1605082893421 8 connected 0-98 5461-11021
9a8e6254af86f278d33108060ef1fac12e287bec 172.30.60.2:8004@18004 slave 23bfc097bf05021ef74e942bcd388d75a1ef2694 0 1605082892412 5 connected
172.30.60.2:8001>

Redis Cluster 小知识

Redis 集群选举原理

当Slave发现自己的Master变为 FAIL(gossip)时,Slave会尝试 FailOver 欲成为新的Master, 但是因为Master 可能拥有多个 Slave, 次数就需要进行选举了.

选举

发现自己的Master 变为Fail,通gossip方式其他节点也会知道该Master已经出现故障.
记录自己选举次数 currentEpoch(新纪元) , currentEpoch +1, 并广播FAILOVER_AUTH_REQUEST给所有节点,尝试获取选票.
只有Master 节点才响应 FAILOVER_AUTH_REQUEST 广播.并检验其合法性,并Master会响应一个FAILOVER_AUTH_ACK, 一次currentEpoch Master只会响应一次FAILOVER_AUTH_REQUEST广播(只投一票)
Slave 统计收到的 FAILOVER_AUTH_ACK ,如果超过半数(Master),就直接晋升为新Master.并广播(PONG)通知其他集群节点,自己晋升为Master了.
如果没超过半数,继续进入下一轮选举,currentEpoch再 + 1;

发起FAILOVER_AUTH_REQUEST广播的时机

当Slave发现自己的 Master 变为Fail时, 不会立马发起选举, 一部分原因是因为gossip协议,让其他节点感知到该Master 已经Fail(和后续验证合法性有关,当其他节点没意识到该Master节点Fail时,当收到选举广播,会拒绝投票),而是延时一定的时间之后再发起选举.

延时计算公式:

1	Delay = 500ms + random (0 ~ 500ms) + SLAVE_RANK * 1000ms

其中SLAVE_RANK表示 Slave节点需要从总Master同步的数据总量,SLAVE_RANK 约下表示需要同步的数据越少,说明Slave 的数据越接近Master, 这种Slave 如果晋升为Master 需要同步的数据最少. Redis 更倾向于将这种Salve 晋升为Master ;
当存在多个Slave的SLAVE_RANK 一样,此时就需要拼random了.

Redis 避免脑裂问题

脑裂问题: 同时存在多个Master,让客户端不知道该选择哪个Master.

1	min‐slaves‐to‐write 1

打个比方来说:

当Slot信息确定了,对应的key-value需要存储到某个IP:PORT的Master节点上,但是因为网络波动,导致这个节点同时出现了多个Master,导致存值出现问题.

和主从模式一样,我们可以设置超时时间clusternodetimeout,不然集群那么迅速的选举.

还可以通过设置min‐slaves‐to‐write 1:表示只有Master 上面至少有一个从节点才认为其是Master.

当出现网络波动,网络瞬断, 导致Slave 晋升为Master ,当网络恢复正常及时原来的 Master 恢复,在设置了min‐slaves‐to‐write 配置的情况下,因为原来的Master 已经没有Slave了,就不会认为其是Master,而会选择新的Master,原master 加入集群也将变为Slave.

Redis Cluster 至少需要3个Master节点,并且推荐节点数为奇数

我们知道集群选举,选出新的Master 需要其他Master的 ACK 数量超过半数,所以需要满足大于等于3个节点的时候,当其中一个Master出现问题,选举才能进行的下去.

所谓的推荐节点数为奇数(2n+1),是相对于偶数(2n+2)来比较的;

这里我们就以 5台和 6台来说;

如果6台Master节点,当挂了 1或者2 台Master时. 选举都能选出新的Master,尽量维持6台Master的局面;

如果是5台呢? 是不是也能达到这种效果,而且还可以少一台机器,所以推荐使用奇数个节点是在节省机器资源的情况下给出的结论.