Consul - 集群

Posted on | In | |

搭建集群

Consul推荐在每个数据中心运行3或5个Servers以避免Server失败导致的数据丢失。

基于此，我们先用4个虚拟机来做试验：
|:—IP Address–:|:—–Role—-:|
| 192.168.244.130 | Consul Server |
| 192.168.244.131 | Consul Server |
| 192.168.244.132 | Consul Server |
| 192.168.244.133 | Consul Client |

Server Configuration

1. 生成Key，创建必要配置目录和文件

1
2
3
4
5
6
[root@Automation ~]# mkdir /etc/consul.d/server
[root@Automation ~]# mkdir /etc/consul.d/consul-server-data
[root@Automation consul.d]# ls
consul-server-data  ping.json  server  web.json
[root@Automation consul.d]# consul keygen
Vz3jcuLn6a6Ekq5PbrJO2w==

Key: It should be the same for all the servers and clients in a datacenter. If it’s different the consul members will refuse to join.

2. 创建Configration

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
{
"bind_addr": "192.168.244.130", 
"datacenter": "dc1",
"data_dir": "/etc/consul.d/consul-server-data",
"encrypt": "Vz3jcuLn6a6Ekq5PbrJO2w==",
"log_level": "INFO",
"enable_syslog": true,
"enable_debug": true,
"node_name": "ConsulServer1",
"server": true,
"bootstrap_expect": 3,
"rejoin_after_leave": true,
"retry_join": [ 
  "192.168.244.130",
  "192.168.244.131",
  "192.168.244.132"
  ]
 }

在每一个Server都要以下相同操作, 唯一要修改的就是IP地址和node_name。

3. 运行Server集群

Server 1:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
[root@Automation server]# consul agent -ui -config-dir /etc/consul.d/server/
WARNING: LAN keyring exists but -encrypt given, using keyring
WARNING: WAN keyring exists but -encrypt given, using keyring
bootstrap_expect > 0: expecting 3 servers
==> Starting Consul agent...
==> Consul agent running!
           Version: 'v1.0.2'
           Node ID: '96c8e751-0019-e213-ac42-4635fe1d89e5'
         Node name: 'ConsulServer1'
        Datacenter: 'dc1' (Segment: '<all>')
            Server: true (Bootstrap: false)
       Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, DNS: 8600)
      Cluster Addr: 192.168.244.130 (LAN: 8301, WAN: 8302)
           Encrypt: Gossip: true, TLS-Outgoing: false, TLS-Incoming: false

==> Log data will now stream in as it occurs:

    2018/01/02 13:17:35 [INFO] raft: Initial configuration (index=0): []
    2018/01/02 13:17:35 [INFO] serf: EventMemberJoin: ConsulServer1.dc1 192.168.244.130
    2018/01/02 13:17:35 [INFO] serf: EventMemberJoin: ConsulServer1 192.168.244.130
    2018/01/02 13:17:35 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)
    2018/01/02 13:17:35 [INFO] raft: Node at 192.168.244.130:8300 [Follower] entering Follower state (Leader: "")
    2018/01/02 13:17:35 [WARN] serf: Failed to re-join any previously known node
    2018/01/02 13:17:35 [WARN] serf: Failed to re-join any previously known node
    2018/01/02 13:17:35 [INFO] consul: Adding LAN server ConsulServer1 (Addr: tcp/192.168.244.130:8300) (DC: dc1)
    2018/01/02 13:17:35 [INFO] consul: Handled member-join event for server "ConsulServer1.dc1" in area "wan"
    2018/01/02 13:17:35 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)
    2018/01/02 13:17:35 [INFO] agent: Started HTTP server on 127.0.0.1:8500 (tcp)
    2018/01/02 13:17:35 [INFO] agent: started state syncer
    2018/01/02 13:17:35 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os scaleway softlayer
    2018/01/02 13:17:35 [INFO] agent: Joining LAN cluster...
    2018/01/02 13:17:35 [INFO] agent: (LAN) joining: [192.168.244.130 192.168.244.131 192.168.244.132]
    2018/01/02 13:17:35 [INFO] agent: (LAN) joined: 1 Err: <nil>
    2018/01/02 13:17:35 [INFO] agent: Join LAN completed. Synced with 1 initial agents

Server 2:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
[root@Automation server]# consul agent -ui -config-dir /etc/consul.d/server/
WARNING: LAN keyring exists but -encrypt given, using keyring
WARNING: WAN keyring exists but -encrypt given, using keyring
bootstrap_expect > 0: expecting 3 servers
==> Starting Consul agent...
==> Consul agent running!
           Version: 'v1.0.2'
           Node ID: 'c3a7562f-3ca5-2447-fa43-1e7ae3a3a6df'
         Node name: 'ConsulServer2'
        Datacenter: 'dc1' (Segment: '<all>')
            Server: true (Bootstrap: false)
       Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, DNS: 8600)
      Cluster Addr: 192.168.244.131 (LAN: 8301, WAN: 8302)
           Encrypt: Gossip: true, TLS-Outgoing: false, TLS-Incoming: false

==> Log data will now stream in as it occurs:

    2018/01/02 13:17:58 [INFO] raft: Initial configuration (index=0): []
    2018/01/02 13:17:58 [INFO] serf: EventMemberJoin: ConsulServer2.dc1 192.168.244.131
    2018/01/02 13:17:58 [INFO] serf: EventMemberJoin: ConsulServer2 192.168.244.131
    2018/01/02 13:17:58 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)
    2018/01/02 13:17:58 [INFO] raft: Node at 192.168.244.131:8300 [Follower] entering Follower state (Leader: "")
    2018/01/02 13:17:58 [INFO] serf: Attempting re-join to previously known node: ConsulServer1.dc1: 192.168.244.131:8302
    2018/01/02 13:17:58 [INFO] serf: Attempting re-join to previously known node: ConsulServer1: 192.168.244.131:8301
    2018/01/02 13:17:58 [INFO] consul: Adding LAN server ConsulServer2 (Addr: tcp/192.168.244.131:8300) (DC: dc1)
    2018/01/02 13:17:58 [INFO] consul: Handled member-join event for server "ConsulServer2.dc1" in area "wan"
    2018/01/02 13:17:58 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)
    2018/01/02 13:17:58 [INFO] agent: Started HTTP server on 127.0.0.1:8500 (tcp)
    2018/01/02 13:17:58 [INFO] agent: started state syncer
    2018/01/02 13:17:58 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os scaleway softlayer
    2018/01/02 13:17:58 [INFO] agent: Joining LAN cluster...
    2018/01/02 13:17:58 [INFO] agent: (LAN) joining: [192.168.244.130 192.168.244.131 192.168.244.132]
    2018/01/02 13:17:58 [INFO] serf: Re-joined to previously known node: ConsulServer1: 192.168.244.131:8301
    2018/01/02 13:17:58 [INFO] serf: Re-joined to previously known node: ConsulServer1.dc1: 192.168.244.131:8302
    2018/01/02 13:17:58 [INFO] serf: EventMemberJoin: ConsulServer1 192.168.244.130
    2018/01/02 13:17:58 [INFO] consul: Adding LAN server ConsulServer1 (Addr: tcp/192.168.244.130:8300) (DC: dc1)
    2018/01/02 13:17:58 [INFO] serf: EventMemberJoin: ConsulServer1.dc1 192.168.244.130
    2018/01/02 13:17:58 [INFO] consul: Handled member-join event for server "ConsulServer1.dc1" in area "wan"
    2018/01/02 13:17:58 [INFO] agent: (LAN) joined: 2 Err: <nil>
    2018/01/02 13:17:58 [INFO] agent: Join LAN completed. Synced with 2 initial agents
    2018/01/02 13:18:04 [WARN] raft: no known peers, aborting election
    2018/01/02 13:18:05 [ERR] agent: failed to sync remote state: No cluster leader

Server 3:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
[root@Automation server]# consul agent -ui -config-dir /etc/consul.d/server/
WARNING: LAN keyring exists but -encrypt given, using keyring
WARNING: WAN keyring exists but -encrypt given, using keyring
bootstrap_expect > 0: expecting 3 servers
==> Starting Consul agent...
==> Consul agent running!
           Version: 'v1.0.2'
           Node ID: 'ee9443b2-4aa1-f2cc-7b76-4b6530a99fe1'
         Node name: 'ConsulServer3'
        Datacenter: 'dc1' (Segment: '<all>')
            Server: true (Bootstrap: false)
       Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, DNS: 8600)
      Cluster Addr: 192.168.244.132 (LAN: 8301, WAN: 8302)
           Encrypt: Gossip: true, TLS-Outgoing: false, TLS-Incoming: false

==> Log data will now stream in as it occurs:

    2018/01/02 13:18:13 [INFO] raft: Initial configuration (index=0): []
    2018/01/02 13:18:13 [INFO] serf: EventMemberJoin: ConsulServer3.dc1 192.168.244.132
    2018/01/02 13:18:13 [INFO] serf: EventMemberJoin: ConsulServer3 192.168.244.132
    2018/01/02 13:18:13 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)
    2018/01/02 13:18:13 [INFO] raft: Node at 192.168.244.132:8300 [Follower] entering Follower state (Leader: "")
    2018/01/02 13:18:13 [INFO] serf: Attempting re-join to previously known node: ConsulServer1.dc1: 192.168.244.132:8302
    2018/01/02 13:18:13 [INFO] serf: Attempting re-join to previously known node: ConsulServer1: 192.168.244.132:8301
    2018/01/02 13:18:13 [INFO] consul: Adding LAN server ConsulServer3 (Addr: tcp/192.168.244.132:8300) (DC: dc1)
    2018/01/02 13:18:13 [INFO] consul: Handled member-join event for server "ConsulServer3.dc1" in area "wan"
    2018/01/02 13:18:13 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)
    2018/01/02 13:18:13 [INFO] agent: Started HTTP server on 127.0.0.1:8500 (tcp)
    2018/01/02 13:18:13 [INFO] agent: started state syncer
    2018/01/02 13:18:13 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os scaleway softlayer
    2018/01/02 13:18:13 [INFO] agent: Joining LAN cluster...
    2018/01/02 13:18:13 [INFO] agent: (LAN) joining: [192.168.244.130 192.168.244.131 192.168.244.132]
    2018/01/02 13:18:13 [INFO] serf: EventMemberJoin: ConsulServer2 192.168.244.131
    2018/01/02 13:18:13 [INFO] serf: EventMemberJoin: ConsulServer1 192.168.244.130
    2018/01/02 13:18:13 [INFO] serf: Re-joined to previously known node: ConsulServer1: 192.168.244.132:8301
    2018/01/02 13:18:13 [INFO] consul: Adding LAN server ConsulServer2 (Addr: tcp/192.168.244.131:8300) (DC: dc1)
    2018/01/02 13:18:13 [INFO] serf: Re-joined to previously known node: ConsulServer1.dc1: 192.168.244.132:8302
    2018/01/02 13:18:13 [INFO] agent: (LAN) joined: 3 Err: <nil>
    2018/01/02 13:18:13 [INFO] agent: Join LAN completed. Synced with 3 initial agents
    2018/01/02 13:18:13 [INFO] serf: EventMemberJoin: ConsulServer1.dc1 192.168.244.130
    2018/01/02 13:18:13 [INFO] serf: EventMemberJoin: ConsulServer2.dc1 192.168.244.131
    2018/01/02 13:18:13 [INFO] consul: Handled member-join event for server "ConsulServer1.dc1" in area "wan"
    2018/01/02 13:18:13 [INFO] consul: Handled member-join event for server "ConsulServer2.dc1" in area "wan"
    2018/01/02 13:18:13 [INFO] consul: Existing Raft peers reported by ConsulServer1, disabling bootstrap mode
    2018/01/02 13:18:13 [INFO] consul: Adding LAN server ConsulServer1 (Addr: tcp/192.168.244.130:8300) (DC: dc1)
2018/01/02 13:18:18 [DEBUG] raft-net: 192.168.244.132:8300 accepted connection from: 192.168.244.130:38206
2018/01/02 13:18:18 [DEBUG] raft-net: 192.168.244.132:8300 accepted connection from: 192.168.244.130:45520
    2018/01/02 13:18:18 [WARN] raft: Failed to get previous log: 1 log not found (last: 0)
    2018/01/02 13:18:19 [INFO] consul: New leader elected: ConsulServer1
    2018/01/02 13:18:19 [INFO] agent: Synced node info

4. 验证集群运行成功

在任一节点都可以验证：

1
2
3
4
5
[root@Automation ~]# consul members
Node           Address               Status  Type    Build  Protocol  DC   Segment
ConsulServer1  192.168.244.130:8301  alive   server  1.0.2  2         dc1  <all>
ConsulServer2  192.168.244.131:8301  alive   server  1.0.2  2         dc1  <all>
ConsulServer3  192.168.244.132:8301  alive   server  1.0.2  2         dc1  <all>

Client Configuration

1. 创建必要配置目录和文件

1
2
3
4
5
[root@Automation ~]# mkdir /etc/consul.d/client
[root@Automation ~]# mkdir /etc/consul.d/client-data
[root@Automation ~]# cd /etc/consul.d/
[root@Automation consul.d]# ls
client  client-data  ping.json  web.json

2. 创建Configration

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
{
"bind_addr": "192.168.244.134",
"datacenter": "dc1",
"data_dir": "/etc/consul.d/client-data",
"encrypt": "Vz3jcuLn6a6Ekq5PbrJO2w==",
"log_level": "INFO",
"enable_syslog": true,
"enable_debug": true,
"enable_script_checks": true,
"server": false,
   "service": {"name": "consul-test", "tags": ["HTTP"], "port": 80,
   "check": {"script": "curl localhost >/dev/null 2>&1", "interval": "10s"}},
"node_name": "ConsulClient1",
"server": false,
"rejoin_after_leave": true,
"retry_join": [
  "192.168.244.130",
  "192.168.244.131",
  "192.168.244.132"
  ]
 }

3. 运行Server集群

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
[root@Automation client]# consul agent -config-dir /etc/consul.d/client
==> Starting Consul agent...
==> Consul agent running!
           Version: 'v1.0.2'
           Node ID: 'c8167e25-7e41-0994-c5b9-8f5f739ecc2a'
         Node name: 'ConsulClient1'
        Datacenter: 'dc1' (Segment: '')
            Server: false (Bootstrap: false)
       Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, DNS: 8600)
      Cluster Addr: 192.168.244.134 (LAN: 8301, WAN: 8302)
           Encrypt: Gossip: true, TLS-Outgoing: false, TLS-Incoming: false

==> Log data will now stream in as it occurs:

    2018/01/02 14:09:02 [INFO] serf: EventMemberJoin: ConsulClient1 192.168.244.134
    2018/01/02 14:09:02 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)
    2018/01/02 14:09:02 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)
    2018/01/02 14:09:02 [INFO] agent: Started HTTP server on 127.0.0.1:8500 (tcp)
    2018/01/02 14:09:02 [INFO] agent: started state syncer
    2018/01/02 14:09:02 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os scaleway softlayer
    2018/01/02 14:09:02 [INFO] agent: Joining LAN cluster...
    2018/01/02 14:09:02 [INFO] agent: (LAN) joining: [192.168.244.130 192.168.244.131 192.168.244.132]
    2018/01/02 14:09:02 [WARN] manager: No servers available
    2018/01/02 14:09:02 [ERR] agent: failed to sync remote state: No known Consul servers
    2018/01/02 14:09:02 [INFO] serf: EventMemberJoin: ConsulServer3 192.168.244.132
    2018/01/02 14:09:02 [INFO] serf: EventMemberJoin: ConsulServer1 192.168.244.130
    2018/01/02 14:09:02 [INFO] serf: EventMemberJoin: ConsulServer2 192.168.244.131
    2018/01/02 14:09:02 [INFO] consul: adding server ConsulServer3 (Addr: tcp/192.168.244.132:8300) (DC: dc1)
    2018/01/02 14:09:02 [INFO] consul: adding server ConsulServer1 (Addr: tcp/192.168.244.130:8300) (DC: dc1)
    2018/01/02 14:09:02 [INFO] consul: adding server ConsulServer2 (Addr: tcp/192.168.244.131:8300) (DC: dc1)
    2018/01/02 14:09:02 [INFO] agent: (LAN) joined: 3 Err: <nil>
    2018/01/02 14:09:02 [INFO] agent: Join LAN completed. Synced with 3 initial agents
    2018/01/02 14:09:02 [INFO] agent: Synced node info

4. 验证集群运行成功

在任一节点都可以验证：

1
2
3
4
5
6
[root@Automation ~]# consul members
Node           Address               Status  Type    Build  Protocol  DC   Segment
ConsulServer1  192.168.244.130:8301  alive   server  1.0.2  2         dc1  <all>
ConsulServer2  192.168.244.131:8301  alive   server  1.0.2  2         dc1  <all>
ConsulServer3  192.168.244.132:8301  alive   server  1.0.2  2         dc1  <all>
ConsulClient1  192.168.244.134:8301  alive   client  1.0.2  2         dc1  <default>

Adding Servers

跟上边一样的步骤创建、配置、运行。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
{
"bind_addr": "192.168.244.135", 
"datacenter": "dc1",
"data_dir": "/etc/consul.d/consul-server-data",
"encrypt": "Vz3jcuLn6a6Ekq5PbrJO2w==",
"log_level": "INFO",
"enable_syslog": true,
"enable_debug": true,
"node_name": "ConsulServer4",
"server": true,
"rejoin_after_leave": true,
"retry_join": [ 
  "192.168.244.130",
  "192.168.244.131",
  "192.168.244.132",
  "192.168.244.135"
  ]
 }

验证

1
2
3
4
5
6
7
[root@Automation ~]# consul members
Node           Address               Status  Type    Build  Protocol  DC   Segment
ConsulServer1  192.168.244.130:8301  alive   server  1.0.2  2         dc1  <all>
ConsulServer2  192.168.244.131:8301  alive   server  1.0.2  2         dc1  <all>
ConsulServer3  192.168.244.132:8301  alive   server  1.0.2  2         dc1  <all>
ConsulServer4  192.168.244.135:8301  alive   server  1.0.2  2         dc1  <all>
ConsulClient1  192.168.244.134:8301  alive   client  1.0.2  2         dc1  <default>

Removing Servers

我们把Leader节点Remove

1
2
[root@Automation ~]# consul leave
Graceful leave complete

验证

1
2
3
4
5
6
7
[root@Automation ~]# consul members
Node           Address               Status  Type    Build  Protocol  DC   Segment
ConsulServer1  192.168.244.130:8301  alive   server  1.0.2  2         dc1  <all>
ConsulServer2  192.168.244.131:8301  left    server  1.0.2  2         dc1  <all>
ConsulServer3  192.168.244.132:8301  alive   server  1.0.2  2         dc1  <all>
ConsulServer4  192.168.244.136:8301  alive   server  1.0.2  2         dc1  <all>
ConsulClient1  192.168.244.134:8301  alive   client  1.0.2  2         dc1  <default>

当Leader Server出问题后，会自动选择新的Leader：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
    2018/01/02 15:14:53 [INFO] serf: EventMemberLeave: ConsulServer2 192.168.244.131
    2018/01/02 15:14:53 [INFO] consul: Removing LAN server ConsulServer2 (Addr: tcp/192.168.244.131:8300) (DC: dc1)
    2018/01/02 15:14:57 [WARN] raft: Heartbeat timeout from "192.168.244.131:8300" reached, starting election
    2018/01/02 15:14:57 [INFO] raft: Node at 192.168.244.136:8300 [Candidate] entering Candidate state in term 52
    2018/01/02 15:14:59 [INFO] raft: Duplicate RequestVote for same term: 52
2018/01/02 15:15:00 [DEBUG] raft-net: 192.168.244.136:8300 accepted connection from: 192.168.244.130:34750
    2018/01/02 15:15:00 [INFO] raft: Duplicate RequestVote for same term: 52
    2018/01/02 15:15:04 [WARN] raft: Election timeout reached, restarting election
    2018/01/02 15:15:04 [INFO] raft: Node at 192.168.244.136:8300 [Candidate] entering Candidate state in term 53
    2018/01/02 15:15:04 [INFO] raft: Election won. Tally: 2
    2018/01/02 15:15:04 [INFO] raft: Node at 192.168.244.136:8300 [Leader] entering Leader state
    2018/01/02 15:15:04 [INFO] raft: Added peer 96c8e751-0019-e213-ac42-4635fe1d89e5, starting replication
    2018/01/02 15:15:04 [INFO] raft: Added peer ee9443b2-4aa1-f2cc-7b76-4b6530a99fe1, starting replication
    2018/01/02 15:15:04 [INFO] consul: cluster leadership acquired
    2018/01/02 15:15:04 [INFO] consul: New leader elected: ConsulServer4
    2018/01/02 15:15:04 [INFO] raft: pipelining replication to peer {Voter 96c8e751-0019-e213-ac42-4635fe1d89e5 192.168.244.130:8300}
    2018/01/02 15:15:04 [INFO] raft: pipelining replication to peer {Voter ee9443b2-4aa1-f2cc-7b76-4b6530a99fe1 192.168.244.132:8300}
    2018/01/02 15:15:04 [INFO] consul: member 'ConsulServer2' left, deregistering

唐胡璐 wechat
欢迎您扫一扫上面的微信公众号，订阅我的博客！
分享创造价值，您的支持将鼓励我继续前行！