设为首页收藏本站

[PHP] mysqlmha高可用架构的安装_MySQL

[复制链接]
查看14 | 回复0 | 2015-6-30 10:05:18 | 显示全部楼层 |阅读模式
MMM无法完全地保证数据的一致性,所以MMM适用于对数据的一致性要求不是很高,但是又想最大程度的保证业务可用性的场景对于那些对数据一致性要求很高的业务,非常不建议采用MMM的这种高可用性架构,那么可以考虑使用MHA。在mysql故障切换的过程中,MHA能够在0-30s内自动完成数据库的故障切换操作,并且MHA能够最大程度上保证数据的一致性,以达到真正意义上的高可用。
MHA软件由两部分组成,Manager工具包和Node工具包,具体的说明如下。

Manager工具包主要包括以下几个工具:
masterha_check_ssh 检查MHA的SSH配置状况
masterha_check_repl 检查MySQL复制状况
masterha_manger 启动MHA
masterha_check_status 检测当前MHA运行状态
masterha_master_monitor 检测master是否宕机
masterha_master_switch 控制故障转移(自动或者手动)
masterha_conf_host 添加或删除配置的server信息

Node工具包(这些工具通常由MHA Manager的脚本触发,无需人为操作)主要包括以下几个工具:
save_binary_logs 保存和复制master的二进制日志
apply_diff_relay_logs 识别差异的中继日志事件并将其差异的事件应用于其他的slave
filter_mysqlbinlog 去除不必要的ROLLBACK事件(MHA已不再使用这个工具)
purge_relay_logs 清除中继日志(不会阻塞SQL线程)


注意:
(1)为了尽可能的减少主库硬件损坏宕机造成的数据丢失,因此在配置MHA的同时建议配置成MySQL 5.5的半同步复制
1.1、搭建环境
用途 主机名 ip server_id 类型
master yaolansvr192.168.0.316803写入
candicate master/monitor host yaolansvr_slave192.168.0.416804 读
slave yaolansvr_slave01192.168.0.516805 读

1.2、yaolansvr安装ftp服务,并上传mha安装软件
(1)关闭selinux,否则vsftpd报226错误
  # mkdir -p /yangsq/ftp # useradd -d /yangsq/ftp -s /sbin/nologin uftp # passwd uftp # chown -R uftp:uftp /yangsq/ftp # yum list all|grep vsftpd # yum -y install vsftpd.x86_64 # cp /etc/vsftpd/vsftpd.conf /etc/vsftpd/vsftpd.conf.bak anonymous_enable=YES改为anonymous_enable=NO local_enable=YES write_enable=YES chroot_local_user=YES # chkconfig vsftpd on # service vsftpd start # yum install ftp.x86_64 -y # sestatus SELinux status: enabled SELinuxfs mount: /selinux Current mode: enforcing Mode from config file: enforcing Policy version: 24 Policy from config file: targeted # setenforce 0 # ftp 192.168.0.3 21  
1.3、所有数据库节点安装Perl模块,同时做
  yum -y install perl-DBD-MySQL yum -y install perl-CPAN.x86_64 cd /yangsq/ftp tar xvf mha4mysql-node-0.54.tar.gz cd mha4mysql-node-0.54 perl Makefile.PL make && make install Installing /usr/local/share/perl5/MHA/BinlogPosFinderXid.pm Installing /usr/local/share/perl5/MHA/SlaveUtil.pm Installing /usr/local/share/perl5/MHA/NodeUtil.pm Installing /usr/local/share/perl5/MHA/BinlogPosFinderElp.pm Installing /usr/local/share/perl5/MHA/BinlogPosFindManager.pm Installing /usr/local/share/perl5/MHA/BinlogPosFinder.pm Installing /usr/local/share/perl5/MHA/BinlogManager.pm Installing /usr/local/share/perl5/MHA/NodeConst.pm Installing /usr/local/share/perl5/MHA/BinlogHeaderParser.pm Installing /usr/local/share/man/man1/filter_mysqlbinlog.1 Installing /usr/local/share/man/man1/purge_relay_logs.1 Installing /usr/local/share/man/man1/apply_diff_relay_logs.1 Installing /usr/local/share/man/man1/save_binary_logs.1 Installing /usr/local/bin/filter_mysqlbinlog Installing /usr/local/bin/apply_diff_relay_logs Installing /usr/local/bin/save_binary_logs Installing /usr/local/bin/purge_relay_logs  

1.4、在yaolansvr_slave安装mha manager
(1)No package perl-Log-Dispatch available.
  # mv CentOS-Base.repo CentOS-Base.repo.bak sftp> put C:\Users\Yaolan\Downloads\CentOS6-Base-163.repo # yum clean all # yum makecache
或者# rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm


  yum -y install perl-Config-Tiny yum install perl-Log-Dispatch -y yum install perl-Parallel-ForkManager -y yum install perl-Time-HiRes -y # tar xvf mha4mysql-manager-0.54.tar.gz # cd mha4mysql-manager-0.54 # perl Makefile.PL # make && make install Installing /usr/local/share/perl5/MHA/ManagerAdminWrapper.pm Installing /usr/local/share/perl5/MHA/ManagerUtil.pm Installing /usr/local/share/perl5/MHA/MasterFailover.pm Installing /usr/local/share/perl5/MHA/MasterMonitor.pm Installing /usr/local/share/perl5/MHA/ManagerAdmin.pm Installing /usr/local/share/perl5/MHA/Config.pm Installing /usr/local/share/perl5/MHA/DBHelper.pm Installing /usr/local/share/perl5/MHA/HealthCheck.pm Installing /usr/local/share/perl5/MHA/FileStatus.pm Installing /usr/local/share/perl5/MHA/MasterRotate.pm Installing /usr/local/share/perl5/MHA/Server.pm Installing /usr/local/share/perl5/MHA/ServerManager.pm Installing /usr/local/share/perl5/MHA/SSHCheck.pm Installing /usr/local/share/perl5/MHA/ManagerConst.pm Installing /usr/local/share/man/man1/masterha_check_ssh.1 Installing /usr/local/share/man/man1/masterha_secondary_check.1 Installing /usr/local/share/man/man1/masterha_conf_host.1 Installing /usr/local/share/man/man1/masterha_check_status.1 Installing /usr/local/share/man/man1/masterha_stop.1 Installing /usr/local/share/man/man1/masterha_manager.1 Installing /usr/local/share/man/man1/masterha_master_monitor.1 Installing /usr/local/share/man/man1/masterha_check_repl.1 Installing /usr/local/share/man/man1/masterha_master_switch.1 Installing /usr/local/bin/masterha_manager Installing /usr/local/bin/masterha_check_ssh Installing /usr/local/bin/masterha_check_status Installing /usr/local/bin/masterha_master_monitor Installing /usr/local/bin/masterha_secondary_check Installing /usr/local/bin/masterha_conf_host Installing /usr/local/bin/masterha_check_repl Installing /usr/local/bin/masterha_stop Installing /usr/local/bin/masterha_master_switch  

1.5、所有mysql服务器配置ssh登录无密码验证,sshd_config已改不生效,万能的重启
注意:
(1)ssh-copy-id: command not found,解决:yum install openssh-clients -y
(2)对自己本身也要执行ssh-copy-id
(3)关闭slave后重启mysql实例,slave相关的进程正常启动
  # ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa) # yum install -y openssh-clients # ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.0.3 # ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.0.4 # ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.0.5  

1.6、搭建主从复制环境
yaolansvr服务器master节点-----------
注意:
(1)datadir和server-id的设置,candidate master和slave只修改server-id
(2)
  # vi /usr/mysql/etc/my.cnf [mysqld] port = 3306 datadir=/data/mysql/data #慢查询设置 slow-query-log-file=/var/log/MysqlQuery.log long_query_time =5 slow_query_log=1 #server-id server-id=16803 #binlog 设置 log-bin = /data/mysql/data/mysql-bin.log binlog_cache_size = 8M binlog_format=mixed #全局 join_buffer_size = 2M sort_buffer_size = 2M read_rnd_buffer_size = 2M read_buffer_size = 2M max_heap_table_size = 64M thread_cache_size=12 thread_cOncurrency= 12 query_cache_type = 1 query_cache_size = 32M ft_min_word_len = 4 thread_stack = 192K tmp_table_size = 64M #myisam内存设置 key_buffer_size=1024M #允许最大的复制传输 max_allowed_packet=64M #跳过dns解析 skip-name-resolve #连接数设置 max_cOnnections= 1000 max_connect_errors = 200 #innodb设置 innodb_buffer_pool_size = 1G innodb_additional_mem_pool_size = 16M innodb_log_buffer_size = 8M innodb_log_file_size = 512M innodb_log_files_in_group = 3 innodb_file_per_table=1 innodb_stats_persistent_sample_pages=1000 innodb_write_io_threads = 8 innodb_read_io_threads = 8 innodb_thread_cOncurrency= 16 innodb_flush_log_at_trx_commit = 2 innodb_lock_wait_timeout = 30 sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES # service mysqld start  
1.6.1、在yaolansvr做备份,并创建复制用户
  # mysqldump -A --flush-privileges --lock-all-tables --events --routines --triggers --master-data=2>/yangsq/ftp/`date +%Y-%m-%d`_all.sql mysql> grant replication slave on *.* to 'repl1'@'192.168.0.%' identified by '123456'; mysql> flush privileges;


1.6.2、使用ftp从yaolansvr获取整库的备份,并恢复
  # mysql show slave status; Empty set (0.00 sec) mysql> change master to master_host='192.168.0.3',master_user='repl1',master_password='123456',master_port=3306,master_log_file='mysql-bin.000008',master_log_pos=120; mysql> start slave; Last_IO_Errno: 1593 Last_IO_Error: Fatal error: The slave I/O thread stops because master and slave have equal MySQL server UUIDs; these UUIDs must be different for replication to work.
解决:两台datadir/auto.cnf一样,select uuid()不同,所以删除candidate master上的auto.cnf,重新启动实例


重启mysql实例后:
  mysql> show slave status\G *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: 192.168.0.3 Master_User: repl1 Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin.000008 Read_Master_Log_Pos: 408 Relay_Log_File: yaolansvr_slave-relay-bin.000003 Relay_Log_Pos: 571 Relay_Master_Log_File: mysql-bin.000008 Slave_IO_Running: Yes Slave_SQL_Running: Yes

1.6.3、其他的slave节点设置read_Only=1(不写入my.cnf,以供candidate master转为主后,提供写)
# mysql -e "set global read_Only=1"


1.6.4、注意******************所有数据库节点都要创建复制用户
(1)candidate master不创建复制用户,则报错:
  Mon Jun 29 17:28:00 2015 - [info] Alive Slaves: Mon Jun 29 17:28:00 2015 - [info] 192.168.0.4(192.168.0.4:3306) Version=5.6.20-log (oldest major version between slaves) log-bin:enabled Mon Jun 29 17:28:00 2015 - [info] Replicating from 192.168.0.3(192.168.0.3:3306) Mon Jun 29 17:28:00 2015 - [info] Primary candidate for the new Master (candidate_master is set) Mon Jun 29 17:28:00 2015 - [info] 192.168.0.5(192.168.0.5:3306) Version=5.6.20-log (oldest major version between slaves) log-bin:enabled Mon Jun 29 17:28:00 2015 - [info] Replicating from 192.168.0.3(192.168.0.3:3306) Mon Jun 29 17:28:00 2015 - [info] Not candidate for the new Master (no_master is set) Mon Jun 29 17:28:00 2015 - [info] Current Alive Master: 192.168.0.3(192.168.0.3:3306) Mon Jun 29 17:28:00 2015 - [info] Checking slave configurations.. Mon Jun 29 17:28:00 2015 - [info] Checking replication filtering settings.. Mon Jun 29 17:28:00 2015 - [info] binlog_do_db= , binlog_ignore_db= Mon Jun 29 17:28:00 2015 - [info] Replication filtering check ok. Mon Jun 29 17:28:00 2015 - [error][/usr/local/share/perl5/MHA/Server.pm, ln382] 192.168.0.4(192.168.0.4:3306): User repl1 does not exist or does not have REPLICATION SLAVE privilege! Other slaves can not start replication from this host.
(2)如果一个数据库节点不会成为备选master,且no_master=1,则不需要创建复制用户




只在candidate master创建复制用户,必须与master的复制用户相同
mysql> grant replication slave on *.* to 'repl1'@'192.168.0.%' identified by '123456';
mysql> flush privileges;




1.6.5、注意******************所有数据库节点都要创建监控用户,监控用户必须要
只在master上创建监控用户:但是会被复制
(1)其他节点必须创建监控用户,否则报错:
Mon Jun 29 18:02:41 2015 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln255] Got MySQL error when connecting 192.168.0.4(192.168.0.4:3306) :1045:Access denied for user 'monitor'@'192.168.0.4' (using password: YES), but this is not mysql crash. Check MySQL server settings.


mysql> grant all privileges on *.* to 'monitor'@'192.168.0.%' identified by '123456';
mysql> flush privileges;


1.7、配置MHA
(1)master_binlog_dir=/home/soft/mysql/3306/binlog必须是binlog所在目录
(2)如果一个数据库节点不会成为备选master,且no_master=1,则不需要创建复制用户
  # mkdir -p /etc/mha/app1 # vi /etc/mha/app1/app1.cnf [server default] manager_workdir=/etc/mha/app1 manager_log=/var/log/manager.log master_binlog_dir=/data/mysql/data/ ssh_user=root user=monitor password=123456 repl_user=repl1 repl_password=123456 secondary_check_script=masterha_secondary_check -s 192.168.0.3 -s 192.168.0.5 ping_interval=3 #master_ip_failover_script=/etc/mha/app1/master_ip_failover #shutdown_script=/script/masterha/power_manager #report_script=/script/masterha/send_report #master_ip_online_change_script=/etc/mha/master_ip_failover [server1] hostname=192.168.0.3 port=3306 #master_binlog_dir=/data/mysql/data candidate_master=1 [server2] hostname=192.168.0.4 port=3306 #master_binlog_dir=/data/mysql/data candidate_master=1 [server3] hostname=192.168.0.5 port=3306 no_master=1  
1.7.2、所有数据库节点 relay log的自动清除
# mysql -e "set global relay_log_purge=0"


1.7.3、所有数据库节点在环境变量中输出mysqlbinlog
# ln -sv /usr/mysql/bin/mysqlbinlog /usr/bin/mysqlbinlog


1.7.4、检查ssh的配置
注意:
(1)# cp /yangsq/ftp/mha4mysql-manager-0.54/samples/scripts/master_ip_failover /etc/mha/app1/master_ip_failover或将master_ip_failover_script=/etc/mha/app1/master_ip_failover注释掉
否则报错:Mon Jun 29 17:05:26 2015 - [info] /etc/mha/app1/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.0.3 --orig_master_ip=192.168.0.3 --orig_master_port=3306
Bareword "FIXME_xxx" not allowed while "strict subs" in use at /etc/mha/app1/master_ip_failover line 93.
(2)没有mysqlbinlog,报错:
Mon Jun 29 17:52:11 2015 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql,/var/log/mysql --output_file=/var/tmp/save_binary_logs_test --manager_version=0.54 --start_file=mysql-bin.000008
Mon Jun 29 17:52:11 2015 - [info] Connecting to root@192.168.0.3(192.168.0.3)..
Failed to save binary log: Binlog not found from /var/lib/mysql,/var/log/mysql! If you got this error at MHA Manager, please set "master_binlog_dir=/path/to/binlog_directory_of_the_master" correctly in the MHA Manager's configuration file and try again.


# masterha_check_ssh --cOnf=/etc/mha/app1/app1.cnf


# masterha_check_ssh --cOnf=/etc/mha/app1/app1.cnf
Mon Jun 29 17:33:39 2015 - [info] Reading server configurations from /etc/mha/app1/app1.cnf..
Mon Jun 29 17:33:39 2015 - [info] MHA::MasterMonitor version 0.54.
Mon Jun 29 17:33:39 2015 - [info] Dead Servers:
Mon Jun 29 17:33:39 2015 - [info] Alive Servers:
Mon Jun 29 17:33:39 2015 - [info] 192.168.0.3(192.168.0.3:3306)
Mon Jun 29 17:33:39 2015 - [info] 192.168.0.4(192.168.0.4:3306)
Mon Jun 29 17:33:39 2015 - [info] 192.168.0.5(192.168.0.5:3306)
Mon Jun 29 17:33:39 2015 - [info] Alive Slaves:
Mon Jun 29 17:33:39 2015 - [info] 192.168.0.4(192.168.0.4:3306) Version=5.6.20-log (oldest major version between slaves) log-bin:enabled
Mon Jun 29 17:33:39 2015 - [info] Replicating from 192.168.0.3(192.168.0.3:3306)
Mon Jun 29 17:33:39 2015 - [info] Primary candidate for the new Master (candidate_master is set)
Mon Jun 29 17:33:39 2015 - [info] 192.168.0.5(192.168.0.5:3306) Version=5.6.20-log (oldest major version between slaves) log-bin:enabled
Mon Jun 29 17:33:39 2015 - [info] Replicating from 192.168.0.3(192.168.0.3:3306)
Mon Jun 29 17:33:39 2015 - [info] Not candidate for the new Master (no_master is set)
Mon Jun 29 17:33:39 2015 - [info] Current Alive Master: 192.168.0.3(192.168.0.3:3306)
Mon Jun 29 17:33:39 2015 - [info] Checking slave configurations..
Mon Jun 29 17:33:39 2015 - [info] Checking replication filtering settings..
Mon Jun 29 17:33:39 2015 - [info] binlog_do_db= , binlog_ignore_db=
Mon Jun 29 17:33:39 2015 - [info] Replication filtering check ok.
Mon Jun 29 17:33:39 2015 - [info] Starting SSH connection tests..
Mon Jun 29 17:34:21 2015 - [info] All SSH connection tests passed successfully.
Mon Jun 29 17:34:21 2015 - [info] Checking MHA Node version..
Mon Jun 29 17:34:31 2015 - [info] Version check ok.
Mon Jun 29 17:34:31 2015 - [info] Checking SSH publickey authentication settings on the current master..
Mon Jun 29 17:34:32 2015 - [info] HealthCheck: SSH to 192.168.0.3 is reachable.
Mon Jun 29 17:34:32 2015 - [info] Master MHA Node version is 0.54.
Mon Jun 29 17:34:32 2015 - [info] Checking recovery script configurations on the current master..
Mon Jun 29 17:34:32 2015 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/data/mysql/data --output_file=/var/tmp/save_binary_logs_test --manager_version=0.54 --start_file=mysql-bin.000008
Mon Jun 29 17:34:32 2015 - [info] Connecting to root@192.168.0.3(192.168.0.3)..
Creating /var/tmp if not exists.. ok.
Checking output directory is accessible or not..
ok.
Binlog found at /data/mysql/data, up to mysql-bin.000008
Mon Jun 29 17:34:32 2015 - [info] Master setting check done.
Mon Jun 29 17:34:32 2015 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Mon Jun 29 17:34:32 2015 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='monitor' --slave_host=192.168.0.4 --slave_ip=192.168.0.4 --slave_port=3306 --workdir=/var/tmp --target_version=5.6.20-log --manager_version=0.54 --relay_log_info=/data/mysql/data/relay-log.info --relay_dir=/data/mysql/data/ --slave_pass=xxx
Mon Jun 29 17:34:32 2015 - [info] Connecting to root@192.168.0.4(192.168.0.4:22)..
Checking slave recovery environment settings..
Opening /data/mysql/data/relay-log.info ... ok.
Relay log found at /data/mysql/data, up to yaolansvr_slave-relay-bin.000003
Temporary relay log file is /data/mysql/data/yaolansvr_slave-relay-bin.000003
Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.
done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Mon Jun 29 17:34:33 2015 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='monitor' --slave_host=192.168.0.5 --slave_ip=192.168.0.5 --slave_port=3306 --workdir=/var/tmp --target_version=5.6.20-log --manager_version=0.54 --relay_log_info=/data/mysql/data/relay-log.info --relay_dir=/data/mysql/data/ --slave_pass=xxx
Mon Jun 29 17:34:33 2015 - [info] Connecting to root@192.168.0.5(192.168.0.5:22)..
reverse mapping checking getaddrinfo for bogon [192.168.0.5] failed - POSSIBLE BREAK-IN ATTEMPT!
Checking slave recovery environment settings..
Opening /data/mysql/data/relay-log.info ... ok.
Relay log found at /data/mysql/data, up to yaolansvr_slave01-relay-bin.000002
Temporary relay log file is /data/mysql/data/yaolansvr_slave01-relay-bin.000002
Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.
done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Mon Jun 29 17:34:43 2015 - [info] Slaves settings check done.
Mon Jun 29 17:34:43 2015 - [info]
192.168.0.3 (current master)
+--192.168.0.4
+--192.168.0.5


Mon Jun 29 17:34:43 2015 - [info] Checking replication health on 192.168.0.4..
Mon Jun 29 17:34:43 2015 - [info] ok.
Mon Jun 29 17:34:43 2015 - [info] Checking replication health on 192.168.0.5..
Mon Jun 29 17:34:43 2015 - [info] ok.
Mon Jun 29 17:34:43 2015 - [warning] master_ip_failover_script is not defined.
Mon Jun 29 17:34:43 2015 - [warning] shutdown_script is not defined.
Mon Jun 29 17:34:43 2015 - [info] Got exit code 0 (Not master dead).




1.8、vip配置
为了防止脑裂发生,推荐生产环境采用脚本的方式来管理虚拟ip,而不是使用keepalived来完成


1.8.1、修改故障切换的脚本
# vi /etc/mha/app1/master_ip_failover
####添加变量
my $vip = '192.168.0.10/24';
my $key = '1';
my $ssh_start_vip = "/sbin/ifconfig eth1:$key $vip";
my $ssh_stop_vip = "/sbin/ifconfig eth1:$key down";


####首先主节点需要启动vip
# /sbin/ifconfig eth0:1 192.168.0.10/24
# /sbin/ifconfig eth0:1 down


####然后检查复制环境状况
# masterha_check_repl --cOnf=/etc/mha/app1/app1.cnf


# masterha_check_status --cOnf=/etc/mha/app1/app1.cnf
app1 is stopped(2:NOT_RUNNING).


# touch /etc/mha/app1/manager.log
# nohup masterha_manager --cOnf=/etc/mha/app1/app1.cnf /etc/mha/app1/manager.log 2>&1 &
# tail -f /var/log/manager.log
Tue Jun 30 10:21:25 2015 - [info] Got terminate signal. Exit.
Tue Jun 30 10:21:58 2015 - [info] MHA::MasterMonitor version 0.54.
Tue Jun 30 10:21:58 2015 - [info] Dead Servers:
Tue Jun 30 10:21:58 2015 - [info] Alive Servers:
Tue Jun 30 10:21:58 2015 - [info] 192.168.0.3(192.168.0.3:3306)
Tue Jun 30 10:21:58 2015 - [info] 192.168.0.4(192.168.0.4:3306)
Tue Jun 30 10:21:58 2015 - [info] 192.168.0.5(192.168.0.5:3306)
Tue Jun 30 10:21:58 2015 - [info] Alive Slaves:
Tue Jun 30 10:21:58 2015 - [info] 192.168.0.4(192.168.0.4:3306) Version=5.6.20-log (oldest major version between slaves) log-bin:enabled
Tue Jun 30 10:21:58 2015 - [info] Replicating from 192.168.0.3(192.168.0.3:3306)
Tue Jun 30 10:21:58 2015 - [info] Primary candidate for the new Master (candidate_master is set)
Tue Jun 30 10:21:58 2015 - [info] 192.168.0.5(192.168.0.5:3306) Version=5.6.20-log (oldest major version between slaves) log-bin:enabled
Tue Jun 30 10:21:58 2015 - [info] Replicating from 192.168.0.3(192.168.0.3:3306)
Tue Jun 30 10:21:58 2015 - [info] Not candidate for the new Master (no_master is set)
Tue Jun 30 10:21:58 2015 - [info] Current Alive Master: 192.168.0.3(192.168.0.3:3306)
Tue Jun 30 10:21:58 2015 - [info] Checking slave configurations..
Tue Jun 30 10:21:58 2015 - [info] read_Only=1 is not set on slave 192.168.0.4(192.168.0.4:3306).
Tue Jun 30 10:21:58 2015 - [warning] relay_log_purge=0 is not set on slave 192.168.0.4(192.168.0.4:3306).
Tue Jun 30 10:21:58 2015 - [info] read_Only=1 is not set on slave 192.168.0.5(192.168.0.5:3306).
Tue Jun 30 10:21:58 2015 - [warning] relay_log_purge=0 is not set on slave 192.168.0.5(192.168.0.5:3306).
Tue Jun 30 10:21:58 2015 - [info] Checking replication filtering settings..
Tue Jun 30 10:21:58 2015 - [info] binlog_do_db= , binlog_ignore_db=
Tue Jun 30 10:21:58 2015 - [info] Replication filtering check ok.
Tue Jun 30 10:21:58 2015 - [info] Starting SSH connection tests..
Tue Jun 30 10:22:00 2015 - [info] All SSH connection tests passed successfully.
Tue Jun 30 10:22:00 2015 - [info] Checking MHA Node version..
Tue Jun 30 10:22:01 2015 - [info] Version check ok.
Tue Jun 30 10:22:01 2015 - [info] Checking SSH publickey authentication settings on the current master..
Tue Jun 30 10:22:01 2015 - [info] HealthCheck: SSH to 192.168.0.3 is reachable.
Tue Jun 30 10:22:01 2015 - [info] Master MHA Node version is 0.54.
Tue Jun 30 10:22:01 2015 - [info] Checking recovery script configurations on the current master..
Tue Jun 30 10:22:01 2015 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/data/mysql/data/ --output_file=/var/tmp/save_binary_logs_test --manager_version=0.54 --start_file=mysql-bin.000009
Tue Jun 30 10:22:01 2015 - [info] Connecting to root@192.168.0.3(192.168.0.3)..
Creating /var/tmp if not exists.. ok.
Checking output directory is accessible or not..
ok.
Binlog found at /data/mysql/data/, up to mysql-bin.000009
Tue Jun 30 10:22:01 2015 - [info] Master setting check done.
Tue Jun 30 10:22:01 2015 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Tue Jun 30 10:22:01 2015 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='monitor' --slave_host=192.168.0.4 --slave_ip=192.168.0.4 --slave_port=3306 --workdir=/var/tmp --target_version=5.6.20-log --manager_version=0.54 --relay_log_info=/data/mysql/data/relay-log.info --relay_dir=/data/mysql/data/ --slave_pass=xxx
Tue Jun 30 10:22:01 2015 - [info] Connecting to root@192.168.0.4(192.168.0.4:22)..
Checking slave recovery environment settings..
Opening /data/mysql/data/relay-log.info ... ok.
Relay log found at /data/mysql/data, up to yaolansvr_slave-relay-bin.000006
Temporary relay log file is /data/mysql/data/yaolansvr_slave-relay-bin.000006
Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.
done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Tue Jun 30 10:22:02 2015 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='monitor' --slave_host=192.168.0.5 --slave_ip=192.168.0.5 --slave_port=3306 --workdir=/var/tmp --target_version=5.6.20-log --manager_version=0.54 --relay_log_info=/data/mysql/data/relay-log.info --relay_dir=/data/mysql/data/ --slave_pass=xxx
Tue Jun 30 10:22:02 2015 - [info] Connecting to root@192.168.0.5(192.168.0.5:22)..
Checking slave recovery environment settings..
Opening /data/mysql/data/relay-log.info ... ok.
Relay log found at /data/mysql/data, up to yaolansvr_slave01-relay-bin.000005
Temporary relay log file is /data/mysql/data/yaolansvr_slave01-relay-bin.000005
Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.
done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Tue Jun 30 10:22:02 2015 - [info] Slaves settings check done.
Tue Jun 30 10:22:02 2015 - [info]
192.168.0.3 (current master)
+--192.168.0.4
+--192.168.0.5


Tue Jun 30 10:22:02 2015 - [info] Checking master_ip_failover_script status:
Tue Jun 30 10:22:02 2015 - [info] /etc/mha/app1/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.0.3 --orig_master_ip=192.168.0.3 --orig_master_port=3306
Tue Jun 30 10:22:02 2015 - [info] OK.
Tue Jun 30 10:22:02 2015 - [warning] shutdown_script is not defined.
Tue Jun 30 10:22:02 2015 - [info] Set master ping interval 3 seconds.
Tue Jun 30 10:22:02 2015 - [info] Set secondary check script: masterha_secondary_check -s 192.168.0.3 -s 192.168.0.5
Tue Jun 30 10:22:02 2015 - [info] Starting ping health check on 192.168.0.3(192.168.0.3:3306)..
Tue Jun 30 10:22:02 2015 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..


# masterha_check_status --cOnf=/etc/mha/app1/app1.cnf
app1 (pid:2243) is running(0:PING_OK), master:192.168.0.3


# masterha_stop --cOnf=/etc/mha/app1/app1.cnf

进行切换后:
# tail -f /var/log/manager.log
Tue Jun 30 10:28:02 2015 - [warning] Got error on MySQL select ping: 2006 (MySQL server has gone away)
Tue Jun 30 10:28:02 2015 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/data/mysql/data/ --output_file=/var/tmp/save_binary_logs_test --manager_version=0.54 --binlog_prefix=mysql-bin
Tue Jun 30 10:28:03 2015 - [info] Executing seconary network check script: masterha_secondary_check -s 192.168.0.3 -s 192.168.0.5 --user=root --master_host=192.168.0.3 --master_ip=192.168.0.3 --master_port=3306
Tue Jun 30 10:28:03 2015 - [info] HealthCheck: SSH to 192.168.0.3 is reachable.
Monitoring server 192.168.0.3 is reachable, Master is not reachable from 192.168.0.3. OK.
Monitoring server 192.168.0.5 is reachable, Master is not reachable from 192.168.0.5. OK.
Tue Jun 30 10:28:03 2015 - [info] Master is not reachable from all other monitoring servers. Failover should start.
Tue Jun 30 10:28:05 2015 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)
Tue Jun 30 10:28:05 2015 - [warning] Connection failed 1 time(s)..
Tue Jun 30 10:28:08 2015 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)
Tue Jun 30 10:28:08 2015 - [warning] Connection failed 2 time(s)..
Tue Jun 30 10:28:11 2015 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)
Tue Jun 30 10:28:11 2015 - [warning] Connection failed 3 time(s)..
Tue Jun 30 10:28:11 2015 - [warning] Master is not reachable from health checker!
Tue Jun 30 10:28:11 2015 - [warning] Master 192.168.0.3(192.168.0.3:3306) is not reachable!
Tue Jun 30 10:28:11 2015 - [warning] SSH is reachable.
Tue Jun 30 10:28:11 2015 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/mha/app1/app1.cnf again, and trying to connect to all servers to check server status..
Tue Jun 30 10:28:11 2015 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Tue Jun 30 10:28:11 2015 - [info] Reading application default configurations from /etc/mha/app1/app1.cnf..
Tue Jun 30 10:28:11 2015 - [info] Reading server configurations from /etc/mha/app1/app1.cnf..
Tue Jun 30 10:28:12 2015 - [info] Dead Servers:
Tue Jun 30 10:28:12 2015 - [info] 192.168.0.3(192.168.0.3:3306)
Tue Jun 30 10:28:12 2015 - [info] Alive Servers:
Tue Jun 30 10:28:12 2015 - [info] 192.168.0.4(192.168.0.4:3306)
Tue Jun 30 10:28:12 2015 - [info] 192.168.0.5(192.168.0.5:3306)
Tue Jun 30 10:28:12 2015 - [info] Alive Slaves:
Tue Jun 30 10:28:12 2015 - [info] 192.168.0.4(192.168.0.4:3306) Version=5.6.20-log (oldest major version between slaves) log-bin:enabled
Tue Jun 30 10:28:12 2015 - [info] Replicating from 192.168.0.3(192.168.0.3:3306)
Tue Jun 30 10:28:12 2015 - [info] Primary candidate for the new Master (candidate_master is set)
Tue Jun 30 10:28:12 2015 - [info] 192.168.0.5(192.168.0.5:3306) Version=5.6.20-log (oldest major version between slaves) log-bin:enabled
Tue Jun 30 10:28:12 2015 - [info] Replicating from 192.168.0.3(192.168.0.3:3306)
Tue Jun 30 10:28:12 2015 - [info] Not candidate for the new Master (no_master is set)
Tue Jun 30 10:28:12 2015 - [info] Checking slave configurations..
Tue Jun 30 10:28:12 2015 - [warning] relay_log_purge=0 is not set on slave 192.168.0.4(192.168.0.4:3306).
Tue Jun 30 10:28:12 2015 - [warning] relay_log_purge=0 is not set on slave 192.168.0.5(192.168.0.5:3306).
Tue Jun 30 10:28:12 2015 - [info] Checking replication filtering settings..
Tue Jun 30 10:28:12 2015 - [info] Replication filtering check ok.
Tue Jun 30 10:28:12 2015 - [info] Master is down!
Tue Jun 30 10:28:12 2015 - [info] Terminating monitoring script.
Tue Jun 30 10:28:12 2015 - [info] Got exit code 20 (Master dead).
Tue Jun 30 10:28:12 2015 - [info] MHA::MasterFailover version 0.54.
Tue Jun 30 10:28:12 2015 - [info] Starting master failover.
Tue Jun 30 10:28:12 2015 - [info]
Tue Jun 30 10:28:12 2015 - [info] * Phase 1: Configuration Check Phase..
Tue Jun 30 10:28:12 2015 - [info]
Tue Jun 30 10:28:12 2015 - [info] Dead Servers:
Tue Jun 30 10:28:12 2015 - [info] 192.168.0.3(192.168.0.3:3306)
Tue Jun 30 10:28:12 2015 - [info] Checking master reachability via mysql(double check)..
Tue Jun 30 10:28:12 2015 - [info] ok.
Tue Jun 30 10:28:12 2015 - [info] Alive Servers:
Tue Jun 30 10:28:12 2015 - [info] 192.168.0.4(192.168.0.4:3306)
Tue Jun 30 10:28:12 2015 - [info] 192.168.0.5(192.168.0.5:3306)
Tue Jun 30 10:28:12 2015 - [info] Alive Slaves:
Tue Jun 30 10:28:12 2015 - [info] 192.168.0.4(192.168.0.4:3306) Version=5.6.20-log (oldest major version between slaves) log-bin:enabled
Tue Jun 30 10:28:12 2015 - [info] Replicating from 192.168.0.3(192.168.0.3:3306)
Tue Jun 30 10:28:12 2015 - [info] Primary candidate for the new Master (candidate_master is set)
Tue Jun 30 10:28:12 2015 - [info] 192.168.0.5(192.168.0.5:3306) Version=5.6.20-log (oldest major version between slaves) log-bin:enabled
Tue Jun 30 10:28:12 2015 - [info] Replicating from 192.168.0.3(192.168.0.3:3306)
Tue Jun 30 10:28:12 2015 - [info] Not candidate for the new Master (no_master is set)
Tue Jun 30 10:28:12 2015 - [info] ** Phase 1: Configuration Check Phase completed.
Tue Jun 30 10:28:12 2015 - [info]
Tue Jun 30 10:28:12 2015 - [info] * Phase 2: Dead Master Shutdown Phase..
Tue Jun 30 10:28:12 2015 - [info]
Tue Jun 30 10:28:12 2015 - [info] Forcing shutdown so that applications never connect to the current master..
Tue Jun 30 10:28:12 2015 - [info] Executing master IP deactivatation script:
Tue Jun 30 10:28:12 2015 - [info] /etc/mha/app1/master_ip_failover --orig_master_host=192.168.0.3 --orig_master_ip=192.168.0.3 --orig_master_port=3306 --command=stopssh --ssh_user=root
Disabling the VIP on old master: 192.168.0.3
Tue Jun 30 10:28:13 2015 - [info] done.
Tue Jun 30 10:28:13 2015 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Tue Jun 30 10:28:13 2015 - [info] * Phase 2: Dead Master Shutdown Phase completed.
Tue Jun 30 10:28:13 2015 - [info]
Tue Jun 30 10:28:13 2015 - [info] * Phase 3: Master Recovery Phase..
Tue Jun 30 10:28:13 2015 - [info]
Tue Jun 30 10:28:13 2015 - [info] * Phase 3.1: Getting Latest Slaves Phase..
Tue Jun 30 10:28:13 2015 - [info]
Tue Jun 30 10:28:13 2015 - [info] The latest binary log file/position on all slaves is mysql-bin.000009:120
Tue Jun 30 10:28:13 2015 - [info] Latest slaves (Slaves that received relay log files to the latest):
Tue Jun 30 10:28:13 2015 - [info] 192.168.0.4(192.168.0.4:3306) Version=5.6.20-log (oldest major version between slaves) log-bin:enabled
Tue Jun 30 10:28:13 2015 - [info] Replicating from 192.168.0.3(192.168.0.3:3306)
Tue Jun 30 10:28:13 2015 - [info] Primary candidate for the new Master (candidate_master is set)
Tue Jun 30 10:28:13 2015 - [info] 192.168.0.5(192.168.0.5:3306) Version=5.6.20-log (oldest major version between slaves) log-bin:enabled
Tue Jun 30 10:28:13 2015 - [info] Replicating from 192.168.0.3(192.168.0.3:3306)
Tue Jun 30 10:28:13 2015 - [info] Not candidate for the new Master (no_master is set)
Tue Jun 30 10:28:13 2015 - [info] The oldest binary log file/position on all slaves is mysql-bin.000009:120
Tue Jun 30 10:28:13 2015 - [info] Oldest slaves:
Tue Jun 30 10:28:13 2015 - [info] 192.168.0.4(192.168.0.4:3306) Version=5.6.20-log (oldest major version between slaves) log-bin:enabled
Tue Jun 30 10:28:13 2015 - [info] Replicating from 192.168.0.3(192.168.0.3:3306)
Tue Jun 30 10:28:13 2015 - [info] Primary candidate for the new Master (candidate_master is set)
Tue Jun 30 10:28:13 2015 - [info] 192.168.0.5(192.168.0.5:3306) Version=5.6.20-log (oldest major version between slaves) log-bin:enabled
Tue Jun 30 10:28:13 2015 - [info] Replicating from 192.168.0.3(192.168.0.3:3306)
Tue Jun 30 10:28:13 2015 - [info] Not candidate for the new Master (no_master is set)
Tue Jun 30 10:28:13 2015 - [info]
Tue Jun 30 10:28:13 2015 - [info] * Phase 3.2: Saving Dead Master's Binlog Phase..
Tue Jun 30 10:28:13 2015 - [info]
Tue Jun 30 10:28:13 2015 - [info] Fetching dead master's binary logs..
Tue Jun 30 10:28:13 2015 - [info] Executing command on the dead master 192.168.0.3(192.168.0.3:3306): save_binary_logs --command=save --start_file=mysql-bin.000009 --start_pos=120 --binlog_dir=/data/mysql/data/ --output_file=/var/tmp/saved_master_binlog_from_192.168.0.3_3306_20150630102812.binlog --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.54
Creating /var/tmp if not exists.. ok.
Concat binary/relay logs from mysql-bin.000009 pos 120 to mysql-bin.000009 EOF into /var/tmp/saved_master_binlog_from_192.168.0.3_3306_20150630102812.binlog ..
Dumping binlog format description event, from position 0 to 120.. ok.
Dumping effective binlog data from /data/mysql/data//mysql-bin.000009 position 120 to tail(143).. ok.
Concat succeeded.
Tue Jun 30 10:28:14 2015 - [info] scp from root@192.168.0.3:/var/tmp/saved_master_binlog_from_192.168.0.3_3306_20150630102812.binlog to local:/etc/mha/app1/saved_master_binlog_from_192.168.0.3_3306_20150630102812.binlog succeeded.
Tue Jun 30 10:28:14 2015 - [info] HealthCheck: SSH to 192.168.0.4 is reachable.
Tue Jun 30 10:28:15 2015 - [info] HealthCheck: SSH to 192.168.0.5 is reachable.
Tue Jun 30 10:28:15 2015 - [info]
Tue Jun 30 10:28:15 2015 - [info] * Phase 3.3: Determining New Master Phase..
Tue Jun 30 10:28:15 2015 - [info]
Tue Jun 30 10:28:15 2015 - [info] Finding the latest slave that has all relay logs for recovering other slaves..
Tue Jun 30 10:28:15 2015 - [info] All slaves received relay logs to the same position. No need to resync each other.
Tue Jun 30 10:28:15 2015 - [info] Searching new master from slaves..
Tue Jun 30 10:28:15 2015 - [info] Candidate masters from the configuration file:
Tue Jun 30 10:28:15 2015 - [info] 192.168.0.4(192.168.0.4:3306) Version=5.6.20-log (oldest major version between slaves) log-bin:enabled
Tue Jun 30 10:28:15 2015 - [info] Replicating from 192.168.0.3(192.168.0.3:3306)
Tue Jun 30 10:28:15 2015 - [info] Primary candidate for the new Master (candidate_master is set)
Tue Jun 30 10:28:15 2015 - [info] Non-candidate masters:
Tue Jun 30 10:28:15 2015 - [info] 192.168.0.5(192.168.0.5:3306) Version=5.6.20-log (oldest major version between slaves) log-bin:enabled
Tue Jun 30 10:28:15 2015 - [info] Replicating from 192.168.0.3(192.168.0.3:3306)
Tue Jun 30 10:28:15 2015 - [info] Not candidate for the new Master (no_master is set)
Tue Jun 30 10:28:15 2015 - [info] Searching from candidate_master slaves which have received the latest relay log events..
Tue Jun 30 10:28:15 2015 - [info] New master is 192.168.0.4(192.168.0.4:3306)
Tue Jun 30 10:28:15 2015 - [info] Starting master failover..
Tue Jun 30 10:28:15 2015 - [info]
From:
192.168.0.3 (current master)
+--192.168.0.4
+--192.168.0.5


To:
192.168.0.4 (new master)
+--192.168.0.5
Tue Jun 30 10:28:15 2015 - [info]
Tue Jun 30 10:28:15 2015 - [info] * Phase 3.3: New Master Diff Log Generation Phase..
Tue Jun 30 10:28:15 2015 - [info]
Tue Jun 30 10:28:15 2015 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
Tue Jun 30 10:28:15 2015 - [info] Sending binlog..
Tue Jun 30 10:28:15 2015 - [info] scp from local:/etc/mha/app1/saved_master_binlog_from_192.168.0.3_3306_20150630102812.binlog to root@192.168.0.4:/var/tmp/saved_master_binlog_from_192.168.0.3_3306_20150630102812.binlog succeeded.
Tue Jun 30 10:28:15 2015 - [info]
Tue Jun 30 10:28:15 2015 - [info] * Phase 3.4: Master Log Apply Phase..
Tue Jun 30 10:28:15 2015 - [info]
Tue Jun 30 10:28:15 2015 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed.
Tue Jun 30 10:28:15 2015 - [info] Starting recovery on 192.168.0.4(192.168.0.4:3306)..
Tue Jun 30 10:28:15 2015 - [info] Generating diffs succeeded.
Tue Jun 30 10:28:15 2015 - [info] Waiting until all relay logs are applied.
Tue Jun 30 10:28:15 2015 - [info] done.
Tue Jun 30 10:28:16 2015 - [info] Getting slave status..
Tue Jun 30 10:28:16 2015 - [info] This slave(192.168.0.4)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(mysql-bin.000009:120). No need to recover from Exec_Master_Log_Pos.
Tue Jun 30 10:28:16 2015 - [info] Connecting to the target slave host 192.168.0.4, running recover script..
Tue Jun 30 10:28:16 2015 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user='monitor' --slave_host=192.168.0.4 --slave_ip=192.168.0.4 --slave_port=3306 --apply_files=/var/tmp/saved_master_binlog_from_192.168.0.3_3306_20150630102812.binlog --workdir=/var/tmp --target_version=5.6.20-log --timestamp=20150630102812 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.54 --slave_pass=xxx
Tue Jun 30 10:28:16 2015 - [info]
MySQL client version is 5.6.20. Using --binary-mode.
Applying differential binary/relay log files /var/tmp/saved_master_binlog_from_192.168.0.3_3306_20150630102812.binlog on 192.168.0.4:3306. This may take long time...
Applying log files succeeded.
Tue Jun 30 10:28:16 2015 - [info] All relay logs were successfully applied.
Tue Jun 30 10:28:16 2015 - [info] Getting new master's binlog name and position..
Tue Jun 30 10:28:16 2015 - [info] mysql-bin.000015:120
Tue Jun 30 10:28:16 2015 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.0.4', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000015', MASTER_LOG_POS=120, MASTER_USER='repl1', MASTER_PASSWORD='xxx';
Tue Jun 30 10:28:16 2015 - [info] Executing master IP activate script:
Tue Jun 30 10:28:16 2015 - [info] /etc/mha/app1/master_ip_failover --command=start --ssh_user=root --orig_master_host=192.168.0.3 --orig_master_ip=192.168.0.3 --orig_master_port=3306 --new_master_host=192.168.0.4 --new_master_ip=192.168.0.4 --new_master_port=3306 --new_master_user='monitor' --new_master_password='123456'
Enabling the VIP - 192.168.0.10/24 on the new master - 192.168.0.4
Tue Jun 30 10:28:16 2015 - [info] OK.
Tue Jun 30 10:28:16 2015 - [info] Setting read_Only=0 on 192.168.0.4(192.168.0.4:3306)..
Tue Jun 30 10:28:16 2015 - [info] ok.
Tue Jun 30 10:28:16 2015 - [info] ** Finished master recovery successfully.
Tue Jun 30 10:28:16 2015 - [info] * Phase 3: Master Recovery Phase completed.
Tue Jun 30 10:28:16 2015 - [info]
Tue Jun 30 10:28:16 2015 - [info] * Phase 4: Slaves Recovery Phase..
Tue Jun 30 10:28:16 2015 - [info]
Tue Jun 30 10:28:16 2015 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase..
Tue Jun 30 10:28:16 2015 - [info]
Tue Jun 30 10:28:16 2015 - [info] -- Slave diff file generation on host 192.168.0.5(192.168.0.5:3306) started, pid: 2658. Check tmp log /etc/mha/app1/192.168.0.5_3306_20150630102812.log if it takes time..
Tue Jun 30 10:28:16 2015 - [info]
Tue Jun 30 10:28:16 2015 - [info] Log messages from 192.168.0.5 ...
Tue Jun 30 10:28:16 2015 - [info]
Tue Jun 30 10:28:16 2015 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
Tue Jun 30 10:28:16 2015 - [info] End of log messages from 192.168.0.5.
Tue Jun 30 10:28:16 2015 - [info] -- 192.168.0.5(192.168.0.5:3306) has the latest relay log events.
Tue Jun 30 10:28:16 2015 - [info] Generating relay diff files from the latest slave succeeded.
Tue Jun 30 10:28:16 2015 - [info]
Tue Jun 30 10:28:16 2015 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase..
Tue Jun 30 10:28:16 2015 - [info]
Tue Jun 30 10:28:16 2015 - [info] -- Slave recovery on host 192.168.0.5(192.168.0.5:3306) started, pid: 2660. Check tmp log /etc/mha/app1/192.168.0.5_3306_20150630102812.log if it takes time..
Tue Jun 30 10:28:18 2015 - [info]
Tue Jun 30 10:28:18 2015 - [info] Log messages from 192.168.0.5 ...
Tue Jun 30 10:28:18 2015 - [info]
Tue Jun 30 10:28:16 2015 - [info] Sending binlog..
Tue Jun 30 10:28:17 2015 - [info] scp from local:/etc/mha/app1/saved_master_binlog_from_192.168.0.3_3306_20150630102812.binlog to root@192.168.0.5:/var/tmp/saved_master_binlog_from_192.168.0.3_3306_20150630102812.binlog succeeded.
Tue Jun 30 10:28:17 2015 - [info] Starting recovery on 192.168.0.5(192.168.0.5:3306)..
Tue Jun 30 10:28:17 2015 - [info] Generating diffs succeeded.
Tue Jun 30 10:28:17 2015 - [info] Waiting until all relay logs are applied.
Tue Jun 30 10:28:17 2015 - [info] done.
Tue Jun 30 10:28:17 2015 - [info] Getting slave status..
Tue Jun 30 10:28:17 2015 - [info] This slave(192.168.0.5)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(mysql-bin.000009:120). No need to recover from Exec_Master_Log_Pos.
Tue Jun 30 10:28:17 2015 - [info] Connecting to the target slave host 192.168.0.5, running recover script..
Tue Jun 30 10:28:17 2015 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user='monitor' --slave_host=192.168.0.5 --slave_ip=192.168.0.5 --slave_port=3306 --apply_files=/var/tmp/saved_master_binlog_from_192.168.0.3_3306_20150630102812.binlog --workdir=/var/tmp --target_version=5.6.20-log --timestamp=20150630102812 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.54 --slave_pass=xxx
Tue Jun 30 10:28:17 2015 - [info]
MySQL client version is 5.6.20. Using --binary-mode.
Applying differential binary/relay log files /var/tmp/saved_master_binlog_from_192.168.0.3_3306_20150630102812.binlog on 192.168.0.5:3306. This may take long time...
Applying log files succeeded.
Tue Jun 30 10:28:17 2015 - [info] All relay logs were successfully applied.
Tue Jun 30 10:28:17 2015 - [info] Resetting slave 192.168.0.5(192.168.0.5:3306) and starting replication from the new master 192.168.0.4(192.168.0.4:3306)..
Tue Jun 30 10:28:18 2015 - [info] Executed CHANGE MASTER.
Tue Jun 30 10:28:18 2015 - [info] Slave started.
Tue Jun 30 10:28:18 2015 - [info] End of log messages from 192.168.0.5.
Tue Jun 30 10:28:18 2015 - [info] -- Slave recovery on host 192.168.0.5(192.168.0.5:3306) succeeded.
Tue Jun 30 10:28:18 2015 - [info] All new slave servers recovered successfully.
Tue Jun 30 10:28:18 2015 - [info]
Tue Jun 30 10:28:18 2015 - [info] * Phase 5: New master cleanup phase..
Tue Jun 30 10:28:18 2015 - [info]
Tue Jun 30 10:28:18 2015 - [info] Resetting slave info on the new master..
Tue Jun 30 10:28:18 2015 - [info] 192.168.0.4: Resetting slave info succeeded.
Tue Jun 30 10:28:18 2015 - [info] Master failover to 192.168.0.4(192.168.0.4:3306) completed successfully.
Tue Jun 30 10:28:18 2015 - [info]


----- Failover Report -----


app1: MySQL Master failover 192.168.0.3 to 192.168.0.4 succeeded


Master 192.168.0.3 is down!


Check MHA Manager logs at yaolansvr_slave:/var/log/manager.log for details.


Started automated(non-interactive) failover.
Invalidated master IP address on 192.168.0.3.
The latest slave 192.168.0.4(192.168.0.4:3306) has all relay logs for recovery.
Selected 192.168.0.4 as a new master.
192.168.0.4: OK: Applying all logs succeeded.
192.168.0.4: OK: Activated master IP address.
192.168.0.5: This host has the latest relay log events.
Generating relay diff files from the latest slave succeeded.
192.168.0.5: OK: Applying all logs succeeded. Slave started, replicating from 192.168.0.4.
192.168.0.4: Resetting slave info succeeded.
Master failover to 192.168.0.4(192.168.0.4:3306) completed successfully.

######192.168.0.4提升为主之后,查看状态:
# masterha_check_status --cOnf=/etc/mha/app1/app1.cnf
app1 is stopped(2:NOT_RUNNING).


mysql> select @@read_only;
+-------------+
| @@read_only |
+-------------+
| 0 |
+-------------+

######修复宕机的机器
首先cat /var/log/manager.log|grep "All other slaves should start"确定change master命令,把宕掉的数据库给启动,登陆进去后,slave status为空,使用change master命令设置应用的主节点,启动slave进程
然后设置read_Only=1,最后检查复制环境,并启动mha manager的监控,并把# mysql -e "set global relay_log_purge=0"




192.168.0.4关闭mysql后,192.168.0.3提升为主的过程中报错:
Tue Jun 30 11:50:37 2015 - [error][/usr/local/share/perl5/MHA/MasterFailover.pm, ln297] Last failover was done at 2015/06/30 10:05:18. Current time is too early to do failover again. If you want to do failover, manually remove /etc/mha/app1/app1.failover.complete and run this script again.
Tue Jun 30 11:50:37 2015 - [error][/usr/local/share/perl5/MHA/ManagerUtil.pm, ln178] Got ERROR: at /usr/local/bin/masterha_manager line 65
并且masterha_manager会立即死掉

注意:
(1)一旦重启slave,记得需要将mysql -e "set global read_Only=1"
回复

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

关键词 游戏源码 源码 网站源码 小程序源码 app源码 源码之家 源码资本 棋牌源码 源码网 源码时代 php源码 阿奇源码 源码下载 刀客源码 源码是什么 源码论坛 易语言源码 论坛源码 源码交易 java源码 源码中国 商城源码 asp源码 影视源码 源码站 安卓源码 时时彩源码 站长源码 a5源码 源码分享 网站源码下载 企业网站源码 android源码下载 php 源码 php源码下载 免费源码 asp源码下载 个人主页源码 网址导航源码 手游源码 源码天空 c源码 源码熊 asp 源码 在线客服源码 商业源码 精品源码 个人网站源码 asp网站源码 博客源码 私服源码 vb源码 aspnet源码 中国源码 cms源码 视频网站源码 易语言源码大全 神马影院php源码 c#源码 net源码 网页游戏源码 软件源码 源码哥 在线考试系统源码 安卓源码下载 源码库 源码爱好者 源码资源网 php网站源码 商业源码网 音乐网站源码 导航网站源码 源码商城 钓鱼网站源码 熊猫烧香源码 门户网站源码 .net源码 web源码 彩票源码 代刷网源码 小说网站源码 delphi源码 在线客服系统源码 整站源码 交友源码 房产网源码 广告联盟源码 aspnet 源码 源码社区 学校网站源码 公司网站源码 775源码屋 源码吧 新闻网站源码 云购源码 源码交易网 淘客源码 vb源码下载 股海网指标公式源码 通达信指标公式源码 源码集合 vip源码 电脑维修网站源码 web游戏源码 酷源码 彩票网站源码 源码程序 创业网站源码 asp.net源码 net源码下载 vbnet源码 亿乐社区源码 易语言源码网 源码超市 卖源码 电影网站源码 404源码社区 旅游网站源码 源码出售 flash源码 h站源码 net网站源码 网络公司源码 android游戏源码 莎莎源码 旅游网站源码下载 asp.net 源码 jsp源码 游戏源码网 网页源码 android源码 表白网站源码 淘宝客网站源码 棋牌源码论坛 易语言qq源码 表白网站源码大全 psd源码 易语言源码大全 卡盟源码带分站 狗扑源码 yy头像源码 淘宝客源码 51aspx源码 手机网站源码 love源码 易语言钓鱼源码 一元云购源码 一元夺宝源码 a5源码网 外星人源码 易语言游戏源码 苍老师高清种子源码 直播源码 云矿机源码分享 装修erp管理系统源码 vscode 源码解析 淘宝asp源码 seaweedfs源码分析 贪吃蛇大作战 ts 源码 陕西挖坑源码 全本书屋免费源码 数据结构java源码 淘客返现源码 网站登陆界面源码 视频学习网站源码 微信小程序源码可靠吗 999朵玫瑰花表白源码 linux fork源码 源码hostsir 友情链接平台 源码 易语言电话源码 源码 数码 商城 易语言 留言源码 淘宝公式源码 溯源码有假的吗 源码 编辑器控件 c 控件源码 小程序源码导入修改 下载站网站源码 dede网站源码 dnf辅助源码附带模块 rss 订阅 源码 qq钓鱼源码怎么用 p2p searcher源码 怎样打开asp源码 qq飞车抽奖辅助源码 t9输入法源码 qq粒子头像源码下载 svip源码论坛 vb源码 2000 仿刀塔传奇 源码 百度vb源码例子 次世代音频源码 仿92直播源码 安卓 电话本源码 天下通源码 如何更该html源码怎么 单页源码程序 淘宝上源码是什么 如何修改exe程序源码 商业进销存源码 软件源码破解 春哥源码邀请码 八位源码表示范围 天书奇谈 辅助 源码 楠秀直播源码 微信 小说 付费 源码 ddx多个周期指标源码 头像psd源码打包下载 童装网站源码 同城交友完美整站源码 网站群源码 网站证书编号查询源码 土豆播放器源码 坦克宝贝全套源码 树形菜单 源码 透明flash源码 十字绣 网站源码 图书销售源码 探花教程网源码下载 网站工具源码 手机psd源码 手机qq 源码 网页gps定位源码 微信消息恢复源码 视频聊天室源码asp 斗牛源码价格 荣耀原生java影视app源码 59pic美女图片源码 php人事管理系统源码 labview源码 易支付网站源码 dnf驱动源码 爱秀直播诱导源码 applem2引擎全套源码 小程序 分销商城源码 cocos2dx lua游戏源码 易语言窗口同步器源码 烈火雄心 gotv 源码 ionic 项目源码 fd源码教程 c p2p 源码 织梦旅游网站源码 foxtable erp源码 jar包源码查看工具 jsp框架源码 醉八仙登录器源码 jquery源码学习笔记 unity开发3d麻将源码 js天猫抓娃娃游戏源码 jdk6.0 源码 java 团购网 源码 lpk.dll源码 javaee.jar源码 flash源码短片下载 java网站 源码 dhcp server 源码 flv播放器源码下载 jsp api源码 html课程表源码 dnf箱子源码 永利娱乐系统源码搭建 ssh项目源码下载 jvm 源码下载 java在线支付源码 java实现发帖页面源码 java捕鱼达人免费源码 jpeg 源码 blog 租房网站源码下载 通达信盘中选股源码 jsp源码出现乱码 中国 商业 源码 征婚交友网站源码合买 google talk 源码 unity3d源码下载 统计语言模型 源码 rockplayer 源码 微信公众号asp源码安装 文件源码 重发软件源码 linux 源码解析 openerp 源码 nehe opengl 源码 中国源码基地 android java源码 文字直播源码 文件下载 源码 物业源码pb csdn qq炫舞卡八音 源码 ons交友源码 phpcms源码文件 ok168源码 代理网站源码 带网上充值功能 源码 安卓相册源码 仿猎聘网源码 八批psd头像源码 出售vb源码 爱购物电子商城源码 个人门户网站源码 安卓新闻客户端源码 高仿蘑菇街源码 底部异动k线源码 爱情黑页源码 超级病毒源码 搜猫x源码下载 淘宝首页 源码 美女 数字卡盟网站源码 天空源码社区 汕头招聘源码 办公app源码 头像 源码 皮肤 全站源码下载 输入法 源码 投资网源码 淘宝出售源码 淘宝相册 源码 深度网站导航源码 java easyui项目源码 20秀诱导直播源码 咸鱼源码 dsa善领源码 ghost远控源码详讲 android源码 打包下载 flapmmo源码 java 源码交易 flash网站整站源码xml h.264源码解析 asp企业源码html 燕窝溯源码 访问次数 apicloud demo源码 javah 源码 flash 源码 翻页 emoi源码 dedecms官方网站源码 hao123整站源码 flex 游戏 源码 ibos 源码 ios手机麻将源码 flash as3 源码下载 数字代币源码 centos源码包下载网站 jsp mvc 源码 java 高效获取url源码 flash留言板源码下载 h站源码共享 flash透明素材源码 jsp网站开发源码 整形美容网站源码 java源码 电子商务 ios 网易新闻 源码 ie 查看源码快捷键 j2me浏览器源码 万挂作坊全套源码 go9go源码 java web完整项目源码 java 地图源码 支撑压力线源码 网络祭祀网站源码 源码天空密码 php 心愿墙系统源码 竞猜 源码 java 中国餐饮技术网源码 网站分类目录 源码 乌龙学院 源码 h5酒店预订系统源码 opc client 源码 psd空间源码 qq三国源码 qq钓鱼软件源码.rar qq屋动态空间源码 直销源码修改 qq 乐园 源码 ps源码图 网上售书系统源码 制作dnf自动喊话源码 qq业务网站源码v3 运行jquery源码 ssh 项目源码 pdf to excel 源码 源码软件下载 python毕设源码 discuz 验证码 源码 移动端h5页面源码下载 网易彩票源码 门店展示 小程序源码 下吧源码 qq刷赞平台网站源码 拼团app源码下载 网站源码wgwzs 系统 订单 源码 微擎会员卡源码 寻源码 建站 网站格子店铺源码下载 源码库 开源中国 夜魅直播源码 网速测试源码 远程木马 源码 优酷 土豆 源码 图像 vb 源码 虚拟网店源码 物通网源码 微品秀 源码下载 dota类网页游戏源码 firefox 源码解析 javascript程序源码 flash宣传动画源码 flash3d游戏源码 g722 源码 gdal 源码 srtm java 8583 源码 flash图片源码 java flex源码下载 装修crm 源码 flash源码查看器 站长吧源码社区 dj网站源码代 e语言验证码源码 中国商业源码 sgl查看工具源码 doom3源码分析 flash引导页源码 filemon 源码 java calendar源码 delphi 应用教程 源码 c 标签打印源码 中国服装网 源码 网络爬虫 源码 xpay 个人收款 源码 在线祝福网站源码 炫 办卡网贷平台 源码 网页源码 改 asp网页 源码 逐浪cms源码 红包流水统计源码 网络公司asp源码 万挂作坊4.x源码 在线录制视频 源码 整人域名源码 php 短信平台 源码 ogre 源码 qq号源码 php天气源码下载 中国龙flash网站源码 python的exe源码分析 育儿网源码 带数据 qq空间皮肤3d源码下载 支撑线源码 直播8源码 qq上线远控源码 协同过滤算法源码 武林delphi源码 校园表白小程序源码 在线视频点播java源码 彩虹代刷网源码4.2 场地预约 php源码 struts2网站源码 android su源码 vb 商业程序源码 2018汪年全家福 源码 html5连连看源码 h5抓娃娃机源码 彩虹易支付app源码 java嗅探源码 html编辑器asp源码 gcov 开源 源码 idc 网站 源码 html页面显示源码 dnf秒拍卖源码 iphoneqq协议源码 flash源码下载网站 找你妹源码 e语言笔记加密源码 ie 10 察看源码 asp店铺源码免费下载 jsp源码库存管理 jetty 源码 下载 asp小游戏源码 java连接数据库源码 java sdk源码 flash带后台源码 html 静态页面源码 idc网站源码支付 ip地址查询 源码 iphoneqq在线软件源码 hibernate3源码下载 jvm源码下载 ftp工具上传网站源码 如何把安卓源码变成apk h5万人猜源码 个股指标公式源码 彩虹代刷网源码美化 动态源码语言排行 dedecms笑话 源码 windows锁屏截屏源码 flash请柬源码 eclipse源码下载 flash站源码 js树形结构源码 html 制作 源码 jsp 网上商城源码 新闻大数据爬取源码 ethtool 源码 ios 股票 源码 招聘职位源码 dnf后台喊话源码 dw聊天室源码 gtk 源码下载 iphone开发源码 android 2.2源码 web表白 程序源码知乎 图标秒点器易语言源码 之道返利源码 ons交友网站源码 psd空间 源码下载 qq网页登录源码 qq 网址 源码 qq解除限制钓鱼源码 php网盘程序源码 英文站b2b 源码 网站源码什么意思 qq秀设计网源码 ps怎么修改源码文字 世界杯足球竞猜源码 虚拟主机评测网源码 彩虹云任务源码7.1.8 小刀娱乐易源码 区块马 源码 微信强制分享群 源码 android源码 ios源码 android互动屏保源码 jsp oa源码 gh0st源码编译 易语言抢票源码 java 串口源码 google maps 源码 div css 源码 杂志迷 源码 java 源码包 网站源码在线翻译 flash文字效果源码 flash 导航源码 jsp oa办公系统源码 dj网站源码代收费功能 asp动网论坛源码 java io源码 java 彩信 源码 jdk1.5源码 html5砸金蛋源码 怎样获取天猫网页源码 最好的源码网 ps红色火焰源码 易拍网 源码 赞不停源码 mysql源码pdf minix 源码 txt mpeg2解码器源码 源码 html php建站源码下载 麻将游戏源码安卓 易语言 网页 源码 ssh开发源码 php webgame 源码 php登陆界面源码 qq酷淘宝客源码 php 源码 投票 struts2的源码 源码天空下载 android 超级画板源码 易语言 百度 源码 tcmalloc 源码分析 易语言基础源码 小猪cms源码2017 易语言语音说话源码 易语言超文本源码 倚天蝴蝶形态指标源码 易语言邮箱扫号的源码 易语言破解源码 易语言网页变速器源码 mybatis源码包下载 android10源码打包下载 zip源码 易语言修改源码教程 易语言客户登记源码 订单文件网站源码 仿58同镇源码 源码如何打包ipa 订货批发系统源码 小高教程网的源码 织梦采集侠源码解析 漫画小说源码带采集 淘宝搜索功能源码 源码售卖网 手机github下载源码 仿5173交易平台源码 传奇评测网源码下载 传奇源码如何泄露的 先锋影音资源码 sql仓库管理系统源码 海外源城系统平台源码 琪琪源码 栗子发卡网源码 机房运维管理系统源码 西游记张卫健tvb源码种子 新闻站源码自动采集 九转指标源码 源码直播免费 大商创源码安装 返利app原生前后端源码 下载 免费社区团购系统源码 图片社交平台源码 淘客导购cms源码 通达信发行价源码 刷脸支付系统源码 在线作图网站源码 区块链钱包系统开发源码 殷保华公式源码 变色龙源码 安卓源码助手 欧傅算命源码下载 黑阔站长吧源码 抄底主图指标公式源码 火速拍卖源码 html5魂斗罗源码 布艺网站源码 在线库app源码 beginthreadex源码 通达信买卖金点源码 建站公司企业源码 免费自助建站平台源码 医院管理信息系统源码 exe怎么查看源码 冒险岛ijl15.dll源码 黑色分类目录源码 模型展示源码 美女一对一加视频app源码 震荡波源码 微信三方平台源码 箱体指标源码 宝塔线公式源码附图 深入浅出mfc源码 实验室设备管理系统源码 云服务器编译安卓源码 ios马甲包老外源码库 flvjs源码 webpack的loader源码 嗨玩源码 刷脸saas系统源码 net源码是什么 qq业务乐园源码 ps动态qq头像源码 netants源码 php地方信息港源码 qq屋模版源码 oa 项目源码下载 制作聊天软件源码 qq业务 源码下载 php服装商城源码 装修网网站源码 linux 0.11源码下载 net 订货系统源码 多城市商城源码 我画你猜源码 tcpdump 源码分析 重庆婚嫁网最新源码 易语言电影菜单源码 md5解密源码php 易语言挂机脚本源码 易语言目录监视源码 易语言本地验证源码 shopex仿天猫商城源码 python源码剖析买不到 dba要研究数据库源码吗 块儿快递查询源码 微信qq机器人整站源码 无极任务平台源码 asp棋牌游戏源码 期货app源码 百度小程序支付回调源码 刮刮卡系统源码 党建网站源码 易语言给软件做激活码源码 算命取名源码 群友通讯录源码 程序员阅读源码 正版满堂彩源码 汽车之家网站源码 创盟整站源码下载 九艺棋牌源码 骑士人才源码 好看的html黑页源码 java怎么查看jar包的源码 商城源码吧 万词霸屏 源码 天使测试工具源码 传奇源码哪几种 impress源码 几年前的php源码 十秒竞拍源码下载 游戏源码及素材下载 ai人工智能主播 源码 ihucms导航网源码 虚拟路由源码 三货币对冲ea源码 霸屏天下源码修改 视频一对一交友源码 方块导航源码 飞狐抛物线公式破解源码 dz论坛资源源码 买了源码能做什么 影视博客源码 源码安装openssh h视频网站源码带裂变 qt彩票源码下载 高清播放器源码 仿百度图片搜索源码 pcm和源码 易语言163邮箱源码 simpledateformat源码 外贸手机网页源码 电影视频网站模板源码 比分源码 安卓客服系统源码 dk买卖点公式源码 1737棋牌源码 java2实用教程源码 notepad源码分析 高清电影下载源码 小夜网php卡盟源码 因酷源码 mysql多表查询源码 填大坑源码 彩虹url转发源码 乞讨网php源码 买卖交割的论坛源码 a5源码opensns 外贸导购源码 asp网站源码 查看 溯源码真伪 dnf怎么源码泄露的 av平台源码 电子地图源码 旅游项目源码 热血虎卫源码 大挂武林源码 快递单号查询系统源码 map源码分析 源码转换成补码的规则 zblog源码分享 php发邮件接口源码 广告平台源码下载 vb 销存源码access 空包单号网源码 voicechatter源码 cc2540串口透传源码 大番薯12人牛牛源码 安卓手机锁屏病毒源码 传奇引擎源码修改教程 linux 源码包 mysql 仿搜狐视频分享源码 tpshop微信小程序源码 服装门户 源码 短地址赚钱源码 微信免费影视小程序源码 易语言采集网页源码 刀疤兔源码 动力社区源码 动网博客源码 源码制作器 下载 茶叶网站源码下载 采购软件源码 飞狐公式源码 捕捞季节公式源码 飘花电影网源码 小鹏 网络摄像头直播源码 zhe800 源码 ws源码群 webgl源码下载 在线礼佛源码 vc项目源码包含引用 十三水源码 qince vb节假日提醒源码 食材配送源码 指定程序走代理 源码 apm飞控源码 小程序识别文字源码 易语言吃鸡源码 边缘源码 农业中子出售网站源码 新闻app 源码下载 云呼源码 php自动收录导航源码 2018世界杯竞猜源码 新闻聚合app完整源码 多城市家教平台源码 短线操盘手指标源码 单位主页网站源码 多功能记事本源码 大漠插件 源码 顶部线的公式源码 仿爱玩源码 国外flash片头源码 黑色音乐网站源码 久久信息网源码 简单的网址导航源码 婚庆 门户 源码 僵尸终结者 源码分析 建站大师 源码 留言本源码下载 极品源码社区 局域网通讯 源码 绿麻雀系统源码 流动字幕源码 大气拍卖公司网站源码 后盾网许愿墙源码 微信同城分类信息源码 three.js机房源码 全民k歌 解析 源码 物业收费系统源码 母婴小程序源码 google keep 源码 定位平台 源码 windowbuilder源码 win8网站源码 机器猫 拖拽 源码 记事本 源码 金字塔魔鬼通道源码 柳州房产网 源码 寒江独钓源码 即时通源码 简单导航网站源码 进进窗口化源码 换客 源码 汉字显示屏 源码 类似树洞外链的源码 绝版动态皮肤源码下载 集团首页 源码 竞拍系统 源码 开盘预警抓涨停 源码 接水管 游戏源码 关键字搜索的asp源码 黑域基地网站源码 慧眼识金指标源码 航海家 源码 寒龙源码 监控买卖指标源码 黑客360 qq刷钻源码 酒吧网站源码下载 酒店在线订房系统源码 集成搜索源码 侯杰stl源码剖析 家政公司源码 婚恋 在线聊天 源码 酷酷导航源码 酒店预定网站源码 卖源码网站 马士兵struts2源码 马甲 指标 源码 黑客源码网 绿色源码之 黑客之门 源码 军事门户源码下载 炫酷html表示源码 linux2.6内核源码注释 车位检测车牌识别源码 扫码挪车小程序 源码 arrsylist源码 帝隆p2p网贷系统源码 麻将胡牌算法源码 广告平台小程序源码 刺激战场源码在哪里找 workerman 游戏源码 cmake编译opencv源码 yy最新活动源码.zip 在线小说 网站源码 办公用品商城 源码 微赞微擎源码最新版 源码寻仙打怪辅助源码 赵鹏下吧 源码 vc 乐谱源码 centos 7 mysql 源码 黑马手机安全卫士源码 彩票门户源码 h5 翻牌抽奖源码 毕向东源码 安卓rom源码下载 仿qq游戏大厅源码 仿思道oa源码增强版 标题栏 源码 点卡源码 网站正在建设中 源码 多空风险指标源码 attention机制源码 vb串口源码下载 java web直播源码 vb源码 保存图像 vc 图形 源码 vb erp源码 vb ie 源码 分时博弈源码 商之翼yii2源码 python3.6源码下载 跑酷游戏源码完整版 采集电影网站源码 编辑 html 源码 采购申请入口系统源码 java类源码怎么看 布林带 源码 俄罗斯方块安卓源码 出售网址导航源码 源码啊 农业 信息网 源码 带源码sae建站 大盘指标 源码 附图 产品展示 php源码 批量修改网站源码 服务器源码如何 简单的游戏源码 莱整形医院源码 空间psd皮肤源码 家装后市场 源码 航海 至尊数据 源码 建站程序源码系统 家装php带源码下载 简易新闻发布源码 计算器 mfc 源码 换客中国源码 股票软件 delphi 源码 猴岛cf窗口化源码 华软源码网 灰色按钮克星源码 经过代理vb源码 竞拍网源码 机械类源码 论文网源码下载 相册 小程序 源码 eclipse关联maven源码 韩国源码下载站 竞价低开池初选源码 yy最新源码 垃圾源码 股票论坛 源码 卡哇猪电影网站源码 酷源码asp调试工具 科讯地方门户源码 空间psd源码 空间闪动标题栏源码 酷跑游戏源码完整版 卡盟源码哪个好 节日祝福源码 局域网 斗地主 源码 公司网站的源码 剑灵服务器emu源码 谷歌android 4.1源码 火爆的商机网整站源码 聊斋ol 源码 好吃猫外卖点餐源码 会计从业考试网站源码 垃圾x站换量用源码 红木家具代源码 喊话器源码 机房管理系统源码 就爱源码社区 黑客 指标源码 rtmp源码是什么 怪老头电脑源码 企业统一通信 源码 地方旅游网整站源码 白狼答题器源码 超级搜索引擎源码 淘宝客网站源码php 冲击波源码 十字消除的源码 淘宝卖家工具箱源码 三国跑酷源码 随意贴源码 旅行社综合业务管理系统源码 随机数生成器源码 威客整站源码 最新易企秀v13.0 源码 人才网源码 下载 网站整站源码下载免费 请销假系统h5源码 同城 聊天室 源码 压力线和支撑线的源码 iapp的源码用什么编译 通达信 波段王 源码 视频网站源码 jsp easyui tab 事件源码 世纪星平台源码 树型论坛源码 2017闲鱼源码吧 视频传输源码 正版彩虹代刷网源码 android gps定位源码 myzly二级域名分发源码 奇摩女孩 源码 网站自动化宣传机器源码 软件试用版 源码 共享单车 源码 xwork源码 yy钓鱼网站源码下载 vb导出导入excel源码 微信机器人投注源码 mac 源码安装mongodb vrml与java源码 vc数据库编程源码 vb.net下载软件源码 微信收款 源码 倒计时软件源码 避庄洗盘指标源码 天高云淡秒赞源码最新 盗取网站源码教程 动站源码下载 筹码监控指标源码 洞穴狂奔源码 查看java源码工具 钓鱼抽奖网页源码 砍价微信小程序源码 源码资本已投项目 求购房产网源码 中值滤波实现源码python 数据商城源码 日志签名档源码 软件营销网站源码 网狐内核源码 神将世界太史慈源码 神经猫源码 图书馆管理系统源码下载 失物招领app源码 小程序房贷计算器源码 ecshop车商城源码 android 足球源码 英语趣配音 php源码 易语言截图软件源码 易语言准心源码 源码能修改嘛 直播刮彩票源码 汽车uds诊断协议源码 android通讯软件源码 思途旅游cms源码 整人yy协议源码.zip 易语言源码编辑框右键 win8网址导航源码 缘来客网站源码出售 qq飞车公子辅助源码 安卓源码包下载 大游戏论坛源码 点卡销售系统源码 傲气战歌网网站源码 床上用品源码 安卓2.2源码 安装软件进度条源码 大麦户网站源码 炒股软件的源码 超级外链 源码 通达信 振幅源码 网 站源码 易企秀v12.7源码下载 网胜b2b 源码 同城交友网 源码 广州南站离白马服装城源码 外卖网源码 网址导航源码带推广 python源码统计爬虫 数字卡源码 贴吧查ip源码 视频聊天k歌源码php 散户至宝指标源码 apk源码调试 安卓手机点餐系统源码 准确出击选股指标源码 下载器源码 微信小程序 源码抓包 团购聚合源码 学成家教网源码 荒野行动方块透视源码 问道手游扫拍源码 winform综合案例源码 web ssm 项目源码 站长统计 源码 学生管理系统源码下载 兔子卡盟网站源码 小说随身听 源码 小姐 威客 源码 网上的源码后门多 源码之家 软件 下载 学校整站源码下载 怎么学 gcc源码 新闻文章系统源码 小游戏php源码 修改mac vb源码 小说源码下载带采集的 淘宝返利api接口源码 源码免杀入口点 在线图片制作源码 形象设计源码 spark1.5源码编译安装 吾爱挂q源码 微信独立商城源码 在线投注 源码下载 黑马网源码 flappy bird 源码 省钱快报源码 小京东淘宝客源码 asp报修系统源码 c winform 源码下载 c#验证码识别源码 快三源码手机版 夺宝源码下载 网页抓包工具源码 团购打折网源码 验证真假qq源码 源码 javamail 幸运28 源码 源码中含有非法链 修改头像源码文字 营销软件下载源码 营销单页源码 下载google源码 图片水印psd源码 新闻发布系统asp源码 小说采集 源码 虚拟摄像头源码 网站源码打包工具 兄弟变速器源码 星辰变 采集源码 微信墙源码免费下载 项目源码打包下载 推广联盟 源码 源码安装openerp 微赞最新源码 电玩源码破解 php 卡片机器人源码 战舰少女资源码 无限弹窗源码 鲜花预定源码下载 能连续涨停公式源码 微擎微赞视频打赏源码 小猪cms源码 2017破解