MHA GTID based failover代码解析

作为以下文章的补充,说明MHA GTID based failover的处理流程。

MHA判断是GTID based
failover需要满足下面3个条件(参考函数get_gtid_status)
所有节点gtid_mode=1
所有节点Executed_Gtid_Set不为空
至少一个节点Auto_Position=1

GTID basedMHA故障切换

  1. MHA::MasterFailover::main()
  2. ->do_master_failover
  3. Phase 1: Configuration Check Phase
  4. -> check_settings:
  5. check_node_version:查看MHA的版本信息
  6. connect_all_and_read_server_status:确认各个node的MySQL实例是否可以连接
  7. get_dead_servers/get_alive_servers/get_alive_slaves:double
    check各个node的死活状态
  8. start_sql_threads_if:查看Slave_SQL_Running是否为Yes,若不是则启动SQL
    thread
  9. Phase 2: Dead Master Shutdown
    Phase:对于我们来说,唯一的作用就是stop IO thread
  10. -> force_shutdown($dead_master):
  11. stop_io_thread:所有slave的IO thread stop掉(将stop掉master)
  12. force_shutdown_internal(实际上就是执行配置文件中的master_ip_failover_script/shutdown_script,若无则不执行):
  13. master_ip_failover_script:如果设置了VIP,则首先切换VIP
  14. shutdown_script:如果设置了shutdown脚本,则执行
  15. Phase 3: Master Recovery Phase
  16. -> Phase 3.1: Getting Latest Slaves Phase(取得latest slave)
  17. read_slave_status:取得各个slave的binlog file/position
  18. check_slave_status:调用”SHOW SLAVE
    STATUS”来取得slave的如下信息:
  19. Slave_IO_State, Master_Host,
  20. Master_Port, Master_User,
  21. Slave_IO_Running, Slave_SQL_Running,
  22. Master_Log_File, Read_Master_Log_Pos,
  23. Relay_Master_Log_File, Last_Errno,
  24. Last_Error, Exec_Master_Log_Pos,
  25. Relay_Log_File, Relay_Log_Pos,
  26. Seconds_Behind_Master, Retrieved_Gtid_Set,
  27. Executed_Gtid_Set, Auto_Position
  28. Replicate_Do_DB, Replicate_Ignore_DB, Replicate_Do_Table,
  29. Replicate_Ignore_Table, Replicate_Wild_Do_Table,
  30. Replicate_Wild_Ignore_Table
  31. identify_latest_slaves:
  32. 通过比较各个slave中的Master_Log_File/Read_Master_Log_Pos,来找到latest的slave
  33. identify_oldest_slaves:
  34. 通过比较各个slave中的Master_Log_File/Read_Master_Log_Pos,来找到oldest的slave
  35. -> Phase 3.2: Determining New Master Phase
  36. get_most_advanced_latest_slave:找到(Relay_Master_Log_File,Exec_Master_Log_Pos)最靠前的Slave
  37. select_new_master:选出新的master节点
  38. If preferred node is specified, one of active preferred nodes will
    be new master.
  39. If the latest server behinds too much (i.e. stopping sql thread for
    online backups),
  40. we should not use it as a new master, we should fetch relay log
    there. Even though preferred
  41. master is configured, it does not become a master if it’s far
    behind.
    get_candidate_masters:
    就是配置文件中配置了candidate_master>0的节点
    get_bad_candidate_masters:
    # The following servers can not be master:
    # – dead servers
    # – Set no_master in conf files (i.e. DR servers)
    # – log_bin is disabled
    # – Major version is not the oldest
    # – too much replication delay(slave与master的binlog
    position差距大于100000000)
    Searching from candidate_master slaves which have received the
    latest relay log events
    if NOT FOUND:
    Searching from all candidate_master slaves
    if NOT FOUND:
    Searching from all slaves which have received the latest relay log
    events
    if NOT FOUND:
    Searching from all slaves

    -> Phase 3.3: Phase 3.3: New Master Recovery Phase
    recover_master_gtid_internal:
    wait_until_relay_log_applied
    stop_slave
    如果new master不是拥有最新relay的Slave
    $latest_slave->wait_until_relay_log_applied:等待直到最新relay的Slave上Exec_Master_Log_Pos等于Read_Master_Log_Pos
    change_master_and_start_slave( $target, $latest_slave)
    wait_until_in_sync( $target, $latest_slave )
    save_from_binlog_server:
    遍历所有binary server,执行save_binary_logs
    –command=save获取后面的binlog
    apply_binlog_to_master:
    应用从binary server上获取的binlog(如果有的话)
    如果设置了master_ip_failover_script,调用$master_ip_failover_script
    –command=start进行启用vip
    如果未设置skip_disable_read_only,设置read_only=0

    Phase 4: Slaves Recovery Phase
    recover_slaves_gtid_internal
    -> Phase 4.1: Starting Slaves in parallel
    对所有Slave执行change_master_and_start_slave
    如果设置了wait_until_gtid_in_sync,通过”SELECT
    WAIT_UNTIL_SQL_THREAD_AFTER_GTIDS(?,0)”等待Slave数据同步

    Phase 5: New master cleanup phase
    reset_slave_on_new_master
    清理New Master其实就是重置slave
    info,即取消原来的Slave信息。至此整个Master故障切换过程完成

启用GTID时的在线切换流程和不启用GTID时一样(唯一不同的是执行的change
master语句),所以省略。

GTID based failover代码解析
作为以下文章的补充,说明MHA GTID based failover的处理流程。

MHA判断是…

前段时间我的同事沈龙星整理了一下MHA故障切换和在线切换的代码流程,在征得其同意后,在此转发。以下是正文

本文是以MySQL5.5为基础的,因此没有涉及到gtid相关内容。MHA的主从切换过程分为failover和rotate两种,前者适用于原Master
down的情况,后者是在在线切换的情况下使用。下面分别讲解

rotate的处理过程

  1. MHA::MasterRotate::main()
    -> do_master_online_switch:
    Phase 1: Configuration Check Phase
    -> identify_orig_master
    connect_all_and_read_server_status:
    connect_check:首先进行connect
    check,确保各个server的MySQL服务都正常
    connect_and_get_status:获取MySQL实例的server_id/mysql_version/log_bin..等信息
    这一步还有一个重要的作用,是获取当前的master节点。通过执行show slave
    status,
    如果输出为空,说明当前节点是master节点。
    validate_current_master:取得master节点的信息,并判断配置的正确性
    check是否有server down,若有则退出rotate
    check master alive or not,若dead则退出rotate
    check_repl_priv:
    查看用户是否有replication的权限
    获取monitor_advisory_lock,以保证当前没有其他的monitor进程在master上运行
    执行:SELECT GET_LOCK(‘MHA_Master_High_Availability_Monitor’,
    ?) AS Value
    获取failover_advisory_lock,以保证当前没有其他的failover进程在slave上运行
    执行:SELECT GET_LOCK(‘MHA_Master_High_Availability_Failover’,
    ?) AS Value
    check_replication_health:
    执行:SHOW SLAVE
    STATUS来判断如下状态:current_slave_position/has_replication_problem
    其中,has_replication_problem具体check如下内容:IO线程/SQL线程/Seconds_Behind_Master(1s)
    get_running_update_threads:
    使用show
    processlist来查询当前有没有执行update的线程存在,若有则退出switch
    -> identify_new_master
    set_latest_slaves:当前的slave节点都是latest slave
    select_new_master:选出新的master节点
    If preferred node is specified, one of active preferred nodes will
    be new master.
    If the latest server behinds too much (i.e. stopping sql thread for
    online backups),
    we should not use it as a new master, we should fetch relay log
    there. Even though preferred
    master is configured, it does not become a master if it’s far
    behind.
    get_candidate_masters:
    就是配置文件中配置了candidate_master>0的节点
    get_bad_candidate_masters:
    # The following servers can not be master:
    # – dead servers
    # – Set no_master in conf files (i.e. DR servers)
    # – log_bin is disabled
    # – Major version is not the oldest
    # – too much replication delay(slave与master的binlog
    position差距大于100000000)
    Searching from candidate_master slaves which have received the
    latest relay log events
    if NOT FOUND:
    Searching from all candidate_master slaves
    if NOT FOUND:
    Searching from all slaves which have received the latest relay log
    events
    if NOT FOUND:
    Searching from all slaves

    Phase 2: Rejecting updates Phase
    reject_update:lock table来reject write binlog
    如果MHA的配置文件中设置了”master_ip_online_change_script”参数,则执行该脚本来disable
    writes on the current master
    该脚本在使用了vip的时候才需要设置
    reconnect:确保当前与master的连接正常
    lock_all_tables:执行FLUSH TABLES WITH READ LOCK,来lock table
    check_binlog_stop:连续两次show master
    status,来判断写binlog是否已经停止

    read_slave_status:
    get_alive_slaves:
    check_slave_status:调用”SHOW SLAVE
    STATUS”来取得slave的如下信息:
    Slave_IO_State, Master_Host,
    Master_Port, Master_User,
    Slave_IO_Running, Slave_SQL_Running,
    Master_Log_欧博国际网站,File, Read_Master_Log_Pos,
    Relay_Master_Log_File, Last_Errno,
    Last_Error, Exec_Master_Log_Pos,
    Relay_Log_File, Relay_Log_Pos,
    Seconds_Behind_Master, Retrieved_Gtid_Set,
    Executed_Gtid_Set, Auto_Position
    Replicate_Do_DB, Replicate_Ignore_DB, Replicate_Do_Table,
    Replicate_Ignore_Table, Replicate_Wild_Do_Table,
    欧博国际平台,Replicate_Wild_Ignore_Table
    switch_master:
    switch_master_internal:
    master_pos_wait:调用select
    master_pos_wait函数,等待主从同步完成
    get_new_master_binlog_position:执行’show master status’
    Allow write access on the new master:
    调用master_ip_online_change_script –command=start
    …,将vip指向new master
    disable_read_only:
    在新master上执行:SET GLOBAL read_only=0
    switch_slaves:
    switch_slaves_internal:
    change_master_and_start_slave
    change_master:
    start_slave:
    unlock_tables:在orig master上执行unlock table
    Phase 5: New master cleanup phase
    reset_slave_on_new_master
    release_failover_advisory_lock

前段时间我的同事沈龙星整理了一下MHA故障切换和在线切换的代码流程,在征得其同意后,在此转发。以…

Author

发表评论

电子邮件地址不会被公开。 必填项已用*标注