I followed the Percona guide to upgrade my XtraDB Cluster (MySQL + Galera) from 5.5 to 5.6, here are some observations of the process.
The upgrade process
I included putting the upgraded nodes into read only mode and I performed an additional step to reconfigure the xinetd process used to run clustercheck after installing the new binaries:
server_args = clustercheck <password> 0 /var/log/mysqlchk.log 0
This sets available when donor and available when read only to off, by default with 5.6 the SST method is Xtrabackup, but my 5.5 cluster had been setup with rsync, more on this later.
The first 2 nodes upgraded just fine, however when I restarted the 3rd node the entire cluster crashed. Restarting MySQL on all nodes got things back up and running again.
After upgrading
Everything was initially working fine, but I noticed some odd behaviour on one node, it was repeatedly crashing when I was purging old data from one of the databases. The node would fail to restart with an rsync error, only resolved when I performed a rolling restart of the other 2 nodes.
Switching from rsync to Xtrabackup
Rysnc clearly is not the best option for a 5.6 based cluster, it worked fine for the last 18 months with the 5.5 cluster but there are good reasons why Xtrabackup is the default option with 5.6. The main benefit that I can see is that Xtrabackup is non-blocking so the node can continue to accept connections during SST.
Xtrabackup requires a username/password to login to another host to perform the transfer, hence you have to create an account with permission to access all databases from each server in the cluster. I simply ran CREATE USER and GRANT statements on a single node and allowed the cluster to replicate the new accounts to the other nodes.
It's important that the datadir variable is defined in my.cnf, or Xtrabackup doesn't work.
A test can be carried out from the shell before updating my.cnf:
innobackupex --host=<another node> --user=<user> --pass=<password> /tmp/
Finally my.cnf can be updated:
wsrep_sst_method=xtrabackup-v2
wsrep_sst_auth=user:password
Checking the MySQL log should show SST now using XtraBackup.