I woud like to know how people are doing rolling upgrade of Casandra clustes when there is a change in native protocol version say from 2.1 to 3.11. During rolling upgrade, if client application is restarted on nodes, the client driver may first contact an upgraded Cassandra node with v4 and permanently mark all old Casandra nodes on v3 as down. This may lead to request failures. Datastax recommends two ways to deal with this:
1. Before upgrade, set protocol version to lower protocol version. And move to higher version once entire cluster is upgraded.
2. Make sure driver only contacts upraded Cassandra nodes during rolling upgrade.
Second workaround will lead to failures as you may not be able to meet required consistency for some time.
Lets consider first workaround. Now imagine an application where protocol version is not configurable and code uses default protocol version. You can not apply first workaroud because you have to upgrade your application on all nodes to first make the protocol version configurable. How would you upgrade such a cluster without downtime? Thoughts?