7/15/20

Distributed Locks are Dead; Long Live Distributed Locks!

“Distributed locks aren’t real”, some like to remind us. “Anyone who’s trying to sell you a distributed lock is selling you sawdust and lies.” This may sound rather bleak, but it doesn’t say that locking itself is impossible in a distributed system: it’s just that all of the system’s components must participate in the protocol. This blog post is the story of how we implemented a distributed locking protocol that gives your components a straightforward way of joining in.
In line with Hazelcast’s tradition, the coordinating component of the protocol is an object that extends the semantics of java.util.concurrent.locks.Lock. We call it FencedLock, following the naming used in Martin Kleppmann’s 2016 post “How to Do Distributed Locking”.
You can also check out my follow-up blog post about how we tested the CP Subsystem APIs, including FencedLock, with Jepsen.
For the impatient reader, here are the takeaways of this blog post:
  • FencedLock is a linearizable distributed implementation of the java.util.concurrent.locks.Lock interface with well-defined execution and failure semantics. It can be used for both coarse-grained and fine-grained locking.
  • FencedLock replicates its state over a group of Hazelcast members via the Raft consensus algorithm. It is not vulnerable to split-brain problems.
  • FencedLock tracks liveness of lock holders via a session mechanism that works in a unified manner for both Hazelcast servers and clients.
  • FencedLock allows 3rd-party systems to participate in the locking protocol and achieve mutual exclusion for the side-effects performed on them. This is the “fenced” part of the story.
  • FencedLock is battle-tested with an extensive Jepsen test suite. We have been testing its non-reentrant and reentrant behavior, as well as the monotonicity of the fencing tokens. To the best of our knowledge, FencedLock is the first open source distributed lock implementation that is tested with such a comprehensive approach.
Concurrency and nondeterminism are the main reasons that make multithreaded programming difficult. When there are multiple threads of execution, even a simple algorithm can produce a different result on every new run. Conceptually, distributed applications are not much different from multithreaded applications, so they need to deal with concurrency and nondeterminism as well. Distributed applications contain another underlying complexity that does not exist in multithreaded applications: partial failures. A distributed application is expected to keep running while some of its components have failed.
We can capitalize on this conceptual similarity to make concurrency primitives useful in distributed applications. Take JDK’s java.util.concurrent.locks.Lock interface for example. It defines an API to coordinate access to a shared resource by multiple threads. Lock implementations can either enforce mutual exclusion to preserve correctness or efficiency, or allow concurrent access to a shared resource in a controlled manner, like ReadWriteLock. Moreover, the Lock interface is general enough to enable a distributed implementation where lock-acquiring threads live in different processes and/or the lock state is replicated over multiple processes.
Hazelcast IMDG 3.12 introduces a linearizable distributed implementation of the java.util.concurrent.locks.Lock interface in its CP Subsystem: FencedLock. It is efficient for both coarse-grained and fine-grained locking. You can use the monotonic fencing tokens provided by FencedLock to achieve mutual exclusion across multiple threads that live in different processes. In this blog post, you’ll discover how we extended the semantics of the Lock interface for distributed execution and covered several failure modes that we can face in a distributed setting.
If you aren’t yet familiar with Hazelcast’s CP Subsystem, you can read our CP Subsystem primer. One basic fact about it is this: a Hazelcast cluster may have not one, but several subclusters that we call CP groups. Each CP group is a cluster on its own with respect to the CP services it provides.
The code samples in this blog post start one JVM and form a three-member CP group within it. We do this strictly for simplicity because the members still communicate over the network and aren’t even aware that they are sharing the JVM. Their behavior is equivalent to each Hazelcast member running concurrently on its own JVM and the events occur in the order shown in the code samples. All code samples shown in this blog post are also placed in our code samples repository.

Basic Semantics

Let’s start with a basic example and proceed by solving one problem at a time. Here’s all the code you need to write to begin using a FencedLock:
Config config = new Config();
config.getCPSubsystemConfig().setCPMemberCount(3);

// Create 3 Hazelcast server instances, they will automatically form the CP Subsystem:
HazelcastInstance hz1 = Hazelcast.newHazelcastInstance(config);
HazelcastInstance hz2 = Hazelcast.newHazelcastInstance(config);
HazelcastInstance hz3 = Hazelcast.newHazelcastInstance(config);

// Obtain a handle to the same lock from two different HZ instances:
FencedLock hz1Lock = hz1.getCPSubsystem().getLock("my-lock");
FencedLock hz2Lock = hz2.getCPSubsystem().getLock("my-lock");

// Acquire the lock on instance 1:
hz1Lock.lock();

// Fail to acquire the lock on instance 2:
boolean lockedByHz2 = hz2Lock.tryLock();
assert !lockedByHz2;

// Release the lock on instance 1:
hz1Lock.unlock();

// Now Instance 2 can acquire the lock:
lockedByHz2 = hz2Lock.tryLock();
assert lockedByHz2;
In a nutshell,
  1. Instance One acquires the lock
  2. Instance Two fails to acquire the lock
  3. Instance One releases the lock
  4. Instance Two acquires the lock
We can conclude that, once a Hazelcast instance has acquired the lock, no other instance can acquire it until the holder explicitly releases it (or the system does it after the holder fails – see below for details). The instance can be either a server or a client instance; it works the same way.
FencedLock is reentrant by default:
// Acquire the lock on instance 1:
hz1Lock.lock();

// Acquire the lock again (succeeds):
boolean heldByHz1 = hz1Lock.tryLock();
assert heldByHz1;

// Release the lock once:
hz1Lock.unlock();

// Try to acquire from instance 2 (fails):
boolean heldByHz2 = hz2Lock.tryLock();
assert !heldByHz2;

// Release the lock from instance 1 again:
hz1Lock.unlock();

// Now instance 2 can acquire it:
heldByHz2 = hz2Lock.tryLock();
assert heldByHz2;
If you try to re-acquire the lock from the same instance, but on a different thread, you won’t be allowed to:
// Acquire the lock on instance 1:
hz1Lock.lock();

// Ask for the lock again but on another thread (fails):
new Thread(() -> {
    boolean acquired = hz1Lock.tryLock();
    assert !acquired;
}).start();
This behavior is vital to the lock behaving the way you expect: protecting a single critical section at a time.
Java’s Lock interface does not enforce a particular approach to reentrancy. Given this freedom, FencedLock offers configurable behavior and is reentrant by default, as shown above. You can disable reentrancy and use FencedLock as a non-reentrant mutex or set a custom reentrancy limit. When the configured reentrancy limit is reached, further lock acquire attempts fail with LockAcquireLimitReachedException.
FencedLock also offers a set of utility methods to query its status. You can check whether the lock is held and how many times it was reentrantly acquired (this fact is also committed to the CP group). You can query the lock-acquire count from anywhere, not necessarily from the lock-holding Hazelcast member and thread. Just keep in mind that you can use these facts only in an informative fashion – you can’t rely on them for correctness because you’d be committing the check-then-act fallacy. For example, between the time you get the response “you have the lock” and the time you take some action on the assumption you have the lock, you may already have lost it.
While we’re on this subject, the same logic applies even to the primary FencedLock.lock() call: at the very next line of code in your program, you may no longer be holding the lock. This is what the fencing token is for and we discuss it later on in this post.
FencedLock provides fairness. If a lock() request from caller1 is committed before another lock() request from caller2, caller1 wins.
We pretty much covered the most relevant parts of FencedLock’s distributed execution semantics. Let’s dive a bit deeper and investigate how FencedLock deals with failures. I have to admit, this is my favorite part!

Server-Side Failures

The good thing about using the Raft consensus algorithm under the hood is not having to worry about the consistency of the lock state. Each operation on a FencedLock gets committed to the majority of the CP group and each change in the lock state is guaranteed to be persisted as long as the CP group majority is alive. This means our FencedLock is invulnerable to split-brain problems. It preserves linearizability in case of network partitions and remains available on the side where the majority of the CP group is present.
FencedLock additionally achieves exactly-once semantics thanks to the idempotent implementations of its internal operations. For instance, even if the implementation retries a lock() operation because of a failed Raft leader, the lock is still acquired just once. The same rule applies to the other methods in the API.
This is a very useful guarantee because CP member failures can create a bit of trouble for in-flight operations. Say the Raft leader commits a lock-acquire request but then fails before sending the response to the caller. If we present the failure directly to the caller, how can it know whether the leader committed the request or not? In the reentrant mode, a naive retry performed by the client-side proxy would increment the acquisition count again and then a subsequent unlock() wouldn’t release it. In the non-reentrant mode, if the first lock() request acquired the lock, the retried one would fail with LockAcquireLimitReachedException.
To avoid these pitfalls, we built the client side of the FencedLock to retry operations internally in an idempotent manner, providing the user with an end-to-end exactly-once execution guarantee.

Client-side Failures

What Happens if a Lock Holder Dies?

We rely on the mechanics of the Raft consensus algorithm and the idempotence technique to tolerate failures of CP members, i.e., replicas of the lock state. However, there is another critical participant: our client, the lock holder! What if a caller acquires a FencedLock and then crashes (or experiences a long-lasting hiccup) while still holding it? If we don’t perform some cleanup, the entire application will suffer because the lock remains forever unavailable. This situation is equivalent to a Java program that calls Thread.suspend() or Thread.destroy() on a thread that is holding a lock. The Java designers got the easy way out: they simply banned thread suspension and destruction. Unfortunately, in the distributed world this is an unavoidable fact of life.
In order to deal with client failures, we introduced a mechanism to track the liveness of Hazelcast servers and clients in a unified manner: CP Sessions. We use this mechanism in the CP data structures that manage resource ownership; currently, there are two: FencedLock and ISemaphore. We create a new CP session when a Hazelcast server or client makes its very first lock or semaphore acquire request. After that, all lock and semaphore requests of the caller are associated with this CP session. Note that a Hazelcast server or a client maintains only a single CP session for a given CP group. Even if it is interacting with dozens of FencedLock instances in the same CP group, they will be all associated with a single CP session.
We keep the CP session alive for some time after we committed the last operation associated with it. If a client doesn’t perform any operations on its own but is otherwise live, its proxy implementation will automatically send periodic heartbeat operations. If the client dies or has a very long hiccup, the session times out. We close it, release all the associated locks and semaphore permits, and cancel all pending requests.
Let’s see the CP session auto-cleanup mechanism in action:
Config config = new Config();
CPSubsystemConfig cpSubsystemConfig = config.getCPSubsystemConfig();
cpSubsystemConfig.setCPMemberCount(3);
cpSubsystemConfig.setSessionHeartbeatIntervalSeconds(1);
cpSubsystemConfig.setSessionTimeToLiveSeconds(10);
HazelcastInstance hz1 = Hazelcast.newHazelcastInstance(config);
HazelcastInstance hz2 = Hazelcast.newHazelcastInstance(config);
HazelcastInstance hz3 = Hazelcast.newHazelcastInstance(config);

// Hazelcast member One acquires the lock
hz1.getCPSubsystem().getLock("my-lock").lock();

// Member One crashes. After some time, the lock 
// will be auto-released due to missing CP session heartbeats:
hz1.getLifecycleService().terminate();
FencedLock lock = hz2.getCPSubsystem().getLock("my-lock");
while (lock.isLocked()) {
    Thread.sleep(TimeUnit.SECONDS.toMillis(1));
    System.out.println("Waiting for auto-release of the lock...");
}

System.out.println("The lock was automatically released");
In this code sample, a Hazelcast member acquires the lock and then crashes. The rest of the CP group realizes the configured timeout has passed without any operations from that member and closes its session. This causes my_lock to be released.
The CP Subsystem also offers the ability to destroy a CP session manually. You can use this when you have the means to find out directly that a CP session owner crashed.
Config config = new Config();
CPSubsystemConfig cpSubsystemConfig = config.getCPSubsystemConfig();
cpSubsystemConfig.setCPMemberCount(3);
HazelcastInstance hz1 = Hazelcast.newHazelcastInstance(config);
HazelcastInstance hz2 = Hazelcast.newHazelcastInstance(config);
HazelcastInstance hz3 = Hazelcast.newHazelcastInstance(config);

hz1.getCPSubsystem().getLock("my-lock").lock();
// The lock holding Hazelcast instance crashes..
hz1.getLifecycleService().terminate();

CPSessionManagementService sessionManagementService = hz2.getCPSubsystem().getCPSessionManagementService();
Collection sessions = sessionManagementService.getAllSessions(CPGroup.DEFAULT_GROUP_NAME).get();
// There is only one active session and it belongs to the first instance
assert sessions.size() == 1;
CPSession session = sessions.iterator().next();
// We know that the lock holding instance is crashed.
// We are closing its session forcefully, hence releasing the lock...
sessionManagementService.forceCloseSession(CPGroup.DEFAULT_GROUP_NAME, session.id()).get();

FencedLock lock = hz2.getCPSubsystem().getLock("my-lock");
assert !lock.isLocked();
Finally, a Hazelcast instance (both server and client) automatically closes its CP session on graceful shutdown:
Config config = new Config();
CPSubsystemConfig cpSubsystemConfig = config.getCPSubsystemConfig();
cpSubsystemConfig.setCPMemberCount(3);
HazelcastInstance hz1 = Hazelcast.newHazelcastInstance(config);
HazelcastInstance hz2 = Hazelcast.newHazelcastInstance(config);
HazelcastInstance hz3 = Hazelcast.newHazelcastInstance(config);

hz1.getCPSubsystem().getLock("my-lock").lock();
hz1.shutdown();

FencedLock lock = hz2.getCPSubsystem().getLock("my-lock");
assert !lock.isLocked();
To be precise, the CP Subsystem resets the CP session timeout in 3 circumstances:
  • When the session owner performs a FencedLock or ISemaphore operation.
  • When the session owner sends a periodic heartbeat.
  • When the current Raft leader of the CP group crashes and a new leader is elected. This must be done in order to give the session owner time to discover the new Raft leader and start sending operations/heartbeats to it.
As veteran distributed systems folks might have already noticed, we borrowed several ideas from Google’s paper “The Chubby lock service for loosely-coupled distributed system”. We believe that the semantics of CP sessions are easy to grasp.

What if a Lock Holder Plays Dead, But Isn’t?

In an asynchronous system, the locking service cannot ensure that only one process thinks it holds the lock because there is no way to distinguish between a slow and a crashed process. Consider a scenario where a Hazelcast client acquires a FencedLock, then hits a long GC pause or becomes partitioned from the cluster. Since it will not be able to commit session heartbeats in the meantime, its CP session will be eventually closed and another Hazelcast client can acquire this lock. If the first client wakes up again or reconnects to the cluster, it may not immediately notice that it has lost ownership of the lock. In this case, multiple clients think they hold the lock. If they attempt to operate on a shared resource, they can break the system. Even if the first client actually crashes in this scenario, requests sent by 2 clients can be reordered in the network and hit the external resource in reverse order.
To prevent such situations, you can choose to use a long or maybe infinite CP session time-to-live duration, but then you’ll get liveness issues. CP sessions offer a trade-off between liveness and safety. Long story short, no timing assumption can be 100% reliable in asynchronous networks.
We offer a fencing mechanism to deal with these situations and achieve mutual exclusion across the whole system without sacrificing liveliness. Namely, we order lock holders by monotonic fencing tokens. Each FencedLock instance contains a fencing token which is incremented each time the lock instance moves from the available state to the locked state. Before attempting any side-effectful actions, the lock holder must pass this fencing token to all external services it will use, thereby fencing off previous lock holders. It must also include the token in every request. The service must persist the largest observed fencing token and reject any incoming request whose fencing token is less than the current one. This approach ensures that no two lock holders can interact with the service concurrently.
Figure 1 below illustrates this idea. In the beginning, Client-1 acquires the lock and receives 1 as its fencing token. It passes this token to the external services. Just after that, Client-1 hits a long GC pause and eventually loses ownership of the lock because it isn’t committing CP session heartbeats. Then, Client-2 comes in and acquires the lock, receiving 2 as the fencing token.
Now the most interesting thing happens. Before Client-2 had the chance to propagate its fencing token to the services, Client-1 wakes up and manages to use its token to execute a request against the service, even though it’s no longer the owner of the lock. This shows us that the fact “Client-N owns the lock” is not linearizable across the whole system (the combination of all the actors: the FencedLock, the clients, and the external services). The services learn the fact indirectly, through the clients’ explicit propagation of the fencing token. Nevertheless, mutual exclusion on the external services is not jeopardized as long as the clients obey the rules and propagate their fencing tokens before taking any side-effectful actions.
Figure 1: Using fencing tokens to fence off stale lock holders
Figure 1: Using fencing tokens to fence off stale lock holders
Note the key message here: all external services must participate in the fencing-token protocol, with guaranteed linearizability, for the whole setup to uphold its invariants. You can check Martin Kleppmann’s “How to do Distributed Locking” blog post to learn more about the fencing token idea. Google Chubby also implements a very similar solution, which they call “sequence numbers”.
We have a code sample that demonstrates how to utilize fencing tokens when talking to external services in our code samples repository. It’s too long to show here.
Lock holders can query the FencedLock API for their fencing tokens. In the following code sample, the first Hazelcast member acquires the lock and then asks for the fencing token by calling FencedLock.getFence(). Then, the second Hazelcast member acquires the lock. Its fencing token is greater than the fencing token assigned to the first member. Note that fencing tokens are incremented only when the lock moves from the available state to the held state. Hence, the subsequent reentrant lock acquire of the second member returns the same fencing token.
Config config = new Config();
CPSubsystemConfig cpSubsystemConfig = config.getCPSubsystemConfig();
cpSubsystemConfig.setCPMemberCount(3);
HazelcastInstance hz1 = Hazelcast.newHazelcastInstance(config);
HazelcastInstance hz2 = Hazelcast.newHazelcastInstance(config);
HazelcastInstance hz3 = Hazelcast.newHazelcastInstance(config);

// The lock switches from the available state to the held state.
FencedLock hz1Lock = hz1.getCPSubsystem().getLock("my-lock");
hz1Lock.lock();
long fence1 = hz1Lock.getFence();
hz1Lock.unlock();

// The lock switches from the available state to the held state.
FencedLock hz2Lock = hz2.getCPSubsystem().getLock("my-lock");
hz2Lock.lock();
long fence2 = hz2Lock.getFence();

assert fence2 > fence1;

// The lock is already held by the second instance.
// Making a reentrant lock acquire.
hz2Lock.lock();
long fence3 = hz2Lock.getFence();

assert fence3 == fence2;

hz2Lock.unlock();
hz2Lock.unlock();

// The lock switches from the available state to the held state.
hz2Lock.lock();
long fence4 = hz2Lock.getFence();
hz2Lock.unlock();

assert fence4 > fence3;
You can use FencedLock.lockAndGetFence() and FencedLock.tryLockAndGetFence() methods to avoid repeated patterns of lock.lock(); lock.getFence(); chains.
There’s another subtlety related to reentrancy and the possibility of losing your lock in a hiccup. At the point your code re-acquires the lock, it may have lost it beforehand, so it doesn’t actually re-acquire it, but gets it afresh. Your critical section was broken and other clients did their actions while you were gone, but everything appears to be going on as planned. The external systems agree that you are the lock holder and you proceed on the assumption that no other requests but your own got through. Eventually your code will try to release the lock twice and get an exception on the second release, but at that point the damage has already been done.
FencedLock implements a further mechanism that prevents this from happening: it makes the lock holder aware that it has lost ownership of the lock the next time it interacts with the FencedLock proxy. In Figure 2, Client-1 acquires the lock and hits a long GC pause. It loses ownership of the lock after some time due to the failure to commit any heartbeat operations. Then, Client-2 acquires and releases the lock while Client-1 is still frozen. Eventually, Client-1 comes back alive and tries to reentrantly acquire the lock for the second time, thinking it still holds it. Now FencedLock throws LockOwnershipLostException and stops it from proceeding with the happy path.
You can use the same mechanism even without re-acquiring the lock, as an optimization: you can find out at any point in the code that you lost the lock, before trying to access the external system.
Figure 2: Failing stale lock holders
Figure 2: Failing stale lock holders
We’ll demonstrate the failure with LockOwnershipLostException on an even simpler scenario. In the following code sample, we forcefully close the CP session of the lock-holding Hazelcast member as if its session was closed due to missing heartbeats. Then, the next lock() call of the kicked-out lock holder fails with LockOwnershipLostException. This scenario could occur when a Hazelcast member comes back alive after it was assumed to be crashed and its CP sessions were forcefully closed. Note that LockOwnershipLostException is still thrown even if the lock is not acquired by anyone else after it is forcefully released in the CP group.
Config config = new Config();
CPSubsystemConfig cpSubsystemConfig = config.getCPSubsystemConfig();
cpSubsystemConfig.setCPMemberCount(3);
HazelcastInstance hz1 = Hazelcast.newHazelcastInstance(config);
HazelcastInstance hz2 = Hazelcast.newHazelcastInstance(config);
HazelcastInstance hz3 = Hazelcast.newHazelcastInstance(config);

FencedLock lock = hz1.getCPSubsystem().getLock("my-lock");
lock.lock();

CPSessionManagementService sessionManagementService = hz2.getCPSubsystem().getCPSessionManagementService();
Collection sessions = sessionManagementService.getAllSessions(CPGroup.DEFAULT_GROUP_NAME).get();

assert sessions.size() == 1;
CPSession session = sessions.iterator().next();
// There is only one active session and it belongs to the first instance.
// We are closing its session forcefully to mimic that
// its session was closed because of missing session heartbeats...
sessionManagementService.forceCloseSession(CPGroup.DEFAULT_GROUP_NAME, session.id()).get();

try {
    // The new lock acquire call of the lock holder
    // fails with LockOwnershipLostException
    lock.lock();
    assert false;
} catch (LockOwnershipLostException expected) {
}

Testing FencedLock with Jepsen

If there is anything harder than building distributed systems, it is validating their behavior. Jepsen has proven to be a very successful tool in this endeavor and became the standard tool to test the safety of distributed systems. It subjects the system to many kinds of failures while running a test case and verifies that the system keeps its promises.
We have been using Jepsen to test the correctness of our new linearizable java.util.concurrent.* implementations under network partition failures. In fact, we followed a process we call “Jepsen-test-driven development”. We wrote our Jepsen tests first. Then, we ran those tests against the implementations to see how they behave and where they fail. After each iteration, we improved both the behavior and reliability of the implementations and then made another test run. This approach has been very fruitful to enable our implementations to preserve safety and remain useful for several edge cases. To the best of our knowledge, FencedLock is the first open source distributed lock implementation that is tested with such a comprehensive approach.
We test FencedLock in 4 different ways:
  • Non-reentrancy: we test if FencedLock behaves as a non-reentrant mutex, i.e., it can be held by a single endpoint at a time and only that endpoint can release it. Moreover, the lock cannot be acquired by the same endpoint reentrantly.
  • Reentrancy: We test if FencedLock behaves as a reentrant mutex. A single endpoint can hold the lock at a time and that endpoint can acquire the lock reentrantly. The reentrant lock acquire limit is 2 for this test.
  • Monotonic fencing tokens for non-reentrant FencedLock: We validate the monotonicity of fencing tokens assigned to lock holders.
  • Monotonic fencing tokens for reentrant FencedLock: Fencing tokens are incremented each time the lock switches from the available state to the held state. However, if the current lock holder acquires the lock reentrantly, it will get the same fencing token. The reentrant lock acquire limit is 2 for this test.
We fixed all the bugs that these tests revealed so far. You can see our tests in the official Jepsen repo. We are planning to extend our suite with more test scenarios and failure cases in addition to network partition failures. We are also integrating our Jepsen test suite into our CI system.

Closing Words

Locking in a distributed system is possible, but only if all of its components participate in the protocol. Hazelcast provides you with the implementation of such a distributed locking protocol and gives your components a straightforward way to participate in it.
We are looking forward to hearing about how developers use FencedLock for their needs. You can check our documentation and code samples to learn more about FencedLock, and use our Google groups to ask for help.
Until next time…

Acknowledgments

I would like to thank Martin Kleppmann for his valuable feedback and insightful comments, Marko Topolnik for his remarkable effort to help me improve this blog post, and Hazelcast folks for their reviews.

Edit Log

(April 4, 2019): We improved Figure 1 and the explanation of how to use the monotonic fencing tokens to achieve mutual exclusion across external services. In short, although no locking service can ensure only one process thinks it holds the lock, we can use monotonic fencing tokens to ensure that only one process can perform side effects on multiple external services at a time.
(June 6, 2019): The follow-up blog post is linked in the intro.

3/13/20

Keycloak HA setup in multinode Kubernetes cluster

Keycloak Cluster Setup

Friday, May 10 2019, posted by 张立强 liqiang@fit2cloud.com
This post shares some solutions to setup Keycloak cluster in various scenarios (e.g. cross-DC, docker cross-host, Kubernetes).
If you'd like to setup Keycloak cluster, this blog may give you some reference.
Two cli script files are added to the Keycloak image as per the guide.
The Dockerfile is below and these two files are the most important matter for this blog, you can find them from TCPPING.cli and JDBC_PING.cli.
FROM jboss/keycloak:latest

ADD cli/TCPPING.cli /opt/jboss/tools/cli/jgroups/discovery/
ADD cli/JDBC_PING.cli /opt/jboss/tools/cli/jgroups/discovery/
First of all we should know that for a Keycloak cluster, all keycloak instances should use same database and this is very simple, another thing is about cache(generally there are two kinds of cache in Keycloaks, the 1st is persistent data cache read from database aim to improve performance like realm/client/user, the 2nd is the non-persistent data cache like sessions/clientSessions, the 2nd is very important for a cluster) which is a little bit complex to configure, we have to make sure the consistent of cache in a cluster view.
Totally here are 3 solutions for clustering, and all of the solutions are base on the discovery protocols of JGroups (Keycloak use Infinispan cache and Infinispan use JGroups to discover nodes).

1. PING

PING is the default enabled clustering solution of Keycloak using UDP protocol, and you don't need to do any configuration for this.
But PING is only available when multicast network is enabled and port 55200 should be exposed, e.g. bare metals, VMs, docker containers in the same host.


We tested this by two Keycloak containers in same host.


The logs show that the two Keycloak instances discovered each other and clustered.



2. TCPPING

TCPPING use TCP protocol with 7600 port. This can be used when multicast is not available, e.g. deployments cross DC, containers cross host.

We tested this by two Keycloak containers cross host.
And in this solution we need to set three below environment variables for containers.
#IP address of this host, please make sure this IP can be accessed by the other Keycloak instances
JGROUPS_DISCOVERY_EXTERNAL_IP=172.21.48.39
#protocol
JGROUPS_DISCOVERY_PROTOCOL=TCPPING
#IP and Port of all host
JGROUPS_DISCOVERY_PROPERTIES=initial_hosts="172.21.48.4[7600],172.21.48.39[7600]"
The logs show that the two Keycloak instances discovered each other and clustered.


3. JDBC_PING

JDBC_PING use TCP protocol with 7600 port which is similar as TCPPING, but the difference between them is, TCPPING requires you configure the IP and port of all instances, for JDBC_PING you just need to configure the IP and port of current instance, this is because in JDBC_PING solution each instance insert its own information into database and the instances discover peers by the ping data read from database.
We tested this by two Keycloak containers cross host.
And in this solution we need to set two below environment variables for containers.
#IP address of this host, please make sure this IP can be accessed by the other Keycloak instances
JGROUPS_DISCOVERY_EXTERNAL_IP=172.21.48.39
#protocol
JGROUPS_DISCOVERY_PROTOCOL=JDBC_PING


The ping data of all instances haven been saved in database after instances started.

The logs show that the two Keycloak instances discovered each other and clustered.

One more thing

The above solutions are available for most scenarios, but they are still not enough for some others, e.g.Kubernetes.
The typical deployment on Kubernetes is one Deployment/ReplicateSet/StatefulSet contains multi Keycloak Pods, the Pods are really dynamic as they can scale up and down or failover to another node, which requires the cluster to discover and remove these dynamic members.
On Kubernetes we can use DNS_PING or KUBE_PING which work quite well in practice.
Besides DNS_PING and KUBE_PING, JDBC_PING is another option for Kubernetes.
On Kubernetes multicast is available only for the containers in the same node and a pod has no static ip which can be used to configure TCPPING or JDBC_PING. But in the JDBC_PING.cli mentioned above we have handled this, if you don't set the JGROUPS_DISCOVERY_EXTERNAL_IP env, the pod ip will be used, that means on Kubernetes you can simply set JGROUPS_DISCOVERY_PROTOCOL=JDBC_PING then your keycloak cluster is ok.

Discussion

Suggestions and comments can be discussed via Keycloak User Mail List or this GitHub Repository.

Nowadays supplying Single Sign On mechanisms becomes more and more important for users convenience. Briefly – single user logged into one system can be automatically logged within other applications used across organization (internally and / or externally). As it is very often used in integration of various systems – reliable operation upon bigger load with good performance of this service becomes important. 

Overview 

Keycloak supports HA mode to provide this functionality. See https://www.keycloak.org/docs/latest/server_installation/index.html#_standalone-mode. As we provide isolation of our environments by containers working in a cluster distributed across different machines and datacenters – another obstacles come in. 
Our Kubernetes cluster setup is multinode, cross server and cross data center, dynamic (from network perspective) installation. This excludes usage of almost every method for discovering cluster members.  We cannot assume on which node pod will be scheduled. Also, IP of other Keycloak cluster members (due to dynamic nature of Kubernetes pods) cannot be hardcoded. 
This excludes usage of DNS_PING protocol (containers are not on the same host), neither TCP_PING cannot be used. We do not know IP of newly created pods.  
We are using Helm chart for provisioning. 

Enabling HA in Keycloak chart 

Codecentric helm chart used to deploy Keycloak solution to Kubernetes cluster has required value in its values.yaml file:  
keycloak : 
  replicas: 2 
This starts Keycloak in HA mode what we can see i.e. from process list on container

Without any further configuration – due to multinode, cross-vm Kubernetes cluster – this results in loss of ability to login to admin page and a lot of similar errors.  
At this moment (May 2019), chart provided by Codecentric has this variable (JDBC_PING) already provided in templates/statefulset.yaml: 

{{- if $highAvailability }} 
            - name: JGROUPS_DISCOVERY_PROTOCOL 
 
              value: "dns.DNS_PING" 

Default single node cluster configuration may produce errors given below: 
10.123.109.80:7600: BaseServer.TcpConnection.readPeerAddress(): cookie 
sent by /10.100.79.133:57980 does not match own cookie; terminating connection 
at org.jgroups.blocks.cs.TcpConnection.readPeerAddress(TcpConnection.java:242) 
at org.jgroups.blocks.cs.TcpConnection.(TcpConnection.java:53) 
at org.jgroups.blocks.cs.TcpServer$Acceptor.handleAccept(TcpServer.java:126) 
at org.jgroups.blocks.cs.TcpServer$Acceptor.run(TcpServer.java:111) 
Additionally, we’re unable to log in to Admin panel of Keycloak. We are still redirected to login page. 
According to https://www.keycloak.org/2019/05/keycloak-cluster-setup.html, in this  particular scenario – JDBC_PING protocol can be used. 
While pods are starting, their IP’s are used while bootstrapping configuration. Cluster discovery is being conducted, and ID of nodes are added to database. This table does not refresh automatically, and unavailable addresses need to be removed dif ferent way (i.e. before deployment). This shortens time needed for pods to start. 

To achieve this method of configuration – JGROUPS_DISCOVERY_PROTOCOL=JDBC_PING environment variable need to be set in extraEnv section, however, this results in another error in logs while starting Keycloak: 
10:26:15,505 ERROR [org.jboss.msc.service.fail] (ServerService Thread 
Pool -- 52) MSC000001: Failed to start service org.wildfly.clustering.jgroups.channel.ee: 
org.jboss.msc.service.StartException in service 
org.wildfly.clustering.jgroups.channel.ee: java.lang.IllegalStateException: 
java.lang.IllegalArgumentException: 
java.security.PrivilegedActionException: 
java.lang.IllegalArgumentException: Unrecognized JDBC_PING properties: [dns_query] 
We need to change default value of this variable since we are not using DNS_PING discovery protocol.  It is defined in statefulset.yaml in Keycloak chart: 
- name: JGROUPS_DISCOVERY_PROPERTIES 
 value: "dns_query={{ template "keycloak.fullname" . }}-
headless.{{ .Release.Namespace }}.svc.{{ .Values.clusterDomain }}"
{{- end }}
Two options are here, one is to download whole chart and remove this condition, other one is to set this environment variable using our values.yaml  in section extraEnv:  
- name: JGROUPS_DISCOVERY_PROPERTIES 
  value: "" 
This value cannot be empty though. Next error will show up: 
11:35:57,491 ERROR [org.jboss.msc.service.fail] (ServerService Thread 
Pool -- 52) MSC000001: Failed to start service 
org.wildfly.clustering.jgroups.channel.ee: 
org.jboss.msc.service.StartException in service 
org.wildfly.clustering.jgroups.channel.ee: 
java.lang.IllegalStateException: java.lang.IllegalArgumentException: 
Either the 4 configuration properties starting with 'connection_' or 
the datasource_jndi_name must be set 
JGroups mechanism need database connection parameters to be provided. Four configuration properties mentioned above include connection string, driver, username and password in plaintext. We can also use datasource_jndi_name which we can get ie. from standalone.xml file from container of single Keycloak instance (replicas set to 1 and JGROUPS_DISCOVERY_PROTOCOL variable removed): 
 java:jboss/datasources/KeycloakDS
" 
Providing this variable will give us hopefully last error: 
12:08:29,323 ERROR [org.jgroups.protocols.JDBC_PING] (ServerService 
Thread Pool -- 58) JGRP000138: Error reading JDBC_PING table:
org.postgresql.util.PSQLException: ERROR: relation "jgroupsping" does not exist 
As we may assume – we’re missing one table in our database. That is table mentioned on the beginning. It is not provided within Keycloak container out of the box and need to be added manually before service start.   
CREATE TABLE IF NOT EXISTS JGROUPSPING (own_addr varchar(200) NOT NULL, cluster_name varchar(200) NOT NULL, ping_data BYTEA, constraint PK_JGROUPSPING PRIMARY KEY (own_addr, cluster_name))

Docker image customization 

Additionally, we need to reconfigure Keycloak TCP stack – disable multicast, and use JDBC_PING as default discovery protocol.  
Reconfiguration of keycloak server can be done in container. We need to start Wildfly server, provision with file listed below, and stop it.
One way is to provide such CLI configuration in Keycloak container and build it using Docker. Later on such container can be used as image in our values.yaml for Kubernetes deployment. 
These steps can be supplied in a wrapper like so
(file standalone-ha-configuration.cli, 3 lines):
embed-server --server-config=standalone-ha.xml --std-out=echo
run-batch --file= /opt/jboss/startup-scripts/jgroups-jdbc-ping.cli
stop-embedded-server 
standalone-ha-configuration.cli is started in Dockerfile before entrypoint like so (one line): 
RUN cd /opt/jboss/keycloak && bin/jboss-cli.sh --
file=/opt/jboss/startup-scripts/standalone-ha-configuration.cli && 
rm -rf 
/opt/jboss/keycloak/standalone/configuration/standalone_xml_history 

Here is database table and TCP stack reconfiguration file which is executed within keycloak CLI (file jgroups-jdbc-ping.cli)
# Make use of the JDBC_PING
/subsystem=jgroups/stack=tcp:remove()
/subsystem=jgroups/stack=tcp:add()
/subsystem=jgroups/stack=tcp/transport=TCP:add(socket-binding="jgroups-tcp")
/subsystem=jgroups/stack=tcp/protocol=JDBC_PING:add()
/subsystem=jgroups/stack=tcp/protocol=JDBC_PING/property=datasource_jndi_name:add
(value=java:jboss/datasources/KeycloakDS)
/subsystem=jgroups/stack=tcp/protocol=JDBC_PING/property=break_on_coord_rsp:add(value=true)
# Statements must be adapted for PostgreSQL. Additionally, we add a 
'creation_timestamp' column. 
/subsystem=jgroups/stack=tcp/protocol=JDBC_PING/property=initialize_sql:add
(value="CREATE TABLE IF NOT EXISTS JGROUPSPING (own_addr varchar(200) 
NOT NULL, creation_timestamp timestamp NOT NULL, cluster_name varchar(200) 
NOT NULL, ping_data bytea, constraint PK_JGROUPSPING PRIMARY KEY (own_addr, cluster_name))") 
/subsystem=jgroups/stack=tcp/protocol=JDBC_PING/property=insert_single_sql:add
(value="INSERT INTO JGROUPSPING (own_addr, creation_timestamp, cluster_name, ping_data) 
values (?, NOW(), ?, ?)") 
/subsystem=jgroups/stack=tcp/protocol=MERGE3:add() 
/subsystem=jgroups/stack=tcp/protocol=FD_SOCK:add(socket-binding="jgroups-tcp-fd") 
/subsystem=jgroups/stack=tcp/protocol=FD:add() 
/subsystem=jgroups/stack=tcp/protocol=VERIFY_SUSPECT:add() 
/subsystem=jgroups/stack=tcp/protocol=pbcast.NAKACK2:add() 
/subsystem=jgroups/stack=tcp/protocol=UNICAST3:add() 
/subsystem=jgroups/stack=tcp/protocol=pbcast.STABLE:add() 
/subsystem=jgroups/stack=tcp/protocol=pbcast.GMS:add() 
/subsystem=jgroups/stack=tcp/protocol=pbcast.GMS/property=max_join_attempts:add(value=5) 
/subsystem=jgroups/stack=tcp/protocol=MFC:add() 
/subsystem=jgroups/stack=tcp/protocol=FRAG2:add() 
/subsystem=jgroups/channel=ee:write-attribute(name=stack, value=tcp) 
/subsystem=jgroups/stack=udp:remove() 
/socket-binding-group=standard-sockets/socket-binding=jgroups-mping:remove() 
/interface=private:write-attribute(name=nic, value=eth0) 
/interface=private:undefine-attribute(name=inet-address) 
These steps will result in a working clustered HA configuration of Keycloak in Kubernetes environment. Below established cluster logs are shown (infinispan is in-memory cache which is clustered here automatically): 
07:34:46,383 INFO  [org.infinispan.CLUSTER] 
(MSC service thread 1-4) ISPN000094: Received new cluster view for channel ejb: 
[keycloak-4|6] (5) [keycloak-4, keycloak-2, keycloak-3, keycloak-1, keycloak-0] 
07:34:46,383 INFO  [org.infinispan.CLUSTER] 
(MSC service thread 1-1) ISPN000094: Received new cluster view for channel ejb: 
[keycloak-4|6] (5) [keycloak-4, keycloak-2, keycloak-3, keycloak-1, keycloak-0] 
07:34:46,383 INFO  [org.infinispan.CLUSTER] 
(MSC service thread 1-3) ISPN000094: Received new cluster view for channel ejb: 
[keycloak-4|6] (5) [keycloak-4, keycloak-2, keycloak-3, keycloak-1, keycloak-0] 
Here 5 clustered instances are seen. Finally, we are able to log in to Administrator panel too. 

Thank you for reading, I hope this knowledge will be useful for you and will help in development of your SSO solution.