Troubleshoot the application
It’s a fact of life that things fail. With this in mind, it’s important to understand what happens when failures occur and how to mitigate them. The following sections cover different failure scenarios:
- Swarm manager failures
- Consul (discovery backend) failures
- Interlock load balancer failures
- Web (voting-app) failures
- Redis failures
- Worker (vote-worker) failures
- Postgres failures
- Results-app failures
- Infrastructure failures
Swarm manager failures
In it’s current configuration, the Swarm cluster only has single manager container running on a single node. If the container exits or the node fails, you will not be able to administer the cluster until you either; fix it, or replace it.
If the failure is the Swarm manager container unexpectedly exiting, Docker will automatically attempt to restart it. This is because the container was started with the --restart=unless-stopped switch.
While the Swarm manager is unavailable, the application will continue to work in its current configuration. However, you will not be able to provision more nodes or containers until you have a working Swarm manager.
Docker Swarm supports high availability for Swarm managers. This allows a single Swarm cluster to have two or more managers. One manager is elected as the primary manager and all others operate as secondaries. In the event that the primary manager fails, one of the secondaries is elected as the new primary, and cluster operations continue gracefully. If you are deploying multiple Swarm managers for high availability, you should consider spreading them across multiple failure domains within your infrastructure.
Consul (discovery backend) failures
The Swarm cluster that you have deployed has a single Consul container on a single node performing the cluster discovery service. In this setup, if the Consul container exits or the node fails, the application will continue to operate in its current configuration. However, certain cluster management operations will fail. These include registering new containers in the cluster and making lookups against the cluster configuration.
If the failure is the consul container unexpectedly exiting, Docker will automatically attempt to restart it. This is because the container was started with the --restart=unless-stopped switch.
The Consul, etcd, and Zookeeper discovery service backends support various options for high availability. These include Paxos/Raft quorums. You should follow existing best practices for deploying HA configurations of your chosen discover service backend. If you are deploying multiple discovery service instances for high availability, you should consider spreading them across multiple failure domains within your infrastructure.
If you operate your Swarm cluster with a single discovery backend service and this service fails and is unrecoverable, you can start a new empty instance of the discovery backend and the Swarm agents on each node in the cluster will repopulate it.
Handling failures
There are many reasons why containers can fail. However, Swarm does not attempt to restart failed containers.
One way to automatically restart failed containers is to explicitly start them with the --restart=unless-stopped flag. This will tell the local Docker daemon to attempt to restart the container if it unexpectedly exits. This will only work in situations where the node hosting the container and it’s Docker daemon are still up. This cannot restart a container if the node hosting it has failed, or if the Docker daemon itself has failed.
Another way is to have an external tool (external to the cluster) monitor the state of your application, and make sure that certain service levels are maintained. These service levels can include things like “have at least 10 web server containers running”. In this scenario, if the number of web containers drops below 10, the tool will attempt to start more.
In our simple voting-app example, the front-end is scalable and serviced by a load balancer. In the event that on the of the two web containers fails (or the node that is hosting it), the load balancer will stop routing requests to it and send all requests the surviving web container. This solution is highly scalable meaning you can have up to n web containers behind the load balancer.
Interlock load balancer failures
The environment that you have provisioned has a single interlock load balancer container running on a single node. In this setup, if the container exits or node fails, the application will no longer be able to service incoming requests and the application will be unavailable.
If the failure is the interlock container unexpectedly exiting, Docker will automatically attempt to restart it. This is because the container was started with the --restart=unless-stopped switch.
It is possible to build an HA Interlock load balancer configuration. One such way is to have multiple Interlock containers on multiple nodes. You can then use DNS round robin, or other technologies, to load balance across each Interlock container. That way, if one Interlock container or node goes down, the others will continue to service requests.
If you deploy multiple interlock load balancers, you should consider spreading them across multiple failure domains within your infrastructure.
Web (voting-app) failures
The environment that you have configured has two voting-app containers running on two separate nodes. They operate behind an Interlock load balancer that distributes incoming connections across both.
In the event that one of the web containers or nodes fails, the load balancer will start directing all incoming requests to surviving instance. Once the failed instance is back up, or a replacement is added, the load balancer will add it to the configuration and start sending a portion of the incoming requests to it.
For highest availability you should deploy the two frontend web services (frontend01 and frontend02) in different failure zones within your infrastructure. You should also consider deploying more.
Redis failures
If the a redis container fails, it’s partnered voting-app container will not function correctly. The best solution in this instance might be to configure health monitoring that verifies the ability to write to each Redis instance. If an unhealthy redis instance is encountered, remove the voting-app and redis combination and attempt remedial actions.
Worker (vote-worker) failures
If the worker container exits, or the node that is hosting it fails, the redis containers will queue votes until the worker container comes back up. This situation can prevail indefinitely, though a worker needs to come back at some point and process the votes.
If the failure is the worker01 container unexpectedly exiting, Docker will automatically attempt to restart it. This is because the container was started with the --restart=unless-stopped switch.
Postgres failures
This application does not implement any for of HA or replication for Postgres. Therefore losing the Postgres container would cause the application to fail and potential lose or corrupt data. A better solution would be to implement some form of Postgres HA or replication.
Results-app failures
If the results-app container exits, you will not be able to browse to the results of the poll until the container is back up and running. Results will continue to be collected and counted, you will just not be able to view results until the container is back up and running.
The results-app container was started with the --restart=unless-stopped flag meaning that the Docker daemon will automatically attempt to restart it unless it was administratively stopped.
Infrastructure failures
There are many ways in which the infrastructure underpinning your applications can fail. However, there are a few best practices that can be followed to help mitigate and offset these failures.
One of these is to deploy infrastructure components over as many failure domains as possible. On a service such as AWS, this often translates into balancing infrastructure and services across multiple AWS Availability Zones (AZ) within a Region.
To increase the availability of our Swarm cluster you could:
- Configure the Swarm manager for HA and deploy HA nodes in different AZs
- Configure the Consul discovery service for HA and deploy HA nodes in different AZs
- Deploy all scalable components of the application across multiple AZs
This configuration is shown in the diagram below.
This will allow us to lose an entire AZ and still have our cluster and application operate.
But it doesn’t have to stop there. Some applications can be balanced across AWS Regions. It’s even becoming possible to deploy services across cloud providers, or have balance services across public cloud providers and your on premises date centers!
The diagram below shows parts of the application and infrastructure deployed across AWS and Microsoft Azure. But you could just as easily replace one of those cloud providers with your own on premises data center. In these scenarios, network latency and reliability is key to a smooth and workable solution.
Related information
The application in this example could be deployed on Docker Universal Control Plane (UCP) which is currently in Beta release. To try the application on UCP in your environment, request access to the UCP Beta release. Other useful documentation:
Please login to continue.