There are several common pitfalls you may run into when running Kubernetes health checks:
HTTP applications should not have TCP health checks, as they will mark your application as healthy on port binding, even if your actual HTTP service is not running. Write a proper health endpoint, where you should check the application’s dependencies and then make it live.
Always put readiness checks in your servers. This ensures that your application does not prematurely receive traffic it cannot serve.
Avoid TCP health checks for databases like Redis. Redis can be live once the Redis server is running, but this doesn’t ensure that Redis has joined the cluster or started as a master slave or its final configurations. In these cases, use the liveness command interface to make sure Redis or the databases are in your desired state.
Avoid verifying dependencies in your health check that are not necessary for the application to be running. Also, avoid health check loops. For example, application X needs applications Y and Z to be online, and Z needs X to be online. In this scenario, if one application goes down, they will all go down, as they are dependent on each other.