Does your backend target server becomes unhealthy when registered with ALB or NLB ? Don’t worry ! We will now discuss the steps to isolate the problem to Target or ALB. After knowing this, you will be able to fix It by yourself and would be in position to reach out to correct support team within AWS (ELB team or Linux/Windows team) for further help.
Unhealthy Targets registered with ALB
ALB health check follows below call to Target hence you can do same from another machine to target machine using curl and thus isolating problem. I will be assuming Linux instance being Target server in this article.
=> http://10.0.0.1:8080/path
Here http is the Health check protocol that is configured (it can be https ). 10.x.x is the IP of Target server, 8080 is the Health check port of server and path is the path configured on ALB for server. Similar curl command would be
=> curl -Iv http://10.0.0.1:8080/path
Simply run this command (feel free to put your Target IP, port path, if you are default ‘/’ path then leave it blank) from another machine in same VPC as of Target and reachable to Target. Check the response code in output. If it is anything else then 200 OK, it means the problem is with your Target server and not ALB.
At the same time, do one more test. That is, run same command in the Target console itself and verify the code in output ! If it returns 200 OK, this means Target server is working fine but something is blocking outside connection to it. Check for IPtables, firewall rule in Target server and check VPC security group rules , NACLs, Route table, NAT/PAT Gateway attached to this target server. This will fix the issue of reachability.
Let’s discuss the common mistakes that you might have done while registering target.
- When you create the Target Group, you configure it with the port number. But when you register Target to that Target Group, you get option to configure port number for it, this means you can override the Group settings. Sample screenshots depicting same is attached below.
You might have mistakenly configured it on incorrect port and your server is listening on another port ! so, please make sure that whatever port you put here, the server must listening on it actively !
To check this, run below command :
netstat -tulpn
if port 8080 is not present in list with LISTEN state, this means server isn’t listening on it and get that fixed by checking your service/application (like on which port it is configured to work on and more). This also isolates the problem outside ELB Scope.
Once you are done with this, you will be able to fix it yourself !
Sometimes everything is fine but ALB still marks my target as Unhealthy ?
Well there could be time when your server is returning 404 or 3xx/4xx or some other response code to Health check, which is expected by you in some cases like API calls that require authentication or simply API calls, so in that case if it is OK code for you and you want ALB to show Target as healthy for it, just put these code as Success code under Health check setting on Target group as shown in screenshot below.
Unhealthy Targets registered with NLB
NLB will also do HC is same way as ALB if it is configured with http/https HC. Hence troubleshooting steps will be same. In case of TCP HC, it will be more simple, just like telnet check / netcat check on port to target.
For example : Talent target-IP port OR nc -zv target-ip port
Hence you can run this command from another machine to Target machine, if it fails, this means problem list at Target thus isolated out of NLB scope. Checks for Security group attached to instances, NACLs, firewall rules etc.
I hope you all enjoyed reading this article and was able to fix the Unhealthy Target group issue. In case if you are seeing any other error, please feel free to comment it and we will try to address it here asap.
Thank you