# OCI Load Balancer - HTTP Health Check Failing Date: 16DEC2024 Operating system: Oracle Linux Version: 8 Image: Oracle-Linux-8.10-aarch64-2024.10.31-0 ## Page Summary: This page will cover troubleshooting a failed health check received on the OCI Load Balancer. The current lab consists of a deployed OCI Load Balancer in a Public Subnet which is fronting 2 web servers sitting in a Private Subnet. The load balancer is listening on Port 80 (http), and will forward requests in round robin to both back-end servers. For added context, both instances sit in two different AD's, but are within a regional subnet. Load balancer sits in a regional subnet as well. Public Subnet CIDR: 10.0.0.0/24 Private Subnet CIDR: 10.0.1.0/24 Snapshot of Health Check Error I am receiving on both deployed back-end servers will be in image below: ![[health-check2.png]] **Health Check Error**: Critical - Connection failed ## Analysis: After checking the load balancer Error Logs and not seeing any errors, I confirmed that the Security Lists are correct and permitting traffic from the load balancer to the back-end web servers and vice versa. <div style="margin-top: 40px;"></div> Once validated, this pushed me in the direction of an issue within the instances themselves. ## Fix #1: I use a bastion instance to remote into 1 of the back-end servers and check firewalld rules to see if http traffic is allowed to the first instance: ``` sudo firewall-cmd --list-all ``` ![[oci-instance-rules.png]] Taking a look at the "services" line, there is no http being allowed to this instance. Lets add a firewalld rule to fix this. Step 1: Add a rule that will permit http traffic. ``` sudo firewall-cmd --zone=public --add-service=http --permanent ``` Step 2: Issue a reload to load the rule into the firewalld table. ``` sudo firewall-cmd --reload ``` Step 3: Verify rule. ``` sudo firewall-cmd --list-all ``` ![[oci-http-rule.png]] As you can see http is now allowed to this instance. Lets check back in with our OCI Load Balancer: ![[oci-lb-error2.png]] We are faced with a new error code "Status code mismatch". ## Fix #2: Referencing [Oracle Docs](https://docs.oracle.com/en-us/iaas/Content/Balance/Troubleshooting/common_load_balancer_errors.htm), here are two solutions: ![[oci-cmmon-lb.png]] Currently my health check path is set to "/" on the OCI Load Balancer. So, I decide to curl to that path on the instance to see what response code I receive: ![[403 curl.png]] I receive a http response code of 403. If you research this response code you will see something similar along the lines of "HTTP 403 is an HTTP status code meaning **access to the requested resource is forbidden**." This tells me that the health check path does not have sufficient permissions. Lets create a new file that we will use for health checks and assign proper permissions to. In regards to Apache, the path `/var/www/html/` is the default **document root** directory. It is the location where Apache expects to find the files it serves to clients when they make requests to the server, such as our load balancer. Create a file called "health" in the html directory, and add a message such as "OK" inside of this file by using a text editor such as nano or vim. ``` sudo touch health (creates the file) sudo nano health (opens the file with nano) ``` ![[nano-ok.png]] Ctrl + O to save file. Ctrl + X to exit file. Now lets adjust permissions on this file so the Apache daemon has necessary permissions to operate this file. ``` sudo chmod 644 /var/www/html/health sudo chown apache:apache /var/www/html/health ``` Chmod 644 changes the files permissions to *rw-owner*, *r--group*, *r--others*. Chown changes the owner and the group to apache. ![[health-file.png]] After adjusting permissions restart the httpd service: ``` sudo systemctl restart httpd.service ``` Now with our new health check file created and adjusted permissions, change the health check path on the OCI Load Balancer to "/health". ![[new-health-file.png]] Save changes, then give the load balancer time to retry its poll on the instance. ![[lb-health.png]] ![[lb-health2.png]] Great! As you can see we now have a successful health check on 1 compute instance. Now apply the same 2 fix's to the second instance.