502 Bad Gateway on Google Cloud Run

Hi there, first of all: thanks for creating these Docker images. They’re looking solid :+1:

One issue I am facing is when running a PHP container with the serversideup/php:8.1-fpm-nginx image on Google Cloud Run, the first request (when the container has a cold start and must boot first) throws a 502 Bad Gateway.
Inspecting the logs, it returns this error:

2022/09/18 15:05:26 [crit] 284#284: *3 connect() to unix:/var/run/php/php-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 169.254.1.1, server: _, request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php-fpm.sock:", host: "*removed*"

I’ve experienced this issue too with our current Docker setup. Cloud Run marks the container/service as ready when the container is listening on the specified port. This issue happens because NGINX is started before PHP-FPM. Once Nginx is ready, Cloud Run will accept connections. The first request fails because PHP-FPM is not ready yet. All next requests work fine!

Did a brief scan through the S6-overlay documentation and saw there is a way to make services dependent on other services? Is there a way to set it up so Nginx is started after PHP-FPM instead of before?

Thanks for reporing this @marickvantuil! This is a solid issue report and helped me find a solution to your problem quickly :raised_hands:

I just pushed up a commit attempting to fix this issue on my “beta” branch: Fixed order of service execution · serversideup/[email protected] · GitHub

Can you confirm everything works for you as expected?

Use this image for testing: serversideup/php:beta-8.1-fpm-nginx

You can see the full changelog for this image here: Major restructure and optimization by jaydrogers · Pull Request #59 · serversideup/docker-php · GitHub

Thanks for the quick reply!!

I’ve tried with the new beta image, and while the order does seem correct, it’s not quite working yet. One problem with Cloud Run containers is that the cold start is kind of… slow :sweat_smile:

It takes PHP-FPM 3 seconds to start after NGINX:

I understand this might be Cloud Run specific, and I don’t want to pollute your image with specific logic on how to handle a slow PHP-FPM boot. Maybe there is a way to override the image to add some logic to make NGINX dependent on PHP-FPM? Though I have no idea how to do that, haha! Any other ideas are much appreciated!

Thanks for the detailed logs again :+1:

I’ve been playing around with this over the last hour and half, and I see where I can improve this…

Unfortunately I ran out of time and I’m heading out for the evening, but I will jump on this again tomorrow. I will keep you posted!

1 Like

I just pushed up another change to serversideup/php:beta-8.1-fpm-nginx

If you could give that a whirl and let me know if that solves your issue.

I am 60% confident it will, but I am curious to see your results.

Whats new:

  • This commit changed the format from “Legacy S6 Overlay” format to “Modern S6 Overlay” format

I am not sure if this will make a difference, but I did confirm on my end that they were executing in order.

The full solution:

  • Use “healthchecks” for each service and require them to be healthy before continuing

The full solution is easier said than done (thanks PHP-FPM :smiley:). I’m currently playing around with that feature right now and will update you once I have more information :+1:

Also, just in case you get that email quickly. I guess I pressed “Submit” too soon…

Make sure this job completes before testing (the images are still building at the moment of me writing this message): Configured working S6 overlay service in new format · serversideup/[email protected] · GitHub

UPDATE: It completed successfully :smiley:

Just adding another update on this thread:

If you want to test what I have now, that would be helpful in the meantime!

serversideup/php:beta-8.1-fpm-nginx

Just adding another update on this…

I have a few conversations opened on how to best solve this. All of this is being tracked on my PR: Draft: Add Readiness Checks by jaydrogers · Pull Request #75 · serversideup/docker-php · GitHub

Alright… One more update…

Good news is, I should have this resolved.

The power of open source really helped out on this one. Without the fine folks over at S6 Overlay and PHP-FPM-Healthcheck, I wouldn’t have any clue how to do this on my own.

I just merged these enhancements to serversideup/php:beta-8.1-fpm-nginx.

Check it out and let me know if you have any more issues!

I will mark this solved for now, unless you report differently :+1:

1 Like

Thanks!! Was a bit busy this week, but I just tested this, and it’s working perfect from what I can see! Will definitely test it some more in the coming days, but I expect that this is solved now. Thanks so much for the help :slight_smile:
This is great, as I can now get rid of our proprietary Docker image and start using the Server Side Up images!

1 Like

Awesome!! I will be merging these images to “stable” in the next few weeks.

I am testing the beta images on our internal apps and will merge once I have full confidence. :+1: