Kubernetes + Nginx : Upstream Sent Too Big Header
My Laravel site was working just fine yesterday but after a code-only update today I was seeing 502 errors on some pages
upstream sent too big header while reading response header from upstream, client:
While the solution was hard to find it was easy to implement.
Explanation
In my case the cause was InertiaJS within Laravel
It all worked fine at first - but I’d been hacking on some React in a Do the simplest thing that could possibly work kind of a way and initially that meant a few big components with some copy paste going on.
But going past a proof of concept stage I had done some refactoring and broken down my code into a number of smaller modules.
Running my code in production mode locally I was able to see the following header
Link: <http://localhost/build/assets/app-UcQWvS3O.css>; rel="preload"; as="style", <http://localhost/build/assets/app-0PJJ3e0W.js>; rel="modulepreload", <http://localhost/build/assets/Show-8VeUuxjo.js>; rel="modulepreload", <http://localhost/build/assets/AuthenticatedLayout-xjIFWOj6.js>; rel="modulepreload", <http://localhost/build/assets/Tabs-RKDiCRbI.js>; rel="modulepreload", <http://localhost/build/assets/Alert-i2JiGJJ9.js>; rel="modulepreload", <http://localhost/build/assets/Preview-f-ElfTPO.js>; rel="modulepreload", <http://localhost/build/assets/Edit-ise_5yE-.js>; rel="modulepreload", <http://localhost/build/assets/Container-pio715qQ.js>; rel="modulepreload", <http://localhost/build/assets/ThemeProvider-nl1uEB85.js>; rel="modulepreload", <http://localhost/build/assets/Nav-1m4InTeO.js>; rel="modulepreload", <http://localhost/build/assets/hook-lH3WC8uy.js>; rel="modulepreload", <http://localhost/build/assets/objectWithoutPropertiesLoose-ISFDu3G3.js>; rel="modulepreload", <http://localhost/build/assets/querySelectorAll-OOFkXFKE.js>; rel="modulepreload", <http://localhost/build/assets/Button-erS29WTu.js>; rel="modulepreload", <http://localhost/build/assets/Anchor-AUwl0HgO.js>; rel="modulepreload", <http://localhost/build/assets/Fade-oWunCIPq.js>; rel="modulepreload", <http://localhost/build/assets/AbstractModalHeader-PszNXT4L.js>; rel="modulepreload", <http://localhost/build/assets/NoopTransition-0KMsi1jl.js>; rel="modulepreload", <http://localhost/build/assets/divWithClassName-TNsvM5Ip.js>; rel="modulepreload", <http://localhost/build/assets/index-DOBjPGd3.js>; rel="modulepreload", <http://localhost/build/assets/import-_iTFD0sX.js>; rel="modulepreload", <http://localhost/build/assets/InputGroupContext-PmE461b8.js>; rel="modulepreload", <http://localhost/build/assets/Button-TKEY_PYu.js>; rel="modulepreload", <http://localhost/build/assets/ElementChildren-vwpaH4QL.js>; rel="modulepreload", <http://localhost/build/assets/Form-dHofC52s.js>; rel="modulepreload", <http://localhost/build/assets/Col-MN3-eVM3.js>; rel="modulepreload", <http://localhost/build/assets/ckeditor-1hthJoEb.js>; rel="modulepreload", <http://localhost/build/assets/InputGroup-9XltiSDt.js>; rel="modulepreload", <http://localhost/build/assets/ImageLibrary-WN9GZHru.js>; rel="modulepreload", <http://localhost/build/assets/Modal-DKx5DwuU.js>; rel="modulepreload", <http://localhost/build/assets/Pagination-2HUjgsvy.js>; rel="modulepreload", <http://localhost/build/assets/Table-mR_I82hO.js>; rel="modulepreload", <http://localhost/build/assets/ImageBlock-tqSxxa9H.js>; rel="modulepreload", <http://localhost/build/assets/VideosetBlock-2Z8NQSdz.js>; rel="modulepreload", <http://localhost/build/assets/Video-CZPDJDQb.js>; rel="modulepreload", <http://localhost/build/assets/TextBlock-g_VzDBUi.js>; rel="modulepreload", <http://localhost/build/assets/ComponentHeader-vqDHx7Ww.js>; rel="modulepreload"
Along with a few cookies and regular headers this was pushing my headers to a size wouldn’t fit in the default nginx configuration.
These headers only became a problem when loading some pages via a direct page load. Regular navigation was OK as long as the first page dion’t have too many modules as they would load a few at a time. But if the first page view was a page with many modules - then the header went over a limit.
Kubernetes Complication
After a bit of searching I had found the Nginx changes, I updated my config and … I still had the same error but this time my nginx logs said everything was fine and claimed to have responded with a 200 OK status …
It took me a while to realise that now I was seeing the same nginx error - but this time from the nginx-ingress.
I was able to confirm this via
kubectl logs -f -n ingress-nginx pod/ingress-nginx-controller-mypodid
I found a good post on fixing this
and comments on it which told me that there is now an easier and non-intrusive way to do this using this annotation
Adding those annotations to my Helm chart fixed it.
Solution
Nginx Container Config
Add these two proxy_buffer lines to your nginx config
location ~ \.php$ {
#NOTE: You should have "cgi.fix_pathinfo = 0;" in php.ini
include fastcgi_params;
fastcgi_intercept_errors on;
fastcgi_pass ${FPM_HOST};
fastcgi_param SCRIPT_FILENAME $document_root/$fastcgi_script_name;
proxy_buffers 8 16k; # Buffer pool = 8 buffers of 16k
proxy_buffer_size 16k; # 16k of buffers from pool used for headers
}
Nginx Ingress Config
In my case this is within Helm Values.yaml file
ingress:
enabled: true
className: ""
annotations:
kubernetes.io/ingress.class: nginx
cert-manager.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/proxy-buffers-number: "4"
nginx.ingress.kubernetes.io/proxy-buffer-size: "16k"