Tangible Bytes

A Web Developer’s Blog

Kubernetes + Nginx : Upstream Sent Too Big Header

My Laravel site was working just fine yesterday but after a code-only update today I was seeing 502 errors on some pages

upstream sent too big header while reading response header from upstream, client:

While the solution was hard to find it was easy to implement.

Explanation

In my case the cause was InertiaJS within Laravel

It all worked fine at first - but I’d been hacking on some React in a Do the simplest thing that could possibly work kind of a way and initially that meant a few big components with some copy paste going on.

But going past a proof of concept stage I had done some refactoring and broken down my code into a number of smaller modules.

Running my code in production mode locally I was able to see the following header

Link: <http://localhost/build/assets/app-UcQWvS3O.css>; rel="preload"; as="style", <http://localhost/build/assets/app-0PJJ3e0W.js>; rel="modulepreload", <http://localhost/build/assets/Show-8VeUuxjo.js>; rel="modulepreload", <http://localhost/build/assets/AuthenticatedLayout-xjIFWOj6.js>; rel="modulepreload", <http://localhost/build/assets/Tabs-RKDiCRbI.js>; rel="modulepreload", <http://localhost/build/assets/Alert-i2JiGJJ9.js>; rel="modulepreload", <http://localhost/build/assets/Preview-f-ElfTPO.js>; rel="modulepreload", <http://localhost/build/assets/Edit-ise_5yE-.js>; rel="modulepreload", <http://localhost/build/assets/Container-pio715qQ.js>; rel="modulepreload", <http://localhost/build/assets/ThemeProvider-nl1uEB85.js>; rel="modulepreload", <http://localhost/build/assets/Nav-1m4InTeO.js>; rel="modulepreload", <http://localhost/build/assets/hook-lH3WC8uy.js>; rel="modulepreload", <http://localhost/build/assets/objectWithoutPropertiesLoose-ISFDu3G3.js>; rel="modulepreload", <http://localhost/build/assets/querySelectorAll-OOFkXFKE.js>; rel="modulepreload", <http://localhost/build/assets/Button-erS29WTu.js>; rel="modulepreload", <http://localhost/build/assets/Anchor-AUwl0HgO.js>; rel="modulepreload", <http://localhost/build/assets/Fade-oWunCIPq.js>; rel="modulepreload", <http://localhost/build/assets/AbstractModalHeader-PszNXT4L.js>; rel="modulepreload", <http://localhost/build/assets/NoopTransition-0KMsi1jl.js>; rel="modulepreload", <http://localhost/build/assets/divWithClassName-TNsvM5Ip.js>; rel="modulepreload", <http://localhost/build/assets/index-DOBjPGd3.js>; rel="modulepreload", <http://localhost/build/assets/import-_iTFD0sX.js>; rel="modulepreload", <http://localhost/build/assets/InputGroupContext-PmE461b8.js>; rel="modulepreload", <http://localhost/build/assets/Button-TKEY_PYu.js>; rel="modulepreload", <http://localhost/build/assets/ElementChildren-vwpaH4QL.js>; rel="modulepreload", <http://localhost/build/assets/Form-dHofC52s.js>; rel="modulepreload", <http://localhost/build/assets/Col-MN3-eVM3.js>; rel="modulepreload", <http://localhost/build/assets/ckeditor-1hthJoEb.js>; rel="modulepreload", <http://localhost/build/assets/InputGroup-9XltiSDt.js>; rel="modulepreload", <http://localhost/build/assets/ImageLibrary-WN9GZHru.js>; rel="modulepreload", <http://localhost/build/assets/Modal-DKx5DwuU.js>; rel="modulepreload", <http://localhost/build/assets/Pagination-2HUjgsvy.js>; rel="modulepreload", <http://localhost/build/assets/Table-mR_I82hO.js>; rel="modulepreload", <http://localhost/build/assets/ImageBlock-tqSxxa9H.js>; rel="modulepreload", <http://localhost/build/assets/VideosetBlock-2Z8NQSdz.js>; rel="modulepreload", <http://localhost/build/assets/Video-CZPDJDQb.js>; rel="modulepreload", <http://localhost/build/assets/TextBlock-g_VzDBUi.js>; rel="modulepreload", <http://localhost/build/assets/ComponentHeader-vqDHx7Ww.js>; rel="modulepreload"

Along with a few cookies and regular headers this was pushing my headers to a size wouldn’t fit in the default nginx configuration.

These headers only became a problem when loading some pages via a direct page load. Regular navigation was OK as long as the first page dion’t have too many modules as they would load a few at a time. But if the first page view was a page with many modules - then the header went over a limit.

Kubernetes Complication

After a bit of searching I had found the Nginx changes, I updated my config and … I still had the same error but this time my nginx logs said everything was fine and claimed to have responded with a 200 OK status …

It took me a while to realise that now I was seeing the same nginx error - but this time from the nginx-ingress.

I was able to confirm this via

kubectl logs -f -n ingress-nginx pod/ingress-nginx-controller-mypodid

I found a good post on fixing this

and comments on it which told me that there is now an easier and non-intrusive way to do this using this annotation

Adding those annotations to my Helm chart fixed it.

Solution

Nginx Container Config

Add these two proxy_buffer lines to your nginx config

location ~ \.php$ {
  #NOTE: You should have "cgi.fix_pathinfo = 0;" in php.ini
  include fastcgi_params;
  fastcgi_intercept_errors on;
  fastcgi_pass ${FPM_HOST};
  fastcgi_param SCRIPT_FILENAME $document_root/$fastcgi_script_name;
  proxy_buffers         8 16k;  # Buffer pool = 8 buffers of 16k
  proxy_buffer_size     16k;    # 16k of buffers from pool used for headers
}

Nginx Ingress Config

In my case this is within Helm Values.yaml file

ingress:
  enabled: true
  className: ""
  annotations:
     kubernetes.io/ingress.class: nginx
     cert-manager.io/cluster-issuer: letsencrypt-prod
     nginx.ingress.kubernetes.io/proxy-buffers-number: "4"
     nginx.ingress.kubernetes.io/proxy-buffer-size: "16k"