NextJS Multisite on Kubernetes

Thu Nov 30, 2023 NextJS Kubernetes

I am working on a NextJS project that has 9 websites with different content and themes but the exact same structure.

We want one codebase and one server to reduce costs (and the carbon footprint of our cloud) and also to streamline the process of releasing new features.

So we’re using middleware to detect which host the incoming request is for and add that as a parameter - which worked fine in development but I had some problems moving to our production system which is in Kubernetes.

Background

As background these are some examples of posts from people working on similar projects

I should note that this project is not live yet and we are still exploring the best way to build this

Middleware

What I have is something like this

import { NextRequest, NextResponse } from "next/server";

export function middleware(request: NextRequest) {
  const url = request.nextUrl;
  if (!process.env.A_HOST) {
    return
  }
  const hostname = request.headers.get("x-forwarded-host") ?? request.headers.get("host") ?? process.env.A_HOST;

  let host = ""
  switch (hostname) {
    case process.env.A_HOST:
      host = "sitea"
      break

    case process.env.B_HOST:
      host = "siteb"
      break
    case process.env.C_HOST:
      host = "sitec"
      break
    default:
      // console.log(request.headers)
      console.log(`no match : ${hostname}`)
      return;
  }



  // check if we already have a path prefix that matches one of our domains
  // don't redirect if we do 
  // this stops an infinite loop
  // NextJS seems to be smart enough to avoid redirects to localhost but we cant use localhost in docker 
  let path_prefix = url.pathname.substring(1)
  path_prefix = path_prefix.substring(0, path_prefix.indexOf('/'))
  const domain = ['sitea', 'siteb', 'sitecc']
  if (!domain.includes(path_prefix)) {
    const local = `http://${process.env.HOSTNAME}:${process.env.PORT}`
    // console.log(local)
    const newURL = new URL(`/${host}${url.pathname}`, local);
    console.log(newURL)
    return NextResponse.rewrite(newURL);
  }


}

export const config = {
  matcher: "/((?!api/|_next/|_static/|_vercel|[\\w-]+\\.\\w+).*)",
};

Then I have multiple domain names pointing to the same server, NextJS doesn’t have the ability to route via domains so the middleware changes the request for https://sitea.com/path to https://localhost/sitea/path and then I can have a route like [site]

This is enough for our site to use the appropriate theme and the site specific content from our headless CMS.

Caveat - NextResponse

I have to admit that I’m not entirely clear what NextResponse is doing here - there seems to be some logic regarding whether it is an internal or external redirect and when to stop redirecting…

Gotcha 1 - localhost

In dev we used localhost as the internal redirect - but I moved to docker and realised that “localhost” refers to the host machine, and in Kubernetes the hostname is autogenerated. Kubernetes also uses different ports on teh pod internally from that exposed publicly.

Since Kubernetes exposes hostname and port as environment variables I found the best solution was to use these in Kubernetes and add them to .env files for docker and local dev (and use docker-compose to set a predictable hostname)

Gotcha 2 - infinite loop

Once I stopped using localhost I found the redirect caused an infinite loop - I figure there must be some logic in NextJS that detects a redirect to localhost (it also works for 127.0.0.1) and stops further redirects. But as above - localhost didn’t work for me and on moving to docker I found I was getting infinite(ish) loops and I got errors on urls like

http://mysite/sitea/sitea/sitea/sitea/sitea/sitea/sitea/sitea/sitea/sitea

The fix for that was to check if I already has a site prefix on my path and skip teh redirect if so

Gotcha 2 - Host headers

Locally and on docker I used the “host” header to determine which site was being redirected

In Kubernetes there is an Nginx TLS proxy which changes this header

I had to use the “x-forwarded-for” header (with the “host” header retained as a fallback for local dev)

Healthcheck

My kubernetes healthcheck doesn’t need to be aware of the multisite domains - so I added a route for a healthcheck page and made sure the middleware exits cleanly if a requests is made that isn’t any of the multisite domains.

Unfinished

This work isn’t complete and isn’t yet ready for production - we may hit more bugs or find more optimisations yet.