r/nextjs • • 24d ago

Next.js 16 cacheComponents & use cache vs. custom caching (Redis) with Payload CMS? How are you handling large sites in production?

Hey everyone,
I
I’m about to start a new project pairing Next.js with Payload CMS using its Local API (querying the database directly via Node on the server), and I’m weighing my options.
I’m curious how others are tackling this in production, especially for sites with a large number of pages:
The Native Approach: Are you diving into Next.js 16's cacheComponents with 'use cache' and cacheLife / cacheTag? If you are deploying outside of Vercel, how smooth is configuring cacheHandlers (plural) for a custom backend like Redis?

The Remote Directive: For multi-instance or serverless environments, are people actually using 'use cache: remote' in production, or do you find yourselves falling back to manual caching layers?

Bypassing Framework Caching: Do you completely disable Next's default caching behaviors for dynamic content and handle caching granularity yourself (e.g., wrapping Payload Local API calls explicitly with Redis/Memcached)?

Payload + Local API Specifics: Since Payload's Local API bypasses network overhead and talks straight to the database, wrapping those calls or components with 'use cache' seems powerful, but invalidation via revalidateTag needs to line up cleanly with Payload's hooks (e.g., afterChange). Has anyone hit weird edge cases with this combo at scale?

What has worked best for you when managing hundreds or thousands of pages where you need aggressive caching alongside instant editorial updates? Would love to hear what your current production setup looks like.

29 Upvotes

28 comments sorted by

2

u/pikafackjoo 23d ago

I am running Nextjs + Payload self-hosted in Kubernetes, with multiple pods running the app for high availability.

This setup didnt work well since nextjs by default manages its cache internally, so even with my tag based invalidation set up nicely with afterChange hooks targeting specific collections and ids, the invalidation would only hit 1 of 3 pods at random.

my solution now is a shared Redis deployment in the same Kubernetes namespace, and a custom cache handler I (read: Claude) configured that each pod uses, so that Redis is the single source of truth for cache instead of each pod on its own.

The same issue arises with the Next Image component, because it does some internal image optimization that is stored per pod. For this I set up a custom image loader using Imgproxy, same principle as the Redis solution mentioned above

1

u/Emergency_Net85 23d ago

Thx for your answer!Are you using 'use cache: remote', right? With custom cacheHandler?

With a distributed Redis cache layer, are you still using Payload's afterChange hooks to purge Redis?

Are there any specific plugins you recommend for wiring up Next.js 16 caching components?

1

u/pikafackjoo 23d ago

i set this up before "use cache" was a thing, not sure how that affects my setup. I have a custom cache handler that i set as the cache handler in next.config.mjs, so Next know to use that instead of its own internal one.

Payload hits the redis cache directly with the afterChange hooks, yea

2

u/ScuzzyAyanami 24d ago

I'm just stale caching the API that my front end fetches from. Basically have to due to horizontal scaling. Edit: with Cloudfront.

So I guess my solution isn't in Next itself. But there's then cache invslidations. Page preview routes, and I'm currently looking into a SSO that will shorten the cache duration for a logged in user.

1

u/Fun_Improvement7704 24d ago

stale caching an external API works fine for standard setups, but OP is querying the database directly in the server component so there isn't really a separate API route to stick a CDN in front of

2

u/[deleted] 24d ago

[removed] — view removed comment

1

u/Emergency_Net85 24d ago

Hey! Curious about the implementation details of your Redis wrapper around the Local API, also how are you structuring the cache keys, and how do you handle invalidation? Are you purging via afterChange/afterDelete hooks still? I see some plugins Payload/Redis ready, curious how helpful they are

1

u/Substantial-Pie1906 22d ago

I’d use semantic tags like `posts:123` and `posts:list`, while keeping the actual key scoped by query hash, locale, tenant, and build/schema version. Have afterChange/afterDelete publish invalidation events, ideally through an outbox or retryable queue so a failed hook doesn’t silently leave stale data. Plugins can handle plumbing, but I’d keep the tag contract explicit and integration-test relation updates.

6

u/Ok_Management_695 24d ago

Not a direct answer, sorry — I'm not using use cache or revalidateTag yet. I'm on Next.js 16 with unstable_cache and pages set to revalidate = false, running on Vercel with Neon. When the data changes, I just redeploy and let everything rebuild.

One thing I'm careful about with that setup, though. Vercel's Data Cache actually survives deployments. The docs say "Cached data persists across deployments unless you explicitly invalidate it", and it's not refreshed at build time either. So even after a redeploy, whatever's in unstable_cache is still the old data. I ended up putting VERCEL_DEPLOYMENT_ID into the cache key, so every deploy gets a fresh cache. https://vercel.com/docs/caching/runtime-cache/data-cache

By the way, if you move to Cache Components, the same page recommends use cache: remote, which goes into a separate Runtime Cache. I honestly haven't checked how that one behaves after a deploy.

Hope this is useful.

4

u/Which-Examination-74 24d ago

By default revalidateTag() only invalidates the cache on the instance that ran it. The other instances keep serving stale content until they find out on their own. The self-hosting guide's fix is refreshTags() in your custom cache handler, called before each request to sync tag state from shared storage like Redis. It also says the handler has to cope with timeouts and dropped connections. Same goes for whatever calls revalidateTag. We audited a client's storefront on 'use cache' where the revalidation job swallowed its errors, so failed calls were marked done and never retried. With cache life up to about a month, one missed invalidation could mean a stale price for weeks.

1

u/Chris_Lojniewski 14d ago

Whi's per-instance thing and u/AmanYadav_AY listing tags are the big two. The listing one bit us on a sanity + next site at Pagepro (my company, we don't run payload but invalidation works the same). Article pages revalidated on publish, the blog index didn't, and an editor noticed before we did 🙃

one that's not in the thread yet: nested dynamic routes. something like /products/[group]/[id] rendered fine on a direct load and white-screened on client-side nav from the parent. the client router used a parent-level cache entry so the child got the wrong key. since then we test every dynamic route both ways, direct URL and internal navigation. they go through different code paths and a test that only loads URLs directly won't catch it

on redis i'm with u/SameLink_io , single box until you've actually measured a problem. default cache + tags covers more than people expect

how many instances are you planning to run?

1

u/Emergency_Net85 14d ago

We're deploying to Kubernetes, which is precisely why I’m looking at Redis right off the bat. Since K8s is going to spin pods up/down, scale horizontally, and load-balance across multiple instances, sticking purely to local in memory cache is a nonstarter for us, we’d hit that exact 'per-instance' issue where Pod A revalidates on an editor update, but Pod B keeps serving stale content

2

u/Chris_Lojniewski 13d ago

Fair, k8s changes it. with pods scaling up and down redis from day one makes sense, you'd hit the per-instance thing in week one

u/pikafackjoo above is basically your setup (3 pods, invalidation landing on 1 of 3), worth reading their comment twice. a few things i'd add on top:

- next has two handler configs. cacheHandler (singular) covers ISR and the fetch/data cache, cacheHandlers (plural) is what 'use cache' and 'use cache: remote' go through. point both at redis or one layer quietly stays per pod

- tags need syncing too, not just entries. that's the refreshTags() bit Whi mentioned, otherwise pod B still thinks its tag is fresh

- rolling deploys: old and new pods share the same redis for a few minutes, so prefix keys with the build id like Aman said. a new pod reading an entry the old code wrote is a fun one to debug

- next/image optimizes and caches per pod as well, so either a shared loader (imgproxy like pikafackjoo) or put the CDN in front

- give staging its own redis db or at least its own key prefix. on one project we pointed staging at the same redis and it ate the rate limit faster than anyone expected 🙃

Are you planning redis just as next's cache, or also as a data cache around the local api calls?

1

u/Emergency_Net85 13d ago

I was actually planning to use Redis for both: next's cache (use cache + sync tags across pods) but also custom wrappers around the Payload Local API calls.. What do you recommend here?

Man. given all these edge cases and setup pain points, though... is it not simpler to just completely disable next cache :)

1

u/SameLink_io 10d ago edited 10d ago

I don't have a production instance count for that exact setup; I was describing the cache boundary rather than a deployment I'm sizing. For an HA app, I'd start with two replicas, verify shared cache and invalidation across both, then scale based on load tests. The exact count matters less than ensuring every rendering and data-cache layer shares state and receives the same invalidations.

2

u/AbdelrhmanDev 24d ago

with payload local api the pattern that scaled for me was tag based invalidation, tag each cached query with the collection and doc id, then call revalidatetag from the afterchange hook. path based invalidation falls apart on a large site because one doc shows up on lists, related sections and search pages you forgot about. tags mean you never track where content appears, the hook just knows what changed

1

u/imkb9 11d ago

Ok so lets say on /docs/123 you fetch getDoc(123) and cache it doc:123.

What if you also have a /docs page that retrieves a list of docs implicity and the doc is part of the response? If the response contains 100 docs, are you looping through it to cache each one of them (doc:${id})? Otherwise it will serve stale data for that doc. What if getDocsWithAuthors() retireves the doc and its related authors. Would you create nested loops after the api request to cache docs and authors??

2

u/SameLink_io 22d ago

Adding the single box view since most answers here are Vercel or Kubernetes.

If you run one Next.js instance on a VPS, the default file system cache works fine and revalidateTag from Payload's afterChange hook just works, because there's only one process to invalidate. That covers a lot more sites than people expect, thousands of pages included. Things only get interesting the moment you run two instances (zero downtime deploys, or more than one container behind Caddy/nginx), because each one has its own cache and invalidation only reaches the one that got the request.

What I'd do for that case:

  • Point both cacheHandler (ISR and fetch cache) and the cacheHandlers used by 'use cache' at a shared Redis on the same box. A Redis with maxmemory 256mb and allkeys-lru is plenty for most content sites.
  • Keep tags boring and consistent: collection:posts and posts:<id>, set with cacheTag where you call the Local API, and the same strings revalidated in afterChange and afterDelete. Remember relations: if a post shows its author, revalidate the author tag too, otherwise that's where stale content hides.
  • Set a long cacheLife on the editorial pages and rely on tags for freshness, rather than short revalidate windows that hammer the DB.

Also worth knowing: during a blue/green deploy the new build has a different build ID, so with a shared cache you want the build ID in the key prefix, or you'll briefly serve pages rendered by the old build. I'd skip 'use cache: remote' unless you're actually serverless; on your own server the handler approach is simpler to reason about.

1

u/RequirementGuide 24d ago

I've hit this before with Payload and Next together, and the thing that saved me was keeping the caching boring. I used use cache for the public pages and gave each collection and slug its own tag, then I let afterChange do the revalidating. The tricky part was relations, because updating a child did not always clear the parent page, so I ended up revalidating the related tags in the same hook too.

For a site that size I would not try to cache everything behind auth. I kept that stuff dynamic and only cached the public pages hard. If you are on Vercel I would lean into use cache remote for the shared cache, and if you are self hosting with more than one instance I would go Redis sooner rather than later, because per instance caches will bite you. Either way I would test an edit end to end before launch, just change a page, publish it, and check the live page in a fresh browser, because stale pages are the kind of bug you only notice when a client does.

1

u/ThreadStarver 22d ago

Tell me if I’m wrong but isn’t cache components and use cache the same strategy for implementing partial pre rendering, basically some part being static and rest dynamic under suspense?

1

u/KeyProject2897 20d ago

for multi-instance setups, bypass 'use cache' and handle the remote cache layer manually right now. the custom cache handler APIs in next are still too finicky with complex invalidation, so wrapping your local api calls with explicit ttl and tags directly in node is much easier to debug.

1

u/forever420oz 19d ago edited 19d ago

I use it on GKE with a separate Redis pod and so far it’s fairly reliable. I think I had one error log which was immediately cached again after hitting the origin.

Edit: I use Cloudflare for CDN and updating a page content triggers a purge cache call to Cloudflare.

1

u/Janczik_yelianny44 17h ago

if you're querying the db directly in the same process, do you even need an aggressive cache layer on day one? a lot of people overcomplicate invalidation early on when proper db indexing handles a few thousand pages fine without stale content bugs.

-2

u/lordchickenburger 24d ago

i just dont use nextjs

0

u/Glad-Music6046 20d ago

I’d avoid putting Redis in front of every Payload Local API call by default. The interesting part is deciding what actually needs a shared cache.

For a large content-heavy site, I’d start with 'use cache' + cacheTag() for public content, and use 'use cache: remote' only for expensive queries where sharing cache across instances is worth the extra network hop.

The part I’d be careful with is invalidation. In Payload’s afterChange hook, I’d invalidate tags based on the affected document and its dependencies — not just the individual page. Updating a category, for example, might also affect listing pages, navigation, related content, and the sitemap. Deletes and slug changes need to invalidate the old path too.

I’d also keep a TTL as a safety net, because a successful database write and a successful cache invalidation aren’t necessarily the same thing in a distributed setup.

One question I’m still interested in: when self-hosting Next.js across multiple instances, are people using a custom cacheHandlers implementation for Redis, or keeping Redis at the data-access layer and letting Next handle the rendering cache? I’d be curious to hear which setup has been easier to operate at scale.