Self-Hosting Payload CMS on a VPS with Coolify: A Field Guide

Most Payload CMS tutorials assume a fresh project and a managed platform. This one is different: I added Payload 3 to an existing Next.js 16 site, moved hosting from Vercel to my own VPS running Coolify, and wired it to a dedicated Postgres 17 database — pair-building the whole thing with an AI coding agent (Claude Code). Below is the exact path that worked, including the four gotchas that cost the most time, and the prompt templates I would reuse next time.

The target architecture

One VPS, one Coolify instance, three moving parts:

• App — Next.js 16 + Payload 3 in a single container, built by Coolify from a docker-compose.yaml in the repo, auto-deployed on every push to main.
• Database — a standalone Postgres 17 resource managed by Coolify, on the shared coolify Docker network.
• Media — a named volume mounted at /app/media so uploads survive redeploys.

Traefik (bundled with Coolify) handles routing and SSL. No nginx, no GitHub Actions — Coolify's GitHub App integration does the deploy-on-push part.

Step 1: Install Payload into an existing Next.js app

Use stable Payload 3.x, not the v4 canary — v3 supports Next 16 and React 19 and only needs Node ≥ 20:

npm install payload @payloadcms/next @payloadcms/ui \
  @payloadcms/db-postgres @payloadcms/richtext-lexical graphql sharp

Restructure src/app into two route groups: your existing site moves into (frontend) (including its layout.tsx and globals.css), and Payload's boilerplate lives in (payload) — admin panel, REST catch-all, GraphQL. Wrap next.config.mjs with withPayload and add output: 'standalone' for the Docker build.

Two things will silently break the Payload CLI if you skip them:

1. Add "type": "module" to package.json. Without it, payload generate:types dies with ERR_REQUIRE_ASYNC_MODULE because the CLI's loader can't require ESM modules that use top-level await.
2. Use relative imports inside payload.config.ts and your collections. The CLI does not read tsconfig path aliases, so @/collections/Users resolves in Next but crashes payload generate:types.

Your existing API routes are safe: a static route like /api/contact wins over Payload's /api/[...slug] catch-all under Next's route priority. Verify it with a curl after the restructure, not by assumption.

Step 2: Model content as Markdown, not rich text

Payload's Lexical editor is excellent for humans in the admin panel, but I write most content through an AI agent talking to the API. Editing Lexical's JSON AST programmatically is fragile; editing a Markdown string is trivial. So each collection's narrative body is a plain textarea field holding raw Markdown, rendered on the frontend with react-markdown + remark-gfm + rehype-highlight. Structured data that renders as UI components — stat tiles, tech stacks, figures — stays in typed Payload fields.

This post is itself a Markdown field in that schema.

Step 3: Local dev with a throwaway Postgres

A docker-compose.dev.yml with postgres:17-alpine bound to 127.0.0.1:5432, plus DATABASE_URL and PAYLOAD_SECRET in .env.local. Pin the same Postgres major version you will run in production. In dev mode Payload auto-creates the schema on boot (remember this detail — it becomes Gotcha #4).

Step 4: Containerize

A standard multi-stage Dockerfile: node:22-alpine, npm ci, build, then copy the standalone output into a runtime stage that runs as a non-root user with /app/media pre-created and chowned. Two non-obvious lines earn their keep:

# Fail the build if sharp's native binary is broken,
# instead of failing at the first image upload:
RUN node -e "require('sharp')"

And in every CMS-backed page:

// No database exists at build time on the build server.
export const dynamic = 'force-dynamic'

Without force-dynamic, next build tries to prerender pages by querying a database that isn't there, and the Docker build dies with payloadInitError.

Step 5: Wire up Coolify

The clicks, in order:

1. Sources → GitHub App → install it on the repo. This grants pull access and registers the auto-deploy webhook in one step.
2. New → Database → PostgreSQL 17. Note the container name (a UUID like o82fvcg…) and password.
3. New → Application → your repo, branch main, build pack Docker Compose.
4. Environment variables on the app: DATABASE_URL, PAYLOAD_SECRET (fresh secret, not your dev one), PAYLOAD_MEDIA_DIR=/app/media, NEXT_PUBLIC_SITE_URL.

Now the two Coolify-specific gotchas:

Gotcha #2 — the compose filename. Coolify looks for /docker-compose.yaml. If your file is docker-compose.yml, the deploy fails with "Docker Compose file not found". Rename it (or change the expected path in the app settings).

Gotcha #3 — the network. A Docker Compose app gets its own isolated network named after its UUID, while a standalone database lives on the shared coolify network. Result: your app cannot even resolve the database hostname — getaddrinfo fails, and a Payload site quietly renders as if it had no content. Fix: enable "Connect To Predefined Network" on the app, then use the database's container name as the host:

DATABASE_URL=postgresql://postgres:<password>@<db-container-name>:5432/postgres

You can verify from the VPS in ten seconds: docker exec <app> nc -z <db-container-name> 5432.

Step 6: The schema gotcha nobody tells you about

Gotcha #4: Payload's push: true schema sync only runs in development. Your production container runs with NODE_ENV=production, so the freshly created database stays empty and every request fails with Postgres error 42P01 ("relation does not exist") — which surfaced for me as a blank blog and a 500 on /admin/login.

Two ways out:

• Proper: generate migrations (payload migrate:create), commit them, run payload migrate on deploy.
• Pragmatic bootstrap: dump the schema your local dev DB already has and apply it to production:

docker exec local-db pg_dump -U dev -d app_dev \
  --schema-only --no-owner --no-privileges \
  | ssh vps "docker exec -i <db-container> psql -U postgres -d postgres"

I used the pragmatic route to go live, with migrations on the roadmap before any serious schema evolution. After the schema lands, /admin greets you with the create-first-user flow, and you are in business.

Prompt templates

The whole build was driven through an AI agent. These four prompts map to the phases that mattered; adapt the bracketed parts.

1. Research before touching anything:

Fetch current documentation for [Payload CMS] and plan its integration into my existing [Next.js 16] app at [repo]. Then inspect my VPS over SSH ([alias]) read-only: list running containers, check what's already installed (Coolify? reverse proxy?), and disk space. Produce a plan with the order of operations and the risks — do not modify anything yet.

2. Constrain the content model to the workflow:

I will author content mostly through [Claude Code / the API], not the admin UI. Choose field types accordingly — prefer plain Markdown over rich-text JSON for narrative bodies, keep structured data (stats, tags, figures) in typed fields, and explain the trade-off you are making.

3. Verify like production, before production:

Build the Docker image locally with no database available — it must succeed. Then run the container against a local Postgres and curl every route: pages, admin, API, sitemap. Show me status codes and grep the actual rendered content, not just 200s.

4. Debug the deploy from both sides:

The deployed app shows [symptom]. SSH into the VPS and diagnose end-to-end: app container logs, whether the app and database share a Docker network, whether the DB hostname resolves from inside the app container, and whether the schema exists. State the root cause before proposing a fix, and prefer fixes that don't require rebuilding.

The pattern behind all four: make the agent prove state (curl it, grep it, query it) instead of assuming success, and separate read-only investigation from mutation. Every real bug in this build — the network isolation, the missing schema — was found by a verification step, not by reading docs.

Closing checklist

• Payload 3 stable, route groups, withPayload, output: 'standalone'
• "type": "module" + relative imports in Payload config files
• Markdown bodies if an AI writes your content
• force-dynamic on CMS pages; sharp check in the Dockerfile
• Compose file named docker-compose.yaml for Coolify
• "Connect To Predefined Network" + DB container name as host
• Schema does not auto-create in production — migrate or sync it
• Fresh PAYLOAD_SECRET in production, never the dev one

Total time from empty repo to a live, self-hosted CMS with auto-deploy: one focused session. The VPS bill: unchanged, because it was already running Coolify for other projects — which is rather the point of self-hosting.

Postscript: I removed Payload from this blog

Shortly after finishing the setup above, I asked myself the question that should have come first: who actually writes here?

The answer was "me, through an AI agent editing files in my repo." Nobody else needs an admin panel. And once that is the honest answer, a database-backed CMS is solving a problem I do not have. Publishing became a git push anyway — the same push that triggers the deploy — so Payload's draft/publish workflow and version history were reimplementing what Git already gave me for free.

So this blog now runs on plain Markdown files in content/posts/, parsed with gray-matter and rendered by the same react-markdown pipeline. The post you are reading is one of those files. What that bought me:

• No database for content. The entire class of bugs in Gotcha #4 — schema drift, 42P01, migrations — simply stops existing.
• Static generation is back. With content in the repo at build time, pages prerender again instead of being forced to dynamic = 'force-dynamic' to dodge the missing-database problem.
• Content is diffable and reviewable. A post is a pull request.

None of which makes the guide above wrong. Payload is genuinely good, and everything documented here is what you need if you self-host it. The lesson is narrower and older than any framework: pick the storage that matches who edits the content. A CMS earns its complexity when non-technical people need a UI, when you need a real media library with transformations, or when several consumers read the same structured data over an API. For a solo developer writing Markdown with an AI agent, files win.

I would still reach for Payload on the next project that has actual editors. I just stopped paying for a feature I was the only user of.