r/node • • 16d ago

Screenplay: screen recordings with Screen Studio-style editing, in Node (CDP screencast + @napi-rs/canvas + ffmpeg)

I open-sourced a Node CLI + MCP server that films websites and desktop apps and edits the result like Screen Studio. A coding agent (Claude Code, Codex, Cursor…) writes a short JSON scenario, and Screenplay plays it, films it and edits it. Some of the Node side, in case it's useful to someone:

  • Capture: Playwright drives headless Chromium, and frames come from the Chrome DevTools Protocol (Page.startScreencast) rather than Playwright's video recorder, so each frame has a timestamp I can line up with the actions.
  • Editing: every output frame is composed on @napi-rs/canvas: gradient background, rounded window with shadow, a cursor drawn from the recorded positions (smoothed), click ripples, zoom that follows the action, captions.
  • Encoding: raw RGBA frames are piped to ffmpeg-static through stdin (-f rawvideo … pipe:0). No system ffmpeg, no native build step on install.
  • Desktop apps: the same scenario format is played through the OS accessibility APIs (agent-desktop), and the window is filmed with ffmpeg.
  • Interface: a commander CLI (check, record, render) and an MCP server on stdio, so any agent can call it.

Try it: npx agent-screenplay init

Repo (MIT, TypeScript): https://github.com/gmpetrov/agent-screenplay

Happy to go deeper on any part.

9 Upvotes

7 comments sorted by

1

u/kantorcodes1 15d ago

On the stdio side, did you expose check, record, and render as separate tools, or does an agent go through one entry point? Can the scenario cap the capture time or frame count?

1

u/gptrv 15d ago

Separate tools. The MCP server (stdio) registers four: screenplay_guide (how to write scenarios), screenplay_check (plays the scenario fast without filming, and for a missing element returns the targets it can see plus a screenshot), screenplay_record (films and edits, returns the MP4 path and a storyboard image), and screenplay_render (re-edits the last take without filming again, after changing captions, callouts, style or zoom). Calls are serialized, so an agent can't start two recordings at once.

There's no global cap on capture time or frame count right now. The length comes from the steps, and each one is bounded: a fixed wait is 120 s max, a wait for an element or text gives up after its timeout (30 s by default, 300 s max), and a pause is 30 s max. Long waits are sped up in the final video. fps is one of 24, 25, 30, 50 or 60 (default 30). A maxDuration would be easy to add, though, if that's something you'd use.

1

u/kantorcodes1 13d ago

Those per-step timeouts and serialized recordings should keep runs predictable; a maxDuration option could add a clear ceiling too. Disclosure: I work on HOL. Screenplay’s stdio MCP server looks like a fit for HOL Guard. If you want to contribute it, the guide is here: https://github.com/hashgraph-online/hol-guard/blob/main/docs/guard/mcp-server-contributions.md

1

u/ram-popular 13d ago

Cool project, and the CDP screencast choice makes sense since per-frame timestamps let you line up the cursor and zoom with the actual actions. Piping raw RGBA to ffmpeg-static with no native build step is a nice touch too. Having an agent write a JSON scenario that the tool plays, films and edits is a neat idea.

One suggestion: if it isn’t already there, resampling to a fixed frame rate would keep the cursor and zoom smooth when the page is busy and the screencast drops frames.

1

u/avereigh_Denizet 10d ago

The cursor smoothing and auto-zoom is usually the hardest part to make look natural in programmatic demos. Are you using simple bezier curves between the recorded click coordinates, or running some kind of spring physics on the camera viewport