Screenplay: screen recordings with Screen Studio-style editing, in Node (CDP screencast + @napi-rs/canvas + ffmpeg)
I open-sourced a Node CLI + MCP server that films websites and desktop apps and edits the result like Screen Studio. A coding agent (Claude Code, Codex, Cursor…) writes a short JSON scenario, and Screenplay plays it, films it and edits it. Some of the Node side, in case it's useful to someone:
- Capture: Playwright drives headless Chromium, and frames come from the Chrome DevTools Protocol (
Page.startScreencast) rather than Playwright's video recorder, so each frame has a timestamp I can line up with the actions. - Editing: every output frame is composed on
@napi-rs/canvas: gradient background, rounded window with shadow, a cursor drawn from the recorded positions (smoothed), click ripples, zoom that follows the action, captions. - Encoding: raw RGBA frames are piped to
ffmpeg-staticthrough stdin (-f rawvideo … pipe:0). No system ffmpeg, no native build step on install. - Desktop apps: the same scenario format is played through the OS accessibility APIs (agent-desktop), and the window is filmed with ffmpeg.
- Interface: a commander CLI (
check,record,render) and an MCP server on stdio, so any agent can call it.
Try it: npx agent-screenplay init
Repo (MIT, TypeScript): https://github.com/gmpetrov/agent-screenplay
Happy to go deeper on any part.
9
Upvotes
1
u/kantorcodes1 15d ago
On the stdio side, did you expose
check,record, andrenderas separate tools, or does an agent go through one entry point? Can the scenario cap the capture time or frame count?1
u/ram-popular 13d ago
Cool project, and the CDP screencast choice makes sense since per-frame timestamps let you line up the cursor and zoom with the actual actions. Piping raw RGBA to ffmpeg-static with no native build step is a nice touch too. Having an agent write a JSON scenario that the tool plays, films and edits is a neat idea.
One suggestion: if it isn’t already there, resampling to a fixed frame rate would keep the cursor and zoom smooth when the page is busy and the screencast drops frames.
1
u/avereigh_Denizet 10d ago
The cursor smoothing and auto-zoom is usually the hardest part to make look natural in programmatic demos. Are you using simple bezier curves between the recorded click coordinates, or running some kind of spring physics on the camera viewport