From video to scroll animation - DIY with ffmpeg, or free with vid2ssq

Whether the clip came from Kling, Veo, Seedance, After Effects, or a phone, the web problem is the same: scroll wants random access to frames. An MP4 wants a decoder and keyframes. That is why video.currentTime = progress * duration jumps - especially scrolling back, especially on iOS.

The industry pattern (Apple product pages included) is:

1. Turn the video into numbered stills.
2. Draw the current still on a <canvas> from scroll progress.
3. Serve a lighter set on phones.

You can do that by hand. Or you can generate a production bundle with Scrollsequence’s free local pipeline.

Part 1 - The generic DIY way

1. Extract frames with ffmpeg

# Example: ~24 fps stills, 1600px wide WebP
ffmpeg -i clip.mp4 -vf "fps=24,scale=1600:-1" frame_%04d.webp

Then thin the set. You rarely need every frame. A practical hero is often ~100–150 frames on desktop, fewer on mobile.

If you insist on scrubbing the MP4 anyway, re-encode with frequent keyframes first (-g 1 or -g 2). That helps seeking; it still will not match true frame sequences for reverse scrub and memory control.

2. Scrub on a canvas (vanilla idea)

<section id="pin" style="height: 300vh;">
  <canvas id="seq" style="position: sticky; top: 0; width: 100%; height: 100vh;"></canvas>
</section>

const frames = []; // preload Image() / createImageBitmap for each frame_%04d.webp
const canvas = document.getElementById('seq');
const ctx = canvas.getContext('2d');
let current = -1;

function draw(i) {
  const img = frames[i];
  if (!img) return;
  // size canvas to CSS * devicePixelRatio (cap at 2 on phones)
  ctx.drawImage(img, 0, 0, canvas.width, canvas.height);
}

function onScroll() {
  const el = document.getElementById('pin');
  const rect = el.getBoundingClientRect();
  const total = el.offsetHeight - window.innerHeight;
  const scrolled = Math.min(total, Math.max(0, -rect.top));
  const progress = total > 0 ? scrolled / total : 0;
  const i = Math.round(progress * (frames.length - 1));
  if (i === current) return;
  current = i;
  requestAnimationFrame(() => draw(i));
}

window.addEventListener('scroll', onScroll, { passive: true });

That is the whole “Apple-style” trick in spirit: progress → index → draw, only when the index changes.

3. What DIY usually forgets (and why demos lag)

You can wire GSAP ScrollTrigger instead of the raw scroll math - same idea: scrub a number from 0→1, map to frameIndex.

Part 2 - Free Scrollsequence path: npx scrollsequence vid2ssq

Scrollsequence v2 is built around the same model on every stack: Scene → Layer → Canvas. The asset side is a pipeline that turns raw .mp4 / .webm into multi-resolution folders plus a scrollsequence-manifest.json that React, vanilla JS, and WordPress all consume the same way.

Free vs optional AI (important)

From the Asset Pipeline docs and credits docs:

• 100% free / local: frame extraction, responsive resizing, format/quality, frame step, manifest generation (CLI uses local FFmpeg via Node / Wasm - no credits).
• Optional / API key + credits: --depth (3D depth maps), --track "sports car" (subject tracking), --vision (scene / camera / subject metadata for agents and automation).

So the default “video → scroll-ready bundle” path is free. Depth maps and vision are extras when you want WebGL tilt or agent-readable metadata.

Run the wizard

# Interactive wizard (beta package URL - see docs for current .tgz)
npx https://docs.scrollsequence.com/scrollsequence-2.0.0-beta.52.tgz vid2ssq

# Or if already installed:
npx scrollsequence vid2ssq

Non-interactive example (free local processing):

npx scrollsequence vid2ssq input.mp4 \
  --name hero-sequence \
  --sizes 1280,1920 \
  --format jpg \
  --quality 80

Optional AI (needs SSQ_API):

npx scrollsequence vid2ssq input.mp4 \
  --name car-reveal \
  --sizes 1280,1920 \
  --depth \
  --track "sports car" \
  --vision

What you get that hand-rolled ffmpeg usually skips

1. Multiple size tiers (e.g. 1280 + 1920) so phones are not force-fed desktop frames. The runtime picks a tier from canvas CSS width × devicePixelRatio.
2. scrollsequence-manifest.json - one file describing paths, frame counts, optional files.d depth maps, tracking, and vision.
3. Depth maps (optional) - grayscale per frame for WebGL 2.5D mouse / gyro tilt (shared lower-res depth across color tiers).
4. Tracking + vision (optional) - bounding data and scene/camera/subject JSON so an AI agent (or your own UI) can pin labels to a moving subject or reason about the shot.
5. Same output for WordPress, React, and vanilla - no separate “Elementor-only” asset format.

Inside Next.js / Vite, if a public/ folder exists, the CLI writes straight to public/<name>/ so the dev server can serve frames immediately.

Drop the manifest into the page

Vanilla (free engine pattern):

<div class="scrollsequence-scene" data-ssq-duration="1500" data-ssq-scrub="1">
  <div class="scrollsequence-scene_inner">
    <div class="scrollsequence-layer">
      <div
        class="scrollsequence-canvas2d"
        data-ssq-media-groups='[{
          "type": "manifest",
          "rootDir": "/hero-sequence/",
          "imageSizes": { /* from scrollsequence-manifest.json */ }
        }]'
      >
        <canvas></canvas>
      </div>
    </div>
  </div>
</div>

<script type="module">
  import ScrollsequenceManager from 'scrollsequence';
  import 'scrollsequence/scrollsequence.css';
  ScrollsequenceManager.initialize().preloadAll();
</script>

React / Next.js:

'use client';
import { Scene, Layer, Canvas2d } from 'scrollsequence/react';
import manifest from '@/public/hero-sequence/scrollsequence-manifest.json';

export default function Hero() {
  return (
    <Scene duration={1500}>
      <Layer>
        <Canvas2d
          mediaGroups={[{
            type: 'manifest',
            rootDir: '/hero-sequence/',
            imageSizes: manifest.imageSizes,
          }]}
        />
      </Layer>
    </Scene>
  );
}

WordPress v2 uses the same Canvas 2D + Media Tools vid2ssq flow (basic extraction free in-browser; AI toggles credit-based).

Install note (public beta): package currently installs from the docs .tgz until the npm name settles — see docs.scrollsequence.com/v2.

Which path should you use?

DIY teaches the mechanism. vid2ssq is the same mechanism packaged so you are not reinventing responsive tiers and a manifest every project — and the basic convert stays free.

Docs: Asset Pipeline & CLI · Vanilla · React · WordPress

Draft for Antigravity — based on Scrollsequence v2 docs (asset pipeline, vanilla, react) provided Oct 2026.