Gemma 4 video understanding

Two videos were analysed by Gemma 4 12B running locally: a screencast and a meeting clip. Only the written results are shown here, not the videos or stills from them.

Names, contact details and the meeting's title have been removed from the meeting clip's results and are marked in square brackets.

Screencast: NotebookLM importing ten sources

Model
Gemma 4 12B (GGUF Q4_K_M, LM Studio, fully local)
Run
Local: LM Studio on an Apple M4 Max with 128 GB
Method
26 frames @ 1 FPS, 1024px wide; two passes (description + scene breakdown); scene boundaries verified frame-by-frame by Claude Opus 4.8

Scenes

  1. Source discovery (0 to 1 s): The notebook opens on NotebookLM's source-discovery results for the query “What are AI use policies are published…”. Ten sources on university AI-use policies are pre-selected, with the Import button ready. The Chat pane still shows the “Let's start your notebook…” welcome. On screen: 10 sources selected · Import · Let's start your notebook…
  2. Sources imported, notebook created (1 to 7 s): Import lands: a “10 sources added” toast confirms, the notebook home shows the title “Russel Group AI Use · 10 sources · 3 Jun 2026”, and a tooltip invites customising the notebook with a cover image. Source entries finish resolving in the left panel. On screen: 10 sources added · Add a cover image and custom note to personalise your notebook · 10 sources · 3 Jun 2026
  3. Overview summary starts generating (7 to 10 s): The Customise tooltip clears and the Chat pane begins composing the notebook overview — skeleton placeholder bars appear where the summary text will land.
  4. Studio tour: data table tooltip (10 to 14 s): Attention moves to the Studio panel. Hovering the Data tool raises the tooltip “Generate a data table from your sources”; the tile highlights as it is selected. On screen: Generate a data table from your sources
  5. Data table generating (14 to 24 s): The Data tool is activated: Studio shows “Generating data table… based on 10 sources”. The overview summary in Chat is still composing — both generations run in parallel. On screen: Generating data table… · based on 10 sources
  6. AI overview appears (24 to 25.75 s): The Chat pane renders the finished overview: academic integrity policies and misconduct procedures for Imperial College London, LSE, and the University of Manchester — academic malpractice, generative AI, contract cheating, plagiarism, and rules for referencing AI usage. The data table is still generating as the recording ends. On screen: academic integrity policies · misconduct procedures · referencing AI usage

Notes on the method and the result

  • 26 frames @ 1 FPS, 1024px wide, via `lmstudio_video.py` → LM Studio `/v1/chat/completions`.
  • Pass 1 prompt: "This is a screen recording. Describe what application and content is shown, what the user does, and how the screen changes over time. Read out any prominent on-screen text, titles, or headings you can see." (8,200 tokens total)
  • Pass 2 prompt: "This is a screen recording of a web app. Break it into scenes. For each scene give the approximate start time in seconds (frames are 1 per second), one line on what is visible, and one line on what changes or what the user does." (9,867 tokens total)
  • Scene boundaries in the deliverable were verified frame-by-frame by Claude Opus 4.8 (the model's own Pass-2 timings ran compressed — content accurate, timing unreliable).
  • Strong: on-screen text reading (notebook title, tooltips, toasts, source counts, university names), workflow comprehension, temporal ordering.
  • Weak: absolute timestamps (compressed ~26s into ~10s); needs external frame review or labelled-frame prompting for timing-critical work.

A meeting clip

Model
Gemma 4 12B (visuals) + Parakeet TDT 0.6B (speech), both fully local
Run
Local: LM Studio on an Apple M4 Max with 128 GB
Method
First 30s of a 2h Teams recording. Visuals: 30 frames @ 1 FPS, 1024px, Gemma 4 12B via LM Studio. Speech: Parakeet TDT 0.6B v3 (MLX) timestamped transcript. Scene boundaries verified frame-by-frame by Claude Opus 4.8.

Scenes

  1. Teams recording title card (0 to 2 s): The recording opens on the dark Microsoft Teams title card for the session: “[session title removed]”, dated [date removed], recorded by [name removed], organised by the [organiser removed]. On screen: [session title removed] · [date removed]
  2. Title slide, welcome (2 to 16 s): The shared screen switches to the opening slide: a robot figure over [place removed] rooftops with “[session title removed]” on a dark blue panel, [organisation removed] and [organisation removed] logos top-left, contact email below. The Teams participant gallery lines the right edge. The presenter welcomes everybody to the session. On screen: [session title removed] · [email removed]
  3. Section slide: getting to know each other (16 to 17 s): A brief interstitial slide — “Getting to know each other” on a plain light background — lands exactly as the presenter says he wants to start by getting to know each other. On screen: Getting to know each other
  4. Presenter introduction slide (17 to 30 s): The presenter slide appears: [name removed], [job title removed] at the [organisation removed], with email and [website removed], alongside the [lab name removed] graphic. The presenter introduces his role and starts on his background as the clip ends. On screen: [name removed] · [job title removed] · [lab name removed]

Notes on the method and the result

  • Clip: `ffmpeg -t 30` re-encode to 1280px / CRF 26 / AAC 64k mono (478 KB).
  • Visuals: 30 frames @ 1 FPS, 1024px → `lmstudio_video.py` → LM Studio. Prompt: "This is the opening of a recorded online meeting or webinar. Describe what is shown on screen: the application, layout, any slides or shared content, visible people, and how the screen changes over time. Read out any prominent on-screen text…" (8,698 tokens)
  • Speech: `speech-to-text` skill, `parakeet_to_srt.py` → 4 SRT segments.
  • Scene boundaries verified frame-by-frame by Claude Opus 4.8.
  • Gemma read the Teams title card metadata, slide titles, presenter details, and even participant names from the gallery thumbnails ([name removed], [name removed]).
  • Parakeet's timestamps and Gemma's visual scenes were produced independently and align naturally (slide change at ~16s lands on "getting to know each other").
  • Speech lane is verbatim incl. disfluencies ("a little bit above me, uh") — fine for sync, run cleanup prompts from the speech-to-text skill for publishable text.