AI Video Auto-Cut in Python: YOLO Detection + Whisper Captions

Scrubbing through 30 minutes of raw footage to find 40 seconds worth posting is the slowest part of editing. A computer can't judge taste, but it can do the first pass: find the moments where your subject is in frame and moving, line them up with what's being said, and cut rough clips with captions already burned in.

This guide builds that first pass in Python with three open tools: Ultralytics YOLO for object detection, OpenAI Whisper for word-level transcription, and ffmpeg for cutting and captioning. The code runs on a local GPU or on a rented one. We haven't published speed benchmarks for this exact script, so treat the timing notes as guidance and measure on your own footage.

What this pipeline does and doesn't do

It does:

• sample frames and score each second by how much of your target subject is in frame and how much it moves;
• transcribe speech with per-word start and end times;
• pick the highest-scoring windows, snap their edges so no word is cut in half, and drop overlaps;
• export each window as an MP4 with burned-in captions, optionally center-cropped to 9:16.

It doesn't understand stories, punchlines, or whether a trick landed. YOLO detects objects. It doesn't recognize actions. "A person and a bicycle are large in frame and moving fast" is a decent proxy for riding footage. It's a poor proxy for a cooking video. Plan to review every clip, and expect to throw some away.

DIY script or an editing app?

If you shoot riding footage specifically, our BMX Clip Extractor takes a different approach to the same problem. It uses a local Qwen vision model to find active riding segments instead of object boxes. The script below is the general-purpose version you can adapt to any subject.

What you need

Hardware. Both models run on CPU, but slowly. An NVIDIA GPU with 6 GB or more of VRAM makes this practical. Whisper's README lists approximate VRAM needs per model:

turbo is an optimized large-v3 with a minimal accuracy loss, which makes it a strong default if you have the memory. For English-only footage, the .en variants (small.en, medium.en) tend to do slightly better at the same size.

YOLO model. The current Ultralytics family is YOLO26. Figures from the Ultralytics README (COCO validation, 640-pixel input):

yolo26s is a sensible starting point: noticeably more accurate than nano and still fast. Move to m if distant subjects get missed.

Licensing. Ultralytics YOLO is released under AGPL-3.0, with a separate enterprise license for commercial use. Using it for your own editing is one thing. Shipping it inside a product or hosted service is another, so read the license before you do that.

Software.

pip install ultralytics openai-whisper opencv-python
sudo apt install ffmpeg        # macOS: brew install ffmpeg; Windows: winget install Gyan.FFmpeg
ffmpeg -hide_banner -filters | grep subtitles   # must list the subtitles filter

The caption step uses ffmpeg's subtitles filter. The ffmpeg filter documentation notes it's only available in builds configured with libass. Most packaged builds include it, but check before you debug anything else.

Step 1: Score every second with YOLO

# autocut.py
import cv2
from collections import defaultdict
from ultralytics import YOLO

def score_video(path, model_name="yolo26s.pt", classes=("person", "bicycle"),
                every_n=3, conf=0.35):
    """Return {second: score}. Higher = subject larger in frame and moving more."""
    model = YOLO(model_name)
    wanted = {i for i, n in model.names.items() if n in classes} if classes else None
    cap = cv2.VideoCapture(path)
    fps = cap.get(cv2.CAP_PROP_FPS) or 30.0
    per_second = defaultdict(float)
    prev, idx = [], 0
    while cap.grab():                       # grab is cheap; decode only sampled frames
        if idx % every_n == 0:
            ok, frame = cap.retrieve()
            if not ok:
                break
            h, w = frame.shape[:2]
            result = model(frame, conf=conf, verbose=False)[0]
            score, centers = 0.0, []
            for box in result.boxes:
                cls = int(box.cls[0])
                if wanted is not None and cls not in wanted:
                    continue
                x1, y1, x2, y2 = box.xyxy[0].tolist()
                area = (x2 - x1) * (y2 - y1) / (w * h)
                score += float(box.conf[0]) * (0.5 + area)
                centers.append(((x1 + x2) / 2 / w, (y1 + y2) / 2 / h))
            if centers and prev:            # motion: distance to nearest previous detection
                score += 5 * sum(min(abs(cx - px) + abs(cy - py) for px, py in prev)
                                 for cx, cy in centers)
            prev = centers
            per_second[int(idx / fps)] += score
        idx += 1
    cap.release()
    return dict(per_second)

Three design choices matter here. Sampling every third frame cuts detection work by two thirds with little loss, because action rarely lasts less than a tenth of a second. Box area rewards shots where the subject fills the frame over specks in the background. And the motion term, which measures how far detections moved since the last sampled frame, separates riding from standing around.

The class names come from the COCO dataset the stock models were trained on, which includes person, bicycle, skateboard, surfboard, skis, snowboard, sports ball, dog, and car. Pass classes=None to score everything.

Step 2: Transcribe with word timestamps

import whisper

def transcribe_words(path, model_size="small"):
    model = whisper.load_model(model_size)
    result = model.transcribe(path, word_timestamps=True)
    return [{"word": w["word"].strip(), "start": w["start"], "end": w["end"]}
            for seg in result["segments"] for w in seg.get("words", [])]

word_timestamps=True is what makes clean captions and clean cut points possible. Whisper reads audio straight from the video file through ffmpeg, so there's no separate extraction step. For a full walkthrough of Whisper setup, model choice, and accuracy tips, see How to Run Whisper Locally for Free Transcription.

Step 3: Pick the highlight windows

def pick_windows(per_second, words, top_n=8, min_len=4.0, max_len=12.0,
                 lead_in=1.0, speech_weight=0.5):
    if not per_second:
        return []
    duration = max(per_second) + 1
    speech = defaultdict(int)
    for w in words:
        speech[int(w["start"])] += 1
    # 3-second rolling action score plus a small bonus for speech
    score = {t: sum(per_second.get(t + k, 0) for k in (-1, 0, 1))
                + speech_weight * speech.get(t, 0)
             for t in range(duration)}
    chosen = []
    for t in sorted(score, key=score.get, reverse=True):
        if len(chosen) >= top_n or score[t] <= 0:
            break
        start = max(0.0, t - lead_in)
        end = min(float(duration), start + max_len)
        for w in words:                     # never cut a word in half
            if w["start"] < start < w["end"]:
                start = w["start"]
            if w["start"] < end < w["end"]:
                end = w["end"]
        if end - start < min_len or any(start < e and end > s for s, e in chosen):
            continue
        chosen.append((round(start, 2), round(end, 2)))
    return sorted(chosen)

The rolling sum stops a single noisy frame from winning, and the lead-in gives viewers a beat of setup before the action. The speech bonus is deliberately small. Raise speech_weight for vlogs, where talking is the content. Set it to zero for silent action footage.

Step 4: Export clips with burned-in captions

import pathlib
import subprocess

def srt_time(t):
    ms = int(round(t * 1000))
    h, ms = divmod(ms, 3_600_000)
    m, ms = divmod(ms, 60_000)
    s, ms = divmod(ms, 1000)
    return f"{h:02}:{m:02}:{s:02},{ms:03}"

def write_srt(words, start, end, path, per_line=4):
    clip = [w for w in words if w["start"] >= start and w["end"] <= end]
    cues = []
    for i in range(0, len(clip), per_line):
        chunk = clip[i:i + per_line]
        cues.append(f"{i // per_line + 1}\n"
                    f"{srt_time(chunk[0]['start'] - start)} --> {srt_time(chunk[-1]['end'] - start)}\n"
                    f"{' '.join(w['word'] for w in chunk)}\n")
    pathlib.Path(path).write_text("\n".join(cues), encoding="utf-8")
    return bool(cues)

def export(src, windows, words, out_dir="clips", vertical=False):
    out = pathlib.Path(out_dir)
    out.mkdir(exist_ok=True)
    for n, (start, end) in enumerate(windows, 1):
        srt = out / f"clip_{n:02}.srt"
        filters = ["crop=trunc(ih*9/16/2)*2:ih"] if vertical else []
        if write_srt(words, start, end, srt):
            filters.append(f"subtitles={srt.as_posix()}:force_style='Fontsize=18,Outline=2'")
        cmd = ["ffmpeg", "-y", "-ss", str(start), "-i", src, "-t", str(round(end - start, 2))]
        if filters:
            cmd += ["-vf", ",".join(filters)]
        cmd += ["-c:v", "libx264", "-crf", "18", "-preset", "fast", "-c:a", "aac",
                str(out / f"clip_{n:02}.mp4")]
        subprocess.run(cmd, check=True)

if __name__ == "__main__":
    import sys
    src = sys.argv[1]
    scores = score_video(src)
    words = transcribe_words(src)
    windows = pick_windows(scores, words)
    export(src, windows, words, vertical="--vertical" in sys.argv)

Run it with python autocut.py raw_footage.mp4 --vertical.

Putting -ss before -i makes ffmpeg seek quickly and reset the clip's timestamps to zero, which is why the SRT times are written relative to each clip's start. The script re-encodes with libx264 instead of stream-copying. Copying with -c copy is faster, but cuts can only land on keyframes, so clips often start early or open on a frozen frame. Burning in captions requires a re-encode anyway.

The vertical crop takes the center 9:16 slice and rounds the width to an even number, because libx264 rejects odd dimensions. If your subject isn't centered, that crop will miss them. Use the YOLO box centers from Step 1 to drive a moving crop once the basic version works.

Adapting it to other footage

The defaults above are tuned for riding footage. Different subjects need different signals:

For talking-head content, a better approach is to send the word-level transcript to a language model and ask it to return start and end times for the strongest self-contained segments. Then reuse export() unchanged. The cutting and captioning code doesn't care where the windows came from.

Make review fast

The script's output is a set of candidates, and review is where your time goes. Two small additions pay for themselves:

• Log every window. Write each window's start, end, score, and transcript snippet to a CSV next to the clips. Sorting by score shows quickly whether the heuristic is working, and the timestamps let you jump straight to the source if a clip needs a wider cut.
• Over-generate, then cut. Ask for 12 windows when you need 6. Discarding weak clips takes seconds. Going back into raw footage for a missed moment takes minutes.

After a few sessions you'll know which parameters your footage needs, and the tuning table below becomes a short list of settings you actually change.

Tuning guide

Running it on a rented GPU

Detection and transcription are the expensive parts, and both scale with footage length. If your own machine is too slow, the same script runs unchanged on a cloud GPU. The Google Drive to cloud GPU rclone pipeline shows how to pull footage in, run the job, verify the output, and tear the instance down. How Remote GPU AI Video Editing Works covers the architecture.

Troubleshooting

No such filter: 'subtitles'. Your ffmpeg build lacks libass. Install a full build from your package manager or the ffmpeg download page, then rerun the filter check from the setup step.

Captions fail on Windows with a path error. The subtitles filter treats the colon in C:\ as an option separator. Run the script from your project folder so the SRT paths stay relative, as the code above does.

Every clip comes from the same few seconds of footage. One long, high-scoring stretch dominates. Lower max_len, or require a minimum gap between chosen windows.

Whisper uses the CPU even though you have a GPU. PyTorch was installed without CUDA support. Install the CUDA build of PyTorch for your driver version, then check that torch.cuda.is_available() returns True.

CUDA out-of-memory errors. YOLO and Whisper are both loaded on one small card. Run scoring and transcription as separate passes, or drop to yolo26n and Whisper base.

Clips start on a black or frozen frame. That's a keyframe problem from stream-copying. Keep the re-encode settings shown above.

FAQ

Can YOLO detect specific tricks or moves? Not out of the box. The stock models detect objects, not actions. You can train a custom model on labeled clips of your own footage, or use a vision-language model that describes frames, which is the approach behind our BMX Clip Extractor.

How long does a 30-minute video take to process? It depends on GPU, model sizes, every_n, and resolution. We haven't published a benchmark for this script. Time a five-minute sample first and scale up from there. Detection time grows roughly linearly with the number of sampled frames.

Does Whisper handle music-heavy footage? Loud music hurts transcription accuracy and can produce invented words during instrumental stretches. For music-backed edits, set speech_weight to 0 and skip burned-in captions, or separate the vocals first.

Is this good enough to post without review? No. Treat the output as a rough cut. The script finds candidates fast. Choosing what's actually good is still your job.

More creator workflows: AI Tools for BMX and Action Sports Content Creators and AI for Short-Form Video: TikTok, Reels, Shorts Strategy.