FaceForensics++, built in 2019 from 1,000 YouTube videos, is still the default training set. Bigger sets lean on paid actors or celebrity footage, and detectors trained on them pick up shortcuts, like who is in a clip or a split second of silence before a fake, instead of what was faked.

On video-call deepfakes at 270p with heavy H.264 compression, 14 detectors score 0.46–0.56 AUC. 0.5 is a coin flip. Part of our model runs on the device, where it sees the stream before it’s compressed.

Invisible perturbations make detectors miss fakes, even after a platform recompresses the file, and some turn out to be learning JPEG compression rather than fakery. Models trained only on real video, checking that lips match speech, avoid those shortcuts.

Trained on FaceForensics++, 14 detectors reach up to 0.86 AUC on Celeb-DF, a 2020 benchmark. On face swaps made with Deep-Live-Cam, a popular real-time tool, they score 0.40–0.70.

FaceForensics++, DFDC and Celeb-DF date from 2019–2020. DF40 and Celeb-DF++ add newer generators, but on deepfakes that circulated in 2024, open-source video detectors score 0.50–0.63 AUC, down from 0.87–0.96 on their own benchmarks.