logo Idiap Research Institute        
 [BibTeX] [Marc21]
Vulnerability of Automatic Identity Recognition to Audio-Visual Deepfakes
Type of publication: Conference paper
Citation: Korshunov_IJCB_2023
Publication status: Accepted
Booktitle: IEEE International Joint Conference on Biometrics
Year: 2023
Month: September
Abstract: The task of deepfakes detection is far from being solved by speech or vision researchers. Several publicly available databases of fake synthetic video and speech were built to aid the development of detection methods. However, existing databases either focus on visual or voice modalities. Also, the databases rarely contain or any evidence is shown that the resulted deepfakes realistically impersonate any real person. In this paper, we present the first realistic audio-visual database of deepfakes SWAN-DF, where lips and speech are well synchronized and video have high visual and audio qualities. We took the publicly available SWAN dataset of real videos with different identities to create audio-visual deepfakes using several models from DeepFaceLab and blending techniques for face swapping and HiFiVC, DiffVC, YourTTS, and FreeVC models for voice conversion. From the publicly available speech dataset LibriTTS, we also created a separate database of only audio deepfakes LibriTTS-DF using several latest text to speech methods: YourTTS, Adaspeech, and TorToiSe. We demonstrate the vulnerability of a state of the art speaker recognition system, such as ECAPA-TDNN-based model from SpeechBrain, to the synthetic voices. Similarly, we tested face recognition system based on the MobileFaceNet architecture to several variants of our visual deepfakes. The vulnerability assessment show that by tuning the existing pretrained deepfake models to specific identities, one can successfully spoof the face and speaker recognition systems in more than 90% of the time and achieve a very realistic looking and sounding fake video of a given person.
Keywords:
Projects Idiap
NAST
Biometrics Center
Authors Korshunov, Pavel
Chen, Haolin
Garner, Philip N.
Marcel, S├ębastien
Added by: [UNK]
Total mark: 0
Attachments
  • Korshunov_IJCB_2023.pdf
       (Accepted version)
Notes