Initial commit

.gitignore

+
+dataset/
+
+**/*.jpg 
+**/*.png 
+!imgs/*.jpg
+!imgs/*.png
\ No newline at end of file

README.md

+<div align="center">
+<img src="./imgs/before_after1.jpg" width="40%">
+
+<h1 align="center"> Synthetic to Authentic </h1> (ECCVw24 Oral)
+
+[Parsa Rahimi](https://parsa-ra.github.io/)<sup>1,2</sup>
+[Behrooz Razeghi](https://www.behroozrazeghi.com/)<sup>2</sup>
+[Sébastien Marcel](https://www.idiap.ch/~marcel/professional/Welcome.html)<sup>2</sup>
+
+<sup>1</sup> EPFL, <sup>2</sup> Idiap
+
+[![Project page](https://img.shields.io/badge/Project-Page-blue.svg)](https://idiap.ch/syn2uath)
+[![Paper](https://img.shields.io/badge/Paper-Arxiv.2407.07627-B31B1B.svg)](https://arxiv.org/abs/2407.07627)
+
+</div>
+
+This is the code for our paper [Syn2Auth](https://arxiv.org/abs/2407.07627).
+In this paper we studied the possibility of using different Image-to-Image transfer methodologies for making the images in the 3D rendered datasets (e.g., [DigiFace1M](https://github.com/microsoft/DigiFace1M)) look more realistic while boosting the performance of the FR system which trained on the more realistic images. This is done without using any identity preservation prior (e.g., another strong FR system) or large datasets used for training FR systems.
+
+<div align="center">
+<img src="imgs/overview.drawio.png" width="70%" align="center">
+</div>
+
+# News
+
+- [2024/08/10] Our paper got accepted as oral presentation in ECCVw24.  
+
+<!-- # TODO
+
+- [ ] Add code for keypoint injection as a soft FR prior. -->
+
+
+# General Setup
+```bash
+    conda create -f env.yaml
+    conda activate syn2auth
+```
+Populate the paths in `paths.sh` to the corresponding location in your filesystem and run.
+```bash
+. paths.sh
+``` 
+to source them.
+
+# Transfer Methodologies.
+We mainly studied the following methodologies from literature.
+* [CodeFormer](https://github.com/sczhou/CodeFormer.git)
+* [VSAIT](https://github.com/facebookresearch/vsait.git)
+* [DECENT](https://github.com/Mid-Push/Decent.git)
+* [UNSB](https://github.com/cyclomon/UNSB)
+
+For generating datasets and train different methodologies please clone the corresponding repositories.
+
+## RealDigiFace
+This dataset is generated by simply passing the images of [DigiFace1M](https://github.com/microsoft/DigiFace1M) to the methods that mentioned above, specifically using the CodeFormer and VSAIT that we report in the paper. 
+```
+ DigiFace1M -> F -> RealDigiFace
+```
+In the `third_party` folder for each of the mentioned methods there exist a folder in which after cloning the original repository you need to override them with the content of these folder, These contains slight modifications of the original repository to generate the being  able to translate the original images in the repository or measure the execution time.
+
+## Preparing Datasets for Training
+Please follow the [Dataset README.md](./datasets/README.md).
+
+Usually datasets for FR are presented in the following form. 
+
+Hierarchy structure (H1)
+```
+root_folder
+|_id1___pic1.png
+|  |____pic2.png
+|       .
+|       .
+|  |____pic_id1_count.png
+      .
+      .
+      .
+|_idk___pic1.png
+   |____pic2.png
+       .
+       .
+   |____pic_idk_count.png
+```  
+
+For example, using `grid_generator.py` you can generate sample grids of different variations of different identities. 
+```bash 
+  python3 grid_generator.py /path/to/digiface1m /path/to/realistic/version/of/digiface
+```
+
+<div align="center">
+<img src="imgs/iter_intra_class.png" width="40%">
+<img src="imgs/iter_intra_class_syn.png" width="40%">
+</div>
+
+
+## General Scripts: 
+This is used to resize the images generated from different translation methods to the target resolution which usually used for FR. 
+* `resize.py` Given an folder which contains images in different hierarchy, it find and resize all the images into a destination folder while preserving the hierarchy of the folders.
+  ```bash
+    resize.py /path/to/source/folder /path/to/dest/folder --size 112 112 
+  ```
+* `side_by_side.ipynb` Notebook used to create the canvas of the pictures from the different methods. Like Fig.3 in the paper. 
+* 
+
+# Face Recognition Experiments: 
+For face recognition experiments we used the modified version of the [AdaFace](https://github.com/mk-minchul/AdaFace) repository. Please follow the instructions in this repository for training your models. With this repository we can train a FR system using the image folder hierarchy format in which  folder corresponds to some samples of certain identities, and different identities are presented in separate folders.
+
+
+### Update
+We also include the `.rec` format to train your models.
+
+
+## Scripts:
+* `generate_fr_confidence.ipynb`: This notebook used to generate the tables for the IJB-C experiments, which given multiple experiments and their corresponding accuracies in different thresholds it aggregate and reports the mean and std of the accuracies.
+* `easy_benchmarks.ipynb`: This notebook used to generate the tables from the benchmarks LFW, CFPFP, CALFW, ... from different experiments and their corresponding reported accuracies.
+* `transferred_vs_orig`: Script use to study the effect of transfer in terms of the cosine similarity after and before applying realism.
+
+## Acknowledgment: 
+This project was funded under SAFER project from Hasler Foundation.
+We also thank the authors of the [DigiFace1M](https://github.com/microsoft/DigiFace1M), [AdaFace](https://github.com/mk-minchul/AdaFace), [CodeFormer](https://github.com/sczhou/CodeFormer.git), [VSAIT](https://github.com/facebookresearch/vsait.git), [DECENT](https://github.com/Mid-Push/Decent.git), [UNSB](https://github.com/cyclomon/UNSB) for sharing their amazing work.
+
+## Reference
+If you find this repository useful for your work, please consider citing our paper:
+```bibtex
+@article{rahimi2024synthetic,
+  title={Synthetic to Authentic: Transferring Realism to 3D Face Renderings for Boosting Face Recognition},
+  author={Rahimi, Parsa and Razeghi, Behrooz and Marcel, Sebastien},
+  journal={arXiv preprint arXiv:2407.07627},
+  year={2024}
+}
+```
+

configs/transferred_vs_orig.yaml

+path_key: img_path 
+
+close_threshold: 0.8 # If the cosine similarity whould be higher than this threshold some random images in this class will be saveds
+distant_threshold: 0.4 # vice versa of above.
+
+original: 
+  path: REALISM_BASE_PATH/datastes/adaface_ir50_webface4m_2023-09-11_00-05-38.hf
+  key: adaface_ir50_webface4m
+
+trans:
+  path: REALISM_BASE_PATH/third_party/CodeFormer/res/DigiFace-1M-2X/final_results/adaface_ir50_webface4m_2023-09-12_16-13-26.hf
+  key: adaface_ir50_webface4m
+
+nbins: 1000
\ No newline at end of file

datasets/README.md

+# Dataset Preparation
+Here you can find some scripts which facilitates training of the Image to Image translation methods which are usually trained in the form of CUT repository A -> B. 
+
+## Scripts
+* `copy_nested_into_one_folder.py`: Given a base folder which contains in order of hierarchy of images, it list all images and randomly select some subset of theses which in the paper we refer to them as shards. We used this script to create shards 5 shards of FFHQ dataset in the resolution of 128x128 as the source (A), and also 5 shards of DigiFace1M dataset with the same resolution to train Image to Image translation methodologies, namely, VSAIT, DECENT and UNSB. 
+
+* `create_symlinks.py`: For training our models to go from the source (A) to target (B) domain we usually need to have `test`, `val` and `train` sets, this scripts using the shards which created before randomly assigns the created shards to non-overlapping `test`, `val` and `train` set to create different combinations.
+

datasets/copy_nested_into_one_folder.py

+import os 
+import glob
+import shutil
+from argparse import ArgumentParser
+from multiprocessing import Pool, cpu_count
+from pathlib import Path
+import numpy as np 
+from PIL import Image
+
+parser = ArgumentParser()
+parser.add_argument("--base_folder")
+parser.add_argument("--output_folder")
+parser.add_argument("--limit", default=20000)
+parser.add_argument("--num_shards", default=5)
+parser.add_argument("--numpy_seed", default=10)
+parser.add_argument("--size", default=128, dtype=int)
+
+args = parser.parse_args()
+
+print(f"Setting numpy seed to {args.numpy_seed}")
+np.random.seed(int(args.numpy_seed))
+
+print(f"Creating the output folder: {args.output_folder} ... ")
+os.makedirs(args.output_folder, exist_ok=True)
+
+input_img_list = sorted(glob.glob(os.path.join(args.base_folder, '**/*.[jpJP][pnPN]*[gG]'), recursive=True))
+print(f"Found #{len(input_img_list)} images ... ")
+
+num_shards = args.num_shards
+if num_shards is None: 
+    print(f"Setting number of shards to 1")
+    num_shards = 1 
+
+for shard_num in range(int(num_shards)):
+    print(f"Working on shard number {shard_num} ... ")
+    if args.limit is not None: 
+        img_count_limit = int(args.limit)
+        print(f"Randomly selecting {img_count_limit} images ... ")
+
+        np.random.shuffle(input_img_list)
+        
+        input_img_list = input_img_list[:img_count_limit]
+
+
+    base_folder = args.base_folder
+    output_folder = args.output_folder 
+    output_folder = output_folder + f"_shard{shard_num}"
+    os.makedirs(output_folder, exist_ok=True)
+
+    def copy2base(file_path):
+        # Getting the filename
+        target_name = file_path[len(base_folder)+1:]
+        target_name = target_name.replace('/', "_")
+        target_path = Path(output_folder) / target_name
+
+        
+        Image.open(file_path).resize((args.size, args.size), Image.Resampling.LANCZOS).save(target_path)
+
+        #shutil.copy(file_path, target_path)
+
+    pool = Pool(max(cpu_count()//2, 1))
+
+    pool.map(
+        copy2base, 
+        input_img_list
+    )
\ No newline at end of file

datasets/create_symlinks.py

+import os 
+from subprocess import run 
+import numpy as np 
+
+base_path=os.getenv("REALISM_BASE_PATH")
+
+num_configs = 5 
+base_dir = f"{base_path}/datasets"
+config_base_name = "cutstyle_digi2ffhq"
+shards = [0,1,2,3,4]
+numpy_seed = 10
+
+shards_base_path = {
+ "A":f"{base_path}/datasets/digiface1m_shard",
+ "B":f"{base_path}/datasets/ffhq128x128_shard"
+}
+
+np.random.seed(numpy_seed)
+for i in range(num_configs):
+    print(f"Working on config {i}")
+    config_name = f"{config_base_name}/config_{i}"
+    cur_config_full_path = os.path.join(base_dir, config_name)
+    os.makedirs(cur_config_full_path, exist_ok=True)
+
+    for sec in ["A", "B"]:
+
+        picked_shards = np.random.choice(shards, 3, replace=False)
+        for picked_shard, part in zip(picked_shards, ["test", "train", "val"]):
+
+            source_link = shards_base_path[sec] + f"{picked_shard}/" # Note the `/` at the nd it is important
+            dest_path = os.path.join(base_dir, config_name)
+            os.makedirs(dest_path, exist_ok=True)
+            dest_path = os.path.join(dest_path, f"{part}{sec}")
+
+            run(
+                [
+                    "ln",
+                    "-s",
+                    source_link, 
+                    dest_path,
+                ]
+            )
\ No newline at end of file

datasets/creating_symlink.sh

+#!/bin/bash
+
+
+ln -s $REALISM_BASE_PATH/datasets/digiface1m_shard0/ $REALISM_BASE_PATH/datasets/cut_style_dataset/trainA
+
+ln -s $REALISM_BASE_PATH/datasets/digiface1m_shard1/ $REALISM_BASE_PATH/datasets/cut_style_dataset/valA
+
+ln -s $REALISM_BASE_PATH/datasets/digiface1m_shard2/ $REALISM_BASE_PATH/datasets/cut_style_dataset/testA
+
+ln -s $REALISM_BASE_PATH/datasets/ffhq128x128_shard0/ $REALISM_BASE_PATH/datasets/cut_style_dataset/trainB
+
+ln -s $REALISM_BASE_PATH/datasets/ffhq128x128_shard1/ $REALISM_BASE_PATH/datasets/cut_style_dataset/valB
+
+ln -s $REALISM_BASE_PATH/datasets/ffhq128x128_shard2/ $REALISM_BASE_PATH/datasets/cut_style_dataset/testB
+

datasets/download_cut_dataset.sh

+set -ex
+
+FILE=$1
+
+if [[ $FILE != "ae_photos" && $FILE != "apple2orange" && $FILE != "summer2winter_yosemite" &&  $FILE != "horse2zebra" && $FILE != "monet2photo" && $FILE != "cezanne2photo" && $FILE != "ukiyoe2photo" && $FILE != "vangogh2photo" && $FILE != "maps" && $FILE != "cityscapes" && $FILE != "facades" && $FILE != "iphone2dslr_flower" && $FILE != "mini" && $FILE != "mini_pix2pix" && $FILE != "mini_colorization" && $FILE != "grumpifycat" ]]; then
+    echo "Available datasets are: apple2orange, summer2winter_yosemite, horse2zebra, monet2photo, cezanne2photo, ukiyoe2photo, vangogh2photo, maps, cityscapes, facades, iphone2dslr_flower, ae_photos, grumpifycat"
+    exit 1
+fi
+
+if [[ $FILE == "cityscapes" ]]; then
+    echo "Due to license issue, we cannot provide the Cityscapes dataset from our repository. Please download the Cityscapes dataset from https://cityscapes-dataset.com, and use the script ./datasets/prepare_cityscapes_dataset.py."
+    echo "You need to download gtFine_trainvaltest.zip and leftImg8bit_trainvaltest.zip. For further instruction, please read ./datasets/prepare_cityscapes_dataset.py"
+    exit 1
+fi
+
+echo "Specified [$FILE]"
+URL=https://people.eecs.berkeley.edu/~taesung_park/CycleGAN/datasets/$FILE.zip
+ZIP_FILE=./datasets/$FILE.zip
+TARGET_DIR=./datasets/$FILE/
+wget --no-check-certificate -N $URL -O $ZIP_FILE
+mkdir $TARGET_DIR
+unzip $ZIP_FILE -d ./
+rm $ZIP_FILE

env.yaml

+name: syn2auth
+channels:
+  - conda-forge
+  - pytorch
+  - nvidia
+  - pytorch3d
+  - huggingface
+dependencies:
+  - python=3.10
+  - dlib
+  - lpips
+  - pydrive
+  - wandb
+  - pytorch-cuda=11.7
+  - pytorch-lightning
+  - torchvision
+  - python_abi
+  - pip
+  - numpy
+  - click
+  - pillow
+  - scipy
+  - dlib
+  - torchmetrics
+  - hydra-core 
+  - omegaconf
+  - datasets
+  - scikit-learn
+  - easydict
+  - pip: 
+    - pyyaml
+    - tqdm
+    - yapf
+    - lpips
+    - gdown
+    - python-json-logger
+    - pyrootutils
+    - pre-commit
+    - rich
+    - pytorch-metric-learning
+    - addict
+    - future
+    - lmdb
+    - numpy
+    - opencv-python
+    - Pillow
+    - pyyaml
+    - requests
\ No newline at end of file

grid_generator.py

+from PIL import Image
+import os
+import numpy as np 
+from pathlib import Path
+from argparse import ArgumentParser
+
+parser = ArgumentParser()
+
+parser.add_argument("dataset_base_orig", help="Path to the original dataset")
+parser.add_argument("dataset_base_syn", help="Path to the translated images")
+args = parser.parse_args()
+
+dataset_base_orig = args.dataset_base_orig 
+dataset_base_syn  = args.dataset_base_syn
+
+num_ids = 2
+num_var = 3
+img_sz = (224,224)
+offset = 1 # Pixels between each lines
+np.random.seed(23)
+
+output_img = Image.new('RGB', (img_sz[1]*num_var, (img_sz[0]+offset)*num_ids), (255,255,255))
+syn_output_img =  Image.new('RGB', (img_sz[1]*num_var, (img_sz[0]+offset)*num_ids), (255,255,255))
+
+selected_dirs = np.random.choice(os.listdir(dataset_base_orig), num_ids, replace=False)
+
+for id_idx, selected_dir in enumerate(selected_dirs): 
+    id_dir = os.path.join(dataset_base_orig, selected_dir)
+    syn_id_dir = os.path.join(dataset_base_syn, selected_dir)
+
+    img_samples = np.random.choice(os.listdir(id_dir), num_var, replace=False)
+
+    for img_idx, img_sample in enumerate(img_samples):
+        img_path = os.path.join(id_dir, img_sample)
+        syn_img_path = os.path.join(syn_id_dir, img_sample)
+        print(img_path)
+        print(syn_img_path)
+
+        output_img.paste(
+            Image.open(img_path).resize(img_sz),
+            (img_idx*img_sz[1],id_idx*(img_sz[0]+offset))
+        )
+
+        syn_output_img.paste(
+            Image.open(syn_img_path).resize(img_sz),
+            (img_idx*img_sz[1],id_idx*(img_sz[0]+offset))
+        )
+
+output_img.save('./iter_intra_class.png')
+syn_output_img.save('./iter_intra_class_syn.png')
\ No newline at end of file

img_list.py

+import glob 
+import os 
+
+input_path = os.getenv("REALISM_DATASET_DIGIFACE1M")
+
+input_img_list = sorted(glob.glob(os.path.join(input_path, '**/*.[jpJP][pnPN]*[gG]'), recursive=True))
+
+print(f"Found {len(input-input_img_list)} images")
+
+with open('./list.txt', 'w') as file:
+    file.writelines(input_img_list)
+