Computer Graphics Society

Art is inherently fragile. Across centuries, masterpieces are subjected to physical trauma, UV bleaching, humidity, and the inevitable yellowing of protective varnish (oxidation). Traditionally, restoring these works requires thousands of hours of painstaking labor by highly trained conservators using chemical solvents and microscopic tools.

While researching applied deep learning techniques within the Computer Graphics Society, a compelling question emerged: Can we digitally reverse centuries of degradation using AI, and can we do it without relying on massive, expensive cloud compute clusters?

Today, I am open-sourcing the result of that research: an end-to-end, multi-stage AI pipeline for classical art restoration, engineered to run entirely offline on a standard 6GB consumer GPU.

🛑 The Problem with "One-Size-Fits-All" AI

When approaching image restoration, the immediate temptation is to throw a massive generative model (like Stable Diffusion) at the problem. However, classical art restoration has a strict constraint: authenticity.

A generalized diffusion model doesn't just restore; it hallucinates. It replaces original brushstrokes with its own statistical approximations. Furthermore, a single end-to-end model struggles to simultaneously understand large structural damage (tears) and high-frequency microscopic details (canvas grain).

To solve this, I moved away from the "black box" approach and engineered a deterministic, sequential 4-stage assembly line where specialized neural networks handle specific types of degradation.

⚙️ The Multi-Model Architecture

The pipeline processes images sequentially, passing the computational graph from one specialized network to the next.

Stage 0: Structural Inpainting (LaMa)

The Problem: Physical canvas tears, deep craquelure (cracking), and missing paint chunks.
The Solution: We utilize LaMa (Large Mask Inpainting). Unlike standard CNNs that suffer from limited receptive fields, LaMa uses Fast Fourier Convolutions (FFCs). This allows the network to "see" the entire image globally, making it incredibly effective at understanding and replicating large, repetitive structures like canvas weave.
The Engineering: To process high-resolution art, I implemented a mathematical tiling function. The image is processed in 512x512 chunks with a 32-pixel overlap, seamlessly blending the structural repairs without exhausting spatial memory.

Stage 1: Color & Light Correction (Custom U-Net)

The Problem: Varnish oxidation (which turns paintings yellow/brown) and UV fading.
The Solution: I trained a custom ResNet34-backed U-Net to act as a digital solvent. The network was trained using a highly specific dual-loss function:
- L1 Loss: Ensures mathematical color accuracy to reverse the yellow tint.
- VGG19 Perceptual Loss: Evaluates the image at a feature level. This forces the U-Net to preserve the artist's authentic brushstrokes rather than smoothing them out into digital plastic.

Stage 2: Super-Resolution (Real-ESRGAN)

The Problem: After structural and color repair, the image often lacks high-frequency crispness, especially when scaled up for modern displays.
The Solution: Real-ESRGAN acts as a digital magnifying glass. It upscales the repaired image by 400%, intelligently hallucinating micro-textures, canvas grain, and paint thickness that were lost to time and low-resolution digitization.

Stage 3: Semantic Face Restoration (GFPGAN)

The Problem: Human visual perception is hyper-sensitive to faces (the uncanny valley effect). Even slight distortions introduced by earlier stages can ruin a classical portrait.
The Solution: The final pass uses Generative Facial Priors (GFPGAN). This model selectively detects facial geometry and reconstructs the features, ensuring the subjects look human, structurally sound, and historically accurate.

💻 The Engineering Bottleneck: VRAM Management

Designing the architecture was only half the battle. The real engineering challenge was running four massive state-of-the-art models on a standard consumer laptop with only 6GB of VRAM.

If you attempt to load LaMa, U-Net, Real-ESRGAN, and GFPGAN simultaneously, PyTorch will immediately throw an Out-Of-Memory (OOM) exception.

To bypass this hardware limitation, I implemented aggressive VRAM management techniques:

Sequential Graph Destruction: Models are loaded into memory, executed, and immediately deleted from the GPU using Python's del operator.
Forced Garbage Collection: Between every single stage, the pipeline calls gc.collect() and torch.cuda.empty_cache() to physically wipe the VRAM slate clean.
Memory Allocation Tuning: I configured the PyTorch CUDA allocator (max_split_size_mb:128) to prevent memory fragmentation during the heavy tensor operations in the ESRGAN upscaling phase.

🛡️ Breaking Free from the Cloud: 100% Offline Inference

During development, a critical flaw emerged. The external dependencies for Real-ESRGAN and GFPGAN (specifically the facexlib library) relied on hardcoded auto-downloaders that pulled weights from Hugging Face and GitHub.

When Hugging Face implemented new authentication tokens, the pipeline crashed with 401 Client Errors. Relying on third-party servers meant the tool was brittle and prone to breaking.

To make the project truly robust, I completely refactored the architecture for 100% offline inference:

I ripped out the auto-downloading logic from the wrapper classes.
I bundled the custom U-Net, the ESRGAN weights, and the facexlib detection models into a single, unified .zip archive.
I rewrote the path-handling logic to explicitly route all sub-libraries to a local checkpoints/ directory.

The result? The pipeline requires zero internet connection to run. No API keys, no server timeouts, no cloud compute bills. It is a completely self-contained local application.

📊 Evaluation and Next Steps

The pipeline utilizes Scikit-Image for rigorous mathematical evaluation against synthetically degraded ground-truth data, tracking both PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index) to ensure pixel and structural integrity.

Building this pipeline was a masterclass in balancing theoretical deep learning with brutal software engineering constraints. By focusing on hardware optimization and system architecture, we can democratize powerful AI tools, bringing research-grade capabilities to standard consumer hardware.

The entire project—including the inference scripts, custom wrappers, and pre-trained weights—is open-sourced.

🔗 Explore the repository, review the code, and try restoring an image yourself on GitHub: [https://github.com/devraj-1234/ArtiFact]