University of Michigan · Computer Vision course project · Sept–Dec 2022

Applying an artistic style to a selected object

Interactive Segmented Style Transfer

A before-and-after sequence made from course-report images. It shows the selected swan changing style; it doesn't show the model running.Course report, Figure 2 · DAVIS585 / WikiArt

I built an application for selecting an object in a photo and applying an artistic style to it. The selected area changes while the surrounding image stays the same.

My role

I proposed the idea and led the design, implementation, model comparisons, and integration into the final application.

stylization models compared
4
sampled evaluation inputs
1,000

DAVIS585 / WikiArt image–mask–style combinations

AdaIN stylization time
4.92 ms

Mean of 600 transfers; excludes segmentation and compositing

Starting point

Selecting the area to change

The application uses clicks to select an object and refine its mask, which marks the area to edit. It then combines the styled object with the original background.

How it came together

  1. 01

    Select with clicks

    Positive and negative clicks guide FocalClick toward the object and away from areas to leave out, so users can adjust the mask without tracing the whole outline.

  2. 02

    Compare the models

    I compared AdaIN, AdaAttN, IEContraAST, and PAMA using ArtFID, then measured stylization time in a separate experiment.

  3. 03

    Keep the original background

    The mask determines which pixels come from the styled image and which come from the original.

Original: Reveal the original swan image and the selectively stylized AdaAttN result. Stylized result: Reveal the original swan image and the selectively stylized AdaAttN result. OriginalStylized object

Drag to compare the original image with the AdaAttN result. These are resized images from the report, shown here as a saved example of the output.
Progress presentation screenshot of a click-based object-selection interface, generated mask, and tower-styling example.
The selection interface, a mask, and a styling example from the progress presentation. I finished combining them into one application later.Original team presentation, 1:35 · DAVIS / WikiArt imagery

Comparing ArtFID and speed

AdaIN had the shortest measured stylization time. AdaAttN had the lowest ArtFID, at 27.419 compared with AdaIN's 27.955. Lower ArtFID is better on this measure.

Stylization time by model

Report measurements · 600 transfers per model · lower is faster

Mean stylization time

AdaIN

4.92 ms

PAMA

9.28 ms

IEContraAST

10.19 ms

AdaAttN

37.36 ms

Bars share a zero baseline · scale 0–40 ms

The timings cover stylization only, excluding segmentation and compositing. The report doesn't specify the hardware or measure how long the full application takes.

The 1,000-input evaluation and timing chart come from the course report. The progress presentation shows an earlier experiment with different scores.

The screenshot shows the interface before I finished the combined application. The full application source hasn't been recovered; the code link points to the segmentation dependency.

Reports, code, and credits

Course reportProgress presentationSegmentation dependency

Course team: Le Qin, Yu Peng, Christopher Yeh, and Yu-Ju Chiu.

I used existing FocalClick and neural style-transfer implementations. My work covered the interaction, integration, and evaluation.

Image examples use DAVIS585 and WikiArt material credited in the original report.

Next projectShowing sound direction with LEDs