AutoPhoto: Aesthetic Photo Capture using Reinforcement Learning

Authors: Hadi AlZayer, Hubert Lin, Kavita Bala · IROS 2021

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

An RL agent moves and turns a camera through an unknown indoor scene until a learned aesthetics model rates the view highly, then takes the photo; trained in Habitat, run on a Jackal UGV.

Why it matters

It shows that a learned, data-driven quality score can drive a robot's viewpoint search end to end, rather than hand-written framing rules tied to a chosen object. The same pattern of a learned scorer, a memory-based policy and a local adaptive threshold applies to other active-perception tasks such as inspection or real-estate photography, which the authors name as a use case.

Problem

Composing a good photo means moving and turning the camera until the view looks right, which takes practice. Prior automatic composition either crops a fixed image or, in drone cinematography, frames human subjects with heuristics such as the rule of thirds. A learned aesthetics score cannot be optimised analytically, and existing aesthetics models were trained to compare crops of one image, not views from different places.

Proposed solution

Photo capture is posed as a POMDP: the agent sees only the current camera view, moves forward or back 0.25 m, turns 10°, 30° or 90°, or takes the photo. An actor-critic policy with an LSTM memory is trained with PPO in AI Habitat on Gibson scenes, rewarded by the change in score from a ResNet18 aesthetics model trained on AVA and CPC with extra losses for robustness to small shifts and poor exposure. The capture reward is +1 only if the photo beats a local threshold set to the mean plus one standard deviation of the scores of the 100 nearest sampled views.

Key insights

Aesthetics models trained only to rank crops of one image are poor at ranking different viewpoints; training on both AVA and CPC with translation and exposure losses gives a scorer suited to a moving camera.
Shaping the reward with the per-step change in aesthetic score plus a decaying exploration bonus is critical; a terminal-only reward is too sparse.
LSTM memory lets the agent avoid re-exploring poor regions and, on the robot, learn that a repeated action with no state change needs a different action.
Setting the success threshold from the local region's own score statistics makes the goal independent of scene size and overall scene quality.

Main results

Captured photos beat the local aesthetic threshold 81.7% ± 0.9 of the time on 20 unseen Gibson scenes and 77.8% ± 1.0 on 18 Replica scenes.
The best baselines reached 57.8% (imitation learning) on Gibson and 56.6% (key frame selection) on Replica; rule of thirds reached 19.1% and 17.3%, random 13.3% and 14.3%.
The aesthetics model scored 84.4% on cross-view ranking (AVA) against 62.2% for VEN, 99.7% on exposure ranking against 81.0%, and a minimal-crop MSE of 0.03 against 0.29, at a small cost on crop ranking (CPC 72.2% against 75.8%).
On a Clearpath Jackal taking 64 indoor photos, 1792 Mechanical Turk judgements gave a mean preference of 0.63 ± 0.01 for the robot's photo over the initial view (t-test against 0.5, p < 0.05).
Ablations on Gibson: removing the LSTM dropped success to 62.7%, using last-layer features only to 62.3%, and removing both reward-shaping terms to 69.3%.

Limitations

The authors state the approach depends on high-quality simulated environments, which may be harder to obtain in some domains.
The authors state that system behaviour depends strongly on the chosen aesthetic function.
Movement is limited to discrete planar steps (0.25 m translations and 10°/30°/90° turns); full 6-degree-of-freedom control, for example on a drone, is left to future work.
Real-world evaluation compares the final photo only with the robot's starting view, since an aesthetic threshold cannot be estimated without dense sampling of the real scene.
On the robot, collision handling is external: if the chosen action would collide, the policy is queried again until it proposes a valid one.

Field impact

IROS 2021 Best Entertainment and Amusement Paper Award finalist, cited 15 times. The authors release source code on GitHub.

Abstract

The process of capturing a well-composed photo is difficult and it takes years of experience to master. We propose a novel pipeline for an autonomous agent to automatically capture an aesthetic photograph by navigating within a local region in a scene. Instead of classical optimization over heuristics such as the rule-of-thirds, we adopt a data-driven aesthetics estimator to assess photo quality. A reinforcement learning framework is used to optimize the model with respect to the learned aesthetics metric. We train our model in simulation with indoor scenes, and we demonstrate that our system can capture aesthetic photos in both simulation and real world environments on a ground robot. To our knowledge, this is the first system that can automatically explore an environment to capture an aesthetic photo with respect to a learned aesthetic estimator. Source code is at https://github.com/HadiZayer/AutoPhoto

Open Source Robotics