Portalware: Free-Hand AR Drawing with a Dual-Display Smartphone-Wearable Paradigm

Jing Qian*, Tongyu Zhou*, Meredith Young-Ng*, Jiaju Ma, Angel Cheung, Xiangyu Li, Ian Gonsher, and Jeff Huang (*equal contribution)

The ACM conference on Designing Interactive Systems (DIS 2021)

Overview

Free-hand interaction enables users to directly create artistic augmented reality content using a smartphone, but lacks natural spatial depth information due to the small 2D display's limited visual feedback. Through an autobiographical design process, three authors explored free-hand drawing over a total of 14 weeks. During this process, they expanded the design space from a single-display smartphone format to a dual-display smartphone-wearable format (Portalware). This new configuration extends the virtual content from a smartphone to a wearable display and enables multi-display free-hand interactions. The authors documented experiences where 1) the display extends the smartphone’s canvas perceptually, allowing the authors to work beyond the smartphone screen view; 2) the additional perspective mitigates the difficulties of depth perception and improves the usability of direct free-hand manipulation; 3) the wearable use cases depend on the nature of the drawing, such as: replicating physical objects, "in-situ" mixed reality pieces, and multi-planar drawings.

ware_teaser.png

(A) Illustration of a user sketching AR content on both the smartphone and the wearable. (B) An AR sketch of virtual flowers on a physical pot with Portalware. (C) The user's view of Portalware.

Bibtex


@inproceedings{Qian2021portalware,
    author = {Qian, Jing and Zhou, Tongyu and Young-Ng, Meredith and Ma, Jiaju and Cheung, Angel and Li, Xiangyu and Gonsher, Ian and Huang, Jeff},
    title = {Portalware: Exploring Free-Hand AR Drawing with a Dual-Display Smartphone-Wearable Paradigm},
    year = {2021},
    isbn = {9781450384766},
    publisher = {Association for Computing Machinery},
    address = {New York, NY, USA},
    url = {https://doi.org/10.1145/3461778.3462098},
    doi = {10.1145/3461778.3462098},
    booktitle = {Proceedings of the 2021 ACM Designing Interactive Systems Conference},
    pages = {205–219},
    numpages = {15},
    keywords = {autobiographical design, wearable, augmented reality, free-hand drawing},
    location = {Virtual Event, USA},
    series = {DIS '21}
}