SmartCanvas: Context-inferred Interpretation of Sketches for Preparatory Design Studies

1ShanghaiTech University     2KAUST     3Yale University     4University College London

Eurographics 2016

Using a background photograph for anchoring, a practicing architect used our system to progressively develop a design. In the background, our algorithm lifts the sketched strokes along with their intersections to 3D using a novel context-based canvas arrangement extraction algorithm. The discovered canvases are painted and used for 3D-like interactions to facilitate preparatory design previews (right).


In early or preparatory design stages, an architect or designer sketches out rough ideas, not only about the object or structure being considered, but its relation to its spatial context. This is an iterative process, where the sketches are not only the primary means for testing and refining ideas, but also for communicating among a design team and to clients. Hence, sketching is the preferred media for artists and designers during the early stages of design, albeit with a major drawback: sketches are 2D and effects such as view perturbations or object movement are not supported, thereby inhibiting the design process. We present an interactive system that allows for the creation of a 3D abstraction of a designed space, built primarily by sketching in 2D within the context of an anchoring design or photograph. The system is progressive in the sense that the interpretations are refined as the user continues sketching. As a key technical enabler, we reformulate the sketch interpretation process as a selection optimization from a set of context-generated canvas planes in order to retrieve a regular arrangement of planes. We demonstrate our system (available at http:/ with a wide range of sketches and design studies.


Context-anchored design examples developed using our system.


@article {177,
  title = {SMART CANVAS : Context-inferred Interpretation of Sketches for Preparatory Design Studies},
  year = {2016},
  month = {05/2016},
  publisher = {Computer Graphics Forum},
  attachments = {},
  author = {Youyi Zheng and Han Liu and Julie Dorsey and Mitra, Niloy}


We thank the anonymous reviewers for their invaluable comments, suggestions, and additional references. We would like to thank A- mal Aboulhassan, Pooya Sareh, Wenwen Zhu, Lubing Fan, Cristina Amati, and Dongming Yan for their great help on the user study. We specially thank Luke Pearson ( for contributing his sketches using the system. The project was supported in part by the ERC Starting Grant SmartGeometry (StG- 2013-335373) and the US National Science Foundation Award (No. 1302267).


Paper (82 MB)

Data (16 MB)

Code (2 MB)