David M. Frohlich, Emily Corrigan-Kavanagh
Mirek Bober, Haiyue Yuancentre
Radu Sporea, Brice Le Borgne
George Revill, Jan van Duppen
Alan W. Brown, Megan Beynonsurrey
Electronic publishing usually presents readers with book or e-book options for reading on paper or screen. In this paper, we introduce a third method of reading on paper-and-screen through the use of an augmented book (‘a-book’) with printed hotlinks than can be viewed on a nearby smartphone or other device. Two experimental versions of an augmented guide to Cornwall are shown using either optically recognised pages or embedded electronics making the book sensitive to light and touch. We refer to these as second generation (2G) and third generation (3G) paper respectively. A common architectural framework, authoring workflow and interaction model is used for both technologies, enabling the creation of two future generations of augmented books with interactive features and content. In the travel domain we use these features creatively to illustrate the printed book with local multimedia and updatable web media, to point to the printed pages from the digital content, and to record personal and web media into the book.
Experiments in augmented paper date back about 28 years to Newman & Wellner’s (1992) digital desk and its variants. These use a variety of image recognition and tagging technologies to bring paper to life with sound, video and other digital associations that can be updated in a way that the printed word cannot (c.f. Signer & Norrie 2010). Despite demonstrable benefits in lab evaluations, only a few of these prototypes have made it into commercial products, and sometimes only temporarily. QR codes are probably the most commercially successful augmented paper technology, followed by full page recognition using a smartphone held over the printed page. An emerging set of augmented reality (AR) apps can be customised to particular paper documents, but this does not allow mainstream publishers to augment their publications in a generic way which is compatible with their publishing workflow.
On the Next Generation Paper project we are tackling this problem in partnership with several book publishers in the travel and tourism domain, where paper and web information are already used together: www.NextGenerationPaper.info This commercial orientation may seem like an unusual one for a research project, but we have argued in a previous paper that user-centred design should pay greater attention to commercial factors and technology uptake in the process of innovation itself (Frohlich & Sarvas 2011). For this reason, on our project, we combine a number of classic design methodologies such as design ethnography, prototyping, interaction design, lab experiments and field trials with agile innovation. Hence, Bradt Travel Guides have published a commercial version of our demonstration in March 2019 for market feedback: https://www.bradtguides.com/cornwall-the-a-book We call this the Cornwall a-book, after Mackay et al (2002) who first used the term to refer to an augmented laboratory notebook. Here we showcase the prototyping and interaction design elements of the work, ahead of lab, field and market evaluation.
The augmented book is created using standard publishing software, InDesign (see Figures 1 and 2). InDesign is usually used to create a printed book in a specialist print format. However, it also allows for the simultaneous creation of an e-book in different file formats (for different e-book readers), including interactive e-books with digital hotlinks. We use this latter feature to author an augmented book initially as an interactive e-book for Android smartphones and tablets in fixed format EPUB3.0. Different hotlink icons can be embedded in the text and linked to integral multimedia content or external websites. The book component of this can be printed as usual, but the links are played on a companion smartphone or monitor. In fact, the smartphone runs a ‘Next Generation Paper’ app which reads the interactive e-book file and supports control actions to play links corresponding to particular pages of the printed book. At present, viewing links on a large monitor is achieve with the smartphone in the loop through screen-casting. Future versions of the app might also run on smart TVs or other devices.
Two alternative control technologies can be used to link the physical book to the digital content through the app, as shown in Figures 1 and 2. In the first case, the identity of the currently open page can be entered manually by typing or speaking the page number, or by image recognition of the page itself from the smartphone camera. Speech recognition is provided by Google’s speech-to-text service. Image recognition is provided by Visual Atoms. We refer to this as second generation or 2G paper.Figure 1. Publishing workflowFigure 2. System architecture
Alternatively, light and touch sensors wired to a low power Bluetooth chip can be printed or embedded in the pages and cover of the book (see red book icon in Figure 2). We refer to this as third generation or 3G paper. Light sensors were printed on one side of plastic sheets for each page. These were laminated with paper having ‘windows’ cut out above the sensors as shown in the top right margin of the book in Figure 3. Touch sensor buttons for a variety of link types were embedded in the left and right margins of the book cover shown in Figure 3. Turning to any page and pressing the button for any available link type sends both page and link identity to the smartphone app. The book hardware was created in collaboration with VTT using specialist organic photovoltaic printing equipment.
Our Next Generation Paper app currently runs in 2G or 3G mode, depending on whether page and link identity is given by optical or embedded sensing. 3G mode establishes a Bluetooth connection between the smartphone and the book. The tile-based app interface is illustrated in Figure 4. The four tiles in the middle panel (i.e. visual recognition, manual page entry, speech-to-text, and bookmarking) allow the user to identify page number and subsequently go to the link screen shown in Figure 5. This screen shows the seven link types used to augment the paper content, and which are available for the current page (highlighted in green). The live location service on the top right tile and twitter feeds and image slide show at the bottom of the screen provide additional interactions.Figure 3. 3G book hardware.Figure 4. Home screen of the Next Generation Paper prototype app
3 INTERACTION DESIGN
Despite the complexity of the system implementation above, we aimed to achieve a simple interaction in which a printed travel guide can be read in conjunction with a smartphone containing additional information. Working with travel writer Kirsty Fergusson and Bradt Travel Guides, we chose to augment a new edition of their Slow Travel Cornwallguide with a range of multimedia and web content (Fergusson 2019). This illustrates the text, keeps it updated, or connects the reader with some of the organizations mentioned. Figure 5. Link screen of Next Generation Paper app.
By default, the user reads the paper guide book and requests information manually on selected hotlinks (icons) displayed in the book. This is done by speaking, typing or photographing the page before selecting the link type on-screen as in Figures 4 and 5 (2G mode), or by pressing the selected link type on the cover of the book as in Figure 3 (3G mode). Users then select the actual link instance from a list to ‘play’ the content at full screen size on the smartphone. This leads to the paper-and-screen reading experience shown in Figure 6. This example shows the smartphone playing a video interview with a member of staff at the National Lobster Hatchery in Padstow. The interviewee is holding up a ‘berried’ lobster to explain how they are farmed. Figure 6. Multimedia illustration within the paper-and-screen reading paradigm
In addition to this default style of following hotlinks from paper to screen, we have been experimenting with reverse links and notifications pointing from screen to paper. For example, users can watch a slideshow cycling through all the additional digital images for the book at the bottom right of the app home screen shown in Figure 4. Touching any of these images, points the user to the printed page to which the image relates by bringing up its highlighted links as in Figure 5. A similar behavior results from touching flags on the map in the top right of Figure 4. Flags appear on the dynamic home screen map when users move into the Cornwall locations covered in the book, and notifications can be sent to the user even when the app is closed. This gives the travel book a kind of agency to call out to its reader through the app, at moments when the book might be useful to them.
A final feature of the app allows the user to personalize the guide book by adding their own links to it. The plus symbol at the top of the link screen shown in Figure 5 can be used to cue Image, Audio, Video or Weblinks for association with that page. For example, users could record their own verbal comments on their visit to the lobster hatchery described on page 15.
A short video illustrating the operation of the 2G and 3G technology is shown in Figure 7. This was based on our first demonstration of an augmented Chapter 2 from the second edition of the Cornwall guide.
Although augmented paper and books have been demonstrated and sold before, their behavior has usually been simple and the process of making them has been highly customized to the technology and content used. In this demonstration we have shown how a more generic approach can be taken to what might be called a-books, in which a variety of link types and two-way connections can be established between paper and screen, through a more standardized creation process for both publishers and readers. We distinguish between optical 2G and embedded 3G paper for a-books because we believe optical page recognition is already robust enough to use in commercial applications today. 3G paper has some way to go before it can be made cheaply and reliably enough to manufacture at scale, yet ultimately offers a more seemless and magical interaction with digital content from the paper surfaces alone. By architecting a solution which is compatible with both technologies, we hope to have future-proofed this approach for some years to come, and laid the foundations for publishers to add value to their paper products and services with digital information.
This work was funded by grant number EP/P02579X/1 from the Digital Economy programme in the UK. We would like to thank Adrian Phillips, Rachel Fielding and Anna Moores at Bradt Travel Guides and Kirsty Fergusson for collaboration on the augmented Cornwall guide.
- Fergusson, K. 2019. Slow travel Cornwall & the Isles of Scilly, 3rd Edition. Bradt Travel Guides. https://www.bradtguides.com/cornwall.html
- Frohlich, D.M. and Sarvas, R., 2011. HCI and innovation. In CHI’11 Extended Abstracts on Human Factors in Computing Systems (pp. 713-728). ACM.
- Mackay, W.E., Pothier, G., Letondal, C., Bøegh, K. and Sørensen, H.E., 2002. The missing link: augmenting biology laboratory notebooks. In Proceedings of the 15th annual ACM symposium on User interface software and technology (pp. 41-50). ACM.
- Newman, W. and Wellner, P., 1992. A desk supporting computer-based interaction with paper documents. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 587-592). ACM.
- Signer, B. and Norrie, M.C. 2010. Interactive paper: past, present and future. In proc of. Ubicomp ’10. Sept 26-29th 2010.