This demonstration is derived from research currently being undertaken by an international team of scientists, artists, and digital humanities scholars about the future of text in XR. In its current stage of development, the project uses WebXR to access a library of documents derived from ACM Hypertext proceedings; select a document or documents to read; manipulate the documents by moving them around in the virtual space; navigate among various linked outputs of the documents, such as abstracts and references; and save the state of the environment.

CCS Concepts: • Human-centered computing → Visualization design and evaluation methods; • Human-centered computing → Visualization systems and tools; • Human-centered computing → Mixed / augmented reality; • Human-centered computing → Virtual reality;

Keywords: XR, VR, Augmented Text, Future of Text, augmentation, academic reading, open source, Hypertext, viewspecs, views, Vision Pro, Meta Quest, transition, interaction, visualisation

ACM Reference Format: Dene Grigar, Frode Hegland, Andrew Thompson, and Mark W R Anderson. 2024. Demonstration of The Future of Text in XR: Year 1. In 35th ACM Conference on Hypertext and Social Media (HT '24), September 10--13, 2024, Poznan, Poland. ACM, New York, NY, USA 2 Pages. https://doi.org/10.1145/3648188.3677047

This demonstration is derived from research currently being undertaken by an international team of scientists, artists, and digital humanities scholars about the future of text in XR. The project, supported by a grant from The Alfred P. Sloan Foundation1 awarded in 2023 [2], explores ways to harness the potential of XR, via VR headsets such as the Meta Quest 2 or 3 or the Apple Vision Pro, for academic communication. In its current stage of development—that is, the eighth month of a two-year period—the project uses WebXR [5, 8] to access a library of documents derived from ACM Hypertext Conference proceedings [1]; select a document or documents to read; manipulate the documents by moving them around in the virtual space; navigate among various linked outputs of the documents, such as abstracts and references; and save the state of the environment. At the heart of our project is an investigation into open-source systems, such as WebXR, so that those interested in using XR for research and creative activities can get the most out of their data without proprietary systems defining methods and structures. Specifically, the demonstration includes:

Full document interactions: Users can interact directly with a document to move, scale, and explore via a map view in a 360-degree environment, thus enabling hypertextual interactions by providing high resolution metadata for viewer software to use.

Document component interactions: Users can interact with the document to put elements from the document in 3D spatial positions either manually or to pre-determined locations, including table of contents and references.

Multi-Document interactions (Connections): Users can interact with references and document tags in one document and see how they link to other documents in their personal library and beyond.

Multiple View Points: Users can choose from several different environments each interpreting the same data but in a unique way. Because the data is open source, anyone is encouraged to create their own environment.

Headset/Traditional Computer Transition: Users can save the state of their interactions in a specific view point in order to return at a later point or pass the information into a different environment.

The project is built with WebXR, an application program that supports augmented and virtual reality devices. It is programmed in JavaScript and uses the Three.js 3D library for rendering objects in the virtual environment. To produce high-quality, readable text in the XR environment, we are also using the Three.js add-on, Troika. For our demonstration we will use the Apple Vision Pro [4] and Meta Quest 3 [12] for showing the advancements we have made during Year 1 of our project. A brief video of the project can be seen at https://vimeo.com/982667578.

The subject matter used in the demonstration is the corpus of papers from Hypertext Conference proceedings. The visualisation is enhanced by additional data developed from the dataset of that corpus published at HT’22 [3]. The latter is needed because some data is only implicit in the context of reading the paper or else is is own visible to the viewer when reading the text in facsimile. Part of the exploration here is to improve the academic's interaction with documentation other than by direct reading, e.g. by making abstracts, heading, references, etc., into elements of the document accessible for direct interaction outside the full document text.

Moving forward, a critical component of XR development will be to ensure that all data and metadata is accessible to human and machine reading in order to ensure a document's provenance and, thus, validity. Visual-Meta [7], a schema that embeds metadata to documents at the same level of the text as an appendix, addresses this need and enables reading systems to provide more hypertextual affordances to the user, such as linking data and navigating among documents and their textual elements, and allow for experimentation with spatial hypertext [6, 9, 10, 11, 13] in XR.

An example of ongoing work is shown in Figure 1 below:

An XR environment shows a number of lists of papers displayed in virtual space. The user is interacting via hand-tracking, shown as simulated human hands within the environment. Visible lines link various interconnected elements of the work. Figure 1: Exploring papers from HT Conference Proceedings

This work has been undertaken via a grant from the Alfred P. Sloan Foundation [2].

We would like to acknowledge members of the Future of Text in XR team: Fabien Benetou, Adam Wern, Brandel Zachernuk, Peter Wasilko, Leon van Kammen, Rob Swigart, and Andrew Thompson, who have participated in the conversations and activities associated with the development of this project.

We would also like to acknowledge the ongoing support of our Advisory Committee, Ismail Serageldin and Vint Cerf, who guided the development of our grant proposal.

⁎Corresponding author

1 https://sloan.org/programs/research

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).

HT '24, September 10–13, 2024, Poznan, Poland

© 2024 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-0595-3/24/09. DOI: https://doi.org/10.1145/3648188.3677047