Materializing data: notes on collaboration and tangible interfaces with excerpts and additions

The visualization of data elucidates trends and patterns in the phe-nomena that the data represents, and opens accessibility to understanding complicated human and natural processes represented by data sets. Research indicates that interacting with a visualization amplifies cognition and analysis. A single visualization may show only one facet of the data. To examine the data from multiple perspectives, engaged citizens need to be able to construct their own visualizations from a data set. Many tools for data visualization have responded to this need, allowing non-data experts to manipulate and gain insights into their data, but most of these tools are restricted to the computer screen, keyboard, and mouse. Cognition and analysis may be strengthened even more through embodied interaction with data, whether through data sculpture or haptic and tangible interfaces. We present here the rationale for the design of a tool that allows users to probe a data set, through interactions with graspable (tangible) three-di-mensional objects, rather than through a keyboard and mouse interaction. We argue that the use of tangibles facilitates understanding abstract concepts, and facilitates many concrete learning scenarios. Another advantage of using tangibles over screen-based tools is that they foster collaboration, which can promote a productive working and learning environment.


Introduction
The visualization of data elucidates trends and patterns in the phenomena that the data represents, and opens accessibility to understanding complicated human and natural processes represented by data sets. Research indicates that interacting with a visualization amplifies cognition and analysis. A single visualization may show only one facet of the data. To examine the data from multiple perspectives, engaged citizens need to be able to construct their own visualizations from a data set. Many tools for data visualization have responded to this need, allowing non-data experts to manipulate and gain insights into their data, but most of these tools are restricted to the computer screen, keyboard, and mouse. Cognition and analysis may be strengthened even more through embodied interaction with data, whether through data sculpture or haptic and tangible interfaces. We present here the rationale for the design of a tool that allows users to probe a data set, through interactions with graspable (tangible) three-dimensional objects, rather than through a keyboard and mouse interaction. We argue that the use of tangibles facilitates understanding abstract concepts, and facilitates many concrete learning scenarios. Another advantage of using tangibles over screen-based tools is that they foster collaboration, which can promote a productive working and learning environment.

Materializing data
Visualization is recognized as an effective means to understand and communicate data, particularly complex data, since visualization leverages our perceptual cognition (Ware, 2012) and can represent large quantities of data that would otherwise be incomprehensible. This makes the visualization of open data a promising endeavor. Representing data by visual means allows us to find patterns which may be obfuscated by non-visual means. With this in mind, various authors argue that the visualization of data represents a key opportunity for enabling engagement to facilitate learning (Zambrano, Engelhardt 2008;Lewis, 2013;Bohman 2015) and critique. This can serve formal and informal purposes, for example civic dialogue.
Public visualizations have been proposed in instrumental ways to help citizens better understand urban issues, with promising results (Moere, Hill, 2012) and a recent study found that visualization interfaces, situated in public spaces led to improved perception, sustained behavior change, increased social awareness and discourse (Valkanova, Jorda, Vande Moere, 2015). In addition, the authors found that public visualizations led to meaningful participation with government as well as a range of social interactions related to locally relevant topics. The visualization of data allows for heightened perception and the identification of patterns and findings.
Some have argued that physical data representations, by existing in the same dimensional space as the viewer, encourages people to reflect on the data's meaning and provides a more enjoyable and engaging experience 4 Materializing data: notes on collaboration and tangible interfaces with excerpts and additions

Echoes
DATJournal v.1 n.2 2016 compared to their 2d graphical counterparts (Zhao, Moere, 2008). One way in which physical representations help us better grasp abstract notions is by using a concrete or metaphoric scale, whereby abstract ideas are mapped onto everyday relatable objects (Chevalier, Vuillemot, Gali, 2013). The aesthetic engagement afforded by physical representations has made it a fruitful subject of many art works. In a recent physical visualization sugar cubes are used to represent the amount of sugar contained in soft drinks and other foods (Chevalier, Vuillemot, Gali, 2013). Seeing the numbers of sugar cubes contained in a CocaCola may have a stronger cognitive impact than reading the grams of sugar contained on the label.
For several decades artists have created data sculpture and other forms of representation such as wearables, textiles and prints, which provide representations of data which are not screen based and deeply engaging. Physical representations can viscerally convey a social message with the use of metaphor. In the CodeZebraOS project Sara Diamond (Diamond et al., 2005) created a chat visualization software which showed users emotional qualities in each of their posts. These were also expressed as interactive images on wearable fashion designs, allowing audiences to interact with clothing which in turn would change images within the software. In May 2015, a pop up art exhibition in Los Angeles entitled 'Manifest Justice' featured an installation of 22 prison uniforms juxtaposed against one university graduation gown, with a sign above it reading: «Since 1980, California has built 22 prisons, and one university» (Posted by Eye Candy on May 12 and Blog 2015) The image of the metaphoric objects and their quantities gives the viewer a visceral understanding of the government's spending priorities.
Recent work by OCAD University students Rachel Hurst (2016) and Emily Neill (2016) finds ways to indicate large scale data sets through highly condensed and visually compelling imagery. In Rachel Hurst's Censor-ship landscapes of Yemen, Somalia and Pakistan are screen captured from Google Earth at dates close to missile strikes based on actual data and then can be inserted into the battle platform grid, transforming into human figures. Emily Neill uses data from energy use, hours of labour and casualties within the emerging world garment industry to create textiles made of environmentally efficient materials that are patterned in abstract forms representing the data. Viewers can cut a piece of a textile as a means to engage with their own consumption of textiles.
Data sculpture is a growing area of artistic practice. Andrew Spitz's (2013) Loci are unique 3D printed outputs of individuals' flight paths in order to spur recollection, reflection and shared memories of travels. Andreas Nicolas Fischer (2014) develops statistical maps with vertical axes that correspond to the financial status of countries. In 2 a.m. to 2 p.m. R. Justin Steward (2011) created a data sculpture of the Minneapolis/St. Paul transit system which displays bus routes that move over intervals of time. Germaine Koh (2015) created Topographic Table, an interactive model of the terrain in the Vancouver area that responds to actual tremors as earth quakes of different scales move through mountains north of Vancouver. Sensors and Internet connected electronics embedded in the table's frame cause it to tremble in response to data about earthquakes in the Pacific Northwest. This piece of furniture models the geology and the psychic condition of living near the Cascadia fault line. Koh's work underlines the power of vibro-tactile displays of data. In recent research with Patrick Crowe Rachel Zuanon and her company, Adam Tindale, Sara Diamond and Zenophile Media (2015) vibro-tactile wrist-bands were created for the game Time Tremors in order to display gameplay activities representing energy expended and direction of play.
Physical representations of data, in addition to being aesthetically engaging, may offer more intuitive approaches to data analysis and lead to insights into data sets (Vande Moere, 2008). Empirical evaluations have found that in some circumstances they out-perform on-screen equivalents when retrieving information and that the component of touch appears to be a key cognitive aid (Jansen, Dragicevic, Fekete, 2013). Jansen et al. compared sculptures of 3-dimensional bar graphs with their onscreen equivalent, and found that people were able to retrieve data more rapidly from the sculptural forms. Other recent studies have found that engaging students with the production of physical representations enhances their understanding of statistics (Gwilt, Yoxall, Sano, 2012).
These experiments underline the capacity that humans have evolved for sensing and manipulating their environment. Therefore to optimize practical digital tools we can learn from artists' work and start designing them to engage our sense of three-dimensional space and extend this to touch (Sharlin et al., 2004). Currently, most of our digital technology is confined to screens, pointers, and keyboards (known as WIMP interfaces), but the standard interface is rapidly moving towards the touch screens, which take advantage of our intuitive gestures and allow us to apply our sense of touch to the task (Wigdor, Wixon, 2011;Weiyuan Liu, 2010). The current phase of development is in ubiquitous computing and the internet of things (Olson, Nolin, Nelhans, 2015), using everyday objects to interact with data and using natural gestures to formulate operations.
The idea of using physical objects to enhance learning dates back to Friedrich W. Fröbel's (1782-1852) pedagogic «gifts» -objects presented to the children to illustrate mathematical concepts (Huron et al., 2014). Jean Piaget (1896Piaget ( -1980, who believed that children learn naturally by manipulating and experimenting with physical objects, later reinforced this notion. More recent studies have shown that handling and interacting with physical objects may also benefit adult learning (Chapman, 1988).

Tangible interfaces
Tactility is an important part of the engagement that physical representations of data offer when the data sculptures are scaled for personal use, as seen in the examples of (Jansen, Dragicevic, Fekete, 2013;Gwilt, Yoxall, Sano, 2012;Stusak et al., 2014) discussed above. While in those examples, users were able to touch the data sculptures, the data itself was ma-6 Materializing data: notes on collaboration and tangible interfaces with excerpts and additions

Echoes
DATJournal v.1 n.2 2016 nipulated digitally before it was manifested physically. In this section, we look at exploring ways to physically manipulate data.
In 1997, Ishii and Ullmer proposed the concept of «tangible bits» (Ishii, Ullmer, 1997), in which computational bits are coupled with graspable physical objects. Since then, many have stepped up to the design challenge of creating a system that extends the affordances of physical objects into the digital domain (Ishii, 2008). For example, at MIT's media lab, the tangible media group has created inForm, an interactive shape-changing system that users can interact with through touch, which also changes form in response to the users' interactions (Follmer et al., 2013).
As the user manipulates data, there is not always clear distinction between input and output: the answer to the query does not appear all at once, but rather the system transforms over time to represent the answer. This is actually a typical feature of analog tangible systems; it is only in the digital world where there is a clear distinction, a dichotomy, between input and output (Sharlin et al., 2004). Tangible user interfaces aim to use tactility to engage the user, but they can also aid cognition by introducing some fluidity between user input and data output .
Tangible User Interfaces (TUI) are broadly defined as graspable 3D physical objects with which to interact with data, often digital data. Currently, there is no standard protocol for using the manipulation of physical objects to interact with digital data. Shaer et al. (Shaer et al., 2004) made an effort to conceptually define a paradigm for standardizing TUI inputs with what they called a Token and Constraint system (Ullmer, Ishii and Jacob 2005). Tokens are physical objects that are handled to access or manipulate the digital data, and constraints limit the way in which the user interacts with the token. Constraints set the framework of how the user manipulates the tokens, and ideally, they should be designed to express the set of operations that can be performed on the digital data (Jacob et al., 2008).
Research in TUI for learning applications seeks to clarify which specific elements of tangible interface design support learning. For example, tangible interfaces may augment student engagement with learning tasks (Shaer and Hornecker, 2010). In one study for example engagement was significantly increased in digital learning tools when children are allowed to use their everyday physical play objects to interact with the digital information. In these cases, learning outcomes improve from the increased engagement with the lessons.
As learning tools, tangible interfaces have been shown to encourage activities and behaviors that augment learning and problem solving. Schneider found that outcomes in solving logic puzzles are improved when interacting with a tangible interface compared to a working on a touch-screen (Schneider et al., 2011). These studies, however, also noted that the participants using the tangible interface worked on the puzzles much more collaboratively than those using the multi-touchscreen, which suggests that the collaboration was the key factor that improved outcomes, and that using the tangible interface fostered the collaboration.

Tangibles and collaboration
We are particularly interested in the role of collaboration, which many studies suggest may be a key differentiator between tangible and screen-based interfaces (Hornecker, Buur, 2006;Lee et al., 2012). Collaboration interests us for two reasons. One, the ability to work collaboratively and organize is essential for meaningful public participation in a democracy. Two, there is an empirical correlation between collaborative problem solving and improved learning outcomes (Schneider et al., 2011). Many of the physical representations of data discussed earlier respond to individual and group and interaction, creating highly collaborative experiences.
Tangible interfaces measurably increase collaborative behavior. A recent study using eye-tracking devices found that participants working in small groups on a problem-solving task experienced more moments of joint visual attention when working with graspable movable objects on a tabletop than when working with a screen-based interface .
It should be noted that the amenability of tangible interfaces for collaborative work has made them promising tools for facilitating collaboration over long distances by augmenting teleconferencing (see for example Bouabid et al., 2014;Gonzalez-Franco et al., 2015), and for strengthening communications in co-located meetings by means of smart boards and digital Post-It notes (see for example Haller et al., 2010). MIT Media Lab's inFORM is a dynamic shape display that can render 3D content physically, so users can interact with the physical world around it, for example moving objects on the table's surface (Leithinger et al., 2014). Remote participants in video conferencing can be physically displayed, allowing a strong sense of presence.
Interestingly, the eye-tracking study by  that measured increased joint visual attention with the use of tangibles also suggested that there was a correlation between joint visual attention and learning outcomes. Previous empirical studies have shown that using tangible interfaces usually results in better task performance of the group, but using them did not always affect the learning outcomes of the individuals (Do-Lenh et al., 2010). Other studies have shown that the collaboration fostered by TUI may improve creative outcomes. Kim and Maher, for example, compared Graphical User Interfaces (GUI) with TUI in study participants that were assigned a collaborative design task, and they found that the groups using TUI performed multiple cognitive actions in a shorter time, made more unexpected discoveries of spatial design features, and exhibited more problem-finding behaviours (Kim, Maher, 2008).

Our tool -a tangible interface for interactive data query
While the use of tangible interfaces has been extensively explored in gaming applications (not discussed here), in pedagogical applications (discussed above), and in communications (mentioned above), there has been comparatively little work done in using tangible interfaces for data 8 Materializing data: notes on collaboration and tangible interfaces with excerpts and additions

Echoes
DATJournal v.1 n.2 2016 query. TUIs share characteristics with other physical representations in their use of metaphor or concrete representations of data content, aesthetics and bodily engagement. In our project we take advantage of the benefits that tangible data representations bring and combine these with graphical representations in a highly interactive environment.
One of the first tangible data query systems was designed to interactively convey historical information at a tourist site, using blocks that can be positioned to form a query (Camarata et al., 2002). The idea of rearranging objects to create data queries was later used in Stackables (Klum et al., 2012), and later in Cubequery (Langner, Augsburg, Dachselt, 2014), whose cubes include a small display screen for the outputs. In contrast to these systems, our system does not require any specialized hardware, a feature we believe to support the idea of data democratization.
Our system only requires standard equipment: a computer, a webcam, and a projector. In our system, users create queries by placing and arranging clearly demarcated objects (that are handheld in size) onto a common tabletop, and the results of the query are displayed onto an overhead screen placed at one end of the table. The visualizations that appear on the screen respond to the configuration of the objects on the table. In our current prototype, there are 4 different types of objects the user can manipulate to discover the data. 1) The category objects let the user decide the subject of the data. For example, each category object could represent a country. 2) The measurement objects let the users determine what data about the category they want to see visualized. For example, one may want to look at the population of a country, or the income per capita.
To illustrate this example, if a user places two category objects on the table: one representing Canada and the other representing the United States, along with a measurement object representing population, or income per capita, then the screen will display a bar chart of the populations, or income per capita, of Canada and the United States. Reordering the category objects on the table will reorder them on the screen. The other two types of objects allow the users to probe the data in more detail.
3) The subdivision objects divide the data into subcategories. For example, the users might want to view the population, or the income per capita, broken down by gender, by age group, or by regions. 4) The detail objects provide a close up view of the data in any subcategory when these objects are placed in close proximity to a subdivision objects. So, for example, the detail object is applied if a user wants to know with precision what the income per capita is of women between the ages of 25 and 35 for each country.
The objects are tracked by means of a camera placed discretely beneath the transparent tabletop. The bottom of each object is marked with a fiducial marker, and the camera placed below the table captures the image of the fiducial markers in real time. The fiducial markers are read using open-source reacTIVision software (Kaltenbrunner, Bencina, 2007). The re-acTIVision software outputs the position of the markers, if they are in the field of view of the camera, and this information is input into our software, 9 Materializing data: notes on collaboration and tangible interfaces with excerpts and additions

Echoes
DATJournal v.1 n.2 2016 which constructs the visualizations from a (user-provided) database, filtered by the user's query.
In our first prototype application we used demographic data on radio listeners and their consumption habits, which was collected on April 1, 2013 for the Toronto area in Canada, and compiled by nLogic Canada, our industry research partner. In this prototype application, each category object represents a radio station, and we have two measurement objects, which allow users to determine whether they view data on the radio stations' number of listeners, or on the number of minutes listened. Each of these data sets can be broken down into age and gender demographics by placing the subdivision objects on the table. The detail objects are used to get the precise number of listeners or minutes listened within any subdivision. For example, the user can determine the exact number of female listeners between the ages of 18 and 25 for each radio station by placing the detail object in proximity to the corresponding subdivision objects. This particular application of the tool will be used as a part of a market research package for advertisers targeting radio air time (Jofre et al., 2015).
While our first prototype is specific to this radio-station data set, we note that it can be adapted to any dataset, as the system allows users to provide their own formatted data. The fiducial markers can be printed and glued onto the bottom of any object the user wishes to use to explore their data set.
Our system has two levels of users, providing two levels of affordances; expert users that provide the data set, and that choose or create the physical objects used to interact with it, and non-expert users that use the tabletop objects to explore the data. This type of system can be directly translated to the classroom, where the teacher provides data for the students to explore collaboratively in constructivist learning exercises.
Results from our first pilot study of test users support our assertion that our tangible interactive tool for data query encourages communication and collaborative data exploration, which is consistent with the literature on tangible interfaces and observations about interactive data sculpture above. We organized participants into small groups of two to four, and gave them problems to solve using the data. Participants exhibited collaborative behavior, and in subsequent surveys, they reported positive feelings towards their teammates and about their interaction as a group.
In addition to encouraging collaboration, the playful nature of the tangible interaction could lead to a greater degree of engagement. Preliminary observations of users are promising -test subjects seem eager to handle the objects, and they take on a playful disposition when interacting with the system. Turning data query into a pleasurable experience can encourage people to spend more time exploring data, which, in the information age, is essential to being an educated and engaged citizen.

Conclusion
Making use of open data (and learning from it) is an important aspect of citizen education and civic participation. We offer here a tool with which users can visualize and investigate their data collaboratively using a graspable tangible interface. We designed our tool to create a pleasurable data exploration experience, and to help users gain insight into their data. The intuitive nature of manipulating blocks to form a query may help bridge any data literacy gap for non-expert users, and a playful collaborative situation may encourage non-expert users to make contributions of their insights into the data (which in the context of civic participation are equally valuable as expert user contributions). We believe that a big part of citizenship engagement should be spent in dialogue, and to this end, we designed this tool to be used by teams of two to four people to collaboratively examine the data. Our design rationale was based on a wealth of literature, reviewed here, which suggests that using tangibles has cognitive benefits, and encourages collaboration, making them a promising technology for better engaging people with data.

12
Materializing data: notes on collaboration and tangible interfaces with excerpts and additions