This dissertation lies at the intersection of (1) artistic experimentation in altering human perception and consciousness through creation of digitally-based or-enhanced interactive art and (2) inquiry into the nature and implications of human- computer interactions (HCI).1 It examines known gaps between these two broad areas (involving user, information, and interface) and questions the validity of current ontological and methodological approaches to reducing these gaps. It utilizes a design approach, focusing on space-time aesthetics in an attempt to connect theory and practice through the framework of cybersemiotics. It reflects on the implications and impact on consciousness of such an integrative framework.
My own entry into this convergent, transdisciplinary conversation was my practice as technoetic artist2 and information designer developing interactive hybrid environments.3 More specifically, the initially uneven reception and limited success of my installation Mixing Realities, a large dynamic and interactive hybrid physical- digital installation begun in 2008, was the impetus for my delving into the theoretical frameworks available to conceptually ground and potentially enhance my artistic practice.
Mixing Realities was developed to focus on time perception as subject matter and content, and as such its design (form4) attempted to hypermediate time perception by “seeking to make the digital interface [and physical environment] ‘transparent’ [to the user]. In this sense, a transparent interface would be one that erases itself, so that the user is no longer aware of confronting a medium, but instead stands in an immediate relationship to the contents of that medium” (Bolter & Grusin, 2000, pp. 23-24). My modest intention was simply, as Roy Ascott states,“to start a dialogue, to set feelings and ideas in motion” (Ascott, 2007, p. 98) concerning time perception and physical embodiment. Nonetheless, observing how so few people actually engaged with the artwork instigated a sense of frustration and puzzlement about how or what people were actually experiencing.The artwork was designed to combine physical and digital elements in a visually enticing, intuitive, user-friendly environment. Cyberperception5 was explored through intentional communication loops between input, output, and observer, following the principles of second-order cybernetics. Materials were carefully selected, considering their form and function, in order to optimize the user’s experience. Nevertheless, it soon became clear that the initial connection between the user and the interface was not intuitively occurring.
Today most of our interaction with the world is mediated by digital technologies. As users, we must frequently interact with a new interface (a gadget, computer application, website, digital artwork, etc.), often experiencing frustration and disconnection from the interface similar to what I observed in Mixing Realities. I also frequently experience a similar dissociation as a user-designer of interactive hybrid environments. My research suggested that the user’s dissociation from the interface is related to a disconnect from an unnamed, but ever-present, third element: information. Users either lack knowledge (information) about the interface itself, or they are unprepared for or simply uninterested in receiving meaningful information from the digital data made available through their interaction with the interface. That recognition led me to an awareness that something more than optimal feedback processes, information design, and user experience guidelines and rules are needed to more fully integrate the user into the work as I originally intended and to more adequately conceptualize the interactive processes at work.
This research attempts to connect theory and practice, to understand the limitations of Mixing Realities, and to gain a better vantage point on human-computer interactions by employing a common method of analysis utilized by art historians, designers, architects, poets, and other creative individuals to evaluate practice through theory. It involves defining the subject matter and observing content and context behind the artwork, while focusing on form.Through this approach, it became possible to observe “Mixing Realities” as a whole while also focusing on the individual elements composing the artwork, their characteristics and relationships.
The creators and theorists of interactive hybrid environments utilize concepts from many fields, which do not necessarily exchange with or complement each other. HCI, cybernetics, information theory, semiotics, design, and new media are some of the fields which attempt to describe these environments, but they don’t necessarily agree on the subject matter, content, or context, much less on the same components or their characteristics and relationships. Such pseudo-transdisciplinarity revealed both ontological and methodological problems.As my research proceeded, I became especially aware of the frequent reliance on conceptualizations of the interface (1) as replicating and reinforcing physical perceptions of embodiment and also (2) as the sole mediator of the digital/physical experience.6
Ontologically, this thesis highlights the problem of the user-interface disconnect that I perceived in my artistic/design practice as well as the inadequate conceptualizations of human-computer (i.e., user-interface) interaction in the theoretical literature by systematically incorporating the third element—information— into the analysis and by introducing the concept of the meta-environment (see Figure 2) as the locus for describing and analyzing the triadic relationship among user, information, and interface.
Methodologically, the problem of the user-interface disconnect and inadequate conceptualizations of HCI is addressed through the analytical framework of cybersemiotics,7 a transdisciplinary theory of information, cognition, meaning, communication, and consciousness that integrates cybernetics and Peircean semiotic paradigms in a common framework (Brier, 2008), which allows for an integrative analysis of the elements in the meta-environment taking into consideration that information can be not only data but “jointly actualized meaning” (Brier, 2008, p. 20).
Cybersemiotics helped me expand the connection among the elements in the meta-environment by offering a common language for classification, description, and exchange among these elements where the following transdisciplinary aspects could be equally considered:
(1) physic-chemical aspects such as spatiality and temporality,
(2) biological and natural science aspects (atoms or bits),
(3) linguistic-cultural-social structuralist aspects related to subjective and/or objective sign interpretation and meaning creation.
(4) aspects of consciousness (qualia), the “phaneroscopic first person point of view” in embodied cognition and meaning creation from “one mental space to another” (Brier, 2008, p. 303).
This transdisciplinary, holistic analysis of the meta-environment highlighted the cognitive dissonance between human perceptions and computer-mediated processes while revealing the potential of the meta-environment to function as a complex adaptive system.
This theoretical approach underpinned a further realization that in the implementation of interactive hybrid environments, information is generally translated to the user through a design architecture that assumes embodiment and relies primarily on physical narratives that privilege space over time. It also showed how the elements in the meta-environment comprise a complex adaptive system with different levels of interactions and processes, affecting and mutually influencing each of the individual elements, similar to the manner in which the elements of an artwork influence and affect perceptions of that artwork.
Applying this new understanding to my own practice laid a basis for understanding why so few people had fully experienced Mixing Realities and, more broadly, supported re-conceptualization of the relationship among user, information, and interface in the meta-environment. By questioning such dichotomous paradigms as physical/digital, space/time, and human perception/logical-mathematical process, cybersemiotics pointed toward the need for a more holistic and integrated qualia experience, described and visually represented here as the Cybersemiotic Experience.
- 1. “Human-computer interaction (HCI) is an area of research and practice that emerged in the early 1980s, initially as a specialty area in computer science embracing cognitive science and human factors engineering” (Carroll, 2014, https://www.interaction- design.org/literature/topics/human-computer-interaction).
- 2. “Technoetics is a convergent field of practice that seeks to explore consciousness and connectivity through digital, telematic, chemical or spiritual means, embracing both interactive and psychoactive technologies, and the creative use of moistmedia.” (Ascott, 2008, p. 204).
- 3. Subject matter, content, and form are the components of art, according to Ocvirk, Stinson,Wigg, Bone, Cayton, 2009, pp. 10-16.
- 5. Cyberperception is the emergent human faculty of technologically augmented perception (Ascott, 2007).
- 6. These assertions are based on different sources, but I share two to illustrate my point:An introduction to embodiment and digital technology research: Interdisciplinary themes and perspectives by William Farr, Sara Price, Carey Jewitt (2012) http://eprints. ncrm.ac.uk/2257/4/NCRM_workingpaper_0212.pdf analyses how concepts of embodiment permeate and mediate HCI.Also in The Encyclopedia of Human-Computer Interaction, 2nd ed. online, John M. Carroll proposes the following description of human-computer interactions. https://www.interaction-design.org/literature/book/the-encyclopedia-of-human-computer-interaction-2nd-ed/human- computer-interaction-brief-intro. “HCI addresses the dynamic co-evolution of the activities people engage in and experience, and the artifacts — such as interactive tools and environments — that mediate those activities [emphasis added]. HCI is about understanding and critically evaluating the interactive technologies people use and experience. But it is also about how those interactions evolve as people appropriate technologies, as their expectations, concepts and skills develop, and as they articulate new needs, new interests, and new visions and agendas for interactive technology. Reciprocally, HCI is about understanding contemporary human practices and aspirations, including how those activities are embodied, elaborated, but also perhaps limited by current infrastructures and tools. HCI is about understanding practices and activity specifically as requirements and design possibilities envisioning and bringing into being new technology, new tools and environments. It is about exploring design spaces [emphasis added], and realizing new systems and devices through the co-evolution of activity and artifacts, the task-artifact cycle” (Carroll, 2014).
- 7. Soren Brier’s cybersemiotic framework is introduced in chapter 4 and discussed in chapter 5