This week I kept up on the Non-Sentences project mostly, I spent most of the time working on the network lines and a new text to speech mode from it that adds a female voice speaking the words as they come out generated. These two things, which on the surface don't look like that much, took a lot of time and coding to make them working. For the lines, essentially I had to calculate the x and y position of each cell, then draw a 3D line based on those points as they're being called by my sentence controller script, to draw it out word by word, then erase after the sentence is being generated. It makes interesting shapes and maybe there is a way to use them in other contexts, either within or outside of this particular project. The text-to-speech took some major edits of this sentence controller as well, as it naturally wants to read each word as fast as possible, resulting in a lot of cutoff from word to word, trying to read the full set of words every time before being cut off when a new word was added. I had to work a lot with timing and spacing of the words being generated, and spacing between the generation of the spoken word, as how that works is it basically creates an mp3 file for each word, then finds that file in my computer, and plays it back out through TouchDesigner. It still reads the last word a couple times before the sentence is erased, but I think it adds a sort of hypnotic element to it, akin to movies like Blade Runner 2049's baseline test (interlinked) and the opening of Under the Skin as the Scarlett Johansson alien is being "built" and learning to talk like a person, which is based on real life phonetic training for speech therapy (the best available video I could find of this I'll have below, as well as the Blade Runner sequence).
As it relates to my thesis, this can bring more insight into the training of a LLM chat or TTS model. Trained on millions of words, this project can be seen as a visualization / audio representation of the back end process of a model both learning how to structure sentences that make sense to us, and the actual behind the scenes of a model stringing together words before they become visible to us. Normally, we see it's best foot forward in this predictive randomness, but not the millions of millions of training nonsense or how it chooses to put words together. I also think the imperfect voice adds to the uncanny-ness to it all, where by lowering the fidelity of the voice (compared to something like ChatGPT's text-to-speech models) it strips back a layer of relateability, where its main goal is trying to mimic human speech patterns, but by it being slightly "off" we might better step back and see it for what it really is: algorithmic decisions trying to make sense.
As a note the audio recording for the voice is pretty quiet, so if you turn up your computer audio to hear it make sure you turn it back down before you play the other videos.
To add more to this specific project I think I will add a word counter that shows how many times a specific word has been used by my algorithm. With it being random you would think over time that all the words would average out to about the same number of times being used, but in building this I've noticed words like "architecture, facilitate" and "dream" being used more frequently than other words. Adding that can speak to the over-reliance of certain words by LLM, which are making their way into our language as well, and in a non-measurable speculative effect of this project for someone viewing it, they might end up using some of the words in the table more frequently than others. Additionally for some stylistic polish I need to add the words being used highlighted in the cells besides just the line drawing to it, and think of some options for the sentence text on the right, thinking of having the characters look like they're "floating" by translating them on the x, y plane smoothly and randomly, to add to the flexibility of meaning as sentences break down to words, break down to characters, which break down to lines.
Among working towards other projects for the Emergent Garden interactive edition I've done some work on the interactive aspects of hand tracking, where I have currently a setup that tracks your index finger and thumb, and a basic interface over it that shows a datapoint of how spread out your fingers are. This has a lot of promise already, as all the data is there to map things like primitives to your fingers or the midpoints between them, maybe pinching to increase / decrease their size, and using other fingers or hand motions to place them down. There is a balance to find here, where you want to give people as much control to build the garden as they please, but with non-traditional methods of building, there can be a chance for people to get confused of how it actually works, which could undermine some of my work on it. This is where potential combinations of midi pads could come in handy I think, where there is a reliable hardware interface that people can use aside from their actual body movements and motions.
Finally, in terms of the previous project with face / emotion tracking, I did some brainstorming and think a good direction to go with that could be a reactive AI-rorshach visual that reacts to your smile or frown, with some accompanying text that's trying to elicit how you feel in the moment, and try to manipulate how you feel to be overly positive. Rorshach tests nowadays are typically seen as outdated and unreliable, bordering on the edge of pseudoscience as a means to determine personality characteristics or even underlying psychopathologies. What else may be unreliable? Maybe using an AI model as a therapy device or emotional crutch. The Rorshach test has a couple of "algorithms" or scoring criteria to determine anything from predisposed schizophrenia to a persons personilty, coping mechanisms, or personal perception. These algorithms provide a means to an end to a predetermined result of a score for whatever may be evaluated in taking a test, but suppose you don't know that and take one from a seemingly trustworthy source, and take at surface value what the Rorshach test says about you or your personality, and you believe it to be true even if it was based on uninformed pretenses or contexts. The same line of thinking can be applied about AI models and concepts of sycophancy, where the AI tends to reinforce beliefs that you have when conversing with it, whether it be your emotions or if you have a great new business idea to combine french fries with salad. (A newer South Park episode used this concept recently, to mock AI, startup funding, and of course the Trump administration).
Finally, on the more research side of my thesis, I've reached out to a couple AI creatives, and have gotten some confirmations on those that would like to be interviewed. One of my favorite creatives using AI was one of the first to reach out, which I am pretty stoked about. I won't say who he is, but he's done some incredible work with AI that explores concepts like the perception of time and fragility of infrastructure across long time, and has been featured on things from electronic billboards to creating visuals for major music festivals and artists, like Future, Metro Boomin, Travis Scott at events like Rolling Loud and Lollapalooza. Within the next couple of days to weeks, I'll have some interviews down and transcripted to be used for my thesis.
What I’ve Read
I managed to get a hold of "HOMO LUDENS A STUDY OF THE PLAY-ELEMENT IN CULTURE", and in reading some of it and using Google Notebook to quickly summarize other parts, it brings up ideas and definitions of play that seems familiar to me, whether I've heard it from an undergraduate philosophy class or YouTube essay. Johan Huizinga describes play as a fundamental and autonomous activity that is free, meaningful, and distinct from ordinary life, emerging before culture or civilization itself, as animals also are observed to play with each other. He argues that play is not merely a biological function or a means to an end, but rather a primary category of life that carries its own intrinsic value and significance. He says play is “in fact freedom,” a temporary stepping outside of real life into a separate sphere governed by its own order and rules. Although it is “not serious,” it can be pursued and used with complete seriousness and absorption. Serious in tone and attitude, but not serious in stakes or survival. Within this self-contained world, play creates order, tension, and beauty, giving meaning and form to human experience beyond survival or utility.
Huizinga says that play is a foundational element of culture itself, not just a pastime or instinct. Civilization, he claims, “arises and unfolds in and as play,” and thus Homo Ludens: “man the player”, is as essential a concept as Homo Sapiens. Through language, ritual, art, and contest, humans express their creative and cultural instincts in the spirit of play. This definition frames play as more than simple amusement, to a cultural force that shapes meaning, community, and the human spirit.
As this relates to my thesis I'll have to do some more in-depth reading in how play can promote learning and AI literacy. The shaping of meaning and community seems like it could be a good stance to explore further, and relating it to the ideas of biology and AI I've explored too seem like a connective tissue that can be formed.
Where the Next Steps are Leading
Again, I'll need to continue refining my projects / installations. My muse 2 has been in the mail for some time now, but it should be arriving in the next couple of days so that I can explore that more. I'll have to refine the emergent garden interactions as well, and in working with the Rorshach idea I'm thinking if there is potential in combining it with the cognitive control to fully encompass the transference of meaning, collaboration, and cognition from emotional, physical, and perceptual levels. Aside from projects, the interviews are coming along. Once I've done a couple of those and have the transcripts, I can start coding them for consistent themes and work towards the workshop content.
This week as most of us know was for our big presentations, so that is where a bulk of my making went, through slides and formatting. In the making of the slides I feel like I had a chance to think in a different context about my work. It hasn't been so much in an isolated circle, but to formally present to those with no context definitely had me thinking more critically about what words go where, and how to convey certain concepts for the best understanding. Before any feedback while I was still developing the slides, I was going back and forth between the words of play, creativity, interaction, experimentation, and experience, and what they actually mean. I think from where I was in the beginning of the semester, these words, their definitions, and what they mean in the context of my thesis are making much more sense, but there is still more to be done in that area. Drawing from Tina and Silvia's feedback, I have done a lot of thinking on what my audience should be, but what I want them to feel is something I'll have to think about more: where play fits into understanding and learning. I think the title "Playful Interaction" steps into a right direction, where it's not entirely play, and places concepts like participation and experience tangentially with the concept of what play is, so maybe play as a word will make less and less appearances in my thesis from this point forward.
Before I step into Rao's feedback, I'll briefly cover some more applicable "things" I've made for the thesis: these tools for the workshop. The idea behind them is to make TouchDesigner more accessible so there's less development and learning of the program itself, and more time to focus on learning an AI workflow and fast-tracking results. I've been building some tools and gathering some from free open sources, with the idea of being able to utilize them in either the pre-processing and/or post-processing areas of my flow-chart. These tools include now but are not limited to: a mouse drawing tool, machine vision made easy for face / hand tracking, filters made easy like halftones, chromatic aberration, displacement and noise, fundamental "mini-scripts" that are essential for movement in TouchDesigner (absTime.seconds()/absTime.frame makes an object move with the seconds passed or amount of frames), drag and drop sine and cosine waves, and a poster tool that makes text, shapes, and borders easy to add for an easy composition. There isn't too much to show of these tools as they all take the appearance of TouchDesigner node blocks, but I'll include screenshots.
From mostly Rao's feedback and some of the other comments, I think that there is some good things to address here. Mainly for quantifying what people take away from AI literacy in the context of the installations. I think from the interview and workshop protocols there is grounds to gauge what working practitioners have gained literacy wise from working with AI, and what those with less experience stand to gain through the workshop, which maybe I could have articulated better in my presentation. This is a possible area of concern for the installations though, as currently there is no metric for what people gain literacy wise from viewing and interacting with the pieces. Perhaps this could be an added leg of research, doing something similar to what Mira Jung did with her soil research, having groups of participants view and interact with her pieces, then answering a questionnaire to see if there was any knowledge gained. I feel like I would have to decide this pretty soon though, since the installations themselves would have to be finished by the end of the semester, a space to set them up would have to be decided, and IRB would have to be approved. I think it would be valuable for my thesis, but I'll have to think a little bit on the next best steps.
What I’ve Read
Continuing my readings in Transcending Imagination, Chapter 8 treads towards more thought experiment and philosophical ideating, with topics like sentience and emergence, which are relevant topics for AI but aren't particularly relevant with current issues with AI, and what my thesis is going towards. He does make a distinction which I air towards in my presentation, the separation of technical processes vs. analytical critical processes. In the case of sentience he talks about embodied functions, like mathematical and logic-driven operations, and emergent functions, such as judgment, pattern recognition, or aesthetic perception. He says that sentience is less about super-intelligence and more about the capacity of AI systems to produce outcomes that evoke understanding, reflection, or emotion in human observers. He then introduces this concept of autopoietic intelligence, which describes a self-maintaining system, and draws more parallels to biology and evolution/adaptation.
I don't think I'll cover this AI sentience in my thesis, as I want to keep the focus on the people who use AI and how to increase their knowledge perceptually. Sentience is still also a very conceptual thing, and fits into the hype/hate cycle as a kind of unknown. Unknown if it would even be possible, from the sheer resources required or otherwise, but I don't think it's particularly important in the here and now.
Where the Next Steps are Heading
After this week the next steps are to keep working on the installations and reach out to interview candidates. I set up a calendly to get times scheduled and make the process easier on them, so hopefully I'll have some scheduled or at least the beginning of conversations coming into the next week. For the installations the first step is to work on getting the network set up for non-sentences. Once I have that, I think I will pivot to the Emergent Garden and getting the hand-tracking and gesture as interface aspects of that working for a live installation setting.
This week was a big focus on getting the Non-Sentences project up and running, and I made some really good progress on it, with most of the backend up and running, minus 1 piece to tie it all together. Quickly on the experience of making it first, it's always nice to learn more about design programs, TouchDesigner being one I am particularly fond of these days. Learning more about it will of course be beneficial in being able to talk about it and teach people about it on a basic level for the workshop, and while there are some things that can be easy for newcomers to pick up and start making things right away, especially with AI involved, this project tested my knowledge on the actual coding and data structure side of things, which was a nice challenge I've mostly overcame.
Below the sketch I made initially just below here to refresh on the beginning of the project vision, it involves the sentences being generated in real-time on a loop (with the words being chosen at random, but in a fixed order that would fit a "grammatically correct" sentence structure) displayed on the right side of a screen, while the left would show the table of words it can choose from and the network connecting all the words together.
A pretty basic sketch, but it's beginning to take form in the short recording below, where I have the table of all possible choices next to the generated sentences being run in real time on a loop.
I think the most interesting one in this video just so happens to relate to the project itself: "Our intelligent network expands into your facilitated language."
Of the progress I made this week, I wasn't quite able to get the network of lines from word to word on the table. It took me the most time to get the sentences to be able to generate word by word, then erase in a loop. As I said before TouchDesigner can give newcomers the ability to make pretty interesting things relatively fast, but with the nature of this project I ended up using a pretty node-less system, having the bulk of the sentence generation in one text block DAT, which took more time than if it would have been a pure node system. Below is a screenshot of the tables and the small number of nodes used, and a screenshot of the 'sentence_controller' text file.
While not specifically using any AI model (for now, I'll discuss near-future plans for this a little further down), the conceptual basis of this project is that it would shed some light on the process of how a chat bot comes up with strings of characters that can be interpreted as sentences by us. This combination of database driven algorithm and human pattern-seeking intuition gives these strings a meaning for us in the form of sentences. Even if they are semi-random (true random being a contentious topic in the world of computers), our meaning-making machines of brains decide when these random strings make sense, and even if they provide some value in the form of poetry. On the other hand, the random nature of the algorithm can spit out sentences that don't make sense: "Non-Sentences", which is where a conversation on AI-hallucinations comes in. Chatbot hallucinations make sense in the fact that they are grammatically correct sentences that you can read, but the information can be wildly and totally incorrect, which leads back to us and makes our ability to be able to detect and interpret these hallucinations for what they are much more important. This is AI literacy not in the technical sense, but the interpretive sense, and the ability to read AI bogus will become a more important skill to have when things like scammers and misleading ads on social media or the news have already begun implementing these chatbots into their writings. That's why in the tables I made it a point to use words that AI prefers to use: latin-english buzzwords of higher than normal complexity, like "delve", "underscore", "bolster", and "transformative" to name a few.
To add onto this more, obviously the next step in my vision for this project is to have the "network" lines generate from word to word along with the sentence, then erase and start over again, along with the sentence. This could be a pretty complex step but from how the tables are set up in TouchDesigner I think I'm in a good position to make it happen relatively quick. To hone more into the linguistic side of it (and maybe how AI is changing our language as per The Verge's recent blog post and the research behind it (paper 1, paper 2)), I've dabbled around with adding a text-to-speech model to read the sentences as they come out. I've experimented with gTTS, googles python based text-to-speech model, which can be easily implemented within TouchDesigner as it's a python based environment, but as I've gotten it to "work", it is pretty glitchy with how the sentence is being spoken, currently stumbling over the sentence over and over again at a fast unintelligible pace. A next step could be getting it to slow down and read word for word by using similar logic for the sentence generation, just going word for word as opposed to combining them all. Another option could be using ElevenLabs for speech, as they have plugin API's that work in TouchDesigner, and may offer some more intuitive, quicker, and more extensive control.
Looking briefly towards other projects, for the Emergent Garden I've thought about modes of interaction and I think that using hand tracking as a form of activation and control would be a good way to expand more on the idea of cultivation of a technological ecosystem, where by using motions like finger pinching or flicking the hand, you can place or remove primitive shapes to add to the ecosystem, and change the AI's parameters to create new evolutions of plant and animal organisms. Adding this semi-tactile form of interaction lends more to the idea of human intervention in the ethical use of AI, speaking more towards the balance between control and collaboration. The act of “tending” to the system through motion could represent the ways humans guide, nurture, or even disrupt technological growth. It would also introduce a more embodied and intuitive relationship with the work, allowing the audience to feel physically connected to the generative processes at play. This connection can enhance immersion and reinforce the metaphor of co-creation, where human gestures influence the evolution of an AI-driven environment, mirroring the ongoing negotiation between human intention and machine autonomy.
What I’ve Read
For readings I've continued with Transcending Imagination, which I think is proving valuable for this project-based portion of my thesis in investigating the literacy of AI as it pertains to its effects outside of technology, especially in this chapter (only one this week, most of my time is now spent on making as per my thesis timeline).
In Chapter 7, the term “narrated economy” is introduced, which reframes the designer’s role from fabricator to narrator, bringing more emphasis to the articulation of intent and meaning as a "new" creative act. I think that is ties into where I'm going with my projects and prototypes, with the idea that interaction itself can become a form of inquiry, with storytelling, gesture, and experience as tools for making AI’s invisible processes and effects visible. By positioning AI as an active participant in creation, Manu says that the designer’s task is no longer to simply use AI but to converse with it and to narrate, test, and reflect on its responses. Through this thinking, the interface becomes a "site of revelation", where AI’s biases, limitations, and assumptions can be surfaced experientially rather than explained abstractly. The resifting of human-centered, narrative-driven design also shows the ethical and aesthetic dimensions of this shift. If designers are now “architects of experience,” as he describes, then we must build spaces where people don’t just observe AI outputs but feel and reflect on how those outputs are shaped, which I can see in my own approach to experiential design as a reflective medium, using interaction, visual form, and sensory engagement to expose the underlying mechanics and ideologies of AI systems.
Where the Next Steps are Heading
Of course in the immediate future I'll be needing to put together my presentation, which I've outlined fully, just a matter of getting everything onto slides and writing a pseudo script to follow the beats to. After that I'll continue on the Non-Sentences project to at least what I had in my original vision of the sketch, expanding on it if it doesn't take too much time in areas of relevant text effects or text to speech plugins in TouchDesigner. After that I plan on ordering my Muse 2 for the brainwave project, then reaching out to my shortlist of creatives who implement AI technologies in their works for interview times, continuing to work on other projects in the meantime, such as refining the emotive-control project I'd done already, or adding hand-tracking capabilities to the Emergent Garden.
This week I did not do a ton of design work as opposed to finalizing my research protocols for the workshop and interviews and getting that submitted, and almost immediately approved by the IRB exemption review wizard. Compared to the last instances I shared on the last report of the interviews and pre/post survey protocols, I was able to refine and reduce the content in a more appropriate manner. For the interviews, I reduced the master list down to 11 questions that should vary in the time it takes to answer them, ranging from longer form process explanations to questions that can be answered relatively shortly. This is to be respectful of their time and give me the time to find a suitable number practitioners using AI, with an interview time of 30-45 minutes allowing me to recruit more participants compared to an hour or more long time. I am aiming for 3-5 practitioners in various fields in order to get some varied responses as far as their outcomes in playing with AI and learning about AI and technology in their creative process of making. For the pre/post workshop surveys, I restructured the questions to make them more suitable for comparison of data before and after the workshop (for some of the questions, they are 1-1 in the pre and post surveys, making it easy for direct comparison), and making them more neutral and less leading, in order to not assume outcomes from the workshop itself. Another thing that Alex suggested was to have a scale of 1-4 for the Likert-style questions in order to actually push participants over a threshold as opposed to having a neutral option that does not give much room for a baseline improvement or regression. The baseline would be if they answer the same choice for the question before and after the workshop. Now that this is out of the way, I can start planning more in terms of the workshop, things like sourcing materials and refining more of the slide/workshop content; and finding participate candidates for the interviews.
Additionally with that out of the way, I began sketching for the design project portion of my thesis. I accomplished a couple sketches, maybe a little less than I was hoping, but it's still good to get the ideas out that I've been putting off for the sake of scoping my ideas down. I'll post the one's I got the furthest with at the time of writing, and briefly explain the ideas behind them, and keep in mind they are really rough. As a whole, these prototypes and what comes after will seek to communicate and teach those engaging with these things about AI that aren't necessarily on a technical level. While there still are technical aspects that would be included in each one so far, in the big conversation of AI it is important to learn about it's affects either psychologically, cognitively, or conceptually. Additionally from the pieces themselves, I think having some standees or something equivalent can help communicate the ideas, similar to what you'd see in a museum or gallery of an artist statement. Lastly, with all of these I think it is important to include some form of process within the final design, either as side-by-side network that showcases what's going on, or an abstracted visual. This would be to further increase transparency and learning of what the model's and processes are actually doing behind the scenes, by bringing them to the foreground.
First is the idea of using a brainwave monitor as a conduit for driving AI Image generation. The monitor (A Muse 2), would be connected to a phone app that creates OSC data on the 4 main brainwaves (Alpha, Beta, Delta, & Gamma) which can be wirelessly communicated to TouchDesigner, and thus an AI image model. Aside from an interesting and interactive installation idea, this can be used to communicate some cognitive effects AI can have on people using it, and to communicate the beginning of the cognitive chain is the user and how they interact with - or don't interact - with an AI. This could also be expanded upon with more interaction modes than just the brainwave monitor. With a web cam and machine vision setup, body and motion detection can be combined with this to provide a whole-body interaction, mind and movement together in combination with an AI model. In a bento-like layout, you'd see your brainwaves next to the visual they are helping to create (and possibly the body tracking nodes if that is to be included).
Second is the concept of a "non-sentences" project. This actually would not inherently use an AI image generation model, as it is primarily focused towards pulling back and making more apparent the processes of text generation. Gathering a database of words and separating them into where they would fall into a common sentence structure (Articles, nouns/verbs/adjectives/, Prepositions, etc.), and having them semi-randomly form a sentence. This would visualize in an abstract way how text generators work in predicting words and characters to make things that sound like sentences to us, pretty convincingly for the most part too, but also showing where things can go wrong with hallucinations that can half make sense or are just nonsense, revealing behind convincing words and images is just an algorithm at heart. Again, there would be an accompanying visual of the process behind this generation, in the sketch envisioned as a network that links these words together as they are spit out. While there is a structured process, it's nature can still output nonsense, or "non-sentences".
The last sort of sketch that I did not get too far in involved creating a more interactive form of the Emergent Garden, or the Emergent Ecosystem, through things like midi pads or a drawing tablet. I'm on the fence about making this piece interactive however, as some of the idea behind it was making parallels to the unconscious algorithm of biological evolution where the only influence is the underpinning search algorithm of favorable mutations for survival. However, an interactive component could lend to the idea of cultivating curiosity and care, highlighting where our choices cause chain reactions that evolve depending on the circumstances. Regardless, I think the emergent garden poster series as an evolving standalone piece with no interaction can still fit well within my thesis as a conceptual learning piece on what values AI images can have in the context of image space and the evolution and iteration of ideas. That alone would involve a little more work too, as the pieces seen in the video I made involved a little bit of trickery due to time to showcase the concept, having them actually live for viewing will involve a little reformatting.
Overall I think there are some good ideas here that I will have fun exploring into next week. What I plan to tackle first is the "non-sentences" project, as I think it is something I can do relatively fast with the resources I currently have on hand.
What I Read
Reading is taking a further backseat for the actual creation of things, and will probably continue to do so, but I have kept up slightly with "Transcending Imagination", getting through chapters 5-6.
In Chapter 5, Bias and Creative Intent, the book explores how artistic creation is a constant negotiation between skill, intention, and limitation. Every decision: form, color, or concept, involves compromise and reflects the artist’s biases, shaped by experience and perception. Contrasting this with AI-generated art, where algorithms operate free from personal tendencies but are guided by data and semantic intent, forming their own kind of systematic bias, arguing that AI changes the artist’s role from sole creator to facilitator, fostering a symbiotic relationship in which humans guide and interpret machine output. This challenges traditional notions of authorship and originality while expanding the scope of creativity. Bias, both human and algorithmic, is reframed not as a flaw but as a force that can inspire transformation and reveal hidden assumptions within creative intent. this dual-bias is something I definitely would like to explore in future sketches for one of my prototypes.
Chapter 6, Maximizing Creativity, focuses on how AI enhances the articulation of human intent and amplifies creative capacity, discussing how generative systems allow for a deeper and more precise expression of ideas, acting as bridges between abstract thought and manifestation. The author argues that by expanding the language of creation, AI strengthens our ability to communicate emotion, philosophy, and narrative with greater clarity and depth. This chapter concludes that AI doesn’t diminish creativity, but multiplies it, transforming how humans express themselves and reinforcing the shared relationship between technology, thought, and imagination. I think there is some validity to this, but in a way all of AI is predicated on human ideas so to what extent it multiplies human creativity I am unsure, which is an area of thinking my workshop plans to tackle.
Where the Next Steps are Leading
I've outlined a couple next steps already, as they are pretty natural based on where I'm at right now. To summarize, I want to confirm the practitioners using AI that I would like to interview, and I already have a list of people I can reach out to, but there's never any guarantee they will reach back out or be interested in an interview. This is why the goal participant target is 3-5, to account for the possibility of people not responding. Aside from that, my main focus for the rest of the month is developing these prototypes further from their sketches, entering November with solid pieces that I can include in the thesis. Additionally, I will begin to start drafting the written portion of the thesis, starting with literature review and working from there. Nothing super finalized, just putting the pieces in place.
This week I primarily focused on narrowing down my research protocol for expert interviews and the workshop - the workshop including a pre- and post-survey to assess participants outcomes in the interpretability and explainability of AI tools / systems. I needed to do this soon for IRB and for my thesis as a whole, but this also lead me to further thinking about my research questions in a different way, and refining them to something I think is more "sticky" that elicits and reinforces the two "big" aspects of my thesis: The body of work that I'll make, and the workshop itself. I think it was a more natural way of coming up with more refined and solid questions as opposed to sort of brute force thinking of questions for the sake of it: What questions can I answer by interviewing partitioning artists using AI and conducting a workshop centered around playing with AI?
The question I think will be answered mostly by the interviews and workshop is: Can experimental play and creation with artificial intelligence (AI) tools and systems make said tools more interpretable, explainable, and transparent?
I think firstly by learning about the processes behind current practitioners, I can gauge how much they "play" with AI, and what outcomes they've had with it, such as how much they've learned about AI and Technology by making with it, or any hard or soft skills they've gained or deem important when using AI. A preview of two questions:
Describe your process from start to finish, from conception to final output. Explain when, where, and how and what AI was used.
What, if anything, have you learned about AI itself (its limits, strengths, inner workings) from making creative work with it?
Did engaging with AI change your understanding of technology or computational systems more broadly? If so, how?
I'll need to reduce the number of questions for this protocol, I plan to make the final protocol and interview take about 30-45 minutes, so I'll either need to combine or cut some of the weaker questions, the master list so far can be found here.
The pre- and post-surveys for the workshop I think came much easier to me, and assess the outcomes the participants will have in the workshop, such as their knowledge and perceptions of AI and play coming into the workshop, and their reflection on AI systems after the workshop - including a creative reflection and artist statement for their body of work. I think with just a little tidying up these can be good to go, and next I'll have to think about how to deliver them in the final workshop (probably a google form is easiest). The pre-survey and post-survey protocols are linked respectively.
The second research question which I seek to explore in the actual body of work - through prototypes and creative works, still needs a little work. Generally, it would be along the lines of: How can experiential design and artistic practice using AI systems... answer or reveal something about AI. I think there are a couple ways I can go about the ending of the question, such as going along with the first question about making AI more transparent and "giving it back to the people", separating it from the institutions that push out these systems, like we discussed briefly in reflecting on the video I made. An alternate leg to this question could be a sort of investigation in things like biases, distortions, and ethical challenges in AI, which is hard to separate from the medium/material/tool/genre and discussions surrounding it, or it's potential as a teaching tool for different areas (thinking of a cross section between sciences and arts), or as a means to bring awareness to social issues, all of which I think are potentially viable directions to go, and are the basis of previously existing research which I'll cover shortly. I also think the "experiential" design aspect is important to include with what I plan to make, which will include things like Machine Vision and body/hand/face tracking, and ultimately the piece using a brainwave monitor. Another aspect of research I'll cover below.
What I Read
Two papers I read this week parallel to the protocol writing helped me think about my research questions, and they primarily focus on the experiential aspect of AI to explain AI (and brought up a term I didn't know about before: Explainable AI or XAI) and make awareness to larger issues.
The first was simply called Experiential AI, and set up the term “experiential AI” as a way for artists and scientists to come together and make algorithms tangible, visible, and more interpretable. It argues that art can serve as a bridge between opaque computational processes and human understanding, offering new ways to make reasoning processes decipherable. The emphasis on art’s ability to map the “inter-agencies” relates well to my own work and thinking so far, connecting directly to my own focus on embodied and playful experiences with AI—things like tracking movement or brainwaves as an entry point into the systems. This was more of a proposal than a body of work, and was made in 2019, and as such the next paper built on this further.
The second, Experiential AI: Between Arts and Explainable AI, written by the same author(s), builds on that foundation. It critiques the limits of technical “explainability” and suggests that experiential, arts-based methods can go further in making AI interpretable. Their “4As” framework, aspect, algorithm, affect, apprehension, was particularly interesting, and I think can be used in further development of the workshop as opposed to isolated installations or works. It frames experiential AI not just as an artistic add-on but as a methodology: one that connects technical models to embodied human experience and allows people to make sense of AI on multiple levels. Their case studies, like Jake Elwes’ The Zizi Show or Anna Ridler and Caroline Sinders’ AI is Human After All, are great examples of how art can expose bias, hidden labor, or socio-technical entanglements in AI.
Reading these papers on experiential AI helps me situate my research in context of what's going on right now in similar veins, where I'm not just making AI “explainable” in a narrow sense, but in creating experiences, through interviews, workshops, and creative works, where participants can get a more hands-on experience with AI through play and alternate interaction, and, in that process, make its mechanisms and implications more transparent.
Where the Next Steps are Leading
Immediately next I need to finish up the protocols to submit to IRB and get interviews going this month as soon as possible. Like I said the interviews will be valuable to know what practitioners have learned about AI themselves through their own experiences in working with AI on their own and for longer than the participants would have, assuming they come into the workshop with 0 knowledge. After submitting the protocols, in the time it takes for IRB to look over it, is when I plan on working on these experiential prototypes, starting out with sketches and then moving into the actual making.
//about
Ryan Schlesinger is a multidisciplinary designer, artist, and researcher.
His skills and experience include, but are not limited to: graphic design, human-computer interaction, creative direction, motion design, videography, video-jockeying, UI/UX, branding, and marketing, DJ-ing and sound design.
This blog serves as a means of documenting his master’s thesis to the world. The thesis is an exploration of AI tools in the space of live performance and installation settings.
un-prompted 