Generative AI in UX/UI design

Abstract

Background
This paper presents insights from a student project conducted within the “Intelligent Machines” module of the Digital Management program at Hyper Island. The project was in cooperation with Samhall.

Method
Generative artificial intelligence (AI) was utilized during the initial design and development phases. The study documents the process of using AI to understand project objectives, generate innovative ideas, and create graphical user interface (GUI) prototypes, contributing to a better understanding of AI's role in design.

Result
The findings highlight the strength of generative AI in the ideation phase, enabling the generation of diverse ideas. However, they also indicate the current limitations encountered with specific generative UI tools.

Conclusion
Generative AI can enhance design efficiency and creativity but poses challenges that require thoughtful management. Designers should employ AI collaboratively, fostering innovation while maintaining critical oversight. Achieving successful integration demands balancing benefits with ethical considerations and addressing biases, thus advancing design practices in the digital age.

Introduction

This paper is written for UX/UI designers that are interested in the application of generative AI in the design process. Specifically, in generating ideas and prototypes. It explores how those tools can enhance the quality and speed of design work and is therefore also of interest for businesses like Samhall.

The paper is structured into four sections. The first section reviews literature from the perspective of a design-researcher. It tries to incorporate a broad set of literature from the perspective of computer science, businesses and design. It gives clarity about the terms and functionality of AI, how generative AI models work, as well as explains various tools used in the process of the project.

The second section proposes how generative AI can be used to design and develop prototypes. Potential benefits and values with this technologies will be outlined and a critically reflected on an ethical, social and privacy level.

In the third section the process is discussed and it is shown how the decision-making was shaped by the application of the proposed tools. The analysis will be informed by insights gained from simple prototypes, providing a clear foundation for the high-level concepts explored in this study.

In the fourth and last section a critical reflection on my experiences working with machines, software, and hardware technologies to develop solutions is stated. I will examine the potential disruptive impacts that these current and emerging digital technologies might have on both society and my own professional practice.

Artificial Intelligence, Machine Learning, Neural Networks and Deep Learning

In this chapter, an introduction to AI is given. It is not covering the underlying mathematics but abstracting it to allow the reader to understand implications in the application of AI.

Artificial Intelligence (AI) is characterized as the capability of machines to emulate human intelligence and cognitive functions, such as problem-solving and learning (Sarker, 2021; AI vs. Machine Learning vs. Deep Learning vs. Neural Networks | IBM, 2023). This definition is one of many available, as elucidated by Collins et al. (2021). Notably, the academic community has yet to establish a definitive definition. To achieve AI, machine learning (ML) is employed, an algorithm specialized in pattern recognition (França et al., 2021; Thantilage, Le-Khac and Kechadi, 2023). França et al. (2021) emphasize the significance of substantial data for ML algorithms to discern patterns. These patterns can encompass various domains, including visual tracking, text classification, and object tracking (Singh, 2023).

A notable advantage of ML is that developers do not need to explicitly program a rule set; instead, the machine autonomously determines the rules through decision trees and linear regression (Deep learning vs. machine learning, no date; Sarker, 2021). ML finds applications in a diverse range of problems, from product recommendations on e-commerce platforms to medical diagnoses in the healthcare sector (Sarker, 2021b; Taye, 2023). Taye (2023) highlights that ML algorithms can be trained on limited datasets and require human intervention during this process to yield accurate results. Sarker (2021) further notes that as the dataset size increases, ML algorithms encounter a performance plateau. This is where deep learning emerges as a solution. Deep learning is defined as the training of multi-layered neural networks to make predictions and decisions (Hashana et al., 2023). While deep learning necessitates more data for training compared to ML algorithms, its primary advantage lies in its ability to learn from raw, unstructured data and rectify its own errors to enhance its performance (Taye, 2023). Hashana et al. (2023) underscore the role of deep learning in the development of tools such as ChatGPT. Deep generative models exhibit diverse strengths and weaknesses (Bond-Taylor et al., 2022). Generative Adversarial Networks (GANs), proposed by Goodfellow et al. (2020), are among these models. They fall into the category of generative AI, which is an deep learning algorithm, that can generate content like text, images, audio or any form of data (What Is Generative AI?, 2023).

Due to the architecture of deep learning algorithms and the collection of data, they encounter several challenges when being applied to real-world applications. Those challenges include the limited diversity of responses, the accuracy and completeness of the generated content, consistency, and the potential for bias (Tyagi and Rekha, 2020; Akter et al., 2021). Aslan (2021) gives the example of only using British accents for the training of a voice recognition model would lead to a technology that has trouble understanding Indian accents.

Deep learning technology is utilized in applications such as ChatGPT (Hashana et al., 2023) and can be employed to generate user interface designs (Nguyen et al., 2018; Zhao et al., 2021). Framer AI and Galileo UI are services that provide this functionality.

The impact of generative AI on the economy is substantial, as it has the potential to significantly enhance productivity (Chui et al., 2023). Al Naqbi, Bahroun, and Ahmed (2024) further emphasize that the interest in generative AI’s ability to improve employee productivity has been steadily increasing over the past few years.

Generative AI in the design process

In this chapter we will look at how generative AI can be used in the design process to understand the client brief, brainstorming ideas and generate UI designs for the following project. Furthermore, we look into how this can be relevant for the chosen organization.

The project, that is used as a reference, was done in collaboration with Samhall. Samhall hires people with different disabilities, educates them and finds the best matching job at regular companies, where they can start working (‘In English | Samhall - Sveriges viktigaste företag’, no date). The goal is that the employees leave Samhall after a certain time and get employed by a regular company. The project’s goal is to help Samhall in this process by streamlining and digitalizing it, specifically in the introduction phase. It is important to know, that Samhall is not using any existing digital tools and did not provide the possibility of talking to staff and clients. Therefore, the following suggestion is focusing on a situation where a project starts without any legacy tools and no possibility of user testings.

Empathize and Define

Understanding the problem is an essential part of the design process. AI algorithms can process vast amounts of information, uncovering patterns and insights that might be missed by human analysis (Csernyei, 2024). One can ask tools like ChatGPT to analyze the client brief and generate follow up questions. Or using it to do user research by listing user’s pain points, creating personas or generating interview questions (Soegaard, 2024). The use of ChatGPT supports the designers in identifying the problem and offers broader perspectives when working collaboratively with AI tools (Papachristos et al., 2024). This allows designers to empathize with unknown situations much easier and faster. In the case of the collaboration partner Samhall, it could be used to generate personas of people with certain disabilities and asking them questions.

Ideate and Prototype

At some point, when designers have the feeling of having understood the problem, the next challenge is to come up with ideas that tackle this problem. Tools like ChatGPT can be a helpful tool to do so (Tholander and Jonsson, 2023; Papachristos et al., 2024). It can used to propose solutions on different levels, from the general strategy down to what specific features a hypothetical app should have. This brainstorming functionality is especially useful, when the designer works on his own, as it can help bringing in an other perspective and eliminate biases.

But one is not limited to the generation of text. With the help of DALL-E or mid-journey images can be generated (Kwon, Jung and Kim, 2024). Those images can be used to develop a concept or an idea. Through describing, an image of an idea or vision can be generated within seconds. This is considered to be of great value according to Kwon, Jung and Kim (2024). A potential benefit of using pictures, compared to text, is that they allow for more interpretation (Papachristos et al., 2024). And sometimes they completely miss the intention of the designer leading to a “creative friction” that inspires (Tholander and Jonsson, 2023).

Last but not least, AI powered tools can be used to generate prototypes of first ideas. Those prototypes help tremendously in the decision making process (Coutts, Wodehouse and Robertson, 2019). Tools like Framer AI or Galileo AI can be used to design prototypes of UI’s from a prompt. Normally, the crafting of UI designs is manual and time-consuming labour (Troiano and Birtolo, 2014). This rapid generation of prototypes brings not only the benefit of saving time, but also of not being limited to a few prototypes or ideas but quickly iterate through multiple ideas and concepts.

There are also certain issues when working with AI in the design process. Although it is true that working with AI can increase the diversity of ideas it can also result in the opposite outcome when not using it carefully. As described in the first chapter, the output of an AI heavily depends on the data used in the training process (Akter et al., 2021). Speaking of material comes a privacy and social issues. A lot of generative AI models are trained on private or copyrighted data (Balaji, 2024). For example Figma, a famous UI design tool, uses work from designers to train their AI models as can be seen on the provided screenshot from the program (Figma AI Content Training Settings, 2024). This data is then used to further train their model, making it superior compared to models that cannot use the same set of data (Radsch, 2024).

Figma AI Content Training Settings (2024)

Besides unfair market circumstances, AI service providers also face privacy issues. Liu et al. (2021) explains that it is possible to inferrer sensible training data. This is especially harmful, if confidential data is used for the training of deep learning models.

Applying AI

In this chapter the process of our project is explained and discussed. With visual examples, evidence is shown of using tools and techniques to ideate and create, test and prototype of a solution for Samhall.

The project was handled by an interdisciplinary remote team, consisting of four members, including me.

Generating ideas

I started using AI in the process to understand the client brief. Todo so, a tool called BoltAI was used, which is a wrapper for different AI models.

The first prompt to ChatGPT-4o was the instruction to propose ideas for a digital tool for Samhall, but before answering, asking clarification questions on the brief, which was also provided in the request (Appendix B, 2024).

The generated clarification questions were very good and aligned well with what the team was thinking. When we got the answers from the client and feed them back to the AI model, six ideas got generated (Appendix B, 2024). The presented ideas were all heading in a good direction. They were rather described at a surface level, but provided a good starting point for discussion.

When the ideas were presented to the team, the response was not very engaging. Every team member had a lot of ideas on how to tackle the challenge. But we had troubles deciding for an idea and start drafting a solution.

Generating images

At one point in the process, out of frustration due to a lack of progress, I started generating images with DALL·E 3, that could show a vision for the potential future at a Samhall interview process. The used prompts and generated images can be found in the appendix C (2024).

When I shared them with the team the reaction was much more engaging, compared to the previously generated ideas in text form. There were statements like “Samhall cannot afford something like this, we need something simpler like an app” (referring to DALL·E 3 transcription of interview by AI) or “we must make sure people are not treated like this” (referring to DALL·E 3 cognitive test).

DALL·E 3 transcription of interview by AI (2024)

Generating Prototypes

We agreed on building an app that helps Samhall organizing the introduction process, I tried to generate different first drafts using framer.com and it’s AI functionality with a prompt (see Appendix A). Unfortunately, the results were not what was expected. Instead of delivering a screen of an audio recording app, with transcription functionality, a generic landing page advertising for a transcription app was created (see Framer AI Result 1 with context, 2024).

Even after changing the prompt, the results stayed the same (see Framer AI Result 2 full screen). I quickly realized, that I expected something from the AI that it was not intended or trained for. Framer is a website builder and most likely the underlying AI’s goal is to support users creating websites.

Therefore, I switched to another tool, called Galileo AI, which is available under https://www.usegalileo.ai. With the same prompt used as before (see Appendix A) the result, visible in the image Galileo AI Result 1 (2024), was generated.

The result much more matched my personal expectations. Nevertheless, the generated concept lacks some depth. It is very simple and not a very elaborated screen. It lacks looking beyond the request and suggesting styles, texts or elements, that inspire me as a designer for further improvement. I assume that the reason for this behavior, is lack of reference or training data. When the instructions is used to generate a mobile e-commerce store for pottery, the generated results are much richer, as can be seen on the image Galileo AI result 2 (2024). I assume again, that there are much more e-commerce designs existing than audio recording and transcription designs, which can be used as training material for the AI.

To summarize this chapter we learn that it is crucial to use the right AI model for the job. Some models are simply not trained to accomplish certain things. Furthermore, we need to acknowledge that solving specific problems, which are unique or very context dependent are harder to solve for AI’s than standardized problems. The tested tools lack the possibility to provide context to a prompt. Last but not least, it is hard to come to a specific result, when using prompts to generate designs. It takes a lot of time to write down detailed instructions. At a certain point it is faster to manually start crafting the prototype in a UI design tool. Nevertheless, the tools are very helpful to start ideating and have something to build on. The final UI designs of the prototype can be found under appendix D (2024). When the generated UI designs are compared with the Figma Final Design 1 (2024), it becomes clear that they are much richer in terms of detail. This is the result of many decisions based on a lot of information and context known to the designers.

AI and I

In this chapter, the impact of generative AI within design practice is critically examined, highlighting both its transformative potential and the disruptive implications for design professionals and broader societal contexts.

Generative AI becomes increasingly integral to modern design workflows. Not only enables it designers and project teams to ideate and challenge ideas quickly, but also to visualize concepts efficiently, producing numerous variations rapidly—an attribute that accelerates the design process significantly, particularly during the initial prototyping stages (Tholander and Jonsson, 2023). This shifts the role of the designer. Instead of spending a lot of time and energy generating first drafts from a client brief, he can automize this process and focus on correcting details and improve those drafts.

However, I also see that the introduction of generative AI into the design workflow poses several challenges. One critical issue is the potential for information overload (Genovese, 2024). The ability to generate design artifacts instantaneously can lead to an overwhelming number of proposals and ideas, which, in turn, increase the already high cognitive load among designers and project teams (Dorst, 2019). In scenarios where multiple designers contribute ideas, the risk of an infinite number of possibilities can stall progress, making it difficult to arrive at a consensus or a final decision (Pilat and Krastev, no date). Therefore, it is crucial that designers do not swamp projects with carelessly generated proposals. A clear intention when using AI is key to success. Another danger I see is the lack of context of generative AI, which hinders it to do a reality check and understanding abstract concepts like justice or democracy (Tyagi and Rekha, 2020; Tholander and Jonsson, 2023). Deep learning is very shallow and does not have the ability to understand good and bad or wrong and correct. It does not have a motivation to perform good nor satisfy the user or the client. As long as the designers is not going to ask for a critical reflection on, for example, a proposed idea, there will be no such answer. AI will never disagree on it’s own with the user unless it was asked to do. As with any other technology, the output is always depending on the human input. Therefore, clear education and training on how to use those tools is key (Papachristos et al., 2024).

Designers must become adept at managing AI as a collaborative tool rather than a replacement, ensuring that while these technologies augment their capabilities, they do not overshadow the core creative and empathetic elements that define effective design. Professional practice in design will need to pivot towards mentorship, ethics, and a continuous learning environment, where understanding and guiding AI is as critical as mastering traditional design skills.

Last but not least, it is crucial for designers to stay informed about the developments of generative UX/UI design as it builds the foundation of outcome-oriented design (Moran and Gibbons, 2024). In the future, it is likely that fixed layouts and user interface designs disappear and are replaced by personalized user interface, that are tailored to the users needs. This could increase the user experience, as it could provide different user interfaces for a cognitive impaired person and a blind person. If this trend emerges, it is going to fundamentally change the job of UX/UI designers.

DavidWollschlegel_IntelligentMachines_RA

Download full research action with appendix

DavidWollschlegel_IntelligentMachines_RA.pdf

26 MB

Reference list

AI vs. Machine Learning vs. Deep Learning vs. Neural Networks | IBM (2023). Available at: https://www.ibm.com/think/topics/ai-vs-machine-learning-vs-deep-learning-vs-neural-networks (Accessed: 17 December 2024).

Akter, S. et al. (2021) ‘Algorithmic bias in data-driven innovation in the age of AI’, International Journal of Information Management, 60, p. 102387. Available at: https://doi.org/10.1016/j.ijinfomgt.2021.102387.

Al Naqbi, H., Bahroun, Z. and Ahmed, V. (2024) ‘Enhancing Work Productivity through Generative Artificial Intelligence: A Comprehensive Literature Review’, Sustainability, 16(3), p. 1166. Available at: https://doi.org/10.3390/su16031166.

Aslan, Ş. (2021) 9 Types of Data Bias in Machine Learning. Available at: https://www.taus.net/resources/blog/9-types-of-data-bias-in-machine-learning (Accessed: 22 December 2024).

Balaji, S. (2024) When does generative AI qualify for fair use?, When does generative AI qualify for fair use? Available at: https://suchir.net/fair_use.html (Accessed: 19 December 2024).

Bond-Taylor, S. et al. (2022) ‘Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models’, IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11), pp. 7327–7347. Available at: https://doi.org/10.1109/TPAMI.2021.3116668.

Chui, M. et al. (2023) The economic potential of generative AI: The next productivity frontier. McKinsey & Company, p. 68. Available at: https://www.mckinsey.com/capabilities/mckinsey-digital/our-insights/the-economic-potential-of-generative-ai-the-next-productivity-frontier#key-insights (Accessed: 21 December 2024).

Collins, C. et al. (2021) ‘Artificial intelligence in information systems research: A systematic literature review and research agenda’, International Journal of Information Management, 60, p. 102383. Available at: https://doi.org/10.1016/j.ijinfomgt.2021.102383.

Coutts, E.R., Wodehouse, A. and Robertson, J. (2019) ‘A Comparison of Contemporary Prototyping Methods’, Proceedings of the Design Society: International Conference on Engineering Design, 1(1), pp. 1313–1322. Available at: https://doi.org/10.1017/dsi.2019.137.

Csernyei, A. (2024) How generative AI is revolutionizing UX/UI design, Tapptitude. Available at: https://tapptitude.com/blog/how-generative-ai-is-revolutionizing-ux-ui-design (Accessed: 19 December 2024).

DALL·E 3 transcription of interview by AI (2024) [Generative AI] (Accessed: 3rd December 2024)

DALL·E 3 cognitive test (2024) [Generative AI] (Accessed: 3rd December 2024)

Deep learning vs. machine learning (no date) Google Cloud. Available at: https://cloud.google.com/discover/deep-learning-vs-machine-learning (Accessed: 18 December 2024).

Dorst, K. (2019) ‘Design beyond Design’, She Ji: The Journal of Design, Economics, and Innovation, 5(2), pp. 117–127. Available at: https://doi.org/10.1016/j.sheji.2019.05.001.

França, R.P. et al. (2021) ‘An overview of deep learning in big data, image, and signal processing in the modern digital age’, in Trends in Deep Learning Methodologies. Elsevier, pp. 63–87. Available at: https://doi.org/10.1016/B978-0-12-822226-3.00003-9.

Figma AI Content Training Settings (2024) [Screenshot] (Accessed: 19th December 2024)

Figma Final Design 1 (2024) [Screenshot] (Accessed: 22nd December 2024)

Framer AI Result 1 with context (2024) [Screenshot] (Accessed: 19th December 2024)

Framer AI Result 2 full screen (2024) [Screenshot] (Accessed: 19th December 2024)

Galileo AI Result 1 (2024) [Screenshot] (Accessed: 19th December 2024)

Galileo AI Result 2 (2024) [Screenshot] (Accessed: 19th December 2024)

Genovese, M. (2024) ‘Avoiding information overload from AI’, Planorama, 12 August. Available at: https://planorama.design/blog/avoiding-information-overload-from-ai/ (Accessed: 20 December 2024).

Goodfellow, I. et al. (2020) ‘Generative adversarial networks’, Communications of the ACM, 63(11), pp. 139–144. Available at: https://doi.org/10.1145/3422622.

Hashana, A.M.J. et al. (2023) ‘Deep Learning in ChatGPT - A Survey’, in 2023 7th International Conference on Trends in Electronics and Informatics (ICOEI). 2023 7th International Conference on Trends in Electronics and Informatics (ICOEI), Tirunelveli, India: IEEE, pp. 1001–1005. Available at: https://doi.org/10.1109/ICOEI56765.2023.10125852.

‘In English | Samhall - Sveriges viktigaste företag’ (no date) Samhall. Available at: https://samhall.se/in-english/ (Accessed: 7 December 2024).

Kwon, J., Jung, E.-C. and Kim, J. (2024) ‘Designer-Generative AI Ideation Process: Generating Images Aligned with Designer Intent in Early-Stage Concept Exploration in Product Design’, Archives of Design Research, 37(3), pp. 7–23. Available at: https://doi.org/10.15187/adr.2024.07.37.3.7.

Liu, X. et al. (2021) ‘Privacy and Security Issues in Deep Learning: A Survey’, IEEE Access, 9, pp. 4566–4593. Available at: https://doi.org/10.1109/ACCESS.2020.3045078.

Moran, K. and Gibbons, S. (2024) Generative UI and Outcome-Oriented Design, Nielsen Norman Group. Available at: https://www.nngroup.com/articles/generative-ui/ (Accessed: 22 December 2024).

Nguyen, Tam The et al. (2018) ‘Deep learning UI design patterns of mobile apps’, in Proceedings of the 40th International Conference on Software Engineering: New Ideas and Emerging Results. ICSE ’18: 40th International Conference on Software Engineering, Gothenburg Sweden: ACM, pp. 65–68. Available at: https://doi.org/10.1145/3183399.3183422.

Papachristos, E. et al. (2024) ‘Integrating AI into Design Ideation: Assessing ChatGPT’s Role in Human-Centered Design Education’. Preprints. Available at: https://doi.org/10.36227/techrxiv.171656320.09963657/v1.

Pilat, D. and Krastev, S. (no date) Choice Overload Bias - The Decision Lab. Available at: https://thedecisionlab.com/biases/choice-overload-bias (Accessed: 20 December 2024).

Radsch, C. (2024) Dismantling AI Data Monopolies Before it’s Too Late | TechPolicy.Press, Tech Policy Press. Available at: https://techpolicy.press/dismantling-ai-data-monopolies-before-its-too-late (Accessed: 22 December 2024).

Sarker, I.H. (2021a) ‘Deep Learning: A Comprehensive Overview on Techniques, Taxonomy, Applications and Research Directions’, SN Computer Science, 2(6), p. 420. Available at: https://doi.org/10.1007/s42979-021-00815-1.

Sarker, I.H. (2021b) ‘Machine Learning: Algorithms, Real-World Applications and Research Directions’, SN Computer Science, 2(3), p. 160. Available at: https://doi.org/10.1007/s42979-021-00592-x.

Singh, C. (2023) ‘Machine Learning in Pattern Recognition’, European Journal of Engineering and Technology Research, 8(2), pp. 63–68. Available at: https://doi.org/10.24018/ejeng.2023.8.2.3025.

Soegaard, M. (2024) ChatGPT for UX Design: 7 of Our Favorite Prompts, The Interaction Design Foundation. Available at: https://www.interaction-design.org/literature/article/chat-gpt-for-ux-design (Accessed: 22 December 2024).

Taye, M.M. (2023) ‘Understanding of Machine Learning with Deep Learning: Architectures, Workflow, Applications and Future Directions’, Computers, 12(5), p. 91. Available at: https://doi.org/10.3390/computers12050091.

Thantilage, R.D., Le-Khac, N.-A. and Kechadi, M.-T. (2023) ‘Healthcare data security and privacy in Data Warehouse architectures’, Informatics in Medicine Unlocked, 39, p. 101270. Available at: https://doi.org/10.1016/j.imu.2023.101270.

Tholander, J. and Jonsson, M. (2023) ‘Design Ideation with AI - Sketching, Thinking and Talking with Generative Machine Learning Models’, in Proceedings of the 2023 ACM Designing Interactive Systems Conference. DIS ’23: Designing Interactive Systems Conference, Pittsburgh PA USA: ACM, pp. 1930–1940. Available at: https://doi.org/10.1145/3563657.3596014.

Troiano, L. and Birtolo, C. (2014) ‘Genetic algorithms supporting generative design of user interfaces: Examples’, Information Sciences, 259, pp. 433–451. Available at: https://doi.org/10.1016/j.ins.2012.01.006.

Tyagi, A.K. and Rekha, G. (2020) ‘Challenges of Applying Deep Learning in Real-World Applications’:, in R. Kashyap and A.V.S. Kumar (eds) Advances in Computer and Electrical Engineering. IGI Global, pp. 92–118. Available at: https://doi.org/10.4018/978-1-7998-0182-5.ch004.

What Is Generative AI? (2023) The Interaction Design Foundation. Available at: https://www.interaction-design.org/literature/topics/generative-ai (Accessed: 19 December 2024).

Zhao, T. et al. (2021) ‘GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks’, in 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), Madrid, ES: IEEE, pp. 748–760. Available at: https://doi.org/10.1109/ICSE43902.2021.00074.