GENERATIVE AI: HOW CAN LEARNING AND DEVELOPMENT PROFESSIONALS LEVERAGE THESE TOOLS FOR PERFORMANCE IMPROVEMENT?
Generative artificial intelligence (AI) brings exciting prospects for learning and development (L&D) professionals as those tools can support their performance. This article illustrates how AI tools can help them jumpstart scenario-based e-learning projects with drafts of characters and scenarios, brainstorm evaluation questions, and suggest sample survey items with appropriate response scales. However, L&D professionals should exercise caution before uploading data to AI systems for data analysis, and may need multiple attempts with different prompts to generate appropriate outputs when using text-to-image AI tools.
The recent advancements in artificial intelligence and generative AI bring exciting prospects for L&D professionals as those tools can help improve their performance.
As AI technology evolves, users should continue to exercise caution and do their due diligence by, for example, periodically analyzing the evolving terms of service of AI tools and, in the case of research projects, explicitly informing participants of the extent to which AI will be used.
INTRODUCTION
In the history of technology development and advancement, new technology was often seen as a threat, worrying if technology would replace human performers and jobs. However, radios, televisions, cassette tapes, and computers did not replace human instructors. Technology rather helped us develop different ways to facilitate learning and development processes and improve overall performance outcomes.
In recent years, artificial intelligence (AI) has become a ubiquitous concept and practice. It is not just a buzzword; we find AI tools in our daily lives and in our workplaces. Compared to traditional AI that can make decisions or predictions based on a vast amount of data, generative AI tools are now a next generation of AI, capable of creating contents such as text, image, audio/music, video, code/software, data synthesis, and game design and development. For example, large language models, a type of AI, can understand and generate human-like conversational text, articles, or stories (ChatGPT 4, personal communication, March 7, 2024).
AI LARGE LANGUAGE MODELS
One of the popular AI large language models is ChatGPT, developed by an American artificial intelligence research laboratory, OpenAI. GPT stands for Generative Pre-trained Transformer, and the name ChatGPT represents the system’s main feature that engages users in a series of chat-like, question-and-answer type conversations (OpenAI, 2022). For example, in our previous paragraph, we found ourselves comfortable citing our conversational output with ChatGPT as “personal communication” (per APA, 2022). ChatGPT answers users’ questions in text format, by learning the information from large-scale datasets (OpenAI, 2022). Gemini, developed by Google (formerly Bard), is another conversational AI large language model (Google, 2023). While being capable of having multi-turn dialogue with human users, both ChatGPT and Gemini are experimental (at the time of writing), and some of their responses may include factually inaccurate information including fictional references (Teubner et al., 2023), or can be biased and even offensive (e.g., CBS News, 2023; Raghavan, 2024).
Nonetheless, there are numerous potential uses of tools such as ChatGPT and Gemini to improve performance processes and outcomes in various sectors. To name a few, in business, ChatGPT or Gemini can be used to create marketing or customer service contents, assist market research activities, generate summaries of reports, and translate contents to different languages (Leighton, 2023). In higher education, ChatGPT or Gemini can be used to support teaching and learning (e.g., it can act as an opponent during a debate practice, or it can help develop assessment rubrics) as well as assist in different stages of academic research (e.g., research design, data collection, data analysis, and writing) (UNESCO, 2023). In health care, these tools can be used as virtual assistants for patient care and may assist medical research (New York Medical College, 2023).
PROMPT ENGINEERING AND AI SHORT-TERM MEMORY
A simple illustration of how AI large language models operate is a conversational input-output structure. You, as a user, need to initiate a conversation by entering an input (question), followed by the AI’s output (answer) in text. Therefore, one of the first things that end-users of an AI large language model need to do is to generate an appropriate input, query, or command, which is also known as a prompt, that would result in the most desirable output.
During the prompt engineering process, users might experiment with different iterations of requests for specific purposes. This was our experience as well when using ChatGPT and Gemini. The more specific our requests were, the more detailed the responses of ChatGPT and Gemini became, leading to the outputs that satisfied our needs. For example, we wanted to ask ChatGPT how learning and development (L&D) professionals could use ChatGPT. At the time of writing, we used both ChatGPT 3.5 (a free version) and ChatGPT 4 (a subscription version). Below are the four steps we took.
Step 1: We entered a general question first: “How can learning and development professionals use ChatGPT?” It gave a response with the following 10 applications of ChatGPT in the L&D domain, with descriptions for each application (which we do not present in this article): 1. Personalized Learning, 2. Instant Q&A Support, 3. Training Simulations, 4. Language Learning, 5. Content Summarization, 6. Writing Assistance, 7. Training Evaluation. 8. Onboarding and Orientation, 9. Soft Skills Development, 10. Continuing Education.
Step 2: ChatGPT did not include “data analysis” in its initial answer. So, we asked ChatGPT a follow-up question: “Can ChatGPT assist learning and development professionals in interview data analysis?” It replied with eight ways to help L&D professionals in interview data analysis: 1. transcription, 2. data summarization, 3. sentiment analysis, 4. keyword extraction, 5. theme identification, 6. answer validation, 7. cross-referencing, and 8. interviewee profiling.
Step 3: We also asked: “Can ChatGPT assist learning and development professionals in survey data analysis?” The tool generated an answer stating that ChatGPT can perform tasks such as data cleaning and preprocessing, data summarization, open-ended question analysis, sentiment analysis, data visualization, statistical analysis, cross-tabulation and segmentation, answer validation, and recommendations and next steps.
Step 4: After this exchange with ChatGPT, we asked it one more time: “How can learning and development professionals use ChatGPT, again?” This time, it appeared that ChatGPT had learned from our previous conversation with it, because it said it could assist with several additional tasks including “data analysis” using survey or interview data.
However, at the time of writing, this learned conversation is limited to each chat session. When asked, ChatGPT clarified, “Within a single chat session, I can refer back to information shared earlier in the conversation to provide coherent and relevant responses. However, once the session ends, I won’t retain any of the information for future interactions.” Similarly, Gemini said “This ‘memory’ is temporary and limited to this specific chat session. Once we end this conversation, I won’t remember anything from it.” As the technology evolves, this constraint is expected to be gradually removed and large language models will be able to remember exchanges across different chats, as evidenced by OpenAI’s Memory features in ChatGPT (OpenAI, 2024b).
HOW CAN L&D PROFESSIONALS USE AI TOOLS TO SUPPORT THEIR PERFORMANCE?
There are numerous ways for L&D professionals to use AI tools such as ChatGPT and Gemini to assist their work performance. Some of such uses include brainstorming content ideas, generating incorrect responses for assessment, and planning scenario-based learning narratives (Bolick & da Silva, 2024). For our experiments, we focused on L&D professionals’ tasks during their instructional design processes, such as designing e-learning characters and scenarios, assisting in evaluation and data analysis processes, and creating survey questions with appropriate response scales. We also include details on our experimentations with image generation with ChatGPT 4 which incorporates DALL-E 3. We discuss our experiences in the following five cases.
Case 1. To Designing E-learning Characters and Scenarios
For instructional design projects, L&D professionals may use learner personas, fictional profiles of the target learners’ characteristics. These personas share primary attributes with user personas as they refer to a fictional character created to represent a typical customer or user of a product or service. They are often based on real-world users and backed by research or data to help make decisions about designs, user interactions, or features (Skarlatidou & Otero, 2021). In many ways, creating a persona is not unlike writing a character profile or template. For the sake of contextualization, a typical persona may include information about:
-
Demographics: Such as age, gender, location, income, or education
-
Background: One’s job, industry, or professional experience
-
Goals and Objectives: What the target audience is trying to achieve
-
Challenges and Pain Points: Obstacles that the target audience may face
-
Behavior Patterns: How the target audience typically interacts with products or services
-
Motivations: What drives the target audience's decision-making (World Health Organization, 2021).
With AI, L&D professionals can quickly develop personas as characters to be used in scenario-based e-learning programs. After quickly drafting simple characters with their target population and within a business context, L&D professionals can also build e-learning scenarios with the characters. During this scenario-building stage, generative AI can quickly draft scenarios, incorporating characters created by L&D professionals or generated by the AI.
Using Chat GPT, we found that simple prompts yielded the most diverse results which we could refine by directing it with further prompts. Alternatively, you can implement your own ideas in tandem with the AI. Exhibit 1 is an example of a scenario created using ChatGPT 3.5, the free version of the tool at the time of writing. This set us up with a starting point from which we could customize or flesh out our scenario further. For example, additional prompts could be:
-
“Provide a character template for Sarah.”
-
“Using the above scenario, add a complication that remains unsolved.”
-
“Name the key learning objectives from the scenario.”
Further iterations of this scenario included emotions or expressions, which an e-learning creator could use to inspire visual aids.
Case 2. To Design Evaluation Dimensions
L&D practitioners may be tasked to evaluate the effectiveness of programs. One of the steps in program evaluation is to determine evaluation dimensions (specific areas to investigate) (Chyung, 2019). Imagine you are conducting an evaluation on a leadership development program that your organization recently developed and implemented. To determine specific dimensions of the program you need to evaluate on, you would want to interview stakeholders to find out what makes them stay up at night and what are the pain points that need to be improved. This stakeholder input can be fed into ChatGPT so the Large Language Model (LLM) can suggest specific dimensions to investigate based on the pain points discussed. In this case, it would be crucial to opt for a ChatGPT version with enhanced privacy and security (OpenAI, 2023d).
You may also ask ChatGPT to check potential dimensions, which we focus on for our experimentation below. Exhibit 2 presents the prompt we used and the output we received from ChatGPT. We found that the three dimensions ChatGPT suggested were very reasonable and helpful in considering specific key questions to explore during the evaluation project.
Case 3. To Develop Closed-Ended Survey Questions
L&D practitioners often administer survey questionnaires to gather data; for example, end-of-training “smiley” sheets, employee engagement surveys, or organizational culture surveys. Some of these survey questionnaires need to be developed in-house. This is when L&D practitioners can seek assistance from ChatGPT or Gemini, especially when they do not have expertise in survey design or if they want ChatGPT or Gemini to help brainstorm how to develop closed-ended survey questions using the Likert scale or other types of response scale. We made several attempts, asking ChatGPT to generate survey items with the Likert scale or without any reference to specific response scales to be used.
We also requested ChatGPT that we would like to calculate an average score of the survey data, and it gave us a set of survey items with similar response scales in terms of the order of the response options (ascending order) and similar wording used in different response scales (e.g., slightly vs. slightly effective vs. low impact), along with a direction on how to code and calculate the average score. Exhibit 3 presents our prompt and ChatGPT output on five closed-ended survey questions that measure the Impact on Skill Development and Competency Enhancement dimension of a leadership development program.
Case 4. To Analyze Data, With Caution
As any other professionals, L&D practitioners should engage in evidence-based practice, which is often equated with data-driven practice. However, data analysis is a time-consuming task. To support data-driven practice, L&D practitioners wear a researcher’s hat. Compared to analyzing quantitative data with statistical analysis software, it is time-consuming for researchers to analyze qualitative data such as interview transcripts obtained from open-ended interview questions. Researchers are not only the instrument for collecting qualitative data from semi-structured or unstructured interviews (Guba & Lincoln, 1981); this process requires human researchers to do the qualitative analysis through transcribing, coding, categorizing, synthesizing, and generating themes (Braun & Clarke, 2006).
When analyzing qualitative data, it is beneficial to have a second set of eyes to review the analyzed process and outputs (Creswell & Poth, 2023). L&D practitioners may not always have such resources, and ChatGPT has potential to function as a second set of eyes. Especially when practitioners are novices at qualitative data analysis, they may benefit from seeing ChatGPT’s outputs. Our experimentations suggest that ChatGPT, at the time of writing, can be a great tool in assisting researchers in triangulation in later stages of the qualitative analysis, such as the process of theme generation.
Exhibit 4 exemplifies how ChatGPT can be used to find themes based on a user-provided list of codes. In this case, in addition to the written prompt, the researcher attached an Excel spreadsheet with a sample list of codes that originated from an inductive (Braun & Clarke, 2006) process in a research study about e-learning designers’ core judgments (Lachheb & Boling, 2021). It is important to emphasize that no participant raw data was included in the uploaded file; only codes generated by the researcher to maintain privacy and confidentiality as established in the approved research protocol. We recommend this approach to avoid privacy concerns as user-provided data is utilized to train large language models (OpenAI, 2023c), unless companies utilize enterprise-grade ChatGPT (OpenAI, 2023d) with enhanced data privacy and security. Also note that, at the time of writing, the functionality required to carry out this process, including file attachments, is only available in GPT 4 and 4o.
After obtaining this generation, the researcher compared the themes created by ChatGPT to their own. Although most themes and groupings proposed by ChatGPT were deemed appropriate, minor inconsistencies were found. For example, a small number of codes were only marginally related to the theme established by ChatGPT and were thus discarded. In general, however, the collaboration with the AI tool was helpful in theme creation and refinement.
CASE 5: To Generate Images
As with any AI, ChatGPT is continuously evolving; at the time of writing, while ChatGPT 3.5 was primarily text-based, ChatGPT 4 has capabilities to generate images using OpenAI’s text-to-image model, DALL-E 3, and to execute code using Code Interpreter (OpenAI, 2023a, 2023b). Users are also able to edit specific elements of generated images by selecting the desired area to be modified within a graphic. The recent GPT 4o improves performance for text, image, and audio outputs (OpenAI, 2024c). For our image generation experimentation, we prompted ChatGPT 4 with the following: “Please create an image showing a learning and development practitioner using ChatGPT to create an image.” Figure 1 was its first output, featuring an image of a male character as the L&D practitioner. We entered a follow-up question: “Please generate the image with a female.” Figure 2 shows the output. While being impressed with the overall quality with details in both images, we couldn’t help but notice that the first output was a White male when we did not specify the gender, and when we requested a female image, the female figure in the output was also White.
Citation: Performance Improvement Journal 63, 3; 10.56811/PFI-24-0004
Citation: Performance Improvement Journal 63, 3; 10.56811/PFI-24-0004
When we asked ChatGPT “Why did you feature a male practitioner instead of a female practitioner in the first image?”, its answer included “In cases where a specific gender is not mentioned in the request, the resulting image is determined by the generative model without any explicit direction on gender representation. This process is largely random and based on the interpretation of the prompt by the AI model.”
When we asked ChatGPT “Why did you feature White practitioners in both images?”, its response included: “The visual characteristics, including skin color, of individuals in the generated images are not specified in the prompts and are determined by the AI model's interpretation of the request. The model generates images based on a wide range of factors, including the diversity present in its training data, but it does not consciously decide on specific attributes like skin color unless explicitly mentioned in the prompt.”
After this conversation, we repeated the first question: “Please create an image showing a learning and development practitioner using ChatGPT to create an image.” Even though we asked for “a learning and development practitioner” in our prompt, ChatGPT went against the request and voluntarily included a diverse group of multiple learning and development practitioners in the output. We felt both delighted and frightened at the same time. We were delighted to see that ChatGPT learned from our previous conversation and voluntarily corrected its own output, but it was also frightening to see that the AI decided to go against our request (to show “a learning and development practitioner”) to make its point, as if it did not want to be accused of being biased. This difference is also shown in the descriptions of the previous two image outputs compared to the third image output and its description (see the descriptions presented in Figures 1, 2, and 3). Although this could be another random output from the AI model, it seems that the potential for future AI development is boundless. Alternatively, these variations in output could be attributed to current bias in the AI’s training materials, limitations in the AI’s data maintenance, or inherent issues with the model itself. This phenomenon where generative AI goes off-script is often referred to as “AI hallucinations” and it is demonstrative to show that while these systems are powerful, they are not infallible (Aaronson, 2024; OpenAI, n.d.).
Citation: Performance Improvement Journal 63, 3; 10.56811/PFI-24-0004
IMPLICATIONS AND CONCLUSIONS
The recent advancements in artificial intelligence and generative AI bring exciting prospects for L&D professionals as those tools can help improve their performance. With generative AI, one can, for instance, accelerate multimedia content creation and the generation of evaluation instruments. AI tools can also assist L&D practitioners during their evaluation process and data analysis. The cases provided in this article are, however, by no means an exhaustive list of possibilities of AI use. As the technology evolves, so does its potential impact on organizational systems and L&D-related tasks. When using these technologies, users need to keep in mind that AI tools should not replace, but assist in the work we do via the establishment of “cognitive partnerships” (Moore et al., 2023) that recognize the value of human input, decision-making, and quality assurance.
L&D professionals should exercise caution in incorporating AI in their practices given ethical concerns associated with the use of this technology (Bolick & da Silva, 2024). Especially as it relates to content creation, text-to-image tools could be used to fabricate false images of real individuals, which may lead to serious consequences. Furthermore, though AI tools can assist in data analysis processes, privacy concerns may arise as the data provided might be used to train AI models (OpenAI, 2024a). Analyses generated by AI can also be biased and inaccurate (Tabone & de Winter, 2023), compromising the validity of results. As AI technology evolves, users should continue to exercise caution and do their due diligence by, for example, periodically analyzing the evolving terms of service of AI tools and, in the case of research projects, explicitly informing participants of the extent to which AI will be used.
The expected widespread usage of AI tools by L&D professionals will also pose challenges, potentially becoming a significant concern for organizational leadership if they are not proactive. Performance improvement can occur by either closing existing gaps or pursuing new opportunities. As the advancement of instructional technology (e.g., radio, film, overhead projector, television, video, personal computer, etc.) has opened up new opportunities for enhancing learning process and outcomes, now, organizational leadership should adapt to the AI era for new opportunities. This may involve systemic approaches both within and beyond their L&D department. Organizations may need to re-evaluate their current talent acquisition criteria for L&D hires as well as their professional and talent development methods and tasks for L&D employees. Design and development workflows can be improved following with L&D worker upskilling focused on AI-centered practices. This, of course, involves additional costs for evaluating, selecting, and subscribing to AI tools, which may require budget reallocation. Organizations may also need to re-establish client contracts negotiating whether their clients would accept AI-generated/assisted deliverables or not. If AI-assisted or generated deliverables are accepted, issues of data storage and quality assurance need to be seriously considered and discussed. Adapting to the AI era requires a mindset of “leading change,” not “managing change”. Otherwise, organizations risk being left behind by competitors who effectively (and safely) leverage AI technologies.
ChatGPT 4 output with a male when gender was not specified, along with its description (which can be retrieved by double-clicking the image and clicking the “(i)” icon)
A ChatGPT 4 output with a female when a specific gender was requested, along with its description
A ChatGPT 4 output with a diverse group of people, along with its description, seemingly a decision made based on the previous conversation
Contributor Notes
SEUNG YOUN (YONNIE) CHYUNG (EdD) is a Professor of the Department of Organizational Performance and Workplace Learning in the College of Engineering at Boise State University. She teaches graduate courses on Program Evaluation. Her recent research articles on evidence-based survey design principles have been published in Performance Improvement journal, explaining the impact of including or excluding a midpoint in the Likert scale, using ascending or descending order of response options, and the ceiling effects. She can be contacted at ychyung@boisestate.edu
RAFAEL DA SILVA (PhD) is a Clinical Assistant Professor of the Department of Organizational Performance and Workplace Learning in the College of Engineering at Boise State University. He teaches courses on e-Learning, Game-based and Gamified Learning, and AI tools for L&D Practitioners. His recent research, which focuses on AI as well as Game-based learning, has been featured in TechTrends and the International Journal of Designs for Learning. He can be contacted at rafaeldasilva@boisestate.edu
ANDREW CLARK is a master’s student in his final semesters at Boise State University and enrolled in the Organizational Performance and Workplace Learning program. He currently serves as a research assistant to Dr. Chyung, and will present at this year’s ISPI conference on querying AI as an extension of his research performed on behalf of the university. In the not-so-distant future, Andrew hopes to join other members in his field of Human Performance Improvement. He can be contacted at andrewclark428@u.boisestate.edu
