LAYING THE GROUNDWORK: AI IN MATURITY MODEL DEVELOPMENT, SERIES PART 1

Improved Data Collection and Analysis

As previously stated, MM development is labor intensive and manually driven. Today, AI has transformed how data is collected, analyzed, and processed. AI is the “digital replication of three human cognitive skills: learning, reasoning, and self-correction” (Rathore et al., 2021, p. 32038). Digital learning consists of a set of rules, executed as a computer algorithm, transforming historical data into operational data. Digital reasoning centers on selecting the appropriate rules to achieve a specific goal. Digital self-correction is the self-referencing process of incorporating the output of learning and reasoning. AI utilizes this process to create an intelligent system capable of performing tasks that typically require human intelligence. Tasks include time-consuming periodic assessments. AI automates data collection from various sources, such as sensors, databases, documents, and user interactions, enabling continuous monitoring. This facilitates the development of MMs to evolve in real time.

AI streamlines data collection by gathering, processing, and analyzing large amounts of data from diverse sources, allowing organizations to build accurate, real-time models. AI tools manage ill-structured data, identify patterns, and perform predictive analysis, enabling organizations to pinpoint areas for improvement. ML algorithms predict future states of maturity based on historical data to help organizations forecast growth patterns and identify potential barriers. AI techniques, such as natural language processing (NLP) and data mining, applied to MM creation facilitate the identification of relevant criteria from documented data, reducing manual effort in model design. For example, Pattathil et al. (2024) conducted a study in which AI was used to increase the power of data analysis in ophthalmology by leveraging electronic health records and electronic medical records to predict eye diseases and improve clinical decision making. AI algorithms were applied to analyze large volumes of complex patient data to generate disease prediction. ML was employed to predict risks, assess complications, and enhance diagnostic accuracy for various eye conditions, such as glaucoma and age-related macular degeneration.

ML is utilized in MM development to predict risks, assess organizational bottlenecks, and improve decision-making accuracy. NLP tools broaden data sources for assessing organizational maturity. The analysis of data, such as emails, instant messages, and reports, brings insights that were previously inaccessible. This expands the scope of input and reduces the need for intensive human intervention, allowing MMs to reflect a more relevant understanding of organizational readiness.

Adaptive and Real-Time Updates

Traditional MMs are static in nature. ML facilitates MM development to become dynamic by continuously learning from new data inputs. This enables real-time updates, making sure that the model evolves with the changing environment and remains relevant to organizational needs. AI automatically adjusts the criteria and dimensions of the MM based on the data it processes, allowing dynamic models to manage complex and evolving scenarios. Gilbert et al. (2021) conducted a study that addressed adaptive and real-time updates in ML-based medical devices, highlighting the significance of allowing ML models to continually learn from new data in real time. The primary focus was on how regulatory frameworks, which were developed for static medical devices, are evolving to accommodate the dynamic nature of ML algorithms. The study emphasized that real-time ML systems improve by retraining models based on new patient data. These updates, although beneficial for performance improvement, raise challenges for stakeholders in maintaining the clinical safety and effectiveness of these models. The paper discusses frameworks such as the FDA’s proposed algorithm change protocols to manage these adaptive updates by predefining acceptable changes and protecting real-world performance monitoring. In summary, adaptive and real-time updates are considered necessary for improving medical devices but require regulatory frameworks that can handle frequent and automatic updates without compromising patient safety.

Predictive Analytics and Scenario Planning

Predictive analytics is a technique used for detecting events or risk factors likely to occur in the future and analyzing response strategies. A variety of statistical techniques such as ML and data mining are used. In predictive analytics, numerical data that represents past operations are generally used to capture significant relationships among various algorithms, statistical models, and patterns not contained in the collected data. AI significantly enhances the predictive capabilities of MMs. Through predictive analytics, AI forecasts how an organization progresses across maturity stages based on historical data, current trends, and emerging patterns. Stakeholders have valuable foresight for strategic planning and resource allocation when AI is used to explore what-if scenarios, assessing potential outcomes, and risks of different strategies.

AI-enabled models are effective in scenario planning for complex systems, in which human analysis might overlook relationships or trends. For example, Hong et al. (2015) conducted a study of a marketing scenario–planning system designed to support decision making for medium-sized business owners using prescriptive analytics on the organization’s data. “The system leverages the powerful ecosystem, Hadoop/HBase-based data infrastructure to store, process and manage large scale structured and unstructured data, such as sales, logistical information, and social media feeds” (Hong et al., 2015, p. 592). The system architecture included data collection, prediction, and prescriptive analytics, which, together, generate marketing scenarios tailored to specific business environments. The study emphasizes the importance of integrating various data types to provide operationalized business strategies and suggests further research to evaluate the system’s prediction accuracy. Scenario planning and MM development are complementary strategic tools that, when combined, provide organizations with valuable information to prepare for future states while understanding their current capabilities and potential pathways for growth.

Customization and Personalization

Traditional MMs are similar in structure and have a universal approach. AI can develop tailored models based on the unique characteristics of a particular context. This customization results in increased accuracy and context-sensitive assessments. AI automates the customization process by analyzing an organization’s specific needs, environment, constraints, and strategic goals and then generating a version of the MM that fits those parameters. An example of customization and personalization is demonstrated in the Marín Díaz et al. (2023) research on AI use for customizing MMs. They used AI to personalize and optimize connections within smart city environments, aiming for tourism digital maturity. They integrated AI with fuzzy logic and the analytic hierarchy process (AHP) to develop a model that assesses digital maturity and generated customized recommendations based on tourists’ activities and preferences. AHP enables organizations to make complex decisions by breaking them down into smaller, more manageable chunks. AHP is useful when decisions involve many criteria to evaluate objectively or to directly compare. Fuzzy logic deals with imprecision and uncertainty by allowing for degrees of truth rather than the traditional binary options found in classic logic. The researchers conducted a digital maturity evaluation that used AI combined with the AHP and fuzzy logic, assessing the digital maturity levels of tourists. This approach prioritized and personalized services and made recommendations according to individual tourist maturity profiles. The researchers used AI decision-making processes by managing mixed information using fuzzy logic. This helped unify data presented in different formats into a managed system, improving precision and making personalized recommendations more reliable. Finally, personalization was achieved by designing a recommendation system that used AI to group tourists based on their digital maturity and tailor interactions appropriately.

The Díaz et al. (2023) findings relate to the broader development of MMs tailoring assessments using AI to evaluate how maturity models can be customized to reflect the unique characteristics and behaviors of different user groups. Assessing individual maturity profiles, organizations develop more relevant and precise MMs that align with the specific needs of their audiences. Customization allows for an in-depth understanding of user capabilities and challenges, leading to more effective interventions. The integration of fuzzy logic and AHP into MM development enables a subtle approach for prioritizing services and recommendations. Similarly, MMs benefit from incorporating decision-making frameworks that help organizations identify and prioritize development areas based on user input and organizational context. This ensures that recommendations are data-driven and aligned with the specific goals of the organization. The enhancement of decision making through AI and fuzzy logic highlights the importance of integrating multiple data formats into an integrated system. MM developers adopt similar approaches to aggregate qualitative and quantitative data from different sources, providing a comprehensive view of an organization’s maturity level. This multifaceted assessment leads to more informed decision making regarding development strategies and resource allocation. Categorizing users and tailoring interactions to their specific needs and preferences create more engaging and relevant experiences. Personalization fosters a sense of ownership and investment in the maturity process. Contextual relevance ensures that the assessments and recommendations provided are actionable and effective in driving performance improvement.

In summary, the concepts of customization and personalization, as demonstrated by the Díaz et al. (2023) research, improves MM development by tailoring assessments and recommendations to individual user needs and contexts, organizations can improve decision making and drive continuous improvement in MM development.

Automation of Assessment Processes

AI systems automate data-driven maturity assessments, which generate insights and recommendations without manual intervention, therefore saving time and increasing efficiency. Researchers Alenjareghi et al. (2024) highlighted key findings on automation of assessment processes by enhancing safety and efficiency through AI integration in risk assessment (RA). AI plays an important role in processing large amounts of data, identifying risks and hazards, and improving decision-making accuracy. The automation of RA provides dynamic and real-time assessments by reducing human error. Alenjareghi et al. (2024) found that using AI in automating risk assessments significantly enhances safety and efficiency in complex environments. This is especially relevant in contexts in which safety and compliance are critical as automated assessments can readily identify risks and facilitate interventions.

The automation of assessments has implications for the development and implementation of MMs, specifically by enhancing efficiency, accuracy, and responsiveness. AI facilitates ongoing self-assessments, enabling organizations to frequently evaluate their maturity levels without the need for external consultants. Continuous feedback helps organizations to adapt quickly to changes in their environment. This responsiveness is significant for organizations operating in fast-paced environments in which conditions change rapidly. By continuously monitoring performance and maturity indicators, organizations adapt their strategies proactively rather than reactively, which enhances their overall agility and promotes a culture of continuous improvement. Blending AI-driven assessments with traditional evaluation methods creates a balanced approach to MM development. AI enhances efficiency and accuracy; however, human judgment and expertise remain necessary to contextualize findings and make nuanced decisions.

Identifying and Reducing Bias

AI is objective due to the lack of influence by subjective opinions or organizational politics. AI algorithms process data from a variety of sources to generate a more holistic view of an organization’s maturity. Mitigating the risk of biased results based on limited or subjective input increases model credibility. Harfouche, et al. (2023) suggested there is one caveat is that the data must be free from bias when used during model development. AI helps to ensure that the decisions made are more equitable by identifying hidden patterns or biases in the data that may have been overlooked through human intervention. Vorisek et al. (2023) explored biases in health care AI systems and how to reduce them. They developed a survey for AI developers about their perspective on AI development, fairness, and algorithms to train AI. Initial findings revealed that the causes of bias are due to a lack of sociodemographic information and algorithm design flaws. For example, an algorithm trained on male data may not apply to female patients. A second finding is the AI developers’ awareness of biases because there is a significant gap in knowledge about the specific preventive measures to eliminate those biases. Only about half of the survey respondents knew of explainable AI methods, and fewer than half were aware of the software tools to evaluate fairness. In this study, the key bias prevention methods included the need for better education, clearer guidelines, and more comprehensive data.

There is a need for MM developers to evaluate fairness in their frameworks. A perception of bias reduces the adoption and effectiveness of the model. One technique to ensure fairness is engagement with a diverse group of stakeholders to ensure that the MM reflects a range of perspectives and is equitable. This parallels the need for MM developers to be educated about biases inherent in their frameworks and the methodologies available to address them. Another technique is to establish clear guidelines that help organizations recognize how to evaluate their current practices against maturity levels and identify areas for improvement. Integrating these techniques into the development of MMs enables organizations to mitigate biases, enhance fairness, and ensure that their maturity frameworks are more effective in driving performance improvement across diverse contexts.

Data Privacy and Ethical Concerns

AI relies on large amounts of data to function effectively, raising concerns about data privacy and ethical use. Organizations ensure that the data used to feed AI-driven MMs complies with privacy rules. The use of AI creates several ethical issues, many of which revolve around ethics, privacy, accountability, and transparency. As previously stated, two primary ethical concerns are bias and discrimination. Biased data fed to AI systems can help reinforce integrated inequalities. Safeguarding against bias is to establish clear objectives that guide the model’s criteria and milestones. Examples of transparent guidelines for bias mitigation are ensuring diversity in data sources and transparency in algorithm design. Another safeguard is to ensure privacy and avoid bias by reviewing how data is collected, labeled, stored, and processed. The best practice for organizations is to integrate tools that evaluate AI fairness such as bias detection software into their procedures.

Data privacy concerns are significant when analyzing large amounts of personal data to make predictions or decisions. Shuaib (2024) identified several concerns about critical data protection and data breaches around the integration of AI in health care. The researcher emphasized the need for robust encryption, strict access control, and cybersecurity safeguards to protect data. Shuaib (2024) stressed the need for visible, accountable, and equitable governance frameworks that address these ethical concerns: “AI systems need to operate under principles of transparency, accountability, equity, and respect for patient autonomy” (p. 1768).

Relating Shuaib’s (2024) findings to MM development, (a) organizations implement and enforce policies around data privacy, ensuring compliance with regulations, and foster trust by transparently managing data; (b) evaluate the degree to which AI systems mitigate biases using diverse data representation and transparent algorithms; and (c) create governance frameworks that are adaptive and context-specific, which includes ensuring that accountability processes are in place to handle AI errors.

Autonomy and Decision Making

AI systems make or influence decisions that affect people’s lives. Who is responsible when AI makes an error or causes harm? A lack of transparency in AI decision-making processes makes it challenging to understand how or why a particular outcome was achieved. Human oversight of AI systems helps to mitigate errors and ensures ethical decision making. For example, in banking industry customer service automation, human representatives intervene in complex cases in which AI misunderstands a client’s needs. This approach, in which humans review and validate AI suggestions, essentially partners with AI support rather than AI dominating the decision-making processes.

Researchers Makri et al. (2022) investigated the aviation industry and the connection of AI and human relationships. They suggest that there is a nuance between automation and autonomy. Automation is preset or fixed instructions, and autonomy permits systems to make contextual modifications. The authors stressed the necessity of keeping the human in the loop (HITL) in decision making, and autonomy highlights balancing human and AI input. The study revealed several implications for the development of MMs. AI integration in MMs is the ability for contextual modifications, constructing a pathway from basic automation to advanced autonomy with HITL checkpoints. The study emphasized precise reliance on structured roles and accountability and safety protocols in which HITL autonomy is the core component in the management of complex systems. The reliance on structured roles informs MM development by creating clear accountability structures, especially in high-stakes decisions in which human oversight is fundamental to mitigate failure. The article highlighted a need for socio-technical solutions, supporting humans as assets rather than problems. This awareness is valued, integrated, and optimized together with AI to create flexible structured systems that reduce errors through a balance of human oversight and adaptive AI. Makri et al. (2022) observed accident reporting for continuous improvement by identifying critical practices, such as role-specific standard operating procedures, that are central to understanding human–machine interaction failures. MM developers benefit from these practices by embedding reporting and continuous learning as stages of progression with mechanisms to review and adjust human–machine interactions in real-time.

Why Humans Matter: Keeping the HITL

The partnership of humans and AI provides balance and synergy to MM development. Human insight is vital to ensure MMs are responsive, practical, and ethically grounded. Humans bring context-specific and tacit knowledge that machines misunderstand. For example, organizational cultures, regional regulations, and industry-specific dynamics require human judgment to understand and incorporate effectively into MMs. The Metcalf et al. (2019) research on the importance of keeping HITL highlights the unspoken view about tacit knowledge and the difficulty of codifying tacit knowledge to others. The authors explored the application of artificial swarm intelligence (ASI) in group decision making. ASI collects group human intelligence in real time to address complex decisions beyond the limits of traditional AI models’ dependence on historical data alone. ASI integrates both explicit and tacit knowledge from varied human groups, modeling decision-making processes observed in biological swarms, such as wolf packs or schools of fish. The authors identified some benefits of ASI, which includes faster decision confluence, minimized impact of individual biases, and growth with varying group sizes. ASI technology shows encouraging results across business contexts, such as forecasting and prioritizing strategic initiatives, often outperforming traditional methods.

Similarly, Drori & Te’eni, (2024) utilized the voice of human experts who understand unique contexts, developers can create models that account for the nuances of each organization’s operating environment. These make sure the models are technically correct and relevant. AI and automated decision-making systems, when left unchecked, unintentionally introduce or reinforce biases. Examples include hiring practices, performance assessments, or resource allocations in which biased outputs foster unfair treatment. Human oversight is important in auditing algorithms, validating fairness, and ensuring that ethical guidelines are followed. HITL enables teams to regularly review model outputs, address any potential biases in real time, and adjust to align with organizational values and ethical standards. Human experts play an important role in translating model outputs into actionable outcomes to explain the “why” behind decisions. HITL is essential for making these models interpretable and ensuring that users at all levels understand and trust the model’s processes and recommendations.

In summary, HITL is required in MM development to balance the power of automation with human insight, creativity, and ethical judgment. Whereas AI and automation accelerate analysis and forecasting, humans are essential in ensuring these models are trustworthy, adaptable, and aligned with human-centered values.