This is the final post in a 3–part series on building an Artificial Intelligence and Machine Learning (AI/ML) capability for the first time. In case you missed our post last week, the second article covered how to set a targeted objective through development and communicating results. We will now focus on deploying and transitioning Artificial Intelligence and Machine Learning capability to operations, governance of the capability, and establishing monitoring and maintenance routines to ensure performance holds with passing time.  

Transition to Operations and managing the Artificial Intelligence and Machine Learning capability. 

The stakeholders have decided to move the AI/ML capability onto operational systems, and now it’s time to move forward. At this point ensure the data utilized to build the capability and the code or model itself are validated. Establish a specific plan with the Operations team to either transfer the AI/ML capability into the system or connect to the system using an API call. Depending on the final data needs, data pipelines may need to be established. 

Use a pre-packaged solution for deployment, monitoring and governance or utilize a custom-built solution out of open-source tools and frameworks (Pachyderm, etc.). There are many pre-packaged solutions that are either optimized for a specific application or that can handle many types of models and be customized. If you have internal existing tools and frameworks already for other models, you may want to expand upon what is currently used to achieve easier adoption of your AI/ML capability. 

Perform Change Management.

With any new business process or capability, change management activities preceding deployment result in greater, more widespread adoption and less user frustration. The outcome for the user is they better understand how to incorporate the new AI/ML capability into their workflow and what insights they can achieve. A combination of training documentation, in-person training, roadshows and even training material inside the operational tool (if applicable) are useful methods to do knowledge transfer and training. Any changes that are made to the capability may impact a user’s workflow, so perform an assessment with each capability update.  

A key to a successful rollout is to have users involved as the capability is transferred to Operations, before it is deployed. They can then provide input on how a user will interact and respond to the new insights generated or give input on how to optimize or modify an impacted business process. 

Maintaining and monitoring the Artificial Intelligence and Machine Learning capability. 

Like any new technology, without the appropriate care and feeding the new capability’s performance will decay over time and eventually wander into dangerous territory. To retain good AI/ML health and avoid using bad insights, there are two aspects to address. The first is ensuring the data integrity is stable: it continues to be updated at an appropriate interval and the data quality suffices. The second component is ensuring the model continues to perform as needed for the organization. These two aspects are key to your success and may be the “make or break” case for utilizing AI/ML in your firm.   

To make monitoring easier, add automated methods for data quality checks and the AI/ML capability performance in Operations. Dashboards with visualizations are good tools to monitor performance and data quality at-a-glance. Use notifications for key individuals to take corrective action quickly. Have a communications plan in place if/when the capability is not available or goes down.  

It is a best practice to hold a major review of the AI/ML capability on regular intervals beyond monitoring activities. The state of business may change enough that the capability does not accurately represent the current state. There may also be new data available or external data sources that can greatly enhanced performance or applicability to the business. 

Identify how to best support the deployed Artificial Intelligence and Machine Learning capability. 

As this is potentially the first foray into AI/ML for your organization, it will likely receive a lot of attention. To ensure a smooth adoption, provide timely support for special training requests or any issues that surface. Identify subject matter experts that are responsible for addressing either data issues or functionality issues. This may be a combination of a business product owner, a data owner and an infrastructure owner to cover any type of problem. It is also wise to earmark maintenance resources for a period to ensure proper support is available if needed. 

Updating the Artificial Intelligence and Machine Learning capability.

There is a repeatable process that occurs with every iteration, starting with training the model with updated data and/or additional data fields. Once proof-of-concept training and testing is complete, the capability is registered as a new configuration, then validated by a team of business and technical subject matter experts. The validated capability then moves to Operations, and monitoring functions start. Once an issue with performance is identified or new features / data are desired, another iteration occurs. 

Example scenarios where your AI/ML capability may need to be retrained are: there was a shift in the market place or your business model/performance (COVID-19 is a great example), third party or internal data is newly available or you a need to have the current business environment captured. 

Connect with Strive! 

Here at Strive Consulting, our subject matter experts’ team up with you to understand your core business needs, while taking a deeper dive into your organization’s growth strategy. Whether you’re interested in AI/ML implementation or an overall data and analytics assessment, Strive Consulting is dedicated to being your partner, committed to success.  

As part of our 3-part series, our next post provides guidance on how to build an Artificial Intelligence & Machine Learning (AI/ML) capability for the first time. In case you missed our post last week, the first article covered three items required before beginning development on an AI/ML initiative. Moving forward, we’ll now focus on the steps to help prepare for development, recommendations on a development and model selection approach, and how to communicate results to stakeholders.

Set a specific objective.

Baseline stakeholder expectations by setting a very clear objective for the project. This will prevent scope creep during development and be more likely to produce results that are meaningful. One example may be to identify typical transactions, or fraudulent, made in 2019 by a group of customers. Can the new capability identify fraud where it was formerly not found? Are there trends you never discovered before? With these new insights, how much loss is avoided by the organization?

When setting the objective, ensure you clarify scope enough that results can be shown to prove the AI/ML capability works. Establish key performance parameters to measure performance, and at the same time, allow a common understanding on what success looks like.

Identify where the Artificial Intelligence & Machine Learning capability will live.

Working with Operations from the project onset prepares internal partners for possible changes to current systems. Their early inclusion enables a smoother handoff and the possibility of a more natural integration into existing systems. The following are a few approaches to consider; keep the AI/ML capability in a separate sandbox until performance is proved out over time, run the capability in parallel with current functionality, or replace/supplement the current functions with the new AI/ML capability. Depending on the degree of integration required, you may want to go as far as enlisting a representative from Operations as a stakeholder.

Prepare for development.

Configuration management for both data and software/models is crucial. If multiple trials are run with various models and sets of data, poorly managing that complexity can lead to costly and time-consuming mistakes. The flavor of development environment is not as important and varies widely across the breadth of data scientists and Artificial Intelligence & Machine Learning practitioners. As your team begins to aggregate data, know that the data preparation may take a significant amount of time depending on the data quality. To give a perspective on the time required for data prep, Anaconda surveyed data scientists and found that 45% of their time is still spent on data loading and data cleansing. ¹

Establish the development approach.

When developing a POC capability, the best approach is iterative and adaptable, and will likely take many trials for success. Establish checkpoints for checking on POC performance and aim to produce a minimum viable product from the first iteration on. An agile methodology lends itself well to this type of effort, with the degree of process formality dependent on the appetite of the organization. Look at each iteration as an experiment and use each hypothesis for each iteration as an avenue to incorporate stakeholder expectations.

Select the software package or tool.

The amount of software development required when building a POC varies greatly depending on the package or tool selected. Free packages generally have a larger lift either setting up your environment, or for preparing the data and visualizing it. The most popular programming languages and packages to use are Python [SciPy, Scikit-learn, TensorFlow, etc.], R [randomforest, CARET, KernLab etc.] and JAVA [WEKA, Java-ML, etc.] but require time and expertise to set up and utilize. Tools like Alteryx are easy to use right out of the box and are great for someone not wanting to code but needing more power than Excel. The expertise of resources, budget available and the timeline desired for results will drive which packages and tools work best.

Develop the Artificial Intelligence& Machine Learning capability.  

The specific model to use is driven by the data that is available, objectives and the expertise level of the practitioner. As a best practice, most practitioners start with a simpler model to baseline performance. Complexity is increased if there is insufficient performance on the business objectives metrics. If performance is not as expected after a few iterations, do not give up hope as there may be promise in expanding the dataset to adjacent features or in utilizing another AI/ML method.

Communicate results.

Circle back on KPIs identified during the project onset showing any progress made. Compare the business performance from a current state perspective to the estimated performance provided by the AI/ML capability. Share how the new capability will work in the operational system, if adopted. If removing or reducing manual labor, highlight the extent of time savings for targeted users. If there appears to be lift, provide the estimated ROI once operational.

Next up in our Artificial Intelligence & Machine Learning Blog Series

Part 3 and final portion, where we’ll guide you through the steps of transitioning the AI/ML capability to operations, establishing maintenance routines, and ensuring performance continues to meet requirements for your business.

Connect with Strive!

Here at Strive Consulting, our subject matter experts’ team up with you to understand your core business needs, while taking a deeper dive into your organization’s growth strategy. Whether you’re interested in AI/ML implementation or an overall data and analytics assessment, Strive Consulting is dedicated to being your partner, committed to success.

Artificial Intelligence and Machine Learning (AI/ML) technology continue to be increasingly accessible with a lower barrier of entry to newcomers. In recent posts, Gartner identifies that one in ten enterprises now use ten or more Artificial Intelligence applications, with the top use cases being chatbots, process optimization, and fraud analysis. ¹ Of the companies reporting Machine Learning usage, Algorithmia identified the following area benefits: reducing company costs, generating customer insights and intelligence, and improving customer experiences. ²

Have you been asked to lead an Artificial Intelligence and Machine Learning initiative to solve a business problem?

If so, you’re at the beginning of an exciting journey in leveraging transformational technology. To give you an idea of potential return, let’s look at Highmark Inc.’s Financial Investigations and Provider Review Department. Highmark successfully applied AI to save over $260 million related to fraud, waste, and abuse in the year 2019 and continues to use AI to enhance fraud identification and prevention to catch potentially fraudulent activities today. ³

Because many have walked the path before you, successful approaches to incorporating AI and ML capabilities are readily available. This blog series will guide you through that process, while highlighting best practices and potential pitfalls. Our first article starts by walking through three key components that must be present for a successful AI/ML initiative.

The following items are considered as entrance criteria:

A specific business need is identified.

• For the first AI/ML project, it is best to choose a well-defined bite-sized proof-of-concept (POC) along with a key metric to evaluate the level of success. A succinct problem statement and hypothesis on outcome(s) will align the team and set stakeholder expectations with each iteration of the POC. Success on a POC enables further endeavors of a more aggressive nature.
• If a specific objective is not identified, the project has the potential to fail based on the growing or changing scope, or even worse, misaligned expectations.

Data exists to support the business objective.

• A full data warehouse does not need to be present, but enough data of sufficient quality must be available to build the knowledge base needed for training the AI/ML capability. Be sure to consider any support data that may be needed to fully address the objective. If data gaps are identified, there may still be a chance to produce a meaningful result, but with a reduced scope of application. To reduce risk, the best first attempt for use cases have the required full dataset available within a single source of truth, or the data is available via existing pipelines.
• Data volume is a primary requirement, closely followed by data quality. In a study that Researchscape International and Trifacta performed, one of the greatest challenges the respondents faced for AI implementation was data accuracy, with only 26% of respondents reporting that their data is completely accurate. Additionally, poor data quality negatively impacts AI/ML initiatives, with more than one-third reporting that projects took longer (38%), were more expensive (36%), or did not achieve anticipated results (33%). ⁴
• If the current state of your data is not known, you may want to perform a data assessment before pursuing the AI/ML initiative further.

A culture of making data-driven decision making is present.

• An AI/ML capability may have the potential to disrupt your business, but without an appetite to steer decisions based on the algorithm outcomes, the initiative will fall flat. If there is skepticism, show the estimated ROI and a plan to kick the tires with A/B testing. An adoption roadmap with milestones and corresponding results will ensure confidence that true diligence is being taken and result in reduced uncertainty.

If these criteria are met, the odds of a successful AI/ML implementation are tenfold.

Tune in next week!

Part two of this series will guide you through the steps of developing the AI/ML capability, from setting an objecting all the way through to managing and understanding your results. Make sure to check out Strive’s Insights page for additional postings and information.

Connect with Strive.

Here at Strive Consulting, our subject matter experts’ team up with you to understand your core business needs, while taking a deeper dive into your organization’s growth plan. Whether you’re interested in AI/ML implementation or a overall data assessment, Strive Consulting is dedicated to being your partner and committed to success.