When productizing AI, there are numerous challenges you can encounter, such as how to apply your AI model to a process or people, stabilizing data and models, how to keep your model accurate in changing environments and over time, scaling, and how to grow or augment the capabilities of your AI model.
Embedding AI
Running a successful machine learning Proof of Concept (PoC) with a new algorithm is only 10% of the effort required to productize it and get actual value from it. The remaining 90% can be divided into things you need to do to make a usable product and things you need to do to make a useful product.
To make a usable product, you need to zoom in on the technical implementation of making the product available to your users. To make it useful, you should look at embedding the product into a process for the users. First, however, what exactly is the difference between a PoC and a usable product?
First of all, PoCs are not meant for production. Products need to work all the time, any time, and under shifting circumstances. During your PoC, you find the data you are looking for, make a copy, and start to clean it up and analyze it. In production, your data source has to be connected to a data platform in real-time, safely, and securely; the data stream has to be manipulated automatically and compared to/combined with other data sources.
During your PoC, you either have the luxury of being able to talk to your future users and work with them to design a solution, or you have no users at all, and you are designing a technical solution. For a product, you have users that need to understand that solution, and people responsible for keeping the technical solution running. Thus, a product requires training, FAQs, and/or support lines for it to be usable. Furthermore, you just create a new version for your one use case in a PoC. Products require updates, and when you have rolled out your product for multiple customers, you need a way to test and deploy your code for production (CI/CD pipelines).
“At Itility, we’ve developed our Itility Data Factory and AI Factory that cover the building blocks and underlying platform for any of our projects. This means we have the usable angle covered from the start, so that we can focus on the useful angle (which is more customer and use case dependent),” the company stated.
Pest detection app – from PoC to useable product
“The Proof of Concept phase of our Pest Detection App consisted of a model that can perform the narrow task of classifying and counting flies on a glue trap based on images taken by greenhouse team members. In case they missed a picture or if something went wrong, they could go back and take another, or directly fix it in the dashboard. Quite some manual checks were needed.
“Our PoC-world was simple, based on one single device, one single user, and one single customer. However, to make it into a useable product, we needed to scale and support multiple customers. Then, the question of how to keep data separated and secure arises. Moreover, each individual customer/machine requires a setup and default configuration. So, how to configure/set up 20 new customers? How do you know when to build an admin interface and automate onboarding? At 2 customers, 20, or 200?”
Of course, you might have questions, such as ‘how does counting flies help my customer? How to create value from this information? How to recommend decisions and take action? How does this AI application fit in the business process?’. Step one is to change your frame of reference from a technical/data perspective to the end-user perspective. This means continuing the conversation with your customer and seeing how the proven PoC fits into daily processes.
“You also have to closely follow the process for a longer period of time, you need to join operational and tactical meetings to really understand what actions are taken every day based on which information, how much time is spent on doing what, and the reasoning behind certain actions. Without understanding how the information from your model is used to create business value, you will not get to a useful product.
“In our case, we discovered what information was used to make decisions. For example, we discovered that for some pests it was more important to follow the weekly trend (for which you don’t need super high accuracies) whilst others require action at the first sign of a pest (which means it’s better to have a couple of false positives than to have even one false negative).
“Additionally, we discovered that our customer had previously had a ‘bad’ experience with a similar tool claiming to have accuracies it could not deliver in practice. Why would they trust ours? We took this trust problem head-on and made accuracy and transparency a key feature of the product. We used this information to make our product useful by adapting the application to the end user’s working methods, and by increasing transparency in the interaction, giving the user more control over the application,” the company continues.
What is the biggest challenge?
“In our fly-counting scenario, we can talk about our accuracy score all we want. However, to be useful, the user (a greenhouse specialist) needs more than percentages. What is needed is to experience it, and to learn to trust it. The worst thing that can happen is when your users compare your results with their own manual results and there is a (large) discrepancy. Your reputation is ruined and there is no room to regain trust. We counteracted this by adding software to the product that encourages the user to look for those discrepancies and correct them.
“Our approach is thus to make the user part of the AI solution instead of presenting it as a system that is going to replace the specialist. We turn the specialist into an operator. AI is augmenting their abilities and the specialists remain in control by continuously teaching and guiding the AI to learn more and make corrections when the environment or other variables drift. As an operator, the specialist is an integral part of the solution – teaching and training the AI with specific actions.”
Click here to see a video with more details on the operator-centric approach.
