Why is moving AI from prototype to production so difficult?

Transcript

Lawrence Ngo: In our experience, we’ve, we’ve felt that, you know, getting to the prototype is about five percent of the work, and then you’re just kind of squashing edge cases after edge case for the rest of that 95 percent.

Keith Raymond: The other things that’s happening here is this iteration, the generative iteration, the GPT iteration of AI was super unique from the standpoint of it. The cost of entry to play around with it was nothing, right? It was very cheap compared to other technologies of the past.

Keith Raymond: So everybody could do a pilot, everybody could do a proof of concept, all that kind of stuff. And it was relatively cheap. But like Lawrence said, once you go to scale, all of the other economics come into play.

Keith Raymond: And so it’s been very interesting to see because AI, at least at its initial stages, 2023, 2024, the GPT version, it kind of blew through traditional ways of assessing projects, right? Because it was in everybody’s hands, it was economically easy to use from a piloting perspective.

Keith Raymond: But now that organizations are starting to understand it more, and now the agentic play and token economics and everything else are impacting things, or coming into question, we’re seeing a full circle back to more of the traditional evaluation of how do you launch any kind of project with capital investment, right? And so that’s happening, that full circle is happening.

Keith Raymond: Still relatively cheap, again, to develop a pilot of proof of concept, as Lawrence stated, but it’s all the other foundational elements. When you talk about having your governance in play, your communication plan in play, your cloud, your data architecture, all that kind of stuff, and layering agentic, then you’re talking about orchestration between systems and all that kind of stuff, your agentic layers and stuff.

Keith Raymond: Yeah, it becomes a much more detailed, like any other applied technology, really, detailed assessment of value for the investment.

Lawrence Ngo: The issue that comes up is that as you prototype a solution, you often come across what is like the most happy path possible. Like there’s a small number of samples you’re able to get through that workflow.

Lawrence Ngo: Let’s say extracting documents or underwriting, as Keith mentioned earlier. You kind of see the happy path and see the AI succeed in a well-controlled few-shot environment. Meaning few-shot, meaning just a small number of samples.

Lawrence Ngo: Once you hit production, you start having a much larger distribution of data. Hundreds, thousands, tens of thousands of data samples, which then brings up, it feels like, as many edge cases as there are samples.

Lawrence Ngo: So those edge cases end up making or breaking the ultimate success of a production enterprise-grade product, though, because there’s very dense business logic in what the insurance industry does, and that business logic is not just readily available to ChatGPT or Claude right out the box.

Lawrence Ngo: It’s very powerful to get to that prototype initially and to get out the gate very quickly. But at the end of the day, being able to get something that, for example, is 80 percent accurate on a set of samples up to what you would want in production, which might be 98 percent or above in terms of accuracy.

Lawrence Ngo: It’s a lot of work and a lot of iteration and making sure that you can validate your samples at scale.