You misunderstand. We are one of the top shops for ML based fraud detection. But when someone is accused of fraud they get to appeal it. Then a human is in the loop and the model scores and all inputs are investigated and compared against many other sets of data and policy etc. LLMs facilitate this effort by making the investigators job considerably easier in navigating the enormous amount of complex information. The LLMs role is not to make decisions but to assist in navigating and understanding a lot of high cognitive load information. We have been doing a lot for many years using traditional techniques to make this process easier. But LLMs unlocked a level of dynamism and responsive UX that has blown the lid off our ability to adjudicate appeals. This has significant economic gain for us as offboarding legit customers for fraud causes a lot of losses over a long term.
The LLM isn’t used for reasoning at all. The human does all the reasoning. The LLMs task is summarization and semantic analysis of relevance which LLMs are fantastic about especially in a well managed and fine tuned environment with guard rails and context scoping. It’s a true copilot scenario and the LLM takes direction from the human and answers questions only. All decisions are investigator driven. This is the right relationship. The LLM coupled with IR tools does information retrieval and summarization and the human makes decisions and reasons.