These days there is a lot of yapping about what AI can and can’t do; it CAN answer bar exam questions so well it’s nearly in the 90th percentile of test-takers, it CAN’T search in real time to find up-to-date answers beyond what it’s been trained on so it won’t be getting stuck in a late-night Wikipedia hole any time soon. Today we’ve learned there’s one more can: it CAN answer tax questions and be broadly correct 84.19% of the time.
The folks at tax app Keeper trained GPT-4 on 2023 tax updates and then set the public loose on it, inviting them to ask their burning tax questions. From there, actual human professionals fact-checked the answers. For example:
One of the reviewers was Isaiah McCoy, a CPA working in Miami. Going into it he tempered his expectations and thought the tool might hit 60/40 right/wrong or even 50/50 just because tax law is so nuanced. “It far exceeded my expectations,” he said. “Its success rate was more like 80/20 or 90/10. I think it did a great job overall, really blew me away.” As for the prospect of getting replaced by AI, he says he feels moderately safe. “I definitely feel like it’s a threat or an opportunity depending on how you look at it,” he said.
By evaluating 215 answers, Keeper learned that the AI does a good job of:
- Knowing which tax forms apply to which situation
- Understanding real estate tax implications
- Naming deductible business expenses for specific industries
And it sucks at:
- Getting tax form details right, including what certain lines and boxes are intended for
- Answering questions about tax year applicability
- Addressing state and local tax nuances
Here’s one it bombed at:
It’s wild how it matched the casual tone of the original question in its response with a “Hey there!”
Feel free to give it a spin yourself, just remember not to input any PII. And don’t threaten it, you don’t want to be on its shit list when the inevitable takeover happens.
Latest Accounting Jobs–Apply Now:
Related
Have something to add to this story? Give us a shout by email, Twitter, or text/call the tipline at 202-505-8885. As always, all tips are anonymous.
