The AI Knowledge Gap

I think I am more of an AI skeptic than most, though I’d rather see myself as an AI pragmatist. I still use AI and re-evaluate its uses from time to time to see how things have changed. It’s often very helpful and I use it probably every day for small things, but I wouldn’t consider myself a big AI enthusiast. I try to look at it pragmatically, not as a solution to all our modern problems, but as another powerful tool. Recently, though, I realized something about LLMs. They can help us be a jack of all trades but won’t help us master anything. Becoming a master will still require hard work and meaningful experience.

I’ve realized how differently I react to AI depending on what I’m asking about. When I ask it about things I know, I can see it’s limitations. I can see all the gaps and oversimplifications. It sounds confident, but I can tell it’s missing the nuances that actually matter. But when I ask it about things I have no idea about it seems like magic. It feels like having a brilliant tutor.

It’s the same phenomenon as when you read an article about things you’re an expert versus an article about something new to you. In one instance you’ll be like this, this, and that is wrong or is slightly misleading. Maybe it leaves out some crucial part. But you don’t pick up on these things when it’s a topic new to you.

The thing is, it’s probably making similar mistakes in both cases.

This realization changed how I use these tools. It is much faster to learn with AI a lot about everything, but it will reach its limits at some point. For broad learning, for getting up to speed on new topics, for connecting ideas across different fields, AI is incredible. But for deep expertise? That still requires the messy work of actual experience and knowledge.

That’s why I am always careful with its answers now. I think about it, ask for more info and tell it to backup its position. I think admitting what you don’t know about yourself and taking this as fuel to dig deeper is an important step in surpassing basic knowledge and becoming an expert. So you can pick up on these things that are wrong or are missing a nuance.

The gap between what AI can teach you quickly and what real expertise requires isn’t going away. And maybe that’s not a bad thing. AI makes the surface level of almost any topic accessible, but it also shows you just how much depth there is beneath that surface. The magic isn’t in getting perfect answers. It’s in learning to ask better questions and knowing when you need to go deeper than any AI can take you.