What Is an LLM and How Does It Work?

If you've been using ChatGPT, Claude, or Gemini and thinking of them as a "very smart search engine," you're not alone — and you're also missing something important. Understanding what an LLM actually is will change how you use it, how you prompt it, and how you build with it.

What is an LLM?

LLM stands for Large Language Model. It's a type of AI system trained on enormous amounts of text — books, websites, code, articles — with one goal: to predict what the next most likely word (or token) should be, given everything that came before it. That's it. At its core, an LLM is a very sophisticated next-word prediction engine. But because it was trained on so much human knowledge, that prediction turns out to be extraordinarily useful — it can write, reason, summarize, translate, generate code, and more. How does it actually work? When you type a message to Claude or ChatGPT, here's what happens at a high level: Your text is broken into tokens. A token is roughly a word or part of a word. "Product manager" might be 2–3 tokens. The model reads all the tokens in context. It considers everything — your message, the conversation history, any system instructions — at once. It generates a response token by token. Each word it outputs is based on a probability distribution: given everything before, what's the most likely next word? It keeps going until it reaches a stopping point — either a natural end, or a token limit. What this means for you as a PM This mental model has three practical implications: Context is everything. The more relevant context you give the model, the better its output. Vague prompts produce vague results — not because the model is lazy, but because it's predicting based on limited information. It doesn't "know" things the way you do. It has no memory between conversations (unless given tools that provide it). It's not searching the internet in real time (unless explicitly connected to it). It's working from patterns learned during training. It can be confidently wrong. Because it's predicting likely text, it can generate plausible-sounding but incorrect information. This is called hallucination. Always verify outputs for anything high-stakes. The key takeaway An LLM is not magic, and it's not a database. It's a pattern-matching system trained on human language at massive scale. Once you internalize that, you start prompting differently — more deliberately, with more context, and with healthier skepticism about the output.