From Tokens to AI Agents
How Large Language Models Evolved
“The Universe is under no obligation to make sense to you.” ― Neil deGrasse Tyson
Large Language Models, or LLMs, did not emerge from a single breakthrough. They are the result of several architectural innovations that changed how machines process language, learn from data and interact with external systems.
Modern AI applications are no longer just neural networks generating text. They can combine language models with retrieval systems, tools, memory, orchestration layers and agentic workflows.
This article explores that evolution, from token processing and recurrent networks to Transformers, retrieval and AI agents.
Basics
What Are Tokens?
Before an LLM can process text, the text must be converted into smaller units called tokens.
A token can represent:
- a complete word
- part of a word
- punctuation
- whitespace
- a frequently occurring character sequence
For example, a tokenizer might divide a sentence like this:
"Transformers are powerful."
["Transform", "ers", " are", " powerful", "."]The exact result depends on the tokenizer. Tokens are converted into numeric identifiers and then mapped to vectors called embeddings before they are processed by the model.
The following video provides an accessible introduction to tokens and how language models generate text:
Tokenizer and neuronal Processing
flowchart TD
subgraph TextPreparation["Text Preparation"]
A["Input Text"]:::input
B["Tokenizer"]:::process
C["Token IDs"]:::data
A --> B
B --> C
end
subgraph NeuralProcessing["Neural Processing"]
D["Embeddings"]:::process
E["Transformer Layers"]:::llm
F["Token Probabilities"]:::process
D --> E
E --> F
end
subgraph Generation["Generation"]
G["Select Token"]:::output
H["Append Token"]:::output
G --> H
end
C --> D
F --> G
H -.->|"Repeat"| E
classDef input fill:#dbeafe,stroke:#2563eb,color:#111;
classDef process fill:#e9d5ff,stroke:#9333ea,color:#111;
classDef llm fill:#c4b5fd,stroke:#7c3aed,color:#111;
classDef data fill:#fef3c7,stroke:#d97706,color:#111;
classDef output fill:#ccfbf1,stroke:#0f766e,color:#111;
Tokenization affects the amount of context a model can process, the cost of API requests and the time required to generate a response.
Recurrent Neural Networks
Before Transformers, language models were commonly built using Recurrent Neural Networks, or RNNs.
An RNN processes a sequence one element at a time. It maintains a hidden state that carries information from one processing step to the next.
RNN - The Early Years
flowchart LR
subgraph Sequence["Sequential Processing"]
T1["Token 1"]:::input
H1["Hidden State 1"]:::process
T2["Token 2"]:::input
H2["Hidden State 2"]:::process
T3["Token 3"]:::input
H3["Hidden State 3"]:::process
T1 --> H1
H1 --> H2
T2 --> H2
H2 --> H3
T3 --> H3
end
H3 --> O["Output"]:::output
classDef input fill:#dbeafe,stroke:#2563eb,color:#111;
classDef process fill:#e9d5ff,stroke:#9333ea,color:#111;
classDef output fill:#ccfbf1,stroke:#0f766e,color:#111;
RNNs introduced learnable sequence processing, but they have several limitations:
- Sequential processing limits parallelization.
- Information must pass through many intermediate states.
- Long-range dependencies are difficult to preserve.
- Training becomes increasingly expensive for long sequences.
Long Short-Term Memory
Empirical Evaluation of Gated Recurrent Neural Networks
Long Short-Term Memory networks, or LSTMs, and Gated Recurrent Units, or GRUs, improved the ability of recurrent networks to preserve information. However, they did not remove the fundamental sequential bottleneck.
Attention
Attention mechanisms allow a model to focus on the parts of an input that are most relevant to the current computation.
Consider this sentence:
Maria submitted the pull request after she completed the tests.
To interpret the word she
, the model should give more weight to Maria
than to pull request
or tests
.
Attention Calculation
flowchart LR
Q["Current Token: she"]:::attention
subgraph Context["Available Context"]
M["Maria"]:::entity
P["Pull Request"]:::entity
T["Tests"]:::entity
end
Q --> R["Attention Calculation"]:::process
M -->|"High Weight"| R
P -->|"Low Weight"| R
T -->|"Low Weight"| R
R --> O["Contextual Representation"]:::llm
classDef entity fill:#dbeafe,stroke:#2563eb,color:#111;
classDef attention fill:#fed7aa,stroke:#ea580c,color:#111;
classDef process fill:#f3f4f6,stroke:#6b7280,color:#111;
classDef llm fill:#c4b5fd,stroke:#7c3aed,color:#111;
Attention Is All You Need
Attention is commonly explained through three learned representations:
- Query: the information the current token is looking for
- Key: the information another token advertises
- Value: the information that token contributes
The compatibility between queries and keys determines how strongly the associated values influence the result.
The Transformer Revolution
The decisive breakthrough came in 2017 with the paper Attention Is All You Need.
The Transformer removed recurrence from the core architecture and used self-attention to model relationships between tokens.
Transformer Architecture (Simplified)
flowchart TB
A["Token Embeddings"]:::input
P["Positional Information"]:::data
subgraph TransformerBlock["Transformer Block"]
B["Multi-Head Self-Attention"]:::attention
C["Add and Normalize"]:::process
D["Feed-Forward Network"]:::llm
E["Add and Normalize"]:::process
B --> C
C --> D
D --> E
end
A --> B
P --> B
E --> F["Next Transformer Block"]:::llm
classDef input fill:#dbeafe,stroke:#2563eb,color:#111;
classDef data fill:#fef3c7,stroke:#d97706,color:#111;
classDef attention fill:#fed7aa,stroke:#ea580c,color:#111;
classDef process fill:#f3f4f6,stroke:#6b7280,color:#111;
classDef llm fill:#c4b5fd,stroke:#7c3aed,color:#111;
This enabled:
- greater parallelization during training
- more efficient use of GPUs and AI accelerators
- better handling of long-range relationships
- training on larger datasets
- scaling to increasingly large model architectures
Multi-head attention allows a Transformer to evaluate multiple types of relationships simultaneously. Different attention heads can learn to focus on positional, syntactic or semantic relationships.
Positional information is added because self-attention alone does not inherently understand the order of tokens.
Encoder, Decoder and Encoder-Decoder Models
The original Transformer contained both an encoder and a decoder. Later models specialized the architecture into three major families.
flowchart TB
A["Transformer Architecture"]:::llm
A --> B["Encoder-Only"]:::encoder
A --> C["Decoder-Only"]:::decoder
A --> D["Encoder-Decoder"]:::combined
B --> B1["Understanding and Embeddings"]:::output
C --> C1["Autoregressive Generation"]:::output
D --> D1["Input-to-Output Transformation"]:::output
classDef llm fill:#c4b5fd,stroke:#7c3aed,color:#111;
classDef encoder fill:#dbeafe,stroke:#2563eb,color:#111;
classDef decoder fill:#fed7aa,stroke:#ea580c,color:#111;
classDef combined fill:#dcfce7,stroke:#16a34a,color:#111;
classDef output fill:#ccfbf1,stroke:#0f766e,color:#111;
Encoder-Only Models
Encoder-only models process the complete input and create contextual representations. They are particularly well suited to semantic search, classification, information extraction, text embeddings and entity recognition.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
BERT popularized the encoder-only approach using masked-language-model pretraining.
Decoder-Only Models
Decoder-only models generate text autoregressively. Each new token is predicted using the tokens that precede it. Typical use cases include conversational assistants, code generation, document creation, question answering and structured tool calls.
Encoder-Decoder Models
Encoder-decoder models first create a representation of the input and then generate an output based on that representation. Typical use cases include translation, summarization, question answering and other sequence-to-sequence transformations.
Scaling and Foundation Models
Once the Transformer made large-scale training practical, researchers investigated how model performance changes when increasing the number of model parameters, training data and available compute.
flowchart LR
subgraph Resources["Scaling Inputs"]
P["More Parameters"]:::parameter
D["More Training Data"]:::data
C["More Compute"]:::compute
end
P --> T["Large-Scale Training"]:::process
D --> T
C --> T
T --> F["Foundation Model"]:::llm
F --> I["In-Context Learning"]:::output
F --> Z["Zero-Shot Tasks"]:::output
F --> W["Few-Shot Tasks"]:::output
classDef parameter fill:#e9d5ff,stroke:#9333ea,color:#111;
classDef data fill:#fef3c7,stroke:#d97706,color:#111;
classDef compute fill:#dbeafe,stroke:#2563eb,color:#111;
classDef process fill:#f3f4f6,stroke:#6b7280,color:#111;
classDef llm fill:#c4b5fd,stroke:#7c3aed,color:#111;
classDef output fill:#ccfbf1,stroke:#0f766e,color:#111;
Scaling produced models capable of performing tasks from instructions or examples supplied in their context. This established prompting and in-context learning as important application patterns.
Scaling Laws for Neural Language Models
Language Models are Few-Shot Learners
Training Compute-Optimal Large Language Models
However, adding parameters alone is not sufficient. The amount of training data and the compute budget must be balanced with the size of the model.
Mixture of Experts
A dense neural network activates the same layers and parameters for every input. A Mixture of Experts, or MoE, introduces multiple expert networks and a router that decides which experts should process each token.
flowchart LR
T["Token Representation"]:::input
R{"Router"}:::decision
subgraph Experts["Expert Networks"]
E1["Expert 1"]:::expert
E2["Expert 2"]:::expert
E3["Expert 3"]:::expert
E4["Expert 4"]:::expert
end
T --> R
R -->|"Selected"| E1
R -->|"Selected"| E2
R -.->|"Inactive"| E3
R -.->|"Inactive"| E4
E1 --> O["Combined Output"]:::output
E2 --> O
classDef input fill:#dbeafe,stroke:#2563eb,color:#111;
classDef decision fill:#fecaca,stroke:#dc2626,color:#111;
classDef expert fill:#c4b5fd,stroke:#7c3aed,color:#111;
classDef output fill:#ccfbf1,stroke:#0f766e,color:#111;
Only a subset of experts is activated for a given token. This allows the model to contain more parameters without using all of them for every inference step.
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
MoE introduces additional engineering challenges, including routing, expert load balancing, memory allocation and communication between devices.
Retrieval-Augmented Generation
A pretrained language model stores statistical patterns in its parameters. This does not give it direct access to an authoritative or current knowledge source.
Retrieval-Augmented Generation, or RAG, searches an external knowledge source and adds relevant information to the model's context before it generates an answer.
flowchart TD
U["User Question"]:::input
subgraph Retrieval["Knowledge Retrieval"]
E["Create Query Embedding"]:::process
VS["Vector or Hybrid Search"]:::retrieval
DOC["Relevant Documents"]:::knowledge
E --> VS
VS --> DOC
end
subgraph Generation["Answer Generation"]
P["Build Grounded Prompt"]:::process
LLM["Language Model"]:::llm
ANS["Grounded Answer"]:::output
P --> LLM
LLM --> ANS
end
U --> E
U --> P
DOC --> P
classDef input fill:#dbeafe,stroke:#2563eb,color:#111;
classDef process fill:#f3f4f6,stroke:#6b7280,color:#111;
classDef retrieval fill:#fde68a,stroke:#d97706,color:#111;
classDef knowledge fill:#fef9c3,stroke:#ca8a04,color:#111;
classDef llm fill:#c4b5fd,stroke:#7c3aed,color:#111;
classDef output fill:#ccfbf1,stroke:#0f766e,color:#111;
Potential benefits of RAG include access to current or domain-specific information, answers grounded in identifiable documents, knowledge updates without retraining and the ability to provide citations.
RAG for Knowledge-Intensive NLP Tasks
Retrieval does not automatically guarantee correctness. Quality depends on document preparation, indexing, query construction, ranking, access control and the instructions given to the model.
Tool Calling
Modern LLM applications can interact with external tools such as search engines, databases, repositories, calculators, compilers, ticketing systems and enterprise APIs.
sequenceDiagram
participant User
participant Orchestrator
participant LLM
participant Tool
User->>Orchestrator: Submit goal
Orchestrator->>LLM: Prompt with available tools
LLM-->>Orchestrator: Propose structured tool call
Orchestrator->>Orchestrator: Validate request
Orchestrator->>Tool: Execute approved call
Tool-->>Orchestrator: Return result
Orchestrator->>LLM: Add observation to context
LLM-->>Orchestrator: Produce response or next action
Orchestrator-->>User: Return result
The model normally proposes a structured tool call, but the surrounding application executes it. Trusted application code should validate permissions, arguments and policy constraints before invoking an external system.
This separation matters because language-model output is probabilistic, while access control and safety rules must be enforced deterministically.
The Rise of AI Agents
A conventional LLM interaction produces a response to an input. An AI agent adds an iterative control loop around the model.
A typical agent repeatedly observes its context, evaluates what should happen next, selects an action, receives the result, updates its working state and continues until it reaches a stopping condition.
Agent Loop
flowchart TD
Goal["Goal"]:::input
subgraph Agent["Agent Loop"]
Observe["Observe"]:::process
Reason["Reason"]:::reason
Decide{"Need Action?"}:::decision
Tool["Use Tool"]:::tool
Memory["Update Memory"]:::memory
Observe --> Reason
Reason --> Decide
Decide -->|Yes| Tool
Tool --> Memory
Memory --> Observe
end
Result["Final Result"]:::output
Goal --> Observe
Decide -->|No| Result
classDef input fill:#dbeafe,stroke:#2563eb,color:#111;
classDef process fill:#f3f4f6,stroke:#6b7280,color:#111;
classDef reason fill:#fed7aa,stroke:#ea580c,color:#111;
classDef decision fill:#fecaca,stroke:#dc2626,color:#111;
classDef tool fill:#dcfce7,stroke:#16a34a,color:#111;
classDef memory fill:#fef3c7,stroke:#d97706,color:#111;
classDef output fill:#ccfbf1,stroke:#0f766e,color:#111;
The ReAct pattern combines reasoning and actions in an interleaved process. Observations from external actions are returned to the model and influence its next decision.
An agent is therefore not necessarily a fundamentally different model. It is often a language model integrated into software that provides tools, memory, state management and an execution loop.
Introduction to the AI agent concept
ReAct: Synergizing Reasoning and Acting in Language Models
An Agent Is a System, Not Just a Model
Calling an application an agent can hide important implementation details. A production-grade agent normally includes several cooperating components.
flowchart TB
User["👤 User / Calling System"]:::input
User --> Gateway
subgraph Governance["Governance & Security"]
Identity["Identity & Authorization"]:::security
Policy["Policies & Guardrails"]:::security
Audit["Audit & Logging"]:::security
end
Gateway["Agent Gateway"]:::process
Gateway --> Identity
Identity --> Orchestrator
Policy --> Orchestrator
subgraph Core["Agent Core"]
Orchestrator["Agent Orchestrator"]:::reason
LLM["Language Model"]:::llm
Memory["Memory"]:::memory
Retrieval["Retrieval"]:::retrieval
end
Orchestrator <--> LLM
Orchestrator <--> Memory
Orchestrator <--> Retrieval
Audit -. monitors .-> Orchestrator
Orchestrator --> Tools
subgraph Tools["Tool Layer"]
direction TB
API["Enterprise APIs"]:::tool
DB["Databases"]:::tool
SCM["Source Control"]:::tool
Search["Search"]:::tool
Code["Code Execution"]:::tool
end
Tools --> Validation
Validation["Output Validation"]:::process
Validation --> User
classDef input fill:#dbeafe,stroke:#2563eb,color:#111;
classDef process fill:#f3f4f6,stroke:#6b7280,color:#111;
classDef reason fill:#fed7aa,stroke:#ea580c,color:#111;
classDef llm fill:#c4b5fd,stroke:#7c3aed,color:#111;
classDef tool fill:#dcfce7,stroke:#16a34a,color:#111;
classDef memory fill:#fef3c7,stroke:#d97706,color:#111;
classDef retrieval fill:#fde68a,stroke:#d97706,color:#111;
classDef security fill:#fecaca,stroke:#dc2626,color:#111;
For enterprise and regulated environments, the surrounding architecture may be at least as important as the selected model.
A production-grade solution should consider explicit tool permissions, identity propagation, input and output validation, protection against prompt injection, handling rules for sensitive information, execution limits, audit logging, human approval for consequential actions and safe recovery from partial failures.
An agent with powerful tools but insufficient controls is not simply an advanced assistant. It can become an unreliable automation system with a substantial attack surface.
The Architectural Evolution at a Glance
flowchart TD
Sequential["Sequential Models"]:::legacy
AttentionStage["Attention-Based Models"]:::attention
Foundation["Foundation Models"]:::llm
Sequential --> AttentionStage
AttentionStage --> Foundation
Foundation --> Extended
subgraph Extended["Extended Model Systems"]
MOE["Mixture of Experts"]:::expert
RAG["Retrieval"]:::knowledge
TOOLS["Tool Calling"]:::tool
end
Extended --> Agentic
Agentic["Agentic Systems"]
classDef legacy fill:#f3f4f6,stroke:#6b7280,color:#111;
classDef attention fill:#fed7aa,stroke:#ea580c,color:#111;
classDef llm fill:#c4b5fd,stroke:#7c3aed,color:#111;
classDef scale fill:#dbeafe,stroke:#2563eb,color:#111;
classDef expert fill:#e9d5ff,stroke:#9333ea,color:#111;
classDef knowledge fill:#fef3c7,stroke:#d97706,color:#111;
classDef tool fill:#dcfce7,stroke:#16a34a,color:#111;
class Extended scale;
The progression is not simply a story of models becoming larger. Each stage changed where computation happens and how information flows:
- RNNs maintained a sequential hidden state.
- Attention created dynamic connections across a sequence.
- Transformers enabled highly parallel training.
- Scaling increased generality and in-context capability.
- Mixture of Experts introduced conditional computation.
- Retrieval connected models to external knowledge.
- Tool calling connected models to deterministic capabilities.
- Agent loops transformed individual responses into multi-step processes.
Conclusion
The Transformer remains the foundation of most modern Large Language Models, but the architecture of useful AI systems now extends far beyond the neural network.
Tokens determine how information enters the model. Attention determines how context is combined. Autoregressive decoding produces output one token at a time. Retrieval supplies external knowledge, tools provide additional capabilities and agent loops coordinate repeated decisions and actions.
Do not evaluate an AI solution only by asking which model it uses. Evaluate the complete system around the model.
That system includes its context strategy, retrieval quality, tool contracts, identity model, permissions, orchestration logic, observability and safety boundaries.
The model may propose decisions, but the surrounding software determines whether those decisions lead to dependable engineering outcomes.
This arcticle and the illustrations were created using AI tools and agents.
Comments