"AI model" has come to mean "chatbot", but the eight acronyms you see in model diagrams (LLM, LCM, LAM, MoE, VLM, SLM, MLM, SAM) are not eight competitors. Some name what a model can do, one names how it is wired inside, one names its size, one names how it was trained, and LCM names two unrelated research ideas. Once you sort them that way, choosing models for a product stops being "which is best" and becomes "which job does each part of the system need done".
Most of the AI features I have shipped in the last two years started as "call the big model" and ended as three or four models with narrow jobs. The big model is still in there. It just stopped doing the work a 3B model on the same box, or an image segmentation model, does faster and cheaper.
What are the types of AI models?
The common types of specialized AI models are best grouped by what each label describes, because the labels answer different questions. An LLM, a VLM, a LAM and SAM describe capability (what goes in and what comes out). MoE describes architecture (how the network routes work internally). SLM describes scale. MLM describes a training objective. LCM is a research label with two meanings.
| Question the label answers | Labels |
|---|---|
| What can it do? | LLM, VLM, LAM, SAM |
| How is it built inside? | MoE |
| How big is it? | SLM |
| How was it trained? | MLM |
| What does it represent or generate? | LCM (both meanings) |
That grouping explains why the categories overlap. Mixtral is a language model and an MoE. A 4B open-weight model that reads images is a small, multimodal language model, and if its feed-forward layers are sparse it is an MoE as well. Asking "should I use an MoE or a VLM?" is like asking whether a car should be a diesel or an SUV.
The biggest model is rarely the right one for every job in a system. Model selection is a cost, latency and accuracy decision you make per task, the same way you pick a queue or a database.
LLMs: how next-token prediction works
A large language model (LLM) is a neural network trained to predict the next token in a sequence, and everything it does, from chat to code, is that prediction repeated. A token is a chunk of text, usually a word or part of one. The architecture underneath almost every current LLM is the Transformer, introduced in Attention Is All You Need (Vaswani et al., 2017).
Tokens, embeddings and self-attention
The tokenizer splits text into subword pieces from a fixed vocabulary, so a rare word becomes a few common pieces instead of an unknown. Each piece is mapped to an embedding (a learned vector that stands in for the token's meaning), and positional information is added so the model knows order.
Self-attention is the part that made Transformers win. For every token, the layer computes how much each earlier token matters to it, and mixes their information in by those weights. In "the invoice the customer disputed was refunded", attention is how "was refunded" connects back to "invoice" across the words in between. A GPT-style model applies a causal mask, so each position can only look left: it must not see the answer it is about to predict.
Generation is a loop
The final layer produces a probability for every token in the vocabulary. The runtime picks one (greedily, or by sampling with a temperature), appends it, and runs the model again. A 500-token answer is 500 forward passes. That single fact explains most LLM engineering: output tokens cost more than input tokens, latency grows with answer length, and streaming exists because the model produces one piece at a time anyway.
Where LLMs break
Hallucination is the predictable result of the objective. The model was trained to produce a plausible continuation, not a verified one, so it will confidently complete a sentence about an API that does not exist. Grounding it with retrieved documents and checking its output against something real is the fix, not a better prompt.
The context window (the number of tokens it can attend to at once) is finite. And an LLM on its own cannot do anything: it emits text. Booking, querying or clicking needs software around it, which is where action models come in.
LCM means two unrelated things
LCM stands for both Latent Consistency Model and Large Concept Model, and the two share nothing except the letters. A Latent Consistency Model makes image generation faster. A Large Concept Model is Meta research into language models that predict whole sentences. If a diagram says "LCM" and shows text, it means the second. If it shows images in four steps, it means the first.
Latent Consistency Models: fewer denoising steps
A diffusion model generates an image by starting from random noise and removing a little of it at a time, guided by your prompt, until a picture remains. Each removal step is a full pass through a large network, and classic samplers for Stable Diffusion use something like 20 to 50 of them. That is why a single image can take seconds even on a good GPU.
Latent Consistency Models (Luo et al., October 2023) attack the step count. "Latent" means the work happens in a compressed latent space (a smaller representation of the image that a decoder turns back into pixels), as in Stable Diffusion. "Consistency" refers to consistency distillation: a student model is trained so that from any point along the noisy path, it predicts the same clean end result the slow teacher would reach. Having learned to jump to the end, it can produce a usable image in roughly 2 to 4 steps.
The follow-up, LCM-LoRA, packages that acceleration as a small adapter for existing Stable Diffusion fine-tunes. The result is image generation fast enough for interactive work: live previews as a user types a prompt, sketch-to-image tools that update as you draw. The trade is some loss of fine detail against a full-step run, so I would use it for previews and drafts before the final render.
Large Concept Models: predicting sentences, not tokens
Meta's Large Concept Models paper (FAIR, December 2024) asks what happens if a language model reasons one idea at a time instead of one token at a time. It treats a sentence as a "concept" and works in the embedding space of SONAR, Meta's existing sentence encoder covering around 200 languages in text, with speech support as well.
The pipeline splits input into sentences, encodes each into one SONAR vector, trains a model to predict the next sentence vector, and decodes that vector back into text with SONAR's frozen decoder. Because the concept space is shared across languages, a model trained mostly on English data can be decoded into other languages. The paper explored several ways to predict the next vector (plain regression, diffusion-based generation and a quantized space), with most experiments at 1.6B parameters and one architecture scaled to 7B.
This is research, not something you deploy next sprint. Sentence boundaries are a crude unit of meaning, and decoding a predicted vector can lose precise wording. It is worth following because it tests whether "next token" is the only way to build a language model, and the code is public at facebookresearch/large_concept_model.
LAM: when the model has to act
A Large Action Model (LAM) is a model built to turn a request into actions in software (clicks, API calls, form fills) rather than into text, and the term is an emerging industry label, not a standard architecture. Rabbit popularized it in January 2024, Salesforce ships function-calling models under the xLAM name, and Microsoft's Large Action Models: From Inception to Implementation (December 2024) describes training one for Windows applications. Three uses of the term, three different things.
What they share is a loop. Text generation ends when the answer is written. Action ends when the world is in the state the user asked for, and you only know that by looking.
An LLM with tools, or a specialized action model?
Almost every production agent I have seen, including the ones I have built, is an LLM with tools. You give a general model a list of functions with JSON schemas, it replies with a structured call instead of prose, your code runs the call, and the result goes back into the next turn. The model plans; your code acts. The browser-automation agents that click through web pages work the same way, with "click", "type" and "read the page" as the tools.
A specialized action model is trained on recorded action sequences for one environment, so it learns, for example, where the "Export" control sits in a given desktop app. It can be smaller and faster on its home turf, and worse everywhere else.
The engineering that matters is the same either way: the narrowest permissions that do the job, confirmation on destructive actions, a log of every call, and verification against the system of record instead of the model's own report. An agent that can refund under a fixed amount and hands anything larger to a human is a feature. One that can do anything the API allows is an incident waiting for a date.
Mixture of Experts: why only some weights run
A Mixture of Experts (MoE) model replaces one large feed-forward block in each Transformer layer with several smaller "expert" blocks and a router that sends each token to only a few of them. The model holds many parameters but uses a fraction of them per token, so it can know more without each token costing more to compute.
In a dense model, every parameter processes every token, so doubling the parameters roughly doubles the compute. MoE breaks that link through sparse activation: the router (a small learned layer) scores the experts for the current token and only the top few run.
Mixtral 8x7B (Mistral AI, January 2024) is the clearest public example. Each layer has 8 experts and the router picks 2 per token. The model has about 47B parameters in total (not 56B, because attention and embedding layers are shared rather than copied 8 times), but only about 13B are active for any one token. That is how it matches or beats much larger dense models on many of the benchmarks in its paper while running at the speed of a far smaller one.
The misconception: experts are not subject specialists
The name suggests one expert for maths, one for code, one for French. The Mixtral paper looked for that and did not find clear topic specialization; the routing patterns it reports follow syntax and token type more than subject. Treat "expert" as a block of weights the router learned to prefer for certain token patterns, not as a department.
What MoE costs you
Sparse compute does not mean sparse memory. Every expert has to be loaded, because the next token might route to any of them. Mixtral needs memory for 47B parameters even though it computes with 13B, which is why it is harder to self-host than its speed suggests. Training is fiddlier too: routers can collapse onto a few favorite experts unless a balancing loss spreads the traffic.
For an application developer, MoE is mostly a property of the model you pick, not something you build. It shows up as "fast for its quality, heavy for its speed", and that is the trade to check against your hardware.
Vision language models: how a model reads screenshots
A vision language model (VLM) is a language model that also accepts images, made by attaching a vision encoder to an LLM through a projection layer that translates image features into the vectors the LLM already understands. It is the "multimodal" in most product announcements: one model, more than one kind of input.
LLaVA (Liu et al., 2023, "Visual Instruction Tuning") is the cleanest architecture to learn from, because it is built almost entirely from parts that already existed.
LLaVA pairs OpenAI's CLIP vision encoder (ViT-L/14) with the Vicuna language model. The vision encoder cuts the image into patches and turns each into a vector. On its own, those vectors mean nothing to the language model; they live in a different space. The projection layer is trained to map them into the LLM's word-embedding space, so to the LLM an image becomes a run of extra tokens it can attend to like any other.
LLaVA trained in two stages: first only the projection layer, on image-caption pairs, to align the two spaces; then the projection and the LLM together on GPT-4-generated conversations about images, so it learned to answer questions rather than only caption. Commercial VLMs are more elaborate, but this shape still explains them.
Where VLMs earn their place
Document understanding is the big one: invoices, receipts and forms where layout carries meaning that OCR text loses. Screenshot analysis is how I have used them most: a user attaches a screenshot of an error, and the model reads the dialog, the URL bar and the console before a human opens the ticket. Product identification, chart questions and visual Q&A fit the same shape.
Where they still fall over
Visual reasoning is weaker than visual description. VLMs miscount objects, confuse left and right, misread small or dense text when the image is downscaled to the encoder's input resolution, and will describe a chart's trend confidently while getting the exact values wrong. For anything with money or compliance attached, have the VLM extract fields into a schema and validate them, rather than trusting a free-text reading.
Small language models and local inference
A small language model (SLM) is a language model small enough to run on modest hardware, usually from a few hundred million to a few billion parameters, and for narrow, high-volume tasks it is often the better engineering choice. There is no official cut-off; the label is about where it can run, not a magic number.
Parameter count drives everything you pay for. Weights must sit in memory, and every generated token touches them. A 3B model stored at 16-bit precision needs about 6GB just for weights (3 billion × 2 bytes). Quantized to 4 bits, the same model needs around 2GB, which fits a laptop, a modest cloud instance or a recent phone. Fewer bytes to move per token also means lower latency.
Quantization is storing weights at lower precision (8-bit or 4-bit integers instead of 16-bit floats), trading a small amount of accuracy for a large cut in memory. llama.cpp and its GGUF file format made this routine, with quantization levels like Q4_K_M as the common default. Ollama wraps llama.cpp-style inference in a local server with a one-line install.
What small models are good and bad at
They are good at classification, extraction into a fixed schema, routing and short rewrites. They are worse at long multi-step reasoning and at knowledge they were never trained on. Microsoft's Phi-3-mini (3.8B), Meta's Llama 3.2 1B and 3B, and the smaller Gemma models target this tier. Local inference also changes the privacy story: a message classified on your own server never leaves it.
A local triage model that escalates
The pattern I reach for most is hybrid. A local SLM labels every incoming message with a category and a complexity, and a JSON schema limits its reply to a few allowed values so it cannot ramble (Ollama's /api/chat takes that schema in its format field). Simple, common requests are answered locally against a help article. Everything else goes to a cloud LLM, with the small model's label attached so the big model starts from that guess.
Run the classifier at temperature 0 so the same message always gets the same label. Log both decisions, and sample the "simple" bucket regularly: that is where a cheap model's mistakes hide.
Masked language models and why BERT still matters
A masked language model (MLM) is trained by hiding some words in a sentence and learning to fill them in from the words on both sides, which makes it good at understanding text rather than generating it. The defining example is BERT (Devlin et al., Google, 2018). Shown "Artificial [MASK] is changing the [MASK]", it learns to recover "intelligence" and "world" from everything around the gaps.
During BERT's pre-training, 15% of token positions are chosen for prediction. Of those, 80% are replaced with [MASK], 10% with a random token and 10% left unchanged, so the model cannot learn to only pay attention when it sees the mask symbol.
Bidirectional versus autoregressive
The difference from an LLM is which way the model is allowed to look. A GPT-style model predicts the next token and can only see the left side. A masked model sees the whole sentence at once, so its reading of "bank" in "the bank by the river flooded" is shaped by "river", which comes after it. That makes it strong at reading and weak at writing: there is no natural loop for producing a long answer one token at a time.
Where masked models still win
Classification, named entity recognition (finding names, dates, product codes and amounts in text), spam and intent detection, and extractive question answering are all jobs where a fine-tuned BERT-sized encoder (BERT-base is about 110M parameters) is fast, cheap and accurate. If you need to label millions of rows a day, an encoder runs on a CPU at a cost an LLM cannot match.
The connection to embeddings, and the limit of it
Encoders are also where most text embedding models came from. An embedding model turns text into a vector so that similar meaning lands at nearby points, which is what semantic search and RAG retrieve by. I covered what that buys a product in what vector embeddings are and where they help.
But a raw BERT is not a good embedding model, and modern embedding models are not all MLMs. BERT's pre-training never asked it to place similar sentences near each other, so its vectors compare poorly out of the box. Sentence-BERT (2019) fixed that by fine-tuning on sentence pairs, and most current embedding models are trained with contrastive objectives (pull matching pairs together, push others apart), several on top of decoder LLMs rather than encoders. MLM is a common starting point for embeddings, not the method that defines them.
Segment Anything: masks from a click
The Segment Anything Model (SAM) is Meta's promptable image segmentation model: you give it an image and a point, box or rough mask, and it returns a pixel-accurate outline of the object you meant. It is the one model on this list that has nothing to do with language in its original form.
Three vision tasks get confused, so separately: classification says "this photo contains a dog", object detection draws a box around the dog, and segmentation marks exactly which pixels are dog. Segmentation is what you need to cut a product out of its background, blur a face, or measure an area.
How SAM works inside
The original SAM (Kirillov et al., April 2023) splits the work into three parts so that the expensive part runs once.
The image encoder is a large Vision Transformer that turns the photo into an embedding once. The prompt encoder turns a click (marked as "include" or "exclude"), a box or a rough mask into a few vectors. The lightweight mask decoder combines the two and outputs masks. Because only the light parts re-run, the paper reports interactive speed in a browser, which is what makes "click to select" tools feel instant.
A single click is often ambiguous (did you mean the shirt, or the person wearing it?), so SAM returns three candidate masks with confidence scores and lets you pick. It was trained on SA-1B, a dataset of 11 million images and over 1.1 billion masks that Meta built with SAM itself in the annotation loop.
SAM 2 and SAM 3
SAM 2 (July 2024) extended the same idea to video. It keeps a memory of the object from earlier frames, so one click on frame one tracks that object through the clip, including when it is briefly hidden. SAM 3 (November 2025) adds what Meta calls promptable concept segmentation: you type a short noun phrase such as "yellow school bus" or give an example image, and it finds and masks every matching instance, in images and video, while still accepting the points and boxes earlier versions used.
Where it fits, and what to watch
Catalog background removal, photo editing, video effects, dataset labeling and robotics perception all use it. The limits: SAM 1 and 2 know where an object's edges are, not what it is, so pair them with a detector or a VLM when you need a label. The image encoder wants a GPU. And hair, wires and glass still come out with rough edges a product photo will show.
How these models work together in one product
In a real AI feature, these models are components with separate jobs, and the architecture is the routing between them. Here is the support platform I keep using as an example, with each type where it fits:
| Component | Job in the system |
|---|---|
| SLM | Classify every incoming message and decide what needs a big model |
| Embedding model | Index help articles and past tickets for retrieval |
| LLM | Write the reply, reasoning over what retrieval found |
| VLM | Read attached screenshots and scanned documents |
| SAM | Isolate the damaged item in a product-return photo |
| LAM-style agent | Look up the order and issue a refund through the API |
| MoE | How the chosen LLM happens to be built, not a separate box |
| LCM | Latent consistency for fast image previews, if the product generates images |
Note the last two rows. MoE is not a component you call; it might describe the LLM you picked. And Large Concept Models do not belong in a production diagram today; they are a research direction you watch.
Walking one request through it
A customer uploads a photo of a scanned invoice and asks why they were charged twice.
- The VLM reads the uploaded invoice and extracts the line items, dates and totals into a schema the code validates.
- The embedding model has already indexed the billing help articles and past resolved tickets.
- Retrieval pulls the articles closest to "duplicate charge" plus this customer's billing history. Retrieval is the step that decides whether the answer is right, which I wrote about in RAG examples where retrieval made the difference.
- The LLM writes the explanation, grounded in what was retrieved and what the VLM extracted.
- Agent tools fetch the live payment records and, if policy allows and the amounts match, issue a refund for the duplicate.
- The SLM handled the first step before any of this, labeling the message "billing, complex" so it went down this path at all, and it handles the far larger stream of "how do I reset my password" messages without touching the cloud.
Why one model should not do everything
A frontier multimodal model could do most of these steps alone. I would not build it that way. Every message would pay frontier prices, including the trivial ones that are most of the volume. One model doing six jobs produces one opaque output instead of six inspectable ones. And swapping the classifier would mean retesting everything. Separate components with typed handoffs are easier to test, cheaper to run and safer to change.
Comparison table of AI model types
| Model / concept | Primary purpose | How it works | Best use cases | Engineering relevance |
|---|---|---|---|---|
| LLM | Generate and reason over text | Transformer predicts the next token, in a loop | Chat, coding assistants, summaries, drafting | The default reasoning component; needs grounding and tools |
| LCM (Latent Consistency Model) | Fast image generation | Distilled diffusion that jumps to the result in about 2 to 4 steps | Live previews, interactive image tools | Cuts image latency and GPU cost, some quality trade |
| LCM (Large Concept Model) | Research into sentence-level modeling | Predicts the next SONAR sentence embedding, decodes to text | Multilingual generation research | Watch, do not deploy yet |
| LAM | Turn requests into actions | Plan, call tools, observe, verify, in a loop | Workflow automation, browser and app agents | Mostly LLM + tools today; permissions and verification matter most |
| MoE | Bigger capacity without more compute per token | Router sends each token to a few expert blocks | Serving large models efficiently | Fast for its quality, but all experts must fit in memory |
| VLM | Understand images alongside text | Vision encoder → projection → LLM | Documents, screenshots, charts, visual Q&A | Validate extracted fields; weak at counting and fine text |
| SLM | Run cheaply and locally | A small LLM, often quantized | Classification, extraction, routing, on-device | First stop in a hybrid router; privacy by staying local |
| MLM | Understand text | Predict masked words from both sides | Classification, entity recognition, search ranking | Cheap at high volume; a base for some embeddings |
| SAM | Segment objects in images and video | Heavy image encoder once, light prompt encoder and mask decoder per click | Background removal, editing, video, labeling | Pair with a detector or VLM for labels; GPU for the encoder |
FAQ
Is an MoE model the same as an LLM?
An MoE model can be an LLM. Mixture of Experts describes how a network is built inside, while LLM describes what it does, so Mixtral is both a large language model and an MoE.
What is the difference between a VLM and an LLM?
A VLM is an LLM with a vision encoder and a projection layer attached, so it accepts images as input as well as text. The language model at its core still generates the answer as text.
Which LCM is the one in AI model diagrams?
If the diagram is about language and concepts, it means Meta's Large Concept Model, which predicts sentence embeddings. If it is about generating images in a few steps, it means a Latent Consistency Model. They are unrelated.
Is a Large Action Model a real model type?
It is an industry term rather than a standard architecture. Some vendors train specialized models on recorded actions, but most production agents are general LLMs calling tools in a plan, act and verify loop.
When should I use a small language model instead of a large one?
Use one when the task is narrow and frequent, such as classification or extraction into a fixed schema, or when data must stay on your own hardware. Escalate to a larger model when the task needs long reasoning or broad knowledge.
The decision rule I use
Start from the job, not the model. Write down what goes in (text, an image, a click, a live system), what must come out (an answer, a label, a mask, a changed record), and how many times a day it happens. Pick the smallest model that meets the accuracy bar for that job, give it a schema to answer in, and only route to the large model what the small one cannot handle.
The caveat: this adds moving parts, and a two-person team with a few hundred requests a day may be better served by one good LLM call until the bill or the latency says otherwise. Split the system when you can name the job a second model would do better.
Which of these have you put into a real product so far, and which one surprised you once it met real users?



