Opens in a new tab
Home / Blog / Artificial Intelligence

A Beginner’s Guide to Natural Language Processing

Natural language processing has changed more in the last few years than in the thirty years before that. Here's a current take on what NLP is, what large language models replaced and what they didn't.

SoftmaxAI EngineeringPublished March 10, 2026Updated August 25, 202612 min read
Topic
Natural language processing, from fundamentals to current practice
Biggest shift
Task-specific models → general models, prompted
Still classical
High-volume classification, extraction, search ranking
Choosing rule
Volume and latency decide; capability rarely does

What is natural language processing?

Natural language processing (NLP) is the field concerned with getting computers to work with human language in a useful way — reading it, interpreting it, searching it, translating it and producing it. It sits at the intersection of linguistics, machine learning and plain old software engineering, and it exists because the vast majority of the information organisations hold is written by people for people, rather than structured for machines.

Every support ticket, contract, clinical note, review, email and recorded call is language. NLP is what turns that from an archive into something a system can actually act on.

  • Understanding. Working out what a piece of text is about, what it is asking for, what sentiment it conveys and which entities it refers to.
  • Extraction. Extracting structured facts from unstructured prose — dates, amounts, parties, obligations, diagnoses, part numbers.
  • Retrieval. Finding the relevant passage among millions, which is the bedrock of search and of retrieval-augmented generation.
  • Generation. Producing fluent, appropriate text (summaries, replies, translations, drafts) that a human can then use or edit.
  • Conversation. Maintaining a coherent multi-turn exchange, which combines all of the above and forms the basis of chatbots and voice agents.

How text becomes something a model can use

All models work on numbers, so every NLP system starts by converting text into a numerical form. Understanding that goes a long way to explaining most of the behaviour that beginners find so surprising.

Tokenization

Text is broken up into tokens — roughly word fragments. "Unbelievable", for instance, might be three tokens, while a common word will generally be one. That's why models are charged and limited by token rather than by word, why a rare technical term takes up more of your context window than a common word, and why some languages are more expensive to process than others.

Embeddings

Each token (and, by extension, each sentence or document) is represented as a vector — a list of numbers — such that texts with similar meaning end up close to each other in the vector space. This is the basis of semantic search: a query about "refund policy" can return a passage titled "returns and money back", which has no words in common with it.

  • Embeddings capture meaning, not spelling. That's precisely why they struggle with part numbers, error codes and acronyms, where the exact characters are the meaning.
  • That weakness has a standard fix. Combine vector search with keyword search. Hybrid retrieval covers the blind spots of each approach and is the default in production systems.
  • Embeddings power more than search. Clustering support tickets, deduplicating records, detecting near-duplicates and routing by topic all come out of the same representation.

Attention and transformers

The architecture underpinning modern NLP allows a model to weigh every token against every other token when interpreting a passage. "It" can thus be resolved to the right noun, and a sentence can be read in the context of the paragraph. This is the change that made general-purpose language models possible, which is explained further in our guide to foundation models.

What changed with large language models

Before general-purpose language models, you'd build an NLP application as a pipeline: a tokenizer, a part-of-speech tagger, a parser, a named-entity model and a task-specific classifier, which you'd have to train on your own labelled examples. Each stage in the pipeline required its own data, its own training and its own maintenance.

A lot of that collapsed. A single model, given clear instructions, can now do classification, extraction, summarisation and translation acceptably without any task-specific training at all. For a team starting out, that's been a massive reduction in the barrier to entry.

SituationBetter approachWhy
Prototype, unclear requirementsPrompted general modelNo labelled data needed; iterate in hours
Millions of documents per dayFine-tuned small model or classical classifierCost and latency dominate at volume
Strict, stable output schemaSmall fine-tuned modelMore predictable and cheaper than prompting
Exact-match retrieval of codes or IDsKeyword search plus embeddingsEmbeddings blur exact tokens
Answers must cite internal documentsRetrieval-augmented generationGrounds output in your own sources
Regulated, must be explainableClassical or interpretable modelDecision path must be defensible
Choosing an approach today

The most common beginner mistake these days is to go straight for a large model on a task that a logistic regression can solve at a thousandth of the cost. The second most common mistake is the reverse: hand-crafting a brittle rule system for a task that a model can do in a single call.

Where NLP is used

  • Customer support. Routing and prioritising tickets, drafting replies, summarising long threads and surfacing the relevant policy — see our support ticket agent case study.
  • Healthcare. Drafting clinical documentation from a consultation, coding notes and extracting structured findings from free-text records.
  • Finance and legal. Extracting obligations, dates and parties from contracts; monitoring communications for compliance; summarising filings.
  • Retail and e-commerce. Understanding product queries, generating product descriptions and metadata, and mining reviews for genuine product signal.
  • Voice interfaces. Speech-to-text, intent understanding and text-to-speech combined into a system that holds a phone conversation — as in our voice agent case study.
  • Knowledge access. Semantic search across internal documentation, so that people find the passage that answers their question rather than a list of files.

Evaluating an NLP system

This is where beginners most often get stuck. Unlike a numeric prediction with one single correct answer, most language tasks have many valid answers, so "is it working?" has to be defined before you can even begin to measure it.

  • Write the test set first. Gather real examples with the expected outputs before you start to build. If you do it afterwards, you end up with a set which just flatters whatever you've built.
  • Classification is the easy case. Precision, recall and the confusion matrix all apply directly. Look at the errors that actually matter, rather than at the overall accuracy.
  • Generation needs criteria, not a score. Define what a good summary or reply should contain and what it should never contain, then assess against that. Automated similarity metrics correlate very poorly with usefulness.
  • Groundedness for anything retrieval-based. Check that assertions trace to a retrieved source. That's what makes hallucination detectable rather than just embarrassing.
  • Track failures in production. The examples that break the system are the most valuable dataset you'll ever have. Capture them deliberately.

How SoftmaxAI builds NLP systems

We build our language systems the same way we build everything else: a written, client-approved specification prior to any code, an agent-accelerated build led by senior engineers, and every decision that matters owned by a human.

  • Define good before building. The evaluation set, the refusal behaviour and the acceptance criteria are all written during the specification phase with your domain experts — not reverse-engineered from complaints.
  • Match the method to the volume. We size the approach against real throughput and latency requirements, as the cheapest adequate approach beats the most capable one at scale.
  • Ground answers in your sources. Where accuracy matters, retrieval anchors the output to your documents with citations — see our RAG development services.
  • Deployed to your cloud. Systems run in your own environment with monitoring and production evaluation. As a Google Cloud Partner we build natively on Vertex AI and Gemini where that makes sense.

Next read: essential NLP techniques for the practical toolkit, and our guide to foundation models for what lies beneath modern systems.

NLPEmbeddingsTokenizationTransformersText classificationNamed entity recognitionSemantic searchLLMs

FAQ

Questions we get asked.

What is natural language processing in simple terms?
It's the engineering discipline of getting computers to work usefully with human language — reading text, understanding what it means, finding the relevant passages, extracting facts, translating and producing readable output. It exists because the vast majority of the information organisations hold is text written for people, rather than structured for machines.
Do I still need to learn classical NLP if large language models exist?
Yes, for production work. Large models do most things acceptably with no training data at all, which is perfect for prototyping, but at high volume, classical and small fine-tuned models are dramatically cheaper, faster and more predictable. Real systems typically use both.
What are embeddings and why do they matter?
Embeddings turn text into vectors such that similar meanings end up close together. They are the basis of semantic search, clustering, deduplication and retrieval-augmented generation. Their Achilles' heel is exact tokens (part numbers, codes, acronyms), which is why production search combines them with keyword matching.
What is tokenization?
Splitting text into the fragments that a model actually processes — roughly word-pieces. It explains why models are priced and constrained by token rather than by word, why rare or technical terms use up more context, and why some languages are more expensive to process than others.
How do I evaluate an NLP system?
Write the test set before you build. For classification, use precision and recall, and work out which errors matter most. For generation, define what the output must contain and must never contain, rather than relying on similarity scores. For anything retrieval-based, check that claims trace back to a real source.
What is the most common beginner mistake?
Using an expensive general-purpose model for a high-volume task that a simple classifier would handle at a tiny fraction of the cost — or the reverse, hand-crafting brittle rules for something a model can do in a single call. Match the method to the volume, latency and accuracy you actually need, and you'll avoid both pitfalls.
Start here

Have a language problem worth solving? Let’s pressure-test it — free.

A 30-minute call with the engineers who build these systems, not a salesperson. Bring your documents and your hardest question and you will get an honest feasibility read and a clear next step.