Background and Problems
When developers first encounter large language models, they often treat them as a smarter text completion API. However, when applying them to question answering, summarization, code generation, or enterprise knowledge bases, three common issues arise: Why does the model give irrelevant answers? Why is information in the latter part of a long document ignored? Why can the same question produce very different results when phrased differently?
These questions usually come back to three foundational concepts: Transformer, context window, and prompt. The Transformer determines how a model understands and generates text; the context window determines how much content the model can see at once; the prompt determines what information you provide to the model and in what structure. Understanding these three concepts is the first step from calling an API to building stable AI applications.
Core Concepts
Transformer: Understanding Sequences with Attention
According to public sources, the Transformer was proposed by Vaswani and others in the 2017 paper Attention Is All You Need. Unlike traditional RNNs that process words one by one, the core of a Transformer is self-attention: when processing a word, the model can simultaneously attend to information from other positions in the sentence and calculate the strength of their relationships.
An engineering analogy may help: if a sentence is viewed as a set of service calls, self-attention is like dynamically querying the relevance of other nodes each time a node is processed, then aggregating the results by weight. A typical pipeline includes converting text into word vectors, adding positional encodings, computing Query, Key, and Value, obtaining attention scores through scaled dot products, and aggregating them into a new representation.
Modern large language models commonly use Encoder, Decoder, or Decoder-only architectures. According to public sources, BERT-style models lean toward Encoder representations, while GPT-style generative models usually use a Decoder-only architecture. Application developers do not need to implement a model from scratch, but they should know that models do not retrieve text word by word; they predict based on contextual probability distributions.
Context Window: The Range of Tokens a Model Can See at Once
A context window usually refers to the maximum number of tokens a model can process in a single inference. A token does not necessarily correspond to a Chinese character or an English word; it may be split into subwords or character fragments by the tokenizer. A larger window allows the model to reference more history at the same time, but it does not mean unlimited memory.
Context windows introduce three engineering constraints:
- Capacity limits: The input, system prompt, conversation history, and reserved output all consume tokens.
- Attention cost: Longer windows usually increase inference latency and memory usage.
- Information decay: Even when the window is long enough, the model may pay insufficient attention to middle or later content, so structured formatting is needed.
Therefore, long-text applications should not simply stuff all materials into the prompt. Instead, use summarization, retrieval, chunking, and ranking.
Prompts: Turning Tasks into an Executable Input Protocol
A prompt is not a magic spell; it is more like API documentation. A good prompt should clearly define the task objective, input materials, output format, constraints, and examples. For complex tasks, it can also include role setting, reasoning-step requirements, or refusal policies.
For example, when asking a model to summarize a technical document, instead of writing Help me summarize it, provide structured fields such as background, solution, risks, and conclusion. This makes the output more stable and easier for programs to parse.
Practical Steps or Checklist
- Step 1: Define the task type. Determine whether it is classification, extraction, summarization, rewriting, question answering, or code generation. The clearer the task, the easier it is to design the prompt.
- Step 2: Estimate the context budget. Reserve tokens for the system prompt, user input, reference materials, and output to avoid exceeding the limit.
- Step 3: Organize the input structure. Use headings, numbering, lists, and separators, and place key information in prominent positions.
- Step 4: Provide minimal but sufficient examples. If the output format is strict, one to three examples are often more effective than lengthy explanations.
- Step 5: Ask the model to indicate uncertainty. For example, ask it to state what information is missing if the materials are insufficient. This reduces hallucination risk.
- Step 6: Run regression tests. Prepare a set of typical questions, edge cases, and abnormal inputs, and continuously compare different prompt versions.
A simple example is to construct a list of conversation messages, separating the system instruction, user question, and context materials:
messages = [{'role': 'system', 'content': 'You are a technical documentation editor and answer only based on the provided materials'}, {'role': 'user', 'content': 'Materials: ... Question: What is a context window?'}]This code is not a complete API call; it illustrates that in engineering practice, roles, rules, and questions should be passed in layers rather than mixed into a single block of natural language.
Common Pitfalls and Suggestions
- Treating a long window as unlimited memory. A larger window only means more tokens can be input; it does not mean every detail will be used equally. For long documents, add a table of contents, summaries, and retrieval augmentation.
- Over-relying on the model for prompt compliance. If JSON output is required, specify field names, types, and null handling, and validate with post-processing when necessary.
- Ignoring token-counting differences. Chinese, English, code, and punctuation are tokenized differently. Do not estimate by character count alone; follow the tokenizer or official documentation of the model you use.
- Frequently changing prompts without an evaluation set. Without a fixed test set, it is hard to know whether an optimization is a real improvement or random fluctuation.
- Treating model output as a source of truth. Large language models may generate plausible but incorrect content. In critical scenarios, combine them with authoritative data, source citations, or human review.
Adopt a prompts as code approach: version control, comments, A/B testing, and turning failure cases into regression tests.
Further Reading Directions
- The original Transformer paper Attention Is All You Need and illustrated explanations can help you understand self-attention, multi-head attention, and positional encoding.
- Tokenizer and token-counting tools help you understand context windows and estimate costs.
- RAG, or retrieval-augmented generation, and vector databases are suitable for long documents, enterprise knowledge bases, and real-time data.
- Prompt evaluation and automated optimization, such as fixed datasets, metric scoring, and prompt version management.
- Basics of model safety and alignment, including prompt injection, sensitive information leakage, and output content moderation.
Overall, building large language model applications is not magic. If you understand how Transformers model sequences, respect the engineering limits of context windows, and use structured prompts to define tasks clearly, output quality usually improves significantly. For developers, the stronger these foundations are, the fewer pitfalls they will encounter when building Agents, RAG, and automated workflows.
References
Disclaimer: This article is compiled from publicly available online sources for educational purposes only and does not represent the official position of Guanshan Academy. If you believe your rights have been infringed, please contact us for removal.
Contact: chenxj.g@gmail.com
