Support bots love long context until the GPU runs out of memory mid-shift. Choosing a default context is a product decision as much as a technical one.
Start from tickets, not from marketing maxima
Sample real tickets: how much history, how many policy paragraphs, how long the user paste tends to be. Design the prompt budget around the eightieth percentile case, not the worst PDF anyone has ever attached.
Techniques that save context
- Retrieve only the top few knowledge chunks instead of whole manuals.
- Summarize older chat turns when a thread grows.
- Ask users to paste the error block, not the entire log file, when possible.
- Keep system instructions short and stable.
Hardware loop
Pick a context, load the model, run a realistic busy-hour script, watch memory. If you are constantly near the ceiling, reduce context or quantize before buying hardware. Use the VRAM table and VRAM guide while planning, and keep eval discipline via methodology. Tutorial entry points live under tutorials.
Rollout note
Publish the chosen limit to agents so they know why the bot truncates. A known limit with retrieval beats an advertised huge window that fails during peak. Revisit the number quarterly as ticket patterns change; write the decision down so the next maintainer does not rediscover it by outage.
Extended product framing
Agents should know the bot summarizes older turns after a set number of messages. Put that in the internal FAQ. If leadership demands unlimited memory, respond with the hardware graph showing diminishing returns and stability risk. Offer retrieval from ticket history as the scalable alternative. Product pressure often pushes context up; engineering must bring the cost curve to the same meeting, calmly and with numbers from your own box rather than internet anecdotes. Keep a one-page decision record with the date, the measured peak memory, and the owner of the next review.
Published by MicroAI One (本地AI 1号网), a Shenzhen team working on local LLM hardware, software and deployment.