A language model on your hardware, tested against a real task.
You get a model chosen for your workload and your hardware, running in your environment, and connected to a bounded task it can be evaluated against — with approved documents, access rules, and review points defined before launch.
What stays inside your environment.
- Model executionPrompts, documents, and generated answers run on your hardware.
- Retrieval, not trainingApproved excerpts are supplied at request time; model weights stay unchanged.
- Approved sourcesOnly current, authorized documents can be retrieved.
- Boundary crossingsUpdates, authentication, and diagnostics cross the dashed line — each is designed and tested.
- Review recordsLogs and evaluation results stay inside the line for inspection.
Match the model to a task.
Consider local AI when running the model on hardware in your environment is a requirement and you need to establish what that environment can support. Begin with a bounded task, such as finding an approved procedure or drafting a response for review, rather than a general request to add AI.
Bring representative questions, suitable sample material, expected user numbers, and an inventory of available hardware. Model size alone does not determine suitability: memory, response length, simultaneous requests, software compatibility, and the quality of answers all affect the choice.
Design the information path.
Find the current procedure, with a source to inspect
Imagine an internal assistant helping a coordinator locate the procedure for returning equipment. It searches approved documents, supplies relevant excerpts to the model, and asks for an answer with source references. The coordinator can inspect the document before acting.
Now suppose the collection contains an obsolete return address. The design must identify which version is approved and exclude superseded material. If the current source does not answer the question, the useful response is to identify the gap and direct the coordinator to the document owner. A fluent guess would fail the evaluation.
Example retrieval boundary
- Question
- Where should this equipment be returned?
- Allowed evidence
- Current, approved return procedure
- Excluded material
- Superseded versions and unauthorized documents
- Reviewable result
- Answer linked to its source, or an explicit information gap
Define the output. Check the behavior.
Representative deliverables
- Model comparison, hardware requirements, and a documented deployment configuration.
- An agreed user interface or application connection, with approved source and access rules.
- Evaluation examples, results, known limitations, and an operating guide.
Acceptance to agree
- Check representative answers, source relevance, missing evidence, and conflicting documents.
- Exercise unauthorized access, unavailable sources, and restart or recovery behavior.
- Measure response times and resource use at an agreed simultaneous workload.
Prepare for operation.
Assign responsibility for the hardware, operating system, model runtime, application access, and source collection. Agree what is logged, where it is stored, who can inspect it, and how long it is retained. These decisions include prompts and generated responses.
Document how to update and restore the deployment. Model changes, prompt changes, and refreshed documents can alter behavior; keep a repeatable evaluation set for reviewing them. Ongoing monitoring and maintenance are defined in the engagement.
Model, hardware, and information.
Choose the model and hardware together
Evaluate candidate models on the intended tasks within the available memory and compute budget. Smaller or compressed models may fit more readily but change answer quality or speed. Measure the actual configuration before committing to additional hardware.
Define every connection
Map how users authenticate, where the application runs, and how it reaches the model and document sources. Include update downloads, diagnostics, and external tools in the review. Keeping model execution local does not establish the path taken by every piece of data.
Separate knowledge from instructions
A prompt instructs the model. Retrieval supplies selected source material at request time; it does not train model weights. Workflow integration connects the result to an application or task. Each needs its own rules for access, inputs, and failure behavior.
Decide when a person must review
An answer with a citation can still misread its source. Agree which outputs need checking and prevent an uncertain response from becoming an automatic operational action. Access restrictions must apply to retrieved documents as well as the visible interface.
Before we begin.
Can this run without an internet connection?
That depends on the complete workflow. Model execution, authentication, source retrieval, and application dependencies must all be available inside the intended boundary. Test disconnected operation if it is required.
Do our documents need to train the model?
No. Retrieval can provide relevant excerpts without changing model weights. Training or fine-tuning is a separate decision requiring suitable data, permissions, and evidence that it addresses a limitation the simpler approach cannot.
Can we use hardware we already own?
Start with its specifications and the expected workload. Evaluation establishes whether it is suitable, whether a smaller model is acceptable, or whether a hardware change is justified.
What would you like AI to do in your environment?
Describe the task, the hardware available, and any connectivity requirement. Mention the information the model needs to use; send representative non-sensitive examples after the appropriate sharing method is agreed.