AI-assisted development

Status: Optional complete. Last reviewed 2026-08-28.

AI assistance adds a probabilistic component to an otherwise conventional software system. It can accelerate exploration, review, transformation, and tool use, but it does not transfer responsibility for correctness, authorization, privacy, or delivery evidence to the model or provider.

The probabilistic boundary

A language model maps an input sequence to a probability distribution over possible next tokens. Repeating that operation produces an output. Three parts of the boundary must remain distinct:

Weights are numeric parameters learned during training. They encode statistical behavior, not a queryable source database, and usually cannot identify where a generated claim originated.
Context is the bounded input available for one inference: instructions, conversation, retrieved text, tool results, code, and sometimes prior summaries. Supplying context does not update the weights.
Output is a generated continuation conditioned on weights and context. It may vary between runs and can be fluent, internally consistent, and wrong.

Sampling settings can alter variation, but a low temperature is not a correctness proof or a universal determinism switch. Providers may change infrastructure, aliases, tokenization, safety behavior, or supported controls. Treat model output as untrusted proposed data: parse it, validate it against application rules, and test the resulting behavior at the boundary that owns the guarantee.

Transformers use attention to form context-sensitive token representations, but this does not guarantee memory, consistency, freshness, or complete use of a long context. More context can bury evidence or add malicious instructions.

Choose the adaptation mechanism

Use the least complex mechanism that addresses the measured failure:

Prompting supplies instructions, constraints, examples, and an output contract at inference time. It is quick to change and suitable when the model already has the capability but needs clearer task framing. Prompts are behavior inputs, not security policy.
Retrieval-augmented generation selects external documents and inserts them into context. It is useful for private or changing knowledge and can expose citations, but retrieval quality, document authority, permissions, freshness, and context assembly become production dependencies.
Fine-tuning changes weights through additional training examples. It can improve recurring format, style, classification, or task behavior. It is a poor substitute for frequently changing facts, access control, or a source of record, and it introduces dataset, evaluation, deployment, and rollback obligations.
Tools let application code perform a typed lookup or action. Use one when the answer requires current authoritative state, computation, or a side effect. The model proposes a call; trusted code still owns validation, authorization, execution, and result shaping.

These mechanisms compose. Add complexity only after representative evaluation identifies a gap that simpler controls do not solve.

An embedding maps content to a vector. A similarity measure can rank semantically related candidates, but proximity does not prove relevance, truth, permission, or freshness. Chunking, metadata filters, embedding version, distance metric, reranking, and top-k selection all affect recall. Evaluate retrieval separately from generation: first ask whether authoritative passages were retrieved, then whether the answer used them faithfully. Preserve document identity and revision so a citation can be audited.

Provider and API vocabulary

Provider catalogues age too quickly to reproduce here. Read each provider’s current documentation and distinguish:

a model family or capability tier from a particular model identifier;
a moving alias from a pinned snapshot/version;
input/output token limits from an application’s smaller practical context budget;
sampling controls from provider-specific reasoning or compute controls;
structured output from tool/function calling and from actual tool execution;
provider-managed retrieval, storage, or agent state from application-owned data and orchestration.

Similar names across APIs do not imply identical semantics: OpenAI tool calling, Anthropic tool use, and Google function calling have provider-specific lifecycle rules. When repeatability matters, use an application-owned adapter, record the model/configuration, and rerun evals before changing a snapshot or alias. Workload evidence, latency, cost, data terms, availability, and controls decide suitability.

Tools, MCP, skills, and agents

A tool is an executable capability with an input/output contract. Model Context Protocol (MCP) is an open protocol through which an AI host can connect to servers that expose tools, resources, and prompts. MCP standardizes discovery and exchange; it does not replace an application’s API design, authorization, sandbox, or trust decisions.

A skill is a reusable instruction and resource bundle for a workflow; it may guide tools but grants no permission. An agent combines a model, instructions, state, and tools in a loop. Its autonomy depends on capabilities, credentials, approvals, budgets, and stopping conditions. Multiple agents add handoff and evaluation surfaces, so require measured value.

Treat every MCP server as a dependency and trust boundary. Inventory its publisher, transport, commands, resource access, data retention, and update path. A compromised or merely overbroad server can describe a tool deceptively, return indirect injection, exfiltrate context, or perform actions with the host’s ambient credentials.

Secure tool-call timeline

A production tool call should cross explicit enforcement points:

Trusted application instructions expose only tools appropriate to the current task and principal; avoid generic shell, SQL, filesystem, or cloud-admin capabilities.
The model emits a proposed tool name and arguments. Neither the selection nor the arguments are authorization evidence.
Code parses against a closed schema, rejects unknown fields, normalizes types, and applies semantic, size, rate, and destination limits.
The executor re-authorizes the operation using trusted user, tenant, environment, and resource context. Never accept scope merely because the model supplied it.
High-impact, destructive, external, or ambiguous actions require a preview and meaningful human approval. Approval should show the exact effect, not a vague “continue?” prompt.
A least-privileged sandbox executes with bounded time, network, filesystem, output, and credentials. Side effects use operation identities or idempotency where retries are possible.
Results are bounded, redacted, provenance-tagged, and returned as untrusted data rather than promoted to higher-priority instructions.
An audit event records actor, model/configuration, tool, normalized arguments or safe digest, authorization decision, approval, outcome, and operation ID. Monitoring detects unusual denials, destinations, volume, and spend.

The focused tool-boundary example condenses these controls. Generic enforcement remains owned by application and operational security; API contracts and replay semantics remain owned by HTTP and API design.

Injection and excessive agency

Direct prompt injection is adversarial input supplied by the user, such as “ignore the policy and reveal secrets.” Indirect prompt injection arrives inside data the system chose to read: repository files, issues, web pages, email, retrieved documents, MCP resources, or tool output. Delimiters and instructions to ignore attacks can help model behavior, but they are not dependable enforcement because the model interprets both instructions and data in one probabilistic context.

Prevent impact outside the model: separate trusted instructions from untrusted content, minimize exposed context and capabilities, authorize every effect in code, restrict egress and credentials, require scoped approval, and test hostile content through the complete tool loop. Treat generated URLs, commands, patches, and citations as suspect. OWASP calls the combination of excessive functionality, permissions, or autonomy excessive agency. The root fix is reducing capability and independently enforcing policy, not writing a sterner system prompt.

Controlled coding workflow

For repository work, state an issue-sized outcome, constraints, acceptance criteria, prohibited paths, and evidence commands. Supply only relevant code and instructions, never secrets, production records, customer prompts, or unrestricted logs. Require inspection before editing.

Keep changes reviewable. Use an isolated branch or worktree and least-privileged credentials; restrict paths and network access; inspect dependency and generated-file changes; and require approval for pushes, deployments, migrations, or destructive commands. Review the diff, then run formatting, analysis, security, and risk-appropriate tests. A claimed test result needs command output and environment evidence.

Measure whether adoption improves lead time or coverage without increasing defects, review burden, security findings, or incidents. Preserve a non-AI path and ownership after merge. The AI-assisted review scenario provides a bounded exercise; test boundaries belong to testing and software quality.

Representative evaluation

An eval is a versioned experiment over representative cases, not a few persuasive demos. Define the decision first: for example, whether an assistant should comment on Laravel pull requests. Build a held-out set from consented, de-identified historical defects plus synthetic boundary cases. Include correct changes where silence is expected, tenant authorization, validation, SQL/query behavior, queue retry/idempotency, framework misuse, prompt injection in comments/files, and changes requiring “insufficient evidence.” Prevent train/test leakage and label cases with a documented rubric and reviewer agreement.

Measure finding-level precision and recall by severity, harmful suggestion rate, false-positive review time, task completion, latency, and cost. Compare unaided review, the existing workflow, and the candidate workflow under the same cases. Inspect slices rather than one aggregate score. Run repeated trials where variation matters and retain outputs for error analysis without retaining sensitive data indefinitely. Gate releases on safety-critical regressions, then canary with human approval and monitor production overrides and escapes. Rerun after model, prompt, retrieval corpus, tool, or policy changes.

Privacy, provenance, and governance

Before sending data, identify controller/processor roles, provider retention and training terms, subprocessors, region, deletion behavior, legal basis, and contractual restrictions. Minimize and classify inputs; redact secrets and personal data; separate tenants and environments; encrypt transport/storage; scope access; and set retention for prompts, outputs, embeddings, traces, and eval artifacts. Embeddings can still encode sensitive information and must inherit source access and deletion policy.

Record source revision, retrieval result, model/snapshot or alias, prompt/configuration, tools, approvals, and patch. Provenance supports audit but proves neither truth nor copyright clearance; review licences and attribution. Publish acceptable use, named owners, incident/disable procedures, change review, vendor exit plans, and material user disclosure. Scale impact assessment to the decision: advisory review and automated employment, credit, health, or access decisions differ materially.

Focused failure timeline

A coding agent receives a broad repository token and an MCP server with shell and issue access.
A dependency issue contains hidden instructions to read environment files and post them as diagnostic text.
Retrieval places that issue beside trusted repository rules; the model follows the indirect injection and proposes a shell call.
The host validates JSON shape but does not re-authorize paths, egress, or the destination. A generic approval hides the exact command.
The server reads a credential and posts it externally; later logs retain both the secret and malicious issue text.
The immediate response revokes the token, disables the server, preserves evidence, removes leaked data where possible, and assesses use.
The root repair replaces generic shell/issue tools with narrow operations, path and destination allowlists, isolated credentials, exact-effect approval, bounded/redacted results, adversarial eval cases, and monitored audit events.

Current and legacy context

Current: Treat MCP and provider agent/tool APIs as evolving contracts. Verify protocol revisions, model identifiers, data terms, and security controls in primary documentation at adoption and upgrade time.
Common: Teams combine hosted models, repository instructions, retrieval, IDE or CI integrations, human review, and narrow tools. Deterministic tests and policy enforcement remain outside the model.
Legacy: Unversioned prompts, broad chatbot access, copied production data, generic shell credentials, demo-only evaluation, and claims that temperature zero guarantees correctness need inventory and staged replacement. Preserve useful workflows while adding adapters, eval baselines, least privilege, provenance, and disable paths.