AI Engineer Interview Questions

The decision comes down to data access patterns, latency requirements, and the nature of the knowledge the system needs. If the task is generative and the base model already has the domain knowledge baked in, prompt engineering with a hosted model like GPT-4o or Claude is usually the fastest path to value. Fine-tuning makes sense when you have high-quality labelled examples of the exact output format you want and need consistent stylistic control, or when latency and cost require a smaller, specialised model. RAG is the right choice when the system needs access to recent, proprietary, or frequently changing information that the base model cannot have seen in training. In practice, many production systems combine all three: a fine-tuned model for tone and format, retrieval for live knowledge, and careful prompt design to wire them together.

I would start with the retrieval layer. For a support use case, the knowledge base typically includes product documentation, past resolved tickets, and policy documents. I would chunk the content carefully, aiming for 300 to 500 token chunks with meaningful overlap, and embed each chunk using a model like text-embedding-3-large from OpenAI or a hosted alternative. I would store embeddings in a vector database such as Pinecone or pgvector, and add metadata filters for product category and language to narrow retrieval before the semantic search runs. On the generation side, I would use a two-stage approach: first retrieve the top five chunks by cosine similarity, re-rank them with a cross-encoder, then pass the top three into the final generation prompt with clear instructions to cite the source chunk. I would also add a query classification step to route obvious FAQ questions directly without retrieval, reducing latency and cost for the majority of queries.

Evaluation for LLM systems has to happen at two levels. The first is offline evaluation against a fixed dataset of inputs and expected outputs, which I build before I build the feature. I create a test set of at least 50 representative cases covering typical inputs, edge cases, and known failure modes. I score outputs using a combination of automated metrics (BLEU or ROUGE for extractive tasks, LLM-as-judge scoring for open-ended generation) and human review of a random sample. The second level is online evaluation in production via user signals: thumb ratings, edit rates if users can modify AI output, escalation rates for support use cases, and session abandonment. I instrument these signals from day one because offline evals never perfectly predict user experience. I also set up a latency and cost dashboard, since a feature that degrades because a model API call starts timing out will not appear in quality metrics at all.

Prompt injection is one of the more serious risks in any system where user input is incorporated into a prompt. My standard approach is a layered defence. At the input layer, I validate and sanitise user inputs before they reach the prompt, flagging strings that contain instruction-like patterns such as "ignore previous instructions" or role-switching language. In the prompt itself, I use strict role framing and XML-like delimiters to separate trusted system content from untrusted user content. At the output layer, I apply output validation to catch responses that deviate from the expected schema or contain suspicious content before they reach the user. For higher-risk applications I also log all inputs and outputs for audit, and I run a periodic red-teaming exercise against the production prompt to surface new injection vectors. None of these defences are perfect in isolation, which is why the layered approach matters.

I built a document summarisation feature for an internal knowledge management tool. In testing it performed well, producing accurate, concise summaries across the sample documents we used. After launch, users reported that summaries of long legal contracts were missing key clauses. The root cause was a chunking strategy that split documents at fixed token boundaries rather than at semantic boundaries like section headings, so the retrieval step was feeding the model partial clauses without context. I rewrote the chunker to respect document structure, using headings and numbered list patterns in the XML source to create logical chunks rather than fixed-size ones. Post-fix, the missing clause rate dropped from roughly 18% of documents to under 3%. The lesson was that chunking strategy is not a footnote in a RAG system: it is as important as the model choice.

The product team at my previous company wanted our AI assistant to answer questions about competitor pricing, using information scraped from public websites. I had concerns about accuracy because competitor pricing changes frequently, and about the reputational risk of the assistant confidently stating figures that were out of date. I put together a short risk assessment covering the accuracy half-life of the data, the support ticket volume we would likely see when users acted on stale prices, and the legal risk of quoting competitor figures in certain regulated markets. The product team accepted the accuracy argument but still wanted some competitor awareness in the product. We agreed on a compromise: the assistant would acknowledge questions about competitors and redirect to a human for detailed comparisons. It was a better outcome than either of the original positions.

We were running all queries through GPT-4o and our monthly API spend had grown to a point where the unit economics did not work at the pricing tier we were targeting. I ran a query classification analysis on a 2,000-query sample and found that around 60% of queries were simple factual lookups that did not need a large model. I introduced a routing layer that sent simple queries to GPT-4o mini and reserved the full model for complex multi-step reasoning tasks and edge cases flagged by a low confidence score. I validated the routing against a human-labelled test set before deploying. The result was a 54% reduction in API cost with quality scores on the evaluation set dropping by less than 1%. The routing logic became a template used across three other AI features in the product.

The key is to separate the concerns of preprocessing, model inference, and postprocessing into distinct, independently scalable stages. For preprocessing I keep transformations stateless so they can run in parallel across request batches. For inference I use an optimised serving framework such as vLLM or TGI for LLMs, which handles continuous batching and KV-cache reuse to dramatically increase throughput compared to a naive HTTP wrapper around a model. I also quantise models where possible: moving from FP16 to INT8 with GPTQ typically halves memory and improves throughput with minimal quality loss for most tasks. I set up request queuing with priority lanes so that interactive user-facing requests are not blocked by batch jobs. Finally I add response caching for deterministic or near-deterministic queries using a semantic cache layer, which reduces both latency and cost for repeated similar inputs.

LLM monitoring differs from classical ML monitoring because you cannot rely on a single numerical metric. I track four categories of signals. First, infrastructure metrics: latency percentiles (p50, p95, p99), error rates, token counts per request, and API spend by model and feature. Second, output quality signals: automated LLM-as-judge scores on a sample of production outputs, flagging scores below a threshold for human review, alongside user feedback like thumbs ratings. Third, data drift indicators: changes in the distribution of input query types, which can reveal a shift in how users engage with the product before it shows in quality scores. Fourth, safety signals: the rate at which the output filter or content moderation layer triggers, segmented by input category. I route all of these into a Grafana or Datadog dashboard and set alerts on metrics that have historically preceded user complaints.

I start by building the training dataset carefully, because dataset quality matters more than the fine-tuning technique for most tasks. I aim for at least 500 high-quality examples in the instruction-following format the base model expects, with a 90/10 train/eval split. I use LoRA or QLoRA for parameter-efficient fine-tuning, which lets me fine-tune a 7B model on a single A100 without full-weight updates, significantly reducing compute cost. I run the fine-tune with Axolotl or the Hugging Face Trainer, tracking training loss and eval loss per epoch to catch overfitting early. After training, I evaluate the checkpoint against the held-out eval set using task-specific metrics such as F1 for classification or ROUGE-L for summarisation, and compare to the untuned baseline. I also run the checkpoint against my safety evaluation set to confirm fine-tuning has not degraded refusals on harmful inputs, which is a common side effect of domain fine-tuning.