Technical approach
Measured, documented, and built to be handed over.
For technical teams who want the detail: how we choose models, size hardware, report performance and secure the deployments we build.
Performance reporting
No figure without its conditions.
Throughput and latency depend heavily on hardware, precision, prompt length and load. A number without its test conditions is not useful for planning, so every figure we report includes the following.
Customer-specific requirements are validated on representative workloads before we commit to a configuration.
- Model & version
- Exact model, revision and licence
- Precision
- FP16 / BF16, FP8, INT8, INT4 or other quantisation
- Hardware
- GPU type and count, CPU, memory, interconnect
- Serving stack
- Inference server, version and key settings
- Workload shape
- Prompt length, output length, context size
- Load profile
- Concurrent requests and arrival pattern
- Metrics
- Time to first token, end-to-end latency (median and p95), tokens/s, error rate
- Tooling
- Load-testing tool and test duration
Sizing
How we size infrastructure
Memory first
Model weights, KV cache for the target context length and concurrency, and overhead determine the minimum GPU memory, and therefore the GPU type and count.
Then throughput
We load-test candidate configurations at your expected concurrency and prompt sizes to confirm response times stay within agreed limits.
Then cost
Each viable configuration is priced against your provider and region, so you can weigh performance headroom against monthly running cost.
Technology
Tools we work with
We choose components per engagement and prefer widely adopted open-source tooling your team can support.
- Serving
- vLLM, Hugging Face TGI, NVIDIA Triton / TensorRT-LLM, llama.cpp, or similar, chosen per workload
- Models
- Open-weight LLMs, embedding models, re-rankers, vision, speech and classical ML models
- Retrieval
- pgvector, Qdrant, OpenSearch or similar; hybrid keyword + vector search
- Infrastructure
- AWS, Azure, Google Cloud, GPU clouds, on-premise Linux with NVIDIA GPUs; Kubernetes or VM-based
- Observability
- Prometheus, Grafana, OpenTelemetry, or your existing tooling
- Delivery
- Infrastructure as code (e.g. Terraform), containerised services, version-controlled configuration
Security
Controls, not promises.
Running a model privately keeps data within your environment, but security and compliance still depend on how the system is configured and operated. We implement and document these controls; responsibility for your compliance assessment remains with your organisation.
- Deployment inside private networks / VPCs, with no public endpoint unless required
- TLS on all endpoints, and authentication on every API
- Least-privilege cloud IAM roles for services and operators
- Encryption at rest for model storage, vector stores and logs
- Configurable prompt and response logging, including the option to log nothing
- Audit of who can access what, documented at handover
Ready to talk through your requirements?
Tell us the use case, where it should run and how many people will use it. We will take it from there.