Abdur Rafay AI Engineer

Available for freelance & full-time roles

I build voice agents and the infrastructure behind them.

I'm Abdur Rafay, Member of Technical Staff at Lythe Labs (San Francisco, remote). I ship production voice agents against a sub‑800 ms time-to-first-audio budget: streaming ASR → LLM → TTS, client-side VAD, barge-in, fully async. I own the CI/CD and AWS stack underneath them too. 25+ agent systems delivered across healthcare, finance, cybersecurity & education.

Abdur Rafay
Islamabad, Pakistan UTC+5 · replies within 24h
Now
MTS, Lythe Labs (SF)
Focus
Voice agents · Agent infrastructure · RAG
Also owns
CI/CD · AWS · GCP
Studying
BS Artificial Intelligence
01 / About

I ship the agent and the pipeline that deploys it.

Most AI work fails somewhere other than the prompt: in retrieval quality, in a blocking call that adds half a second to a phone conversation, in the deploy that nobody automated. I work on that whole surface, not just the model call.

At Lythe Labs I build production voice agents: real-time pipelines that stream ASR → LLM → TTS over a WebSocket so audio starts playing before the sentence is finished, client-side VAD that decides when a caller has actually stopped talking and cuts playback the moment they start speaking over it, and the tool-calling agent infrastructure underneath. The open-source Voice Agent Starter Kit came out of that work. I also own the parts nobody volunteers for: CI/CD with automated test integration, the AWS production stack, MCP server development, and test plans for telephony-side tooling.

Before this I delivered 25+ agent systems end-to-end across healthcare, finance, cybersecurity and education, and built RAG pipelines in production. I'm completing a BS in Artificial Intelligence at Bahria University alongside the work, and I publish the tools I need but can't find: an MCP latency auditor, a dependency-free agent framework, an LLM cache.

What I take on

  • Real-time voice engineering

    Streaming ASR → LLM → TTS pipelines, WebSocket audio transport, VAD, endpointing and barge-in, latency budgets and telemetry.

  • Agent infrastructure

    Tool-calling and multi-agent orchestration, MCP servers, memory and retrieval layers, evaluation harnesses.

  • Cloud & DevOps

    CI/CD pipelines with automated tests, AWS and GCP production stacks, Docker, release and deploy ownership.

  • Backend delivery

    FastAPI and Flask services, PostgreSQL, MCP servers, and the API surface agents actually call.

  • QA & reliability

    Test plans, pytest and API testing, regression suites in CI, defect triage through Jira and Linear.

02 / Skills

The toolkit, grouped by what it's for.

If it is on this list, it has shipped in something real.

Languages

  • PythonAgents, backend APIs, automation, data pipelines
  • SQLComplex queries, schema design, optimisation
  • JavaScriptFrontend scripting, async patterns, Node tooling
  • C++Systems fundamentals

AI, LLMs & agents

  • LangChainAgentic pipelines, tool calling, agent-to-agent coordination
  • LLM APIsOpenAI, Anthropic Claude, Groq, Google Gemini
  • Voice agentsStreaming ASR → LLM → TTS, WebSocket transport, VAD and barge-in, latency budgets
  • RAGEmbedding pipelines, hybrid retrieval, relevance tuning
  • MCPModel Context Protocol server development
  • TensorFlow & PyTorchCNNs, deep learning training loops
  • Scikit-learn & XGBoostRegression, classification, clustering, recommenders
  • Hugging FaceSentence Transformers, embeddings, fine-tuning

Cloud & DevOps

  • AWSProduction stack ownership at Lythe Labs
  • Google CloudDeployment and managed services for AI workloads
  • CI/CDPipelines with automated test integration, end to end
  • Docker & GitContainerised services, branching and release workflow

Backend & data

  • FastAPI & FlaskAsync REST services for agent workloads
  • PostgreSQL & MySQLSchema design, session and metadata storage
  • ChromaDB & SQLiteVector and local memory stores for agents
  • Pandas & NumPyAnalysis, feature work, reporting pipelines
  • Streamlit & Looker StudioDashboards and rapid data app delivery

QA & testing

  • pytest & API testingTest plans and suites for pipelines and agents
  • Regression in CIAutomated test integration on every change
  • Jira & LinearIssue tracking and defect triage across teams
03 / Selected work

Four problems worth explaining.

Every entry links to real source. One of them you can pip install.

  1. 01 Open source

    Voice Agent Starter Kit Lythe Labs

    A complete real-time voice loop, microphone to spoken reply, with the latency work already done and the ASR, LLM and TTS providers left swappable.

    Role
    Designed and shipped it. Pipeline, transport, adapters, telemetry.
    Problem
    A phone conversation tolerates well under a second of silence before it stops feeling like a conversation, and real callers interrupt. Every new voice project was rebuilding the same latency-critical scaffolding to get there: audio transport, turn detection, barge-in, streaming across three providers that each want to be awaited differently.
    Approach
    Stream everything. Audio moves over a WebSocket, the LLM emits text deltas, and TTS chunks start playing before the sentence is finished, so time-to-first-audio stops depending on total response length. Client-side VAD decides when the caller has actually stopped rather than waiting on a fixed timeout, and doubles as barge-in: the moment the caller starts speaking over the agent, queued audio is cut and playback stops, so the agent yields the turn instead of talking over the person it is meant to be listening to. The whole path stays async so no provider call can block another, and every stage is instrumented so a regression shows up as a number instead of a complaint.
    Outcome
    Open-sourced and Dockerised. New voice projects start from a working loop with a latency summary endpoint, instead of from an empty main.py.
    • FastAPI
    • WebSockets
    • Deepgram
    • Groq
    • Cartesia
    • VAD + barge-in
    • asyncio
    • Docker
  2. 02 On PyPI

    Relay Arclat

    An MCP server that audits Python voice-agent codebases for the async mistakes that quietly add latency to a phone call.

    Role
    Author. AST analysis, MCP server, packaging and release.
    Problem
    Voice agents live or die on response time, and the worst offenders are invisible in review: one blocking call inside an async path, awaits that should have run in parallel, a tool chain doing three round trips where one would do.
    Approach
    Static analysis at the AST level rather than runtime profiling, exposed as an MCP server so the audit runs inside the assistant's own tool loop, so the agent can inspect the codebase it's working on.
    Outcome
    Published to PyPI as relay-arclat. Latency review becomes an automated pass instead of a careful read-through.
    • Python
    • MCP
    • AST analysis
    • asyncio
    • PyPI
  3. 03 Open source

    Crucible

    A multi-agent framework that runs entirely on your machine: hybrid retrieval over SQLite, local models through Ollama, zero core dependencies.

    Role
    Author. Framework design, memory layer, retrieval.
    Problem
    Most agent frameworks assume a hosted vector database and a deep dependency tree. That's fine for a demo and painful for anything that needs to be reproducible, private, or still working in six months.
    Approach
    Memory in plain SQLite with hybrid BM25 + embedding retrieval, so keyword and semantic matches both land; local model support via Ollama; and a hard rule of no core dependencies.
    Outcome
    Agent workflows that are self-hosted end to end and reproducible. No external services to sign up for, nothing to drift out from under the run.
    • Python
    • SQLite
    • BM25 + embeddings
    • Ollama
    • Multi-agent
  4. 04 Open source

    Agent Testing Suite Lythe Labs

    An evaluation dashboard that makes multi-agent pipelines legible: where the time goes, where the money goes, which tools actually get called.

    Role
    Built at Lythe Labs. Instrumentation and dashboard.
    Problem
    Multi-agent systems are opaque. When a pipeline is slow or expensive, there's rarely evidence pointing at which agent or tool call is responsible, so tuning turns into guesswork.
    Approach
    Instrument every run for latency, API cost and tool-call frequency, then render the pipeline as an interactive flow so a bottleneck is something you can see rather than suspect.
    Outcome
    Agent tuning becomes data-driven: regressions surface as numbers, and optimisation targets the step that's actually costing.
    • Python
    • LangChain
    • Streamlit
    • Observability

Also built

04 / Experience

Six roles, five countries, one throughline.

Every role has come down to the same thing: make the model useful, then keep it running.

  1. Mar 2026 - PresentSan Francisco · Remote

    Member of Technical Staff Lythe Labs

    Building and shipping production voice agents, multi-agent systems, and the infrastructure that runs them: system design through deployment, solo, across a distributed remote team.

    • Designed and shipped the open-source Voice Agent Starter Kit: a streaming ASR → LLM → TTS loop over WebSockets with client-side VAD, barge-in and built-in latency telemetry, cutting voice agent integration time on new projects.
    • Built real-time voice pipelines and tool-calling agent infrastructure against a sub-800 ms time-to-first-audio budget, keeping the whole path async so no provider call blocks another.
    • Engineered a long-running autonomous agent supporting weeks of continuous runtime, using LLM planning, sandboxed execution, persistent memory, resumable sessions.
    • Own CI/CD infrastructure with automated test integration, the AWS production stack, and MCP server development end-to-end.
    • Maintain test plans for voice pipelines and agent systems; triage failures across telephony tooling and LLM integrations via Jira and Linear.
    • Voice agents
    • LangChain
    • FastAPI
    • AWS
    • CI/CD
    • MCP
  2. Jan - Mar 2026Georgia, USA · Remote

    AI Intern School of AI

    Delivered 25+ AI agent systems independently across healthcare, finance, cybersecurity and education, each end-to-end with minimal supervision.

    • Built full-stack multi-agent platforms with persistent memory, voice control, workflow automation and document analysis, cutting manual document processing time by an estimated 70% in pilot tests.
    • Implemented agent-to-agent communication, context-aware task coordination and structured LLM outputs in LangChain.
    • Shipped a medical note structurer (100% of unformatted clinical notes converted to standard templates), an earnings call analyzer, an incident response assistant, an AI tutoring platform and a collaborative document analyzer.
    • Reported system designs and progress weekly to cross-functional leads across three time zones.
    • LangChain
    • Multi-agent
    • Python
  3. Nov 2025 - Jan 2026Lahore, Pakistan

    AI Intern AABS

    Built and optimised production RAG pipelines on Groq and ChromaDB, holding low latency at scale without losing retrieval accuracy.

    • Improved retrieval precision by 40% through embedding pipeline tuning and relevance optimisation.
    • Developed PDF chatbots and memory-augmented conversational agents; integrated PostgreSQL for session and metadata storage in multi-agent MLOps workflows.
    • Applied user-intent classification and multi-agent orchestration to route queries accurately, reducing irrelevant LLM responses.
    • Groq
    • ChromaDB
    • PostgreSQL
    • RAG
  4. Sep 2025Cairo, Egypt · Remote

    Machine Learning Intern Elevvo Pathways

    Six distinct ML projects delivered independently in one month, each through the full lifecycle: cleaning, feature engineering, training, evaluation and visualisation.

    • Linear and logistic regression, K-Means clustering, CNNs, XGBoost, and a collaborative-filtering recommender.
    • Scikit-learn
    • XGBoost
    • TensorFlow
  5. Aug - Sep 2025Dubai, UAE · Remote

    Data Analyst Intern Excelerate

    Led a team of four interns analysing social media marketing data.

    • Identified three underperforming ad campaigns and presented cost-reduction recommendations that the finance team adopted.
    • Built interactive Looker Studio dashboards on Pandas and NumPy pipelines, with SQL extracting and aggregating campaign data across sources.
    • SQL
    • Pandas
    • Looker Studio
  6. Sep - Nov 2024Islamabad, Pakistan

    Data Science Intern Securely Innovations

    Data work and QA for a security analytics team.

    • Preprocessed large structured and unstructured datasets in Pandas, NumPy and SQL; automated reporting workflows that cut manual processing time by 30%.
    • Applied statistical modelling and data mining to surface actionable patterns for analysts.
    • Ran functional, regression and mobile-responsiveness QA across the company's main web platform, tracking defects in Jira.
    • Pandas
    • SQL
    • QA
    • Jira
05 / Credentials

Education & certifications.

Education

BS Artificial Intelligence

Bahria University, Islamabad · 2024 - Present

AI fundamentals through to deep learning, NLP, computer vision and intelligent systems design, studied alongside full-time engineering work.

Certifications

  • Generative AI with Large Language Models

    AWS · DeepLearning.AI

    Transformer architecture, model selection, fine-tuning and RLHF, and the deployment considerations behind LLM-powered applications.

  • AI Infrastructure & Operations Fundamentals

    NVIDIAJul 15, 2025

    Concepts, tooling and practice for deploying, managing and operating AI systems at scale, covering the infrastructure side of supporting AI workloads.

    Verify
  • Supervised Machine Learning: Regression & Classification

    Stanford · DeepLearning.AI

    Foundations of supervised learning: linear and logistic regression, gradient descent, regularisation and model evaluation.

  • AI Developer Professional Certificate

    IBMAug 20, 2025

    The full journey of AI-powered software development: software engineering and generative AI foundations through to shipping applications with Python and Flask.

    Verify
06 / Writing

Notes from the work.

07 / Contact

Let's build something that ships.

Open to full-time AI engineering roles and freelance work: real-time voice agents, agent infrastructure, RAG pipelines, and the cloud stack to run them. Tell me what you're building.

abdurrafay.tech@gmail.com

Currently available · replies within 24 hours

Location
Islamabad, Pakistan (UTC+5)
Résumé
View CV