Sumanth G.

LLM Inference Engine

Continuous Batching from scratch

A local-first LLM serving engine focused on efficient inference, concurrent scheduling, and real-time streaming.

• PagedAttention-inspired KV-cache management with continuous batching — serves 8 concurrent requests while cutting memory fragmentation.
• Adaptive chunked-prefill scheduler that adjusts chunk size to workload pressure, reducing time-to-first-token by 61% on mixed prompts.
• Trie-based prefix KV cache with LRU eviction & shared prompt reuse — +44% throughput vs sequential HuggingFace inference.

• Python
• PyTorch
• FastAPI
• Transformers
• Trie/Radix Cache

Agentic Investment Intelligence Platform

A microservices RAG platform that answers financial questions with AI-backed market & news context.

• Agentic query router (Qwen-3.5-8B via vLLM) that classifies queries and dispatches between Text-to-SQL and RAG workflows.
• Hybrid retrieval + cross-encoder reranking over ChromaDB embeddings and PostgreSQL data — 76% top-k accuracy on a 50-company NIFTY benchmark.
• Semantic caching layer cut repeated-query latency from seconds to milliseconds; Dockerized microservices with parallel news ingestion.

• Python
• FastAPI
• vLLM
• PostgreSQL
• ChromaDB
• Docker

LLM Quantization Lab

A quantization toolkit for compressing LLMs with measurable trade-offs.

• Implements absmax and zero-point quantization from first principles.
• GPTQ support with GGUF file parsing.
• Benchmarking dashboard to compare accuracy vs size/speed across methods.

• Python
• PyTorch
• GPTQ
• GGUF
• Quantization

RL-Driven Auto-Scaling Orchestrator

A cloud orchestrator that learns to scale heterogeneous microservices from real-time metrics.

• Uses reinforcement learning (PPO / DQN) to make dynamic scaling decisions.
• Reacts to real-time load and resource metrics across heterogeneous services.
• Aims to balance latency and cost without hand-tuned scaling rules.

• Python
• Reinforcement Learning
• PPO/DQN
• Docker