-
[WIP] Benchmarking Local LLMs Against Coding Agent Harnesses
I’ve been running a small benchmark,
harness-bench, that pairs local LLMs (served viallama.cpp’sllama-server) with agent harnesses (Aider, Claude Code, OpenCode, Pi, Qwen CLI) on 16 software-engineering tasks across Python, PyTorch, JAX, C, C++, Rust, and SQL. Each(model, harness, task)cell is sandboxed: the agent only sees a scratchworkspace/and grading is done by a hiddentest.shthat the agent never sees. The current sweep is 17 model-quants × 5 harnesses × 16 tasks = 1360 runs on a single M3 Max / 128 GB laptop. …
-
Deep Research MCP
To make my life a bit easier, I built
deep-research-mcp, a small Python agent that exposes several “deep research” backends through a single Model Context Protocol server (Anthropic, 2024). It allows Claude Code, Codex, Gemini CLI, or any MCP client to fire off long-running research tasks against whichever backend the user prefers. …
-
Some Notes on Gradient Estimation
Assume we have a scalar function $f(x)$ of interest, such as a reward we want to maximise or a loss we want to minimise, and that $x$ is drawn from a distribution $p_{\theta}(x)$ parameterised by $\theta$. A natural quantity to study is the expected value of $f$ under this distribution, …
-
Real-World Impact of Our Research
Academic research sometimes risks to be disconnected from real-world applications. Research from our group demonstrated significant real-world impact across multiple domains, from improving the efficiency of LLM inference and training to new state-of-the-art evaluation protocols, and contributing to several industry products. …
-
March 2025 in Research
We have been working on language model evaluation, knowledge utilization, efficiency, and multimodal reasoning. We had papers at ICLR 2025, NAACL 2025 (x3), AAAI 2025, and others, along with several ongoing works. …
-
Postdoc Position in Multimodal Foundation Models
Amazing opportunity to join our team at the School of Informatics, University of Edinburgh! The School of Informatics is seeking a Postdoctoral Research Associate to work on evaluating and improving multimodal foundation models, with a particular focus on Vision-Language Models (VLMs). …
-
November 2024 in Research
My amazing collaborators will be presenting three papers at EMNLP 2024 (main track), a leading conference in natural language processing, happening in Miami later this month! A few weeks ago I also blogged about our ACL 2024, ICML 2024, and CoLM 2024 papers – you can check the post here. …
-
July 2024 in Research
My amazing collaborators will be presenting several works at ACL 2024, ICML 2024, and CoLM 2024 in the upcoming weeks/months! …
-
Looking for Postdocs, June 2024 Edition
We have an opening for a 3-year postdoc – more details are available here – on a project funded by Huawei via the Huawei-Edinburgh Joint Lab initiative, with me as the Principal Investigator (PI). …
-
Looking for Postdocs!
We have an opening for a 2-year postdoc – more details are available here – on a project titled Gradient-based Learning of Complex Latent Structures, with me as the Principal Investigator (PI), and Antonio Vergari (IANC) and Edoardo Ponti (ILCC) as co-PIs. The position is entirely funded by the Edinburgh Laboratory for Integrated Artificial Intelligence (ELIAI) – if you want to know more, feel free to reach out! …