Designing Agentic LLMs for Task Solving: A Multi-Agent Cybersecurity CTF Solver

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

Large language models can generate fluent reason-ing traces, but fluent output is not the same as effective tasksolving. This paper presents a Python-based agentic system forautonomous Capture-the-Flag (CTF) problem solving, designedas a multi-turn workflow instead of a single-round prompt.The system uses a planner, a category-aware specialist, and averifier connected through shared scratchpad memory and a toolexecution layer. The implementation is built with LangChain,LangGraph, and Ollama-compatible model serving, and exposesa command-line interface for solving and benchmarking CTFtasks. We evaluate the system on NYU CTF Bench. Acrossthe full 200-task benchmark, the system solved 41 challenges,corresponding to an overall solve rate of 20.5%. Performancewas strongest in miscellaneous tasks, forensics, cryptography, andreverse engineering, while pwn remained the weakest category.The main failure mode was not only hallucination, but inefficientsearch, where the agent continued generating plausible reasoningwithout sufficient operational progress. These results show thatagentic decomposition helps structure behavior, but does not byitself solve long-horizon tool use, exploit development, or attackselection.