It took around 59 hours of API time and $350 of Gemini 2.5 Pro credits to produce the initial implementation, which passed 117/145 (80.6%) of the h2spec HTTP 2.0 compliance tests. The majority of failures were due to the server failing to return an error response when it should, as these cases weren't captured by integration tests since the http clients used for testing don't send unusual/invalid inputs to the server. A series of refactoring runs, in which the LLM was shown the h2spec failures then asked to make and follow a plan to fix them, brought it up to 128/145, then to 137/145, then to 142/145, then to 144/145, and at last to 145/145. Finally, after 60 API hours and $281 of refactoring, Llmahttap (pronounced lmah-tahp) was born! Weighing in at around 15k lines of source code and 32k lines of unit tests, from start to finish Llmahttap took in total around 119 hours of API time to build, and $631 of API credits. In clock terms it took around two weeks to build; it would have been around half of that if not for the initial Gemini 2.5 Pro rate limit of just 1k requests per day.

I wouldn't recommend anyone use it in production, as there are probably plenty of security issues (the TLS support is poorly tested as h2spec requires a proper cert for TLS testing, and I don’t have one lying around, plus it completely lacks HTTP 1.1 support), but the code is interesting to read from the perspective of seeing what a 100% Gemini-built application looks like. It's also a demonstration of the power of structured LLM workflows to achieve results that are difficult (or at least more expensive) with a purely free-form agentic approach.

Note that while 100% of the application code is AI-written, only around 99.9% of the unit test code is. A couple of times it set the component it was testing to log to null and hence made very slow progress on getting the unit tests it wrote to pass, so I manually modified those tests to instead log to stdout so it'd be properly visible in the `go test` failure output. And when some unit test files got too large I split them up into multiple smaller files to save time and money, as the model doesn’t need five thousands lines of unit tests in context. I also git reverted to a recent checkpoint a few times when it looked like it was getting really off-track, and when it hit a syntax error it was pathologically unable to fix (misplaced braces in a long slice of structs literal). Also note that 100% refers to the HTTP 2.0 spec implementation; for HPACK (a separate spec from the main HTTP 2.0 spec), the Go stdlib is used, similarly for TLS.

The code was written by a program in a higher-order NOn-DeTerministic Programming Language (NOTPL) called Promptyped, which is a DSL embedded in Haskell. The NOTPL program's rough structure was as follows:

• First, with the whole spec in context, generate a list of files that must be created. Alongside each filename, include a high-level description, and list the dependencies of each file (including both project files and spec subjection files), then topologically sort the result so the files are listed after the files they depend on.

• Then, for each file to be created, generate a detailed list of tasks to be completed, keeping the relevant dependencies for that file in context.

• Then, for each task, put the LLM in a code-compile-test loop until the change is complete (as verified by a separate LLM call), the project builds successfully and all unit tests pass.

The refactoring NOTPL's structure was similar, except instead of generating a list of files to create, it generated a list of files that might need changes, then for each such file generated a list of actual changes needed. The Promptyped logs from development are included compressed in the Llmahttap repo, for anyone who’s interested in the exact prompts and responses.

I did encounter one significant issue in the initial NOTPL program. The topological dependency sort didn't place the unit test files directly after the files they tested, instead many ended up right at the end of the list, and the integration test was placed earlier (it's not ideal to start integration testing when there aren't even any unit tests yet). Note that as described in my previous post, the framework memoises/caches the results of all tasks on disk, and restarting the program will automatically continue from where it left off. So to address the issue, I manually modified the cached JSON task-list on disk to re-order the unit tests such that each was listed right after the files it tested, then restarted. There was also a smaller issue: it created staticserver.go as part of general end-to-end testing, but then later in the task to create a static file server it made a separate static_file_server.go with duplicate functionality. This was because the task that created staticserver.go didn't have vision of the overall work plan, so didn't know what the upcoming static file server file would be called. The fix for this was to run a separate refactor task afterwards to merge them (and to update the NOTPL program to also give the overall plan as context to individual file tasks).

To achieve this task the Promptyped framework required some significant upgrades in functionality (relative to the version described in the previous post). The key improvements made, in roughly descending order of importance, were as follows.

• Having a separate LLM check to verify that each task was actually done. This makes it much harder for the LLM to falsely report completion for an incomplete task.

• Rejecting diffs that would lead to syntactically incorrect code (would cause `go fmt` to fail due to inability to parse the code). I found Gemini (even 2.5 Pro) would sometimes completely fail to be able to fix compilation errors in a large context resulting from imbalanced braces (e.g. after an off-by-one error in a diff causes it to overwrite a function's closing brace, or add an extra closing brace). Rejecting such diffs avoided this issue as the LLM would never see such malformed code. Interestingly, in my testing OpenAI models seemed not to get stuck in the same situation, and did not need this functionality, but I used Gemini 2.5 Pro for the project due to better long-context handling and significantly cheaper API calls compared to o3. The model is also sent a message showing what the rejected code post-diff would have looked like, to help it see how the diff is wrong

• Clearing the message history after 5 failed attempts to produce syntactically correct tool calls. This prevents the LLM from getting stuck continuously trying small variations of the same syntactically invalid approach.

• Checking for progress after every 5 diffs resulting in failed compilation, and if there's no progress (same compiler errors still), then running a separate query with a smaller amount of context to suggest a solution. This helps avoid situations where the LLM gets stuck on a stupid hallucination (e.g. it thinks the function name is fooFunc instead of FooFunc even though the context contains the code showing it is indeed named FooFunc).

• When the `go test` failure output is more than e.g. 600 lines, parsing it to extract the actual failed tests, then running just one and giving the LLM the truncated output of that (first and last 400 lines). This makes it simpler for the LLM by only including logs from the relevant failed unit test in context, and saves time and money by reducing the overall context size. The names of all failed unit tests are also passed to the LLM alongside this, so the LLM doesn't incorrectly assume a failing test is passing.

• Each time the LLM calls a tool, requiring it to also include a summary of what it's doing, why it's doing it, and its future plans after it. These are put into an event list alongside other events like compilation successes and unit test failures, and the model is shown the last 70 or so events. This helps the model keep a consistent train of thought/action, and also reduces the chance of the model getting stuck in loops (e.g. fixing test A in a way that breaks test B, then fixing test B in a way that breaks test A, and continuing this way indefinitely). It also makes it easier for a human to see the approach the model's taking to the current task without needing to read the code diffs.

• Keeping all relevant source code in context, minified/"unfocused". I added support for focusing/unfocusing to the framework, such that an unfocused source file shows only type definitions, top-level comments and function headers. This significantly saves space in the context, while still giving the LLM the information it needs to use those types and functions in the files it's currently working on. I automatically focus the most recently modified files, and also give the LLM itself a tool to focus/unfocus files, with the maximum number of focused files limited to 4 (at which point, focusing a new file will unfocus the least recently modified one).

Some other changes that were not strictly necessary, but useful for reducing costs and saving time:

• Using editing by regex instead of by line number. I found all LLMs often make mistakes with line numbers, even if I include regular line-number comments in the code shown to the LLM, particularly off-by-one errors. What seems to be a more reliable approach is instead of line numbers, asking the LLM to provide a regex to match the first line and a regex to match the last line, along with nearest-line-numbers to the first and last line (so if there are multiple potential matches, it picks the right one, and potential matches too far from that line are rejected). When the LLM fails this, generally it fails by providing a regex that matches nothing at all and does nothing, which is less destructive to code than the common failure mode of line-based editing where an off-by-one error leads to e.g. a closing brace being accidentally deleted or duplicated.

• Being very liberal in the tool call syntax accepted from Gemini. Often it ignores the syntax in the prompt and instead goes with its own syntax, and isn't always consistent with what syntax it uses, so being liberal avoids wasting time and money re-querying it just to make it change the tool syntax. I deliberately avoid using the explicit tool-calling functionality offered by LLM providers as I find it cleaner to treat the LLM as just a function from Text to Text, and this approaches generalises better across providers.

• On tool syntax errors, running a separate query with a small context to attempt to correct the syntax. Although it doesn't always work, when it does it avoids unnecessarily re-running big slow queries just because of easily fixable syntax errors.

• Using rawtext literal syntax for code rather than json. For tools like append to and modify file, instead of requiring the LLM to include the new text/code as an escaped string in the JSON object, a textBoxName field is used instead. The model then separately provides a textbox with that name, and the text to be included in C++ raw string literal syntax. This avoids the need for the model to escape special characters in the JSON, reducing the amount of errors it makes.

In my view, the success of this task demonstrates that LLMs are already capable of generating non-trivial applications from scratch, given a sufficiently detailed specification and an external means of correctness testing. The human's challenge is then properly specifying, and the hardest part: deciding what to build. Maybe structuring the architecture also requires human input; to my eyes, Gemini generally tends to architect code worse than a human, at least without detailed prompting, although perhaps more expensive LLMs like o3 or Claude Opus can do better.

In the long run, I suspect people will move away from the free-form agent approaches being heavily promoted by LLM providers (who have a huge financial incentive to make you put as much in the context window as possible). Structured workflow/NOTPL approaches with a carefully managed context have the potential to be significantly more efficient, in terms of both cost and time (since both are directly proportional to context size), and ultimately economic law dictates that firms with the most cost-effective approach (lowest cost of production) will eventually win. Even if we had human-level agents capable of coding a task from start to finish entirely unsupervised, it'd likely still be more cost-efficient for them to delegate the coding to a cheaper, dumber model following a structured workflow, than to code it all themselves.

The next thing I plan to implement is more granular focusing: the ability to just focus specific individual functions. So the LLM can e.g. open a 5k line unit test file without needing to keep the whole file in context, instead just the function bodies for the tests it's currently working on are shown in full. And a clean way to run and manage multiple tasks simultaneously. I also plan to add support for Haskell as a project language, so people unfamiliar with Haskell can use Promptyped itself to add new custom workflows to Promptyped.

The program in question (the first test-case I picked for this approach) is a market data recorder: an application that connects to an exchange's live data stream (in this case Binance websockets) and records the data to parquet files, for entirely non-nefarious purposes. While not particularly difficult, it nevertheless requires the LLM coding it to correctly utilise multiple external libraries (gorilla websockets and parquet), and write code that correctly interfaces with an external API. Something that would take a human at the very least a few hours to write and debug (and much more when doing the same thing for many exchanges). With the relatively simple framework I developed for typed workflows (available here), which uses the "LLM in a loop" approach to force the model to bash its head against the compiler and unit tests (which it's compelled to write), I managed to get o3 mini to write a largely correct* implementation (available here), for the cost of only around $5.10 (1824376 tokens in, 826883 tokens out), significantly cheaper than the human labour to produce equivalent code. It took around 2 hours and 40 minutes, quicker than a human (especially given the number of unit tests it wrote), with around 2.6 minutes spent in compilation and 2.5 spent running unit tests, 95 compilation failures, 34 test failures, and 23 incorrect JSON responses from the LLM.

The key components of this approach are as follows:

1. Embedding the LLM in the host programming language as a domain-specific language (DSL), so that it can be integrated into control flow and return structured values (via JSON, which get checked and automatically converted to values in the host language).

2. Smart context management in the host language, showing the LLM only what's necessary for the task at hand, which not only reduces costs but also reduces the inaccuracy that results from larger context lengths.

3. Breaking up the project into smaller chunks (workflow) managed by the host language control flow.

4. Automatic validation of code quality via compilation and unit test checks, refusing to allow the LLM to progress to the next task until all code compiles and tests pass (miraculously, it never decided to just "delete the failing unit test", which is more than can be said of some human developers).

5. Providing explicit line numbers to the LLM as comments at the start of lines, to facilitate easier editing (I don't know if other tools do this, but if not they really should, as having an LLM manually count lines is a huge waste of processing power/attention, much as it would be for a human).

To illustrate the above, the exact program was roughly as follows. First, have the LLM design an architecture for the project given the spec (just return as text). Next, have it plan the source files for the project and their dependencies, returning a list of (sourceFile, [dependency]) pairs. This is checked to ensure no circular dependencies, and the LLM forced to retry in the case where it introduced any. Explicitly listing the dependencies allows us to only provide the relevant dependencies in the initial context when creating a file.

After that, we iterate over all the planned files, and for each file, first get it to write the file (returning a description), and ensure it compiles. Next, the LLM is asked to plan a list of unit tests, and we iterate over them one-by-one, getting the LLM to write them and not allowing it to finish until everything compiles and the tests pass. At each new task, we refresh the context to just what is initially relevant to the task at hand. After this is done, the project is largely complete, requiring only human checking to ensure it actually works as expected and that the unit tests cover everything important.

To assist in this we provide the LLM with multiple "Tools", specified in the context, which it may call with a specific JSON format. We provide a list of AvailableFiles (all relevant files in the project, including documentation on the APIs/libraries used), with optional descriptions that it adds upon file creation, that it can Open, which will add them to context, and Close if already open, which removes them from the context. For files in context we allow it to edit them (based on line numbers), insert into a file at a specific line number, and append to a file (appending can also be used to create a new file). Compiling and running tests is done automatically after any source file modification, and in the case of failure the error message is returned to the LLM. We also provide a "Panic" tool, for when the LLM deems the task is completely insolvable given the resources at hand, which proved very useful in the initial development of the framework.

For context management, we limit the context returned to the model to the last ten messages, and strategically truncate messages. In particular, we truncate all but the last compile/test failure message (so it only sees the relevant one), and all but the last message from the LLM (so if a tool operation files due to e.g. a syntax error in the JSON, it can see what it wanted to do and try again with correct syntax, but the context isn't bloated with all previous file operations it made, instead it just sees the current file state via the OpenFiles in the context).

To assist it in persisting state/intention across calls, we encourage it to write to a journal.txt detailing what it's currently attempting and what it plans to do next, especially when debugging failed unit tests. Quite cleverly, the model decided to use this as the logfile for the application's logs, so that it would always see the most recent unit test logs in the journal without need to explicitly open them. It did however forget to make this logfile path configurable..

We also memoise (cache to disk) the values successfully "returned" by the LLM, so that we can stop the process and restart, and it will continue from where it left off. This is useful if it goes off the rails and needs help to recover, which happened a few times during framework development as a result of bugs in the framework that presented unclear/incorrect context to the LLM. It also happened when I was trying to use Deepseek R1 (over OpenRouter, so potentially quantised). It for instance got confused when missing the closing ) for an import statement and seemed unable to fix the issue, and sometimes was unable to correctly get the go local import syntax correct.

I feel like at least in the short term a workflow-based approach like the above is optimal for squeezing the most productivity out of LLMs, so I'm open-sourcing the small, awkward library I built in hope it might prove useful for someone. While free-form agents may be the future, workflows are the present. And even in future, it'll likely be cheaper to use a workflow-based approach (which works with smaller LLMs) than fully autonomous agents. Such agents, if sufficiently human, might even in future decide to use a workflow-based approach themselves to save time, delegating to smaller LLMs. It's my hope that such workflow approaches, alternatively known as NOn-deterministic exTremely-high-level Programming Languages (NOTPLs) become widely used, such that LLMs will train on them and themselves become better at writing workflows, simplifying my work even further.

One important note: the library I wrote is in Haskell, the most convenient language for writing EDSLs. The actual usage is however relatively simple, by Haskell standards, and I welcome any attempts to tidy it up further. Ideally in future it would provide a full DSL, a simple language that wraps the underlying implementation while providing a nicer user interface, and then more and more programs could be written in/by such NOTPLs, saving a significant amount of time.

The main interface to the library is the following:

runAiFunc ::
  forall bs a b.
  (FromJSON a, ToJSON a, Show a, BS.BuildSystem bs) =>
  Context ->
  [Tools.Tool] ->
  a ->
  (a -> AppM (Either (MsgKind, Text) b)) ->
  RemainingFailureTolerance ->
  AppM b

a is a type parameter, the type of the object we want the LLM to return, which must be convertible to JSON. bs is a type representing the build system (essentially an interface), which allows changing the backend used for compiling and running unit tests. b is a type we postprocess the a into (we can just use a for it if we don’t need postprocessing).
Context is a struct with the background text for the task, and [Tools.Tool] is a list of tools that task is allowed to use. Currently the library doesn’t support externally defined tools, but new tools can easily be added to the library in Tools.hs.
We provide a value of type a to the function as an example/dummy value, to illustrate to the LLM how it should look.
RemainingFailureTolerance is just an integer representing how many syntax errors (LLM returns syntactically incorrect JSON etc.) before aborting.
The most complex type is (a -> AppM (Either (MsgKind, Text) b)). This is the type of a validator: a function that takes an a as input, and returns either a Text error and error kind, or a value of type b (some post-processed version of a). The result is however wrapped in AppM, a monad, which just means the validation function has access to a state and the ability to do IO; e.g. to validate that a file really exists on disk, or that compilation succeeds and unit tests pass. The runAiFunc function will keep looping until the LLM returns a value for which the validator passes (or RemainingFailureTolerance reaches zero).

Other LLMS:
I also attempted to use DeepSeek, but came across a few issues. The biggest is WRT availability: the official DeepSeek provider is under heavy load, so I had to use OpenRouter instead, which dynamically routes to many providers, who may use different levels of quantisation. Overall DeepSeek seemed to do worse at following instructions, and at fixing code; in particular it kept failing to notice when a closing bracket ) was missing in the import list. I'm not sure however whether this is just the result of quantisation, as OpenRouter doesn't seem to require DeepSeek providers to list what degree of quantisation they're using. DeepSeek over OpenRouter was generally also significantly more expensive; there are a couple larger providers, Fireworks and Together.ai, which tended to be more stable/reliable/fast, but charged around 4x more than the official DeepSeek API (and than OpenAi o3-mini-high). There were also some smaller providers with more competitive prices, but they tended to get overloaded very frequently and rarely be available (often returning empty strings as a response).

I tried using Gemini Flash 2.0, but it seemed to struggle too much at returning code as properly escaped JSON, which is forgivable given it's 10x cheaper than o3 mini and not a thinking model. I tried Gemini Flash Thinking too, but the only version available on OpenRouter was the free version, which was heavily throttled to the point of being unusable. Since even o3-mini sometimes gets the JSON escaping wrong, I’m going to look at changing the tool call syntax so the model can use raw text blocks for code, which would remove the need for it to correctly JSON-format/escape the code and hence hopefully significantly decrease the rate of syntax errors.

Other Languages:
I also tried using C++ as a backend, but came across an issue where the LLM would occasionally miss/delete a closing angle bracket, causing the compiler to vomit up so many errors that the context got completely overwhelmed and the LLM couldn’t fix it. Given the C++ also took significantly longer to compile, and isn’t necessary for this project, I decided to go with Go instead. Go seems particularly well suited to LLM usage because of the simplicity of its design and the large, high-quality standard library (meaning the LLM needs to be familiar with fewer external libraries to get a task done).

Footnote:

*I said largely correct because I identified three bugs in the final response, two of which were technically my fault for not specifying more clearly, and one is a mistake I'd have also made myself. It didn't explicitly set the parquet page size, row group size and compression type, which is best practices but not something I directly asked of it. For merging snapshots with orderbook updates, it didn't explicitly request a snapshot upon startup, instead waiting for the regular one-snapshot-per-minute timer (technically I didn't explicitly ask it to request an initial snapshot, but it's a sensible thing to do). And most significantly, it had a bug due to this issue. It tried to read from the gorilla websocket on a timeout, so that it could regularly check for a cancellation request from the context, but rather un-intuitively the library trashes the connection after timeout, so after the first timeout all further reads fail. This wasn't caught in the unit tests because the tests only checked if it could receive a single correct JSON object from the websocket, so succeeded before the connection timed out. I'd argue that the original developers of gorilla/websocket (who've now abandoned it) deserve more blame than the LLM here, for not providing a non-destructive way to read with timeout, but fortunately some brave soul has taken over maintainership of the library so maybe we'll see a fix sometime in future.

Subscribe now