Programmer’s guide

This goes through what a program does with the library, from its first request to shutting down. The code is taken from docs/examples and docs/snippets.cpp, which are compiled with every build, and test/test_docs.cpp runs it over all three protocols. What is about how it all works rather than how it is used has a page of its own, starting with I/O runtimes: Asio and Corosio.

Getting the library 

It takes C++23, CMake 3.30 and Boost 1.88 or later, spdlog, nghttp2, ngtcp2 and nghttp3. The Dockerfiles in .devcontainer build all of them, and those are the versions that are tested, see Development container. Whatever else goes into the process has to agree with anyhttp on the standard library and the TLS library.

As a subproject, which builds the library and nothing else, on both I/O runtimes:

include(FetchContent)
FetchContent_Declare(anyhttp
   GIT_REPOSITORY https://github.com/pgit/anyhttp.git
   GIT_TAG <commit>
   SYSTEM)
set(TLS_LIBRARY OpenSSL) # the options of the runtime and the libraries, if the defaults won't do
FetchContent_MakeAvailable(anyhttp)

target_link_libraries(my_program PRIVATE anyhttp::anyhttp)

ANYHTTP_BUILD_TESTS, ANYHTTP_BUILD_APPS, ANYHTTP_BUILD_DOCS and ANYHTTP_INSTALL default to OFF that way. To develop against a checkout, set FETCHCONTENT_SOURCE_DIR_ANYHTTP to it.

Or installed, which is for the ASIO runtime only: Corosio writes install rules only when it is the top-level project, so a library that links it can’t be exported.

cmake --preset build-deploy
cmake --build --preset build-deploy
cmake --install build-deploy --prefix /opt/anyhttp

find_package(anyhttp 0.0 REQUIRED) # with /opt/anyhttp in CMAKE_PREFIX_PATH
target_link_libraries(my_program PRIVATE anyhttp::anyhttp)

anyhttp_API, anyhttp_TLS_LIBRARY and anyhttp_H2_LIBRARY then say what that copy was built with. The package has the public headers only, those of the protocol backends stay behind. It looks for its dependencies where they were when it was built.

A build is on one I/O runtime or the other, -DANYHTTP_API=ASIO, the default, or COROSIO, which also takes -DTLS_LIBRARY=OpenSSL. I/O runtimes: Asio and Corosio has the difference. Everything below uses what both have and builds with either, and what is for ASIO only says so.

A first server 

The whole of a program, which answers every request:

#include <anyhttp/server.hpp>

#if ANYHTTP_COROSIO
#include <boost/corosio/io_context.hpp>
using IoContext = boost::corosio::io_context;
#else
#include <boost/asio/io_context.hpp>
using IoContext = boost::asio::io_context;
#endif

#include <string_view>

using namespace anyhttp;

static Task<void> hello(server::Request request, server::Response response)
{
   constexpr std::string_view body = "Hello, World!\n";

   if (auto [ec] = co_await response.submit(200, fields({{"Content-Length", body.size()}})); ec)
      co_return;

   std::ignore = co_await response.write_eof(boost::asio::buffer(body));
}

int main(int argc, char* argv[])
{
   IoContext context;

   server::Config config{.port = 8080};
   if (argc == 3)
   {
      config.tls_certificate_chain = argv[1];
      config.tls_private_key = argv[2];
   }

   server::Server server(context.get_executor(), config);
   server.on_request(hello);

   context.run();
}

A server::Server is made on an executor, and from then on everything it does happens there. There is no thread behind it, so nothing happens unless the context runs. It starts listening as it is made, on TCP and UDP, on the same port, and serves HTTP/1.1, HTTP/2 and HTTP/3 from there, with TLS and without. So there is nothing to choose: try it with curl, which takes the test PKI that the build makes in pki/out.

build/docs/hello_server &
curl http://localhost:8080/
curl --cacert pki/out/root.pem https://localhost:8080/
curl --cacert pki/out/root.pem --http3-only https://localhost:8080/

The handler gets a server::Request and a server::Response, by value: each one a move-only handle, whose end is what the peer gets to see. Each request is a coroutine of its own, and on HTTP/2 and HTTP/3 any number of them run at the same time.

Every server loads a certificate, also one that is meant to serve cleartext only. The defaults of tls_certificate_chain and tls_private_key are the test PKI, relative to the working directory, which is good for development and for nothing else. Without either, making the server throws.

A first client 

The whole of a program, which makes one GET and prints what comes back:

#include <anyhttp/client.hpp>
#include <anyhttp/session.hpp>

#if ANYHTTP_COROSIO
#include <boost/corosio/io_context.hpp>
using IoContext = boost::corosio::io_context;
#else
#include <boost/asio/io_context.hpp>
using IoContext = boost::asio::io_context;
#endif

#include <print>
#include <string>

using namespace anyhttp;

static Task<void> fetch(client::Client& client, boost::urls::url url, int& result)
{
   auto [connect_ec, session] = co_await client.connect();
   if (connect_ec)
   {
      std::println(stderr, "connect: {}", connect_ec.message());
      co_return;
   }

   auto [get_ec, message] = co_await session.get(url);
   if (get_ec)
   {
      std::println(stderr, "get: {}", get_ec.message());
      co_return;
   }

   std::println("{} over {}: {}", message.result_int(), to_string(session.protocol()),
                message.body());
   result = 0;
}

int main(int argc, char* argv[])
{
   if (argc < 2 || argc > 3)
   {
      std::println(stderr, "usage: {} <url> [<CA file>]", argv[0]);
      return 2;
   }

   IoContext context;

   boost::urls::url url(argv[1]);
   client::Client client(context.get_executor(),
                         {.url = url, .tls_ca_file = argc == 3 ? argv[2] : ""});

   int result = 1;
   launch(context.get_executor(), fetch(client, url, result));
   context.run();
   return result;
}

A client::Client makes connections, and connect() gives each as a Session, which is what requests are submitted on. There is no error to catch: like every operation that both runtimes have, these yield a tuple with the error first, and report what goes wrong instead of throwing it. The library has something to say about it all, too, which goes to the default logger of spdlog, see Logging.

build/docs/hello_client http://localhost:8080/
build/docs/hello_client https://localhost:8080/ pki/out/root.pem

Serving requests 

Reading and writing a body 

This echoes what it gets, as it gets it:

Task<void> serve_echo(server::Request request, server::Response response)
{
   if (request.content_length())
      response.content_length(request.content_length().value());

   if (auto [ec] = co_await response.submit(200, {}); ec)
      co_return;

   std::array<uint8_t, 64_k> buffer;
   for (;;)
   {
      auto [ec, n] = co_await request.read_some(boost::asio::buffer(buffer));
      if (ec == errors::eof)
         break;
      if (ec)
         co_return;

      if (auto [write_ec, written] = co_await response.write(boost::asio::buffer(buffer, n));
          write_ec)
         co_return;
   }

   std::ignore = co_await response.write_eof();
}

Writer::content_length() says how long the body will be, before submit() sends the header. It can be said with a header field as well, and what is said neither way goes out as a chunked body over HTTP/1.1.

The end of an incoming body is reported the way the runtime reports it everywhere else: errors::eof with zero bytes. A body cut short by a reset stream, or a connection that went away in the middle of a message, completes with errors::partial_message instead, so the two stay distinguishable.

The end of an outgoing body is stated explicitly, with Writer::write_eof(). It takes a buffer of its own, so the last of the body and the end of it go out together: one DATA frame with END_STREAM, one QUIC STREAM frame with FIN, one last chunk, instead of costing a second, empty write.

Task<void> respond_with(server::Response response, std::string body)
{
   if (auto [ec] = co_await response.submit(
          200, fields({{"Content-Length", body.size()}, {"Content-Type", "text/plain"}}));
       ec)
      co_return;

   std::ignore = co_await response.write_eof(boost::asio::buffer(body));
}

fields() is how to write a few header fields in place, and takes numbers as well as strings. The end of a body has what every operation answers after the end.

Telling requests apart 

A server has one handler, and what it does with the method, the URL and the fields of a request is up to it. get_param_as() looks up a parameter of the query and converts it:

Task<void> route(server::Request request, server::Response response)
{
   const std::string path = request.url().path();
   if (path == "/echo")
      co_await serve_echo(std::move(request), std::move(response));
   else if (path == "/repeat")
   {
      const auto count = request.get_param_as<size_t>("count").value_or(1);
      co_await respond_with(std::move(response), std::string(count, '*'));
   }
   else
   {
      std::ignore = co_await response.submit(404, fields({{"Content-Length", 0}}));
      std::ignore = co_await response.write_eof();
   }
}

Handing the request and the response on, as this does, is all it takes, as they are nothing but handles. So a handler may also give the reading half to one coroutine and the writing half to another, and server::Request converts to a Reader for the purpose, and server::Response to a Writer.

Limits 

max_header_size bounds the header section of a request, which is answered with 431 if it has more, without the handler seeing it. max_concurrent_streams bounds the requests that one connection has in progress over HTTP/2 and HTTP/3, see Concurrent requests. Bodies are bounded by flow control: what nobody reads, nobody sends more of.

Making requests 

One request, one operation 

Session::get() does a whole GET as one operation, and hands back the response as a plain Beast message, with status, fields and the body as a std::string:

Task<std::string> get_body(Session& session, boost::urls::url url)
{
   auto [ec, message] = co_await session.get(url);
   if (ec)
      throw_error(ec);

   std::println("{}: {} bytes", message.result_int(), message.body().size());
   co_return std::move(message.body());
}

A response that says 404 is not an error but a response, and arrives as one. The convenience is paid for with memory, as the body is buffered in full. Anything that wants to look at the body while it arrives, or to send a body of its own, goes through Session::submit().

A request in four steps 

Session::submit() sends the header and yields a client::Request, whose body Writer::write_eof() ends, and client::Request::get_response() waits for the header of the response. That gives a client::Response, which is a Reader for the rest:

IoTask<std::string> post(Session& session, boost::urls::url url, std::string body)
{
   auto [submit_ec, request] =
      co_await session.submit(url, fields({{"Content-Length", body.size()}}));
   if (submit_ec)
      co_return std::tuple{submit_ec, std::string{}};

   if (auto [ec, written] = co_await request.write_eof(boost::asio::buffer(body)); ec)
      co_return std::tuple{ec, std::string{}};

   auto [response_ec, response] = co_await request.get_response();
   if (response_ec)
      co_return std::tuple{response_ec, std::string{}};

   co_return co_await read_all(response);
}

IoTask<std::string> read_all(Reader& reader)
{
   std::string body;
   std::array<char, 16_k> buffer;
   for (;;)
   {
      auto [ec, n] = co_await reader.read_some(boost::asio::buffer(buffer));
      body.append(buffer.data(), n);
      if (ec == errors::eof)
         co_return std::tuple{error_code{}, std::move(body)};
      if (ec)
         co_return std::tuple{ec, std::move(body)};
   }
}

submit() is a POST unless it is given a method. A request without a Content-Length has a chunked body over HTTP/1.1, except for GET and HEAD, which go out with Content-Length: 0 unless their fields frame a body.

Stopping early 

Reading a response from a server that may send more than you want stops wherever you do, with status and header fields in hand:

Task<size_t> read_at_most(Session session, boost::urls::url url, size_t limit)
{
   auto [submit_ec, request] = co_await session.submit("GET", url);
   if (submit_ec)
      co_return 0;
   std::ignore = co_await request.write_eof();

   auto [response_ec, response] = co_await request.get_response();
   if (response_ec)
      co_return 0;
   std::println("{} {}", response.status_code(),
                std::string_view(response.fields()["content-type"]));

   size_t total = 0;
   std::array<char, 16_k> buffer;
   while (total < limit)
   {
      auto [ec, n] = co_await response.read_some(boost::asio::buffer(buffer));
      total += n;
      if (ec)
         break;
   }

   session.reset(); // the rest of the body is never read
   co_return total;
}

What the server gets to send in the meantime is bounded by flow control, and over HTTP/1.1 by the socket buffers. Releasing the response early also resets its stream with HTTP/2 and HTTP/3, leaving the session usable, but it is the end of an HTTP/1.1 connection.

Writing and reading at the same time 

Writing a whole body and only then reading the response is fine for a body that is small, or a server that reads it to its end first. For one that answers while it reads, as the echo above does, the two would wait for each other as soon as all the buffers on the way are full, so the two have to happen at the same time, which when_both() is for:

Task<void> upload(Writer& writer, std::string_view body)
{
   for (; !body.empty(); body.remove_prefix(std::min(body.size(), size_t{16_k})))
      if (auto [ec, written] = co_await writer.write(boost::asio::buffer(body.substr(0, 16_k))); ec)
         throw_error(ec);

   if (auto [ec, written] = co_await writer.write_eof(); ec)
      throw_error(ec);
}

Task<std::string> download(client::Request& request)
{
   auto [response_ec, response] = co_await request.get_response();
   if (response_ec)
      throw_error(response_ec);

   auto [ec, body] = co_await read_all(response);
   if (ec)
      throw_error(ec);
   co_return std::move(body);
}

Task<std::string> round_trip(Session& session, boost::urls::url url, std::string_view body)
{
   auto [ec, request] = co_await session.submit(url);
   if (ec)
      throw_error(ec);

   co_return co_await when_both(upload(request, body), download(request));
}

The helpers in here report what goes wrong by throwing, as that is what makes when_both() cancel the other one.

Several requests on one session 

HTTP/2 and HTTP/3 multiplex requests, up to a limit that the peer sets, and HTTP/1.1 has a limit of one, with some pipelining. In all of them, submit() at the limit does not wait: the caller may be the one who has to finish the request that is in the way. It fails with errors::would_block, and a caller who knows that something else will finish that request can retry later:

IoTask<client::Request> submit_patiently(Session& session, boost::urls::url url)
{
   for (;;)
   {
      auto [ec, request] = co_await session.submit(url);
      if (ec != errors::would_block)
         co_return std::tuple{ec, std::move(request)};

      std::ignore = co_await delay(10ms);
   }
}

Concurrent requests has the rules, and over TLS, Session::protocol() says what the server has picked. There is also a limit on the header section of a response, client::Config::max_header_size.

What a client connects to 

An https URL means TLS, with the server’s certificate verified against the default trust store of the TLS library:

client::Client client(executor, {.url = boost::urls::url("https://example.org")});

or against tls_ca_file, as a PEM file, which is read on every connect:

client::Client client(executor, {.url = boost::urls::url("https://localhost:8080"),
                                 .tls_ca_file = "pki/out/root.pem"});

protocol then says what to offer with ALPN. The default h2 also offers HTTP/1.1, and the server picks. Any other scheme means cleartext, and exactly the protocol given, which for HTTP/2 means prior knowledge. HTTP/3 is TLS over QUIC with any scheme:

client::Client client(
   executor, {.url = boost::urls::url("http://localhost:8080"), .protocol = Protocol::h1});

The URL says what the connection is for and, unless told otherwise, also where it goes. A caller with a resolver of its own, or following a DNS HTTPS record, gives the addresses in endpoints, which are tried in order. Nothing is resolved then, while the certificate still has to be valid for the host of the URL, and requests still say they are for it:

client::Client client(executor, {.url = boost::urls::url("https://example.org"),
                                 .endpoints = {TcpEndpoint(address, 8443)}});

To get from HTTP/1.1 or HTTP/2 to HTTP/3, see Moving to HTTP/3: Alt-Svc. Over TLS, a body that ends with the connection is complete only if the server has said so with a close_notify: otherwise, it may have been cut by anyone, and reading it fails with errors::partial_message.

Errors 

What goes wrong is an error_code. Where the two runtimes spell one differently, the namespace errors:: names it, see the table, and anything else compares against errc::: ec == errc::connection_reset. The ones that belong to the contract of an operation are in its reference and in The end of a body and Concurrent requests.

Requests and responses may outlive their session. It keeps track of them, and detaches them when it goes away, and from then on everything they are asked to do completes with an error, without touching the connection that is gone.

What a handler throws ends the coroutine of that request. On HTTP/1.1, which has nothing smaller to abort, it also ends the connection, so a handler that means to go on has to catch it.

Cancellation 

How an operation is cancelled depends on the I/O runtime: with a cancellation slot, from a completion token, or through the std::stop_token of the awaiting coroutine. What works in both is racing it against something else, with when_either(), which cancels the loser:

Task<std::optional<client::Message>> get_or_give_up(Session& session, boost::urls::url url,
                                                    std::chrono::milliseconds limit)
{
   auto result = co_await when_either(session.get(url), delay(limit));
   if (result.index() == 1)
      co_return std::nullopt; // out of time, and the request has been cancelled

   auto [ec, message] = std::get<0>(std::move(result));
   if (ec)
      co_return std::nullopt;
   co_return std::move(message);
}

A cancelled operation completes right away, with an error that compares equal to errc::operation_canceled, which is the way to ask, as a timer of the runtime has its own spelling of it. A cancelled write also says how much of its buffer it has taken. Whether the message can go on afterwards depends on the operation and the protocol: a cancelled read can always be followed by another, a cancelled write over HTTP/1.1 ends the connection. Cancellation has all of it, and how to finish a message after a cancelled write.

A coroutine that has been cancelled and wants to clean up by awaiting something else, as a last write_eof(), needs reset_cancellation() and shielded() to get there.

Completion tokens 

ASIO only. Every operation is also an asynchronous operation of Asio, named async_..., which takes any completion token. There is no need for one in a coroutine. Without one, what goes wrong is thrown:

Task<std::string> token_exception(Session& session, boost::urls::url url)
{
   auto message = co_await session.async_get(url);
   co_return std::move(message.body());
}

With as_tuple it is reported, the way the coroutine spelling does:

Task<std::string> token_tuple(Session& session, boost::urls::url url)
{
   auto [ec, message] = co_await session.async_get(url, {}, boost::asio::as_tuple);
   if (ec)
      co_return ec.message();
   co_return std::move(message.body());
}

A callback is a token as well:

void token_callback(Session& session, boost::urls::url url)
{
   session.async_get(url, {}, [](error_code ec, client::Message message) {
      if (!ec)
         std::println("{}", message.body());
   });
}

So is cancel_after, which is the short way to a timeout on this runtime:

Task<bool> response_within(client::Request& request, std::chrono::milliseconds limit)
{
   auto [ec, response] =
      co_await request.async_get_response(boost::asio::cancel_after(limit, boost::asio::as_tuple));
   co_return ec != errc::operation_canceled;
}

Ending a session 

There are two ways to end a session, and neither waits for what is in progress, which fails right away:

Session::shutdown() ends the connection in an orderly way: over HTTP/2 with a GOAWAY, over TLS with a close_notify in both directions, then a FIN in both, with each wait limited to two seconds. It fails with errors::connection_aborted only if a write had to be interrupted, which rules out a close_notify. HTTP/3 sends its CONNECTION_CLOSE.
Destroying the session, or Session::reset(), is over as fast as possible, and does nothing asynchronous. The peer sees a connection that was cut.

Task<void> say_goodbye(Session session)
{
   if (auto [ec] = co_await session.shutdown(); ec)
      std::println("the connection could not be ended in an orderly way: {}", ec.message());
}

Running it 

Stopping 

Destroying the server::Server stops it listening and ends every connection it has. Do that on the server’s own executor, and then let the context run on until it returns:

Task<void> serve_for(std::optional<server::Server>& server, std::chrono::milliseconds duration)
{
   std::ignore = co_await delay(duration);
   server.reset(); // on the server's own executor, which this coroutine has been started on
}

inline void run_for_a_while(server::Config config)
{
   IoContext context;

   std::optional<server::Server> server;
   server.emplace(context.get_executor(), config);
   server->on_request(route);

   launch(context.get_executor(), serve_for(server, 100ms));
   context.run(); // returns when all of it has come to an end
}

A context has to run dry, on both I/O runtimes. What is suspended when it is destroyed mostly gets unwound, but not what waits on an object that only comes apart by running, and sessions do. HTTP/2 and HTTP/3 leak without that drain.

Threads 

Everything is on one thread unless told otherwise, which needs no locks and no strands. With several, there are two ways:

one context, run by all threads, and use_strand to give each connection a strand of its own, on which its handlers run:

inline void run_on_threads(server::Config config, size_t count)
{
   IoContext context(static_cast<int>(count));

   config.use_strand = true;
   std::optional<server::Server> server;
   server.emplace(context.get_executor(), config);
   server->on_request(route);

   launch(context.get_executor(), serve_for(server, 100ms));

   std::vector<std::thread> threads;
   for (size_t i = 1; i < count; ++i)
      threads.emplace_back([&context] { context.run(); });
   context.run();
   for (auto& thread : threads)
      thread.join();
}

a context on each thread, each with a server of its own, and reuse_port to have them all on the same port. On Asio, this is what scales best, see docs/benchmark.md. The kernel spreads connections and QUIC datagrams over them, by address, so all of them have to be bound before traffic arrives, and a QUIC client whose address changes lands on another server, which does not know it.

Either way, a Session and everything it gives out are to be used on its executor only. The server program does both, see --threads and --independent in src/apps/server_main.cpp.

On Corosio, one context on several threads has a use-after-free in its reactor, whose window is narrow but real, see step 8 of docs/corosio-port-plan.md.

Logging 

The library and the programs log through spdlog’s default logger, so its level and its sinks are what to set. The logger is also where to look first: each line carries the protocol, the peer and the stream it belongs to, which is the same tag as Reader::log_prefix() gives.

Plain QUIC streams 

QUIC streams without HTTP in them, for a protocol of your own told apart by its ALPN, run on the UDP port HTTP/3 is on. A quic::Stream is a Reader and a Writer at once, and server::Server::on_quic() is what gets each connection:

server.on_quic({.alpn = "echo"}, [](quic::Connection connection) -> Task<void> {
   for (;;)
   {
      auto [ec, stream] = co_await connection.accept_stream();
      if (ec)
         co_return; // the connection is over
      launch(connection.get_executor(), quic_echo(std::move(stream)));
   }
});

Task<void> quic_echo(quic::Stream stream)
{
   std::array<uint8_t, 16_k> buffer;
   for (;;)
   {
      auto [ec, n] = co_await stream.read_some(boost::asio::buffer(buffer));
      if (ec == errors::eof)
         break;
      if (ec)
         co_return;

      if (auto [write_ec, written] = co_await stream.write(boost::asio::buffer(buffer, n));
          write_ec)
         co_return;
   }
   std::ignore = co_await stream.write_eof();
}

and quic::connect() is the other side of that:

IoTask<std::string> quic_hello(Executor executor, std::string host, std::string port,
                               std::string ca_file)
{
   auto [connect_ec, connection] = co_await quic::connect(
      executor, std::move(host), std::move(port), {.alpn = "echo", .tls_ca_file = ca_file});
   if (connect_ec)
      co_return std::tuple{connect_ec, std::string{}};

   auto [open_ec, stream] = connection.open_stream();
   if (open_ec)
      co_return std::tuple{open_ec, std::string{}};

   if (auto [ec, written] = co_await stream.write_eof(boost::asio::buffer("hello", 5)); ec)
      co_return std::tuple{ec, std::string{}};

   co_return co_await read_all(stream);
}

The ends map onto the contract of The end of a body:

QUIC	Writer’s side	Reader’s side
FIN	`write_eof()`	`errors::eof`
RESET_STREAM	`reset_stream()`, or dropping the Writer before the end	`errors::partial_message`
STOP_SENDING	writes fail with `errc::connection_reset`	`stop_sending()`, or dropping the Reader before the end

The two directions are independent. Opening a stream never waits: at the peer’s limit, open_stream() fails with errors::would_block. A connection is closed when its handle goes away, on the server when the handler returns. quic::Config has the flow control limits and how writes reach the stack, and docs/quic-streams-plan.md has how all this came about.

Deploying the programs 

cmake --install build-deploy --prefix <dir> --component Runtime gives bin/anyhttp-server and bin/anyhttp-client, plus every shared library they need in lib/, except for glibc and libgcc_s. The tree can be moved or copied to another machine that has a glibc at least as new as the build host’s. The bundled libraries have licenses of their own, to be passed on along with them.

anyhttp-server -p 443 --cert /etc/anyhttp/chain.pem --key /etc/anyhttp/key.pem

The server always loads a certificate, also to serve only cleartext: without --cert and --key it looks for the test PKI (pki/out/, relative to the working directory), and exits with an error if that isn’t there. scripts/check_package.sh verifies all of this from the outside.