An engine manages connections, processes incoming packets, and schedules outgoing packets. It can be instantiated in either server or client mode. If your program needs to have both QUIC client and server functionality, instantiate two engines. (This is what we do in our LiteSpeed ADC server.)
In addition, HTTP mode can be turned on for gQUIC and HTTP/3 support.
Connection
----------
A connection carries one or more streams, ensures reliable data delivery, and handles the protocol details.
In client mode, a connection is created using a function call, which we will cover later in the tutorial.
In server mode, by the time the user code gets a hold of the connection object, the handshake has already been completed successfully. This is not the case in client mode.
Stream
------
A connection can have several streams in parallel and many streams during its lifetime.
Streams do not exist by themselves; they belong to a connection. Streams are bidirectional and usually correspond to a request/response exchange - depending on the application protocol.
The HTTP support is included directly into LSQUIC. The library hides the interaction between the HTTP application layer and the QUIC transport layer and presents a simple, unified way of sending and receiving HTTP messages. (By "unified way," we mean between Google QUIC and HTTP/3). Behind the scenes, the library will compress and decompress HTTP headers, add and remove HTTP/3 stream framing, and operate the necessary control streams.
UDP datagrams are passed to the engine using the :func:`lsquic_engine_packet_in()` function. This is the only way to do so.
A pointer to the UDP payload is passed along with the size of the payload.
Local and peer socket addresses are passed in as well.
The void "peer ctx" pointer is associated with the peer address. It gets passed to the function that sends outgoing packets and to a few other callbacks. In a standard setup, this is most likely the socket file descriptor, but it could be pointing to something else.
The ECN value is in the range of 0 through 3, as in RFC 3168.
::
/* 0: processed by real connection
* 1: handled
* -1: error: invalid arguments, malloc failure
*/
int
lsquic_engine_packet_in (lsquic_engine_t *,
const unsigned char *udp_payload, size_t sz,
const struct sockaddr *sa_local,
const struct sockaddr *sa_peer,
void *peer_ctx, int ecn);
Why specify local address
-------------------------
The local address is necessary because it becomes the source address of the outgoing packets. This is important in a multihomed configuration, when packets arriving at a socket can have different destination addresses. Changes in local and peer addresses are also used to detect changes in paths, such as path migration during the classic "parking lot" scenario or NAT rebinding. When path change is detected, QUIC connection performs special steps to validate the new path.
-``EAGAIN`` (or ``EWOULDBLOCK``) means that the packets could not be sent and to retry later. It is up to the caller to call :func:`lsquic_engine_send_unsent_packets()` when sending can resume.
-``EMSGSIZE`` means that a packet was too large. This occurs when lsquic send MTU probes. In that case, the engine will retry sending without the offending packet immediately.
- Any other error causes the connection whose packet could not be sent to be terminated.
Outgoing Packet Specification
-----------------------------
::
struct lsquic_out_spec
{
struct iovec *iov;
size_t iovlen;
const struct sockaddr *local_sa;
const struct sockaddr *dest_sa;
void *peer_ctx;
int ecn; /* 0 - 3; see RFC 3168 */
};
Each packet specification in the array given to the "packets out" function looks like this. In addition to the packet payload, specified via an iovec, the specification contains local and remote addresses, the peer context associated with the connection (which is just a file descriptor in tut.c), and ECN.
The reason for using iovec in the specification is that a UDP datagram may contain several QUIC packets. QUIC packets with long headers, which are used during QUIC handshake, can be coalesced and lsquic tries to do that to reduce the number of datagrams needed to be sent. On the incoming side, :func:`lsquic_engine_packet_in()` takes care of splitting incoming UDP datagrams into individual packets.
When to process connections
===========================
Now that we covered how to initialize the library, instantiate an engine, and send and receive packets, it is time to see how to make the engine tick. "LSQUIC" has the concept of "tick," which is a way to describe a connection doing something productive. Other verbs could have been "kick," "prod," "poke," and so on, but we settled on "tick."
There are several ways for a connection to do something productive. When a connection can do any of these things, it is "tickable:"
- There are incoming packets to process
- A user wants to read from a stream and there is data that can be read
- A user wants to write to a stream and the stream is writeable
- A stream has buffered packets generated when a user has written to stream outside of the regular callback mechanism. (This is allowed as an optimization: sometimes data becomes available and it's faster to just write to stream than to buffer it in the user code and wait for the "on write" callback.)
- Internal QUIC protocol or LSQUIC maintenance actions need to be taken, such as sending out a control frame or recycling a stream.
/* Returns true if there are connections to be processed, in
* which case `diff' is set to microseconds from current time.
*/
int
lsquic_engine_earliest_adv_tick (lsquic_engine_t *, int *diff);
There is a single function,
:func:`lsquic_engine_earliest_adv_tick()`, that can tell the user whether and when there is at least one connection managed by an engine that needs to be ticked. "Adv" in the name of the function stands for "advisory," meaning that you do not have to process connections at that exact moment; it is simply recommended. If there is a connection to be ticked, the function will return a true value and ``diff`` will be set to a relative time to when the connection is to be ticked. This value may be negative, which means that the best time to tick the connection has passed.
The engine keeps all connections in several data structures. It tracks each connection's timers and knows when it needs to fire.
Example with libev
------------------
::
void
process_conns (struct tut *tut)
{
ev_tstamp timeout;
int diff;
ev_timer_stop();
lsquic_engine_process_conns(engine);
if (lsquic_engine_earliest_adv_tick(engine, &diff) {
if (diff > 0)
timeout = (ev_tstamp) diff / 1000000; /* To seconds */
else
timeout = 0.;
ev_timer_init(timeout)
ev_timer_start();
}
}
Here is a simple example that uses the libev library. First, we stop the timer and process connections. Then, we query the engine to tell us when the next advisory tick time is. Based on that, we calculate the timeout to reinitialize the timer with and start the timer.
If ``diff`` is negative, we set timeout to zero.
When the timer expires (not shown here), it simply calls this ``process_conns()`` again.
Note that one could ignore the advisory tick time and simply process connections every few milliseconds and it will still work. This, however, will result in worse performance.
Processing Connections
----------------------
Recap:
To process connections, call :func:`lsquic_engine_process_conns()`.
This will call necessary callbacks to read from and write to streams
and send packets out. Call `lsquic_engine_process_conns()` when advised
by `lsquic_engine_earliest_adv_tick()`.
Do not call `lsquic_engine_process_conns()` from inside callbacks, for
this function is not reentrant.
Another function that sends packets is
:func:`lsquic_engine_send_unsent_packets()`. Call it if there was a
previous failure to send out all packets
Required Engine Callbacks
=========================
Now we continue to initialize our engine instance. We have covered the callback to send out packets. This is one of the required engine callbacks.
Other required engine callbacks are a set of stream and connection callbacks that get called on various events in then connections and stream lifecycles and a callback to get the default TLS context.
Here we mention some optional callbacks. While they are not covered by
this tutorial, it is good to know that they are available.
- Looking up certificate and TLS context by SNI.
- Callbacks to control memory allocation for outgoing packets. These are useful when sending packets using a custom library. For example, when all packets must be in contiguous memory.
- Callbacks to observe connection ID lifecycle. These are useful in multi-process applications.
- Callbacks that provide access to a shared-memory hash. This is also used in multi-process applications.
- HTTP header set processing. These callbacks may be used in HTTP mode for HTTP/3 and Google QUIC.
Please refer to :ref:`apiref-engine-settings` for details.
Stream and connection callbacks
===============================
Stream and connection callbacks are the way that the library communicates with user code. Some of these callbacks are mandatory; others are optional.
They are all collected in :type:`lsquic_stream_if` ("if" here stands
for "interface").
The mandatory callbacks include calls when connections and streams are created and destroyed and callbacks when streams can be read from or written to.
The optional callbacks are used to observe some events in the connection lifecycle, such as being informed when handshake has succeeded (or failed) or when a goaway signal is received from peer.
When a connection object is created, the "on new connection" callback is called. In server mode, the handshake is already known to have succeeded; in client mode, the connection object is created before the handshake is attempted. The client can tell when handshake succeeds or fails by relying on the optional "handshake is done" callback or the "on connection close" callback.
::
/* Return pointer to per-connection context. OK to return NULL. */
In the made-up example above, a new per-connection context is allocated and returned. This context is then associated with the connection and can be retrieved using a dedicated function. Note that it is OK to return a ``NULL`` pointer.
Note that in client mode, this is a good place to request that the connection make a new stream by calling :func:`lsquic_conn_make_stream()`. The connection will create a new stream when handshake succeeds.
On new stream
-------------
QUIC allows either endpoint to create streams and send and receive data on them. There are unidirectional and bidirectional streams. Thus, there are four stream types. In our tutorial, however, we use the familiar paradigm of the client sending requests to the server using bidirectional stream.
On the server, new streams are created when client requests arrive. On the client, streams are created when possible after the user code has requested stream creation by calling :func:`lsquic_conn_make_stream()`.
In a pattern similar to the "on new connection" callback, a per-stream context can be created at this time. The function returns this context and other stream callbacks - "on read," "on write," and "on close" - will be passed a pointer to it. As before, it is OK to return ``NULL``.
You can register an interest in reading from or writing to the stream by using a "want read" or "want write" function. Alternatively, you can simply read or write; be prepared that this may fail and you have to try again in the "regular way." We talk about that next.
On read
-------
When the "on read" callback is called, there is data to be read from stream, end-of-stream has been reached, or there is an error.
ssize_t nr = lsquic_stream_read(stream, buf, sizeof(buf));
/* Do something with the data.... */
if (nr == 0) /* EOF */ {
lsquic_stream_shutdown(stream, 0);
lsquic_stream_wantwrite(stream, 1); /* Want to reply */
}
}
To read the data or to collect the error, call :func:`lsquic_stream_read`. If a negative value is returned, examine ``errno``. If it is not ``EWOULDBLOCK``, then an error has occurred, and you should close the stream. Here, an error means an application error, such as peer resetting the stream. A protocol error or an internal library error (such as memory allocation failure) lead to the connection being closed outright.
The "on write" callback is called when the stream can be written to. At this point, you should be able to write at least a byte to the stream.
As with the "on read" callback, for this callback to be called, the user must have registered interest in writing to stream using :func:`lsquic_stream_wantwrite()`.
By default, "on read" and "on write" callbacks will be called in a loop as long as there is data to read or the stream can be written to. If you are done reading from or writing to stream, you should either shut down the appropriate end, close the stream, or unregister your interest. The library implements a circuit breaker to stop would-be infinite loops when no reading or writing progress is made. Both loop dispatch and the circuit breaker are configurable (see :member:`lsquic_engine_settings.es_progress_check` and :member:`lsquic_engine_settings.es_rw_once`).
When reading and writing ends of the stream have been closed, the "on close" callback is called. After this function returns, pointers to the stream become invalid. (The library destroys the stream object when it deems proper.)
This is a good place to perform necessary cleanup.
When either :func:`lsquic_conn_close()` has been called; or the peer has closed the connection; or an error has occurred, the "on connection close" callback is called. At this point, it is time to free the per-connection context, if any.
To reduce buffering, most of the time bytes written to stream are written into packets directly. Bytes are buffered in the stream until a full packet can be created. Alternatively, one could flush the data by calling :func:`lsquic_stream_flush`.
It is impossible to write more data than the congestion window. This prevents excessive buffering inside the library.
Inside the "on read" and "on write" callbacks, reading and writing should succeed. The exception is error collection inside the "on read" callback.
Outside of the callbacks, be ready to handle errors. For reading, it is -1 with ``EWOULDBLOCK`` errno. For writing, it is the return value of 0.
More stream functions
---------------------
Here are a few more useful stream functions.
::
/* Flush any buffered data. This triggers packetizing even a single
* byte into a separate frame.
*/
int
lsquic_stream_flush (lsquic_stream_t *);
/* Possible values for how are 0, 1, and 2. See shutdown(2). */
int
lsquic_stream_shutdown (lsquic_stream_t *, int how);
int
lsquic_stream_close (lsquic_stream_t *);
As mentioned before, calling :func:`lsquic_stream_flush()` will cause the stream to packetize the buffered data. Note that it may not happen immediately, as there may be higher-priority writes pending or there may not be sufficient congestion window to do so. Calling "flush" only schedules writing to packets.
:func:`lsquic_stream_shutdown()` and :func:`lsquic_stream_close()` mimic the interface of the "shutdown" and "close" socket functions. After both read and write ends of a stream are closed, the "on stream close" callback will soon be called.
Stream return values
--------------------
The stream read and write functions are modeled on the standard UNIX read and write functions, including the use of the ``errno``. The most important of these error codes are ``EWOULDBLOCK`` and ``ECONNRESET`` because you may encounter these even if you structure your code correctly. Other errors typically occur when the user code does something unexpected.
Return value of 0 is different for reads and writes. For reads, it means that EOF has been reached and you need to stop reading from the stream. For writes, it means that you should try writing later.
If writing to stream returns an error, it may mean an internal error. If the error is not recoverable, the library will abort the connection; if it is recoverable (the only recoverable error is failure to allocate memory), attempting to write later may succeed.
Scatter/gather stream functions
-------------------------------
There is the scatter/gather way to read from and write to stream and the interface is similar to the usual "readv" and "writev" functions. All return values and error codes are the same as in the stream read and write functions we have just discussed. Those are actually just wrappers around the scatter/gather versions.
The scatter/gather functions themselves are also wrappers. LSQUIC provides stream functions that skip intermediate buffering. They are used for zero-copy stream processing.
Here is the callback itself. You can see it is very simple. If there is data to be processed,
it is printed to stdout.
Note that the data size (``len`` above) can be anything. It is not limited by UDP datagram size. This is because when incoming STREAM frames pass some fragmentation threshold, LSQUIC begins to copy incoming STREAM data to a data structure that is impervious to stream fragmentation attacks. Thus, it is possible for the callback to pass a pointer to data that is over 3KB in size. The implementation may change, so again, no guarantees.
When the fourth argument, ``fin``, is true, this indicates that the incoming data ends after ``len`` bytes have been read.
Read with callback: Example 2: Use FIN
--------------------------------------
The FIN indicator passed to the callback gives us yet another way to detect end-of-stream.
The previous version checked the return value of :func:`lsquic_stream_readf()` to check for EOS.
Instead, we can use ``fin`` in the callback.
The second zero-copy read example is a little more efficient as it saves us
an extra call to ``tut_client_on_read_v2``.
Here, we package pointers to the tut struct and stream into a special struct and pass it to
Now the callback becomes more complicated, as we moved the logic to stop reading from stream into it. We need pointer to both stream and user context when "fin" is true. In that case, we call :func:`lsquic_stream_shutdown()` and begin reading from stdin again to grab the next line of input.
We must specify not only the function that will perform the copy, but also the function that will return the number of bytes remaining. This is useful in situations where the size of the data source may change. For example, an underlying file may change size.
The :member:`lsquic_reader.lsqr_read` callback will be called in a loop until stream can write no more or until :member:`lsquic_reader.lsqr_size` returns zero.
The return value of ``lsquic_stream_writef`` is the same as :func:`lsquic_stream_write()` and :func:`lsquic_stream_writev()`, which are just wrappers around the "writef" version.
Writing to stream: Example 2
----------------------------
Here is the second version of the "on write" callback. It uses :func:`lsquic_stream_writef()`.
In our case, ``count`` is never larger than the number of bytes still left to write.
This is because the caller - the LSQUIC library - gets the value of ``count`` from the ``lsqr_size()`` callback. When reading from a file descriptor, on the other hand, this can very well happen that you don't have as much data to write as you thought you had.
Client: making connection
=========================
We now switch our attention to making a QUIC connection. The function :func:`lsquic_engine_connect()` does that. This function has twelve arguments. (These arguments have accreted over time.)
::
lsquic_conn_t *
lsquic_engine_connect (lsquic_engine_t *,
enum lsquic_version, /* Set to N_LSQVER for default */
- The first argument is the pointer to the engine instance.
- The second argument is the QUIC version to use.
- The third and fourth arguments specify local and destination addresses, respectively.
- The fifth argument is the so-called "peer context."
- The sixth argument is the connection context. This is used if you need to pass a pointer to the "on new connection" callback. This context is overwritten by the return value of the "on new connection" callback.
- The argument "hostname," which is the seventh argument, is used for SNI. This argument is optional, just as the rest of the arguments that follow.
- The eighth argument is the initial maximum size of the UDP payload. This will be the base PLPMTU if DPLPMTUD is enabled. Specifying zero, or default, is the safe way to go: lsquic will pick a good starting value.
- The next two arguments allow one to specify a session resumption information to establish a connection faster. In the case of IETF QUIC, this is the TLS Session Ticket. To get this ticket, specify the :member:`lsquic_stream_if.on_sess_resume_info` callback.
- The last pair of arguments is for specifying a token to try to prevent a potential stateless retry from the server. The token is learned in a previous session. See the optional callback :member:`lsquic_stream_if.on_new_token`.
specifying list of versions to enable in engine settings using :member:`lsquic_engine_api.ea_versions`.
There are a couple of more places in the API where this technique is used.
Server callbacks
================
The server requires SSL callbacks to be present. The basic required callback is :member:`lsquic_engine_api.ea_get_ssl_ctx`. It is used to get a pointer to an initialized ``SSL_CTX``.
In case SNI is used, LSQUIC will call :member:`lsquic_engine_api.ea_lookup_cert`.
For example, SNI is required in HTTP/3.
In `our web server`_, each virtual host has its own SSL context. Note that besides the SNI string, the callback is also given the local socket address. This makes it possible to implement a flexible lookup mechanism.
Engine settings
===============
Besides the engine API struct passed to the engine constructor, there is also an engine settings struct, :type:`lsquic_engine_settings`. :member:`lsquic_engine_api.ea_settings` in the engine API struct
can be pointed to a custom settings struct. By default, this pointer is ``NULL``.
In that case, the engine uses default settings.
There are many settings, controlling everything from flow control windows to the number of times an "on read" callback can be called in a loop before it is deemed an infinite loop and the circuit breaker is tripped. To make changing default settings values easier, the library provides functions to initialize the settings struct to defaults and then to check these values for sanity.
You can instruct the library to generate a timestamp and include it as part of the message.
Several timestamp formats are available. Some display microseconds, some do not; some
display the date, some do not. One of the most useful formats is "chromelike,"
which matches the somewhat weird timestamp format used by Chromium. This makes it easy to
compare the two logs side by side.
There are eight log levels in LSQUIC: debug, info, notice, warning, error, alert, emerg,
and crit.
These correspond to the usual log levels. (For example, see ``syslog(3)``). Of these, only five are used: debug, info, notice, warning, and error. Usually, warning and error messages are printed when there is a bug in the library or something very unusual has occurred. Memory allocation failures might elicit a warning as well, to give the operator a heads up.
LSQUIC possesses about 40 logging modules. Each module usually corresponds to a single piece
of functionality in the library. The exception is the "event" module, which logs events of note in many modules.
There are two functions to manipulate which log messages will be generated.
::
/* Set log level for all modules */
int
lsquic_set_log_level (const char *log_level);
/* Set log level per module "event=debug" */
int
lsquic_logger_lopt (const char *optarg);
The first is :func:`lsquic_set_log_level()`. It sets the same log level for each module.
The second is :func:`lsquic_logger_lopt()`. This function takes a comma-separated list of name-value pairs. For example, "event=debug."
Logging Example
---------------
The following example is adapted from a tutorial program. In the program, log messages
are written to a file handle. By default, this is the standard error. One can change
that by using the "-f" command-line option and specify the log file.
There are three functions: one to open a file, one to write a line into the file, and one to close the file. The lines are not interpreted.
In the engine API struct, there are two members to set: one is the pointer to the struct with the function pointers, and the other is the context passed to "kli_open" function.
A QUIC connection has two sets of connection IDs: source connection IDs and destination connection IDs. The source connection IDs set is what the peer uses to place in QUIC packets; the destination connection IDs is what this endpoint uses to include in the packets it sends to the peer. One's source CIDs is the other's destination CIDs and vice versa.
The CID returned by :func:`lsquic_conn_id()` is that used for logging: server and client should return the same CID. As noted earlier, you should not rely on this value to identify a connection!
You can get a pointer to the connection from a stream and a pointer to the engine from a connection.
Calling :func:`lsquic_conn_get_sockaddr()` will point ``local`` and ``peer`` to the socket addressess of the current path. QUIC supports multiple paths during migration, but access to those paths has not been exposed via an API yet. This may change when or if QUIC adds true multipath support.
