Changes in Varnish 6.5.0

For information about updating your current Varnish deployment to the new version, see Upgrading to Varnish 6.5.0.

A more detailed and technical account of changes in Varnish, with links to issues that have been fixed and pull requests that have been merged, may be found in the change log.

varnishd

Access Control Lists (ACLs)

The VCL compiler now emits warnings if network numbers used in ACLs do not have an all-zero host part (as, for example, "192.168.42.42"/24). By default, such ACL entries are fixed to all-zero and that fact logged with the ACL VSL tag.

Parameters

A new vcc_acl_pedantic parameter, when set to true, turns the ACL warnings documented above into errors for the case where an ACL entry includes a network prefix, but host bits aren’t all zeroes.

The solaris jail has been improved and can reduce privileges even further. There is now a new optional -j solaris,worker=... argument which allows to extend the effective privilege set of the worker (cache) process.

Other changes in varnishd

Some error messages improved in the VCL compiler.

Changes to VCL

VCL variables

A new obj.can_esi variable has been added to identify whether the response can be ESI processed.

Once resp.filters is explicitly set, trying to set a resp.do_* field results in a VCL failure. The same rule applies to beresp.filters and beresp.do_* fields.

The BACKEND VCL type now has a .resolve() method to find the effective backend directly from VCL. When a director is selected, the resolution would otherwise be delayed until after returning from vcl_backend_fetch or vcl_pipe:

# eager backend selection
set bereq.backend = bereq.backend.resolve();

It is now possible to manually set a Connection: close header in beresp to signal that the backend connection shouldn’t be recycled. This might help dealing with backends that would under certain circumstances have trouble managing their end of the connection, for example for certain kinds of resources.

Care should be taken to preserve other headers listed in the connection header:

sub vcl_backend_response {
    if (beresp.backend == faulty_backend) {
        if (beresp.http.Connection) {
            set beresp.http.Connection += ", close";
        } else {
            set beresp.http.Connection = "close";
        }
    }
}

Other changes to VCL

A failure in vcl_recv could resume execution in vcl_hash before effectively ending the transaction, this has been corrected. A failure in vcl_recv is now definitive.

There is a new syntax for BLOB literals: :<base64>:. This syntax is also used to automatically cast a blob into a string.

Behavior for 304 responses was changed not to update the Content-Encoding response header of the stored object.

VMODs

A new std.blobread() function similar to std.fileread() was added to work with binary files.

The shard director’s .add_backend() method has a new optional weight parameter. An error when a backend is added or removed now fails the transaction (or the vcl.load command in vcl_init), but an invalid weight does not result in a hard failure.

The shard director no longer outputs the (unused) canon_point property in backend.list commands.

varnishlog

The BackendReuse log record has been retired. It was named inconsistently with other places like stat counters where we use the words reuse and recycle (it should have been named BackendRecycle if anything).

The BackendOpen record can now tell whether the connection to the backend was opened or reused from the pool, and the BackendClose record will tell whether the connection was effectively closed or recycled into the pool.

varnishadm

The backend.set_health command can be used to force a specific state between sick and healthy or restore the automatic behavior, which depends on the presence of a probe. While forcing a backend to be sick would prevent it from being selected by a director, a straight selection of the backend from VCL would still attempt a connection. This has been fixed, and the command’s documentation was clarified.

varnishstat

A help screen is now available in interactive mode via the h key.

Again in interactive mode, the initial verbosity is now chosen such that fields selected via the -f or -I options are actually displayed without manually increasing the verbosity level.

Filtering using the -f option is now deprecated in favor of -I and -X options that are treated in order. While still present, the -f option now also works in order instead of exclusive filters first and then inclusive filters. It was also wrongly documented as inclusive first.

The JSON output slightly changed to more easily be consumed with programming languages that may map JSON objects to types. See upgrade notes for more details.

There are two new MAIN.beresp_uncacheable and MAIN.beresp_shortlived counters.

varnishtest

The process -expect-text command will wait an order of magnitude longer for the text to appear. It used to be too sensitive to any kind of timing disruption.

Changes for developers and VMOD authors

Build System

VMOD authors who would like to generate VCC files can now use the VARNISH_VMODS_GENERATED() macro from varnish.m4 for autotools builds.

Workspace API

The workspace API saw a number of changes in anticipation of a future inclusion in VRT. The deprecated WS_Reserve() function was finally removed, after the functions WS_ReserveSize() and WS_ReserveAll() were introduced in Varnish Cache 6.3.0.

On the topic of workspace reservation, the WS_Front() function is now deprecated in favor of WS_Reservation(). The two functions behave similarly, but the latter ensures that it is only ever called during a reservation. There was no legitimate reason to access the workspace’s front outside of a reservation.

In a scenario where a reservation is made in a part of the code, but used somewhere else, it is possible to later query the size with the new WS_ReservationSize() function.

The return value for WS_Printf() is now a constant string.

Other VRT / cache.h changes

  • Added VRT_DirectorResolve() to resolve a director

  • Added VRT_BLOB_string() for the default BLOB folding documented above

libvarnishapi

There are three new VSC arguments that can be set with the VSC_Arg() function:

  • 'I' to include counters matching a glob pattern

  • 'X' to exclude counters matching a glob pattern

  • 'R' to include required counters regardless of 'I' and 'X'

The 'f' argument is now deprecated and emulated with 'I' and 'X'. Filtering with 'f' used to check exclusions first and then inclusions, they are all tested in order and the first to match determines the outcome.

The 'R' argument takes precedence over regular filtering and can be used to ensure that some counters are present regardless of user configuration.

libvarnish

A VSA_BuildFAP() function has been added as a convenience to build a struct suckaddr (aka VCL_IP) from a Family, Address and Protocol components.

We added VRE_quote() to facilitate building literal string matches with regular expressions. It can be used to ensure that a user-defined string literal put inside a regular expression may not accidentally change the behavior of the overall expression.

The varnish binary heap implementation has been added with the VBH_ prefix for use with VMODs (via include of vbh.h).

VSB support for dynamic vs. static allocations has been changed:

For dynamic allocations use:

VSB_new_auto() + VSB_destroy()

For preexisting buffers use:

VSB_init() + VSB_fini()

VSB_new() + VSB_delete() are now deprecated.

eof