Version 5.6 released

Version 5.6 of the speedata Publisher is out. The last stable release (5.4) was at the end of February, so it has been about 100 days of work, 25 small patch releases in between, and quite a few new things. Get a coffee, there is a lot in this one.

I have to admit that I sat on some of these changes a bit longer than planned. Originally I wanted a stable release after about six weeks, but then one feature led to the next. You fix one thing and suddenly see three other things that should also be done. So here we are, three months later, with changes in XPath, fonts, tables, accessibility, HTML, and a few internal things.

XPath got serious

This is probably the biggest single change. The XPath engine now understands arrays and maps, the || operator for string concatenation, and fn:serialize(). If you have ever wanted to keep a small lookup table in your layout file without dropping into Lua. Now you can, directly in XPath:

<SetVariable variable="prices"
    select="map { 'small': 9.90, 'medium': 14.90, 'large': 19.90 }"/>
<Value select="$prices('medium')"/>

Arrays and maps are real XPath 3.1 data types, so they work the way the spec says they should. For me this is something that was missing for a long time. Many layouts have a few constants that do not really belong in the data file, but you also do not want them as string literals all over the layout. Now you put them in a map at the top and that is it.

The || operator is a small thing but very nice in daily use:

<Value select="'Hello, ' || $name || '!'"/>

concat() still works of course, but || is nicer to read when you have just two or three parts.

Variable fonts

The Publisher can now use variable fonts. Internally this works via harfbuzz subsetting (which is included in LuaTeX 1.24, see below). You specify the axis values, and harfbuzz produces the instance for you. This means one font file gives you the full design space of weight, width, optical size, whatever the font supports.

I wanted this for a long time and finally did it. Variable fonts are everywhere now (Inter, Recursive, all the new releases), and not supporting them felt more and more like a real gap.

A new match attribute on Record

If you have written XSLT, you know how match works on <xsl:template>: you write a pattern, and the engine picks the right template for each node. The <Record> element in the Publisher now has the same thing:

<Record match="product[@type='discount']">
    ...
</Record>

So far you had to write <Record element="product"> and then check inside the record what kind of product you are dealing with. Now you can split the cases at the top level. Much cleaner for catalogs with many different record types.

Tables: small things that add up

A few table changes worth mentioning:

• colspan="*" means “use the rest of the row”. No need to count columns when you just want a cell to go to the end.
• Alternating row colors now restart on each page. This was really a bug. On long tables the colors could drift, so the first row of page 2 had the wrong color if page 1 ended on the wrong row. Now it resets where the table continues.
• HTML tables support colspan and rowspan. Yes, you might say, of course they do, but until now, no, they did not. Now they do.
• Td rotation keeps images inside the cell instead of letting them escape. That one was fun to debug.

None of these are huge, but if you build tables every day, you notice.

Progress bar

You can now run sp --progress and get a small progress bar instead of looking at a terminal that seems to do nothing:

sp --progress

For large jobs this is a much better experience. Before this, if you had a 500-page catalog, you basically had to trust that the Publisher was working and not stuck somewhere. Now you see something happen.

Placeholder images

A small but very useful one: you can now use placeholder:// as a pseudo-URL for image sources:

<Image file="placeholder://400x300"/>

This gives you a grey placeholder of the requested size. Useful when you want to lay out a page before the final images are there, or for testing layout behaviour with images you do not have yet. No more folder full of dummy.jpg files.

PDF/UA accessibility

Accessibility is better now. The structure tree handling for Output/Text works properly, and InsertPages / SavePages (the trial typesetting commands) no longer break the structure tree. If you make accessible PDFs, this release fixes some real problems.

Memory and performance

Memory usage for data files is reduced. If you process big XML data files, you should notice. Also a few stack overflow issues with origin references in tables (<Tr>) are gone. Those were causing slowdowns on very large tables.

sp filter now runs on Lua 5.4

If you use sp filter for preprocessing, your scripts now run on Lua 5.4 instead of Lua 5.1. This is a big jump version-wise (integer/float distinction, goto, bitops, …). For most filter scripts nothing changes, but if you do anything tricky with numbers you may want to test.

The change is that filter mode switched from gopher-lua (5.1) to go-lua (5.4). The Publisher itself still runs LuaTeX with its own Lua 5.3, so this only affects the filter mode.

LuaTeX 1.24.0

Updated to the current LuaTeX. Mostly invisible from the layout side, but everything runs on top of it. Needed for variable fonts.

New API version (v1)

There is now a v1 of the HTTP API. The old v0 still works, so existing integrations keep running. v1 has cleaner endpoint naming and better error reporting. If you are starting a new integration, use v1.

Things that are gone

Two things got dropped:

• Debian package: I am not packaging .deb files anymore. The binary tarball works everywhere and the Debian package was too much extra work for what it gave. If you are on Debian/Ubuntu, just extract the tarball.
• macOS Intel binaries: Apple Silicon has been the default for a while now, and keeping Intel builds for the few users still on Intel was not worth it. If you still need Intel, you can run the ARM binary via Rosetta, or build from source.

If either of these affects you, let me know: I want to hear it.

New documentation

The documentation has been rebuilt. New look, better search, and the changelog entries now link directly to the commits. The static site is now generated with Hugo (was something custom before), which makes contributing much easier. If you find something wrong in the docs, the source files are markdown/adoc and you can open a PR.

There were also a lot of small things going into the documentation: spelling fixes, better examples, more cross-links. Boring work, but it adds up.

Plus all the small stuff

Things that did not get their own section but are in the release:

• Better error messages in several places (data.xml not found, splib loading failures, end-of-document)
• Box border, box height in lines, vertical alignment for images and boxes in paragraphs
• CSS ::marker pseudo-element support
• --generate-completion flag for shell completion (bash, zsh, fish)
• jardir option for system-provided Java JARs (Saxon integration)
• A bunch of XPath fixes: ForAll with empty selections, matches() with 3 arguments, the legacy xpath.lua getting nil handling right
• And quite a few small bug fixes that you can find in the detailed changelog.

Downloads as always at download.speedata.de. The full changelog is in the manual.

If you find a regression, please open an issue on GitHub. With 135 commits since the last stable release, there is always a chance that something slipped through.