vcswatch reports that
this package seems to have new commits in its VCS but has
not yet updated debian/changelog. You should consider updating
the Debian changelog and uploading this new version into the archive.
Here are the relevant commit logs:
commit fa17498a1e605123241237ec3ba2d9da9142004d
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 22 13:57:34 2026 +0200
Name the cookie size caps in htsbauth.h (#2004)
This names five cookie size caps in the installed `htsbauth.h`, because
the front ends read these values and should not copy the literals. No
value changes, so the object code of `htsbauth.o`, `htscore.o`,
`htsbundle.o` and `htslib.o` is identical and the exported symbols are
unchanged.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 1737d72e7d11c3c5397d7d59717e02e64897826b
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 22 11:36:30 2026 +0200
Read a login bundle named by a start URL (#1999)
A start URL ending in `#httrack-bundle=<name>` crawls the hosts in its
scope with the user agent, headers and cookies of `<name>.hts`. A
missing, broken or conflicting bundle stops the run before any request,
and no message shows a value from a bundle line. The headers stay off
plain http when any bundle URL is https, and a public suffix such as
`s3.amazonaws.com` scopes to that host alone.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit d44f7ae14121f40d323eaceaccf35eeb69e187cc
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 22 10:43:41 2026 +0200
Hide passwords in URLs and -%H headers from the logs (#2000)
The `-%H` log, `hts-log.txt`, the `-%v` display and the argument
messages hide login headers, the FTP `PASS` command and a URL password
(`user:***@`). As a backstop, every log hides each known password in a
login, and each header token of 8 bytes or more anywhere. The archive
and report outputs keep URLs as given (`doit.log`, `new.txt`, WARC,
WACZ, `--changes`, `-%M` and the `-#C` dump), because they record what
was crawled.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 28167ddd02e2efedabdad319197704ca4d0b1b07
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 21 18:34:30 2026 +0200
Copy the rewritten tag with memcpy, which writes no NUL (#2003)
The parser overwrites a tag in place with exactly its own length, so it
must not write a terminating NUL. `memcpy` says that, while `strncpy`
made CodeQL flag the size (alert 315), and the bytes copied are the
same.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 6a0aed8cef013cbd04576673033a1d766b9f9fc2
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 21 18:30:07 2026 +0200
Read dates after 2038 correctly on 32-bit builds (#2002)
On a 32-bit `time_t`, `timegm()` fails after 2038-01-19, so mirrored
files kept their download time and Retry-After dates were ignored. Every
date now goes through `hts_utc_seconds()`, the 64-bit day count from
#1982, and a file time past `time_t` is clipped to its last second. On
32-bit builds, a date before 1901 now stamps the file at 1901-12-13 and
gives a Retry-After of 0, as on 64-bit.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 0885bc79fbc577633936c50268aaa86eb6e511f7
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 21 16:08:05 2026 +0200
Make doit.log, the crawl logs and the progress lock owner-only (#2001)
On POSIX, doit.log, hts-log.txt, hts-err.txt, hts-ioinfo.txt, the
progress lock, the cache's new.txt and new.zip, and WebHTTrack's
winprofile.ini are now owner-only, because each can hold passwords. They
use the cookie jar's open helper, now `hts_fopen_private()`, which also
tightens an old doit.log that an earlier run left at 0644.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 3bbb56660a02fa6192cb802d0ea2ec8207908e08
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 21 13:56:21 2026 +0200
Use a log-out word for Clear login in nineteen catalogs (#1998)
Clear login forgets the saved login, so nineteen catalogs that said
"clear" or "delete" now say "log out", and Swedish gets the grammatical
"Webbläsaridentitet". Only value lines change, and a byte-wise check
shows every key line matches master.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 1d00f37c69fe8279f0173671b6322ba472b3c92d
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 21 13:01:33 2026 +0200
Drop framing headers and earlier duplicate Host, Cookie and Authorization lines given with -%X (#1996)
HTTrack now drops Content-Length, Transfer-Encoding, Connection and four
similar headers given with `-%X`, and prints a warning. A `-%X` Host,
Cookie or Authorization still replaces the engine's own line (#1337),
even when written with a space before the colon. A blank before the
colon is removed, and a malformed or leading folded line is dropped.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit d329ab7a048819d0cc0812434447abfaf94464f9
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 21 12:22:01 2026 +0200
Keep a public-suffix cookie from reaching every host under it (#1995)
A cookies.txt line such as `.com TRUE / FALSE 0 SID x` is now loaded
host-only, because its cookie would go to every `.com` host the crawl
reaches. Refusing it instead would lose the engine's own
`Domain=localhost` cookies on reload, so every public-suffix cookie is
stored host-only.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 104955ccd08babafec29864e91207c2dd6816cda
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 21 12:15:49 2026 +0200
Add the browser login strings to the catalogs (#1997)
This adds the 14 WinHTTrack browser login strings
(xroche/httrack-windows#257) to `lang.def` and all 30 catalogs, with the
translations made for that work. The cookies-file tooltip now says the
file is written back too. Its English text changes in every catalog,
because each catalog finds a translation by its English text.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit c96688103d200ca8ffe38bb9795fddf7ee0b7313
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 21 11:09:31 2026 +0200
Read a Retry-After date without a weekday as a date, not as seconds (#1994)
A server may send a `Retry-After` date without the weekday, such as `06
Nov 1994 08:50:37 GMT`. httrack read its first word as a number of
seconds and waited 6 seconds. Now the value counts as seconds only when
nothing but spaces follows the digits, so `12 x` is also refused rather
than read as 12.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit c07ab0bfb149cfce65f6b8ae1dfed46e27a99b81
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 21 08:57:19 2026 +0200
Make the Modern theme find main buttons by class and show a custom --field (#1993)
Each page's main button now carries a `primary` class, and the Modern
theme selects it by that class. It no longer matches `name="nextBtn"`,
an `onclick` substring or an `OK` value. The page glow is now
translucent and painted on `body`, so a `--field` set in user.css on
`:root` or `body` shows through.
I rendered all 22 pages in headless Chrome, and the same buttons are
highlighted before and after. The default glow colours match the old
ones within a few RGB units. Test 594 now fails when the class is
missing, misplaced or doubled, or when the theme matches by name again.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit bb8a0abb15c8345ee9507f9d2364d0e5acc182ba
Author: Xavier Roche <roche@httrack.com>
Date: Tue Oct 20 22:11:07 2026 +0200
Name the leftover cookies temp files and the Android login on the login page (#1992)
The login page now names the `.tmp` files a stopped run leaves next to
the cookies file. Each one holds a full copy of the cookies. The page
also tells Android readers that the app handles the cookies file, so
they do not look for an export step.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit da8f7bd3f80c3649d97f8a1c10031b3969153194
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date: Tue Oct 20 11:28:31 2026 +0000
Bump github/codeql-action from 4.38.3 to 4.39.0 (#1991)
commit 19f3f16a8bbd0e242cc96c404ab2fcc1f46eabaf
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date: Tue Oct 20 11:18:18 2026 +0000
Bump src/coucal from `9f797c9` to `9f49c5b` (#1990)
commit ba6f6bc6de143f48959e2b2d35869a7dab712093
Author: Xavier Roche <roche@httrack.com>
Date: Tue Oct 20 10:54:03 2026 +0200
Read each tag once when choosing Sec-Fetch-* headers (#1989)
This fixes the fuzz-htsparse timeout in OSS-Fuzz issue 575485238. The
code choosing the Sec-Fetch-* headers rescanned a tag for every link in
it, and read unclosed tags to the end of the page. It now reads each tag
once, and an unquoted `<` ends the tag.
The `<` rule changes how malformed HTML is read, including an unquoted
value that holds a `<`. It only affects request headers, so it can
change a Sec-Fetch-Dest but never the links httrack finds.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 108dee813f3d8e1f7397858efcfe479e81eb6576
Author: Xavier Roche <roche@httrack.com>
Date: Mon Oct 19 16:05:34 2026 +0200
Mark the current option tab with aria-current (#1988)
Each option page now marks its own tab with `aria-current="page"`, and
both themes style that tab with one rule. The Modern theme no longer
needs a selector per page. The default theme's rule sits in `:where()`,
so a plain `.tabCtrl a` rule in a user stylesheet still overrides it.
Closes #1987
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 97b3804a90d35ab92152b108bab3738fa1f68599
Author: Xavier Roche <roche@httrack.com>
Date: Mon Oct 19 13:24:50 2026 +0200
Let --cookies-file read a pasted document.cookie string (#1986)
`--cookies-file` now also reads a document.cookie string, the `a=1; b=2`
line a browser console prints, and drops the quotes the console puts
around it. HTTrack sends those cookies as session cookies to the domain
of the first start URL, and never rewrites that file. The project's own
`cookies.txt` is still read as a Netscape jar only. A Netscape line with
an empty expiry, as Python's `MozillaCookieJar` writes a session cookie,
now loads, so one row of test 651 is inverted. An early exit such as
`--why` now leaves the `--cookies-file` untouched, where master rewrote
it. The Netscape loader now skips a UTF-8 BOM and knows wget's header.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 7b0f376418698c48885d0de802280ef4d2b4dbd1
Author: Xavier Roche <roche@httrack.com>
Date: Mon Oct 19 12:15:04 2026 +0200
Document how to mirror a site behind a login (#1985)
This adds `html/login.html`, which explains how to mirror a site behind
a login or a CAPTCHA with the cookies file and User-Agent of a browser.
The log warning for a bot challenge now also names `--user-agent`, and
test 487 checks it.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit d99292ef327dfb6a3c91a6d021f00ef8d3ae9ad1
Author: Xavier Roche <roche@httrack.com>
Date: Mon Oct 19 12:02:28 2026 +0200
Restore the select chevron and button polish in the Modern theme (#1984)
This builds on the first commit of #1979, which restores CSS-only polish
such as the select chevron and the blue primary buttons. Our commits
give focus outlines the primary colour, and put page and attribute tests
in `:where()`. A plain `#pageContent` rule in `~/.httrack.css` then
overrides them without `!important`. The second commit of #1979 is left
out, because the review rejected its French spacing changes and its
layout tuning.
Co-authored-by: Alain BROYE <alain.broye@live.fr>
---------
Signed-off-by: Alain BROYE <alain.broye@live.fr>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Alain BROYE <alain.broye@live.fr>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit c87b17fca6dbcb03cba6248517e5cabbc65f5f28
Author: Xavier Roche <roche@httrack.com>
Date: Mon Oct 19 09:51:24 2026 +0200
Check the cookie jar's mode on the opened file in the self-test (#1983)
The cookie self-test now checks the jar's file mode with `fstat()` and
`fchmod()` on an open stream, instead of `stat()` and `chmod()` on its
name. This clears CodeQL alert 313, a time-of-check race.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 67510a96aa79061ba6990cd764216d9470621735
Author: Xavier Roche <roche@httrack.com>
Date: Sun Oct 18 23:16:07 2026 +0200
Keep each cookie's scope, expiry and flags, and save the jar back to --cookies-file (#1982)
The cookie jar now keeps each cookie's Secure and HttpOnly flags,
whether subdomains get it, and its expiry, capped at 400 days. Secure
cookies go out over https only. A cookie loaded from a file may carry
4096 bytes of name and value, and HTTrack logs a warning when the jar is
full.
With `--cookies-file`, that file alone is the jar. HTTrack rewrites it
when the crawl ends, keeps the old file as `<file>.bak`, and never
leaves the path without a complete file. If the rewrite would lose a
line, HTTrack leaves the file unchanged and logs a warning. A repeated
cookie, an unreadable file and an unknown comment are examples. curl
reads the written file, and so does Python's `http.cookiejar` with
`ignore_expires=True`.
Replace and delete match on name, domain and path only, as RFC 6265
says, because older builds marked every saved cookie as a domain cookie.
The WebHTTrack tooltip `LANG_COOKIEFILETIP` still says "preload" and
changes with the catalogs.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 7c5452dddd93986fdb6bccebae2abb64b8bff399
Author: Xavier Roche <roche@httrack.com>
Date: Sun Oct 18 13:05:50 2026 +0200
Ignore cookies a site sets for another domain, and delete expired ones (#1981)
A crawled host could set a cookie for any other host with `Domain=`,
even for a public suffix such as `com`. Now such a cookie is ignored,
and an expired `Max-Age` or `Expires` deletes the stored cookie.
Three smaller changes follow RFC 6265. An unknown attribute such as
`SameSite` is no longer stored as a second cookie. An empty value like
`a=` is now kept and sent. A two-digit year below 70 means 20xx.
Test 570 changes one expectation. A quoted `domain="example.com"` was
stored with its quotes and never sent. Now the cookie is ignored,
because that Domain does not match the host.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit a4e5f8bff464bb6b3b259a3a7cc5a5b735313422
Author: Xavier Roche <roche@httrack.com>
Date: Sun Oct 18 11:37:36 2026 +0200
Load the HttpOnly cookies a cookies.txt export marks with #HttpOnly_ (#1980)
`--cookies-file` skipped every `#HttpOnly_` line of a browser or curl
export as a comment, so the login cookie was lost. The loader now strips
that prefix, and other `#` lines are still comments.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 20e63c9c4e1d7713a351500475d1e03750095b31
Author: Xavier Roche <roche@httrack.com>
Date: Sat Oct 17 16:58:33 2026 +0200
Show the WinHTTrack Accept headers window in the guide (#1978)
The WinHTTrack Browser ID image now shows the Accept headers button, and
the guide's Accept headers entry shows the new window, both from
httrack-works 3d1dfbc. Rerunning `tools/doc-images.py` also changed two
Android images, from the earlier Android reshoot 4b9abf7. I left those
out on purpose, so this PR touches only the Windows images.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 67d3a17479e056be97059a7cec0ee6cac426c686
Author: Xavier Roche <roche@httrack.com>
Date: Sat Oct 17 16:25:09 2026 +0200
Document the Accept headers box in the guide (#1977)
The guide now has an Accept headers entry on the Browser ID page, under
the anchor `opt-accept-headers`. httrack-android will point its new
Accept page's help at `#droid/opt-accept-headers`, so test 239 now pins
that id with the other Android targets.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit b6d8d69b62a0983ae67231eb02b391e420368da1
Author: Xavier Roche <roche@httrack.com>
Date: Sat Oct 17 14:31:26 2026 +0200
Send a real browser's Accept header for each kind of link (#1976)
`--accept 'KIND=VALUE'` now sets the Accept for one kind of link.
Wherever `-%Q` already sends Sec-Fetch-*, a Chrome, Edge or Firefox
User-Agent also gets that browser's Accept values, header order and
`en-US` Accept-Language. A User-Agent with no table still sends the old
header block byte for byte, which test 603 pins. WebHTTrack gains the
matching box on its Browser ID page, the twelve `Accept*` profile keys
and the strings for the other front ends.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 7bccc61f95720bcb1c62d5ca8c0e1acc75786ea6
Author: Xavier Roche <roche@httrack.com>
Date: Sat Oct 17 10:49:41 2026 +0200
Write the -%H header log into a new output directory (#1975)
With `-%H`, httrack now creates the `-O` directory before it opens
`hts-ioinfo.txt`, so a first crawl into a new directory gets its header
log. A failed open now logs a warning, where it used to fail silently.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit d7aeadc58b133eebb4f55b7fa34bb17f60726931
Author: Xavier Roche <roche@httrack.com>
Date: Sat Oct 17 10:16:27 2026 +0200
Keep the https scheme in the Referer header (#1974)
A page fetched over https sent its subresources an `http://` Referer,
because the header always wrote `http://` before the host. The header
now keeps the referring page's scheme, so an ftp page's links carry
`ftp://` too. The rule that sends no https Referer to an http URL does
not change.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 752f815e316503469d96cd988216b70bcff7a1ac
Author: Xavier Roche <roche@httrack.com>
Date: Sat Oct 17 08:35:44 2026 +0200
Build the Public Suffix List into httrack in place of libpsl (#1973)
httrack now carries its own copy of the Public Suffix List, so
`Sec-Fetch-Site` gives the same answer on Windows as elsewhere. This
replaces the optional libpsl from #1972. A distribution passes
`--with-psl-file` so that httrack reads the packaged list, which its
updates keep current, and Debian now does. A weekly canary reds once the
built-in copy is 60 days behind upstream.
`src/htspsl_data.c` is generated by `tools/psl-gen.py`, so review the
generator rather than the data. Test 599 runs the list's own test
vectors.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit bf0c3e56fff2a610c6a1894886474515c07cccaf
Author: Xavier Roche <roche@httrack.com>
Date: Fri Oct 16 19:28:49 2026 +0200
Use the Public Suffix List to decide same-site when libpsl is installed (#1972)
With libpsl, `Sec-Fetch-Site` now treats `a.github.io` and `b.github.io`
as different sites, as a browser does. Without libpsl, httrack keeps
comparing the last two labels of the host name. `configure` finds libpsl
on its own, and `--with-psl` makes it required. It also fixes host
parsing from #1970. `example.com.` now counts as a host of its own, the
userinfo ends at the last `@`, and a host such as `0x7f.0.0.1` matches
only itself.
Windows keeps the old rule for now, because vcpkg's libpsl port needs
ICU there. Debian's `libpsl.pc` does not list libidn2 and libunistring,
so test 222 adds them itself to run its static link.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit afc5da298a5185d94caad2d1c70c3a14ae514842
Author: Xavier Roche <roche@httrack.com>
Date: Fri Oct 16 18:38:19 2026 +0200
Show theme names capitalised in the WebHTTrack theme menu (#1971)
The theme menu on the welcome page now shows "Modern" instead of the
folder name "modern". The label is the folder name with a capital first
letter and with `-` and `_` turned into spaces. The menu still posts the
folder name, so saved choices keep working.
The label needs a new `${listlabel:}` directive. `${liststr:}` is
unchanged, because a project category name can contain a tab.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 2b61ffdcd343ea0cb072cc51cedd06a15e9a8970
Author: Alain BROYE <contact@alainbroye.fr>
Date: Fri Oct 16 17:55:57 2026 +0200
Add optional Modern theme to WebHTTrack (#1965)
First CSS-only step for #1960, updated to use the native theme folders
introduced by #1966 and #1968.
## What this PR does
- Ships Modern as `html/server/themes/modern/theme.css`.
- Uses the existing `--ui-*` hooks, the shared `palette.css` tokens and
the native `page-*` body classes.
- Leaves theme discovery, selection and persistence entirely to
WebHTTrack's server-side theme system.
- Keeps `user.css` after the selected theme, so user overrides still
win.
- Leaves Help and Manual unchanged and with their own presentation.
- Adds no Modern JavaScript, loader, browser storage or theme picker.
Dialogs and the stepper are intentionally not part of this PR; those
remain for a separate JavaScript step after agreement in #1960.
## Packaging
`html/Makefile.am` installs the built-in Modern stylesheet under
`html/server/themes/modern/`, and the install manifest includes it.
The existing theme-discovery test removes shipped themes from its
isolated "no installed themes" fixture before testing that case.
## Tests
`592_palette-shared.test` now also checks every shipped
`themes/*/theme.css` `var()` reference against the shared palette, the
native UI variables and the theme's own variables. It continues to check
that UI pages load `style.css` → `/server/theme.css` → `user.css`.
`594_webhttrack-modern-theme.test` checks the theme behaviour itself:
- Classic remains the default even when Modern is installed.
- Modern appears in the native theme menu.
- The server accepts and persists the Modern choice.
- `/server/theme.css` imports `themes/modern/theme.css`.
- The Modern stylesheet is served as CSS.
- The Modern choice survives a server restart.
- Switching back to Classic removes the theme import.
Targeted upstream tests pass: `592_palette-shared.test`,
`594_webhttrack-modern-theme.test`, and `595_webhttrack-themes.test`.
I also checked all 21 native WebHTTrack pages at 1440 px, 720 px and 390
px in both light and dark mode (126 states): no overflow, missing theme
resource, remote dependency or JavaScript error.
---------
Signed-off-by: Alain BROYE <alain.broye@live.fr>
commit b9f5d12f77f2d12ec2b5b03da1aae2b9c6fc804e
Author: Xavier Roche <roche@httrack.com>
Date: Fri Oct 16 17:51:18 2026 +0200
Send the Sec-Fetch-* headers a browser sends for each link (#1970)
With a browser User-Agent, HTTPS and loopback requests now carry the
`Sec-Fetch-*` headers. Their values come from the tag or stylesheet rule
that named the link. HTTPS page loads also carry
`Upgrade-Insecure-Requests`, and `-%Q0` turns all of them off. A link
the parser cannot classify sends none, because a wrong value can get a
request refused where no value passes.
Same-site compares the last two host labels, so `a.github.io` and
`b.github.io` read as same-site where a browser says cross-site. The
error only goes that way, so a request never looks more cross-site than
a browser would make it. `lien_url` in `htsopt.h` gains a tail field,
which keeps the soname.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 64b164c4164c35efca64e753ffb1d66ac134a201
Author: Xavier Roche <roche@httrack.com>
Date: Fri Oct 16 14:14:27 2026 +0200
Let WebHTTrack load themes from ~/.httrack/themes (#1968)
A user can now add a theme without root, by copying its folder into
`~/.httrack/themes/`. When it has the same name as an installed theme,
it wins, and all of that theme's files come from the user's folder.
A theme folder holds styles and assets, so the server refuses HTML files
in one. Otherwise it would expand them as page templates, session id
included.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit af8e28337e628f9c0ec2cc236296631a39692bfb
Author: Xavier Roche <roche@httrack.com>
Date: Fri Oct 16 13:54:07 2026 +0200
Send the client hints that match a Chrome or Edge User-Agent (#1969)
When `-F` names Chrome or Edge, HTTPS requests now carry the three
client-hint headers that browser sends by default. Without them, a
browser User-Agent contradicts the rest of the request. The brand list
follows Chromium's own formula for the User-Agent's major version, and
Chrome older than 108 gets no hints because its formula differed. Other
browsers get no hints either, and `-%Q0` turns them off.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit f64895db3e4bcbe8b65e29c3922ce70effad1b66
Author: Xavier Roche <roche@httrack.com>
Date: Fri Oct 16 13:38:52 2026 +0200
Stop asking for gzip when resuming a partial download (#1967)
A resumed download no longer asks for gzip, because a compressed reply
cannot be appended to the partial file. `back_add()` set the "Range in
use" flag and then reset the request before sending it. The flag is now
set after that reset, and test 596 checks the headers the server
receives.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 2bd3a3d70a2132c56bd20369a3b34e3f5040ebb8
Author: Xavier Roche <roche@httrack.com>
Date: Fri Oct 16 12:28:20 2026 +0200
Let WebHTTrack list themes from a folder and remember the choice (#1966)
A folder under `html/server/themes/` that holds a `theme.css` is a
theme, and the welcome page lists the installed ones next to the
language menu. Saving a project rewrites `~/.httrack.ini` whole, so the
server keeps the choice in `~/.httrack-ui.ini`.
Every page links `/server/theme.css`, which the server generates as an
`@import` of the chosen theme, so the theme's own relative URLs keep
working. A chosen theme whose folder is gone falls back to Classic, and
the choice is kept until the folder comes back.
The stylesheet of the Modern theme from #1965 can move into this folder.
Refs #1960.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit b38c7f7356ed482fc1da041ba85625adc611b866
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 15 16:20:13 2026 +0200
Let WebHTTrack themes restyle fonts, bars and single pages (#1964)
A user stylesheet can now change the UI font and size, the title and tab
bars, the background rings and the wordmark through `--ui-*` variables.
Each styled page's `<body>` also has a `page-<name>` class, so a theme
can target one page. This prepares for the Modern theme proposed on
#1961, which can then rely on variables rather than on selectors that
may change.
The default look does not change: all 21 styled pages render
pixel-identical to master in light and dark mode. The `#siteNavigation`,
`hr` and `.tbl*` rules style no page, so this PR deletes them. Test 592
now requires every colour in style.css to come from a variable.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit eea78a86d93be960b62b87a67de0ca02f62e816b
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 15 14:39:50 2026 +0200
Read a cached record's headers in linear time (#1963)
ProxyTrack's ARC and ZIP readers now append header lines in linear time,
because each append used to measure and copy every header read so far.
The oss-fuzz testcase took over two minutes and now takes 0.4s, and the
bug dates from 2012, so every release has it.
Test 593 converts an archive with three million header lines under a 60s
deadline. Master's code already overruns it at one million lines, and
the fix takes 2s.
Fixes oss-fuzz issue 573879282.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit dfaf398955f876f394b59cb07d6b783da2d2759d
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date: Wed Oct 14 18:06:06 2026 +0000
Bump github/codeql-action from 4.38.2 to 4.38.3 (#1962)
commit 44993927c5e5fd1f22dade90f94e604f95103bb0
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 14 15:36:58 2026 +0200
Add dark mode and a user stylesheet to WebHTTrack (#1961)
WebHTTrack now follows the system dark mode, as the documentation
already does, and loads a user stylesheet after its own. That file is
`~/.httrack.css`, or else `.httrack.css` in the projects folder, or else
`httrack.css` next to `httrack.conf`.
The docs and the UI now take their colours from one file,
`html/palette.css`. It is installed twice, because Debian ships the docs
and the UI in separate packages. Packagers who list files one by one
need the two new ones: `$(datadir)/httrack/html/palette.css` and
`server/user.css`. Light mode renders pixel-identical to master on all
15 UI pages and on the doc pages I checked.
This answers the request in #1960.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit fa8eccaf5b61ab083274f0858d6d0b4609b58cf9
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 14 10:36:02 2026 +0200
Find the host in a legacy save name whatever case the link spelled it in (#1959)
A cache written before 3.40 stores an absolute save name, so the reader
recovers the relative part by looking for the URL's host inside it. The
search was byte-exact while `htsname.c` lowercases the host directory it
writes. A mixed-case host in any of a site's own links therefore left
the name unreducible, and after the mirror moved the record was dropped.
Check the bound first. `index_join_root` re-validates whatever substring
the search offers, with `hts_path_is_relative_contained`, before
joining. A wider search changes which substring reaches that gate and
never bypasses it.
Two limits worth knowing. `src/htscache.c:736` reads the same field
byte-exactly with no host fallback, so httrack's own reader still drops
such a record. That wants its own issue. And test 590 drives the `.ndx`
reader only, so test 589's token-identity pin is all the coverage the
new.zip copy has.
Closes #1958
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 66702dbfef26f959ee571c8737efa9c176598de7
Author: Xavier Roche <roche@httrack.com>
Date: Tue Oct 13 22:55:06 2026 +0200
Refuse a cache or catalog name that leaves its own directory (#1956)
Two readers took a filename off disk and joined it to a root without
checking it first:
- ProxyTrack opened an absolute save name out of a cache record and sent
that file to the client. The guard above the join looked only for a `..`
component.
- htsserver read a catalog from outside `lang/` when `lang.def` named
one with a separator. It also overflowed a 1024-byte stack buffer on a
name past about 1013 bytes.
Every join now goes through one helper per file, so the six ProxyTrack
sites and the two htsserver branches cannot disagree again. A refusal
also reaches the log. It used to be computed and dropped, so a store
that quietly lost records gave an operator nothing to go on.
One loss is deliberate. A pre-3.40 cache holding an absolute save name
drops those records when the store is named with no parent directory.
That shape is the escape, so the join cannot tell the two apart.
Tests 587 and 588 fail on master, and under ASan 588 reports the
overflow as a 2010-byte write. Only the `.ndx` format is reachable from
a test. Test 589 pins the new.zip reader as the copy store.c already
calls it.
That closes the three open `cpp/path-injection` alerts in code scanning
(286, 297, 298). Two more were dismissed as reviewed, each with its
reason on the alert.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit bce4f1a0c7c89c98212273bbb7a846c5d3b3e6d1
Author: Xavier Roche <roche@httrack.com>
Date: Tue Oct 13 22:31:14 2026 +0200
Stop cache repair allocating the size a damaged zip header claims (#1957)
When httrack repairs a damaged cache zip, it allocated the compressed
size read from each entry's header. A truncated cache could make it ask
for up to 2GB per entry. It now copies the data in 8KB chunks, and a
truncated entry fails the repair with no large allocation.
One behavior changes. A size of 2GB or more used to become negative, so
the entry was counted as recovered with its data skipped. Now it fails
like any other truncated entry. The repair also fails now when closing
an output file fails, which it used to ignore. The test covers the
repaired zip but not the temporary file. The PR also moves the `p < 64`
test in `htsparse.c` before the `html[p]` read, which CodeQL flagged.
That read was in bounds already.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
commit aa918855a249945ba54b58e98aeae3615afda905
Author: Xavier Roche <roche@httrack.com>
Date: Tue Oct 13 18:29:42 2026 +0200
Delete the legacy query parsers the query differential kept (#1955)
The `-#test=querydiff` differential kept frozen copies of the query
parsers that #1877's shared iterator replaced. This deletes them. Test
541 now pins `url_query_value` directly:
- the first field with an `=` wins,
- the value is clipped to the buffer,
- the buffer is emptied when nothing matches.
The differential also fuzzed that clipping, and nothing in `fuzz/`
reaches it. So `-#test=querysweep` keeps the fuzzing half, with no
second parser. It reads every random query at eight buffer sizes behind
a poisoned buffer. The widest read is the oracle for the narrow ones.
The review found a second gap, where dropping an empty query field
turned no test red, and that case went to `htsname_selftest.c`.
`legacy_hts_query_key_stripped` stays, because the span differential
still compares against it. The iterator has not shipped in a release
yet, which the issue asked to wait for.
Closes #1878
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit b1847bfc88e2f6d50988c84d2b4a2a0df9b522a7
Author: Xavier Roche <roche@httrack.com>
Date: Tue Oct 13 14:58:53 2026 +0200
Fail the suite when a script pipes a builtin into grep -q (#1954)
Both sites that shipped with a pipe into `grep -q` were written after
AGENTS.md banned it. Test 425 then failed on a working binary, and the
man-page gate printed "man/html sync OK" for the mistake it guards
against. This scans the tracked scripts and workflows for a `printf` or
`echo` piped into `grep -q`. A builtin producer is the one that bites,
because bash flushes it per line where an external producer
block-buffers its few lines into one write.
The scan also allows an env prefix, since `LC_ALL=C grep` is this tree's
house style. That found one more site, in `62_lang-integrity.test`,
which now matches a here-string.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 7d8212c7e7395f69dca0487746f2e34777d962dd
Author: Xavier Roche <roche@httrack.com>
Date: Mon Oct 12 23:04:13 2026 +0200
Mirror every font and image a stylesheet names (#1953)
A stylesheet file lost its assets two ways. The `//` in an absolute
`url()` put the script scanner into a line comment. Only a newline ends
one, so no later `url()` on that line was read, and minified CSS is one
line. A sheet pulled in by `--near` is also parsed at depth 1. A link
survives that depth only when its tag and attribute name an embedded
asset. A `url()` has no tag, so the font stayed external.
Both halves key on the document's own mime, not on the parser's current
CSS state. A page holding `<textarea><style></textarea>` leaves that
state set over real javascript, where keying on it disabled a genuine
`//` comment. A `<style>` block therefore keeps the old behaviour.
Test 586 fails on master for the font and for the second `url()`. It
also carries three decoy urls: one behind a CSS comment, one behind a
javascript `//`, one behind that unclosed `<style>`.
Closes #139
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit a4c0b3b37d7dbda9e529d94da586ffda3d1d927c
Author: Xavier Roche <roche@httrack.com>
Date: Mon Oct 12 22:04:40 2026 +0200
Fix the man-page gate passing when only httrack.1 changed (#1952)
The gate asked `printf | grep -qx` whether `man/httrack.1` had moved.
grep exits on the match, so printf takes SIGPIPE on a later path.
pipefail then reports a miss, and the gate prints "man/html sync OK" for
the mistake it guards against. Measured here at 1 run in 2000 on a
60-path PR, and the same shape made test 425 fail on the FreeBSD leg.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 4343242c6048b09df6b56c78dbbdcbc58742c44b
Author: Xavier Roche <roche@httrack.com>
Date: Mon Oct 12 21:53:59 2026 +0200
Fix the race test 425 loses on the FreeBSD leg (#1951)
The `printf | grep -qx` assertion in test 425 can fail on a shim that
works. bash writes one line per write(2), so grep exits on its match,
the second of three lines, and printf takes SIGPIPE on the third.
`pipefail` makes that the test's failure, and the tier-2 FreeBSD leg
lost that race on master, reported as issue #1942. Matching from a
here-string removes the pipe.
Two cases also got wider, because each let a shim mutant through. The
comma list now carries three pieces, which tells the translation loop
from a pair handled on its own. The option case adds a single-dash
spelling, which the shim translates too.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 6a2dcbaeefbed9afbc63aa93339a420e00bd2966
Author: Xavier Roche <roche@httrack.com>
Date: Mon Oct 12 19:32:19 2026 +0200
Run the four sanitizer legs from one reusable workflow (#1950)
The four sanitizer legs ran the same eight steps. They differed in the
compiler, the flags, which tests they run and what their runtime's
options are. `sanitizer.yml` holds the steps now and takes those four as
inputs. Bumping an action or changing the collect step is one edit
rather than four. The legs now sit side by side in about twenty lines
each.
This does not shrink the tree: the typed inputs cost roughly what the
duplication saved. The gain is that the mechanism exists once.
The check names move, because a reusable workflow prefixes its caller:
`sanitize (ASan+UBSan, gcc)` reports as `sanitize (ASan+UBSan, gcc) /
check`. None of the four is a required context.
Two tests had to follow. 432 read `ASAN_OPTIONS` out of the job's steps,
and a calling job has none, so it reads the options input too. 496
wanted a callable workflow named by tier2.yml, and ci.yml calls this
one, so any caller that runs on its own now counts. Both gained a
mutant, and the commits say what each one proves.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit cfe87a647d235678c3a779a3fceea1a43f8e49ee
Author: Xavier Roche <roche@httrack.com>
Date: Mon Oct 12 17:15:57 2026 +0200
Run the test suite under clang's UndefinedBehaviorSanitizer (#1949)
gcc and clang share the `-fsanitize=undefined` name and not the check
list. Pointer arithmetic on a null pointer is one clang has and gcc does
not. That is how #1946 sat in proxytrack for years with the `sanitize`
leg green, and why it reached us from OSS-Fuzz, which builds with clang.
Reverting #1947 reds this leg on five tests and names the line. It runs
about six minutes against a 17 minute critical path, so it costs a
runner and no wall clock.
The `fuzz` leg already builds clang with UBSan. It only replays corpora
though, and reaches no proxytrack code, so the same coverage there would
have needed a new harness per subsystem.
The runtime is not in the shared libraries, so the build is static. 561
of the 591 tests run. Three that assert an installed shared object
needed a new skip, and the job floors the pass count so a later flag
cannot mass-skip quietly.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 7c6abd21531d1009f72797a67241cefa09d125a4
Author: Xavier Roche <roche@httrack.com>
Date: Mon Oct 12 15:55:15 2026 +0200
Export the check for whether a '-' token is an option (#1948)
Adds `hts_rule_would_be_read_as_opt()` to `httrack-library.h`. The
Windows GUI can then ask the engine whether a `-` filter reads as
options, rather than keep its own copy of `cmdl_opt()`
(xroche/httrack-windows#240). The function replaces the internal
`cmdl_opt()` with the same behavior, so this only adds an export and the
soname does not change.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
commit 6bdf09b74d80d2e6d50ab023ff4275fcdf1b046b
Author: Xavier Roche <roche@httrack.com>
Date: Mon Oct 12 14:56:22 2026 +0200
Build the URL list without offsets on a null pointer (#1947)
`PT_Enumerate()` built each URL's slot offset on a null pointer, at four
sites, which is undefined. Clang's UBSan killed five proxytrack tests on
master. gcc cannot see the class, which is why CI always passed, and
OSS-Fuzz builds with clang. Walking the NUL-terminated strings in slot
order gives the same pointers with no offsets at all. A 31-entry listing
comes back byte for byte identical to master.
Driving the enumerator from `fuzz-arc` is what puts this under a clang
build in CI, and it found two more bugs there. `subtree=1` handed coucal
a NULL duplicate set, which it reads, so any non-empty entry was a hard
SEGV. No caller in the tree passes 1. The set is always built now, and
the dedup is skipped under subtree, where two distinct keys can reduce
to one item. Both String buffers also leaked whenever nothing matched,
which a request for an absent path reaches.
The corpus replay reds on reverting any of the three. Tests 147 and 386
pin the listing's order, and test 120 gains the absent-path request.
Closes #1946
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 3cdc23f647f83d85968551a8ff200b2da2cd38f5
Author: Xavier Roche <roche@httrack.com>
Date: Mon Oct 12 12:24:25 2026 +0200
Fall back to the next address before the timeout expires (#1945)
A connect that hangs is meant to abandon its address and try the next
one. Its deadline measured from its own clock, which started after the
resolve, while the timeout racing it measured from the request. On a
short timeout the deadline then fell due only once the timeout had
abandoned the slot. FreeBSD lost the fallback that way where Linux kept
it.
The duplicate clock is deleted, so the deadline reads the base the
timeout reads and the two cannot drift again. Nothing in the suite drove
that path before, so test 19 now pins a dropped SYN as well, which only
Linux can produce. It covers the path and not the skew, since a
cache-hit resolve leaves no skew to see.
A TLS handshake longer than the deadline can still abandon a healthy
connection on a multi-address host. That predates this change, which
narrows the window it needs.
This fixes the one new failure in #1942, which closes itself on the next
green Tier-2 run.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 18f2ca63f4d4f6cd6406096b281d4cc256e23475
Author: Xavier Roche <roche@httrack.com>
Date: Mon Oct 12 11:42:20 2026 +0200
Stop reallocating a typed array that already has the room (#1944)
`TypedArrayEnsureRoom()` called the allocator on every append. Where the
capacity already covered the request, the realloc asked for the byte
size the block already had. ASan's allocator has no same-size path and
copied the whole store every time. The HTML rewriter appends once per
parser loop iteration, which on a page dense in `<` is once per byte.
The cost therefore grew with the square of the page size. The 550KB
reproducer takes 480s on 3.50.5 and 122s on master, against the 25s
OSS-Fuzz allows, and 0.3s here. Without a sanitizer it takes 0.14s
before and after, because glibc hands the same block back, so no
released binary was affected.
Skipping the realloc leaves a never-grown array's store NULL, where
`realloc(NULL, 0)` used to make it non-NULL. `TypedArrayAppend()`
therefore refuses a zero count now, rather than taking the address of a
store that is not there. Only clang's UBSan reports that, and the suite
runs under gcc.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit c60b0b43957bd21a66a0262b29d9005d0fb341c6
Author: Xavier Roche <roche@httrack.com>
Date: Mon Oct 12 09:09:49 2026 +0200
Stop following Vue, Angular and Alpine binding attributes as links (#1943)
httrack no longer treats the value of a framework binding attribute as a
link. This covers Vue `:x`, `v-x` and `@x`, Angular `[x]` and `(x)`, and
Alpine `x-bind:` and `x-on:`. Their values are JavaScript, so
`:data-src="item.src"` used to request `/item.src` and rewrite it to
`item.html`. Now it stays as written. As a side effect, text that only
looks like a binding, such as `alt='see :x="t1.gif"'`, is no longer
fetched either.
Test 571 pinned `v-bind:src="vbind.gif"` as a followed link, and this PR
reverses that expectation on purpose. `ng-src` stays followed, because
it takes a URL or an interpolation where a binding takes an expression.
Every other attribute is unchanged, and the whole `html/` doc tree
mirrors byte-identically to master.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit f9f8e64cb4ec766cb1d822e2a69da74b43b754d8
Author: Xavier Roche <roche@httrack.com>
Date: Mon Oct 12 08:51:46 2026 +0200
Stop fetching data-src placeholders, and follow img data-src with --near (#1941)
This narrows one thing and widens another: httrack stops requesting
lazy-load placeholders in `data-src` and `data-srcset`, and `--near` now
follows real `<img data-src>` images. A placeholder such as `{{url}}`, a
JSON blob or the word `lazy` stays in the page as written. The `--near`
half completes #136. On test pages for lazysizes, lozad, WordPress, Vue
and Angular, requests fell from 34 to 24 and junk files from 10 to 0.
The same real images were saved, and the lazysizes, lozad and WordPress
pages are byte-identical to master's.
The rule refuses a value holding `{ } < > "` before any `?`, or equal to
one of nine placeholder words. Two kinds of real file are therefore no
longer fetched through `data-src`. One is a file named exactly `lazy`,
which the test pins on purpose. The other is a raw path holding braces,
such as `img/{p2}.gif`.
Closes #136
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 2faa046b34b766de1a20dd4d0881936a30ac2a42
Author: Xavier Roche <roche@httrack.com>
Date: Sun Oct 11 03:28:39 2026 +0200
Name the first saved page as the root of a .mht archive (#1939)
`index.mht` carried no RFC 2387 `start=` parameter, so a reader takes
the first part as its root. The engine saves a page after some of its
own images, so the archive opened on a GIF. It now names the first page
it saves.
The header is written before the part, so the part's file is opened
first. Otherwise a page the engine cannot read back gets named as a root
the archive does not hold. Parts arriving before the first page wait in
`~index.mht.part`, and `url_savename` drops a `~`, so no mirrored file
can collide with it.
`html/cmdguide.html` also said `-%M` keeps each resource's original URL,
which it never did.
Closes #223
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
commit 3b00fe13d2a92d8929669f3e5659311fc80be2a2
Author: Xavier Roche <roche@httrack.com>
Date: Sun Oct 11 03:09:21 2026 +0200
Keep test 424's pty open until the reader has drained it (#1940)
Test 424 closed its pty slave before reading the master, so the child
held the last one. macOS flushes a pty on that final close, so the
master then reads zero bytes. The parent read nothing, and the test
reported `asked 0 times` on a correct exit status. It failed once in 455
macOS suite runs since 2026-09-25, and never in about 4,990 runs
elsewhere.
The reader now keeps its slave and stops on an idle poll once the child
is reaped, since EOF never arrives while it holds one. Linux cannot
reproduce the flush, so no leg of the suite guards this fix. The `typed`
arm also now checks for the option list it exists to prove, which
nothing asserted.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
commit 14874e1732918c491c10ee813ba706a1da20c748
Author: Xavier Roche <roche@httrack.com>
Date: Sun Oct 11 00:56:09 2026 +0200
Document that -%s reuses same-size files on a first crawl (#1938)
The `-%s` help text now says the option keeps a file already on disk
when its size matches, even on a first crawl. The man pages are
regenerated from it. This is what #113 asked for, but the old wording
described the option only as an update hack. Test 343 already covers the
behaviour.
Refs #113
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 8fc1e9589a52c85590b7609304809e40be5667be
Author: Xavier Roche <roche@httrack.com>
Date: Sat Oct 10 17:02:22 2026 +0200
Scan modern image, font and video types from foreign hosts by default (#1937)
The default scan rules named only png, gif, jpg, jpeg, css and js. A GUI
crawl therefore threw away webp, avif, svg, fonts and video sitting on
another host. Issue #139 is titled for ttf, so the font set is the point
of it rather than a bonus.
The test deserves more attention than the list. This value is one string
typed out in four places, three of them in other repositories. Every
check either repo has compares the copies against each other. A typo
propagated to all of them therefore keeps them in agreement, and
silently stops that type ever being fetched again. Test 574 reads the
value out of `winprofile-keys.tsv` and splits it token by token the way
the GUI does. It then crawls a two-host fixture, so `+*.wof2` fails a
crawl instead of failing nothing.
A profile that omits `WildCardFilters` resolves it to a literal compiled
into whichever front end reads the file, `htsserver.c` here. Such a file
therefore changes meaning across this bump. The generated table supplies
no runtime default, and only proves the front ends' literals agree.
WinHTTrack always writes the key explicitly, so its own profiles are
safe. The exposure is a file written by something that omits defaults,
or written before the key existed.
One new reach worth knowing: `+*.svg` admits foreign-host SVG. The
engine parses `image/svg+xml` for links, so those files are now
rewritten the way a same-host SVG already was.
`httrack-windows` validated this commit with their own bindings harness,
against a negative control, and their PR #235 carries the identical
literal.
Part of #139. The front ends land their copies separately.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit a91b0665ac1a6d1982fe2124e3f1dbdd4eb01159
Author: Xavier Roche <roche@httrack.com>
Date: Sat Oct 10 16:08:56 2026 +0200
Save images named only in img srcset or data-srcset with --near (#1936)
With `--near`, httrack now also saves images that a page names only in
`<img srcset>` or `<img data-srcset>`. The parser already scans and
rewrites both attributes, so only the near-link table changes, and the
output without `--near` is identical to master.
This is part of #136. `<img data-src>`, the commonest lazy-loading
attribute, stays refused. Its values are often placeholders, and master
already fetches those as links on in-depth pages, so following it needs
a parser fix first.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit d867581fce5fa13b5c4718909f5dbe86e2d2cf4b
Author: Xavier Roche <roche@httrack.com>
Date: Sat Oct 10 15:31:15 2026 +0200
Keep the files of a mirror whose cache is gone (#1933)
With the cache deleted, the previous mirror's own copy is the only
record of what a link was named. `-%N0` computed that name and resumed
from it. The default `-%N2` named the link `.delayed` instead, so the
on-disk resume never fired and every file came down again.
A link already in the mirror now keeps the name the previous mirror gave
it. So a server that retypes it keeps the old extension. `-%N0` behaves
the same way, and the resume this unblocks is `back_add()`'s own,
unchanged.
Closes #1926
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit b8067dd1d5c705d7decfb674fd7ded12fc339b34
Author: Xavier Roche <roche@httrack.com>
Date: Sat Oct 10 15:24:32 2026 +0200
Send every stored cookie in the request, not the first eight (#1932)
The request `Cookie:` header carried at most eight cookies. A site whose
login sets more than eight got only the first eight back, so it answered
as if nobody had signed in. The line is now bounded by bytes rather than
by a count. A pair that does not fit whole is dropped whole and logged,
because half a pair reads as the next cookie's value.
The cookie line goes out last, after the mandatory headers and the
`--header` set. The caller says how many bytes still follow it, the
blank line that ends the request plus the POST body. Without that, a jar
big enough to reach the buffer's edge left the blank line no room, and
the engine refused its own request. Test 573 is that case.
Closes #101
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 34263cb492b41b5c802cb139f6add800d2a10918
Author: Xavier Roche <roche@httrack.com>
Date: Sat Oct 10 15:23:54 2026 +0200
Document that hts_cancel_file_push takes the save path (#1935)
The `hts_cancel_file_push` comment now says the function takes the
file's local save path, which is what the engine matches and what every
front end passes. A cancel that matches no transfer is now logged
instead of silently ignored, and the engine still matches the same name
as before. Test 253 first pushes the URL, which must be reported, and
then the save path, which must not.
Closes #146
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 6313697f95be214f354691886ee1409606c9c3cc
Author: Xavier Roche <roche@httrack.com>
Date: Sat Oct 10 15:19:55 2026 +0200
Drop a link that cannot be allocated instead of aborting the crawl (#1934)
When httrack runs out of memory while recording a link, it now drops and
logs that link instead of aborting. A link whose string copy fails is
removed whole, so the link table never holds an entry with NULL fields.
The self-test sits in `htsback_selftest.c` beside `backnew`, because
`htscore.c` has no self-test file of its own.
Closes #57
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 5dea190fac40ffc3254c58f222b0dd00881b5c06
Author: Xavier Roche <roche@httrack.com>
Date: Sat Oct 10 15:05:31 2026 +0200
Keep the previous cache when the last one cannot be repaired (#1931)
When the last run's cache cannot be repaired, httrack now keeps the
previous run's cache instead of overwriting it with the damaged one. The
damaged `new.zip`, `new.lst` and `new.txt` move to
`hts-cache/damaged.*`, so `old.zip`, `old.lst` and `old.txt` still
describe one run. `--clean` deletes the damaged files too. When there is
no older cache, nothing changes. The new cases in the reconcile
self-test fail if either the zip or the `.lst` guard is removed.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 2e683a2618efbe19726ddc1b0161702e7466a030
Author: Xavier Roche <roche@httrack.com>
Date: Sat Oct 10 14:16:24 2026 +0200
Stop configure when the coucal submodule is missing (#1927)
configure now stops with an error when `src/coucal` is empty, and the
error names the fix. Before this, a clone made without its submodules,
or a GitHub tag archive, passed configure and then failed at `make` with
`coucal.h: No such file or directory`. Users of the `make dist` release
tarball are not affected, because it already contains coucal.
Closes #247
Closes #294
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit a7f8286dc6ebee17e839aa8790912e9d4f100994
Author: Xavier Roche <roche@httrack.com>
Date: Sat Oct 10 14:00:41 2026 +0200
Give every .mht part its own Content-ID (#1930)
The Content-ID generator escaped `%` to `X`, and a path can carry `X`
literally. So `/p.gif` and `/pX2egif` came out with the same id, and one
MIME part served both images. The marker is now `-`, because
`escape_in_url()` always escapes it.
Every id changes shape, which reaches only archives written from now on,
because nothing reads a `.mht` back. Test 577 is the first test in the
tree that writes one.
Closes #1925
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 5881d3b7d6837c8f49c5be0a14b3ab91c79fc790
Author: Xavier Roche <roche@httrack.com>
Date: Sat Oct 10 12:17:31 2026 +0200
Remove the WebHTTrack Capture URL button that only said not implemented (#1929)
WebHTTrack's add-URL page loses its Capture URL button, which only
showed a "not yet implemented!" alert, and the guide now describes
capture for WinHTTrack only. Wiring the button up is not a small change,
because `--catchurl` runs a blocking proxy that the user's browser must
be pointed at. Users who need it can still run `httrack --catchurl` from
a terminal. The `lang.def` strings stay, because WinHTTrack's catalogs
share them.
Closes #300
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 16447b272f3c4e00936742033ac98fd12fff0e57
Author: Xavier Roche <roche@httrack.com>
Date: Sat Oct 10 11:52:36 2026 +0200
Send the 404 page body when WebHTTrack's index.html is missing (#1928)
WebHTTrack now sends the 404 page body for `GET /` when
`html/server/index.html` is missing. Before this, it declared
`Content-length: 17` and sent nothing, because that request takes the
POST code path. Now only HEAD goes without a body. The server reads HEAD
from the request line, so a refused HEAD gets no body either.
The two `send()` calls also ignored a short write, so they now loop
until the whole buffer is out. That is hardening only: the socket is
blocking, so the reporter could not have hit it.
Closes #83
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit da907917ffa95e4eaacdf8cac404a0fb819a53d3
Author: Xavier Roche <roche@httrack.com>
Date: Sat Oct 10 10:37:13 2026 +0200
Accept a lang.def symbol as a catalog key (#1923)
htsserver now accepts a catalog line keyed by its `LANG_` symbol when
the lookup by English text finds nothing. WinHTTrack uses the same rule
since httrack-windows#229. Catalogs can move to symbol keys one at a
time once httrack-android's string generator accepts them too, because
it still keys on the English text. htsserver reads each catalog twice,
symbol lines first. A repeated English text such as "Start" binds by its
occurrence, so it could otherwise take a symbol's slot. No catalog has a
symbol key yet, so nothing changes on merge.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 8bf5f6c1ffb68a4086c8702ba54bca875e241755
Author: Xavier Roche <roche@httrack.com>
Date: Sat Oct 10 10:07:48 2026 +0200
Override lintian's font warnings for the manual's web font (#1924)
Lintian 2.142.0 now warns about the web font in the manual, which reds
the deb job on every branch. The manual's pages load that font by a
relative path, so the font stays in place and the two tags are
overridden.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit a5ea4e20e325c410afa3cbab5a85c5f69c898cf2
Author: Xavier Roche <roche@httrack.com>
Date: Sat Oct 10 00:00:00 2026 +0200
Delete the openSUSE acknowledgement step their spec no longer needs (#1922)
openSUSE Factory's spec now names the pkgconfig file and the metainfo,
and its icon line globs the scalable SVG. All three acknowledged paths
are covered, so rpm's own unpackaged-files check carries the whole
assertion. The remaining guard could not fire anyway, because it greps
for a literal name where their coverage is a glob. Worse, an `%exclude`
line naming the icon would make it report the opposite of the truth.
A Docker Hub outage reds five other tier-2 legs tonight, so the canary
cannot confirm this. If the glob does miss the SVG, the next weekly run
reds naming that file, on a job that gates nothing.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit a442479dd56600aa51b7adda6ee173445f40d6fb
Author: Xavier Roche <roche@httrack.com>
Date: Fri Oct 9 18:03:20 2026 +0200
Run the parser differential's self-check weekly in tier-2 (#1921)
Tier-2 now runs `tools/parser-diff.sh --self-check`, which plants a
link-finding mutant in each direction and fails unless the differential
reports both. Nothing ran it before, so the harness behind every parser
change had no check of its own. It builds the engine four times and took
6m25s on four cores, which is why it goes here and not onto the
pull-request path.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 6563f99a1bbc8ab9edfbdc301feeda26ad95703e
Author: Xavier Roche <roche@httrack.com>
Date: Fri Oct 9 14:36:45 2026 +0200
Diff the mirrors of two httrack builds to check a parser change (#1917)
`tools/parser-diff.sh` crawls `tools/parser-corpus/` with the newest
release tag's httrack and with the working tree's. It then compares the
two mirrors, the `hts-changes.json` manifest and a normalized
`hts-log.txt`. `--self-check` proves it can see a change at all. It
plants a widening and a narrowing parser edit and requires both to be
reported, then bumps the version and requires silence. `make check`
stays green on either parser mutant, so this corpus is the only thing
that sees them.
Every corpus target has one referrer, so a construct the parser stops
recognising loses its own file rather than being reached by a sibling.
The scripts come from the server's own `/jsmime/` routes, because a
static `.js` is only ever `text/javascript`. Three of the four spellings
`is_javascript_mime_type` accepts would otherwise go uncovered.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 329a784255e8ef2708b2705c929a08a94dcffaa2
Author: Xavier Roche <roche@httrack.com>
Date: Fri Oct 9 14:04:54 2026 +0200
Test which attribute values the parser follows and which it leaves alone (#1919)
A new crawl test pins which attribute values `src/htsparse.c` follows
and which it leaves alone. Every target has exactly one referrer, so a
construct the parser stops recognising loses its own file instead of
being reached by a sibling.
Two of the four crawls turn the `-%P` last-resort scan off. With the
scan on it picks up the same values the contract parser does, so a dead
`href` parser still passed every quoting assertion. The `data-args`
fixture pins the `(` and `,` arms of the scan's opener set, so dropping
either character from `=(,` reds the test.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 5ef0856a2fafd560f982969be5a486fb2e75570c
Author: Xavier Roche <roche@httrack.com>
Date: Fri Oct 9 09:53:10 2026 +0200
Read a quoted HTML attribute value whole (#1918)
`HTS_SPACES` holds both quote characters, so the scan ate the opening
quote and cut a quoted value at its first inner space. `rel="alternate
stylesheet"` read as `alternate`, and single-file mode then treated the
sheet as a plain asset rather than CSS. The fix trims real whitespace on
both sides of the quote, so `http-equiv=" Refresh"` still yields
`Refresh`. `strfield()` matches a prefix and ignores the length it is
handed, so one stray leading byte would have broken that match.
One case moves for the `htsparse.c` caller. `http-equiv=""` used to take
its value from the attribute behind it, so `http-equiv="" refresh` armed
the refresh state. It now matches nothing, like any unknown pragma. Test
569 pins that, and test 568 pins the whitespace class on both sides of
the quote. The Set-Cookie value start shares the root cause and goes in
its own PR. `is_space()` trims the makeindex anchor title the same way,
and I left that alone because changing it would reword the index page of
every mirror.
Closes #1915
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit f2dd831ee2641ead3d7364c641a07735aa3f4a4c
Author: Xavier Roche <roche@httrack.com>
Date: Fri Oct 9 09:47:40 2026 +0200
Keep the quotes of a quoted cookie value (#1920)
RFC 6265 4.1.1 makes the DQUOTEs part of a quoted `cookie-value`, 5.2
strips only whitespace, and 5.4 sends the value as stored. So
`Set-Cookie: sess="a b"` has to come back as `sess="a b"`. The scan
skipped `is_space()`, which counts a quote as whitespace, so the jar
stored `a b"` and sent that. Web Platform Tests pins the unbalanced case
in `cookies/value/value.html`, under "Set value with unbalanced leading
quote". `Set-Cookie: a="` now stores `"` where master stored no cookie
at all.
The same line governs every cookie-av value, so a quoted `path="/d"`
keeps its quotes now. Test 570 pins that. Both spellings fail to match
`/d`, so nothing a server sees moves. One trap is older than this change
and gets easier to hit. `cookie_add()` refuses a value over 1024 bytes
and `treathead` discards its return. The two kept quotes therefore drop
a cookie whose quoted value is 1023 bytes, silently.
#1915 names this scan as one of its siblings.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 6a5ab63825d795871599bdfc6f5a9b95a19e8ee3
Author: Xavier Roche <roche@httrack.com>
Date: Fri Oct 9 00:38:17 2026 +0200
Skip whitespace with strspn() instead of hand-written loops (#1916)
htslib.h, htsserver.h and proxy/proxytrack.h each defined the same four
character classes, and htsescape.c carried a private fifth copy of one.
htslib.h now holds the only definition, and 24 skip loops across eight
files call `strspn()` or `strcspn()` over the matching set string in
htssafe.h instead. Two htssitemap.c loops of the same shape stay as they
are on purpose. That parser reads a sized buffer with no terminator, so
`strspn()` would run past its end.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit dc31e31e34afe3b55de2c52851d4708363941b5a
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 8 19:40:25 2026 +0200
Decode scan rules the way crawled links are decoded (#1914)
Scan rules are now decoded the way the parser decodes a crawled link.
The part before any `?` is decoded, so `%41` becomes `A` and `%20`
becomes a space, and the query part is kept as is. This holds for rules
on the command line, in a `-%S` file, and those applied mid-mirror
through `hts_setfilters()` or `hts_addfilter()`.
On the command line, a rule holding a space was stored as `%20` and
never matched a crawled link. In exchange, a rule with no `?` that used
`%20` to match a query now matches the path. A query rule must now
include the `?`, as in `-*?*x%20y*`. `--why` decodes its URL path the
same way.
Closes #1889
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 054b62207e6792e835923a9b0027a90f5bd4689b
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 8 19:06:10 2026 +0200
Let a lone -N take its template in the first argv pass (#1913)
Makes a lone `-N` take its template in the first argv pass, so the
template is never read as a URL. `-N '' URL` now mirrors with the
default structure (it used to take the URL as the template). `httrack -N
T` with no URL inside an existing project now resumes it instead of
overwriting its doit.log.
Four self-test rows for the `-N` template are inverted on purpose: they
used to expect `used == 1` and now expect 2.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit e3730853f7b06d8accc906368cd64df9ac46836f
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 8 18:04:49 2026 +0200
Treat the path after a clustered -O as the option's value (#1912)
The first argv pass and the `-O` pre-pass now walk an option cluster the
way the main loop does, through one shared walker. So `httrack -qO PATH`
prints usage like `-q -O PATH`, `&` reads as `%` (`-q&P URL`), and `-%O
URL` no longer writes into a directory named after the URL. The pre-pass
also gave `-PO PROXY OUT URL` the proxy as its path, and master has the
same bug. It no longer looks for `-O` inside a word another option took
as its value (`--user-agent -Oz`). A clustered `-q%g` or `-q%C` now
refuses a following `-` word, but a lone `-%g -utm` still works.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 1840af31eac99b4823610290c478b2b17169ede9
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 8 16:14:12 2026 +0200
Keep the case of .httrackrc values (#1911)
An option in `.httrackrc` keeps the case of its value, so a user-agent,
footer, filter or path is used as written. The keywords on, off and
(none) are still read in any case. The `-%A` and `-%w` values are still
lowercased, because the engine compares them in lowercase. A `=` inside
a value is kept, and `key = value` with spaces around the `=` now works.
A quoted `"off"` on a switch such as `index` now turns it off, where it
used to turn it on.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 2b4953ad8486e3ad4b9cc9e0bc05614991a06727
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 8 14:07:11 2026 +0200
Renumber the argv quote test to 562, because #1907 also took 561 (#1910)
#1907 and #1909 both added a test numbered 561, and
`tests/check-test-names.sh` fails when two tests share a number. This
renames the one #1909 added, which landed second.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit eaba65b00b5b8fa0b726da5c54b44e682c16a393
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 8 13:34:09 2026 +0200
Stop removing quotes from command-line arguments (#1909)
httrack no longer removes quotes from its command-line arguments,
because the front ends now pass final arguments. `.httrackrc`,
`hts-cache/doit.log` and the WebHTTrack command line still decode their
own quotes.
A quoted URL now stops httrack with an error, and a quoted `-%L` line is
logged and skipped. A quoted number is refused too. `-O` keeps its own
format, where a quote groups a comma, and in `-O1` mode a quote is plain
data. The engine no longer reads the path after `-O1` as a URL.
WebHTTrack sends its path with `-O1`, so a project name with a comma
stays one directory.
Only someone who types literal quotes on the shell sees a change.
WinHTTrack stopped adding its own quotes in
[httrack-windows#220](https://github.com/xroche/httrack-windows/pull/220).
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 51862ac958395c619c62f886efb78c928c793e99
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 8 13:21:43 2026 +0200
Keep what was already mirrored when the -#L cap is reached (#1907)
Reaching the `-#L` cap made the parser unwind out of the page it was
rewriting, so that page was lost too. A crawl could therefore end having
kept nothing, against the man page's promise that what was mirrored
before the abort is kept. On a 13-page fixture at `-#L6`, master keeps
zero files.
The cap keeps the abort, the message and the threshold PR #1419 gave it,
so `htsback.c` and tests 240, 360 and 364 are untouched. Only the
teardown changes. The parser stops following links once the heap has no
room, and the rest of the page then points at the live site. The leaves
it had already caught stay local and dangle, as they do in any aborted
mirror.
`hts_maxlinks_no_room()` holds back two slots rather than one, because
one parser step records a host's `robots.txt` and then the link that
named it. With one slot the engine takes it for the `robots.txt`, has
none left for the link, and unwinds again. Test 560 drives that pair
through a second host name.
The add-url drain gets the same verdict. A URL injected through
`hts_addurl()` into a full heap used to be dropped with the rest of the
queue. Nothing recorded a verdict, so the mirror finished green. Test
561 drives that through WebHTTrack, injecting link-free pages so the
parser's gate never sees the cap first.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 4b4071c11a9288cb6b56d5d79faf36e136344c8b
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 8 02:59:58 2026 +0200
Re-run a Windows leg whose wsl --update failed, and revive the lost-runner re-run (#1908)
When `wsl --update` fails and the distro then does not import, a new
step stops the Windows leg in seconds, and `windows-rerun-killed.yml`
re-runs it. The 403 behind it comes from one runner VM. This removes the
cached kernel package from #1656, because it installs kernel 5.10.16.3.
All 7 legs in the last 100 runs that used it failed the suite after 25
minutes.
This also revives the re-run of lost runners, dead since #1838. The
`windows gate` job fails on every red run, so the grader read it as a
real failure and refused every run. The grader now ignores that job when
its own gate step failed. A lost runner on the gate job still counts as
lost.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 402d890da43c8b542474abf78d1530062156cc93
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 8 02:09:52 2026 +0200
Keep TAB in doit.log, .httrackrc and the postfile request line (#1903)
These three readers deleted TAB, so `a<TAB>b` became `ab`. Now doit.log
keeps a TAB inside a value, while `.httrackrc` and a `>postfile:`
request line treat it as a space. `.httrackrc` also skips an indented
comment now, and the other readers (page templates, catalogs, profiles,
proxytrack, prompts) still drop TAB.
Section 6 of `01_engine-doitlog.test` asserted the old TAB drop, so its
expected line changes on purpose. A postfile request line of 999 bytes
or more no longer breaks the request. A 999-byte line used to end the
headers early, and a longer one sent its tail as extra headers. Now that
tail is dropped.
Form feed is still dropped, and a TAB now counts toward the 250-byte
`.httrackrc` and 1000-byte postfile line limits.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 3e3f39d8e36498e3e8f926a65b5b0d2ae9a98bde
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 8 01:10:31 2026 +0200
Strip an option value's quotes once, with or without -O (#1906)
Without `-O`, httrack stripped two pairs of quotes from an option value
instead of one, and it read the rc file twice. So `-F '""ua""'` sent
`ua` without `-O` and `"ua"` with it, and now both send `"ua"`. Every
front end passes `-O`, and the old and new binaries agree on the
argument lists they build, so only command-line runs without `-O`
change.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit c0de0ccfd9da94f061c1345fe942dc24f332e8a7
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 8 01:09:17 2026 +0200
Accept WebHTTrack form posts longer than 32 KiB instead of cutting them (#1902)
WebHTTrack now reads form posts of up to 8 MiB whole. A larger post gets
413, and an unreadable or empty Content-length gets 400 (master read an
empty one as up to 32 KiB).
The cap leaves room for the 1 MiB profile line of #1899, which step 4
posts twice and percent-encodes. The request buffer shrinks back to 32
KiB after a large post.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 43b442e988c7ba50111a8d0fe9666db34fd44bba
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 8 01:05:37 2026 +0200
Refuse an out-of-range level glued to -s, -K, -C and six other options (#1905)
`-K`, `-b`, `-s`, `-o`, `-u`, `-C`, `-%I`, `-%v` and `-%N` now read
their glued level with the same bounded reader as the other glued
options. A level above the highest one the engine uses is now refused
with "Value out of range in option". For example, `-s9`, `-o2` (or
`--generate-errors 2`), `-b2`, `-u3` and `-K4294967298` used to be
stored and the mirror ran. `-%I2` is still accepted, although the help
lists only `-%I` and `-%I0`.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit a6c8417281f21a24dd4e5518911f589cf904fb1d
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 8 00:16:21 2026 +0200
Share the filter list growth check with mirror start, and test it (#1904)
The scan rules read at mirror start had their own copy of the filter
list growth check, so they now call the shared `filters_grow()` helper.
Test 558 passes 1202 rules at `-#F128` and checks that none is dropped.
A real run cannot reach the `HTS_FILTERS_MAX` cap without a 2 GB list,
so the `filtergrow` self-test in test 538 still covers the cap. The only
change in behaviour is that the `-#F` hint printed when memory runs out
is now logged at info level rather than notice.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 58e7d9d3d9c61a5bab8816826aecb88a64a71d5d
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 7 19:43:45 2026 +0200
Skip over-long lines in config files instead of reading their tail as a new line (#1899)
The config files share one line reader, `htslines.c`. It consumes a line
too long for its buffer whole, so the tail is never read back as a new
line.
Behaviour changes:
- An over-long line in `.httrackrc`, `cookies.txt` or a lang catalog is
skipped with a warning.
- httrack reads `doit.log` and the WebHTTrack profile whole, up to 1 MiB
per line. A longer `doit.log` line stops the resume, and a longer
profile line is skipped.
- Trailing whitespace past the limit does not make a line too long.
- Continuation lines join the same way at every buffer size.
- A line that exactly fills the buffer no longer gives an extra empty
line.
- A resume no longer appends duplicate options to `doit.log`.
- `linput_trim`, `linput_cpp`, `rawlinput` and `cache_binput` are
removed. None was exported.
httrack-android lists its sources by hand, so its `Android.mk` must add
`htslines.c` and `htslines_selftest.c`.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 9bd0a0f8f1d06e6a66f441207c5cf131d8844942
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 7 18:45:47 2026 +0200
Stop the update purge from deleting a file named by the tail of a long path (#1901)
A hostile site could make `--update` delete a file in the mirror. It
only needed a URL whose `old.lst` line is over 1000 bytes, because the
purge read the tail of that line as another path. The purge now reads
`old.lst` whole, as it already reads `new.lst`, and skips any line that
does not fit. The clean-up of emptied directories now climbs only inside
the mirror, and it climbs with the log off too.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 4945dbf270e3c6acee82a8402fd9bf829da284c0
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 7 17:51:16 2026 +0200
Let a path given on the htsserver command line take effect (#1900)
A `path` pair given to `htsserver` now sets the projects folder, and it
wins over the `path=` line saved in `~/.httrack.ini`. So `webhttrack` no
longer passes `path ~/websites` on every launch, because that would
override the folder saved in the interface. The `lang` key keeps its old
precedence.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 53485c08461d5b2e974a3091b793e09ee78e988d
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 7 16:41:00 2026 +0200
Bind proxytrack's ICP socket to the ICP address it was given (#1898)
proxytrack now binds its ICP socket to the address given in its second
argument, rather than to the proxy address. A setup that gives two
different addresses now gets ICP on the second one, as the usage text
says. The new test needs a second loopback address (127.0.0.2), so it
skips on hosts such as macOS that route only 127.0.0.1 to loopback.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 27b9020f15f9d24b30f013558b8213d63d78f369
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 7 16:32:36 2026 +0200
Generate the htsserver, proxytrack and webhttrack man pages from their --help (#1897)
`make -C man regen-man` now builds all four man pages from each
program's `--help`. So `htsserver.1`, `proxytrack.1` and `webhttrack.1`
document their options, and test 02 fails when any page falls behind.
Each page keeps its fixed prose in `man/<page>.tmpl`, and `httrack.1`
comes out byte-identical to the committed one.
Most of the `makeman.sh` diff is re-indentation, so read it with `git
diff -w`. The three smaller programs print their `--help` as one option
per line now, because the generator reads that layout.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 413c0e1e8afc49313e05964c1030e10f0c1b1d41
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 7 15:27:13 2026 +0200
Read header parameters like charset and filename in one place (#1896)
Adds `hts_header_param()`, one reader for the parameters of a header
value, and uses it for the Content-Type charset, the Content-Disposition
filename and the Keep-Alive numbers. Two results change on purpose: a
charset that is not the first parameter is now found, and
`filename="a.txt"; size=3` now saves `a.txt` instead of `a.txt_;
size=3`.
These results also change:
- A Content-Disposition value of 250 bytes or more is now parsed, so a
`filename` followed by a long `filename*` now saves the `filename`.
- A charset value of 256 bytes or more is now dropped, where the old
code kept its first word.
- A Keep-Alive value of 64 bytes or more is now ignored, so the default
applies.
- Keep-Alive names match in any case.
- Keep-Alive values separated only by a space lose the second value.
A follow-up deletes the legacy parser copies in the self-test once this
lands.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
![[bug history graph]](https://qa.debian.org/data/bts/graphs/h/httrack.png){kind=link}