vcswatch reports that
this package seems to have new commits in its VCS but has
not yet updated debian/changelog. You should consider updating
the Debian changelog and uploading this new version into the archive.
Here are the relevant commit logs:
commit 5ef0856a2fafd560f982969be5a486fb2e75570c
Author: Xavier Roche <roche@httrack.com>
Date: Fri Oct 9 09:53:10 2026 +0200
Read a quoted HTML attribute value whole (#1918)
`HTS_SPACES` holds both quote characters, so the scan ate the opening
quote and cut a quoted value at its first inner space. `rel="alternate
stylesheet"` read as `alternate`, and single-file mode then treated the
sheet as a plain asset rather than CSS. The fix trims real whitespace on
both sides of the quote, so `http-equiv=" Refresh"` still yields
`Refresh`. `strfield()` matches a prefix and ignores the length it is
handed, so one stray leading byte would have broken that match.
One case moves for the `htsparse.c` caller. `http-equiv=""` used to take
its value from the attribute behind it, so `http-equiv="" refresh` armed
the refresh state. It now matches nothing, like any unknown pragma. Test
569 pins that, and test 568 pins the whitespace class on both sides of
the quote. The Set-Cookie value start shares the root cause and goes in
its own PR. `is_space()` trims the makeindex anchor title the same way,
and I left that alone because changing it would reword the index page of
every mirror.
Closes #1915
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit f2dd831ee2641ead3d7364c641a07735aa3f4a4c
Author: Xavier Roche <roche@httrack.com>
Date: Fri Oct 9 09:47:40 2026 +0200
Keep the quotes of a quoted cookie value (#1920)
RFC 6265 4.1.1 makes the DQUOTEs part of a quoted `cookie-value`, 5.2
strips only whitespace, and 5.4 sends the value as stored. So
`Set-Cookie: sess="a b"` has to come back as `sess="a b"`. The scan
skipped `is_space()`, which counts a quote as whitespace, so the jar
stored `a b"` and sent that. Web Platform Tests pins the unbalanced case
in `cookies/value/value.html`, under "Set value with unbalanced leading
quote". `Set-Cookie: a="` now stores `"` where master stored no cookie
at all.
The same line governs every cookie-av value, so a quoted `path="/d"`
keeps its quotes now. Test 570 pins that. Both spellings fail to match
`/d`, so nothing a server sees moves. One trap is older than this change
and gets easier to hit. `cookie_add()` refuses a value over 1024 bytes
and `treathead` discards its return. The two kept quotes therefore drop
a cookie whose quoted value is 1023 bytes, silently.
#1915 names this scan as one of its siblings.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 6a5ab63825d795871599bdfc6f5a9b95a19e8ee3
Author: Xavier Roche <roche@httrack.com>
Date: Fri Oct 9 00:38:17 2026 +0200
Skip whitespace with strspn() instead of hand-written loops (#1916)
htslib.h, htsserver.h and proxy/proxytrack.h each defined the same four
character classes, and htsescape.c carried a private fifth copy of one.
htslib.h now holds the only definition, and 24 skip loops across eight
files call `strspn()` or `strcspn()` over the matching set string in
htssafe.h instead. Two htssitemap.c loops of the same shape stay as they
are on purpose. That parser reads a sized buffer with no terminator, so
`strspn()` would run past its end.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit dc31e31e34afe3b55de2c52851d4708363941b5a
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 8 19:40:25 2026 +0200
Decode scan rules the way crawled links are decoded (#1914)
Scan rules are now decoded the way the parser decodes a crawled link.
The part before any `?` is decoded, so `%41` becomes `A` and `%20`
becomes a space, and the query part is kept as is. This holds for rules
on the command line, in a `-%S` file, and those applied mid-mirror
through `hts_setfilters()` or `hts_addfilter()`.
On the command line, a rule holding a space was stored as `%20` and
never matched a crawled link. In exchange, a rule with no `?` that used
`%20` to match a query now matches the path. A query rule must now
include the `?`, as in `-*?*x%20y*`. `--why` decodes its URL path the
same way.
Closes #1889
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 054b62207e6792e835923a9b0027a90f5bd4689b
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 8 19:06:10 2026 +0200
Let a lone -N take its template in the first argv pass (#1913)
Makes a lone `-N` take its template in the first argv pass, so the
template is never read as a URL. `-N '' URL` now mirrors with the
default structure (it used to take the URL as the template). `httrack -N
T` with no URL inside an existing project now resumes it instead of
overwriting its doit.log.
Four self-test rows for the `-N` template are inverted on purpose: they
used to expect `used == 1` and now expect 2.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit e3730853f7b06d8accc906368cd64df9ac46836f
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 8 18:04:49 2026 +0200
Treat the path after a clustered -O as the option's value (#1912)
The first argv pass and the `-O` pre-pass now walk an option cluster the
way the main loop does, through one shared walker. So `httrack -qO PATH`
prints usage like `-q -O PATH`, `&` reads as `%` (`-q&P URL`), and `-%O
URL` no longer writes into a directory named after the URL. The pre-pass
also gave `-PO PROXY OUT URL` the proxy as its path, and master has the
same bug. It no longer looks for `-O` inside a word another option took
as its value (`--user-agent -Oz`). A clustered `-q%g` or `-q%C` now
refuses a following `-` word, but a lone `-%g -utm` still works.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 1840af31eac99b4823610290c478b2b17169ede9
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 8 16:14:12 2026 +0200
Keep the case of .httrackrc values (#1911)
An option in `.httrackrc` keeps the case of its value, so a user-agent,
footer, filter or path is used as written. The keywords on, off and
(none) are still read in any case. The `-%A` and `-%w` values are still
lowercased, because the engine compares them in lowercase. A `=` inside
a value is kept, and `key = value` with spaces around the `=` now works.
A quoted `"off"` on a switch such as `index` now turns it off, where it
used to turn it on.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 2b4953ad8486e3ad4b9cc9e0bc05614991a06727
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 8 14:07:11 2026 +0200
Renumber the argv quote test to 562, because #1907 also took 561 (#1910)
#1907 and #1909 both added a test numbered 561, and
`tests/check-test-names.sh` fails when two tests share a number. This
renames the one #1909 added, which landed second.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit eaba65b00b5b8fa0b726da5c54b44e682c16a393
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 8 13:34:09 2026 +0200
Stop removing quotes from command-line arguments (#1909)
httrack no longer removes quotes from its command-line arguments,
because the front ends now pass final arguments. `.httrackrc`,
`hts-cache/doit.log` and the WebHTTrack command line still decode their
own quotes.
A quoted URL now stops httrack with an error, and a quoted `-%L` line is
logged and skipped. A quoted number is refused too. `-O` keeps its own
format, where a quote groups a comma, and in `-O1` mode a quote is plain
data. The engine no longer reads the path after `-O1` as a URL.
WebHTTrack sends its path with `-O1`, so a project name with a comma
stays one directory.
Only someone who types literal quotes on the shell sees a change.
WinHTTrack stopped adding its own quotes in
[httrack-windows#220](https://github.com/xroche/httrack-windows/pull/220).
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 51862ac958395c619c62f886efb78c928c793e99
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 8 13:21:43 2026 +0200
Keep what was already mirrored when the -#L cap is reached (#1907)
Reaching the `-#L` cap made the parser unwind out of the page it was
rewriting, so that page was lost too. A crawl could therefore end having
kept nothing, against the man page's promise that what was mirrored
before the abort is kept. On a 13-page fixture at `-#L6`, master keeps
zero files.
The cap keeps the abort, the message and the threshold PR #1419 gave it,
so `htsback.c` and tests 240, 360 and 364 are untouched. Only the
teardown changes. The parser stops following links once the heap has no
room, and the rest of the page then points at the live site. The leaves
it had already caught stay local and dangle, as they do in any aborted
mirror.
`hts_maxlinks_no_room()` holds back two slots rather than one, because
one parser step records a host's `robots.txt` and then the link that
named it. With one slot the engine takes it for the `robots.txt`, has
none left for the link, and unwinds again. Test 560 drives that pair
through a second host name.
The add-url drain gets the same verdict. A URL injected through
`hts_addurl()` into a full heap used to be dropped with the rest of the
queue. Nothing recorded a verdict, so the mirror finished green. Test
561 drives that through WebHTTrack, injecting link-free pages so the
parser's gate never sees the cap first.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 4b4071c11a9288cb6b56d5d79faf36e136344c8b
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 8 02:59:58 2026 +0200
Re-run a Windows leg whose wsl --update failed, and revive the lost-runner re-run (#1908)
When `wsl --update` fails and the distro then does not import, a new
step stops the Windows leg in seconds, and `windows-rerun-killed.yml`
re-runs it. The 403 behind it comes from one runner VM. This removes the
cached kernel package from #1656, because it installs kernel 5.10.16.3.
All 7 legs in the last 100 runs that used it failed the suite after 25
minutes.
This also revives the re-run of lost runners, dead since #1838. The
`windows gate` job fails on every red run, so the grader read it as a
real failure and refused every run. The grader now ignores that job when
its own gate step failed. A lost runner on the gate job still counts as
lost.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 402d890da43c8b542474abf78d1530062156cc93
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 8 02:09:52 2026 +0200
Keep TAB in doit.log, .httrackrc and the postfile request line (#1903)
These three readers deleted TAB, so `a<TAB>b` became `ab`. Now doit.log
keeps a TAB inside a value, while `.httrackrc` and a `>postfile:`
request line treat it as a space. `.httrackrc` also skips an indented
comment now, and the other readers (page templates, catalogs, profiles,
proxytrack, prompts) still drop TAB.
Section 6 of `01_engine-doitlog.test` asserted the old TAB drop, so its
expected line changes on purpose. A postfile request line of 999 bytes
or more no longer breaks the request. A 999-byte line used to end the
headers early, and a longer one sent its tail as extra headers. Now that
tail is dropped.
Form feed is still dropped, and a TAB now counts toward the 250-byte
`.httrackrc` and 1000-byte postfile line limits.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 3e3f39d8e36498e3e8f926a65b5b0d2ae9a98bde
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 8 01:10:31 2026 +0200
Strip an option value's quotes once, with or without -O (#1906)
Without `-O`, httrack stripped two pairs of quotes from an option value
instead of one, and it read the rc file twice. So `-F '""ua""'` sent
`ua` without `-O` and `"ua"` with it, and now both send `"ua"`. Every
front end passes `-O`, and the old and new binaries agree on the
argument lists they build, so only command-line runs without `-O`
change.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit c0de0ccfd9da94f061c1345fe942dc24f332e8a7
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 8 01:09:17 2026 +0200
Accept WebHTTrack form posts longer than 32 KiB instead of cutting them (#1902)
WebHTTrack now reads form posts of up to 8 MiB whole. A larger post gets
413, and an unreadable or empty Content-length gets 400 (master read an
empty one as up to 32 KiB).
The cap leaves room for the 1 MiB profile line of #1899, which step 4
posts twice and percent-encodes. The request buffer shrinks back to 32
KiB after a large post.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 43b442e988c7ba50111a8d0fe9666db34fd44bba
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 8 01:05:37 2026 +0200
Refuse an out-of-range level glued to -s, -K, -C and six other options (#1905)
`-K`, `-b`, `-s`, `-o`, `-u`, `-C`, `-%I`, `-%v` and `-%N` now read
their glued level with the same bounded reader as the other glued
options. A level above the highest one the engine uses is now refused
with "Value out of range in option". For example, `-s9`, `-o2` (or
`--generate-errors 2`), `-b2`, `-u3` and `-K4294967298` used to be
stored and the mirror ran. `-%I2` is still accepted, although the help
lists only `-%I` and `-%I0`.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit a6c8417281f21a24dd4e5518911f589cf904fb1d
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 8 00:16:21 2026 +0200
Share the filter list growth check with mirror start, and test it (#1904)
The scan rules read at mirror start had their own copy of the filter
list growth check, so they now call the shared `filters_grow()` helper.
Test 558 passes 1202 rules at `-#F128` and checks that none is dropped.
A real run cannot reach the `HTS_FILTERS_MAX` cap without a 2 GB list,
so the `filtergrow` self-test in test 538 still covers the cap. The only
change in behaviour is that the `-#F` hint printed when memory runs out
is now logged at info level rather than notice.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 58e7d9d3d9c61a5bab8816826aecb88a64a71d5d
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 7 19:43:45 2026 +0200
Skip over-long lines in config files instead of reading their tail as a new line (#1899)
The config files share one line reader, `htslines.c`. It consumes a line
too long for its buffer whole, so the tail is never read back as a new
line.
Behaviour changes:
- An over-long line in `.httrackrc`, `cookies.txt` or a lang catalog is
skipped with a warning.
- httrack reads `doit.log` and the WebHTTrack profile whole, up to 1 MiB
per line. A longer `doit.log` line stops the resume, and a longer
profile line is skipped.
- Trailing whitespace past the limit does not make a line too long.
- Continuation lines join the same way at every buffer size.
- A line that exactly fills the buffer no longer gives an extra empty
line.
- A resume no longer appends duplicate options to `doit.log`.
- `linput_trim`, `linput_cpp`, `rawlinput` and `cache_binput` are
removed. None was exported.
httrack-android lists its sources by hand, so its `Android.mk` must add
`htslines.c` and `htslines_selftest.c`.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 9bd0a0f8f1d06e6a66f441207c5cf131d8844942
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 7 18:45:47 2026 +0200
Stop the update purge from deleting a file named by the tail of a long path (#1901)
A hostile site could make `--update` delete a file in the mirror. It
only needed a URL whose `old.lst` line is over 1000 bytes, because the
purge read the tail of that line as another path. The purge now reads
`old.lst` whole, as it already reads `new.lst`, and skips any line that
does not fit. The clean-up of emptied directories now climbs only inside
the mirror, and it climbs with the log off too.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 4945dbf270e3c6acee82a8402fd9bf829da284c0
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 7 17:51:16 2026 +0200
Let a path given on the htsserver command line take effect (#1900)
A `path` pair given to `htsserver` now sets the projects folder, and it
wins over the `path=` line saved in `~/.httrack.ini`. So `webhttrack` no
longer passes `path ~/websites` on every launch, because that would
override the folder saved in the interface. The `lang` key keeps its old
precedence.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 53485c08461d5b2e974a3091b793e09ee78e988d
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 7 16:41:00 2026 +0200
Bind proxytrack's ICP socket to the ICP address it was given (#1898)
proxytrack now binds its ICP socket to the address given in its second
argument, rather than to the proxy address. A setup that gives two
different addresses now gets ICP on the second one, as the usage text
says. The new test needs a second loopback address (127.0.0.2), so it
skips on hosts such as macOS that route only 127.0.0.1 to loopback.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 27b9020f15f9d24b30f013558b8213d63d78f369
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 7 16:32:36 2026 +0200
Generate the htsserver, proxytrack and webhttrack man pages from their --help (#1897)
`make -C man regen-man` now builds all four man pages from each
program's `--help`. So `htsserver.1`, `proxytrack.1` and `webhttrack.1`
document their options, and test 02 fails when any page falls behind.
Each page keeps its fixed prose in `man/<page>.tmpl`, and `httrack.1`
comes out byte-identical to the committed one.
Most of the `makeman.sh` diff is re-indentation, so read it with `git
diff -w`. The three smaller programs print their `--help` as one option
per line now, because the generator reads that layout.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 413c0e1e8afc49313e05964c1030e10f0c1b1d41
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 7 15:27:13 2026 +0200
Read header parameters like charset and filename in one place (#1896)
Adds `hts_header_param()`, one reader for the parameters of a header
value, and uses it for the Content-Type charset, the Content-Disposition
filename and the Keep-Alive numbers. Two results change on purpose: a
charset that is not the first parameter is now found, and
`filename="a.txt"; size=3` now saves `a.txt` instead of `a.txt_;
size=3`.
These results also change:
- A Content-Disposition value of 250 bytes or more is now parsed, so a
`filename` followed by a long `filename*` now saves the `filename`.
- A charset value of 256 bytes or more is now dropped, where the old
code kept its first word.
- A Keep-Alive value of 64 bytes or more is now ignored, so the default
applies.
- Keep-Alive names match in any case.
- Keep-Alive values separated only by a space lose the second value.
A follow-up deletes the legacy parser copies in the self-test once this
lands.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit b7138da7ad59176880f67da17e4cb6ee552ade68
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 7 15:03:26 2026 +0200
Build test 120's padding without a slow bash 3.2 substitution (#1895)
Test 120 built its 8 KB request body with `${pad// /x}`. Bash 3.2, the
macOS `/bin/bash`, runs that in quadratic time, so the test now uses
`repeat_chars` as test 317 does. Under bash 3.2 the test drops from 4.9s
to 1.2s, but this may not be what caused #1893's 600s macOS timeout.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 8ab0c96cb4e453ac92e409ba82a0fc8de3edee86
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 7 15:02:28 2026 +0200
Quote and split command-line arguments in one place (#1893)
Six copies of the command-line argument grammar now share four helpers
in `htscmdline.c`. Each way the old copies differed is an explicit
`HTS_SPLIT_*` flag, so no site changes behavior. `-#test=quotediff`
checks this against frozen copies of the old code.
An option's value still loses two surrounding quote pairs without `-O`
and one with it. The quote strip runs once per pass of the `-O` loop. A
fix would change what a front end's doubly quoted argument means, so it
needs its own decision.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit d55e9687aaa814b4f7740813e836bc7fd33584e1
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 7 14:20:54 2026 +0200
Make htsserver, proxytrack and webhttrack answer --help, -h and --version (#1894)
`htsserver`, `proxytrack` and `webhttrack` now print their usage for
`-h` or `--help`, and their version for `--version`, on stdout with exit
0. Before this, `htsserver --help` started a server on port 8080 with
`--help` as its HTML root, and `webhttrack --help` waited for that
server to report a URL.
The new flags must come first on the command line. `htsserver`'s usage
line now puts the HTML root first, because the code always expected that
order. Test 549 runs every flag, and puts stub programs on `PATH` that
fail the test if `webhttrack` starts a browser or `htsserver`.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 608225f5d35b4ab01b7b2061407c677cbe27f8b0
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 7 13:21:42 2026 +0200
Parse cookies, WARC headers and WebHTTrack profiles with the span iterator (#1890)
This moves `cookie_get`, the WARC header walks, the alias loop check and
WebHTTrack's `ini_rebase_lists` onto `hts_span`. Frozen copies of the
old code check each site in `-#test=spandiff`, `cookiespandiff` and
`warcspandiff`, and test 322 checks the rebased profile byte for byte.
`http_headers_have_field` is not moved, because it treats a bare CR, or
a run of CR and LF, as a line break. The robots.txt sitemap walk is not
moved, because a frozen copy would need its private state.
`warc_http_header_value` now returns at once when its buffer size is 0,
where it used to write one byte and wrap the length.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 7700f1c2e4e79f3ad9a33244eb09559c5214f813
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 7 11:52:46 2026 +0200
Time test 58's watchdog check to the kill decision, not the reap (#1892)
Test 58 now checks how long the starved watchdog takes to decide to
kill, rather than the total time including the reap. `run_with_timeout`
records that time in `WATCHDOG_FIRED_AT`, and the test requires it under
12s. A correct watchdog fires at 4s, and the original poll-counting loop
from eb46d8d4 fires at 12s and fails. A slow reap, like the 12s run on
the Windows x64 leg, no longer fails the test.
Closes #1891
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
commit 5fd5afb3e928252c9e39405e827bcdaa6ecc7fca
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 7 10:49:52 2026 +0200
Export hts_scan_token() so front ends can split rules like the engine (#1888)
WinHTTrack splits the scan-rules box itself before it calls
`hts_setfilters()`, so it must split on the same bytes as the engine.
With this export it can call the engine's function instead of keeping
its own copy. Test 207 now fails if the export is dropped, because the
declaration moved to the installed `httrack-library.h`.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
commit 5834bcd615dac4b1baf078a60e87e002be7bcb21
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 7 09:30:32 2026 +0200
Parse --strip-query and --host-alias rules with one span iterator (#1887)
This adds `hts_span`, a read-only field iterator, and moves the
`--strip-query` and `--host-alias` rule parsers in `htslib.c` onto it.
The alias matcher and the alias validator now split a rule with one
shared function. Behaviour does not change, and `-#test=spandiff` checks
each site against a frozen copy of the old code. A too-long
`--strip-query` pattern is still clipped while a too-long alias is
skipped, because changing that is a separate fix.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 01adbc9ba2f27bdd79bf9fd44558d80a9f850eb0
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 7 07:59:05 2026 +0200
Count bytes read, not calls, in the filter matcher's work budget (#1879)
The budget capped 2,000,000 recursive calls, but one call rescans its
whole character class. A pattern whose '(' never closes therefore cost
about 1.1KB of scanning per call. A strjokerfind sweep read some 1.5GB
without ever reaching the cap: 120s under ASan, against the 25s OSS-Fuzz
allows. Charging that scanning to the same counter means the cap now
bounds the work actually done.
The cap moves with its unit, to 24,000,000 quanta. A call is weighed at
16 of them, because a frame costs far more than a byte of scanning. That
rejects some matches master accepted, which is the trade rather than an
oversight. Master allowed 2,000,000 calls of unbounded work each, so no
work-bounded budget can accept them all without restoring the timeout.
The shapes affected are long chains of the *(x) form, from about 36 deep
against a 2KB subject. 40,000 random realistic pairs show no change, and
the heaviest realistic filter spends 11% of the cap.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 622fc3a5f143a9cabd31f5bb79ee52e2ed952a9c
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 7 00:39:05 2026 +0200
Refuse out-of-range numbers in glued short options such as -c8 (#1886)
The 21 short options with a glued number (`-c8`, `-T30`, `-#F5000`) now
use the bounded parser from #1885. A value too large for its field is
refused with "Value out of range in option ...", where it used to wrap.
The `scanOptInt` sites are not changed, because their enum fields would
each need an int temporary and a maximum of their own.
- `-#F` is capped at 522263, and the filter list grows at most one step
of 1000 rules past it. Before this, enough filter rules crashed
`filters_init` (master too). A full list now drops the rule with a
warning. The self-test covers the growth bound in `filters_make_room`,
but not the copy in `httpmirror`, which needs half a million rules to
reach.
- `filters_init` uses `calloct`, because a large `-#F` wrote zeros
through a NULL pointer when the allocation failed.
- `-r` takes at most 2147483646, because the first link is queued at
depth + 1.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit b044404825704f8156b3697b3569bc5e3282c748
Author: Xavier Roche <roche@httrack.com>
Date: Tue Oct 6 23:41:13 2026 +0200
Parse header and FTP numbers with one bounded parser that refuses signs and overflow (#1885)
Header, FTP reply and date numbers now go through one parser,
`hts_parse_llint()`, which refuses a sign and any value past the field's
bound. Test 544 compares each site with a frozen copy of its old
`sscanf` parse, and only signed or oversized numbers read differently.
Its htsserver check runs on a copy of the htsserver parse, not on
htsserver itself.
Each site does this with a refused number, and a signed one such as `-0`
or `+5` is always refused:
- `Content-Length` and the FTP `SIZE` reply leave the size unknown.
- `Content-Range` reads as `0 0 0`.
- A `Keep-Alive` `timeout=` or `max=` turns keep-alive off.
- A date ignores the token, but a `+0100` zone still reads as before.
- htsserver reads no request body.
- proxytrack answers 500.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit b2e47397cff7914181b8d00938ab9a71435ca55e
Author: Xavier Roche <roche@httrack.com>
Date: Tue Oct 6 21:26:11 2026 +0200
Read and write the FTP byte count atomically (#1884)
An FTP worker thread writes a slot's byte count and total size while the
crawl thread reads them for the progress display. Every access to those
two fields is now a relaxed 64-bit atomic, so the last ThreadSanitizer
suppression goes. The new `relaxedcounter` self-test
(`tests/01_engine-relaxed-counter.test`) races a writer thread against a
reader, so the TSan leg fails if the helpers become plain accesses
again.
32-bit hosts other than MSVC still use a plain access, which can tear,
because a 64-bit atomic there may need libatomic.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 88f41ad2d7f2f28b06424d98d210799d47497073
Author: Xavier Roche <roche@httrack.com>
Date: Tue Oct 6 20:48:09 2026 +0200
Stop ThreadSanitizer flagging reads of a backing slot status (#1883)
The crawl thread now reads every backing slot's status through
`back_status()`, an acquire load that pairs with the release store an
FTP worker publishes. Writes stay plain, because the crawl thread only
writes a slot no worker owns. `back_transferred` keeps its TSan
suppression, because it also sums `r.size` while a worker is still
adding to it.
Closes #1868
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit ccaade265c1f175763cbff8e6ac0ad5b7d17dde5
Author: Xavier Roche <roche@httrack.com>
Date: Tue Oct 6 20:22:46 2026 +0200
Test that WebHTTrack keeps the fields after one without '=' (#1882)
This test pins the form-split fix from #1877, which nothing covered. A
POSTed field without `=` must be skipped, and the fields after it must
still reach the server.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 645ccd7900eabbd400086859bc6fcea1d1abe8ec
Author: Xavier Roche <roche@httrack.com>
Date: Tue Oct 6 19:59:20 2026 +0200
Refuse FTP PASV and EPSV replies with out-of-range numbers (#1880)
The FTP client now accepts a PASV reply only when all six fields are
plain numbers from 0 to 255. It reads the address bytes as decimal, so a
zero-padded byte like "010" means 10 rather than octal 8. An EPSV port
must be a plain number from 1 to 65535, and port 65535 is no longer
refused. Any other reply fails the link like a malformed one.
Closes #1876
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 551367124253f558f00bb90e120de8dd085a76fd
Author: Xavier Roche <roche@httrack.com>
Date: Tue Oct 6 19:13:41 2026 +0200
Parse '&'-separated queries with one shared iterator (#1877)
This replaces six hand-written parsers of `&`-separated `key=value`
queries with one iterator, `hts_query_next()`. The six are URL
normalization, the `--strip-query` filter, the `%[param]` save-name
template, and three WebHTTrack request parsers.
Three behaviours change on purpose:
- `%[param]` matches a field's key exactly, so a second `?` inside the
query no longer starts a field.
- A `%[param]` name holding `=` or `&` never matches, because no key can
hold either.
- The WebHTTrack form parser skips a field without `=`. It used to pair
that key with the next field's value and stop.
The new `-#test=querydiff` runs the old URL and `%[param]` parsers next
to the new code on 291200 cases, and allows only the two `%[param]`
changes. A crawl with a `%[id]` template renamed only the four links
whose query holds a second `?`.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 125cb7e8efa3572ef7df4a3c6b7069f0fd997cb9
Author: Xavier Roche <roche@httrack.com>
Date: Tue Oct 6 18:37:29 2026 +0200
Refuse a proxy CONNECT reply whose status code wraps to 200 (#1875)
A proxy's reply to CONNECT was read with `sscanf("%d")`, so a status of
4294967496 wrapped to 200. httrack then used a tunnel the proxy had
refused. The reply now goes through the exactly-three-digits check
`treatfirstline()` already used, lifted into `hts_status_code()` so both
share it.
The CONNECT reply also gains `treatfirstline()`'s leniency: leading
whitespace, a lowercase `http/` and any version token. Each still needs
a real 2xx code.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit f7a079fb04f2c62ad8e165ea3c0de119bfc8fdf3
Author: Xavier Roche <roche@httrack.com>
Date: Tue Oct 6 18:00:45 2026 +0200
Build the Windows engine against OpenSSL 3.6.5 (#1874)
This moves the Windows build to OpenSSL 3.6.5, which fixes
CVE-2026-84782, an out-of-bounds read in DTLS retransmission. HTTrack
never speaks DTLS, but 3.6.4 is still affected. The new baseline is
vcpkg's commit e182cb4, and httrack-windows PR #205 pins the same one.
The Windows build now also checks out the baseline's `versions/`
directory. Without it, vcpkg reads the runner image's older version
database and finds no 3.6.5.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit ba3d1121d0eb4e378607635ea6206e0c58ae3622
Author: Xavier Roche <roche@httrack.com>
Date: Tue Oct 6 17:28:27 2026 +0200
Reword the live scan rules note for WinHTTrack's whole-box update (#1873)
This rewords `LANG_LIVERULES` for WinHTTrack's switch to
`hts_setfilters()` (httrack-windows PR 209), keeping the symbol and
retranslating all 30 catalogs. It should land with that PR, since the
old text describes the append-only panel.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit d39ae32c73bc1db5ee6b6a63aaa1bfcab1cde9a0
Author: Xavier Roche <roche@httrack.com>
Date: Tue Oct 6 16:08:09 2026 +0200
Add hts_setfilters() to replace a running crawl's scan rules (#1872)
This adds `hts_setfilters()`, which replaces a running crawl's scan
rules with a list applied in the order given. A front end can then apply
an edited rule box as a fresh run would. `hts_addfilter()` now appends
after the user's rules rather than after every rule, so an engine host
ban still outranks a live rule.
A queued link is dropped when a new rule refuses it, so an unrelated
rule never drops a start URL the old rules already refused. A link a
removed rule already refused does not come back in this run. This
follows #128.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 3452b6e52dce1f1fbec7ad08e84ce07a640f9106
Author: Xavier Roche <roche@httrack.com>
Date: Tue Oct 6 14:47:36 2026 +0200
Add the catalog string WinHTTrack shows for an invalid live scan rule (#1871)
This adds `LANG_LIVERULESBAD`, the message WinHTTrack shows when
`hts_filter_rule_ok()` refuses a scan rule while its dialog is still
open, translated in all 30 catalogs. WinHTTrack appends the refused rule
on its own line, so the text carries no format specifier.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 739eb1abcde5dde5d0d6476de26863c3ffb624c7
Author: Xavier Roche <roche@httrack.com>
Date: Tue Oct 6 14:06:49 2026 +0200
Add the catalog string for WinHTTrack's live scan rules note (#1869)
This adds `LANG_LIVERULES`, the note WinHTTrack's mid-mirror options
panel shows next to its scan rules box, translated in all 30 catalogs.
Until it exists, every translated WinHTTrack shows that note in English.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit d2d17e391f4a4d3175f2e00777cbac035a5bdc49
Author: Xavier Roche <roche@httrack.com>
Date: Tue Oct 6 14:01:51 2026 +0200
Export hts_filter_rule_ok() so a front end can check a live scan rule (#1870)
This exports `hts_filter_rule_ok()`, which says whether
`hts_addfilter()` would accept a scan rule. WinHTTrack can then refuse a
bad rule while its dialog is still open. It follows
`hts_host_alias_rule_ok()`, and `hts_addfilter()` now calls it, so the
two cannot disagree.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit a481d0dc69bc87d0dfb042a948680a563e99ff34
Author: Xavier Roche <roche@httrack.com>
Date: Tue Oct 6 13:50:53 2026 +0200
Fix the thread races ThreadSanitizer reports in the engine and WebHTTrack (#1867)
This makes the flags that one thread sets and another polls atomic, in
the engine and in WebHTTrack. They are the stop and exit requests, the
FTP stop flag, the log counters, an FTP slot's status and WebHTTrack's
run flags. ThreadSanitizer reported ten such races on master.
The TSan leg now also runs the FTP crawl tests, test 65, and the
WebHTTrack tests that drive a crawl. On master that set fails six tests
with TSan reports, and with this change it runs clean. Review also found
WebHTTrack freeing the run's return message twice, which this fixes.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 6788f7f76a129a8170b57d32eea2524c50289175
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date: Tue Oct 6 11:38:40 2026 +0000
Bump vmactions/freebsd-vm from 1.5.8 to 1.5.9 (#1866)
commit ecfd01cc80aa21c64b0448240cb98c169f84a0eb
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date: Tue Oct 6 11:29:47 2026 +0000
Bump src/coucal from `5855b6e` to `9f797c9` (#1865)
commit f7854c8474181b12695e968a49270b9175f9e1f3
Author: Xavier Roche <roche@httrack.com>
Date: Tue Oct 6 12:45:59 2026 +0200
Stop WebHTTrack using the crawl options after freeing them (#1864)
WebHTTrack now creates the crawl options before it marks a crawl as
running, and clears them under the server lock before freeing them. A
control command checks the options pointer itself, so one sent as a
crawl ends is ignored.
There is no new test because the window is a few instructions wide.
Neither 300 start-then-pause rounds nor ThreadSanitizer hit it on
master.
Closes #1863
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit ce4fedff4022954d389f2b0877a6455a4686434d
Author: Xavier Roche <roche@httrack.com>
Date: Tue Oct 6 12:44:22 2026 +0200
Add scan rules to a running crawl (#1862)
This adds `hts_addfilter()`, a new export that queues a scan rule for a
running mirror. WebHTTrack's control channel gets an `add-filter=`
command that calls it. The engine puts the rule after all the others so
it wins. It then drops the matching links that are queued, or fetched
and not yet saved. A page already saved keeps its local link to a
dropped file, as it does after a host ban.
A `+` rule only affects links found after it arrives. The rule is not
written to `doit.log`, so a later `--update` forgets it.
Closes #128
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
![[bug history graph]](https://qa.debian.org/data/bts/graphs/h/httrack.png){kind=link}