vcswatch reports that
this package seems to have new commits in its VCS but has
not yet updated debian/changelog. You should consider updating
the Debian changelog and uploading this new version into the archive.
Here are the relevant commit logs:
commit 7a349927026edaa3b5301049200b50d6bfddde27
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date: Tue Aug 25 13:46:52 2026 +0200
Bump github/codeql-action from 4.37.7 to 4.37.8 (#1441)
Bumps [github/codeql-action](https://github.com/github/codeql-action)
from 4.37.7 to 4.37.8.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/github/codeql-action/releases">github/codeql-action's
releases</a>.</em></p>
<blockquote>
<h2>v4.37.8</h2>
<p>No user facing changes.</p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/github/codeql-action/blob/main/CHANGELOG.md">github/codeql-action's
changelog</a>.</em></p>
<blockquote>
<h2>4.37.8 - 21 Aug 2026</h2>
<p>No user facing changes.</p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/github/codeql-action/commit/db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28"><code>db488dd</code></a>
Merge pull request <a
href="https://redirect.github.com/github/codeql-action/issues/4102">#4102</a>
from github/update-v4.37.8-9ee088e13</li>
<li><a
href="https://github.com/github/codeql-action/commit/1845f5ba8b4057590f49ee8e246c95ef2ba4b53f"><code>1845f5b</code></a>
Update changelog for v4.37.8</li>
<li><a
href="https://github.com/github/codeql-action/commit/9ee088e13615f8d1eaef4766f9dde95d3356a8f6"><code>9ee088e</code></a>
Merge pull request <a
href="https://redirect.github.com/github/codeql-action/issues/4080">#4080</a>
from github/henrymercer/studious-giggle</li>
<li><a
href="https://github.com/github/codeql-action/commit/1aef003397c876c0ab5bd118e1b1f34c175622e9"><code>1aef003</code></a>
Address review feedback on overlay disk flags</li>
<li><a
href="https://github.com/github/codeql-action/commit/508b83bc415e8df76ce8ea08c0cf42c2529ebc63"><code>508b83b</code></a>
Merge main into overlay minimum disk feature branch</li>
<li><a
href="https://github.com/github/codeql-action/commit/d97b3428e8eebbb1810cf454d6397886d136b4ba"><code>d97b342</code></a>
Merge pull request <a
href="https://redirect.github.com/github/codeql-action/issues/4098">#4098</a>
from github/mbg/permission-error-as-configuration-error</li>
<li><a
href="https://github.com/github/codeql-action/commit/47fa6222231b12097f83215dd7a6b4a0915841fd"><code>47fa622</code></a>
Make <code>EACCES</code> a <code>ConfigurationError</code></li>
<li><a
href="https://github.com/github/codeql-action/commit/45693cc6882bb175b58a06818c91876e201037c7"><code>45693cc</code></a>
Refactor <code>ENOSPC</code> check into
<code>isDiskConfigurationError</code> function</li>
<li><a
href="https://github.com/github/codeql-action/commit/c2fd8f54d19fa46c94ed79cb92e6dd6606d61762"><code>c2fd8f5</code></a>
Merge pull request <a
href="https://redirect.github.com/github/codeql-action/issues/4081">#4081</a>
from github/mario-campos/version-cache-to-disk</li>
<li><a
href="https://github.com/github/codeql-action/commit/c56f48e9bd458a387eb68a68534459e503e56b17"><code>c56f48e</code></a>
Log unexpected conditions during caching CLI output</li>
<li>Additional commits viewable in <a
href="https://github.com/github/codeql-action/compare/v4.37.7...v4.37.8">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
commit d707ad4365c72d86a9b19809fe2b7e3c4090f8f9
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date: Tue Aug 25 13:46:36 2026 +0200
Bump src/coucal from `edebab8` to `0392e05` (#1440)
Bumps [src/coucal](https://github.com/xroche/coucal) from `edebab8` to
`0392e05`.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/xroche/coucal/commit/0392e05e70c78e017ba5d79607d4c6a70366be80"><code>0392e05</code></a>
Bump github/codeql-action from 4.37.6 to 4.37.7 (<a
href="https://redirect.github.com/xroche/coucal/issues/38">#38</a>)</li>
<li>See full diff in <a
href="https://github.com/xroche/coucal/compare/edebab8f4213cdc264ac1bbfae99edaa13128f8e...0392e05e70c78e017ba5d79607d4c6a70366be80">compare
view</a></li>
</ul>
</details>
<br />
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
commit 96fee4fd1b9e183837b00cc9180ae26daa4e4a01
Author: Xavier Roche <roche@httrack.com>
Date: Tue Aug 25 12:10:07 2026 +0200
153 reads proxytrack's stderr banner as output from serving (#1438)
153 merges proxytrack's stdout and stderr onto one pty and cuts the
startup banner at the `PID=` line, which stdout carries. The archive
proxytrack loads is announced on stderr, and the sanitizer legs run the
engine through `tests/stderrwrap.c`, which relays stderr from a separate
process — so that line reaches the pty whenever the relay gets
scheduled, and under load that is after the `PID=` line the cut fences
on. It then read as something the PROPFIND wrote, and the test went red
on the sanitize leg only.
Latent since the shim landed, not a regression from anything recent: it
reproduces on master at `1a5a8d2c` 13 times out of 16 with the shim on a
loaded box, and 16 out of 16 pass at the same load with the shim off.
Both PROPFIND leaks a mutant plants, on stdout and on stderr, still fail
the patched test.
Closes nothing; #1428 and #1429 were only downwind of it.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit c4cd49c0aa101a2ec37f94fcb0c145f9b8121d80
Author: Xavier Roche <roche@httrack.com>
Date: Tue Aug 25 12:08:29 2026 +0200
A param option's value is bounded by length where a magnitude is meant (#1439)
`optalias_param_takes()` (#1427) bounded a `param` row's value at 16
characters, which is a length where a magnitude is meant. It refused
`--depth=0000000000000000001` for its padding alone, and accepted
`--sockets=1234567890123456`, sixteen digits that `sscanf("%d")` then
read as 1015724736 simultaneous connections. Each digit run is now
converted with `strtoll` and range-checked against what the option's own
parser holds: `INT_MAX` for the rows reading an `int`, `INT64_MAX` for
`-m`, `-M` and `-G` on their `LLint` and for `-%c`, whose float takes
any run `strtoll` converts. The check runs per operand, so the second
half of `-m N,N2` and `-%c 0.5` is covered too. `optalias_suffix()`
carried a copy of the same bare 16, so the six `level` rows get the same
treatment.
The behaviour change is the fix rather than a side effect of it, and it
goes both ways. Refused from now on: a value on an int-valued long form
whose digit run exceeds 2147483647 and which the old cap let through, so
no longer than 16 characters. `--sockets=1234567890123456`,
`--max-rate=3000000000`, `--depth=2147483648`, `--timeout=99999999999`,
`--generate-errors=3000000000`, and every value of that shape on the
other int rows. All of them were undefined at `sscanf("%d")` and wrapped
in practice: 1015724736 sockets, a rate of -1294967296, a depth of
-2147483648, a timeout of 1215752191 seconds. None landed on a sane
value by accident either, since `-c` clamps its wrapped negative back to
one socket, which is still not what was asked for.
Accepted from now on: a zero-padded value wherever its magnitude fits,
and on `-m`, `-M`, `-G` and `-%c` any 17-to-19-digit run up to
`INT64_MAX`. `--max-size=9223372036854775807` is refused on master and
taken here, which is right, because that is what the `LLint` behind it
holds.
The old cap was also the only thing keeping a value short enough to glue
into the 1024-byte expansion buffer, where `strlcatbuff` aborts rather
than clips. So a length bound stays, at the glue site and sized to the
room the short form leaves: without it `httrack --depth $(printf '0%.0s'
$(seq 1022))` exits 134 with a backtrace where master printed a syntax
error. It now also reaches a class the old cap never covered, the
`param0` rows, where `httrack --allow <1023 chars>` aborts on master and
draws the same syntax error here.
369 asserts both directions, through the engine's own expansion and on
disk, and reds on six mutants: the length cap put back,
`optalias_run_fits` forced true or stripped of its `max`, the `LLint`
arm narrowed to `-M` and `-%c`, a leading sign accepted, the capacity
guard removed, and `optalias_suffix` reverted. `st_optalias`'s
table-wide loop carries the padding, `INT_MAX` and sign probes across
all 27 `param` rows.
Closes #1437
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 718caeb1d8971a4e161cb54f2f71041651ff2396
Author: Xavier Roche <roche@httrack.com>
Date: Tue Aug 25 11:28:55 2026 +0200
Windows builds carry 14 uncleaned compiler warnings (#1432)
This is the follow-through of #1354, the last MSVC warning sweep: the
Windows legs have been compiling with warnings ever since it closed.
Harvesting the `libhttrack (x64, Release)` and `(Win32, Release)` logs
off master turns up 14 distinct sites: 3 that only x64 sees (a 64-bit
`size_t` narrowing into MSVC's `unsigned int` or `int`), 10 that only
Win32 sees (a 64-bit `LLint`/`__int64` narrowing into a 32-bit `size_t`,
or a signed/unsigned comparison), and 1 common to both. Running both
widths matters: the two lists overlap in a single entry, so either width
alone hides most of the sweep.
Thirteen are ours and all thirteen are fixed. Where the value simply had
the wrong type I changed the type rather than casting — `ampargs` in
`fil_normalized_ex` and the bogus-link loop index in `htsparse` become
`size_t`, and the emergency signal-handler writers in `httrack.c` and
`htsbacktrace.c` count in `unsigned int`, which is what MSVC's `write()`
takes and what their fixed 256-byte buffer and string literals fit in
anyway. Casts appear only where a guard on the same path already bounds
the value: `opt->maxlink` is cast under its own `> 0` test, and
`cache_rstr_addr`'s length is cast under the clamp to `[0, 32768]` two
lines above it, matching the cast the very next line already had.
`r.size` gets `(size_t)` at two call sites in `httpmirror` — it counts
the bytes held at `r.adr`, so it fits `size_t` by construction, and line
1649 of the same function already spelled it that way.
Two chained assignments are split apart, because the chain was what
produced the narrowing: `opt->maxtime = opt->maxsite = 0` in
`htsselftest.c` (an `int` fed from an `LLint`, landed in #1079) and
`gotquery = ampargs = 1` in `htslib.c`. The second sat inside a
braceless `if`, so splitting it needs braces — worth flagging since
dropping them would have made `ampargs = 1` unconditional.
The one I left is `src/minizip/mztools.c(194,29)` C4267. `src/minizip/`
is patched in place here rather than kept pristine, with each delta
recorded in an `.orig`/`.diff` sidecar pair under `EXTRA_DIST`, so this
is fixable -- it is just not worth it: the warning is harmless, the code
is upstream's, and touching it would mean regenerating that pair for no
gain.
No real bug fell out of this. The one candidate worth naming is the ICP
reply path in `proxytrack.c`, where a message length lands in an
`unsigned short`: the reader caps an ICP message at 16 KB per RFC2186
and NUL-terminates its buffer, so `strlen() + 1` cannot reach 65535 and
the field cannot truncate. The cast there is now explicit and carries
that reasoning.
Verified on Linux, since MSVC is not available locally: gcc and clang
both build clean at `-Wall -Wextra`, and a warning-set diff against an
`origin/master` baseline built with identical flags shows nothing new
and five sign-compare classes removed — gcc was seeing the same
signedness bugs MSVC reports as C4018. `make check` is green at 389
tests. The Windows legs on this PR are the real check.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 38dd23f75a6dcc0d3ae425cbf41b543d46183de1
Author: Xavier Roche <roche@httrack.com>
Date: Tue Aug 25 11:23:01 2026 +0200
Option values that mirrored the wrong thing at exit 0 (#1436)
Two of the three spellings in #1434 mirrored the wrong thing at exit 0,
both because the value mapped onto the bare short form.
`--structure=on` expanded to a bare `-N`, which reads the next word as a
user template, so `--structure=on <URL>` swallowed the URL and mirrored
nothing. "on" now names the default preset (`-N0`), the layout "off"
already selected; #1418's glue rules are otherwise untouched.
`--long-names=` expanded to a bare `-L`, whose `sscanf` converted
nothing and left the switch to re-map the value already stored, landing
on DOS 8.3 names (`-L1 -L` did the same). Rather than refuse the empty
value I fixed the short form: a bare `-L` now selects L1, the default
the help and man page already star. Refusing would have taken `sockets=`
and the config-file `long-names=` spelling with it, and this way the
empty value means the option's own default. `-L0`, `--long-names=off`
and `--long-names=2` are unchanged.
Rewriting that parse surfaced a third bug, pre-existing and folded in
here: the digit-run bound #1418 added counts digits rather than reading
the value, so a zero-padded run falls off it. `-L0000000001` selected
ISO9660, `-N0000000001` lost its preset, and `--structure=0000000001`
was refused. `optalias_digits_fit` skips the leading zeros first, shared
by the four sites that read such a run.
The 27 `param` rows keep #1427's 16-character cap: its bound is a
magnitude, not a run length, so `optalias_digits_fit` does not fit there
— 9 significant digits would refuse `--advanced-maxlinks=2000000000`,
and it cannot see the second operand of `-m N,N2`. Filed as #1437.
The third spelling in the issue, `--structure=I0`, has been refused
since #1418 landed after the issue was measured, and test 368 pins that.
What survives is `--structure=8I0`: the digits-then-cluster form #1418
glues on purpose, and the same shape as the `--structure=1L0` that
#1427's test pins as accepted. No rule separates the two, so that one
wants a decision rather than a patch.
Closes #1434
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit d872e62fc05ffa225c2c6cf572effabf7722dc5b
Author: Xavier Roche <roche@httrack.com>
Date: Tue Aug 25 11:22:22 2026 +0200
An undeclared Content-Type let a blob hold back the update purge (#1430)
An empty or absent `Content-Type:` leaves `hts_header()`'s
`HTS_UNKNOWN_MIME` sentinel in place, the sentinel reads as hypertext,
and a blob that failed to re-fetch on `--update` then held the purge
back as if it carried links. Being read as hypertext also ran the
NUL-blanking hack over the body, so an untyped PNG, PDF or ZIP reached
disk with every NUL rewritten to a space.
The fix is local to the purge and to that corruption. Neither the
sentinel's meaning nor `is_hypertext_mime()` moves, so nothing about
what gets crawled changes. HTTrack blanks the NULs of everything it
parses as hypertext, so a NUL surviving in a mirrored copy proves no run
ever read links out of it, whatever the recorded type and name claim,
and `hts_link_may_carry_links()` now consults the copy. The untyped-body
binary check gains NULs, with the same ratio and floor as the `nspec`
arm beside it, so a page carrying the odd NUL stays a page.
Both halves are needed and a purge-only one-liner does not close the
issue alone. The savename is `blob.bin.html`, derived from the sentinel
through `hts_effective_mime()`, so a name-based test in the guard reads
a page; the recorded type reads a page too; and the copy's bytes, the
only evidence left, had been rewritten by the parse the guard is trying
to detect. Stopping the corruption is what makes that evidence readable.
An earlier revision of this PR typed the sentinel by the URL extension,
and it is withdrawn. `guess_httptype_sized(flag=1)` never fails, so the
predicate collapsed to a ten-string list and untyped `.aspx`, `.do` or
`.py` pages lost their subtrees; and `get_ext()` copies past the query
string, now filed as #1433, which killed the `is_dyntype` carve-out for
any URL carrying parameters. Neither failure mode exists in what is
here.
Test 365 covers an empty header, an absent one,
`application/octet-stream` as the control that fails on nothing this
branch touches, an untyped blob failing two updates in a row, an untyped
dynamic page whose child must still be mirrored, three pages carrying
NULs that pin the ratio, the floor and the undeclared-type scoping, and
a page whose subtree must survive its own failure. Seven mutants, one
per arm, each red on its own assertion, with the NUL term in the
answered arm and the one in the #746 fallback killed separately. The
twice-failing blob is what kills the second, and it is served at
`blob.html` on purpose: a savename recomputed on a failed pass drops the
tail an undeclared type had added, so `/blob.bin` never gets past
`fexist_utf8()` to reach that arm at all.
#1414 has landed and its conflict in `hts_link_may_carry_links()` is
merged here. Its three arms stay exactly as they landed, ordering
included, and the NUL evidence joins the two that type a file by its
name: the answered 2xx arm, and the #746 fallback a second consecutive
failure drops to. The redirect and Location arms name no local file, so
there is nothing to read there. Dropping my term leaves 348 and 354
green and only 365 red, which is the proof that no case of #1414's leans
on it; forcing every copy to read as binary reds the holding cases in
all three, which is the proof it is not dead code on those paths.
#1414's own arms still carry their tests: removing the Location arm, the
#746 fallback, or the ordering that keeps Location below the answered
arm each reds 354 on its own case. Its `HTTP_IS_REDIRECT` arm is
reachable by no test either before or after this merge, since a
headers-only redirect entry comes back with no usable status and the
Location arm catches it.
It also inserts at the same anchor in `tests/local-server.py` as #1431,
which is an adjacent insertion on both sides.
Closes #1415
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 2a309a5bb065066aa57ffe68b3909718e1a337c1
Author: Xavier Roche <roche@httrack.com>
Date: Tue Aug 25 10:51:57 2026 +0200
A mirror that aborts on a fatal error still exits 0 (#1419)
`hts_main2()` returned 0 whatever the mirror did, so a script could not
tell a finished mirror from one the engine gave up on halfway. It now
returns 3 for a mirror that started and did not finish. This changes an
exported API's documented return, which Xavier approved ahead of the
work.
By class:
| status | what happened |
| --- | --- |
| 0 | the mirror ran to the end; or the caller stopped it (`^C`,
`hts_request_stop()`, a callback returning 0, WebHTTrack's cancel); or
nothing transferred and the session was rolled back to the last good one
|
| 3 | the engine gave up mid-mirror: a write it cannot retry (a full
disk, a cache write failure) or a link table it cannot grow past `-#L` |
| 1, 255 | the command line was refused, or a `-#E` / `-#R` cache
operation failed. No mirror ran |
Not every user-set limit aborts, and that is worth your eye: `-#L` exits
3 while `--max-time` and `--max-size` exit 0, because the caps stop
through `hts_request_stop()`, the same channel as `^C`, on the reading
that meeting a budget is the outcome asked for while a link cap cuts
short work that was asked for. Both the man page and the header say so,
since a user who meets one and then the other would otherwise call the
status unreliable. (The log calls a capped mirror aborted either way,
and stdout says `Done.` for both, which is #1420.)
**3 and not 2**, on Xavier's call after review. `hts_is_exiting()`
already returns 2 for the rolled-back session, which exits 0, so 2 would
have meant the opposite thing on the other channel and an embedder
reading one as the other would get it backwards. 3 is free:
`hts_main2()` only ever returned -1, 0 and 1; `main()` returns that
value and nothing else; `exit_xh`, which is what `hts_is_exiting()`
hands back, holds only -1, 0, 1 and 2, so the two value spaces are now
disjoint rather than merely renumbered. The header and the man page both
say the two are separate channels, so a fourth value does not reopen
this.
Nine live sites set `exit_xh = -1` and all reach that status. Eight
already existed; the ninth is added here, because the parser hit the
`-#L` cap at `htsparse.c:3182`, logged `Too many URLs`, and returned
quietly, while `htsAddLink`'s identical refusal aborted. One limit, two
answers. `htsserver.c` branches on this status to render its error
block, which is why an exit the caller asked for stays 0: an abort there
would put a red error page on the cancel button. `hts_errmsg()` was
empty on this path, so the abort now fills it; without that the status
arrived as a bare `* ` on stderr.
No soname bump, and I do not read this as an ABI break: the signature is
unchanged, no installed struct changed shape (the `htsopt.h` edit is a
comment), and `HTS_EXIT_MIRROR_ABORTED` is a new macro. Only the value a
caller reads back moved.
Three tests carry it, all through the constant rather than the literal.
360 asserts the abort across four producers and 0 for a mirror that
finishes and one that fits under the link cap, and requires a refused
command line to exit 255 exactly. 350 asserts it on its `/dev/full`
arms, including the four #1417 added, and 0 on both its EPIPE and
SIGTERM arms. 240 already called its `-#L` pass an aborted mirror in its
own header, so it asserts the abort there and 0 on the dead-server pass,
where the session is rolled back rather than left half written. Mutants:
reverting the return reds all three files, always aborting reds the
clean-mirror control, and widening the check to `exit_xh != 0` reds
350's SIGTERM arm. The `httpmirror() == 0` branch also sets the status
and is live rather than dead, but all six returns behind it are
allocation or size-overflow guards, so no black-box arm reaches them.
`Done.` still goes to stdout for an aborted mirror: that is #1420, whose
fix lands on the `opt->shell` `TRANSFER DONE` token WinHTTrack and
htsserver parse.
Closes #1394
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 1a5a8d2cf7b7607275632186dd8e88d41d00f064
Author: Xavier Roche <roche@httrack.com>
Date: Tue Aug 25 09:53:33 2026 +0200
A param option's value is glued onto its short form without being checked (#1427)
The `param` class glued a long option's value onto its short form
without looking at it. A value the short form cannot read converts short
or not at all, so the option keeps a value nobody asked for, and the
tail spills into the cluster loop as more short options.
`--long-names=yes` and `--long-names=none` mirror in DOS 8.3 names,
which is what `--long-names=0` means, and `--sockets=8I0` mirrors with
no top `index.html` because the `I0` reached `-I0`. Both exit 0. A
`param` value is now a bounded digit run, the `on`/`off` mapping the
class already had, or one separator with a digit after it where the
parser reads one (`-m` takes `N,N2`, `-%c` a rate with a decimal point).
Anything else draws the refusal the other classes have used since #1195,
across all 27 `param` rows, asserted table-wide in the `optalias`
self-test so a new row is covered the day it is added.
Refusal rather than a warning, because carrying on does not preserve the
run: the value's characters turn other options on, so the warning would
be followed by a wrong mirror under exit 0, which a script cannot see.
Refusal is not a new outcome here either, since an uppercase junk value
already exits 255 on every one of these rows, `OFF` reaching `-O`, with
a message naming an option nobody typed. `--cache=OFF` now says `Option
--cache does not take the value OFF`. Along the way: `--cache`'s help
string was `--retries`'s, and 19 of these rows carry no help string at
all, whose dangling tab read as a truncated message.
`--structure` is out of scope, having its own `paramn` class since
#1418, which this branch is merged onto. Three gaps stay open behind
that class, all pre-existing and all surviving #1418, which keeps
`8I0`-shaped gluing by design: `--structure=I0` is byte-identical to
`-I0`, exit 0 with `index.html` and the two GIFs silently missing;
`--structure=on <URL>` maps to a bare `-N`, which eats the next
argument, so the crawl mirrors nothing and still exits 0; and
`--long-names=` with an empty value is the bare `-L`, which is 8.3, so
the headline symptom stays reachable by that spelling.
Closes #1426
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 4775971ef446a84df0a346dd08f03040516543f0
Author: Xavier Roche <roche@httrack.com>
Date: Tue Aug 25 09:19:35 2026 +0200
An update purge still deletes live children behind a redirect, and after two failed runs (#1414)
An update purge still deleted live files in two shapes #1390 left open.
A hub that is a 301 has its cache entry stored headers-only, and the
read invalidates it for naming no local file, so nothing said the hub
had carried its target and everything below it: one interrupted transfer
took the whole subtree. A hub that fails on two consecutive runs has no
cache entry at all the second time, because a failed fetch writes none,
so a child that lost its other parent in that window went with it.
Neither is a regression; a build of master from before #1390 loses the
same files.
The guard now reads the redirect off the Location the entry recorded,
which outlives that invalidation, and when no entry is left it falls
back to the copy #746 kept on disk and the name it was saved under. Both
new arms sit below the check that the entry answered a status at all,
because `Location:` is parsed and cached whatever the status: read above
that check, a 200 blob carrying a stray Location suppresses the whole
run's purge. Requiring the kept copy to be present is what stops a
brand-new dead `.html` link from holding the purge forever, the trap
#1390 named when it rejected typing a link by its savename.
The issue proposed keying on `old.lst`'s `(from URL)` column instead.
There is no such column: `filenote()` writes `[savename]` and nothing
else, and `(from URL)` belongs to `hts-cache/new.txt`, a debug log. It
could not carry either case anyway, since a redirect's child records the
grandparent as its referer, and a run that failed never fetched the
children at all.
The trade worth your decision. A hold covers the whole run rather than
the failed page's subtree, because the subtree is exactly what the run
does not know, and `opt->links_unqueued` is set once with nothing that
releases it. Until now that hold could not outlive one run: the second
failure wiped the cache entry the guard keyed on, so a page that stayed
unreachable started purging again on the next update. With the fallback
surviving the missing entry, one previously mirrored `.html` link that
is permanently unreachable holds `--purge-old` off for the life of the
mirror, which then keeps growing with files the site really did drop.
The engine says so once per run in `hts-log.txt`, without naming the
page. I took that over the alternative because the files a purge would
take in this state are still reachable from the local copy the engine
deliberately kept, and a mirror whose pages point at deleted files fails
its reader worse than one carrying stale files it reported. Bounding it
is implementable: `old.lst`, `new.lst` and the kept copy all survive the
failure that loses the cache entry, so a per-link counter has somewhere
to live. What the bound should be is a policy call about how long a
mirror protects a subtree behind an unreachable parent, so the hold is
unbounded here and the choice is yours.
Test 354 covers the redirect hub, the double failure, a blob that must
hold nothing back, a 200 carrying a Location, and a hub saved under a
name its URL does not give. Eight mutants of the guard ran against 348
and 354; seven red the intended assertion, always-on and always-off
reding both files. The eighth, dropping the `HTTP_IS_REDIRECT` arm on
its own, stays green: every redirect entry reachable through the engine
is invalidated on read and caught by the Location arm below it, and the
arm stays as the honest predicate for a 3xx that did name a local file.
348 gains the markers its survival checks never grepped, and premises
for the two shapes that had none, since dropping `-m,4000` used to leave
the size-cap shape green and the answered-500 shape asserted nothing
about the cache recovery that happens ahead of the give-up arm.
Two findings from this work are filed rather than fixed. #1415: an empty
`Content-Type` reads as the no-declared-type sentinel and therefore as
hypertext, which falsifies the negative control #1395 named and is why
354 controls with an explicit `application/octet-stream` blob. #1421: a
page whose local name comes from its cached type loses that name after a
failed fetch, so #746's keep misses and this PR's fallback inherits the
miss; 354 now carries a non-`.html` hub through the first failure, so
the suite can see the shape.
Closes #1395
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 3188d73f8ecd0a1831b1d86f3ba57c676edea7dd
Author: Xavier Roche <roche@httrack.com>
Date: Tue Aug 25 08:30:42 2026 +0200
--structure silently ignores a template and mirrors with the default layout (#1418)
`-N` reads a glued value as a preset number and a detached one as a user
template, and the alias table classed `--structure` as `param`, which
always glues. `--structure "%h%p/%n%q.%t"` therefore became
`-N%h%p/%n%q.%t`, which the engine never consumes: the token fell
through to the URL list and the crawl ran with the default structure at
exit 0. It is a new `paramn` class rather than a reuse of `param1`
because #1379 tried that, and it turned `--structure=1` into a template
literally named `1`, collapsing the mirror onto one colliding filename,
also at exit 0.
`paramn` glues what `-N` can actually read glued: `on`/`off`, and a
digit run of at most nine digits whose remainder holds no path separator
and no extension dot. Every preset and every cluster `param` used to
glue still glues, so `--structure=1L0` remains preset 1 plus `-L0`,
while a template reaches the engine detached even when it opens with a
digit. The short arm uses a stricter test, a bare digit run and nothing
else, because `-N 2col/%n.%t` has to stay a template. Anything left over
has to carry a `%`: a value without one maps every URL onto a single
local name, so `--structure=flat` is a typo rather than a template and
is now refused, along with `OFF`, `none`, `default`, `yes` and `-1`,
which used to mirror the whole site as `./flat` and `./flat-2`.
`--user-structure` still takes such a value verbatim.
The nine-digit bound exists because a longer run does not read portably.
`sscanf("%d")` past `INT_MAX` is undefined and glibc answers differently
per width: on LP64 the value truncates negative, which is "userdef" with
an empty template and crashes `url_savename`, while on ILP32 scanf
clamps to `INT_MAX`, a preset whose sub-layout flattens the whole mirror
onto one file. `optreal_find` matches `-N` exactly, so the alias table
never sees the glued spelling and the engine counts the digits itself,
calling `atoi` only on a run short enough to fit. `-N4294967295` and
`--structure=4294967295` both abort on master.
`-N 1` is preset 1 now instead of a template named `1`. That semantics
change is deliberate and approved; a template named `1` has no practical
use. `--user-structure`, the spelling that always worked, was documented
nowhere and now reaches the help text, and from there the man page, plus
guide.html and cmdguide.html.
Test 359 asserts the whole mirrored file set on disk for every spelling
rather than just the wanted path, so a value that fell through to the
URL list shows up as an extra file. Each assertion was proven red
against a mutant of the guard it covers, including a
first-character-only predicate, an unbounded digit run, and an engine
still reading the run with `sscanf`. `-N2147483647` is what pins that
last one on a 64-bit host: it is in range for an int and out of range
for `-N`, so it fails the same way the i386 leg did.
Three of the test's own assumptions were POSIX-only and each showed up
on a different leg. It read `-N` past `INT_MAX` as portable, which i386
contradicts. It compared against a path a trailing-slash `TMPDIR` gave a
`//` the engine normalizes away, which macOS contradicts, fixed in
`test-timeout.sh` so every test benefits. And it built its two
absolute-path expectations from the POSIX form, while `url_savename`
maps what Windows forbids in a filename to `_` on every platform, so a
drive-letter `TMPDIR` reaches disk as `D_`; those now derive through the
same substitution rather than a hardcoded `D_`.
Closes #1380
The same failure family on the `param` class, which this does not touch,
is filed as #1426.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 9b083e68289a7efdc69eea1650de87b0eb7870c8
Author: Xavier Roche <roche@httrack.com>
Date: Tue Aug 25 07:15:06 2026 +0200
A best-fit substitute made the strict UTF-8 converter lossy on Windows (#1410)
On Windows, `WideCharToMultiByte` substitutes a lookalike character when
the target codepage lacks a code point, and does not report that
substitution through `lpUsedDefaultChar`. U+00A5 comes out of CP932 as
`0x5C`, so `hts_convertStringFromUTF8Strict` saw no loss and handed back
a path separator where the document wrote a yen sign. Both
`WideCharToMultiByte` calls now pass `WC_NO_BEST_FIT_CHARS`, gated on
`cp_reports_default_char`. That predicate already picks out the
codepages needing `dwFlags == 0`, so it needed no change. The yen
becomes the codepage's default character, `usedDefault` is set, and
strict returns NULL.
Setting the flag is not quite enough on its own. Sizing with it reports
the length of the substitute, but the converting pass still wants room
for the form it replaced, so a buffer sized from the flagged pass is
refused: U+00B5 sizes as 1 byte on CP932 while its best-fit takes 2. The
exact-length guard then rejected a conversion that had succeeded, and
`hts_convertStringFromUTF8` returned NULL for ten Latin-1 code points.
The buffer is now sized for the larger of the two passes and the written
length is taken from the conversion. The Windows CI leg is what caught
this; the numbers above are measured there, not inferred. The max() is
defensive: per code point the flagged size is never larger than the
best-fit size, since no codepage we know of has a default character
longer than a best-fit substitute, so the `fsize > bsize` arm is
believed unreachable and the bug fixed here is the reverse, `fsize <
bsize`, which the flagged pass reports and the converting pass cannot
fit.
The non-strict callers change too (`hts_convertStringFromUTF8`,
`hts_convertStringUTF8ToSystem`, hence WinHTTrack's ANSI entry points).
Approximations that used to pass through (folding full-width to
half-width, or dropping an accent) now come out as `'?'`. This is
intended, but the change is visible well beyond the dangerous cases. It
also brings the Windows build to where the iconv build already stood:
failing instead of approximating.
The new `-#test=nobestfit` self-test asserts the best-fit mappings and
the two sizing lengths themselves before it tests the engine, so it
cannot pass vacuously if any of them ever changes. U+00A5 alone would
not have found the sizing bug: it is one byte either way. The
`syscharset` oracle needed the flag as well, or it disagrees with the
engine on any ACP that best-fits. Found by the httrack-windows session
while finishing its `newlang.cpp` UTF-8 work.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit ecaf54eb22591c892aea307d6e9546d502ec0196
Author: Xavier Roche <roche@httrack.com>
Date: Tue Aug 25 02:07:17 2026 +0200
WebHTTrack keeps option fields it never sends, and mutes a blank User-Agent (#1422)
The wizard's type-check list and its eight extension/MIME pairs were
written to `winprofile.ini` and read back into the form, but
`step4.html` never put them on the command line, so the crawl never saw
them. The browser-id box had the opposite problem: cleared, it emitted
`--user-agent ""`, and an empty `-F` switches the header off instead of
falling back to the engine's own value. The command block now emits
`--check-type=N`, one `--assume "ext=mime"` per filled pair, and nothing
at all for a blank browser id. This is engine-side only: the keys
already exist in the shared winprofile block, `hts_optalias` already
carries `check-type` and `assume`, and `winprofile-keys.tsv` is
untouched.
The second commit fixes a footgun the first one opened and a twin
already on master. `${arg:mimeN}` sat outside the `${test:extN:}` guard,
so emptying an extension while leaving its MIME type filled dropped a
bare token onto the command line, which the engine reads as options of
its own: a MIME field holding ` -O /some/path` silently relocates the
whole mirror. `${arg:portprox}` has the same shape and predates this PR.
Nine sites in all, now wrapped in `${do:if-not-empty:<guard>}` the way
the proxy scheme already was. htsserver is a local UI and the person
filling the form is the person whose crawl it is, so this is a way to
lose work rather than a privilege boundary anyone crosses, but emptying
one box and not its neighbour is an ordinary thing to do. Test 358
asserted the absence of `=<mime>` rather than of the value, so it went
green on the injecting tree; it now asserts the value is gone and probes
the shape directly.
`windebug` is the original bug mirrored, reaching the engine as
`--debug-headers` and never saved, so it reverts on reload. That one
needs a new key in the shared block and WinHTTrack's own spelling has to
decide it, so I left it to the httrack-windows side. An extension
beginning with a dash still kills the run before any log is written,
which is #1179's narrow allowlist rather than anything here: filed as
#1425. Test 274 asserted `--user-agent ""` as the observable for #1186,
that a key present in the file beats the wizard's startup default; the
property still holds and is now checked on the rendered box, which is
what #1186 was about.
Verified against a running htsserver rather than by reading templates:
the new test posts browser-shaped forms, starts a real crawl and reads
`hts-cache/doit.log` for the argv the engine got. The `--assume` half
also has to change the mirror, since `data.foo` is only retyped and
parsed once the rule arrives.
Closes #1386
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 82aad613bd47afad784307a150f274f04a1e0115
Author: Xavier Roche <roche@httrack.com>
Date: Tue Aug 25 02:07:13 2026 +0200
A write error outside the fatal class still read as the end of the body (#1417)
`http_xfread1()` advances `r->size` before the `fwrite`, so a body the
failing write happened to complete still satisfied the size test at the
bottom and came back as a clean EOF. #1391 stopped that for the fatal
class alone; the non-fatal class now has a code of its own,
`STATUSCODE_IO_ERROR`, and the four size-based sites in `back_wait()`
test for either. The same function also discarded `fflush()`'s return,
and glibc's later `fclose()` reports nothing once that flush has taken
the error, so a body with no Content-Length and no chunking (the one
shape with no completion test to fall back on) was recorded as mirrored
with nothing on disk, the fatal class included. The decode step is the
third door onto the same bug: `hts_codec_unpack()` and `hts_zunpack()`
now leave a local failure's errno behind and clear it for a body the
decoder refused, which is what `back_finalize()` needs to stop blaming
our own disk on the server.
Report rather than abort is the call for the non-fatal class, since a
full disk means the next file fails too while a broken pipe is about one
destination. 350 and 355 cover both classes with controls, and every
guard was reverted in turn and reds.
Closes #1398
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 7938d709ac2926117c3359cfdabc4d3eea6ba33e
Author: Xavier Roche <roche@httrack.com>
Date: Tue Aug 25 02:06:52 2026 +0200
Pin the sanitizer collector's pattern arms, close its trailing-dir hole (#1416)
`tools/ci-sanitizer-report.sh` is the only path from a sanitizer finding
to a red build, and its pattern also gates `msan` (MemorySanitizer,
clang), a required check. Before this PR only the `runtime error:` arm
(gcc's UBSan) had a fixture. Mutation testing against
`tests/349_sanitizer-stderr-capture.test` found ten more pattern pieces
that could be dropped with the test staying green. That covers every
other name in the ERROR-group, the whole WARNING arm, the SUMMARY arm,
DEADLYSIGNAL, and the two combinations that fully blind the collector to
LeakSanitizer or MemorySanitizer. This predates #1404, so it isn't a
regression, just newly load-bearing now that msan is required.
Each gap gets its own fixture, carrying only the marker for the arm it
pins so a different arm can't catch it and hide the gap. ASan and
LeakSanitizer's own ERROR lines and a SUMMARY-only line cover the
ERROR-group and SUMMARY arms. MemorySanitizer and ThreadSanitizer get
both their WARNING form (their own findings) and their ERROR form, since
compiler-rt's crash-reporting path is shared and a crash makes either
one report through the ERROR-group instead. A bare `DEADLYSIGNAL` marker
line comes from a real ASan crash report, where it appears on its own
with no `ERROR:` prefix. Two full multi-line reports pin the
LeakSanitizer- and MemorySanitizer-blind combinations. `ERROR:
UndefinedBehaviorSanitizer` and `ERROR: libFuzzer` stay unpinned on
purpose: gcc's UBSan never emits the former, and the fuzz job never
calls this script, so the latter can't fire here either.
`need_dir` already refused a missing `-s STDERR_DIR` or `LOG_PATH_DIR`
(#1374); a missing trailing `TEST_LOG_DIR` was silently skipped instead,
so a typo'd log path read as clean rather than broken. Closed the same
way, with a test row for it.
Closes #1405
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 8c91eae788db4f5caa259f001ab74adfa3223049
Author: Xavier Roche <roche@httrack.com>
Date: Tue Aug 25 01:42:25 2026 +0200
The --wide- and --tiny- alias prefixes drop the connection count they exist to add (#1424)
The `--wide-` and `--tiny-` prefixes build the connection count they
exist to add and then drop it: `optalias_check()` writes `c32` or `c1`
into a local buffer nothing reads, so `--wide-mirror` is `--mirror` at
the default four connections. The write is dead in the initial 3.20.2
import, so this is an unfinished feature rather than a regression, and
nothing documents the prefix: cmdguide lists `--wide`, `--tiny` and
`--ultrawide` as standalone presets, and neither the man page nor the
help mentions the prefixed spellings. They are live all the same, since
`optreal_or_alias()` already treats `--wide-<alias>` as an option name
and 275 passes one around as one.
The count is glued onto the short form the alias expands to, the way the
table writes its own clusters: `--wide-mirror` becomes `-wc32` and
`--tiny-spider` becomes `-p0C0I0tc1`. The value comes from the `--wide`
and `--tiny` rows rather than a second copy of 32 and 1. Gluing is only
safe where the expansion is a plain letter-and-digit cluster, and an
earlier draft of this branch glued everywhere, which broke three ways:
`-#hc32` misses the whole-word match on `-#h`, so `--wide-version`
printed the usage dump instead of the version; `-N%h%p/%n.%tc32`
corrupts the user's template and drops the count anyway; and
`--wide-sockets 8` became `-c8c32`, silently 32. The rule now admits 44
of the 172 alias names, excluding the classes whose value shares or owns
the word, long forms, the `-%` and `-#` families (whose option names are
not self-delimiting, so `-%r` plus a `c` is the `-%rc` of `--warc-cdx`),
a cluster already carrying `-c`, and `-h`.
On the other 128 the alias applies without the count and HTTrack says so
on stderr, on the command line and in a config file alike, rather than
dropping it in silence as before:
* Warning: the wide- prefix cannot add its connection count to --path,
so --path runs without it; write --wide --path instead
**A choice to make, and the branch is on the safer side of it.** The
alternative is to refuse those spellings outright, which reads well ("do
not silently ignore what the user asked for") but turns 256 spellings
shipped since 3.48.10 from running to exit 255: `--wide-path /out`,
`--tiny-allow '*.gif'`, `--wide-user-agent X`, `--wide-clean`,
`--wide-sockets 8`, `--wide-structure`, `--wide-cache`, `--wide-proxy`,
`--wide-version`, `--wide-warc` and 118 more names. As pushed, nothing
that runs today stops running and no output changes beyond the warning
line. The refusal path is still there and tested: `#define
OPTALIAS_PREFIX_STRICT 1` at src/htsalias.c:347 switches to it, and the
expectations in `st_optalias` (`WARNS` back to `REFUSES`) and in 361
follow.
The new test asserts the argv echo hts-log.txt writes on its second
line, which is the word the engine was handed, then the security
limiter's clamp line, which is the count reaching the option rather than
just the rebuilt argv. It matches the echo by pattern rather than by
whole word where the spelling is not this test's to fix, so
`--structure`'s value is checked as a template followed by a word break
and the assertions hold whether #1380 leaves it glued or detaches it;
the merged tree against #1380's head was built and run to confirm. A
`--sockets 3` pair pins the glued digits, since 32 clamps only if both
are read. It covers one warned spelling from each excluded class and
checks the base alias still did its job: `--wide-structure` keeps laying
out by its template, `--wide-sockets 8` stays at 8, `--wide-warc` still
writes an archive, `--tiny-wide` still gets 32, `--wide-version` still
prints the version. `-#test=optalias` pins the exact expansion of every
accepted and warned spelling, and asserts no unprefixed alias warns. All
172 names were swept twice against the built binary, with and without a
URL: the 44 glue with the right echo and no warning, the 128 warn and
are otherwise indistinguishable from the unprefixed run.
Closes #1423
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 38f0cc820e7f0aa32b601e88960bb3cc5f615986
Author: Xavier Roche <roche@httrack.com>
Date: Tue Aug 25 01:15:44 2026 +0200
-#C faults on the first cache entry it names (#1412)
`url_savename()` opened by loading `sback->lnk` into a local that
nothing reads until a branch the `-#C` lister never enters. That lister
has no backing and passes NULL, so listing a legacy `hts-cache/new.ndx`
faults on the first entry its pattern matches, before it prints
anything. The local now comes from the block that uses it, and the
type-probing request further down, the last thing on that path to touch
the backing, is skipped when there is none.
That second half is not belt and braces: reading `sback->lnk` up front
is undefined behaviour when it is NULL, so the compiler may delete any
NULL test of `sback` that follows, and at -O2 it does. A mutant carrying
the load with the guard in place still crashed, in the guarded branch.
The new test drives `url_savename()` with no backing through a
`nosback=` knob on `-#test=savename`, and reds on either mutant. End to
end, a hand-built legacy ndx over a one-entry cache kills master at
htsname.c:413 and lists cleanly once patched.
Closes #1393
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 17634476f90ad8d20eaef93519322e14ef63c207
Author: Xavier Roche <roche@httrack.com>
Date: Tue Aug 25 01:07:48 2026 +0200
A long-path self-test aborted or passed depending on the length of $TMPDIR (#1411)
Four long-path self-tests built their directory by appending fixed-size
segments while a counter stayed under 300, then asserted the result had
passed MAX_PATH. The starting length is the base directory's, so the
final value lands somewhere different for every `$TMPDIR`, and for a
band of lengths it lands on exactly 260, where the `> 260` assertion
aborts. Two of the four asserted on the directory alone, so they could
fire; the other two had a leaf name's worth of slack and never did. It
reads as an intermittent CI abort because where a machine's build tree
happens to live is what decides whether it trips.
The four copies are now one helper that loops on the MAX_PATH comparison
itself, so no base length can leave it short. Both tests sweep 41
consecutive base-dir lengths, one segment's worth, so every residue is
covered whatever `$TMPDIR` is. The pre-fix engine aborts at two of the
41 and the fixed one at none, standalone and through `make check`.
The sweep on its own would not have caught a regression. The assertion
it replaced was the only thing pinning that the path ever exceeded
MAX_PATH, so reverting the loop with that assertion gone reintroduces
the bug and the suite stays green. The postcondition is now asserted
inside the helper and kept at both call sites, and both mutants go red.
One thing found while in here: the bound was in bytes, but Windows
measures MAX_PATH in UTF-16 units, so the 5-byte, 2-unit non-ASCII
segment let an exit at 261 bytes be 259 units, under the limit these
tests exist to cross. The helper now charges that segment by code point.
That part has no local red test, since only the Windows path can observe
it.
Closes #1409
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 5bf74ae4f120d71c67946afec8bc953ebadee4ec
Author: Xavier Roche <roche@httrack.com>
Date: Tue Aug 25 00:59:47 2026 +0200
zipAbandonFileInZip reports success when nothing was truncated (#1413)
`zipAbandonFileInZip()` rolls a partial cache member back by rewinding
to its local header and truncating there, and both ends of that were
incomplete. The Win32 filefunc tables in `src/minizip/iowin32.c` set
neither `ztruncate64_file` nor `zflush_file`, leaving a caller who fills
one from an uninitialized struct with a wild pointer; and where a
backend carries no truncate at all, `call_ztruncate64()` degrades to a
no-op that the rollback then reported as `ZIP_OK`. The caller keeps an
archive whose leftover tail hides the central directory from any reader
once it outgrows the 64KB backscan, and hears nothing about it.
`call_ztruncate64()` also collapses any non-zero backend return to a
failure, since `_chsize_s()` reports one as an errno rather than -1,
which would otherwise read as "this backend has no truncate".
This implements `win32_truncate64_file_func` (`SetFilePointerEx` plus
`SetEndOfFile`, putting the file position back where the caller had it,
as `ftruncate` leaves it) and `win32_flush_file_func`
(`FlushFileBuffers`), wires both into all four tables, and adds a
`ZIP_NOTRUNCATED` return for a rollback that could only rewind. The
no-op degradation stays, since failing a rollback the caller has already
performed is worse, but it is no longer silent: the cache logs the
incomplete rollback. Of the three callers only
`cache_zip_write_failed()` ignored the return. `iowin32.c` gains the
`.orig`/`.diff` pair the other patched minizip files carry, which is
most of the added line count.
`-#test=cache-writefail`, the self-test that drives the production
abandon path, was opening its injected ZIP through the 32-bit `zipOpen2`
where the entry is NULL, so it made zero truncate calls and could not
have seen any of this regress. It now opens through the table the cache
itself uses (factored out as `hts_zip_filefunc64()`) behind a counting
interposer, and asserts both that the table carries a truncate and that
the abandoned member went through it. It also drives a table carrying no
truncate at all, to hold the deliberate degradation: the entry drops,
the mirror lives, and the warning is printed. Test 356 drives a new
`-#test=zip-abandon-notrunc`, building the same archive with and without
the entry and grading the return, the leftover tail, and whether
Python's `zipfile` still opens it.
The Win32 half is latent and unexercised by Linux CI: nothing in the
tree fills those tables today, the engine reaches minizip through
`hts_zipOpen_utf8` on the 64-bit fopen table, and `iowin32.c` only ships
in `EXTRA_DIST` for the Windows projects, so no leg here compiles it.
Being `_WIN32`-only it moves no POSIX ABI. I did compile it locally
against a stub `windows.h` and ran a truncate through the sequence the
abandon uses: the file shortens, the position ends up where the caller
left it, and the next write lands at the rewound offset. Dropping
`SetEndOfFile` and dropping the position restore each red that probe. It
stays out of the tree, since nothing else here builds against a stub
Win32 and an uncompiled probe rots unnoticed. What the suite covers on
Linux is the reporting contract and the cache path, not the Win32 code.
Closes #1402
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 02f25f85e5180aa729f154bf0432a0709c407756
Author: Xavier Roche <roche@httrack.com>
Date: Mon Aug 24 21:59:20 2026 +0200
htsserver refuses the host's own name since the Host check landed (#1408)
#1372 gave the panel a Host check, and with it a regression against
3.49.23. `host_is_self()` vouches only for `localhost`, an address
literal, or the exact string passed to `--bind`, so `htsserver --bind
192.168.1.5` browsed as `http://myhost.lan:8080/` now answers 403 where
3.49.23 served it. Default WebHTTrack never saw this: it binds loopback
and opens `127.0.0.1`. Binding by name (`--bind myhost.lan`) already
works and stays the exact workaround.
The vouched set now also holds the names this host answers to. Those are
`gethostname()`, the canonical name it resolves to, and the name the
bound address reverse-resolves to, kept only when that name resolves
back to it. All of it is collected once, before the first request is
served, which is what keeps rebinding closed: a name presented by a
request is never looked up, so a page cannot point a name of its own at
our address and have the lookup vouch for it. Accepting any name that
merely resolves to the bound address would have reopened the hole #1372
was written to close.
The collection runs at the top of `smallserver()` rather than at bind
time, because the launcher prints the URL as soon as
`smallserver_init()` returns. With the collection on the bind path and a
blackholed nameserver, that line took 20s to appear, against a 20s cap
in `65_port-siblings.test`.
Test 341 adds this host's own name as an accepted authority, upper-cased
as well, and four near-misses as refused ones, one on each side of the
name. Where the resolver has one, it also sends `ip6-localhost` to an
`::1`-bound server: that name points at the bound address and is still
refused. Six mutants pin the rows. Drop the lookup, match by prefix or
by suffix, compare case-sensitively, or resolve the request's own name,
and each turns one row red.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit eeba324c9b5d0536b8f4c2ce2f657658c610e970
Author: Xavier Roche <roche@httrack.com>
Date: Mon Aug 24 20:49:15 2026 +0200
Move the catalogs to UTF-8 and retire LANGUAGE_CHARSET (#1407)
Each of the 30 catalogs named its own legacy charset in a
`LANGUAGE_CHARSET` key, and every consumer had to decode by it: the 24
served pages copied the value straight into their `<meta>`,
`htsserver.c` converted both the POST body and the language menu with
it, and four tests read it back. Nothing could catch a file that
disagreed with its own declaration, because a single-byte charset
decodes anything. Each catalog is now converted from the charset it
declared, the key is gone, the pages declare `utf-8` outright, and both
conversions in `htsserver.c` go with it.
Stacked on #1403, and that order is not optional. Greek declared
ISO-8859-7 with cp1253 bytes, so no blanket rule converts it correctly,
and once a file is UTF-8 there is no declaration left to reinterpret:
the wrong reading would have become the text. Retarget this to master
once #1403 lands.
The conversion round-trips: re-encoding every catalog to the charset it
used to declare, through iconv rather than the codec the conversion
used, reproduces the original bytes exactly for all 30. **Two codec
choices are load-bearing and pinned deliberately, not left to whoever
re-runs the conversion.** Japanese decodes with Python's `shift_jis`,
which maps 0x5C to a backslash; glibc's `iconv -f SHIFT_JIS` maps it to
the yen sign, which measured out at 101 lines gaining `¥` and all 93
lines carrying a `\n` escape losing it. Chinese-BIG5 decodes with
`cp950`, since it holds `f9d8` four times and a strict `big5` decode
errors at offset 8534, while `cp950`, `big5hkscs` and iconv agree it is
U+88CF. The round trip passes under either choice because the encode
direction is not where the asymmetry lives, so these two are stated
exceptions rather than facts the gate establishes.
The repertoire check in `62_lang-integrity` was replaced rather than
adapted. Its table listed letters produced by reading a catalog's legacy
bytes as Latin-1, a mistake that cannot happen once the file is UTF-8,
so adapting it would have meant tuning the control until it fit. What
replaces it tests the round trip per line: mojibake is exactly text that
survives being encoded back to Latin-1 and read as UTF-8 again. A byte
pattern cannot do this job, which took two attempts to establish.
Narrow, it caught 4 of 243 misread values in Francais; widened, it
flagged an accented letter before a guillemet or a degree sign, and
French requires a no-break space after an accent before `! ? : ; »`. One
class stays uncovered: a catalog converted from the wrong *legacy*
source, Latin-2 read as Latin-1, is valid UTF-8 with plausible letters,
and only the round-trip proof above catches it.
`tools/build_strings.sh` in httrack-android asserts `LANGUAGE_CHARSET`
on line 9 and exits 1, then reads line 10 as the source charset. This
removes both lines, so Android string regeneration breaks on the next
engine sync; that session has the fix and is landing it with the pin
rather than before it. httrack-windows needs `conv_printf()` fixed
first, since it pairs bytes on the ANSI codepage before #144's UTF-8
decode.
One consequence is accepted rather than solved, and documented in
`doc/winprofile-ini.md` rather than fixed here. WebHTTrack now posts
UTF-8 into `winprofile.ini`, where WinHTTrack writes ANSI codepage bytes
and the file declares no encoding, so the two disagree on every
non-ASCII value. They agreed before only when the host codepage matched
the catalog's charset, which was common but never guaranteed, so this
widens an existing divergence rather than creating one.
`hts_getcategory` is exported with a documented "Not UTF-8" contract
that this makes false, and on Windows its caller converts already-UTF-8
bytes a second time. Fixing it properly needs a `Charset=` key in the
profile, shipped to readers a release before any writer emits it.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 1f9829cb39bd76fea88583c2a8231187928e7f8d
Author: Xavier Roche <roche@httrack.com>
Date: Mon Aug 24 20:08:47 2026 +0200
A translated combo list with the wrong row count picks the wrong option (#1400)
WinHTTrack fills eight combo boxes from `LISTDEF_*` lists and reads each
back as a bare row index that selects an engine option: robots policy,
the three travel modes, build structure, filter type, check type, log
type, wizard action. A catalog whose list has one row fewer or more than
English shifts everything after the gap, so the mirror runs with the
option next to the one the user clicked, silently, with the dialog still
showing their choice. It only goes wrong in the locale that drifted, so
it never reproduces in English, and the GUI cannot catch it: its
selftest compares against the English list, and a runtime check on the
live combo would fire on the user rather than on CI.
Exact per string, unlike the line-break counts it overlaps with, which
are pinned per catalog and total over every string in the file. That
total is the problem: a list gaining a row while unrelated prose loses a
line break leaves the pin unmoved. Clean on every catalog today, so it
pins the state rather than fixing anything, and the number of lists is
pinned too, since a new `LISTDEF_` is a new combo box whose indices want
looking at.
Found by the httrack-windows session, which measured the blast radius
after the dropdown fix in #1375 and showed the count-parity gap was
wider than I had sized it.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 3efd717957d725aa7afd8eb9b4d18b8159673bf8
Author: Xavier Roche <roche@httrack.com>
Date: Mon Aug 24 19:30:56 2026 +0200
Greek.txt declares ISO-8859-7 but is written in cp1253 (#1403)
lang/Greek.txt declares ISO-8859-7, but five of its values hold 0xa2,
which is capital alpha with tonos in cp1253 and a right single quote in
ISO-8859-7. Every front end that obeys the declaration renders the
Cancel button as a quote mark followed by kappa, and the same happens to
the start-page action, the &Open menu item and the unknown-operation
error.
The file is cp1253, so the declaration is the part that is wrong.
Declaring `windows-1253` and changing the one remaining ISO-8859-7 byte,
the 0xb6 that opens the host-alias help text on line 1050, makes it
uniform. Two lines, and all six values then render correctly on both
front ends. The other repair, rewriting the five bytes to 0xb6 and
keeping the ISO declaration, fixes WebHTTrack and breaks WinHTTrack,
which decodes catalog bytes with the system ACP rather than the
declaration: it would take Greek there from one wrong string to six, in
the release where that GUI is what ships.
The charset check in `62_lang-integrity` could not see this. What #963
added catches a codepage byte only inside C1, U+0080 to U+009F, and 0xa2
sits above it. The test now asserts that no value opens with a curly
quote that never closes, pins Cancel to its cp1253 bytes, and requires
no 0xb6 to remain. Each sits behind a positive control, so none can be
deleted quietly.
This predates 3.49, so it is not a regression against 3.49.23.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit bd2a5b27f67af3fd8aa6335fa2c102e446fdfc20
Author: Xavier Roche <roche@httrack.com>
Date: Mon Aug 24 18:52:01 2026 +0200
The emulated leg spends a third of its time compiling headers for a property that cannot vary by target (#1399)
The emulated s390x leg takes 31 minutes, and 28% of that is two tests:
`269_install-headers-order` at 5m45s and `206_install-headers-c99` at
2m46s. Both sweep compilers, one process per case (269 is every ordered
pair of installed headers, so n² of them), and qemu user-mode emulation
pays a cold start on every `execve`. Measured from the job log: apt
141s, bootstrap and configure 173s, build 410s, `make check` 1138s, of
which those two are 511s.
Neither property can vary by target. 269 asks whether the headers
survive any include order; 206 asks whether they compile as strict ISO C
and as C++. An include order that works on x86-64 works on s390x, and
both already run on every native leg.
What does vary by target is `config.h`, which is in `DevIncludes_DATA`,
so compiling the installed headers on s390x does exercise that
architecture's generated config. That is why only these two are skipped:
`205_install-headers` still runs there, still compiles every installed
header standalone in both `HTS_INTERNAL_BYTECODE` states, and so keeps
the part of the coverage that is actually architecture-specific.
`278_install-headers-msvc` is untouched.
The guard splits the way `is_windows` and `skip_on_windows` do:
`is_emulated` reads `EMULATED_ARCH`, which `tools/emulated-suite.sh`
already requires, and `skip_on_emulated` is the errexit-safe wrapper.
Proved both ways: both tests PASS on a native run and SKIP with
`EMULATED_ARCH` set, 205 and 278 still PASS with it set, and the
emulated pass floor has 91 to spare.
The guard itself is tested, because its worst failure is silent. Wired
backwards it skips on every leg, and a skip is green, so no job would go
red and the emulated pass floor is a lower bound that cannot see two
extra skips either way. `257_testlib-asserts` gains a `skips` verdict
helper and three cases: the guard holds when `EMULATED_ARCH` is unset,
holds when it is exported but empty (the shape a workflow typo leaves),
and names both the arch and the reason when it fires. Each kills a
distinct mutant: the plain inversion, an inversion that cannot trip `set
-u`, and an `is_emulated` that counts set-but-empty as emulated.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit c1784971872f285884f4be8b74022dc09bb1ce31
Author: Xavier Roche <roche@httrack.com>
Date: Mon Aug 24 18:45:57 2026 +0200
The DNS resolver declares after a statement, where our CI cannot see it (#1406)
#1392 added a write above the `#if HTS_INET6`, so the declaration
opening each arm now follows a statement. It is legal as this tree
builds. `configure.ac` leaves `-Wdeclaration-after-statement` off on
purpose, since nothing here sets `-std=` and we compile as gnu17, so no
build of ours sees it. The Android NDK build sets `-std=` and warns.
The write becomes a local, set by whichever arm compiles, with one write
through the caller's pointer beside the single `return`. Two branch
conditions lose their `permanent != NULL` test with it. A single-exit
rewrite only holds if nothing leaves early: this function has one
`return` and no `goto`, and the wrapper's own guard writes the flag
itself.
Building the tree under the flag found the other two sites this batch
introduced. `htsparse.c`'s `unanswered` is read once, so it goes into
the condition it feeds. `htscore.c`'s is subtler: that declaration was
fine until #1390 put a log statement above it, so blame names #686 while
the defect is ours.
The same census says the flag cannot simply be turned on. 11 further
engine sites and 27 in the self-tests predate this batch, which is its
own change, after the freeze. No test here: behaviour is unchanged and
the DNS self-test already drives every branch.
Reported by the httrack-android session against the NDK build, in-window
for 3.50.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 6e29ab3bf216a503823fef14689b11c61c5303d9
Author: Xavier Roche <roche@httrack.com>
Date: Mon Aug 24 10:54:25 2026 +0200
Every CI leg pins the OpenSSL that brew installs do not use (#1401)
The homebrew-core httrack formula depends on `openssl@4`, and Homebrew is where most macOS users get httrack. Our three macOS jobs pinned `openssl@3`, so no leg of ours ever built what those users run: a break there would have reached us as a bug report, not a red check. This points them at `openssl@4`.
The engine needed no changes. Every OpenSSL identifier we use is still declared in the 4.0.1 headers with no deprecation attribute, certificate verification is still off by default (`verify_mode = SSL_VERIFY_NONE` in `SSL_CTX_new_ex`), the default security level is still 2, and the two `#if OPENSSL_VERSION_NUMBER` guards in `htslib.c` resolve the same way for 4.0.1's `0x40000010L`. I built master against a from-source OpenSSL 4.0.1 and ran the suite: zero warnings, 370 pass / 12 skip / 0 fail, with `14_local-https.test` and the other TLS tests running instead of skipping, plus a live https crawl. Details in httrack-works `design/openssl4-compat.md`.
3.x keeps its coverage from every Linux leg, the Debian package build and distcheck, so nothing is lost by moving. `macos-release.yml` stays on `openssl@3` on purpose: it decides which dylibs the signed DMG carries, which is a shipping decision and not a CI one, and the bundling mechanics the `macos-app` job exercises are soname-agnostic.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit acc828507d24a287867923870655f7c08eb881f1
Author: Xavier Roche <roche@httrack.com>
Date: Mon Aug 24 10:49:12 2026 +0200
The stderr capture's tee is reaped by nobody (#1397)
#1374's PATH shim backgrounds a `tee` and then execs the program, so nobody reaps the tee. The stderr the caller sees is the copy tee writes, so a test doing `cmd 2>FILE` and reading FILE back can find it short or empty: the program had exited while the tee was still draining. `tests/local-crawl.sh` runs every crawl pass as `>FILE 2>&1`, the same shape. Measured 8 shims wide, which is how `make check -j` runs them, 50 lines expected: 17 of 320 runs short before, 0 of 320 after.
Keeping the exec is not an option. The caller waits only on the shim's pid, and the process feeding the tee is the one the tee is waiting on, so no fan-out can finish before the process the caller is watching does. That leaves making the shim a parent that reaps the copy before it exits, and a shell cannot be that parent: POSIX leaves SIGINT and SIGQUIT untrappable in a backgrounded shell, and 255, 262 and 263 each `kill -INT` the engine while it is backgrounded. So the shim is now a small C program that forks, pumps stderr both to the caller and to the log, and hands back the exit status, the death by signal, the pid-directed signals, stdin and the process group, plus `PR_SET_PDEATHSIG` so a `kill -9` on the shim still takes the engine with it.
The exec'd shim got two things for free that this one has to ask for. It `_exit()`s, because an interposer a test preloaded lands on the shim too and 113's rewrites its report file from every process whose destructors run. And it stops pumping once the child is reaped rather than at EOF, which an htsserver child still holding the pipe never gives (289).
Inter-descriptor ordering is what a fan-out cannot give back: stderr takes a hop stdout does not, so `>FILE 2>&1` can swap a line or two, as it already did before. The full suite is green with the capture on and off, same tally and same skip list, so nothing reads interleaved output. 349 gains the case that was missing, a capture-on `2>FILE` read back with a shell builtin, 8 wide and four rounds: red on the old shim five runs out of five, green here ten out of ten.
The forwarding gate first named `SI_USER`, which Darwin does not report for a `kill(2)` (XNU leaves `si_code` at 0 and defines `SI_USER` as 0x10001), so on macOS the shim relayed nothing and 349's pid-directed SIGTERM sat until the 600s guard fired. It reads `si_code <= 0` now, the convention both kernels honour. 349 bounds that wait as well, so a shim that swallows a signal is named in 30 seconds rather than arriving as a timeout with no case attached.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit f114d2ba7898d99ebcda333c31cb38816554e00f
Author: Xavier Roche <roche@httrack.com>
Date: Mon Aug 24 10:16:42 2026 +0200
The WARC index offset, and three writers that fail without saying so (#1381)
Four ways the WARC and cache writers fail without saying so.
The CDXJ record offset was read with `ftell()`, whose `long` tops out at
2GB and is 32-bit even in 64-bit Windows builds, so past that ceiling
every record is indexed at a wrong offset. The same counter gates
`--warc-max-size` rotation. It now goes through `warc_stream_offset()`,
built on the portable `ftello`/`_ftelli64` the tree already uses for
`fpsize()`.
Copying an on-disk body into the cache read to EOF and never looked at
`ferror()`, so a failing disk committed a truncated entry under its own
declared `X-Size` and the mirror carried on. Detecting that was only
half of it: the handler then passed the open member to
`zipCloseFileInZip()`, which finalized the partial body into the ZIP
anyway, so the log said "entry not cached" while the entry was cached,
short. Minizip cannot take a central-directory record back once it is
appended, so the rollback has to happen before that.
`zipAbandonFileInZip()` ends the compressor, drops the pending central
header and rewinds to the member's local header,. The read-back side had
the matching hole: the direct-disk branch stopped on the ZIP's EOF and
raised nothing when `X-Size` was still short, where its in-memory
sibling compares the read length. It fails the entry the same way now.
Two cache read-error paths built their message with `sprintf(r.msg,
"...%s", strerror(errno))`. `msg` is 80 bytes inside an installed-header
struct and `strerror()` is locale-sized, so that is an ABI-visible smash
on a long enough error string, which `_FORTIFY_SOURCE` catches as an
abort. Both go through the bounded `htsblk_failf()` that 31 other sites
use.
WACZ packaging opened its ZIP with minizip's default filefunc, whose
plain `fopen` mangles a non-ASCII path on Windows: the bug #630 fixed
for the cache, reintroduced in a second writer. The cache's UTF-8
filefunc moves to `htszlib.c` as `hts_zipOpen_utf8`/`hts_unzOpen_utf8`
and both writers share it.
These are compatibility contracts, so the change was checked to write
the same bytes: one crawl through a master build and this branch
produces an identical cache ZIP, WARC, CDXJ and WACZ, with a
master-vs-master run as the control for the per-run UUIDs and timestamps
and a one-byte mutation as the negative control.
Test 351 drives two self-tests. `-#test=warc-offset` table-tests the
tell across the 2GB and 4GB boundaries up to 1TB by seeking, which costs
no disk, and pins the emitted CDXJ text past 4GB; static asserts pin the
tell and the offset field at 64 bits, since a narrowing mutant is
invisible on LP64, where it is not a defect. `-#test=cache-readfail`
drives `cache_add()` itself over an unreadable source, asserts the entry
is absent, that a sibling stored after the rollback still round-trips,
and that a short entry read back through `cache_readex()` fails; it also
overruns the failure message with a poisoned canary in the neighbouring
field, since `hts_init_htsblk()` already sets the status the assertion
used to check for.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit d07c53449dbdb20b75eff8e02b5a51cc9214abfa
Author: Xavier Roche <roche@httrack.com>
Date: Mon Aug 24 10:02:05 2026 +0200
A one-tick clock straddle reddens the suite-timeout test under emulation (#1396)
`105_suite-timeout` compares two reads of `SECONDS` for exact equality,
an assertion latent since #1231 landed it on 2026-08-13. The fixture is
`echo "left=$(budget_left) at=$SECONDS"`, and `budget_left` computes `60
- SECONDS` inside the command substitution, so the two reads happen at
different instants. `SECONDS` floors, so when they fall either side of a
second boundary the test reports `left` one higher than it expects and
fails.
Natively the window between the reads is about 250 microseconds and it
almost never fires. On the emulated s390x leg, where every instruction
is interpreted and `make check -j` adds contention, a scheduling hiccup
of a few hundred milliseconds in that window is ordinary: it reddened
#1391 with `a 60s budget left 58 with 3 gone, want 57`, on a branch that
had passed the same leg twice earlier the same day and that touches
neither this test nor the timeout helpers.
The fixture now samples the clock either side of the call and the
assertion accepts the one-tick range those two reads bracket. That is
not a tolerance: it is still exact against the clock the child itself
read, so a wrong epoch stays a failure. Proved both ways. Replaying the
fixture with the 0.4s straddle that reproduces the flake passes five
times out of five, and making `budget_left` drop the elapsed term fails
with `left 60 with 2 to 2 gone, want 58..58`.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 5b5e3d252279fb0c1d2d931708da0a083050999e
Author: Xavier Roche <roche@httrack.com>
Date: Mon Aug 24 00:00:33 2026 +0200
A full disk reads as a dropped connection, and the mirror carries on (#1391)
A body written straight to disk goes through `http_xfread1()`, which
advances `r->size` before the `fwrite`. When the read completing the
body is also the one whose write fails, the size test at the bottom of
that function returns `READ_EOF` and the error is gone before any caller
sees it. `back_wait()`'s chunk-end and keep-alive tests do the same on
the way back. The disk error surfaced as "Interrupted transfer" and then
"Incorrect length", the engine retried a full disk twice as if the
network had blinked, and the mirror carried on until its end-of-update
purge measured the site against a truncated copy. Bodies of 4096 to
16384 bytes were the window, which covers most of a page's assets. A
sweep of 17 sizes across keep-alive, `Connection: close` and chunked
puts 19 laundered cells on the unfixed build and none on the fixed one.
`errno` cannot be read where that decision is taken, since `free()` and
`fflush()` run in between, so the verdict is taken at the write and
carried in a new internal status code. Ordering is the fix: the
body-complete tests now run after that status is checked rather than
before it. Both halves are load-bearing, since the chunk-end sibling on
its own still lets a chunked 4097-byte body through.
Three closes had the matching hole. stdio holds a small body until
`fclose`, and the parsed-page flush, the direct-to-disk close and
`filesave()` all threw that return away, so a file that never reached
the disk counted as written: a two-page crawl onto `/dev/full` ended "3
files written, No errors". They check it now and report through one
helper. `back_set_finished()`, which owns the direct-to-disk close,
takes the `httrackp` it lacked; all 50 call sites already had one.
Test 350 covers that window and its chunked twin, a `--tolerant` arm
proving a disk-cut body is not also blamed on the server, an `EPIPE` arm
proving a write error outside the fatal class still leaves the mirror
running, and an assertion that the files mirrored before the abort
survive it. It skips where a write through a `/dev/full` symlink
succeeds, which is what MSYS does, so the guard probes the mechanism
rather than the node. Test 352 drives `filesave()` through
`-#test=filesave`, since no crawl reaches its only caller.
The new status code is an added enumerator in an installed header: no
layout change, and nothing outside the engine matches on it. The process
still exits 0 after a fatal abort, which is #1394 and not this change to
make. Found by the httrack-android robustness audit.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 80fb2153d33a53a12b99888ab898b8eb1ad6dcdd
Author: Xavier Roche <roche@httrack.com>
Date: Sun Aug 23 20:52:59 2026 +0200
A host that fails to resolve during an outage stays dead for the whole crawl (#1392)
The DNS cache reads a resolve count of 0 as a valid answer and stores it
with no expiry, so a name that did not resolve is never asked about
again. A resolve that times out escapes this, and the comment says why.
A fast failure does not, and that is what a device with no connectivity
returns: every host first met during an outage stays unreachable for the
rest of the crawl, long after the network is back. On a single-host
mirror the remainder fails, and the per-link retries are spent on cache
hits without ever touching DNS.
Negative answers now expire, and each consecutive failure for the same
host doubles the wait, from a minute up to a ceiling of fifteen. That is
what bounds the cost: a host that is simply gone is asked about a
handful of times across a long crawl rather than once a minute. Positive
answers are unchanged and last the crawl.
The stamp is wall-clock, since the tree has no monotonic clock, so a
clock moving behind the moment a record was stored, after an NTP
correction or a suspend and resume, would delay every pending retry by
the size of the step. At the ceiling that leaves a merely unreachable
host dead for the rest of the crawl, which is the bug this branch
removes. Introducing a monotonic clock across the tree's platforms for
one cache is not worth it, so the store time sits beside the expiry and
a clock behind it counts as expired. Retrying early is the safe
direction.
Where the resolver distinguishes them, `EAI_NONAME` is an answer about
the name itself and still stands for the whole crawl, so the common
dead-link case behaves exactly as it does today. Every other code says
the resolver could not answer, which is what an outage produces for any
host. Android maps all of them, NXDOMAIN included, onto `EAI_NODATA`,
measured on API 25 and 36 with an online control; there the class buys
nothing and the doubling is what protects the crawl. Compare symbols,
not numbers: Bionic and glibc order these differently.
The self-test asserted the old behaviour, one backend call for a host
that does not resolve. It now pins the expiry, the failure count that
lengthens it, the ceiling, that a positive answer does not expire, that
`EAI_NONAME` still does not, and that a backward clock step re-asks. Its
outage host had to change from `EAI_NONAME` to `EAI_AGAIN`, the code
that case actually returns. Nothing sleeps for a wait any more: a
test-only hook moves a cached record's stamps, so the doubling is
table-tested and the "still inside the wait" check no longer races a
stalled runner.
`t_dnscache` grows three fields at its tail. `htslib.h` is not an
installed header, so this is internal.
Found by the httrack-android robustness audit, where this compounds with
the update purge: a host that dies during an outage decides which files
the purge then removes.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 13339734863975d4cae9194a5043242eee05978e
Author: Xavier Roche <roche@httrack.com>
Date: Sun Aug 23 20:52:51 2026 +0200
A vanished suite worker is reported as a clean run (#1371)
The Windows suite's fork-failure counter reads the console log for the
messages bash prints when MSYS fork emulation gives out, so a worker
that dies without bash saying anything is invisible to it: run
32484281897 ended on "no MSYS fork failure in suite-console.log" about a
leg that had just lost `86_local-proxytrack-cache-longfields.test`. The
issue's account of the mechanism holds, with one correction: the suite
step was not green. The tally already counted a missing `.rc` as a
failure, so the leg exited 1. What was indistinguishable was the class,
in the counter's verdict and in the `failing:` list.
The tally now has three outcomes instead of two. A failed test keeps
`FAIL` and its place in `failing:`; a worker that left no status gets a
`LOST` verdict saying how far it got (no log, a 0-byte log, or a log its
test had written), a `lost=` field in the tally line, and an `::error::`
naming the leg as one to re-run; a clean run says so with the lost count
included. The counter reads those `LOST` lines back and annotates them,
so its clean sentence is never printed bare again. A lost worker stays
fatal, since a leg that tested less than it reports must not ship green,
but exits 3 rather than 1 when nothing else failed: the status is all
the workflow keeps of the difference between a leg to repeat and a red
to investigate (#1228). The parent's wait status is deliberately not
used, because `wait -n` reaps workers the pool can no longer name; the
log the worker opened is what survives.
Classification moved into `ci_read_outcome` and `ci_lost_reason`, above
the driver's source guard, so `337_ci-lost-worker.test` can drive it off
the Windows runner: the four `.rc` states, the counter over a clean, a
fork-failure, a lost-worker and a mixed console, a mutant with the
`LOST` rule removed proving the old counter misread the same log as
clean, and a driver run over 115 stub tests where one stub kills its own
worker, the control leg exiting 0 against the killed leg's 3 with
`lost=1` and nothing in `failing:`.
Closes #1352
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit a513123d1afe308ffec91c2fcb53852258332ded
Author: Xavier Roche <roche@httrack.com>
Date: Sun Aug 23 20:31:36 2026 +0200
An update purge deletes the children of a page that failed to fetch (#1390)
On `--update` the engine unlinks what the previous mirror had and this
run did not write. A page that gives up after its last retry keeps its
copy (#746) but is never parsed, so its links never reach `new.lst`, and
the purge reads that gap as the site dropping them: one dropped
connection on a hub page costs its whole subtree, silently, while the
run reports success. An HTTP error already escapes this, being masked to
a 304 and recovered from the cache. A transport failure has no such
path.
The purge now holds off for the run when a link that could carry links
fails to transfer, keyed on `back_transfer_failed()` so it cannot drift
from the predicate that decides whether the copy is kept. A mangled
`Content-Encoding`, a bad chunk length, a disk write error and an
aborted external wrapper all report `STATUSCODE_INVALID`, keep their
copy, and now keep their children with it. Two cases deliberately do not
hold: an answered HTTP error, which is recovered and re-queues its
links, and a page the size cap skips, whose own copy #746 drops so its
children go with it rather than being stranded. The guard also requires
that the previous run actually mirrored the page, since only then is
there a subtree at risk. Typing the link by its savename instead let any
new permanently dead `.html` link disable the purge on every run.
Masking the failure as a 304 was the alternative and does not work this
far down the loop: that masking sits ahead of `back_finalize`, and doing
it at the give-up point would disarm the "no data transferred, restore
the previous session" rollback, so a mirror whose host had vanished
would report success.
`httrackp` gains one field at its tail, live state `copy_htsopt` leaves
alone. It lands in the padding after `wizard_filters` on LP64, so
`sizeof` is unchanged there; on ILP32 the struct has no trailing padding
and grows by four bytes, as every earlier tail field did.
`size_httrackp` is asserted against `sizeof` at each entry point, so a
stale-header consumer aborts rather than corrupts.
Known gap, not fixed here: a failed fetch writes no cache entry, so a
hub failing on two consecutive runs has no previous entry the second
time and stops holding the purge.
Test 348 drives five shapes past the guard and reads `hts-changes.json`
rather than a log line, the two boundaries above among them. Found by
the httrack-android session's overnight robustness audit, reproduced
against 3.49.23.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 8c4c2d8de205b65a20ddedd28bff8563bdd6f1e6
Author: Xavier Roche <roche@httrack.com>
Date: Sun Aug 23 19:35:35 2026 +0200
A gcc UBSan finding dies with the test that discarded stderr (#1374)
#1357 gave the sanitizer legs two places to look for a report a passing
test hid: the ASan/MSan log_path directory and the per-test logs. gcc's
UBSan lands in neither. It ignores log_path and writes to stderr, and
most of the suite sends the engine's stderr to /dev/null or into a
tmpdir it later deletes. An intra-object overflow is the ordinary case:
ASan cannot see it, UBSan reports it as an out-of-bounds index or a
store with insufficient space, and nobody ever read the report.
The harness now runs the engine through a PATH shim that tees its stderr
into a directory the two sanitizer jobs name, and
`tools/ci-sanitizer-report.sh` scans it afterwards with the signature
list it already carries. The shim execs the program, so the pid, the
process group and the exit status the tests kill and read stay the
engine's, and it tees rather than redirects, so a test still gets every
byte on the stderr it chose, /dev/null included. Without a capture
directory it is a bare exec and PATH is left alone, so a plain `make
check` is unchanged.
Teeing at the redirect sites instead would have covered only what exists
today, and reopened with the next test that writes `2>/dev/null`.
`UBSAN_OPTIONS=log_path` is no help either: it works for a standalone
UBSan build but is ignored in the asan+ubsan runtime this leg builds
(measured on gcc 14.2). Two tests probed for a sanitized build with
`ASAN_OPTIONS=help=1` alone, which drops the job's `detect_leaks=0`;
their probe run left a leak report in the stderr nobody read and would
have failed the leg now, so they keep leak detection off.
Checked with an intra-object overflow injected into a self-test error
path that `01_engine-charset.test` reaches with stderr discarded and the
status ignored. On master the test is green, master's collector reports
nothing, and ASan's log_path directory stays empty. With the change the
test is still green and the collector fails the leg, naming the test
whose engine wrote the report. A full sanitized suite on the unmutated
tree stays clean.
### Open, and the reason this should not merge as it stands
Review found the caller's copy of stderr is written by a `tee` nobody
waits for: the shim `exec`s away, so when a test reads a file it
redirected stderr into, that file can still be short or empty. Measured
over 60 runs each, 50 lines expected: capture on, 6 runs short, four of
them with nothing at all; capture off, none. It is specific to the
redirection form. `cmd | …` and `out=$(cmd 2>&1)` are safe, because
`tee` inherits fd 1 and the reader blocks on it. `cmd 2>FILE` and `cmd
>FILE 2>&1` are racy, and the second is `tests/local-crawl.sh:307`, i.e.
every crawl test. With capture on that form is also fully reordered: all
stdout, then all stderr, where capture off interleaves.
Nothing depends on it today, and the full sanitized suite is identical
with capture on and off, same tally and same skip list. But a test
asserting stderr is empty would pass vacuously, and one asserting
content would flake, on the sanitize leg alone. That is the harness
lying, which is the failure this change exists to end.
The fix is a real choice rather than an oversight. Waiting for `tee`
means dropping `exec`, which changes the pid the shim presents, and
tests kill the engine by pid and by process group. Keeping `exec` means
finding another way to make the caller's copy synchronous. I would
rather that decision were made deliberately than picked at four in the
morning, so it is written down here instead.
Test 342 cannot currently see it: its one capture-on passthrough case
uses `out=$(…)`, the form that serialises. A `2>FILE`-then-read case
would expose it, and is worth adding with whatever fix is chosen.
One correction to the prose above: the `114`/`283` leak regression is
created by this change, not pre-existing. On master the probe captures
into a shell variable, so the report never reaches a log and master's
collector could never have seen it.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit b8d35b7c0bf88a46e7f999c9bccda5f80e1aee28
Author: Xavier Roche <roche@httrack.com>
Date: Sun Aug 23 18:40:50 2026 +0200
A bodyless 308 drew a bogus big-file warning, and five checks that decide nothing (#1382)
The mirror loop hand-listed the statuses whose empty body is expected
(301, 302, 303, 307, 412, 416) and never grew 308 when HTTP_IS_REDIRECT
did, so a 308 carrying a Content-Length over --max-file logged "Big file
cancelled according to user's preferences" for what was only a redirect
to follow. The list moves into hts_body_missing_unexpectedly() where it
tracks the macro, and the same call site loses a copy of istoobig()
whose operands have been byte-identical to the first since 3.20.2.
The rest decides nothing. Both IDNA emitters test their byte against -1,
but EMIT_UNICODE only ever hands them an unsigned char. url_savename()'s
%h case branched on short_ver into two identical arms, and htswizard's
second "interdiction de monter" block recomputed the two lienrelatif()
calls of the block above it, discarded both results and only duplicated
its error log.
-#C compared r.contenttype, an array, against 0 where every neighbouring
header tests [0]. That one ships untested: the lister enumerates
hts-cache/new.ndx, which the engine stopped writing in 3.31, so it finds
nothing in a modern cache, and with a hand-made ndx it segfaults in
url_savename() on a NULL sback before reaching the header.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 6fe532b97e3e9b4380723b53206af21c558cb422
Author: Xavier Roche <roche@httrack.com>
Date: Sun Aug 23 15:55:25 2026 +0200
59 hand-rolled log dumps, and the six fail() overrides that could not go without one (#1384)
* tests: one fail_dump helper for the 59 hand-rolled log dumps
The suite had 59 copies of `cmd || { echo MSG; cat LOG; exit 1; }` across 22
files, which round 1's fail() could not absorb because it cannot dump a log.
testlib.sh gains fail_dump (message plus the files the failure is about) and
fail_dump_var, which registers a log by variable name so fail() itself dumps
it -- that is what lets the six tests that shadowed fail() drop the override
and stop diverting every assert_*, require_httrack and run_with_timeout call
through a private copy.
webhttracklib.sh gains htsserver_cleanup_dir and htsserver_reap_vars, folding
34 byte-identical work-dir teardowns into one cleanup_push line apiece.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
* tests: cover fail_dump and fail_dump_var in 257
Both run only on a failure path, so a green suite exercises neither. Six
mutants killed: a plain cat for the indenting sed, a header over an empty
file, a fail_dump that stops after the first file, a dump on stdout, a
fail_dump_var recording the value instead of the name, and fail() dumping
nothing at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 76e96d0fc6a06ccea817b0584e65bf13811f953e
Author: Xavier Roche <roche@httrack.com>
Date: Sun Aug 23 15:53:48 2026 +0200
The server decodes a malformed percent-escape to a NUL (#1376)
* Reject a malformed percent-escape instead of decoding it to a NUL
ehexh() exists three times. The engine's copy returns -1 for a character
that is not a hex digit, and its caller leaves the escape literal. The two
copies behind the WebHTTrack server return 0, so "%ZZ" decodes to a NUL
byte that truncates the value wherever it is next read as a C string. That
reaches posted form values, the User-Agent field and the winprofile.ini
reader.
The copies exist because libhttrack builds -fvisibility=hidden and these
helpers are not HTSEXT_API, so htsserver cannot link them even though
htslib.h declares them. Move the decoders to src/htsescape.{c,h} and
compile that into libhttrack and htsserver the way htsurlport.c and
htscmdline.c already are, so the fix has one place to live. A fourth copy
in proxy/proxystrings.h had no callers at all and is deleted.
The '+'-to-space rule the query parser depends on is kept; only the hex
validation changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Fold the review findings: one hex decoder, one separator set, MSVC wiring
The Windows legs caught the real gap: the Visual Studio projects carry their
own hand-written source lists, so htsescape.c linked everywhere except MSVC.
Add it to libhttrack.vcxproj and webhttrack.vcxproj, and add a test that
compares each automake _SOURCES against its .vcxproj so the next new file
fails locally instead of on a Windows runner.
Review also found the dedup was incomplete. htsencoding.c had get_hex_value(),
byte-identical to the new helper and invisible to the regression guard, so the
hex helpers now live in htsencoding.h, which already owned the concept, and
get_hex_value is gone. The private hts_is_retorsep() was a fourth spelling of a
set htslib.h already defines, and its justification named a build that does not
exist; use is_retorsep() and drop the copy in htsserver.h, which the deleted
unescapeini() had been its only consumer.
Test gaps found by auditing the table against the spec: every malformed row
used an upper-case non-hex digit, so widening the 'a'-'f' arm to 'z' passed all
of them while %zz still decoded to a byte. Add the lower-case row, %00 (which
is well-formed and does decode to NUL), an empty input, a separator run longer
than two, and an assertion on the terminator, which callers rely on because
they read the result with strcmp().
Both new files carried the old GPLv2 header; use the current SPDX GPL-3.0
template the other recently added files use.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Keep htsserver.h's is_quote byte-identical to htslib.h's
Deleting the neighbouring is_retorsep brought is_quote into git-clang-format's
range, and the reflowed spelling is no longer token-identical to htslib.h:528.
Both headers reach htsserver.c and htsweb.c, so C11 6.10.3p2 stops treating the
pair as a benign redefinition and warns twice per build.
The four remaining macros stay: htsserver.h uses is_realspace() itself, and
htsweb.c never includes htslib.h, so deleting them there does not compile.
Collapsing that set belongs to a change that gives htsserver.h a real home for
them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Leave htsserver.h's macro block exactly as it was
The previous commit did not do what it said. Deleting the dead is_retorsep put
the neighbouring lines inside git-clang-format's range, and the formatter
reflowed is_quote on the way past; re-spelling is_quote by hand did not help,
because the line stayed in range and was reflowed again on the next format pass.
Either way htsserver.h and htslib.h stop being token-identical and C11 6.10.3p2
warns twice per build.
Restore the block, blank line included, so nothing in it is a changed line. The
dead macro is worth less than a clean build, and it goes away for real when the
linput/linput_trim copies leave this header and take the block's last consumer
with them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Keep htsescape.c clear of the engine headers
MSVC failed the x64 and Win32 legs with a sockaddr redefinition in ws2def.h:
htsencoding.h includes windows.h, which brings in winsock.h, so the htslib.h
added for is_retorsep() reached winsock2.h second. POSIX builds never see it.
Go back to a local CR/LF/TAB predicate, this time saying why it is not simply
using the engine's macro.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Do not fail the MSVC leg for having no automake environment
The Windows legs were red because of these tests, not the diff they cover.
The MSVC job runs the scripts directly, with no automake around them, so
abs_top_srcdir is unset and ${abs_top_srcdir:?} killed the run; 205, 269 and
their neighbours already handle this by stepping aside.
01_engine-unescape-form still drives the self-test there, and only its
source-tree check needs the variable, so the decoder stays covered on the
platform whose header order broke it. 342 needs the tree for both halves and
skips.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 8c312ca9ff3a84b784dcb7140ab8188e2abf14b8
Author: Xavier Roche <roche@httrack.com>
Date: Sun Aug 23 15:53:16 2026 +0200
Let the WebHTTrack cache boxes be turned off (#1385)
Neither the "cache" box on option3.html nor "cache2" on option9.html had the
hidden companion the other option pages carry, so an unticked box posted
nothing and the stored "1" survived: the cache could not be switched off from
the UI, and --store-all-in-cache could never be taken back off once set.
The seed "Cache" in htsserver.c's initOn[] is the second half. step2.html
copies that profile key onto the field, so the first step2 render after a user
cleared the box put the cache straight back on. The field itself is seeded
right beside it, so the key's seed only ever overwrote a user's choice; a
loaded winprofile.ini still supplies its own value and still decides.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit eb8aec7c6ea9e935e741868ececf247d66bb3f31
Author: Xavier Roche <roche@httrack.com>
Date: Sun Aug 23 15:53:04 2026 +0200
Two file-list options were classed as taking no value (#1379)
* Three long options were classed as taking no distinct argument
--structure took its template as a URL. The alias table holds two rows for the
name, the first classing -N as "param", which glues the value onto the short
form; -N%h%p/%n%q.%t is then not an option the engine consumes, so it landed in
the URL list. Only --user-structure worked, and nothing documents it. The
second row already had the right class and was unreachable, because the first
row wins.
--filelist=FILE and --filterlist=FILE were refused outright ("does not take the
value"), while --list=FILE and --urllist=FILE worked. Each pair shares a short
option, so the classes have to match; -%L and -%S both take a file.
Reverse lookup by short option reads the same table, so the rows have to keep
their order: deleting the first "structure" row instead of fixing its class
renamed -N to --user-structure in the generated man page. Regenerated, and the
table now says which row wins.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Regenerate html/httrack.man.html for the option change
The man page and its HTML rendering are checked against each other, and only
the man page was regenerated. Same three lines, no other drift.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Assert the structure took effect, not that argv was rewritten
Review found the oracle was wrong: doit.log is the invocation rebuilt from
argv, so grepping it for "-N <template>" proves the alias layer split the token
and nothing about whether -N consumed it. Assert on the file the template
names instead, and check the exit status, which is what catches the glued form
(it exits 255 on a template with no slash).
The synonym check now fails an empty log rather than reading it as acceptance,
and the duplicate-name check says what it does not catch: two rows sharing a
class but naming different short options, which --test does today.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Revert the --structure reclass: it turned the presets into templates
Review measured the expansion end to end over 1008 runs and caught a regression
this PR introduced. --structure takes numeric presets, and htscoremain.c:1270
reads a DETACHED -N value as a user template whatever it holds, so classing -N
as param1 made --structure=1 mean a file literally named "1". Every page
collapsed onto one colliding name, at exit 0.
My own earlier check missed it because it asserted on hts-cache/doit.log, which
is the invocation rebuilt from argv: it proved the alias split the token and
nothing about what -N did with it. The test now asserts the preset layout, and
fails on the reclass.
Keep the parts that measured clean: one row for structure instead of two,
--filelist and --filterlist reclassed to match the synonyms they share a short
option with. --structure's own problem needs a CLI decision and is #1380.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Assert the layout each preset produces, not the absence of a file
Re-review broke the preset check with a second mutant: drop the parsed value
(savename_type = 0) and --structure=5 exits 0, writes no file named "5", falls
back to the default layout, and the test passed. Absence was never the right
oracle. Assert the layout instead: -N1 puts the page at web/, -N5 at web/html/,
-N0 under the host directory, and a dropped value cannot imitate any of them.
The file-list check had the same shape: --filterlist=/nonexistent exits 255 with
a non-empty log and no syntax error, so "not refused" passed on a dead run.
Check the exit status and name a real URL in the list.
Drop the description added to the structure row: the param class accepts any
value, so it never prints, and it was wrong anyway (--help documents N1001-N1099
as well).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 2fe02bd399a3902423048ff1799420e0388a2888
Author: Xavier Roche <roche@httrack.com>
Date: Sun Aug 23 15:52:30 2026 +0200
A bracketed IPv6 authority loses its port (#1383)
jump_toport_const() looked for the authority's closing ']' with a limit taken
from source rather than from the post-scheme pointer, so the scheme's own "//"
ended the scan before the bracket was ever reached. The colon it then returned
was one from inside the literal. Through a CONNECT proxy that made httrack emit
"CONNECT [::1]" with no port at all, since the caller reads a non-NULL result as
"the host already carries its port".
The FTP path split the authority by hand on the first ':', which for "[::1]:21"
lands inside the literal too and refused every IPv6 URL with "Invalid port".
That walk now lives in ftp_jump_authority() and ftp_split_hostport(), which
reuse jump_protocol() and jump_toport_const(); the port echoed in the error
message is wire data, so it is clipped into the buffer rather than aborting.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 38540fd5c8f57ce13915703cbec0b48ee4c97e5c
Author: Xavier Roche <roche@httrack.com>
Date: Sun Aug 23 15:31:59 2026 +0200
Catalog values that lost a placeholder or broke an escape (#1375)
* Catalog values that lost a placeholder or broke an escape
Two checks in 62_lang-integrity.test, and the sixteen strings they find.
A translation must carry the same %-tokens as its msgid, printf conversions
and HTTrack's own build-string letters alike, and every backslash escape must
be one conv_printf() knows. Nothing asserted either, so both had drifted.
Norsk:906 is the one that matters: \n typed as \m gave the local-structure
dropdown 14 rows instead of 15, so every entry past it was off by one and the
user got a different disk layout than the row named.
The escape scan reads decoded text; 0x5C is a legal trail byte in BIG5 and
Shift-JIS, and scanning raw bytes reports false hits in Japanese and Chinese.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
* Review fixes: minimal Italian edit, controls that pin what they claim
Italiano:766 was rewritten wholesale from the audit's suggested text, which
also reworded five rows that needed nothing, dropping the only mention of the
DOS short-name convention. Rebuilt from master with the three missing token
rows inserted and nothing else touched.
The five controls all passed a scanner that tokenised one character after %
instead of two and accepted every escape letter the tree actually got wrong.
They only pinned how many % tokens a value had, never which, and used \T,
a letter no real defect uses. Added a %s -> %d control, moved the lost-n
fixture to \m, made each control assert the letter reported, and gave the
escape loop the file-count floor the placeholder loop already had.
Macedonian:656 also regains an opening "(" its msgid has and master lost.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
* Compare the placeholders in order, and reattach the Polish key that moved
A doubled review agent made the case that order is not a style choice here:
these are plain sprintf() formats with no positional specifiers, so a %s that
trades places with a %d binds to the other's argument, which is a crash, not
wrong text. The multiset comparison was rationalising that away as grammar.
Comparing the sequence instead drops the sort helper and flags exactly one
line in the tree, Polski:766, where "%s?" had been hoisted six rows above the
row it labels, leaving that row keyless and the "%s?" row without a
description. Moved the key back rather than taking the audit's full rewrite.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
* Romanian was decoded as the wrong charset, and two more checks
Romanian declared ISO-8859-1 and its bytes are ISO-8859-2, so every a-breve,
s-cedilla and t-cedilla in it rendered as a different letter: LANGUAGE_NAME
alone read "Romanan" where it should read "Romana". htsserver copies the field
into the page <meta> and decodes POST bodies with it, and WinHTTrack reads it
for its codepage, so both front ends were showing mojibake. The declaration is
the only possible fix; the letters do not exist in Latin-1. Its value for the
msgid OK was the literal string LANGUAGE_WINDOWSID, which is what every OK
button in that catalog said.
Line-break parity, pinned per catalog. Each \n in a combo-box string starts an
item, so a value carrying a different number of breaks than its msgid shifts
every row after the first one that moved. Most of the drift is prose reflowed
to a different width, hence the pin; three were real. Macedonian:908 had its
\n typed as Cyrillic "жн", Portugues:902 carried a trailing \r\n that added a
twelfth, empty filter type, and Portugues:916 used \r where \n belongs, which
is a valid escape and so invisible to the escape check.
Literal tokens, zero tolerance. hts-cache, robots.txt and cgi-bin are named on
disk and on the wire; a translated one sends the user to a path that does not
exist. Polski:208 told them to erase "htscache/new.*", Cesky:916 and
Nederlands:562 followed "robot.txt", and Romanian:88 gave "/cgi/bin/". Example
hostnames are deliberately not on the list: localising www.someweb.com is the
translator's call.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
* The structure dropdown named folders the engine never creates
HTTrack builds web/, web/html and web/images on disk, and the dropdown that
offers those layouts translated them: Hungarian promised web/kepek/, Turkish
web/resimler/, Brazilian Portuguese web/imagens/, Bulgarian and Ukrainian
Cyrillic, and Russian and Ukrainian dropped web/ from five rows outright.
Whichever row the user picked, the files landed somewhere else. 25 catalogs
fixed, paths made literal and the prose around them left translated.
The literal-token check counts occurrences rather than looking for one, which
is what found this: a value keeping one web/ of eleven passed a presence test
while ten rows still pointed at nothing. Counting also turned up Magyar:916
following "robot txt" and Slovak:916 writing the ditto mark -//- where the
combo box shows it literally.
The escape check was blind to Japanese. Shift-JIS puts the yen sign at 0x5C,
so decoding turned all 101 of its escapes into a character the scan does not
recognise and the catalog passed without being read. It is scanned now, and a
planted \m in it fails as it should.
site_name and www.domain.xxx are placeholders rather than real directories;
they are kept literal so every catalog reads the same, not for correctness.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
* Ship the line-break pin in EXTRA_DIST
Its sibling 62_lang-untranslated.counts is listed; without this one a dist
tarball drops the file the test reads and the out-of-tree build goes red.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
* Cut the comment blocks the review flagged as over-long
Four blocks in the test and the pin file's header, each carrying more lines
than its point needs. No behaviour change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
* Controls that were passing broken scanners, and a test for the charset fix
The review built three checkers that were wrong and still went green.
A literal-token counter reduced to presence passed, because the fixture only
ever exercised one occurrence against zero; it now names a token three times
and drops one. An escape scanner that stopped after the first pair passed,
because both fixtures put their defect on line 2; the scan now counts what it
read and compares that against the catalog size, which is what already made
the placeholder check hard to fool. And nothing at all covered the Romanian
charset fix: a single-byte charset decodes all 256 values, so no existing
check could see a wrong declaration. Reverting it left the suite green.
That last one is now caught by naming letters a language does not use.
Romanian has no a-tilde, so reading it as Latin-1 lights up on the second
line. The table holds only pairs that have gone wrong; a language absent from
it is simply unchecked, which is worth knowing before trusting a pass.
Also: the literal check accepts a translation naming a path more often than
the msgid, since a gloss is not a defect, and it no longer covers site_name or
www.domain.xxx, which are placeholders rather than paths and were only ever
made literal for consistency. lang/README.md now states the rules the checks
enforce, so a translator meets them before CI does.
Turkish:906 row 5 also gets a prose fix rather than a path one: it read
"images in web/" where English says "Html in web/", naming the opposite
layout.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
* Say which literals the check enforces, not which the rule covers
A bare web/ cannot be checked: msgid 98's example URL contains it inside
'fromsomeweb/', so five catalogs light up as false positives. The rule still
covers it, the test only reaches the compound forms.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
* Controls that pinned one instance where the check covers a set
A second adversarial pass found the same shape twice more. Widening the escape
check's valid set by any letter other than the two the controls happened to
use passed green, hiding every escape defect in the tree; so did narrowing the
literal token list to the one token its fixture named.
The escape controls now name each of the five letters the catalogs actually
got wrong. The token list is pinned outright rather than sampled, because a
fixture can only ever exercise the tokens it happens to mention, and dropping
any other one would go unnoticed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
* Five of the six charset rows could never have fired
The table listed letters those languages do not use, which is true and beside
the point: the letters have to be what the mistake produces. Reading a Latin-2
catalog as Latin-1 turns Czech r-caron into o-slash and Hungarian o-double-acute
into o-tilde, never into the a-tilde and eszett the table named. Only Romanian
worked, and Polish only because line 906 is Portuguese pasted in with Latin-1
bytes. So the check covered one language while reading as though it covered
seven.
The rows now carry what a Latin-1 misread actually makes, measured rather than
reasoned, and a loop reads each catalog that way and fails any row that stays
silent. An inert row is the failure mode here, and it looks exactly like a
clean catalog.
Also back to exact counts for the literal tokens. A gloss was the argument for
allowing more than the msgid, but naming web/html twice where the msgid names
it once duplicates a dropdown row, so it promises a layout it does not build.
Neither direction has a finding in the tree today.
README now says which rules the test checks and which it leaves to the
translator, rather than claiming all of them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
* Pin the other direction of the literal count too
Nothing exercised a translation naming a token more often than its msgid, so a
one-sided comparison passed. The fixture now carries that case as well.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
* Close the three sampling gaps the last review left standing
None was a defect, all three were the same weakness: a control pinning what its
fixture happened to name.
The charset loop proved a row fires, so a row keeping one live letter of three
passed with the rest dead; it now proves each letter. The literal fixture damaged
three of five tokens, so dropping either of the others from the list and the pin
together passed; it now damages all five, which makes the pin belt-and-braces
rather than the only guard. And the escape valid set was pinned by the five
letters its probe used, so widening it by a sixth was invisible; it is now pinned
by string equality, the way the token list already was.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
* Read the letters as lines, in the current shell
The per-letter loop word-split a command substitution, which shellcheck flags
and which master does not do. A pipe would have been worse: the loop would run
in a subshell and its exit would not end the test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 5a1635255e979b5a2a059ce52ff0a86e77a46505
Author: Xavier Roche <roche@httrack.com>
Date: Sun Aug 23 11:56:01 2026 +0200
CI checked a build without the brotli and zstd decoders, a tarball missing three fuzz vectors, and a test that skipped itself (#1387)
* Close four gaps that let CI check less than it looked like it did
CodeQL installed neither libbrotli-dev nor libzstd-dev, and codec detection
defaults to auto, so the scanned build had the br and zstd wire decoders
compiled out. Add the packages and the config.h assertion ci.yml already
carries, so the gap cannot reopen quietly.
The fuzz corpus and the shell-script lint list were both kept by hand and had
already drifted: three vectors never reached the tarball, and
fuzz/run-fuzzers.sh sat outside shellcheck and shfmt. Derive each from the
tree instead, the corpus at configure time (automake expands no EXTRA_DIST
glob) and the script list from git ls-files, the way the neighbouring Python
step does. Both refuse to run on an empty list.
tools/doc-chrome.py was missing from EXTRA_DIST, so 271_doc-chrome-year
skipped itself inside distcheck rather than failing. Ship the file, and treat
a missing input as fatal there like 324 does. That skip was hiding a second
one: distcheck hands the test a read-only srcdir and cp -a carries the mode
over, so the copy it edits was unwritable. Make it writable after the copy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Give the two derived lists an oracle, not just a floor
Review found the new lists guarded by `-gt 0`, which is the same class of hole
this PR is about: typo one arm of the shebang scan and it yields 3 files, not
400, and the lint stays green while checking almost nothing. Measured, not
supposed.
The shell-script list now floors on the half that cannot shrink, the plain
extension glob, and names the three extension-less scripts the scan has to
reach. The corpus glob gets what tests/check-test-names.sh already gives the
test glob: a comparison against the tracked set in both directions, plus the
whitespace and wrong-depth cases that would drop a vector from the tarball
without failing anything. Each of those five failures is mutant-checked.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 3949926ea2edde7d147123eaed7cf14c1c7c0061
Author: Xavier Roche <roche@httrack.com>
Date: Sun Aug 23 11:51:09 2026 +0200
A cache size the reader cannot hold reaches malloc() (#1389)
* A cache size the reader cannot hold reaches malloc()
The engine writes X-Size as a signed 64-bit value; ProxyTrack read it with
sscanf("%d") into an int and assigned that to a size_t, with none of the
validation htscache.c applies. X-Size: -1 therefore became SIZE_MAX, malloc()
of SIZE_MAX + 1 returned a one-byte buffer, and the entry's body was inflated
into it. A legitimate headers-only entry above INT_MAX (the engine stores those
for bodies over 2GB) wrapped to INT_MIN and was dropped.
Decode into a signed 64-bit local, judge it before it becomes a size_t, and
drop the writer's (int) cast: ZIP_FIELD_INT was already 64-bit, so the cast was
the only truncation left. Valid caches decode and re-encode byte-identically.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Pin the two boundaries the engine's selftest already pins
The review built a guard that passed every case in tests/347 and still broke
the spec three ways: `> INT_MAX` where the engine uses `>=`, the negative
check scoped under X-In-Cache, and the width arm dropped. Two cases from
htscache_selftest.c were never ported over: an in-cache X-Size of exactly
INT_MAX, and a headers-only negative one. Both are here now, and each kills
one of those mutants.
The comment claimed to mirror htscache.c. On a 32-bit build it deliberately
does not: the engine holds the size in an int64 and keeps a headers-only
entry above 4GB, where PT_Element.size is a size_t that cannot represent it,
so the proxy refuses rather than truncating to a body it would then report as
complete. Say that instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 9de73a4472b000ca9069f49b591054104f40ab37
Author: Xavier Roche <roche@httrack.com>
Date: Sun Aug 23 11:20:14 2026 +0200
Nothing records why the soname survived an ABI break (#1368)
* Say why the soname stayed at .so.3 across an ABI break
#1365 widened t_cookie.max_len from int to size_t in htsbauth.h, which is
an installed header, so data[] moved from offset 4 to 8 on LP64. The rules
in this very comment block call for current++ and the Debian package
rename; neither happened, by decision.
Nothing in the tree said so. A release audit reconstructed it as a missed
bump, and the next reader would have too. The rationale block already
carries a line per VERSION_INFO value, so the deviation belongs there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Amend the 3:16:0 entry rather than stacking a second one
Review caught two things. VERSION_INFO was set to 3:16:0 by the 3.49.23
release and #1365 landed after it, so the existing entry's "does not move
a layout" is false for that same value; a second entry beside it left the
block self-contradicting. And "the rules above" pointed at a per-version
log that contains no rules, which live in CLAUDE.local.md.
It also measured what I had taken on trust: data[] moves 4 to 8, while
auth and sizeof(t_cookie) are unchanged, because the old padding hole
absorbs the growth. That makes the break narrower than the first wording
implied, and the entry now says so.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 5d71f8ba5e3161908e3c0ca15facb0deee35bd24
Author: Xavier Roche <roche@httrack.com>
Date: Sun Aug 23 11:13:37 2026 +0200
Remove code nothing calls, and name the unused-code warnings (#1388)
Fifteen non-static functions, three unused cache_back fields with the
struct behind them, two ProxyTrack cache structs never instantiated, four
header macros, a dead else-branch in back_clean(), two write-only loop
counters and a -DNO_MALLOCT nothing tests. None is exported: the shared
library keeps the same 171 symbols.
-Wunused-function, -Wunused-variable and -Wunused-but-set-variable are
implied by -Wall today; naming them keeps a future -Wall from quietly
dropping the check that pins this.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 379fa25fa37e2820f2b0940177123d8f5e49f489
Author: Xavier Roche <roche@httrack.com>
Date: Sun Aug 23 10:54:23 2026 +0200
Files kept from disk instead of fetched are never mentioned (#1378)
Four places in `back_wait()` break the connection and keep the file already on disk rather than fetching it. All four logged at `LOG_DEBUG`, which a default run never prints, so a mirror can be assembled largely out of local files and say nothing about it.
That silence has a cost. Investigating #113 measured a run with `--delayed-type-check=0 --sizehack` keeping three files that had been deliberately filled with garbage but were the right size, and reporting no errors. The acceptance rests on size and mtime, and a freshly copied asset has an mtime that satisfies the precondition trivially.
Each site now logs at `LOG_NOTICE`, which is the lowest level a default run shows, naming the file, its size, and which criterion matched.
What the engine accepts is deliberately unchanged. Whether size and mtime are sufficient evidence that a local file matches the remote is a product decision, and quietly tightening it here would change what existing mirrors do. `httrack-works/design/113-reuse-existing-assets.md` has the measurements and ranks the options.
Refs #113.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit f9ab38076026706d485575d29a373b5da869e509
Author: Xavier Roche <roche@httrack.com>
Date: Sun Aug 23 10:54:19 2026 +0200
A script served as text/javascript is never scanned for links (#1377)
An external `.js` file was link-scanned only when the server spelled its Content-Type `application/x-javascript`. `text/javascript` is what IANA recommends, what most servers send, and what httrack's own mime table emits for `.js` and `.mjs`, so the common case was fetched, saved, and never parsed: every URL inside it stayed out of the mirror. Against the local test server on identical script bytes, `application/x-javascript` captured all 7 links; `text/javascript`, `application/javascript`, a `text/javascript; charset=utf-8` header and a `.mjs` module captured none.
The four spellings now live in one `is_javascript_mime_type()` in `htslib.h`, read by `is_hypertext_mime__()`, both `htsparse.c` dispatch sites, `htsindex.c` and the four-way list in `htssinglefile.c`, so the copies cannot drift apart again.
The risk runs the other way: more files get link-scanned now, and the JS scan does pull link-shaped string literals out of ordinary code. I measured that too. A script full of paths with real extensions produces the same four spurious fetches under `application/x-javascript` as under `text/javascript`, so this is the existing scan reaching more files, not new behavior. A link-free script comes back byte-identical: `345_local-js-content-type.test` mirrors one under `text/javascript`, compares it with `cmp` against the same bytes served under a type nothing scans, and pins an `application/json` decoy whose link-shaped string must stay unfetched. The test goes red on master at the first `text/javascript` target.
`36_local-bigcrawl` had a fixture named `decoy.js` asserting that a `text/javascript` script is never scanned. It is now `modern.js`, and the test asserts its target is present.
Surfaced by the JS/SPA work in #302; this is one increment of it, not the whole epic.
The legacy spelling's own count is unchanged: 7 of 7 before and after, on the same fixture. `htsindex.c` does gain behaviour, since a `text/javascript` file previously returned before the keyword loop and now enters it, which matches what the legacy spelling has always done.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit d58e16766549139612f66ef4b857108838b73830
Author: Xavier Roche <roche@httrack.com>
Date: Sun Aug 23 07:52:25 2026 +0200
Wait for the release rather than racing it for the DMG (#1367)
The tag push starts this workflow, but the release it should attach to is
cut by hand minutes later, so the attach step lost the race in 2 of the
last 4 releases. Losing it leaves the signed DMG on the run's artifacts,
where it needs an authenticated request and expires, and 3.49.16 shipped
that way.
It now waits up to 30 minutes for the release to appear instead of testing
once. A tag with no release ever coming costs that wait and warns; the
step still cannot fail the build, and a labelled beta build is still never
attached.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 7619f9aaa1af26d2316f9738ff79e67f9e79d4d9
Author: Xavier Roche <roche@httrack.com>
Date: Sun Aug 23 07:23:51 2026 +0200
Two file sizes still reach an allocator at the wrong width (#1373)
The audit behind #1354 named six C4244 sites and one C4477 as the real defects. All seven went away in #702, which merged the same day the audit was written, so the issue's premise is stale. The Windows build of current master (100786cc, run 32594588602) emits 18 distinct sites, not 74, and none of the seven is among them.
Re-deriving those 18 from the run log leaves two whose value is not provably in range. Both are leftovers of the class #702 declared a bug and fixed everywhere else: an `LLint` file size reaching an allocator without `llint_to_size_t()`. `index_finish()` sizes its keyword-index buffer from `fpsize()`, and htsselftest's hashtable reader sizes its pattern buffer from `fsize()`. On a 32-bit build a file past 4GB wraps the `malloct()` short. The hashtable loop then keeps its 64-bit bound and walks off the end of the buffer; the index just builds from a short read. Both now narrow once, through the helper `htscache.c`'s `readfile2()` already uses, and that width carries into the read and the walk instead of a cast at each use.
Neither has behaviour CI can watch. `llint_to_size_t()` is the identity whenever `size_t` is 64 bits, so on LP64 the change is a no-op by construction, and the trigger wants a >4GB file on a 32-bit target. The helper itself can be pinned, and had no direct test, so the `growsize` self-test gains a table for it. Its negative cases fail here under a mutant that refuses negatives; the over-4GB case only bites on the `linux i386` leg.
The other 16 sites stay as they are. They are clamped, guarded, or vendored, and casting them quiet is what the issue asks not to do. This does not close #1354: the counting and the ratchet are still open.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 702e6c1c5d9edc85d7fdbbaf2623fe5aff5cb70b
Author: Xavier Roche <roche@httrack.com>
Date: Sun Aug 23 07:23:46 2026 +0200
htsserver answers any Host, and advertises URLs a browser cannot open (#1372)
htsserver's control panel is unauthenticated: whatever opens its socket gets a session id and can drive it, up to writing files at a posted path. It binds loopback, so nothing off the machine reaches it, but a page in a browser on the machine can present a name of its own for our address and have the browser treat the panel as same-origin (DNS rebinding). The server now refuses a Host it does not answer for: `localhost`, an address literal, or whatever `--bind` was given. Rebinding always presents a name, so only names need vetting and an address literal is simply whoever opened the socket, which leaves an operator who deliberately widened the bind working. The check runs ahead of the dispatcher rather than inside a handler, so a GET is covered too, and it is the GET that hands out the session id; a guard on one handler is the shape of #1359. No Host at all still passes: HTTP/1.1 mandates one, so only a non-browser client omits it.
Two malformed advertised URLs ride along, both pre-existing. `--bind 0.0.0.0` printed `http://0.0.0.0:<port>/` and an IPv6 literal was printed unbracketed, neither of which a browser opens, and `webhttrack` hands that line's URL straight to the browser. A wildcard bind now advertises loopback, and a literal is bracketed.
Test 341 drives a real server both ways: every authority the panel is reachable at still completes a POST end to end, a foreign one is refused on GET and on POST, and the refused body is checked against the key store rather than against the suppressed reply, with a paired accept so a server that ignored every body could not pass. Its advert half fetches the URL the server printed. Both halves are red on master.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit ea41cb0bd64ab9864166f0132223e78a8a57529b
Author: Xavier Roche <roche@httrack.com>
Date: Sun Aug 23 07:23:42 2026 +0200
Windows-only paths in FAQ answers that are not Windows-specific (#1369)
The FAQ told everyone to look for their mirrors under `C:\My Web Sites` and gave a `file://C:\...` example, on two answers that Linux and macOS users read as well. Both now show the Windows form as one case rather than the only one, and a field comment in `htsmodules.h` that used the same path as its example goes with them. The remaining `C:\temp\` examples sit inside questions about Program Files and the Windows installer, where they are correct.
That is the whole safe half of #105. The rest needs a decision, not a patch.
The Windows default output directory is not set in this repo. On POSIX the engine builds it from the home directory (`htshelp.c`, `htsserver.c` and `webhttrack.in` all land on `<home>/websites`). `C:\My Web Sites` reaches WinHTTrack through `LANG_S30` in `lang.def`, and the code that acts on it lives in httrack-windows. Moving it under Documents, or renaming it so it says HTTrack, is a WinHTTrack change plus a migration story: `winprofile.ini` keeps whatever path the user last browsed to.
The example paths in `lang/*.txt` are not cosmetic either. Each catalog is a list of pairs whose odd line is the English string, and that string is the lookup key; `62_lang-integrity.test` rejects any msgid missing from `English.txt`. Rewriting `C:\\My Web Sites` or `Example:\t%h%p/%n%q.%t\n->\t\tc:\\mirror\\www.someweb.com\\someimages\\image.gif` in English orphans all 29 translations of it unless every catalog is edited in the same commit, each in its own legacy charset. The translated halves are what WinHTTrack displays, so changing those is a user-visible change to a value whose consumers we cannot grep.
Open for Xavier: should the Windows default move to something like `%USERPROFILE%\Documents\HTTrack Web Sites`, and if it does, what happens to profiles already pointing at `C:\My Web Sites`? And is a coordinated 30-file catalog edit worth it to neutralize those two example strings, given it rewrites the join keys and every translation of them?
Refs #105.
One fact needed to decide the first question, which I could not establish from this repo: whether WinHTTrack falls back to `LANG_S30` when `winprofile.ini` carries no path key, or leaves the field empty. That decides whether existing profiles migrate silently or have to be touched.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 100786ccd7e8a75d3626960d648b1922e0a767cc
Author: Xavier Roche <roche@httrack.com>
Date: Sat Aug 22 21:44:01 2026 +0200
Refuse a posted project path that climbs out of its directory (#1359)
The WebHTTrack panel composes a project path from two posted fields, `path` and `projname`, and a `..` in either one escaped the directory the user picked. The `strstr(fspath, "..")` check sitting there only decided whether the mirror got bound for serving; the `structcheck()` and the `fopen(..., "wb")` two lines below it ran regardless, so a POST with `projname=../escape` created `hts-cache/` and wrote a client-supplied `winprofile.ini` one level up. The load half carried no check at all: `loadprojname=../outside` read a `winprofile.ini` from outside the path and rendered its values back into the form. Both are reproducible against a running htsserver, and both CodeQL alerts (#247, #270) point at exactly these two `fopen` calls.
Both sinks now go through `hts_path_is_contained()`, which rejects a `..` path *component* rather than the substring, so a project named `ok..name` keeps working where the old check quietly refused to serve it. A refused save reports on the error page instead of half-executing. An absolute `path` remains the user's own choice and is untouched: naming your own mirror directory is what the form is for.
`330_webhttrack-path-traversal.test` drives a real htsserver, one POST per shape, and carries the `ok..name` control so a guard that refused everything could not pass. Every row is red on master and green here.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit c0d03e2c9d7ff7f80bc7b02f674dd5e30789274e
Author: Xavier Roche <roche@httrack.com>
Date: Sat Aug 22 21:22:58 2026 +0200
Refuse a fortify level the build cannot be shown to have (#1362)
The `_FORTIFY_SOURCE` probe only asked whether the define links, so a build whose `CFLAGS` carries no `-O` got a "yes" and no fortification at all. glibc gates the checked builtins behind `__OPTIMIZE__`, and 2.41 no longer warns about it. Measured on master with `CFLAGS=""`, `libhttrack.so` goes from 16 checked libc imports and 1124 checked call sites to zero, with nothing on stderr. The probe also assigns a string literal to a `char *` under a `-Werror` it adds itself, so a `CFLAGS` carrying `-Wwrite-strings` fails it, and since the `=2` fallback reuses the identical body that combination loses fortification outright rather than downgrading.
`HTS_ADD_FORTIFY_SOURCE` replaces the autoconf-archive macro. It requires `__OPTIMIZE__`, warns and adds nothing when the define would be inert, and accepts a level only once `nm` has found a `__*_chk` in the probe binary. Neither the tree nor CI has ever checked a mitigation against a real binary, so `331_fortify-source.test` counts the checked imports in the engine and the library, with a fortified and an unfortified control compiled from the build's own flags so the count cannot quietly stop meaning anything.
`debian/rules` carried `DEB_CFLAGS_MAINT_APPEND=-O3`, `CFLAGS += -DNOSTRDEBUG` and `CFLAGS += -g3` as plain make variables. None of them is exported, so the `$(shell dpkg-buildflags --export=configure)` below never saw them and no Debian build has used them. They are deleted rather than exported: `STRDEBUG` is read by no file in the tree, the architectures the workaround names are long gone from Debian, and switching `-O3` on today would change codegen everywhere on the strength of a 2004 comment. The `nostrip` block goes with them, since it appended to an `INSTALL_PROGRAM` nobody consumes and `dh_strip` already honours the option.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit a4023efbd0a925089319bde5796cf12ca0a94400
Author: Xavier Roche <roche@httrack.com>
Date: Sat Aug 22 21:13:52 2026 +0200
Three warning sites fixed at the type, one of them an installed struct (#1365)
* Give reconcile_put the type its callers already hold
Its callers pass TINY, MID and SOLID, which are LLint because they are file
sizes the same callers hand to reconcile_expect() and fsize(). The
parameter was size_t, so every one of those 29 call sites narrowed on
Win32, where size_t is 32 bits, and MSVC said so 29 times.
Taking LLint moves the single narrowing inside the loop, where the ternary
has already bounded it by sizeof(filler), instead of leaving it at every
caller. That is 29 of the 56 distinct warning sites on the Windows build.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Make the cookie capacity a size, and clamp a negative read remainder
htsbauth.h declared max_len an int while every check against it sums
strlen() results, so MSVC flagged both comparisons as signed/unsigned. The
field is a capacity; size_t is what it is. That drops the two C4018 sites
and the (int) casts three selftests and cookie_load carried to match it.
The checks themselves now measure each part against the room left rather
than summing first. No constructible input distinguishes the two forms,
since the sum only wraps past SIZE_MAX, so this is conformance to the
bounds rule rather than a fix for a reachable bug.
htslib.c computed totalsize - size into a size_t. A body longer than its
announced totalsize makes that negative, and as a size_t it would have
driven a read past the allocation; it is now computed signed and clamped.
Unreachable today, since each read is bounded by the same remainder.
Test 335 pins the capacity contract, which nothing asserted before: an
oversized value is refused, the store is untouched, and a canary past
max_len survives. Filled with 'C' rather than zero, so a stray NUL from an
off-by-one terminator cannot hide in the fill.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Pin the capacity boundary, and say why one check looks dead
Review showed the capacity test proved almost nothing: its inputs were 511
into 512 and one byte, so every off-by-one survived and deleting either
check individually survived too. It now derives the exact fitting length
from the fixture strings and the reserved headroom, and asserts both that
value and one byte past it. That kills deleting the first check and its
off-by-one.
The second check's mutants still survive, and the reason is worth a
comment rather than a stronger test: the first reserves 256 bytes while
the assembled record adds about 30 of literals, so the first passing
always implies the second. It is a backstop against that assembly growing,
not a live gate, and no input can distinguish it.
Also drops the (int) cast fuzz-header.c carried for the old field type,
which the previous commit missed in three other places.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit caba8121248b7ca7fddf997c42ce96fc82ef3116
Author: Xavier Roche <roche@httrack.com>
Date: Sat Aug 22 21:10:57 2026 +0200
Drop a CodeQL path filter that never applied (#1364)
The CodeQL config claims to exclude `libtest/` and `src/proxy/` from analysis, and never has. GitHub honours `paths-ignore` only for interpreted languages, or for a compiled language it analyses without building; this job runs `build-mode: manual` with a real `make`, and both trees are still compiled and extracted. Thirteen alerts in those paths are open right now, and the master SARIF lists all six files under `artifacts`.
`query-filters`, in the same `config:` block, does work: both rules excluded there sit at zero open alerts. So the config is parsed, and it is the path filter alone that is inert. This drops the dead block and leaves one line saying why, without excluding anything: whether ProxyTrack gets security analysis is a separate decision.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 9997235b31cc58010763de10d6317d129b49e5d1
Author: Xavier Roche <roche@httrack.com>
Date: Sat Aug 22 20:56:45 2026 +0200
Read and write exact blocks through one helper each (#1363)
63 call sites compared an `fread` or `fwrite` return against the count they asked for, most of them casting one side so the types would line up. `hts_fread_exact` and `hts_fwrite_exact`, header-only in the new `src/htsio.h`, do that once and hand back an `hts_boolean`. `-Wsign-compare` drops from 39 to 31, and the MSVC C4018 twins go with it.
Read-to-EOF loops, slurps and the minizip ioapi callbacks still need the count `fread` returns and are untouched. `proxy/store.c` keeps its comparison too: it assigns the short count to `r->size` inside the test, and folding that away would change what a truncated cache entry reports. One behaviour does change: `verif_backblue()` passed the byte count as `fwrite`'s `nmemb` and compared the result against the length, so it reported a write error on every successful run. Every caller ignores the return, which is why nobody noticed.
Nothing here is exported and no installed struct moves. `tests/332` drives `-#test=ioexact` over a short read, a short write, a zero-length transfer and a stream opened the wrong way; all four obvious mutations of the helper bodies abort it.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 0226a2eff61763ebe909fe83c5bbba5d78512d73
Author: Xavier Roche <roche@httrack.com>
Date: Sat Aug 22 20:54:40 2026 +0200
Let the server own its listen address, not its caller (#1366)
htsserver's control panel is unauthenticated: whatever can open its socket gets a session id and can drive it, so the listen address is a security decision the server itself should own. `smallserver_init` defaulted to the IPv4 wildcard and left the choice to its caller; only the literal in `htsweb.c`'s `main()` kept the shipped binary off the network. The default now lives in the server (a new `SOCaddr_initloopback` beside `SOCaddr_initany`, IPv4 so it binds with or without IPv6), and `--bind` remains the deliberate way to widen it, `0.0.0.0` included.
The advertised host follows the socket: a default run prints `http://127.0.0.1:<port>/`, which is what `webhttrack` greps out of the server's output before launching the browser. That path previously advertised the machine's hostname, which would have been unreachable under a narrower bind; it was not reachable in the shipped binary, since `main()` always passed a literal and an empty `--bind` is refused. The two hostname helpers behind it are gone with it. The default also no longer goes through the resolver, so a container with a broken `getaddrinfo` now starts where it used to fail.
It does not restrict which directory the client may name. On a loopback socket the client is the local user, who is entitled to choose their own mirror directory; rooting the path would break mirroring to an external disk. A loopback socket is also still reachable by any local process, and by DNS rebinding from a browser on the machine. A Host-header check is the follow-up for that, not this change.
New test 335 asserts the default socket refuses a connection on a real interface address while accepting one on `127.0.0.1`, using a `--bind` run to that address as its control, and skips on a host with loopback only. It passes on master too, because the `htsweb.c` literal already produced that behaviour: it guards against a future re-widening rather than fixing a present one, and it does fail against a build that restores the wildcard default. Test 77 already covered the same default against a `127.0.0.2` alias, which cannot see an off-host bind.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit e2d71a3a09713cae1951aea0ca06c72c8475d396
Author: Xavier Roche <roche@httrack.com>
Date: Sat Aug 22 19:45:30 2026 +0200
Test the index bound before reading the character it guards (#1360)
Five scans in the wildcard filter engine and in `host_ban()` put the index bound last in their conjunction, so the character read happens before the range check meant to authorize it. CodeQL reports all nine instances as `cpp/offset-use-before-range-check`.
No site reads out of bounds today. In `strjoker_impl()` a pattern only reaches the scan past a `joker[1]`/`joker[2]` guard, and `len` is `strlen(joker)`. The body's largest jump is itself gated on `joker[i + 2] != '\0'`, so the NUL retires the loop before the bound can matter. `heap(i)->adr` has one assignment in the tree, an arena strdup, so the read at index 1020 stays inside the allocation. Five million pattern/subject pairs compared against the pre-change build give byte-identical verdicts, and injecting a `len` longer than `strlen` does diverge, which is how I know the comparison can see this class at all.
The three identical body-less scans collapse into one `joker_class_end()`. The other two loops keep their bodies with the bound moved to the front.
Test 334 covers the class scan, terminated and not. The unterminated rows alone were vacuous for the helper: for an unterminated class every return from `len - 1` up gives the same verdict, so a helper written as `return strlen(joker)` passed all of them. The terminated rows make the returned index decide where the pattern resumes, which kills a helper returning `i + 1` or scanning for the wrong bracket. The remaining mutant, scanning to `i <= len`, changes no verdict at all because it reads `joker[len + 1]`; ASan catches that one, and the test does fail under the sanitize leg.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 35a84bc1e4c675a6246660c9560f046106d809c5
Author: Xavier Roche <roche@httrack.com>
Date: Sat Aug 22 19:45:27 2026 +0200
Check the format string at the log forwarders' call sites (#1358)
The log and debug forwarders take a printf format and hand it to a va_list sink, but none carried `HTS_PRINTF_FUN`, so gcc and clang never checked the format string at any call site. This adds it to eleven forwarders in `htslib.c`, `httrack-library.h`, `proxy/main.c`, `proxytrack.h` and `htsselftest.c`.
Nothing is silenced by this. gcc emits no `-Wformat-nonliteral` for a va_list sink, so the baseline was already zero warnings; what the attribute buys is checking at the call sites. Mutating two of them with a wrong specifier compiles silently on master and warns with this change.
`hts_template_formatv` and its wrappers are left alone on purpose. Their `%s`-only language ignores other specifiers and takes a mandatory NULL sentinel, so annotating them produces eight warnings on valid existing call sites.
The change is attribute-only: `nm -D --defined-only libhttrack.so` is byte-identical before and after, and `HTS_PRINTF_FUN` expands to nothing outside gcc and clang.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit d5c93e2279032d3a438f8f6d472a5bf38480d789
Author: Xavier Roche <roche@httrack.com>
Date: Sat Aug 22 14:52:21 2026 +0200
Two front ends send the update endpoint two different language parameters (#1361)
* Name the language the way WinHTTrack does in the update link
WebHTTrack hand-built the update URL with LanguageId=<index> while
WinHTTrack sends Language=<catalog basename>, so one endpoint received two
different parameters from two front ends and update.php only ever saw one
of them. WinHTTrack's spelling wins, as elsewhere.
The value is LANGUAGE_FILE rather than LANGUAGE_NAME for the reason
#1353 records: a display name is localized, is in the catalog's own legacy
codepage, and cannot survive the endpoint. LANGSEL already resolves any
catalog key through ${attr:}, so no engine change is needed.
Test 330 renders help.html under a catalog whose LANGUAGE_NAME differs
from its LANGUAGE_FILE and carries a high byte, and kills three mutants:
the old key, the endonym, and the ISO code.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Exercise a second language, and assert LANGUAGE_FILE across the catalogs
Review found the test could not tell "renders the basename" from "prints
the constant Francais": one language exercised makes those two
indistinguishable, so a hardcoded value passed. Test 330 now renders under
two, the fixture catalog and the shipped Castellano, and the hardcoded
mutant dies.
It also found the loader backfills an absent key from English, so a
catalog shipped without LANGUAGE_FILE would send Language=English rather
than nothing, silently and only for that language. 62_lang-integrity now
requires the key in every catalog, ASCII, and equal to the basename.
Mutant-checked both ways: emptied and renamed each fail it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 1c5402e854ddb3d9f140f2931d6c03105dc75e4e
Author: Xavier Roche <roche@httrack.com>
Date: Sat Aug 22 13:26:27 2026 +0200
Fail the sanitizer legs on a report a passing test hid (#1357)
Automake sends each test's output to `tests/<name>.log`, and `test-suite.log` keeps only the failing tests. ci.yml prints that log on failure alone, so a sanitizer abort inside one of the 16 test scripts that lack `set -e` left no trace in a green run.
ASan and MSan reports now go to a log_path directory. A new step scans it and the harness logs after `make check`, fails on a hit, and uploads both with `if: always()`. Both sources matter: a test that redirects its own stderr keeps a report out of the per-test log, and gcc's UBSan writes to stderr and ignores `log_path`, unlike clang's.
A sanitized run of the full suite stays clean, with `237_engine-arrays`' deliberate over-large allocation printed but not counted. Injecting a swallowed heap overflow and a swallowed signed overflow turns it red, each caught by a different source.
`236_local-ftp-teardown.test` takes its own `log_path` back: the job-level one diverted the ASan report its `#1051` check greps for.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit d757343ae9d258faebb05a3878ed127b649113cb
Author: Xavier Roche <roche@httrack.com>
Date: Fri Aug 21 21:07:56 2026 +0200
Clip the option-error diagnostics an oversized replayed token can overflow (#1356)
* Pin the argv length bound that three diagnostic buffers rely on
CodeQL reads htscoremain.c:784, :1800 and :1810 as unbounded writes, but an
argv length pre-check at the top of hts_main_internal bounds every one of them.
Nothing tested that guard, and argv is reassigned to a grown array at three
points after it has run.
Adds a test pinning both sides of the boundary, and moves the three sinks onto
slprintfbuff_clip so the alerts close rather than needing a dismissal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Clip the option-error diagnostics an oversized replayed token can overflow
The argv length pre-check runs once, over the original argv. doit.log is
replayed into the grown array afterwards with an 8000-byte line budget, so a
3002-byte token reaches the 1280-byte message buffer at htscoremain.c:783
unchecked and the engine aborts.
Also moves htsalias's five return_error sites onto the same clip primitive:
htslib.h remaps snprintf to the legacy msvcrt _snprintf on Windows, which leaves
the buffer unterminated on truncation and aborted both Windows legs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Tighten the clip assertions and isolate the test's working directory
The length window was loose enough that a message which merely fit would pass,
so assert the " not recognized" suffix is absent: it survives only when nothing
was clipped. Both runs read ./hts-cache/doit.log and $HOME/.httrackrc, so give
them a scratch directory rather than the shared build tree.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit e7f3a8ca84e62e2a22483573e5384461caf9ebce
Author: Xavier Roche <roche@httrack.com>
Date: Fri Aug 21 18:23:10 2026 +0200
Say what the update URL's Language parameter must carry (#1355)
Nothing in the engine calls HTS_UPDATE_WEBSITE, so the only statement of
what its %s means lived in the one caller, WinHTTrack, which fills it with
LANGUAGE_NAME. #1313 then changed LANGUAGE_NAME to endonyms in 14 catalogs
and the receiving page began rendering an empty field, because the
catalogs are per-language legacy codepages that no single declared charset
downstream can decode.
The macro now names the contract where a caller reads it.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
![[bug history graph]](https://qa.debian.org/data/bts/graphs/h/httrack.png){kind=link}