vcswatch reports that
this package seems to have new commits in its VCS but has
not yet updated debian/changelog. You should consider updating
the Debian changelog and uploading this new version into the archive.
Here are the relevant commit logs:
commit ecf31b2c8047a5340138b70ca7ccfb2bb5a3a187
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 4 22:47:11 2026 +0200
An .arc record's declared length can reach into the record stored after it (#1526)
An `.arc` record's declared length is read out of the archive. A record
that inflates it is served bytes belonging to the record stored after
it. That page's URL line, its `Content-Type`, its `Location`, its `Etag`
and its body all go out under the first record's URL. `proxytrack
--convert` copies them into the archive it writes, and a client GET
receives them. Whoever hands ProxyTrack an `.arc` to serve picks that
length.
`PT_LoadCache__Arc` now refuses a record whose declared data holds
another record's line, counted by where that line starts. A line ending
past the declared end still has its front served, so where it ends
decides nothing.
It hit the same trap three times, because the loader and the scan
disagreed on what a record line is. The scan's was narrower. A record
the loader indexes and serves under its own URL was then invisible to
the scan, and a neighbour could swallow it. Three fields rather than
five was one way through. So was a length `sscanf` reads but a digit run
does not. So was one byte `linput` drops, sitting in front of an
ordinary line.
The scan now asks the question the loader asks, `getArcLength` plus a
scheme it can serve. On top of that it takes the arc shape, for the
records the loader never serves. Heritrix writes one `dns:` record per
host, and those bound their neighbour all the same.
The scan also builds each line the way the loader's own `linput` does.
That is the one in `proxytrack.h`, not the `htslib.c` one proxytrack
never links. It drops NUL along with TAB, CR and form feed. A record
line carrying a NUL is therefore indexed and served, and a scan keeping
the NUL sees a C string that ends at it.
Widening the scan rather than narrowing the loader is deliberate.
Narrowing the loader would drop records httrack reads today out of
third-party archives, and the shapes involved are ones real writers
emit. The cost of widening is named below instead.
Refusing a record no longer costs the archive the records after it. The
walk resumes at the declared end and judges what it finds there as it
judges every other record. Nothing about the refusal decides whether the
walk goes on. A fixture whose next record line carries a trailing space
loaded nothing at all under the previous attempt. It now loads
everything except the greedy record itself.
The scan reads the record's data forward rather than seeking back over
it. Over an archive of 10000 small records, 658969 bytes, it reads
672676 bytes, exactly what the current release reads, against 1990127
for the previous attempt. That parity holds only where the release
already reads the data. A record large enough for the release to seek
past its body is read in full instead. One 64MB record costs 67115953
bytes against the release's 10985.
A page holding a line the loader would itself take for a record loses
its place in the archive. Over an archive built from every
documentation, man and HTML file on one machine, 2 records of 6827 are
refused. The check does not cover a record the walk parses but never
indexes, such as an undated `gopher:` line. A greedy neighbour can still
swallow that. The boundary the check draws is narrower than the release
behaves today.
Test 414 crafts an honest archive and the lying ones beside it, then
runs both consumers over each. It covers a length aligned to a later
record's separator, and one ending inside the next record's line. It
covers dropped bytes in front of a record line, three-field and
ten-field lines, and lengths a digit run rejects. It covers a line
straddling the scan's buffer or closing the file, and archives where the
record after a refusal must survive. Both the current release and the
previous attempt fail it. Of 500 generated mutants, 305 build and 171 of
those die. Sixty of the survivors sit in the reader's header parsing,
which this change does not touch. The rest mostly widen the check, which
costs pages rather than leaking them. One mutant makes the scan loop
forever, and only the suite's wall-clock guard reports it. That guard is
the mechanism for non-termination, so the reader needs no assertion of
its own.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 8e3aefd72fce55f70ade29226a658384bb8dc182
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 4 20:06:22 2026 +0200
The emulated leg times out where it used to pace itself, and reads argv[0] off a child (#1538)
The `emulated suite (s390x)` leg has been red on master since #1534. It
is the only place the suite runs big-endian. It is also not in the
required set, so a red one loses that coverage without blocking
anything.
Two tests fail, for unrelated reasons.
`397_configure-auto-features` hits its 900s guard. It already paced
itself with `skip_if_out_of_budget`, but only after its last three
steps, and #1534 added a debian/rules section ahead of them. A configure
run costs about thirty times more under qemu-user than it does natively.
The first paced step stopped fitting, so the guard fired before any
pacing ran. It now paces after every step, and projects against the
costliest step so far rather than the one just run. These steps are
ordered by what they mean rather than by what they cost, so a cheap
neighbour under-projects the slow step after it. On master at a 25s
budget the test runs 34s before skipping, which the guard would kill.
With this it skips at 16s.
`257_testlib-asserts` pinned the relay's image name to `sh`. binfmt_misc
leaves the interpreter in argv[0] and shifts the program one place
right, which is the shape `tools/emulated-suite.sh` asserts this leg
runs on. So under qemu the relay's cmdline names the emulator. The pid
half is what the case is for, and it was never in doubt. The test now
asserts the pid always, and the image only where a child's own cmdline
names the program it runs. A mutant giving that child a binfmt-shaped
argv[0] takes the other branch.
Nothing native changes: the suite still runs all fourteen steps of 397.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 823d98b0ed6fb7ab2e6e5a0b5a1de9fd14452f7e
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 4 17:32:32 2026 +0200
Twenty tests ask the shell what platform it is, and under WSL2 the shell is not the target (#1536)
Under the wsl2 backend the shell answers `Linux` while the binary under
test is still a native httrack.exe. Around twenty tests were keyed on
that shell answer, so they would have picked the POSIX expectation and
passed against a Windows binary. A test that passes while asserting the
wrong thing is worse than one that fails. The skip ratchet cannot see it
either, because those tests never skip.
Two kinds, and they want opposite predicates. Eight files chose a
Windows-or-POSIX expected value from `HTS_OS`, and now ask
`target_is_windows`, tested before the `Linux)` arm a wsl2 shell would
otherwise match. Nine guarded a Linux-only mechanism: ptrace, /proc, or
an LD_PRELOAD interposer sitting in the engine's own process. Those ask
the new `target_is_linux`, because a Linux shell is not enough when the
target is an exe.
Three sites were deliberately left alone, and the reasons matter more
than the count. `19_local-connect-fallback.test` switches on GNU/Hurd
having a single loopback address, which has nothing to do with Windows.
`218_crash-nopie-frames.test` and `372_crash-signal-number.test` compile
their own ELF probes with the local compiler and never invoke httrack,
so they genuinely want the shell.
The ratchet holds one skip list per backend now. An unknown backend is a
loud error rather than an empty list. An empty expectation makes every
skip look expected, so the instrument would stop working without saying
so. The wsl2 list starts as a copy of the msys one, labelled in the file
as a prediction. The first real wsl2 run is what corrects it.
Nothing here changes behaviour on Linux or under msys, which is the
point. `target_is_linux` reduces to the old `HTS_OS` test on both, and
the msys skip list is byte-identical. The suite reads 433 total, 419
pass, 14 skip, 0 fail, against 432 and 418 before, the difference being
the new test.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 96986df3bc1fc77cb5bdd32451a317813aadc4ff
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 4 16:46:55 2026 +0200
The Windows suite can only run under MSYS, whose fork emulation breaks first under load (#1535)
MSYS's fork emulation is what breaks first when the Windows suite is
under load (#1273). This adds a second harness against the same native
httrack.exe: a Linux shell under WSL2. It spawns 500 processes in 1s
where MSYS takes 7s on windows-2022 and 28s on windows-2025. WSL2 comes
up in about 35 seconds with no reboot, because the three Windows
features it needs are already enabled on the hosted images.
Those numbers time spawns to completion, so they measure throughput and
say nothing about a job that never finishes. #1531 is still what decides
whether forks provoke the runner wedges. This is worth doing either way,
because a harness that does not depend on fork emulation survives that
question concluding anything.
Both backends stay live behind `HTTRACK_SUITE_BACKEND`, so the msys path
is untouched and a regression is a variable flip rather than a revert.
Deleting the MSYS side is a later change.
`is_windows()` answered two questions at once. A survey of all 135 call
sites found 90 asking whether the binary under test is a Windows exe,
which stays true under WSL2. Only 6 ask whether the shell is MSYS, which
does not. It is deleted rather than redefined, so a site nobody
converted is a command-not-found instead of a silently wrong answer.
The one primitive WSL2 lacks is `/proc/<pid>/winpid`, because interop's
Linux-side child is only a relay. A spike on a hosted runner proved two
things. The Windows process can be found through a marker already on its
command line, and the whole tree then dies. Run:
https://github.com/xroche/httrack/actions/runs/33855598109. That marker
travels to PowerShell in the environment rather than in the argument
list. Passed as an argument it would sit on the querying process's own
command line, and the query would match itself.
Two bugs of the same shape turned up on the way. `proclib.sh`'s
forensics kept a bare `tasklist`, which under WSL2 returns nothing and
calls the host clean. `58_watchdog.test` ran a bare `ping`, where WSL's
Linux one reads `-n` as numeric output rather than a count.
The suite's skip ratchet measures the rest, because it diffs the skip
set both ways. The new leg therefore reports exactly which tests changed
status, rather than leaving it to be noticed. That leg does not gate the
job yet, since a red read against no history tells nobody anything. Flip
`continue-on-error` once it has a few days behind it.
The cost is one more Windows job per run, and every added Windows job is
one the runner can kill.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit efebf062b8bfac2dccc2218934b04973685300a9
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 4 15:32:40 2026 +0200
Measure which workload provokes the Windows runner kill (#1531)
#1228 ended on a hypothesis it called unsettleable from our side, that
the kill tracks MSYS process creation over loopback socket churn. It is
settleable as an A/B, and this is that experiment.
Three arms spend the same event budget on a different kind of event. One
makes MSYS forks alone. One makes loopback connections alone from
PowerShell, so no MSYS process exists. One makes a process and a
connection per iteration, which is the shape the suite has when it
spawns httrack.exe. No arm builds the engine, so a probe job is dose and
nothing else. Only one arm runs at a time. The job pool is shared and
already queues, so three more Windows jobs an hour would cost the
control its samples.
The control is the live windows-build suite, and
`tools/kill-probe-report.sh` prints it beside the arms, because a window
whose control shows no kill measures nothing. At roughly 5% per
uncensored job, hourly runs give about 40 exposures per arm in two days.
That separates an arm that kills from one that does not, and it will not
split two arms that both do.
Writing the census turned up two things. GitHub drops a job's step
records after about a week, and the wedge signature lives in them. An
older kill used to come back as a clean job, so it is recognised by
runtime instead. The lease reaps a lost job at around 45 minutes, where
a real failure ends in fifteen. The rate has also moved. Over 28 August
to 4 September the control reads 47 kills in 364 uncensored jobs,
against the 5.1% #1228 measured in July.
The probe stops itself on 2026-10-06 rather than leaving an hourly cron
nobody remembers. A PR touching these files runs the arms at a smoke
dose. The first such run caught two arms of three spending nothing at
all.
One correction to `windows-runner-death/MEASUREMENTS.md` in
httrack-works: Defender is out of the picture, because the runner images
put it in passive mode by design. What is left of that hypothesis is the
Windows Filtering Platform and the MSYS fork path.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit c70871df11dd7f0e3080f89bdef09487a56284d5
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 4 14:59:40 2026 +0200
Nothing checks that debian/rules still names every optional feature (#1534)
Three follow-ups a review of #1509 left open.
`debian/rules` passes `--disable-auto-features` and then names the four
optional features by flag, which makes that list the whole optional set.
Nothing checked the list, so deleting `--with-zstd` from it would have
shipped a package that cannot decode zstd and stayed green. 397 reads
the flags out of `debian/rules` now and requires every feature it knows
to be named.
Six review rounds went at this check and the first five each got through
it. Three readers in a row lost to make and shell syntax, so the check
no longer reads `debian/rules` at all. It runs the recipe with every
`dh_*` stubbed and records what `dh_auto_configure` is handed. Make
expands the variables and picks the recipe, the shell splits the words,
and only a list of tokens is left to judge.
What walked past one reader or another, all of it now pinned by a
fixture:
- `--with-zstd=no`, `--without-zstd` behind the positive flag, and a
value behind a make variable
- a `--with-zstd` commented out mid-continuation, and a backslash with a
space after it that make does not join
- a fallback invocation that never runs, and a second one that overrides
the first
- `--with-zstd=auto`, which hands the decision back to the buildd and is
the lottery the flag list exists to end
- a `dh_auto_configure` under `build-arch`, which is what
`dpkg-buildpackage` runs for an arch:any package, or behind a `dh`
sequencer's override target
- a flag behind an `ifeq` on `DEB_HOST_ARCH_OS`
- `--disable-https` and `--enable-https=no`, which no amount of checking
the four auto features would see
- a `dh_auto_configure` in an arch-only override or an `execute_after_`
hook, and a `./configure` run that is not `dh_auto_configure` at all
Three bounds are deliberate. The recorded flags are this host's
expansion. A second pass under a foreign `DEB_HOST_ARCH_OS` is a sample
rather than a proof. A flag gated on a file in the source tree escapes
it. A feature dropped by removing its `Build-Depends` makes the build
fail loudly, so it needs no catching here. Every recorded invocation has
to carry the whole list, which would red a rules file that configures a
second time for something else. A configure hidden in `binary` or
`install` is out of reach, because those recipes move files no scratch
tree has and stop before it.
Thirty-two fixtures pin the reader, written as literal text. Built by
expansion instead, the `--with-zstd="no"` case came out with an escaped
quote under bash 3.2 and read as passed on the macOS leg alone.
Four of them measure rather than assert. A broken recipe reaches make
either as a failure or as a silently shorter flag list, and which one
depends on the host. A comment inside a continuation reds on Linux and
grades as a short list on macOS. Those four run their makefile through a
plain make first, then require the reader to report what that run did.
GNU make 3.81 built from source agrees with 4.4.1 here on all
thirty-two, under `/bin/sh`, bash and dash alike. So the difference is
neither the make version nor the shell, and I have not identified it.
Two error arms configure has and 397 did not exercise. A bare
`--with-iconv` has to fail the build where `iconv_open` does not link,
rather than fall back to the codepage tables. The fixture is a header
declaring the symbol under a name nothing provides. That is what a
renaming libiconv header without its library looks like.
`--enable-backtrace=bogus` has to be refused the way
`--enable-auto-features=maybe` already is.
`feature_public` returned an exit status where its five siblings echo a
string. It echoes now, and it echoes nothing for a feature whose macro
stays out of the installed header. That is the shape `feature_libs` and
`feature_pc` already use for the same idea.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit c9840a479a2200b269ee26d354c9f0e473bfb5de
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 4 14:21:35 2026 +0200
Three option defaults documented wrongly, and a FAQ that denies SOCKS exists (#1520)
Six factual corrections to the option documentation. They come from
cross-referencing every switch in `htscoremain.c` against the man page,
the `html/` guide pages and both GUI front ends.
`--help` starred the wrong line twice. `-R`'s default is 2 (`opt->retry
= 2` in htslib.c), not the `*R1` shown. The default log mode is
`httrack_logmode = 3`, which is what `-f2` selects, so that star
belonged on `f2` rather than on `f`. Both were checked by running crawls
rather than only by reading the initialiser: a default run leaves
`hts-log.txt` alone, and `-f` also leaves `hts-err.txt`.
`man/httrack.1` and `html/httrack.man.html` are generated from `--help`,
so they carried both errors. Regenerating them is why the diff touches
four files for three text changes.
`html/fcguide.html` answered "Is there any SOCKS support?" with "Not
yet!". SOCKS5 and HTTP CONNECT have shipped and `faq.html` was corrected
for them, but fcguide was missed. The same page gives the maximum filter
count as 500 against a default of 200, and spells the footer option
`-&F`. That spelling works, because `-&` is an undocumented synonym for
`-%`, but it appears nowhere else in the tree.
Some findings were left alone on purpose. The `-C` default is genuinely
ambiguous. C1 and C2 cannot be told apart on a first crawl, and the
value from the second run on is what the help already says. Four
alias-table rows point at the wrong option. `--extract-cache` reaches
the cache lister rather than the extractor, and `--autotest` runs a
mirror rather than the autocheck. Those change what a command line does,
so they belong in a code PR. Most of fcguide's remaining wrong numbers
sit inside a verbatim paste of the 2001 help text. Correcting one number
there is an editorial decision about the page rather than a fact fix.
One earlier finding did not survive checking. `verbosedisplay` is set to
`HTS_VERBOSE_SIMPLE` on a terminal before any option is parsed, so the
`* %v1` star is right.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit d04b597c3888311e2c100c6c75ca415374f4f441
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 4 12:03:29 2026 +0200
A test that cannot run its subject reports PASS instead of a skip (#1533)
`399_deb-build-deps.test` ends at `exit 0` when PyYAML is absent, so
automake records a PASS for a test that asserted nothing.
`113_engine-threadattr-leak.test` does the same off Linux and on a
`--disable-shared` build, and `253_local-ftp-close-once.test` does it on
a `--disable-shared` build. All six sites call `skip` now, which exits
77.
`278_install-headers-msvc.test` is a seventh, and a review found it
rather than the scan below. Its staging leg needs a C compiler and its
workflow audit needs PyYAML. With both absent, the only thing still
running is two file-existence checks. Those are real assertions, which
is why nothing mechanical can see it. It counts its legs now and skips
when neither ran.
The line is whether anything ran before the gate. A test that asserts,
then bails out of a leg it cannot reach, has proved something, so PASS
is honest there. About twenty do that and stay untouched. A test that
bails before its first assertion has proved nothing.
I enumerated every `exit 0` under `tests/`, then reproduced each missing
prerequisite and read the verdict back:
- a `PYTHONPATH` pointing at a `yaml.py` that raises `ImportError`
- a `python3` shim that exits 1
- a second tree configured `--disable-shared`
- `HTS_OS=Darwin`
- an `abs_top_srcdir` carrying neither `debian/` nor `.github/`
Each of the six sites reported PASS before and reports SKIP after.
The Windows job pins its skip set, which makes it a control the Linux
runs cannot be. The sweep moved exactly one test into it.
`113_engine-threadattr-leak.test` was greening there without running its
subject. It counts a pthread_attr init/destroy imbalance through an
LD_PRELOAD interposer, and MSYS has no equivalent. The job sets no
`THREADATTRFAIL_LA` either, so the Linux gate is the only thing keeping
the test off its own "was not built" arm.
`253_local-ftp-close-once.test` interposes the same way and has been in
that list since it was written.
`418_test-vacuous-skip.test` is the guard. It reports an `exit 0` a test
can reach before anything that could have failed that test. What runs
inside the gate's own `if` counts as part of the gate, not as evidence.
Over the 430 tests on master it names those three files and nothing
else, and 278 is the case it cannot reach.
Its controls earn their place. The first scan was itself vacuous,
because awk carried its tally across files, so the first file's
assertions vouched for every file behind it. An adversarial review then
broke it eight more ways in both directions, `|| { exit 0; }` and a
`case` arm and a trailing comment among them. Twelve fixtures pin it
now, six that must be reported and six that must not, and gawk, mawk and
busybox awk agree on all twelve.
Two bounds remain. An assertion written on the gate's own line still
counts as prior work, and the word `ok` or `fail` inside a string reads
as one. A test can slip past on either. The remedy is the one a real
finding gets: write `skip` instead of `exit 0`, or teach the scan the
idiom.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 3fe462e75a0a94031ce9f8227853f138228e000a
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 4 09:30:36 2026 +0200
A cached X-Size sizes the buffer ProxyTrack reads a disk-held body into (#1517)
ProxyTrack reads a cache entry's `X-Size` out of the zip index and hands
it to `malloc(r->size + 1)` for entries whose body sits on disk
(`X-In-Cache: 0`). Nothing caps it between the header and the
allocation, so the header decides the size. Where `size_t` is 32 bits,
`X-Size: 4294967295` wraps that sum and `malloc(0)` returns a region
with no usable bytes. `hts_fread_exact` then fills it with the file's
contents, and `r->adr[r->size]` writes the terminator below the
allocation.
The engine already refuses this. `htscache.c` gates the same read at
`r.size >= INT_MAX` before it opens the body file, because either way
the body is read whole into memory. ProxyTrack reads the engine's own
caches and had no such gate. #1389 validated the decode (negative, wider
than `size_t`, in-cache above `INT_MAX`) and left the on-disk body path
uncapped. The gate now sits where the engine puts it, after the
save-name check and before the open. A declared body of 2 GB or more
comes back as `Cache Read Error : Bad Size` instead of a multi-gigabyte
allocation.
`tests/402_local-proxytrack-cache-bodysize.test` forges four sizes.
Three are the honest one, the `INT_MAX` boundary and `SIZE_MAX` on a
32-bit build. The fourth is a size the reader can still hold that the
file is far too short to satisfy. It matters because a fix that dropped
every short-file entry would pass the other three.
The wrap is 32-bit only and this box has no 32-bit toolchain, so it was
pinned outside the suite. A probe runs the reader's arithmetic with
`size_t` narrowed to 32 bits and allocates flush against a `PROT_NONE`
page. At `X-Size: 4294967295` the computed size is 0 and the fill takes
SIGSEGV on its first byte, while the honest size does not fault. ASan
cannot stand in for that, because it does not police a write into a
`malloc(0)` region.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 6c392115b3fbba644ebc805aec5e20721d6efa42
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 4 08:34:47 2026 +0200
Test 84 waits for a directory that the profile save already created (#1529)
`84_webhttrack-mirror-verbatim.test` failed on master's `build (Ubuntu
devel, uutils coreutils)` leg with "the crawl ended before the probe, so
this proves nothing". The uutils coreutils angle is a red herring,
because the assertion greps a file and none of the SIGPIPE differences
apply.
The test has to fetch `/website/` while a crawl runs. A finished crawl
serves that path too, so the probe would pass on an unfixed server. The
test waited for the crawl's project directory to appear, then read the
pane once. But `command_do=start` saves the profile before it launches
the engine, and the save on its own creates that directory. Sending
`command_do=save` alone confirms it. The directory appears, and the pane
correctly reports nothing in progress. The wait therefore never proved
the engine was running, and a slow start on a loaded runner failed the
single read.
It now waits on the pane, which moves only once the engine runs, and
dumps the server log if the crawl never shows up. Mutated to never start
the engine, the guard still fires.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 1c950aa80bedc8a8ee1538a275391c2014e4cffa
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 4 08:32:56 2026 +0200
hdiutil create loses a race on the new macOS runner image (#1530)
Master's `macOS app bundle (arm64)` leg failed twice tonight at "Pack
again under a channel label" with `hdiutil: create failed - Resource
busy`. The 62 runs before those were green. The break lines up with the
runner image moving to macos-15-arm64 20260829.0321.1. The last green
run on the old image was 2026-09-03 18:25, and on the new image the leg
has gone red, green, red.
The job runs `tools/macos-release.sh` three times back to back, and the
second pack is the one that dies. hdiutil attaches the image while it
fills it, and that attach collides with the previous run's helper. The
runner then reports an orphan `diskimages-help` process at cleanup.
The same script builds the DMG the release attaches, so this costs a
release build and not only a CI leg. It now retries up to four times,
and a permanent failure still stops with hdiutil's own message. No
Linux-runnable test covers a macOS packaging path, so I proved the loop
against a stub hdiutil. It succeeds on the first try, survives three
transient failures, and still exits 1 when hdiutil never succeeds.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 0340bdff563107a78931d9db10c3a7bfa3c1be21
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 4 07:45:10 2026 +0200
htsweb: stale Winsock-init comment on an empty callback (#1511)
`htsshow_init()` has an empty body. The comment above it, `/* Initialize
the Winsock */`, describes work the function no longer does.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit efc1514a1fa068e67bd253845fd1cd731351102f
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 4 06:07:52 2026 +0200
Ten early exits from hts_main_internal free nothing (#1528)
`hts_main_internal()` allocates a 64 KB `url` buffer and a token block
for the rebuilt command line, and `htsmain_free()` releases both. Every
error exit expands the macro. Ten exits that end the run without an
error do not. Under an ASan build, `-#h` reports 98392 bytes leaked in 5
allocations, against a clean `-h`. `-#C`, `-#E`, `-#R` and `-#t` report
the same block. Seven of the ten return 0 and three return 1, so what
they share is skipping the macro, not the status they exit with.
The macro now expands `freet` instead of raw `free`. The tree's
conventions ask for that anyway, and it makes a second expansion a
no-op. `cmdl_free()` was already idempotent, and the tail of the
function already called it twice.
Four more leaks sit on those same paths, and the macro covers none of
them, so they are fixed here. The `-#C` handler never releases the
coucal hashtable it builds, nor `cache.zipInput`, the archive
`cache_init()` opens, which costs another 352 bytes against a real
cache. `hts_extract_meta()` drops the meta-data handle when the cache
will not open, and the cache handle when the meta-data file cannot be
created.
One leak is left alone. The `-#C` loop calls `cache_read_ro()` per entry
and never frees the `r.adr` its caller owns. That loop is gated on
`hts-cache/new.ndx`, a legacy index no code in the tree writes any more.
No fixture reaches it, and a fix there would ship unexercised.
`tests/283_engine-cmdline-leak.test` was written for this class and
could not see it, because both of its cases were error exits. It now
drives all ten sites and all four extra leaks. Each case asserts a
message only that handler prints, plus the exit status. The closing
"Thanks for using HTTrack" line must be absent, which proves the run
left early instead of reaching the exit that already frees. Deleting one
release at a time reds the test for all fourteen.
Two cases needed a fixture rather than a flag. The aborted-mirror notice
is guarded by `!opt->quiet`, and a run whose stdout is not a terminal
sets `quiet`. That case therefore goes through `script(1)`. Its
util-linux and BSD spellings do not overlap. The test probes both with a
command only a terminal passes, and skips the case when neither works.
The `-#E` case that has to leave the cache open makes `hts-cache`
read-only. It verifies the denial rather than assuming it, and skips
when the write still succeeds, as it would for root.
The `-#R`, `-#E` and `-#C` cases that need a cache write their zip with
`printf`, which keeps python out of the test.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 5415fa2e62d4a1bedde72cba2f8437a92b526f49
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 4 05:36:52 2026 +0200
--sizehack on an update decides from the cache index alone (#1527)
`back_wait()` decides the same thing in four places: what we already
hold is the size the server announces, so keep it and force a 304. Three
of them measure the file on disk, and those same three refuse hypertext.
The fourth, the one that runs on an update crawl, reads the size out of
the cache index and guards nothing else.
The cache index records a past fetch, not what the mirror holds now.
Both consequences below reproduce against the local test server.
An HTML page is kept whenever its length holds, so a server that always
answers 200 hands the user last week's page. The same crawl without a
cache entry re-fetches it, because that sibling block refuses hypertext.
`--sizehack` still skips the transfer for every data file, and an honest
server still answers the conditional request with a real 304. What the
guard drops is the guess, and it drops it for the file type a reader
actually reads.
The second consequence loses the file. Say the mirrored copy no longer
has the recorded size, because it was edited by hand or the mirror was
rebuilt over an older cache. The forced 304 then sends `cache_read`
looking for a file of that size. The entry is refused, and the socket is
already closed. The file leaves the mirror and is never re-fetched. The
log carries `warning: file size on disk (24) does not have the expected
size (16))` and the run reports no errors.
So the block now asks the two questions its siblings ask, and keeps its
own cache read on top. One gap stays open, and master has it too.
`cache_read` validates the save name the cache recorded, where this
block measures the save name in force now. A crawl whose `-N` changed
between runs therefore still compares two different files. The change
narrows the class rather than closing it.
`tests/417` pins three outcomes: the HTML page must be re-fetched, the
drifted file must come back, and the data file must still take the
shortcut. That last arm is the one that fails if the option is simply
switched off. Crawling with `--footer ''` leaves the mirrored HTML
byte-identical to its source. So the HTML arm needs the mime guard, not
the change of size source. A second pass re-derived the mutants from the
diff. It killed six of seven: each added condition dropped in turn, each
inverted in turn, and the whole hunk reverted. No test can kill the
seventh, which drops `strnotempty(url_sav)`, because `fsize_utf8`
returns -1 on an empty name and the size comparison refuses on its own.
One divergence I found and left alone. The non-update sizehack block
omits the `/robots.txt` exclusion its two siblings carry. It sits on a
path this change does not touch, and nothing measured it. Folding an
unreproduced behavior change into this one would put two claims under
one title.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 36c4795740a1c888bd5ea2f8c4a76ee21bc433c3
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 4 03:36:45 2026 +0200
httrack-library.h is not enough to use CHAIN_FUNCTION() (#1522)
`CHAIN_FUNCTION()` is how html/plug.html tells a plugin author to
install a callback, and html/library.html sends that author to
`httrack-library.h` for the API. A translation unit that includes only
that header does not compile. The macro dereferences `httrackp` and
allocates a `t_hts_callbackarg`, and the header declared neither, so gcc
reports `unknown type name 't_hts_callbackarg'`. A `requires:
htsdefines.h` note sat above the macro, and nothing enforced it.
The fix includes `htsdefines.h` and `htsopt.h`, both already in
`DevIncludes_DATA`, so no new file is installed and no declaration
moved. `nm -D --defined-only libhttrack.so` reports the same 172 symbols
before and after.
The doc's own worked example includes all three headers, which is
probably why nobody hit this. Only a reader who followed the prose
rather than the example would.
`htsopt.h` pulls `htsnet.h` and therefore `winsock2.h`, and that broke
the MSVC build. `htsthread.c` took `windows.h` first through
`htsthread.h`, so `winsock.h` won and MSVC redefined `sockaddr`,
`fd_set` and the socket calls. The defect is the header rather than one
file's include order. Five other headers of ours had the same shape, and
nothing had yet made their translation units reach `winsock2.h`. New
`src/htswin32.h` establishes the order once, and six sites include it
instead of `<windows.h>`. It is not installed and it defines no macro
that hides a declaration. An include-graph walk over the Visual Studio
project files counts 58 translation units. One reached `windows.h` first
before the change, and none did after.
Test 411 stages `DevIncludes_DATA` into a private directory and compiles
`tests/chainplug.c` against it. No include path reaches the source tree,
because a `-Isrc` makes the bug vanish. The test also checks that the
same compile fails without the staged path, so a stray `-I` cannot make
it vacuous. Reverting the header change turns it red on the compiler
error above.
That reach means a consumer also needs OpenSSL's own headers. Where
OpenSSL ships a `.pc` file, `Requires.private` already carries the path.
Where configure finds OpenSSL only through `CPPFLAGS`, as on a keg-only
Homebrew prefix, nothing does, and `222_pkgconfig-consumer` caught it on
macOS. `libhttrack.pc` now names that directory, but only when the
compiler cannot find the header unaided. It takes an absolute path only,
because a relative one in an installed `.pc` resolves against whatever
directory the consumer builds in.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 0fbe7d6a4fb9f9eaf1cbd9759c886bef8b3a18d7
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 4 02:36:40 2026 +0200
html/library.html documents no symbol and points at a file an integrator cannot build (#1523)
`html/library.html` sent an integrator to `src/httrack.c` for an
example. That file includes six headers the development package does not
install: `htsbase.h`, `httrack.h`, `htslib.h`, `htscharset.h`,
`htsbacktrace.h` and `htsthread.h`. So it cannot compile outside the
source tree. The page named no symbol, no include path and no link flags
either.
It now lists the headers `DevIncludes_DATA` installs, gives the
`pkg-config` line, and points at the `libtest/` examples. Those examples
install to `$(datadir)/httrack/libtest` and include nothing outside that
set.
Both lists are copies of build-system facts, so they would rot the same
way the old text did. `tests/412` parses them back out of
`src/Makefile.am` and `src/httrack.c` and compares. A second agent
re-measured the mutants on a separate checkout and killed 7 of 7. Each
one edits a list the test compares.
A full API reference is a separate question, and this PR does not
address it.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 6a84bc6604b328587864e7dd0c5973198b4c3a23
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 4 02:34:21 2026 +0200
ProxyTrack ships the uninitialised tail of a buffer its cache body was too short to fill (#1521)
ProxyTrack sizes the body buffer from the cache entry's declared
`X-Size`, then fills it from the file that entry names. When the file is
shorter the read fails, but the buffer stays, and both consumers emit
`element->size` bytes whatever the status code says.
`proxytrack_process_HTTP` sends them to the client and the ARC writer
puts them on disk. The tail of that buffer is whatever the heap last
held.
An attacker controls both halves, because the declared size and the body
path are fields of the cache entry. Anyone who can hand ProxyTrack an
archive to serve (a downloaded mirror, a shared cache) picks how many
bytes leak. A 4096-byte `X-Size` over an 18-byte file leaked 4078 bytes,
and nothing but memory bounds that number. That heap belongs to a
process that has just parsed other clients' requests and other cache
entries. The bytes can carry URLs, headers and cached page content.
I reproduced it before fixing it, under `MALLOC_PERTURB_=165` so the
tail reads 0x5a rather than the zeros a fresh page hands out. 4078
sentinel bytes reached the client over the wire, and the same 4078
landed in the ARC.
The fix frees the body on a failed read, which is what the in-zip branch
beside it already does. The pre-3.31 `.ndx` reader carries a second copy
of the same allocate-then-fill and takes the same change. Two of the
four sites the new helper replaces already dropped the body, so those
hunks only remove the duplication. With no body to send, the reply still
announced `Content-length: element->size`, and the log line still
reported it. Both now ask for the bytes the element holds.
Test 406 puts the zip reader through both consumers and the `.ndx`
reader through the ARC writer. It requires the record to be present and
no consumer to announce more bytes than the file held. An implementation
that clips to the bytes it read therefore passes too, which is
deliberate. Ten mutants, all killed. Three revert one fix each, and two
refuse every disk-held body, one per reader. The rest zero-fill the
tail, clip to a wrong length, drop the body only past a threshold,
refuse the entry, or never serve one. The heap poison is a bonus rather
than the gate, and the two real-bug mutants still fail where the
allocator ignores `MALLOC_PERTURB_`.
PR #1517 narrows this window but does not close it, and its own test
deliberately leaves the holdable-but-short case out. The two branches
merge cleanly and both tests pass on the merge.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 7db1ea5b04d067e81886e51731ecb40aa6d15400
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 4 02:15:50 2026 +0200
Three alias rows expand to the option next to the one they name (#1524)
Three rows of `hts_optalias[]` expanded to a short option that neither
the alias name nor its own help string means.
`--extract-cache` reached `-#C`, which lists the cache, so it printed
"No cache entry found" and extracted nothing. Extraction is `-#E`.
`--autotest` reached `-#T`, the transfer-ops log, so `--autotest URL`
mirrored the site. The autocheck that prints `** AUTOCHECK OK` is `-#t`,
which no alias reached. `--debug-testfilters` still named `-#0`. #427
removed that arm with the other single-letter self-tests, so the option
reported itself unrecognized, mirrored its two filter arguments as URLs
and exited 0.
The first two rows are repointed. The third is deleted because what
replaced `-#0` is a self-test name (`-#test=filter`) and not a short
option. `--debug-testfilters` now reports an unknown option instead of a
wrong one. #427 already called the row vestigial. It survived only
because editing the array made clang-format reflow the table, which the
`/* clang-format off */` block has since fixed.
I checked all 175 rows against the case labels htscoremain.c carries,
and no other target is unreachable or misdirected. Two rows I left alone
on purpose. `--urllist` names `-%S`, the scan-rules list, where `-%L` is
the URL list. It is the first row for `-%S`, so the man page, the guide
and the generated help print it as that option's long name. history.txt
records a fix to it under that name too, so renaming or repointing it is
a product call. `--fast-engine` and `--debug-overflows` name `-#X`,
which still answers "warning: option has no effect".
The reverse lookup gives `-#E` a long name it did not have, so
`man/httrack.1` and its html render are regenerated.
Test 410 pins both directions. The `optalias` self-test asserts what
each name expands to, and what `-#E`, `-#C`, `-#t` and `-#T` resolve
back to. The script then asserts at runtime what each corrected option
does, against what its neighbour still does. It killed all seven
single-row mutants I ran, including repointing a neighbour and keeping
the right target under the stale value class.
One spelling changes class of outcome. `--extract-cache` moves from the
`level` class to `single`, because `-#E` takes no value.
`--extract-cache=2` used to be accepted and to list the cache. It now
exits 255 with `Option --extract-cache does not take the value 2`. The
bare and detached spellings both extract, and `-#C2` is untouched.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit d2aebf446faab33f557b714f70da8a3232006a68
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 4 02:14:20 2026 +0200
A cache body whose CRC does not match is served without complaint (#1525)
minizip verifies a member's CRC inside `unzCloseCurrentFile()`, once the
whole member has been read. The cache read path threw that return away,
so a body that inflated cleanly with a wrong CRC went into the mirror
unremarked. That case now reports a cache read failure, which costs one
refetch.
The index load in `cache_init()` discards the same return, and that one
stays. It reads no body, so the only member minizip can verify there is
an empty one, true of every headers-only entry. That CRC covers no
bytes, so checking it would drop a usable entry while saying nothing
about the file on disk.
The new case rides `tests/01_zlib-cache-corrupt.test`. It flips the
victim's stored CRC in both the local and the central header. The two
headers must agree, or the open fails first and the case would test the
coherency check instead. Mutation testing killed three of the four
mutants. The survivor keeps an earlier error message that no case in the
file distinguishes. A second agent re-measured this on its own checkout
and confirmed the corrupted member still inflates to full length before
the CRC verdict.
Two other items from the same sweep got a verdict and no change.
`copy_htsopt()` has no caller in the tree, but it is `HTSEXT_API` in the
installed `httrack-library.h`, so it counts as the front ends' API.
`-#C` enumerates from `hts-cache/new.ndx`, which nothing has written
since 3.31. A fresh mirror leaves only `new.zip`, so `httrack -#C '*'`
prints "No cache entry found" over a cache holding two entries.
Restoring that means enumerating the ZIP, and dropping the option
removes a documented feature, so this is a ruling, not a cleanup.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 2d269b26385aa6813a1f174d4f68049efe7119df
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 4 02:06:58 2026 +0200
Three paths fill an in-memory body past the bound the receive guard enforces (#1515)
`http_xfread1()` refuses an in-memory body at `INT32_MAX`, because
`r->adr` is indexed with an int downstream and the allocations around it
add a trailing NUL. Three other places fill `r->adr` without going
through that guard.
The chunk reader refused one value later, at `> INT32_MAX`. Cumulative
chunk sizes of exactly `INT32_MAX` therefore bought a 2 GiB `realloct`
out of `back_wait()`. Only the next `http_xfread1()` call tore the
transfer down. On a 32-bit build that allocation fails outright.
The decode path reaches the one size the guard rejects.
`hts_codec_maxout()` caps a decoded body at `INT_MAX` inclusive, so
`back_finalize()` can commit `r->size = INT32_MAX` and hand
`readfile_utf8()`'s buffer to `htsparse.c`.
`LOAD_IN_MEMORY_IF_NECESSARY()` reads a saved file back into memory to
parse it, with no bound at all. `htsparse.c` casts that size to `int
cSize` and passes `(int) r->size` to `check_html`. At line 3463 it
divides by `(int) (r->size)`, under an `if (r->size)` test made in the
wide type. At exactly 4 GiB the cast is zero and the guarded division
divides by zero.
That site is reached on an ordinary crawl. `back_finalize()` slurps a
written real-media body back into memory only below 8192 bytes
(htsback.c:1232). An `audio/x-mpegurl` response above that stays on
disk, `r.adr` stays NULL, and the load runs. An 11108-byte playlist logs
`File successfully loaded for parsing`. Nothing in the suite happened to
serve one that large, which is why no existing test covers the route.
One predicate, `hts_inmem_size_fits()`, names the bound for all four
sites. The decode refusal takes the route the neighbouring decode
failures take. It classes the slot `STATUSCODE_INVALID`, names it, and
drops the still-coded body so nothing can commit it as the page.
The local file is the one that cannot be bounded from outside.
`readfile2()` stats the file itself, and sizes both its `malloct` and
its `hts_fread_exact` from that stat. A caller that checks `fsize()`
first therefore leaves a window. A first draft of this PR did exactly
that. A 4-byte file that grew to 20 between the two calls came back as a
20-byte buffer. `readfile2_inmem()` applies the bound on the single stat
that sizes the read. `readfile2()` and `readfile()` keep the unbounded
behaviour they had. The cache index read in `htscoremain.c` and the two
in `htscache_selftest.c` are untouched. Whether they want a bound of
their own is a separate question.
`01_engine-xfread.test` pinned less than it looked like. It read `r.adr`
inside the `printf` argument list that also called `http_xfread1()`. gcc
sampled the pointer before the call, so the `adr=` column could never
see an allocation the guard made. Every case also started from a memset
`htsblk`, so `r.adr` was always NULL and a per-call guard was
indistinguishable from a first-allocation-only one. That second gap
hides a real memory bug, because the unknown-length arm would then reach
`realloct(r->adr, (int) r->size + bufl + 1)` with `r->size` past 2^31.
The test now reads `r.adr` and `r.statuscode` after the call in
statements of their own. It enters one case holding a buffer, and keeps
accept controls throughout, because a refusal-only assertion cannot tell
a bound from a brick.
Sixteen mutants, fourteen killed, with a semantic no-op green throughout
as the harness control. Nine are of the receive guard, graded against
the test each tree ships. That is 7 of 9 before and 9 of 9 after, and
the two that used to survive are the two gaps above. The chunk bound put
back to `>` dies in `130_local-chunked-truncated.test`, which now serves
a `7fffffff` chunk-size line. Neutering the decode predicate dies in the
self-test alone. The local-file bound dies on a sparse `INT32_MAX`-byte
file made under the build directory. The fixed build refuses it in 0.01s
without touching a page. A build without the bound reads all of it, in
1.45s and 2.1 GB. Making that bound refuse every file instead dies on
the small file beside it. The sparse case skips where the filesystem
will not hand one out.
Two mutants survive. Dropping the call to `back_set_decoded_size()`
along with the size it commits dies in `50_local-contentcodings.test`,
because the page is then written at its compressed length. Replacing
that same call with the bare size assignment is the survivor. It keeps
the page correct and drops only the bound, and nothing catches it.
Reverting the local-file call site to plain `readfile2()` restores that
whole bypass, and nothing catches it either. Telling either survivor
apart needs a body at the bound. One costs a real 2 GiB decode, the
other a 2 GiB file mirrored to disk. `local-crawl.sh` has no hook to
grow a mirrored file between passes, so that one would cost harness
plumbing for a single mutant.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit ae1de6e50acc808335f5faea59400446927d5f9f
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 4 00:58:32 2026 +0200
abortf_ never returns but nothing tells the compiler (#1518)
`abortf_` always ends in `abort()`, on every path including the
`HTSSAFE_ABORT_FUNCTION` callback branch and the empty override the
three ProxyTrack translation units install. Nothing in the source said
so. This adds `HTS_NORETURN` beside `HTS_DEPRECATED` in `htsglobal.h`
and applies it. The macro is spelled the way GCC, Clang and MSVC each
want, and placed where all three accept it.
The claim that this retires 21 static-analyzer findings did not survive
measurement. scan-build and `gcc -fanalyzer` report the same counts and
breakdowns before and after. Each already inlines the static `abortf_`
and sees the `abort()` for itself. CodeQL is what CI runs, and I could
not run it here.
No new warning appears, and a `-Wall -Wextra` build gives the same count
and breakdown either way. No test is added, because nothing observable
changes.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 0d3c871385cea1a44a1df15a0a5b628e99bf7c58
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 4 00:48:53 2026 +0200
A reference on a document's first byte makes --single-file read in front of it (#1519)
`--single-file` asks `singlefile_ref_class()` whether a reference
htsparse found may be inlined. Inside a stylesheet or a script there is
no enclosing tag, so the answer comes from `rech_tageq(attr, "import")`.
That macro starts by reading the byte in front of the token, to decide
whether the token starts a word. When the token is the document's first
byte, the read lands one before the response body on the heap.
A server picks both the mime type and the body it sends. A six-byte
`text/css` reply of `url(x)` is enough, and the same body behind one
space is clean. Under ASan the process dies. Without a sanitizer the
stray byte decides whether a body opening with `import = ` counts as a
stylesheet. The mirror's output then follows whatever the allocator left
there. `--single-file` is off by default.
The tree already owned the safe primitive. `html_prevc()` returns a
space sentinel at a buffer start instead of reading `[-1]`. That is also
the right answer here, because a token opening a document does sit on a
word boundary. `singlefile_ref_class()` now receives the document start
and uses it for all three of its boundary tests. No path through the
function reads below the body any more.
`tests/401_local-single-file-byte-zero-ref.test` crawls a stylesheet
whose first byte opens a `url()` and checks the reference still inlines.
It passes on the fix and fails on the reverted one under ASan, the leg
that sees this class of bug.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit a65612a3c04dc4b47444406c1106830c30c23a78
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 4 00:44:16 2026 +0200
A ZIP extra field picks its own copy length in an unreachable minizip helper (#1514)
`zipRemoveExtraInfoBlock` in the vendored `src/minizip/zip.c` walked a
ZIP extra field without ever validating the block size it read out of
that field. Three defects follow from the one missing check. The loop
guard `while(p < (pData + *dataLen))` admits a partial block, so the two
`*(short*)` reads opening each iteration run past the buffer. The
`memcpy(pTmp, p, dataSize + 4)` after them is never compared against
`*dataLen`, so two bytes taken from the archive choose both a source
offset and a copy length. A declared `dataSize` of 0x7ff0 writes 32756
bytes into an allocation sized `*dataLen`, which is a controlled-offset,
controlled-length heap write rather than an overread. `dataSize` is also
a signed `short`, so a declared 0x8000 makes `dataSize + 4` negative and
wraps the length. Those same two loads are misaligned, which faults on
sh4, m68k, hppa, sparc, mips and alpha.
Nothing reaches any of it today, so the fix is deletion rather than a
patch. Bounding a function with no caller means inventing its contract,
because every limit you pick is a guess about a caller that does not
exist. Removing it retires the whole class.
The reachability verdict comes from the linker rather than from grep.
Across the 141 objects a full build produces, `nm --defined-only` finds
the definition in `src/minizip/proxytrack-zip.o` and
`src/minizip/libhttrack_la-zip.o` (plus libtool's PIC copy). `nm
--undefined-only` finds no reference in any of those objects, nor in
`httrack`, `htsserver`, `proxytrack`, `libhttrack.so` or the ten test
plugins. `nm -D --defined-only libhttrack.so.3.0.20` lists 172 exported
symbols and this is not one of them. That count is the control, because
a probe reading an empty symbol table would report the same absence.
`minizip/zip.h` is not in `DevIncludes_DATA`, so it never reaches an
installed prefix. Upstream minizip does not call it either, since the
`.orig` copies in the tree carry the same definition and prototype with
no call site. The two apparent calls in `zip.h` sit inside the comment
block that opens at line 359 and closes at line 374.
`tests/405_minizip-no-extra-parser.test` scans the built library and
every object under `src/`, and it requires `zipClose` in that same set
before it trusts an absence. Without that control the test would pass on
a stripped or empty table. It kills the mutant: a stub definition reds
it and names all four products carrying the symbol.
The tree keeps a `.orig` plus `.diff` record per vendored file, so
`zip.c.diff` and `zip.h.diff` are regenerated. Both were checked by
applying them to the `.orig` and comparing against the tree copy. The
pre-existing hunks (the `next_in` const fix, `zipFlush`,
`zipAbandonFileInZip`) survive unchanged.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 54293d872a8e02593617235cb02ca2a40998c89e
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 4 00:39:38 2026 +0200
A skipped crash-symbolize test names no suspect (#1513)
The emulated s390x leg skips `80_engine-crash-symbolize.test` with
`addr2line names no hidden symbol in this build`. That one message
covers three failures that read identically in the log:
- the trace parsed no frames,
- no frame's module resolved to a file on disk,
- addr2line ran and named something other than the hidden frames.
The test now prints how many frames it parsed and how many it resolved,
and it dumps the raw trace and the oracle before skipping. A negative
control with a silent `addr2line` on the PATH confirms the dump fires.
That diagnostic then ran on this branch and named the cause. The s390x
trace stops at `linux-vdso64.so.1(__kernel_sigreturn)` after two frames,
and addr2line resolved both to a file and a line. The engine frames sit
below the signal frame and never reach the trace, so the old message
blamed the wrong component. The second commit skips with that reason
when the trace carries a sigreturn entry. It stays a skip: crossing a
signal frame under qemu-user is not something a test can fix.
`218_crash-nopie-frames.test` skips on that leg for what looks like the
same reason.
The overnight architecture sweep had blamed a missing `-g`, on the
theory that `tools/emulated-suite.sh` runs a bare `configure` and
addr2line therefore resolves nothing. Both halves are wrong.
`configure.ac` never assigns `CFLAGS`, so `AC_PROG_CC` supplies gcc's
default of `-g -O2`, and the emulated log carries `checking whether gcc
accepts -g... yes`. I also built this tree with `CFLAGS=-O2`, where the
test passes.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 4411fc73003180b3e08551d1a19f25a0c4980a28
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 4 00:38:50 2026 +0200
Two is_space definitions in src/, with different character sets (#1516)
`src/htscharset.c` defines a static `is_space()` over SPACE, TAB, CR and
LF. `src/htslib.h` defines a function-like macro of the same name over
eight characters: that set plus VT, FF and both quote characters.
`src/htsserver.h` carried a verbatim copy of the macro. Nothing in
`htscharset.c` reaches either header today, so the local definition wins
and the sniffer behaves as intended.
I checked what a colliding include would actually do, because that
decides how urgent this is. A macro that arrives before the definition
turns `static int is_space(char c)` into a syntax error, so a
top-of-file include fails loudly. A macro that arrives after it compiles
clean, and every call site below silently switches to the
eight-character set. `htscharset.c` already has includes at lines 495 to
504, so either ordering is reachable. Renaming the predicate to
`is_html_space` removes the trap in both cases.
Behaviour does not move. The macro keeps its eight characters and every
caller keeps the set it had. Only the sniffer's three static helpers
change name. The new cases in `tests/01_engine-charset.test` pin both
halves of the sniffer's set. Six cover the characters the two sets
disagree on, and each flips under a mutant whose `is_html_space` body is
the eight-character set. Three more feed TAB, CR and LF, which kill a
mutant matching a bare space. Those three need the batch path, because
argv rewrites them to a space before the self-test sees them.
`htsserver.h` loses only `is_space`, so one definition of the
eight-character set survives. Its other four macros stay, because
`htsweb.c` reads `is_realspace` from that header without including
`htslib.h`, and because they are byte-identical to `htslib.h`'s.
`htsserver.c` includes both headers, so a reworded copy would draw a
redefinition warning.
The macro's `/* Spaces: CR,LF,TAB,FF */` comment named neither the
quotes nor VT, so it is gone. Correcting it in place was not possible:
clang-format 19 rewrites about 35 lines of that hand-formatted region
for any line added there.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit b5c8db9ee5bf4206852dff1408a05e92ecdaa9ef
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 4 00:18:04 2026 +0200
-N's digit encoding is a parity test, not a threshold (#1512)
`src/htsname.c` twice carried `// >1000 signifie "pas de www/"` over
`((opt->savename_type / 1000) % 2) == 0`. The code tests that digit's
parity rather than a threshold. So `-N2000` keeps the host directory the
comment says it drops, and `-N3001` drops the `web/` directory a
threshold reading would keep. No shipped preset is affected, because
every one the man page documents stays below 2000.
Behaviour is unchanged. Both comments now say what the test does, and
the encoding itself moves to the field in `htsopt.h`, which recorded
none of it. `% 100` picks the tree, an odd hundreds digit names the top
directory after the host, and an odd thousands digit drops that
directory. Site structure is the exception the man page's `-N100`
already hints at, because it reads no hundreds digit beyond that literal
value.
The new `400_engine-savename-type-digits.test` pins each of those claims
through `-#test=savename`. I ran it against both threshold mutants, one
per digit, and each one reds it.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 77a97c81342efb6be2f8ea23d87973d113c9e3e6
Author: Xavier Roche <roche@httrack.com>
Date: Thu Sep 3 23:55:06 2026 +0200
Optional dependencies are decided by the build machine, not by the recipe (#1509)
Four optional dependencies default to auto, so this source ships the
`br` and `zstd` codings on Gentoo and drops them in a FreeBSD jail.
Turning iconv off means the built-in codepage tables, which read no
multi-byte charset, so `debian/rules` asks for iconv back. Those tables
could never find their own `cp1252` entry from the `windows-1252` label
every HTTP header uses, which this fixes.
Building without iconv also exposed a crash in the same tables. A
leading NUL stopped the decode loop before it had allocated anything, so
the function stored the terminator through a null pointer.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit e5f372ab7c9944c40a04c3cfd4b7db7e21f0a76a
Author: Xavier Roche <roche@httrack.com>
Date: Thu Sep 3 20:23:59 2026 +0200
HTS_FIND_LIBDIR never looks under a multiarch triplet (#1510)
On Debian and Ubuntu `--with-brotli=/usr` and `--with-zlib=/usr` both
hard-error on master, because the shared libraries live under
`/usr/lib/<triplet>`. Clang rejects `-print-multiarch` and several CI
legs are clang, so detection falls back to the directory the compiler
resolves libc in. The triplet is guessed after `lib` and `lib64`, so no
prefix that resolves today changes its answer.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit c8a61f44c1f48b78a28e8a0b17d283ddb3e14947
Author: Xavier Roche <roche@httrack.com>
Date: Thu Sep 3 19:01:26 2026 +0200
The WebHTTrack UI installs under docdir, which packagers strip (#1508)
htsserver serves the WebHTTrack interface at runtime, and every
packaging system treats `$(docdir)` as documentation it may strip. The
interface now installs under `$(datadir)/httrack/html/server`, and a
relative `doc` symlink beside it reaches `$(docdir)` so the Help links
still work. The documentation does not move, so #1477's layout holds,
and a `symlink_to_dir` maintscript carries Debian past the symlink dpkg
will not replace.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit e67a8661e17c0e9d8764e75725f17921b42f2198
Author: Xavier Roche <roche@httrack.com>
Date: Thu Sep 3 14:48:34 2026 +0200
hts_self_path() has no BSD arm, so FreeBSD falls back to argv[0] (#1507)
FreeBSD, NetBSD and DragonFly mount no /proc, so `hts_self_path()`
returned NULL and the caller fell back to the argv[0] it exists to
avoid. Each mib stays symbolic and comes from that BSD's own headers,
because 9 is the pathname on DragonFly and `KERN_PROC_SV_NAME` on
FreeBSD. The self-test sweep then caught the Windows arm leaving a
terminated clipped path behind, so all four arms now share one refusal
exit. No CI leg here executes a line of the sysctl arm, so the FreeBSD
canary merged as #1505 is its only real verification.
Closes #1506
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 6676a1f13b9983d3ebaaabc5d28f840f40257c57
Author: Xavier Roche <roche@httrack.com>
Date: Thu Sep 3 10:55:45 2026 +0200
Nothing rejects two tests claiming the same number (#1504)
The exemptions are listed by family name rather than by a magnitude
threshold, because the tree's sub-100 duplicates were accidents rather
than a convention. So `02_update-cache` and `13_crawl_proxy_https` move
to 394 and 395 instead of being grandfathered. This catches a duplicate
that lands and stays, but not two open PRs picking 389, since the
ruleset lets a PR merge behind master.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 004b88452870497b88f9a54b4e8a193ca6318a82
Author: Xavier Roche <roche@httrack.com>
Date: Thu Sep 3 10:10:55 2026 +0200
Nothing in our CI compiles this tree against a BSD libc (#1505)
Nothing in our CI compiles against a BSD libc, and macOS is not close
enough to stand in for one. The first runs turned up eight FreeBSD
failures, so the job pins those names and reds on a ninth or one that
stops failing. It skips the FreeBSD port on purpose, because that port
does not build against 3.50.1 and `tools/downstream-drift.sh` already
watches it.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit d5a9a707101bdbebb43191cf9f8792c7d793404a
Author: Xavier Roche <roche@httrack.com>
Date: Thu Sep 3 09:44:37 2026 +0200
Only Fedora and Termux build master the way a packager does (#1500)
Two defects in 3.50.0 (#1487, #1488) were visible only from a downstream
build, and reading a recipe statically cannot find that class. These
three canaries build master weekly through each packager's own recipe,
and they block nothing.
The openSUSE leg packages three files their `%files` predates, so rpm's
unpackaged-file check stays armed for a fourth. Each acknowledgement
reds once they package that file themselves. The tarball step moved into
a shared action, which is why the merged Termux canary is in the diff.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 4a50a42af4cbafca5259a79d5628ed61ff177b34
Author: Xavier Roche <roche@httrack.com>
Date: Thu Sep 3 09:41:42 2026 +0200
A crafted ready-table size header writes outside its allocation, plus two mistyped-pointer writes (#1503)
A `SIZE_MAX` size header in a backlog ready-table file wrapped
`malloct(*size + 1)` to zero, so the guard byte wrote outside the
allocation. The same helper let a `void *` out-parameter travel through
a `char **` and a `lien_back **`, the tree's only `-Wstrict-aliasing=2`
site. Five enum option fields scanned through an `(int *)` cast are
written out of the object under `-fshort-enums`, and new tests cover all
three.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit bad1278e99390454148f926d0d9e7b807a94948b
Author: Xavier Roche <roche@httrack.com>
Date: Thu Sep 3 09:28:14 2026 +0200
--with-brotli=DIR and --with-zstd=DIR look for the library in DIR/lib and nowhere else (#1502)
`--with-brotli=DIR` and `--with-zstd=DIR` appended `DIR/lib`
unconditionally, so a lib64 or multiarch prefix left configure reporting
the coding as enabled against a system copy. #1472 fixed the same shape
next door in `CHECK_ZLIB`, and the subdirectory search now lives in one
macro both call. PACKAGING.md gains the flags that pin the two codings,
which default to `auto` and have been following whatever the builder
happened to hold.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 9250384b605b6f161bfe57298052ca7602b16b71
Author: Xavier Roche <roche@httrack.com>
Date: Thu Sep 3 07:14:44 2026 +0200
Nothing in our CI builds the way a Termux packager does (#1499)
Two defects in 3.50.0 (#1487, #1488) were visible only from a
Termux-style build. This runs master through their own recipe weekly, in
their builder image. It applies their patches verbatim and blocks
nothing, so it stays red until termux/termux-packages#31457 removes the
five that no longer fit master.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
![[bug history graph]](https://qa.debian.org/data/bts/graphs/h/httrack.png){kind=link}