vcswatch reports that
this package seems to have new commits in its VCS but has
not yet updated debian/changelog. You should consider updating
the Debian changelog and uploading this new version into the archive.
Here are the relevant commit logs:
commit 0b84a75ef2f7bb101e29c2334be1e399f94047a5
Author: Xavier Roche <roche@httrack.com>
Date: Mon Sep 28 22:14:40 2026 +0200
Skip the deliberate-crash tests on GNU/Hurd (#1831)
Hurd's ext2fs translator has twice asserted and taken the whole VM down
right after a test crashed a process on purpose. No test had failed
either time, so the tier-2 hurd leg went red for the host's reason. The
eight tests whose point is a deliberate crash now skip there.
The cause is not proven. A probe ran faulting processes against the same
image for 75 minutes and never reproduced the assertion. This trades
crash coverage on a tier-3 platform for a leg that finishes.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 61c594c19abf5e4f8567369050fb441057158bfe
Author: Xavier Roche <roche@httrack.com>
Date: Mon Sep 28 16:39:36 2026 +0200
Reap a watchdog that outlived the driver that started it (#1828)
A process that outlives the test that started it keeps writing to a log
the teardown has already unlinked. The space is then held by a file with
no name, which `du` cannot see at any privilege. Such orphans pinned
11.6 GB of a 16 GB tmpfs here, and the next `make check` reds on
unrelated tests.
Two producers need different mechanisms. The watchdog outlives a driver
whose EXIT trap never ran, so `ci_start_native_watchdog()` now records
its pid beside the log. That trap and test 172's teardown both reap it.
The sink's own parent is the test, where no trap survives a SIGKILL. It
now holds a pipe the test owns, and exits when that closes.
A probe decides whether the interpreter can wait on such a pipe. Where
it cannot, as under MSYS, the sink is the one master starts.
Each fix carries a leg that SIGKILLs the parent and asserts the log is
released.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <xroche@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 9323cb66979ef3babcb4a9995a75f7d2c3b752fe
Author: Xavier Roche <roche@httrack.com>
Date: Mon Sep 28 12:03:42 2026 +0200
End the Hurd run when its VM stops answering (#1829)
A Hurd VM that goes down under the suite used to block the run until the
step timeout, half an hour later. It then read as a suite failure
although no test had failed. A watchdog now ends the run once the suite
has gone quiet and ssh stops answering, and the error says the VM went
down. The leg still goes red, because the ext2fs translator asserting
under the suite is Hurd's bug and not ours.
Silence alone must not decide this, because a test may legitimately run
1200s without printing, so the unanswered ssh probes are what call it.
Test 524 pins that control. This branch's own hurd leg then wedged for
real and the watchdog called it after 17 minutes rather than 40.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit e0b4ea90784361b1461a608b6f14367fe18681bb
Author: Xavier Roche <roche@httrack.com>
Date: Mon Sep 28 08:33:45 2026 +0200
Keep a savename hook from writing outside the mirror (#1826)
`fil_simplifie()` stripped `../` from the save path before the
`savename` hook ran, and nothing checked what the hook wrote back. A
hook returning `<host>/../../ESCAPED/page.html` wrote the page outside
the mirror, at any depth. With no `-O` the root a name joins to is
empty, so an absolute name escaped with no `../` at all.
The engine now keeps its own name whenever the hook leaves an absolute
one, or one carrying a `..`, and logs that at LOG_WARNING. It checks
every name the hook returns. Its own name has to be relative there, so
it strips leading slashes again right before the callback. `-N100` on a
`//` URL path leaves one, and `cleanEndingSpaceOrDot()` puts one back by
emptying a dots-only component. The check is lexical, so a hook aiming
at a symlink that already sits inside the mirror still writes wherever
that symlink points.
Closes #1823
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit e1394df122b4471fbf025d60f31023fef839f5b4
Author: Xavier Roche <roche@httrack.com>
Date: Mon Sep 28 08:11:47 2026 +0200
Stop sending a request the sendhead hook refused (#1825)
A `sendhead` callback returning anything but 1 closed the socket and set
"Header refused by external wrapper". Control then fell through to
`sendc()` and sent the request anyway. Over HTTPS that reaches
`SSL_write()` with the `ssl_con` that `deletesoc_r()` had already freed
and NULLed, because it leaves `r->ssl` set. OpenSSL 3.5.7 returns an
error rather than faulting, so the dereference is latent and
version-dependent, not a reproduced crash.
Stopping the send is only half of it, and that half is invisible on its
own. `back_wait()` marked the slot STATUS_WAIT_HEADERS whichever way the
send went, so its next pass found the closed socket and replaced the
reason with "Receive Error". The caller now reads the return value and
finishes the slot, which is what makes the refusal reach the log. The
status code and retry count are unchanged.
Closes #1822
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 66ce5765204d0ea1275e9c21604d5c1451ed8dbd
Author: Xavier Roche <roche@httrack.com>
Date: Mon Sep 28 07:17:29 2026 +0200
Stop hts_getcategories() returning two fake categories (#1824)
`hts_getcategories(path, 1)` prepended the literals "Test category 1"
and "Test category 2" to every result. A front end listing mirror
categories therefore showed two projects that do not exist, WinHTTrack's
category menu being the visible case.
The two strings never reached the deduplication table the real
categories use, so a project genuinely named "Test category 1" came back
twice as well. The new self-test pins both: the exact set a caller sees,
and that name appearing once.
PR #1820 documented the placeholders as intended behaviour a few commits
ago. That sentence goes too, or the installed header contradicts the
fix.
Closes #1821
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 6ffb8f2caf54300bbaf4284da2ff56f0fc369216
Author: Xavier Roche <roche@httrack.com>
Date: Sun Sep 27 21:53:22 2026 +0200
Count received bytes under a lock, so no FTP transfer's bytes go missing (#1812)
Every FTP transfer counts its received bytes on its own worker thread,
so the plain `+=` they all did on one shared total lost adds. `-M` reads
that total to decide when to stop, so a lost add lets a crawl run past
the size the user asked for.
The total now sits in a static behind a lock, and
`HTS_STAT.HTS_TOTAL_RECV` becomes a copy published beside the other
fields `engine_stats()` refreshes. It uses a lock rather than a 64-bit
atomic add, because gcc calls libatomic for that where the target has no
8-byte compare-and-swap.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit e9e603ef4ee9c0469e412f39a66e470cd791f07e
Author: Xavier Roche <roche@httrack.com>
Date: Sun Sep 27 21:38:51 2026 +0200
Add a ThreadSanitizer CI leg (#1819)
On x86-64 a missing release/acquire pair behaves exactly like a correct
one, so nothing in CI could guard the pairs #1808 and #1809 added. This
adds ThreadSanitizer as a non-required leg over the engine self-tests,
plus 467 and 503, the FTP thread tests the `01_engine-*` glob misses.
`tools/tsan-suppressions.txt` names the benign readers and never the
writer. Suppressing `ftp_worker_release` would be one tidier line, but a
mutant confirmed it also lets a plain publish pass clean. #1809
described these races and introduced neither. The leg's blind spot is
the race with a user-visible effect. An FTP worker bumps
`HTS_TOTAL_RECV` while `back_maxsize_reached` reads it to decide `-M`,
and this fixture transfers no bytes, so #1812 stays the fix.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 2edb4ec94af72058d6cf734f1db7fb25f377d028
Author: Xavier Roche <roche@httrack.com>
Date: Sun Sep 27 21:38:41 2026 +0200
Name the buffer's capacity where these four write into it (#1815)
Four places wrote into a fixed buffer without naming its capacity. The
conditional-request headers in htsback.c are built from an etag and a
date read back out of the cache. Only a clip two functions away bounds
them. htsindex.c scanned a keyword with a bare "%s". htsserver.c could
write 260 bytes into the buffer its own header documents as 258, so
smallserver_init now takes the destination size.
The one with a reachable effect is the backing table. -c is clamped only
when --bypass-limits is off. A large -c then overflowed the signed
maxsoc * 32 + 1024, and the engine asked its allocator for a negative
count. The keyword width has to be a literal, so a static assertion
beside the buffer is what keeps the two in step.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit bb6fc0d69aa15166440cf878cb23f13263bef3f1
Author: Xavier Roche <roche@httrack.com>
Date: Sun Sep 27 19:45:18 2026 +0200
Document every entity in the installed headers (#1820)
Every function, struct field, enum value, macro and typedef in the
fourteen headers HTTrack installs now carries a contract comment. Each
says what a return value means, who frees a pointer, and what size a
caller buffer needs.
Many comments already there were wrong rather than thin. `check_link`
and `check_mime` documented 0 as "drop the link", where 0 accepts and 1
refuses, on the callback surface a front end implements. `strclipbuff`
and `slprintfbuff` were documented as never aborting, which is the
property that makes them safe for hostile input. Each aborts on a zero
size.
One line in `htslib.c` changed for the same reason: `no_high & 1` keeps
high bytes escaped, where the comment said it decodes them.
The change is comments only. Each file's comment-stripped token stream
and its preprocessor macro table are unchanged against the merge base,
in both `HTS_INTERNAL_BYTECODE` arms.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit ae733ca408637c063b4a202565c11bda0c226857
Author: Xavier Roche <roche@httrack.com>
Date: Sun Sep 27 16:41:05 2026 +0200
Bound the copies in the installed example wrappers (#1810)
These wrappers are installed twice over: as shared libraries, and as the
sources users copy. contentfilter had a reachable bug with no CodeQL
alert on it. It splits its keyword list into a 128-entry pointer array
and kept writing past the end. 200 one-byte keywords reach that inside a
single argv element.
The three cpp/unsafe-strcat alerts are a different matter. Those
wrappers joined an engine buffer to a second string with strcpy plus
strcat. That is worth bounding in code people copy, but measuring for
the runtime probes showed no crawl can overflow it. The engine drops a
link past HTS_URLMAXSIZE before a wrapper sees it: 926 bytes arrives,
1000 does not. Both destinations are lien_adrfil's own HTS_URLMAXSIZE *
2, so the worst join is 1534 bytes into 2048. That measurement also
caught an over-rejection here. The first version sized both buffers at
1024 and so refused joins the old code handled.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 53a94374e5539bf44b6b325a755508613d40b96a
Author: Xavier Roche <roche@httrack.com>
Date: Sun Sep 27 16:40:53 2026 +0200
Refuse a cached save name that leaves the mirror (#1813)
X-Save is a filename read back out of a cache the process did not write.
Both readers joined it to the mirror root, so a name that walks out of
that root reaches other files. ProxyTrack opens that path and sends the
bytes to its client. It binds whatever address argv[1] names, so an
archive from an untrusted source becomes an arbitrary file read. The
engine's own reader reaches a rename with the same value.
Rejecting ".." alone is not enough. With no -O the root is "" and the
cached name is used verbatim, so an absolute one walks straight through.
A blanket rejection is wrong too, because an absolute name already under
the mirror is the pre-3.40 shape the engine's own self-test stores on
purpose. So a name carrying the root has the remainder checked, and
anything else has to be relative. ProxyTrack keeps the ".."-only test at
its top, since it routes absolute names into that pre-3.40 branch by
design. That branch now checks the suffix it actually joins, rather than
the whole string it was handed.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 3a93fe8be8bc39e0139c58550eb2e3945ab54482
Author: Xavier Roche <roche@httrack.com>
Date: Sun Sep 27 16:40:46 2026 +0200
Quit on the second Ctrl+C without running the atexit chain (#1817)
`sig_term()` called `exit()`, which runs every atexit handler in signal
context. OpenSSL registers its teardown there, so a second ^C freed
thousands of objects from inside the handler. A `free()` on that path
can deadlock on the malloc lock the interrupted thread already holds.
That would wedge the one escape a user has from a mirror that will not
stop, so `_exit()` replaces it.
`exit()` was also the only thing flushing the log's closing verdict, so
that write now flushes itself. The cache is byte-identical either way.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit d097ef22650f747c1a3fa426ca4390e15dd0e7d1
Author: Xavier Roche <roche@httrack.com>
Date: Sun Sep 27 14:22:27 2026 +0200
Pin that a banned host's FTP slot is left to its worker (#1818)
A slot at STATUS_FTP_TRANSFER belongs to its worker thread, which is
still writing the file. No sweep on the crawl thread may end it.
back_is_live() says so in its comment, and host_ban() does not use it,
gating on a raw `status >= 0` that 1000 passes.
Nothing is broken today, because the next condition saves it by
accident. host_ban() gets a bare host name while an FTP url_adr keeps
its scheme, so the two never match. Normalizing url_adr, or passing a
full URL, would close a file under a running worker.
This pins the outcome instead. The http slot beside it must still be
ended, which is what stops the test passing when host_ban() does nothing
at all.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 6be25c12720c78ca2cebc497f76c604e283ecde7
Author: Xavier Roche <roche@httrack.com>
Date: Sun Sep 27 08:14:06 2026 +0200
Answer for every allocation these four functions dereference (#1814)
Four functions built something out of a fresh allocation and used it on
the next line with no NULL check. The debug resolver behind
HTTRACK_DEBUG_RESOLVE has six of them, and it also handed a half-copied
address chain back as success. openFunctionLib and hts_mutexinit have
one each. CodeQL reached the resolver through
cpp/suspicious-allocation-size, whose own claim about the sockaddr size
is wrong. A 28-byte sockaddr_in6 behind a sockaddr pointer is the
getaddrinfo contract, and the reader bounds itself by ai_addrlen.
hts_mutexinit aborts rather than reporting, because it returns void and
is exported, so reporting would break the ABI. The test reads the source
rather than starving a run: RLIMIT_AS cannot reach allocations this
small, and a probe that tried came back reporting success. The answer
has to name the variable that was just allocated. A first version did
not require that, and a nearby test on another variable then hid a
dropped check.
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 89f0d65d17c74716330c731c82a5faee30153ffb
Author: Xavier Roche <roche@httrack.com>
Date: Sun Sep 27 08:13:54 2026 +0200
Set a file's mode through its open descriptor (#1807)
chmod() looks the name up a second time. In the window between the open
and the chmod, a local process can swap that name for a symlink. We then
set the mode on the symlink's target. fchmod() on the descriptor we
already hold cannot be redirected, and htsbauth.c already uses it.
CodeQL flagged three of these. htscache.c and htssinglefile.c carry the
same shape without an alert, so they move too. The new test then has to
allow only the two htsback.c calls that chmod after a rename, with no
descriptor left.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 6d1fc9e0bfb08329475a0601c84d5d35e962fe89
Author: Xavier Roche <roche@httrack.com>
Date: Sun Sep 27 08:13:49 2026 +0200
Take printf and the log out of the signal handlers, and restore errno (#1811)
`printf` locks stdout and `hts_log_print` allocates, so a ^C landing
while the mirror held either lock could wedge the process. Every handler
now writes with `write()` and puts the interrupted thread's errno back.
The "Exit requested by shell or user" log line still appears. The engine
writes it the next time its loop runs, instead of the handler writing it
from inside the signal.
The DNS self-test also swapped the resolver backend while an abandoned
worker was still reading it, which is the race ThreadSanitizer blamed on
the engine.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 9e9d3cac0a257c7d99c0973f4f0868dae3c77558
Author: Xavier Roche <roche@httrack.com>
Date: Sat Sep 26 23:58:47 2026 +0200
Keep the timeout sweep off a slot an FTP worker owns (#1816)
back_is_live() exists so no sweep in back_wait() touches a slot its FTP
worker owns. The timeout sweep tested a raw `status > 0` instead, so its
exclusion was an accident.
Nothing changes today. A slot only reaches STATUS_FTP_TRANSFER in
back_add()'s direct-FTP branch, which returns before any timeout is
assigned. So the sweep's own `timeout > 0` gate never opened for one.
There is no test because the sweep runs only once a non-FTP slot raises
gestion_timeout (src/htsback.c:3595), which needs st_backstop's
live-connect fixture. One edge remains, a corrupt spool pairing status
1000 with a timeout, where the old code ended the slot and this one
leaves it live.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 3e3a4c8da3fd7c8a5ce6318ce2905cd714069cc6
Author: Xavier Roche <roche@httrack.com>
Date: Sat Sep 26 23:27:16 2026 +0200
Synchronize the FTP slot handoff between the worker and crawl threads (#1809)
An FTP worker published its finished slot with a plain store, and the
crawl thread polled it with a plain load. On arm64 that lets the crawl
thread see the ready status and then read a stale `r.msg`. The store is
now a release, and the one reader that acts on it takes an acquire load.
The publish also moves under the worker-list lock, which matters only
with two engines in one process, because `ftp_workers` is static.
Every other read of that word stays plain, because none of them tells
1000 from 1001, so ThreadSanitizer still reports them. It also still
reports `back[i].info`, which the worker writes throughout a transfer
while the display thread reads it. Fixing either one needs a lock rather
than a barrier.
Test 503 guards the ordering in the source text, because x86 cannot
observe it and no CI leg builds ThreadSanitizer.
---------
Signed-off-by: Xavier Roche <xroche@gmail.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit cb5f9b758870d0b3ab824ecd8e426ff91f38e1e2
Author: Xavier Roche <roche@httrack.com>
Date: Sat Sep 26 23:00:29 2026 +0200
Give every engine header a unique include guard (#1806)
httrack.h reused `HTSTOOLS_DEFH`, htstools.h's own guard, so whichever
header a file included first emptied the other. htsparse.h had no
whole-file guard at all. Nothing changes in today's build, because
htstools.h's declarations sit behind `HTS_INTERNAL_BYTECODE`, which
neither includer defines. The preprocessed output of the four affected
files is byte-identical before and after.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
commit 9ed333c15005ef8fe593057d0cc503e00de6dd9e
Author: Xavier Roche <roche@httrack.com>
Date: Sat Sep 26 18:27:07 2026 +0200
Make a thread see all of a lock another thread just built (#1808)
`hts_mutexlock()` publishes the lock it builds on first use with a
compare-and-swap, but read it back with a plain load. A thread could get
the pointer and still read stale bytes inside the lock. Both reads are
acquire loads now, which changes memory ordering only and leaves every
type and signature alone. That makes `htsthread_init()`'s own
plain-store guard redundant. Deleting it removes a second race, where
two engines in one process could overwrite a lock one of them already
held.
Test 501 races eight threads for 64 unbuilt locks. That half cannot fail
on x86, which never reorders two loads, and a relaxed-ordering mutant
still prints OK. So the test's source guard is what catches an ordering
regression in CI, because no leg runs ThreadSanitizer. A local TSan
build does see it, and on arm64 it is real.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 9e449f5a862a4690c1396be526d1a5ed37803dcf
Author: Xavier Roche <roche@httrack.com>
Date: Sat Sep 26 12:55:32 2026 +0200
Check that a forged cache header line replaces the whole line (#1805)
`corrupt_patch` matched its placeholder as a prefix of the header line,
and never checked the forged replacement's own length. A placeholder
longer than the replacement left its tail on the forged value. A
replacement shorter than `patlen` made memcpy read past the literal and
splice those bytes into the zip. Either way `-#test=cache-corrupt` gave
the same verdict whichever length the fixture used. Both lengths and
both edges of the line are now asserted.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
commit d71e98faca4c9df221370c18e1682f1741d8fe9d
Author: Xavier Roche <roche@httrack.com>
Date: Sat Sep 26 12:06:18 2026 +0200
Fail our own CI build on a compiler warning (#1804)
`--enable-werror` appends `-Werror` to `DEFAULT_CFLAGS`, and the four
pinned Ubuntu build legs pass it, so a new compiler warning reds a PR.
It defaults to off, so no distribution build and no other CI job
changes. Turning it on found three real clang warnings, two of them
fixed here. The third was coucal's own missing format attribute, so it
went upstream as xroche/coucal#47 and this bumps the pin.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 7e4491b89f960a50b04b0c2b2e5aca0b6c13b6ef
Author: Xavier Roche <roche@httrack.com>
Date: Sat Sep 26 10:34:27 2026 +0200
Add one switch-fallthrough marker gcc and clang both accept (#1802)
`HTS_FALLTHROUGH` expands to the `fallthrough` attribute, and
`-Wimplicit-fallthrough=5` stops gcc from taking a comment as the
marker, which clang never accepted. The engine has no unmarked
fallthrough, so the macro has no caller yet. Test 499 compiles one
switch per marker form against the build flags it reads out of
`src/Makefile`.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 61f0566fef29170cfe6effd3abc3b7671e2839a7
Author: Xavier Roche <roche@httrack.com>
Date: Sat Sep 26 10:12:29 2026 +0200
Say why ProxyTrack builds plaintext http:// strings (#1803)
The four `cpp/non-https-url` alerts on `proxytrack.c` are dismissed as
false positives, and the comment records why. The `http://` prefix
completes an archive lookup key rather than a fetch target, and
`proxytrack` imports neither `connect` nor any `SSL_` symbol.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit bd926ac5eef2c651baf9728b25d7a7d5453ab9d1
Author: Xavier Roche <roche@httrack.com>
Date: Sat Sep 26 01:34:22 2026 +0200
Check unsigned wrap in the fuzz build (#1801)
`-fsanitize=undefined` leaves `unsigned-integer-overflow` out, because
the wrap is defined behaviour. It is also how a bounds test written as
`controlled + untrusted < limit` passes when it should fail, which is
the form AGENTS.md bans. The fuzz leg already replays hostile input
through the parsers under clang, so the check goes there and costs no
new job.
Two hashes wrap on purpose. MD5 and MurmurHash3 are third-party and
coucal is a submodule, so they go in an ignorelist. Each path is listed
twice, because make names a source relatively in-tree and absolutely out
of tree. The entity table's FNV is ours, so it says so at its own
definition.
The one remaining `lastI + 2` now tests `lastI` against its unset value.
That changes no output, which is why only this check sees it.
The control: put that guard back to its wrapping form and
`fuzz-htsparse` aborts, in-tree and out. With it in place all 14
harnesses replay clean both ways.
Closes #1796
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 9dfea22d81ae07aee7596ce56ec4e78cfbe9df96
Author: Xavier Roche <roche@httrack.com>
Date: Sat Sep 26 00:55:26 2026 +0200
Turn on seven compiler warnings this code wants (#1798)
Seven warnings this code wants were not on. A full build of master with
gcc 14 is silent on every one of them, so each is a ratchet rather than
a cleanup. The two that were not silent are what #1789 and #1793 already
fixed.
`-Wvla` and `-Walloca` are the ones that matter most here. They refuse a
stack allocation sized by something read off the wire, and the tree uses
neither construct, so nothing can start.
Clang keeps four of the seven and the `AX_CHECK_COMPILE_FLAG` probes
drop the rest, so a clang build gains no new output. `-Wshadow`,
`-Wnull-dereference`, `-Wlogical-op` and `-Warray-bounds=2` were
measured too, at 97, 22, 16 and 2 warnings, and stay out.
Closes #1797
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit a4f99cfbeed26e9261a110a483ffa53324fd6bb8
Author: Xavier Roche <roche@httrack.com>
Date: Sat Sep 26 00:46:14 2026 +0200
Zero an uninitialized stack variable, and bound a trailing char[1] (#1800)
`-ftrivial-auto-var-init=zero` zeroes a stack variable that has no
initializer, so a read before the first write returns 0. MemorySanitizer
finds those reads in CI and nothing mitigates them in a shipped build.
`-fstrict-flex-arrays=3` tells `_FORTIFY_SOURCE` that only a `[]` member
is a flexible array, so a trailing `char x[1]` is bounded as written.
The trap a reviewer would want checked is the first flag reaching the
MemorySanitizer build, where it would hide exactly what that job looks
for. It sits under the same `*-fsanitize=*` guard the fortification
above it already uses, and a `CFLAGS=-fsanitize=memory` configure run
confirms it is dropped.
Measured with gcc 14: text grows 1.4% and the suite is green. Three
interleaved rounds of the htsparse fuzz corpus put the runtime
difference inside the run-to-run spread. Neither flag adds a warning on
gcc or clang.
Closes #1799
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit da33402bd1cab82d6e4ee44eac7030089c0d50f8
Author: Xavier Roche <roche@httrack.com>
Date: Sat Sep 26 00:45:17 2026 +0200
Open the query on a '?' in the first byte of a URL (#1792)
`hts_unescapeUrlSpecial` keeps the offset of the last `%` in `lastI`,
which starts at `(size_t) -1`, so `lastI + 1` wrapped to 0. At `i == 0`
the branch that records a `?` was skipped, so `?a+b` unescaped to `?a+b`
where `x?a+b` gave `x?a b`. A relative link such as `<a href="?q=a+b">`
is the common way to reach it.
A leading `?` now behaves exactly as a `?` one byte in always did. Over
999 million pairs, the new output for `?t` equals the old output for
`x?t` without its leading `x`. Opening the query also cancels a pending
UTF-8 sequence at a `+`, so an output can change by more than one byte.
Closes #1791
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit b38ed9c3adf7e0eb7bfbb9642faf7435da03b844
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 25 23:58:07 2026 +0200
Make ProxyTrack build clean under -Wsign-compare (#1795)
Seven comparisons tested a signed value against a `size_t` or an
`unsigned long`. Four are `send()` and `sendto()` results. The wrap
already took the error branch the code wants, so the cast writes down
what the compiler was doing.
`store.c`'s `fread` site also loses an `(int)` cast on the way into a
`size_t`. `fetchSize` is bounded by an `int` a few lines above, so the
truncation was never reachable. Dropping the cast is what lets the
comparison be typed.
The cast at `proxytrack.c:1291` goes on the `send()` result, not on
`element->size`. That size comes from the cache archive. One whose low
32 bits are all ones would compare equal to a `-1` error, reading a
failed send as a complete one.
Closes #1793
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit cb09f0afedbd36d9856c3a1b89d70a0beea9a3d0
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 25 23:51:45 2026 +0200
Drop the unreachable raw IPv6 form from SOCaddr_copyaddr (#1790)
`SOCaddr_copyaddr` tested `data_size == sizeof(struct sockaddr_in)`
before `data_size == 16` for a raw IPv6 address, and those are the same
number on every target, so the IPv6 branch was dead. No caller reached
it, because every raw-length caller sits under `#if HTS_INET6==0` where
the length is always 4. The doc comment offered the form all the same.
The new `socaddr` self-test pins the forms that do work. It fails when
the two branches are reordered, the shape that would break a live
caller, and `-Wduplicated-cond` catches the dead branch coming back.
Closes #1789
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 25650ea5e3bbef248c203b1dc5a4de475b03e0b3
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 25 23:36:41 2026 +0200
Make the engine build clean under -Wsign-compare (#1794)
Fourteen comparisons tested a signed value against a `size_t` or an
unsigned enum, so the signed side converted to unsigned. A guard
excludes a negative value at every one of them today, but the conversion
is what would hide the next one.
The object code is unchanged in `htscache`, `htsparse` and `punycode`,
and in every `htslib` function but `hts_log_vprint`. The other three
move a comparison from unsigned to signed or widen a loop index, so the
instructions change and the result does not. `hts_log_vprint` casts
`opt->debug` to int rather than retyping `level`, because an enum-typed
`level` reds `-Wswitch` on the switch below it.
The seven sites in `src/proxy/` follow separately.
Part of #1793
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 46c0181357d09965f5157f5b1b0604071876634a
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 25 21:50:48 2026 +0200
Say wontfix in the two Hurd skips, rather than promise a fix (#1788)
#1784 and #1785 are closed as wontfix, so the skip in `471` should not
still carry a `TODO` saying to drop it once #1784 is fixed. A TODO
nobody can discharge is worse than none.
Each skip now names why it is permanent. `471` because every binary we
ship sets its own SIGPIPE handler, so nothing we distribute is exposed
to that defect. `36` because nothing pinned the missing files on the
engine rather than on a one-processor VM.
Comments only, and both tests still run on every other platform.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 826fe49e989c271907702887e7ee5784c3f5da32
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 25 20:37:16 2026 +0200
Build the cache-corruption fixture once instead of three times (#1787)
`corrupt_build`, `corrupt_build_longetag` and `corrupt_build_etag` were
the same seventeen lines three times over. They differed only in the
Etag stored on the victim entry, so the Etag is now an argument and the
six call sites pass it.
That Etag's byte length is what the forged, same-length header line
overwriting it must match. The pairing is now visible at the call site
instead of three hundred lines away.
Two mutants, because a passing test after a fixture refactor proves
little. Ignoring the new argument kills `-#test=cache-corrupt`. Swapping
the 20-byte Etag for the 100-byte one does not kill it. It does not on
master either, so that insensitivity belongs to the test and predates
this change.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit ee86c3756636f8521c4f0fc087e6748f46a6336d
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 25 20:05:41 2026 +0200
Split the engine self-tests into one file per module (#1786)
Every engine self-test lived in one 17,412-line `htsselftest.c`. That
file was the most-edited in `src/`: 232 commits in 90 days, against 119
for the runner-up. 151 of those commits appended a row to the same
registry array, so concurrent branches collided there.
Each module now carries its own `hts<module>_selftest.c` publishing a
`selftests_<module>[]` table. The tables are declared in a shared
prelude, `htsselftest_int.h`, and walked by `hts_selftest()`. Adding a
test now means one file instead of three.
The tests moved unchanged, so review the proof rather than the 17k
lines:
- the chunker that partitioned the file round-trips byte-for-byte
- a check confirms all 405 moved chunks and all 184 registry rows sit in
their target file
- `-#test` lists the same 184 names, with the same usage and description
strings
- `make check` matches the pre-split baseline at 519 total, 504 pass, 15
skip
New code is limited to the prelude, `htsselftest_util.c` for the
fixtures more than one module needed, and the dispatcher walking a list
of tables. `git clang-format` touched only those lines.
`htsparse.h`, `htsmodules.h`, `htsarrays.h`, `htscodec.h`, `htsproxy.h`
and `htszlib.h` carry no include guard. That is why the three older
`*_selftest.c` files now take their includes from the prelude rather
than keeping their own lists. Adding the guards is left for a separate
change.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit cde0be49e582aecd94e1c1a53ca97671a153ac1e
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 25 18:20:31 2026 +0200
Run the test suite on GNU/Hurd in a qemu VM (#1779)
Four package builds broke on the hurd buildds and every one failed a
test rather than the compile (#1668, #1717, #1719, #1775). `cross-arch`
stops at compile, so it sees none of that class. The Debian buildd was
the first machine to run the suite on a Hurd each time, which is after
the upload.
`tools/hurd-vm.sh` fetches Debian's prebuilt hurd-amd64 image, grows the
root filesystem and drops an ssh key into it. It then boots the image
under KVM and runs the suite. The job sits in `tier2.yml`, and also on
any pull request touching the driver, so this one runs it.
The image is pinned by date and checked against the archive's own
SHA512. A boot that never reaches ssh uploads a qemu screendump, because
the image uses the VGA console and the serial log comes back empty.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 91973e5eb4ba5fb1393cc76077f9c41c8e0fd8f5
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 25 15:15:59 2026 +0200
Cover the library's socket writes against SIGPIPE, not just the OpenSSL ones (#1783)
The engine names three mechanisms that stop a broken pipe from killing
whoever embedded it (#1689), but the plain `send()` path carried only
one. GNU/Hurd defines `MSG_NOSIGNAL` and ignores it, so nothing covered
the write there and httrack killed its own process. The library's six
raw sends now hold the mask.
The proof is on Linux rather than on trust. With `HTS_MSG_NOSIGNAL`
forced to 0, `471_engine-sigpipe` passes with this change and fails
without it. The failure reports the same "the engine died on SIGPIPE"
the Hurd run did.
Review caught a bug the first version introduced. `sigpipe_release()`
runs `sigtimedwait()`, which reports `EAGAIN` when nothing is pending,
and two writers in `htsproxy.c` read `errno` on the next statement.
Every fatal error would have read as a would-block. The wrapper saves
`errno` now, and the self-test pins it.
Two tests were asserting something the host supplies rather than
something httrack does. `89` needs `RLIMIT_FSIZE` to stop a write, and
now measures whether it does. `130` needs the host to tell an RST from a
clean end of file, which pfinet does not.
`htsserver` and `proxytrack` still send raw. They are separate binaries
rather than the embedded library. `htslib.h`'s helper does not reach
those two translation units, so they are left for their own change.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 59fc059e253aee9b2a69aab024990da228a03fa8
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 25 15:12:37 2026 +0200
Arm the DNS resolve deadline on a clock NTP cannot step (#1782)
The bounded resolve armed its deadline on the wall clock. A backward NTP
step therefore extended the resolve by the size of the step, and
`--timeout` stopped firing for that long. That is the #606 wedge the
deadline exists to prevent. chrony steps rather than slews past its
`makestep` threshold, one second on a default install, and a step at
boot can be far larger.
`mtime_monotonic()` is the DNS self-test's own clock, promoted to the
engine and now shared with it. The rate limiter's window gets it too.
The DNS cache does not, because `t_dnscache.stored` is compared against
a wall-clock age and a backward step has to expire those stamps.
A test may not step the host's clock, so `01_engine-dnstimeout` reads
the source the way 133 does for gmtime. A deadline armed on one clock
and compared on the other is caught at runtime by the existing
self-test.
Closes #1769
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit d3a62fcd5fe5eb6020352dd65c48f86ed0ee34f3
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 25 15:12:34 2026 +0200
Read a CSS ident the way CSS does before taking url() (#1781)
The CSS scanner ended a name on anything but an ASCII letter, a digit or
`_`. So it read `image-url(x)` as a `url()`, fetched `x` and rewrote the
declaration. CSS Syntax 3 makes `-` an ident code point, so a browser
reads `image-url(` as one function-token and never resolves that
argument. Compass and the Rails asset pipeline both define
`image-url()`, `font-url()` and `asset-url()`. U+1680 and U+FEFF fail
the same way from the other side. JavaScript reads them as spaces, while
CSS puts them inside its non-ASCII ident ranges.
Only CSS gets the wider test. JavaScript subtracts across `-`, so
`image-url("x")` there really is a call to `url()` and stays a link.
Closes #1754
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 282520a08c875b0d94281d540d04da5dd40be7da
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 25 13:58:22 2026 +0200
Open an issue when a scheduled canary fails (#1780)
Ten workflows run on a cron and only one of them says anything when it
fails. `termux-build` went red on 2026-09-24 and the fedora spec canary
twice the week before, and no issue was opened for any of them.
This watches scheduled runs through `workflow_run` and keeps one issue
per workflow, labelled `ci-canary`. A red opens it or comments on it,
and the next green closes it. Only scheduled runs report, because a
dispatch has somebody reading it and a pull-request run already reports
on the pull request.
`workflow_run` only fires for workflows on the default branch, so this
starts working when it merges, not on this PR. `Tier-2` is on the watch
list ahead of #1778, and naming a workflow that does not exist yet is
inert.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit e084565b98876731d278d81b553fa375617a89d6
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 25 13:28:52 2026 +0200
Move the slow CI legs into a Tier-2 workflow (#1778)
Six configure-variant and packaging jobs leave the pull-request path for
a new `tier2.yml`. Tier-2 also absorbs the emulated s390x suite and the
five distro packaging canaries. Measured on this branch, CI goes from 29
jobs to 23 and from 171 job-minutes to 123.
No required status context moved, so no open PR loses a check.
`496_ci-tier2-split.test` pins the required list and reds if one ever
does. It also reds when a workflow absorbed into tier-2 stops being
called, because one with neither a schedule nor a caller never runs
again.
What that buys is fewer occupied runner slots, not a shorter pull
request. A PR still waits about 30 minutes on `libhttrack (x64,
Release)`, which this does not touch. Run tier-2 against a release
branch with `gh workflow run tier2.yml --ref <branch>`. The weekly run
takes master.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 52381d34ec833d162881f65cfcf580b9cc97df7d
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 25 10:04:08 2026 +0200
Clip an over-limit --max-retry-after instead of killing the crawl (#1777)
`back_set_retry_after()` clips a server's oversized `Retry-After` and
logs a notice. `case 'J'` did the opposite with the user's own value,
and panicked above `HTS_MAX_RETRY_AFTER_LIMIT`, so the crawl never
started. WebHTTrack's Max Retry-After box carries no bound, so typing
5000 there cost the whole crawl rather than the option.
An over-limit value is clipped with a notice now. A malformed or
negative one is still refused.
That upper bound was also an accidental overflow guard. `sscanf("%d")`
wrapped 99999999999999 to 276447231, which the bound then caught.
`strtoll` and `ERANGE` take over, so a digit string too wide even for
`LLint` is refused as malformed.
Test 01 pinned the old behaviour, so 3601 and 99999 move from its
refused table to a clipped one. That inversion is the review point,
because #1755 chose refusal deliberately and this reverses it.
Closes #1776
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
commit 6b74afcb475610e8023cc9c3f31da2ad48164b14
Author: Xavier Roche <roche@httrack.com>
Date: Fri Sep 25 05:34:06 2026 +0200
Fix the SIGPIPE self-test on GNU/Hurd (#1775)
`3.50.4-1` failed to build on both hurd buildds, because the SIGPIPE
self-test waits for the peer to answer a write and pfinet never answers.
The self-test now ends its own write side instead, which gives EPIPE
whatever the peer does. The arm still catches a regression, because with
a FIN-only peer on Linux, dropping `HTS_MSG_NOSIGNAL` from `sendc` kills
the process with SIGPIPE.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
![[bug history graph]](https://qa.debian.org/data/bts/graphs/h/httrack.png){kind=link}