vcswatch reports that
this package seems to have new commits in its VCS but has
not yet updated debian/changelog. You should consider updating
the Debian changelog and uploading this new version into the archive.
Here are the relevant commit logs:
commit b026c80bb6fc1366159f94956c78a94adebe31db
Author: Xavier Roche <roche@httrack.com>
Date: Tue Jul 14 20:03:40 2026 +0200
Run the engine test suite on Windows (#577)
* Cap the wildcard matcher's recursion depth (#574)
strjoker() recursed once per pattern segment, bounded only by the length
cap: a hostile filter of 1023 stars reached 2046 frames, ~900KB of stack.
That fits Linux's 8MB but not the 1MB a Windows thread gets, so
-#test=filterbounds died silently on MSVC x64 and Win32 instead of
rejecting the pattern it is meant to reject.
Cap the depth at 256 (real filters use fewer than ten). A cut branch does
not memoize its failure: the same pair may still match when reached at a
shallower depth.
The self-test now asserts the depth reached equals the cap, and the .test
re-runs it under a 512K stack, which segfaults without the cap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Take argv as UTF-8 on Windows, not the ANSI codepage (#573)
httrack.exe is an MBCS build, so the CRT transcodes the UTF-16 command
line down to the machine's ANSI codepage before handing us char **argv.
Non-ASCII arguments are lossy, and anything outside that codepage is
destroyed: a Cyrillic or CJK URL on a Western-codepage box becomes '?'.
Everything else in the engine already treats char* as UTF-8 on Windows -
FOPEN, STAT, UNLINK are hts_*_utf8 wrappers converting to UTF-16 at the
syscall boundary. argv is the one thing that never got the memo.
Decode the real UTF-16 command line at the entry point instead. The
manifest's activeCodePage would fix it in one line, but it needs Windows
10 1903+ and is silently ignored below that, which is precisely the
population running the codepages that mangle non-Latin URLs today.
Windows-only, so no POSIX symbol and no soname change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Make the engine tests pass on Windows
Three of them could never pass there, and none of the three was an engine
bug:
- rcfile wrote .httrackrc, but HTS_HTTRACKRC is "httrackrc" on Windows,
so the engine never found it. Write both names.
- filelist made a file unreadable with chmod 000, which Windows ignores.
Probe whether the mode is enforced instead of assuming, which also
subsumes the root special-case.
- ftp-line was #ifndef _WIN32 for its socketpair(), though get_ftp_line()
itself is portable. Pair over loopback TCP and register it everywhere.
The charset self-tests fed raw non-UTF-8 bytes through argv to probe the
decoders. Those bytes cannot survive any Windows command line, whatever
argv does, so take them as hex - the convention the sniff self-test
already uses - and keep the coverage on every platform.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Run the engine test suite on Windows CI
The ~90 tests only ever ran on Linux and macOS, so a Windows-only
regression was invisible until a user hit it - and the first run of the
offline ones found a third of them failing.
Drive them from Git Bash against the native MSVC httrack.exe. Two things
that silently ruin the run: MSYS rewrites any argument shaped like a POSIX
path, and a URL path is shaped exactly like one, so "-#test=mime /a/b.html"
reached the engine as "C:/Program Files/Git/a/b.html"; and a suite that
degrades to all-skipped would report green having tested nothing, hence
the floor on tests actually passed.
This subsumes the codec and cache self-tests the workflow ran inline. The
.gitattributes pins the test scripts to LF: a converting checkout rewrites
them to CRLF and bash then dies on $'\r' on every line.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Log the launch banner argv without re-encoding it
The banner ran hts_convertStringSystemToUTF8() over each argv element on
Windows, which was right when argv came in the ANSI codepage. argv is
UTF-8 on every platform now, so that pass double-encoded a non-ASCII
argument into mojibake in hts-log.txt. Log the bytes as they are.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Fix the Win32 CLI build: include htscharset.h after winsock2.h
htscharset.h pulls in <windows.h>, which drags in the legacy <winsock.h>
unless <winsock2.h> was included first. In httrack.c it sat ahead of the
net headers, so the x86 build redefined every sockaddr/winsock symbol
(C2011). Move it below htslib.h, where winsock2.h is already in.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Fix two more tests the Win32 CI leg exposed
Both are test bugs surfaced by running on 32-bit Windows for the first
time, not engine bugs:
- strsafe drove its overflow through a char[4]. On a 32-bit build that
equals sizeof(char*), so htssafe's array-vs-pointer heuristic
(sizeof(A) != sizeof(char*), the MSVC path) reads the array as a
pointer and skips the bound: the copy then genuinely overflows and
crashes instead of aborting cleanly. Size the buffer off the pointer
width so the checked path runs everywhere.
- idna's malformed-UTF-8 rejection case passed the bad bytes through
argv. They cannot survive a Windows command line: the UTF-16 round-trip
turns them into valid U+FFFD, which the encoder then accepts. Feed them
as hex so the exact bytes reach the bundled punycode encoder, which
rejects them identically on every platform.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit e0648beff98c1e6aa8efbc1d081c10f540ca2346
Author: Xavier Roche <roche@httrack.com>
Date: Tue Jul 14 19:49:39 2026 +0200
Cap the wildcard matcher's recursion depth (#575)
* Cap the wildcard matcher's recursion depth (#574)
strjoker() recursed once per pattern segment, bounded only by the length
cap: a hostile filter of 1023 stars reached 2046 frames, ~900KB of stack.
That fits Linux's 8MB but not the 1MB a Windows thread gets, so
-#test=filterbounds died silently on MSVC x64 and Win32 instead of
rejecting the pattern it is meant to reject.
Cap the depth at 256 (real filters use fewer than ten). A cut branch does
not memoize its failure: the same pair may still match when reached at a
shallower depth.
The self-test now asserts the depth reached equals the cap, and the .test
re-runs it under a 512K stack, which segfaults without the cap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Pin the matcher depth cap from below in the self-test
The self-test asserted the cap is reached, which a cap set absurdly low
would also satisfy: real filters use fewer than ten segments, so a cap of
5 would break them and still ship green. Assert a 32-segment pattern still
matches, so the cap cannot be lowered past real use unnoticed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit ee1aebd935b2b12026dcab22044e2e1231075e7d
Author: Xavier Roche <roche@httrack.com>
Date: Tue Jul 14 17:58:39 2026 +0200
Harden the ProxyTrack cache readers against corrupt input (#572)
proxy/store.c carried its own copy of htscache.c's old-format (.dat/.ndx) cache readers without the bounds hardening the engine copy got in #568. cache_brstr/cache_rstr trusted a length prefix from the cache file and wrote it into fixed buffers unbounded, and cache_rint/cache_rLLint used an unchecked sscanf result as a size or offset, so a crafted cache overflowed firstline[256] and the PT_Element fields and could wrap a malloc(size+1).
Ports the engine's discipline into the proxy copy: thread the destination capacity through the readers and clamp while consuming the declared length, check every sscanf, allocate the body through a bounded helper, fix an index[index_id] off-by-one, and bound the startUrl/previous_save concatenations. Also casts the coucal void* key at the %s snprintf. The .dat/.ndx format is dead (zip only now), so this is defense-in-depth on the legacy reader; a corrupt-cache test guards it under the sanitizer CI.
commit b4957f0625c25abc266990bd9602b1805c720de5
Author: Xavier Roche <roche@httrack.com>
Date: Tue Jul 14 16:21:51 2026 +0200
Extract the proxy tunneling code into its own module (#571)
* Extract the proxy tunneling code into htsproxy.c
htslib.c is the engine's catch-all "subroutines" file; the two proxy
tunnels living in it (HTTP CONNECT and the SOCKS5 handshake) are one
coherent thing: blocking raw-socket negotiation with the proxy, run from
the same back_wait callsite before TLS. Move both, plus proxy_getline and
the socks5 statics, into htsproxy.c/.h, which also gives the socks5_test_io
self-test seam a home away from the public-ish htslib.h.
Pure code motion: the moved bytes are identical, and hts_proxy_is_socks
stays in htslib.c next to hts_parse_proxy and jump_protocol_const, being a
URL-scheme predicate rather than handshake logic.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Keep the vcxproj source list alphabetical
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit c9d36c82c714df0f4649c6b8bbbdd95131308be6
Author: Xavier Roche <roche@httrack.com>
Date: Tue Jul 14 15:49:59 2026 +0200
Files of 2 GB or more are mishandled on Windows, and on any 32-bit build (#568)
* Report file sizes as 64-bit, so Windows stops mishandling files past 2GB
MSVC's off_t is a 32-bit long in x64 builds as well as Win32, and struct
_stat expands to _stat64i32, whose st_size is that same 32-bit long. So
fsize(), fsize_utf8() and fpsize() truncated twice over: once in the stat,
once in their own return type. A mirrored file of 2GB or more got a wrapped,
often negative size, which fed resume/206 handling, the cache body-size
reads, and readfile2()'s allocation.
Use the 64-bit _stat64 family on Windows and return LLint from the three
size helpers, widening the callers' locals to match. INTsys and HTS_FSEEKO
now cover MSVC too, so readfile2() and fpsize() reach the _fseeki64/_ftelli64
aliases htslib.h already declared. htsback.c's two 'not (off_t)' comments
described this trap but the value feeding that path was still 32-bit.
STRUCT_STAT is in an installed header, so hts_stat_utf8()'s signature changes:
a deliberate Windows-only ABI break, agreed with Xavier. The DLL ships next to
the exe with no soname contract. POSIX is untouched, where off_t is already
64-bit under LFS.
The new -#test=fsize self-test builds a 3GB sparse file and asserts both the
reported size and the 8-byte return width. Forcing the helpers back to a
32-bit return reproduces the Windows failure exactly (-1073741824) and the
test catches it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Report errno when the fsize self-test cannot create or reopen its file
The linux i386 leg fails with 'fpsize is -1', which only says the reopen
returned NULL. Print strerror(errno) on both the create and the reopen path,
and check fclose() so a failed flush stops being silent.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Size files with fseeko/ftello, never ftell: ftell is 32-bit on i386
fpsize() gated fseeko/ftello behind HTS_FSEEKO, which is defined from
'#ifdef LFS_FLAG'. LFS_FLAG is a configure make variable carrying the
-D_FILE_OFFSET_BITS=64 flags, never a C macro, so that test is dead and the
fallback ftell/fseek branch is the one every autotools build compiled. ftell
returns long: 64-bit on x86-64, which hid the bug, but 32-bit on i386, where
it fails with EOVERFLOW past 2GB and fpsize() returns -1.
Call fseeko/ftello unconditionally, as htsback.c already does; MSVC gets them
via the _fseeki64/_ftelli64 aliases in htslib.h. HTS_FSEEKO had no other user.
The dead LFS_FLAG test also leaves INTsys as int on POSIX, capping readfile2()
at 2GB. Widening it changes a typedef in an installed header, so that is left
for a separate change; the FIXME records it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Fix the INTsys widening's fallout, and make the fsize test bite
Widening INTsys to 64-bit on MSVC broke two call sites that assumed it was
an int, both found by an adversarial review of this branch:
- proxy/store.c's cache_rstr_addr() parsed it with sscanf("%d"), writing 4
bytes into an 8-byte object and leaving the high word indeterminate; that
value then sized a malloc() and an fread(). Its sibling cache_rstr() already
used INTsysP. proxytrack.vcxproj builds this file, so it was live on MSVC.
- readfile2()/readfile_utf8() fed a now-64-bit length to malloct() and fread(),
which take size_t: 32-bit on Win32 and i386. The two truncate independently,
so a file of 4GiB-1 gave malloc(0) with a 4GiB read into it. Route the size
through llint_to_size_t() and fail closed when it does not fit.
The self-test only exercised fsize_utf8(), so reverting fsize()'s _stat64 hunk
left it green, and fsize() is what sizes file:// bodies and --list. It now
checks both variants, the absent-file -1 contract, and uses 5GB rather than
3GB: a 32-bit truncation of 3GB is negative and obvious, of 5GB it is
1073741824, a plausible size. NTFS needs FSCTL_SET_SPARSE to leave the hole
unallocated; POSIX gives it for free.
Windows CI now runs the cache, cache-corrupt and fsize self-tests. The cache
fixes in #567 cannot be verified on POSIX at all, where fconv() is a no-op.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Check the sscanf return in proxytrack's cache reader
cache_rstr() and cache_rstr_addr() declared an uninitialized INTsys and never
checked whether sscanf matched. On a truncated or malformed cache entry the
length stays indeterminate, and it then sizes an fread() and, in the addr
case, a malloc(). Initialize it and require a match, so a bad entry reads as
length 0.
Flagged by CodeQL (cpp/missing-check-scanf) once the widening touched that
line; the alert predates it on master, and its sibling had the same hole.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit 1fa018446883c70c620a255e66ca2e29d87fc932
Author: Xavier Roche <roche@httrack.com>
Date: Tue Jul 14 15:38:47 2026 +0200
SOCKS5 proxy support (#570)
* Tunnel crawls through a SOCKS5 proxy (RFC 1928/1929)
-P socks5://[user:pass@]host[:port] now negotiates a SOCKS5 tunnel to the
origin instead of speaking HTTP to the proxy. The handshake runs on the raw
socket right after connect, so it covers plain http as well as https (TLS then
runs end-to-end over the tunnel, no CONNECT); the origin request stays
origin-form, and the proxy credentials ride the RFC 1929 sub-negotiation rather
than a Proxy-Authorization header.
The origin hostname is sent as-is (ATYP=domain): the proxy resolves it, like
curl's socks5h, which is what makes .onion and split-horizon names work and
keeps the crawl from leaking DNS. ftp:// links fail under a socks proxy rather
than silently dial the origin direct.
Reply frames are drained exactly per ATYP, since the socket is shared with the
origin stream: one byte too few and the leftovers corrupt the first TLS record.
-#test=socks5 asserts that byte count against scripted replies, plus the
truncated/hostile cases; 52_local-socks5.test crawls http and https through a
local SOCKS5 server that only answers for a .invalid name, so a locally-resolved
origin could not reach it.
Closes #563
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Don't re-run the SOCKS5 handshake on a reused keep-alive socket
back_wait re-enters the connect-completed block for a keep-alive socket
revived by back_trylive (plain http; https is covered by the ssl_con guard).
The socket is already tunneled, so re-sending the SOCKS greeting injected
\x05\x01\x00... into the established origin stream and corrupted every request
after the first. Gate the handshake on !keep_alive, which back_connxfr carries
onto the revived slot and which is zero on a fresh connect.
52_local-socks5.test now crawls a keep-alive origin (advertising max>1) with
-c1: the whole plain-http crawl rides one tunnel, exactly one handshake, every
subpage intact. Without the gate the reused stream desyncs and the crawl hangs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Use scheme-less origins in the SOCKS5 self-test (CodeQL cpp/non-https-url)
The scripted handshake strips the scheme and takes the TLS flag as a parameter,
so the "http://origin.test" fixtures were inert cleartext-URL literals that
tripped CodeQL. Pass the bare authority instead; the parsing is identical and
the real scheme handling is covered end-to-end by 52_local-socks5.test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit 79e46eecebc75b58dbcfcc576ec33784f98223c5
Author: Xavier Roche <roche@httrack.com>
Date: Tue Jul 14 14:29:12 2026 +0200
Store the cache self-tests' X-Save verbatim, and feed the sniff BOM as hex (#567)
The cache self-tests built the X-Save path with fconcat(), which rewrites
every '/' to '\' on Windows -- including the path_html_utf8 prefix the
reader matches on. cache_readex then took the pre-3.40 compat branch and
re-rooted the name into a path that does not exist, so -#test=cache's
disk-fallback read and -#test=cache-corrupt's headers-only and in-memory
cases all failed on MSVC builds. The engine keeps save names in '/' form and
DOS-ifies only at the syscall boundary; the self-tests now do the same.
The sniff test passed a UTF-8 BOM as raw bytes through argv, where Windows
routes it via the ANSI codepage and U+FEFF has no mapping, so it arrived as
'?'. Use the hex: body form the self-test already accepts.
Both are self-test bugs: neither path is reachable from the crawler, which
never sniffs argv and never DOS-ifies a stored save name.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit 436ac5379312c66aa342dd24f7f0e1c931dbd1ac
Author: Xavier Roche <roche@httrack.com>
Date: Tue Jul 14 14:11:57 2026 +0200
Parse -P proxy URLs scheme-aware, recognize socks5 schemes (#566)
* Parse -P proxy URLs scheme-aware, and recognize socks5 schemes
The -P argument parser scanned backward for the port ':' from the end of the
string with no notion of a scheme, so a scheme-prefixed proxy with no explicit
port ("-P http://proxy") mistook the scheme's own colon for a port separator:
the name came out as "http" with a garbage port. A port or a "user:pass@" prefix
happened to shield the scan, so this only bit the no-port case.
Extract the parse into hts_parse_proxy(), which skips any "scheme://" before
hunting the port, so the scheme colon is never read as a port. Teach
jump_protocol_const the socks5/socks5h schemes (so the proxy host resolves with
the scheme stripped) and default the port by scheme (1080 for socks, 8080
otherwise). This is the parsing groundwork for SOCKS5 support (#563); the
handshake lands separately.
Add a -#test=proxyurl self-test and tests/01_engine-proxyurl.test covering bare
hosts, user:pass, the scheme-colon regression, and the socks defaults.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* hts_parse_proxy: scan from one-past-end, avoid forming arg-1
The backward port scan started at arg+strlen(arg)-1 and decremented while
a >= authority, so on any argument with no ':' or '@' in the authority (every
bare-host proxy, and -P "") it stepped a to arg-1: a pointer formed before the
string, which is undefined behavior. It is not a memory-safety bug (never
dereferenced, no wrap, so UBSan stays green), but it is latent UB reachable from
-P.
Scan from one-past-end and test a[-1] with a > authority instead, so no pointer
below the string is ever formed. Guard name_size == 0 against a future
small-buffer caller (the name_size - 1 truncation would otherwise underflow).
Add a userinfo-free socks explicit-port case and a lone ':' boundary case to the
self-test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* proxyurl test: percent-encoded specials and UTF-8
Pin that the host/port split is byte-transparent: a percent-encoded '@' (%40) or
':' (%3A) in the userinfo stays encoded and never mis-splits host or port, a
second raw '@' is handled by the last-'@' host rule, and UTF-8 bytes in the
username and host label pass through intact. Decoding happens later at auth time,
not in the parser. Raw UTF-8 CLI args match the existing 01_engine-idna.test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit bc4b2c7b8eaddacbec5f62003a50a408fdcff760
Author: Xavier Roche <roche@httrack.com>
Date: Tue Jul 14 13:10:28 2026 +0200
Keep the mirrored file when a coded re-fetch fails to decode (#560)
* Keep the mirrored file when a coded re-fetch fails to decode
An --update re-fetch of a content-coded URL decoded straight into url_sav: the
live mirror was truncated (filecreateempty) before the decode was even attempted,
so a truncated gzip, a corrupt br/zstd stream, a decode-size overrun or a coding
we have no decoder for destroyed the good copy we were merely refreshing. The
uncompressed path has been rollback-safe since #522, but its backup is explicitly
skipped for coded bodies, so nothing covered them.
Decode into a temporary file instead and only commit it over the mirror once the
decode succeeded; on failure url_sav is never touched. The failure path also has
to filenote() the surviving copy, or the end-of-update purge (in old.lst, absent
from new.lst) would delete the very file the decode took care not to overwrite.
Closes #557
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Never drop the decoded body when the mirror replace fails, and probe the good path
replace_file() unlinks the destination and retries when the first rename is
refused (Windows never clobbers). If that retry also fails, the previous copy is
gone, so deleting the decoded temp as well destroyed the last copy of the body:
keep it and log where it went, the way the backup restore does. That restore is
the same remove-then-rename, so it now shares the helper.
The test grew the missing positive control: freshdisk.bin decodes on both passes
on the direct-to-disk path, so the update pass renames a decoded temp over an
existing mirror file, which no case exercised. The leftover-temporaries sweep in
local-crawl.sh now covers the .z/.u decode temps for every crawl test, not just
the three names this one asserted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Restore the 0644 mode on a decoded direct-to-disk file
Committing the decode by renaming the temp bypassed filecreate(), which is what
chmods the mirrored file to HTS_ACCESS_FILE. Under a restrictive umask a coded
binary landed 0600 while every other mirrored file stayed 0644, so a mirror
served from a web root or shared with a group silently lost read access. chmod
it at the rename, as the plain direct-to-disk path already does.
The test now crawls under umask 077 and asserts the mode, via a --file-mode
assertion in local-crawl.sh; without the chmod it fails with 0600.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Pin the codec-update assertions to the URLs and to fresh content
The exact error count did not say which links failed: an engine that logged one
URL twice while another failed silently still counted three and passed. Match the
three decode errors per URL instead. The content checks only looked for the new
marker, so a fix that appended the decoded body to the old file rather than
replacing it would pass too; assert the pass-1 marker is gone. Also assert the
mode on a kept file, not just on the two rewritten ones.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit 4fafe283b416beeb6e11504b0d7186731d148542
Author: Xavier Roche <roche@httrack.com>
Date: Tue Jul 14 12:42:19 2026 +0200
Store the .rc blobs LF-normalized, as .gitattributes declares (#561)
.gitattributes marks *.rc as "text eol=crlf": the blob is LF, the checkout is
CRLF. The two resource scripts added in #558 went in with CRLF blobs, so git
renormalizes them on every add and every checkout reports them as modified.
Renormalized with git add --renormalize; version.rc was already correct. The
working-tree files stay CRLF, which is what the Windows toolchain reads.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit 9ec5a3a067c663351a04b85a429633e46331ef5d
Author: Xavier Roche <roche@httrack.com>
Date: Tue Jul 14 08:26:20 2026 +0200
Decode brotli and zstd content codings (#556)
* Decode the br and zstd content codings, and bound every decode
httrack advertised gzip and deflate only, so it took the oldest coding on
offer while every browser negotiates brotli (46% of CDN traffic) or zstd.
Worse, the response side had no codec identity at all: Content-Encoding was
collapsed into a boolean and hts_zunpack guessed the framing from the body
bytes, so a br or zstd body sent unsolicited (mis-keyed Vary caches do this)
fell through the identity path and was saved as the page, coded bytes and all.
Content-Encoding now maps to a codec (htscodec.c), and the decode dispatches on
it: brotli and zstd get streaming decoders, a known coding we cannot undo fails
the fetch instead of saving garbage, and an unrecognized token still means
identity, because servers do put charsets in that header. br and zstd are
advertised over TLS only, as browsers do.
On any decompression failure the undecoded body is now dropped rather than left
in memory for the writer to commit as the page. That closes the coded-bytes-as-
page hole for the new unsupported-coding path, and with it a latent leak on the
gzip path, where a body that failed to inflate (a truncated stream) was written
to disk verbatim.
Every decode is now bounded: 4096x the coded body, floor 1 MiB, ceiling INT_MAX.
Nothing capped the decoded size before, which was survivable while deflate was
the only codec (it cannot pass 1032x) but not with brotli and zstd, which reach
a million to one. The zlib accumulator also moves to LLint, so a body over 2 GiB
fails instead of overflowing an int.
libbrotlidec and libzstd are optional at configure time and required on Windows,
where libhttrack.dll links them from vcpkg and CI runs the self-tests to prove
the decoders are in.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Fix copyright year on the new codec files (2026, not 1998)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* tests: probe the direct-to-disk decode-failure discard, and assert absence
The wire test only exercised the in-memory discard path (bad.html is HTML).
Add a non-HTML body under an unsupported coding (bin.dat) so the is_write
branch is covered too, and assert both are absent from the mirror rather than
grepping a file that the fix now removes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Factor the budget-enforced write shared by the brotli and zstd decoders
The two streaming decoders duplicated the decoded-size budget check, the
security-critical part of each loop. Pull it into one codec_sink helper so the
bomb ceiling lives in a single place. Behavior-preserving; the codec self-test
(exact decoded bytes, truncation and bomb rejection) proves equivalence.
Also compile the streaming-decode body only when a streaming codec is built,
which drops an unused-variable warning in the --without-brotli --without-zstd
build.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit cd38e6e68dd5f446745722085adcac089bfba953
Merge: 7d9d4ed e692e56
Author: Xavier Roche <roche@httrack.com>
Date: Mon Jul 13 20:51:23 2026 +0200
Merge pull request #559 from xroche/fix/robots-default-comment
Put the robots.txt "(default)" label on the option that is the default
commit e692e565921dfb96b508053fc71b3dad03f37b53
Author: Xavier Roche <roche@httrack.com>
Date: Mon Jul 13 20:37:03 2026 +0200
Put the robots.txt "(default)" label on the option that is the default
The enum said HTS_ROBOTS_SOMETIMES was the default. It is not: hts_init_opt
sets opt->robots = HTS_ROBOTS_ALWAYS (htslib.c:5990), and SOMETIMES is what
the -s option selects when the user asks for it (htscoremain.c:1234).
So HTTrack obeys robots.txt fully by default, and the header said the opposite.
Anyone reading it to answer "does this crawler respect robots.txt" got the
wrong answer from the one place they would look.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
commit 7d9d4ed8904ec2c9c52cac8426a498fe48326222
Author: Xavier Roche <roche@httrack.com>
Date: Mon Jul 13 20:20:02 2026 +0200
Give the Windows binaries a version resource (#558)
libhttrack.dll and httrack.exe have never carried a VERSIONINFO: no product
name, no version, nothing. Right-click either one on Windows and Properties
has no Details worth reading.
That becomes a hard problem now that we are signing them. SignPath enforce
file metadata restrictions: every binary in a release must name the same
product and the same product version, and a binary that names nothing cannot
satisfy the rule. It fails the signing request, on release day, rather than
the build.
So add src/version.rc, and two wrappers that set the parts which differ per
binary. The version is spelled out there as numbers, because a VERSIONINFO
needs 3,49,12,0 and cannot take a string apart, and because htsglobal.h drags
in C declarations that a resource compiler has no business parsing.
That leaves the version written twice, which is how a version resource quietly
starts lying about which release it is. tests/01_engine-version-macros.test
fails the build if version.rc and htsglobal.h ever disagree, on any of the four
values, and on the product name too. I checked that it fires rather than
assuming: drift the numbers and it fails, restore them and it passes.
The product name is HTTrack Website Copier, on all of them. WinHTTrack is the
name of the Windows GUI, not of the command line program or of the engine.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit 573167e8dd93532206c031be5aadcd833b744dcf
Author: Xavier Roche <roche@httrack.com>
Date: Mon Jul 13 10:50:23 2026 +0200
Port the CLI, webhttrack and proxytrack to VS2022, and retire the VS2008 project files (#555)
Follow-up to #554, which ported libhttrack. The CLI, webhttrack and proxytrack were still VS2008 .vcproj, which v143 cannot open, so WinHTTrack.sln still could not be opened as a whole. They are recreated as v143 .vcxproj (x86+x64, /MD, vcpkg, no hardcoded C:\Dev paths), and the legacy .vcproj retire along with the VC6-era .dsp/.dsw beside them. EXTRA_DIST now ships the .vcxproj and vcpkg.json, which #554 had missed, so the tarball no longer carries a project file with no manifest beside it.
Building them turned up two more bugs no Windows compiler had seen. httrack.c's main() calls WSAStartup and WSACleanup itself, so the CLI needs Ws2_32 of its own. proxytrack's getGMT reads the POSIX global timezone, which the UCRT only exposes as _timezone; the existing BSD branch grew an MSVC arm rather than switching to timegm, which would have changed the date arithmetic.
CI builds all four projects and checks the binaries land where WinHTTrack expects. It also asserts, through dumpbin, that libhttrack.dll and httrack.exe import the same vcruntime140.dll and are built for the expected machine: a CRT split across that boundary links cleanly and only crashes later, on the first cross-heap free. The assertion runs against cmd.exe first, which links the old msvcrt, so a detector that matched anything would fail the job instead of quietly blessing a mismatch. A malformed project file is now caught by an XML check in the Linux lint job rather than by a Windows runner minutes in.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit e26a5f5c6a05f2313a289cc1de524319a0d9fcf5
Author: Xavier Roche <roche@httrack.com>
Date: Mon Jul 13 00:00:39 2026 +0200
Build libhttrack on VS2022 with OpenSSL 3.x from vcpkg (#554)
The MSVC build had no CI and had rotted: libhttrack.vcproj is a VS2008 file that v143 cannot open, pinning OpenSSL 1.0.1j and zlib from hardcoded C:\Dev paths. WinHTTrack, ported to VS2022, links libhttrack.dll and takes OpenSSL and zlib through it, so the GUI could not link at all: nothing produced a modern libhttrack.lib.
Adds src/libhttrack.vcxproj (v143, x86+x64, /MD dynamic CRT, DLL) beside the legacy .vcproj, which stays until the CLI, webhttrack and proxytrack are ported. OpenSSL 3.x and zlib come from vcpkg on dynamic triplets, so an OpenSSL CVE is a DLL swap rather than a rebuild. The source list comes from libhttrack_la_SOURCES rather than the old .vcproj, which predated htssniff.c and the selftest files and pulled in httrack.c, the CLI main(). Also adds a Windows CI job, which the repo never had; that gap is how the project file drifted this far.
The C needed no OpenSSL port, but the build did surface three bugs no Windows compiler had ever seen:
- MSVC has no fseeko/ftello/timegm. They are aliased to _fseeki64/_ftelli64/_mkgmtime in htslib.h, which is not an installed header: an object-like macro of that name in an installed one would rewrite the identifier in a consumer's own declarations and C++ member names.
- Winsock's getaddrinfo/freeaddrinfo are __stdcall, while hts_resolver_backend declared its function pointers plain. That compiles on x64, which has a single calling convention, and fails on x86. All three backends now carry HTS_RESOLVER_CALL. hts_dns_set_resolver_backend is not exported (hidden visibility), so despite htsnet.h being installed this is not an ABI change and owes no soname bump.
- The --continue resume path cast a 64-bit offset to off_t, which is 32-bit on MSVC, for both the truncate and the seek. A resume past 2GB wrapped, and _chsize_s then destroyed the partial file.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit 627ae2b04331ca9df54100c336558d857dcc8f0e
Author: Xavier Roche <roche@httrack.com>
Date: Sun Jul 12 18:10:57 2026 +0200
tests: assert the strjoker budget fired, not just that work stayed bounded (#553)
The filterbounds self-test added in #551 asserted `steps < 10*maxsteps`, which
catches a budget whose enforcement is dropped (the counter still climbs to
~1.26e9). But the counter and the enforcement are the same statement, so
deleting the budget line outright leaves steps at 0: `0 < 10*maxsteps` holds and
the probe passes. Only the trailing strjokerfind call then catches it, by
hanging into a harness timeout, which is exactly what #551 set out to remove.
Add the floor `steps > maxsteps`: the budget must push the counter past the cap.
Deleting the budget now aborts the test in milliseconds instead of timing out.
Verified both regressions (line deleted; enforcement dropped) trip an assertion.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit 71ab3574ef86cca1a39b11ffc022830cdc7dfbf2
Author: Xavier Roche <roche@httrack.com>
Date: Sun Jul 12 18:09:56 2026 +0200
Remove the obsolete Java-applet parser and dead SWF module remnants (#552)
* Remove the obsolete Java-applet .class parser and dead SWF vestiges
Java applets stopped running in every mainstream browser years ago (NPAPI
dropped by Chrome in 2015 and Firefox in 2017; the JDK browser plugin gone
since JDK 11), so htsjava -- a hand-rolled parser of hostile .class bytecode
fetched off the network -- chased dependencies no live site serves while
carrying real attack surface for no functional gain. The SWF module was
already gone; only vestiges remained (a dead libhtsswf.so.1 dlopen entry and
help-text examples).
htsjava was a dlopen plugin, never linked into libhttrack, so its removal
leaves libhttrack's ABI untouched: libhttrack.so.3 is unchanged and the
libhttrack3/-dev packages just stop shipping libhtsjava.so.3*. That is a
plain file drop dpkg removes on upgrade -- no package rename, no
Replaces/Breaks, and nothing ever linked the library (it was dlopened), so
there are no reverse dependencies. The parsejava/-j option stays; it also
gates JavaScript parsing.
Drops the plugin sources, the build/config/vcproj wiring, the java self-test
and its two engine tests, and the now-unused hts_count_fits helper (whose
only caller was the plugin), and regenerates the man page. The <applet> URL
rewriting in the HTML parser and the .class codebase bookkeeping are left
intact, so applet pages are still mirrored as plain files.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Scrub the removed java/swf modules from the shipped HTML docs
Follow-up to the htsjava/SWF removal: the pre-generated docs in html/ (shipped
in httrack-doc) still named the gone modules. Update the two stale --help
mirrors in httrack.man.html by hand (a full groff regen would rewrite the whole
file under a newer groff) and drop the htsjava.c plugin-example reference from
plug.html.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Finish scrubbing the java-class docs across the guides and man source
The -j option still exists (it drives JavaScript parsing), so every "parse
Java Classes" summary in the HTML guides becomes "parse scripts", and the
obsolete .class-parsing descriptions and java-applet troubleshooting entries
are reworded to drop the removed capability. The "Some java classes may not
find..." engine-limit line is dropped at its source (README, from which
makeman.sh regenerates the man LIMITS section), and httrack.1 regenerated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit 076127ddea63203218120dfeb92dd39dd93a4280
Author: Xavier Roche <roche@httrack.com>
Date: Sun Jul 12 15:59:38 2026 +0200
tests: make the strjoker work-budget test deterministic, not timeout-reliant (#551)
The filterbounds self-test's step-budget case (900 stars, subject cut to 900)
stayed under the 2,000,000-step budget: the #501 failure memo alone bounded it,
so the budget never fired and the `== NULL` assertion held with or without it.
Only a harness timeout would have caught the budget's removal.
Size the star-heavy dead-end to the length cap (1023 stars, 2048-char subject),
where the memoized matcher runs ~1.26e9 steps (~6s) unbounded, and assert
through a new test-only strjoker_steps() that the work stays near the cap.
Deleting the budget now trips the assertion in milliseconds instead of timing
the suite out.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit dced0208e1b3caa466f49aea8c010c6c160930c7
Author: Xavier Roche <roche@httrack.com>
Date: Sun Jul 12 15:21:21 2026 +0200
Bound the cookie default-domain copy against a hostile over-long host (#550)
treathead seeds each Set-Cookie's default domain from the request host with a
strcpybuff into domain[256]. A 256+ byte host, reachable via a redirect
Location (sized HTS_URLMAXSIZE), overflows the fail-safe copy and aborts the
whole mirror. Drop such a Set-Cookie instead, matching the existing too-long
handling of the explicit domain= attribute.
Adds a cookies self-test driving treathead with a 599-byte host (aborts
without the guard) plus a normal-host control.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit 6e34c30a7c22cbbc8919e1813d4fd4607b889f7c
Author: Xavier Roche <roche@httrack.com>
Date: Sun Jul 12 14:37:52 2026 +0200
Prune signals that can't fire from the webhttrack and engine cleanup handlers (#549)
* Prune signals that can't meaningfully fire from the cleanup handlers
Follow-up to #547. webhttrack's trap listed the whole CPU/program-fault
class (ILL TRAP ABRT BUS FPE SEGV, plus the now-removed STKFLT) alongside
ALRM/XCPU/XFSZ. None of these do anything useful in a shell wrapper: bash
doesn't raise them, and a crash in the htsserver child delivers SIGCHLD to
the parent, not the fault signal. Keep the signals that actually ask the
wrapper to quit and warrant tearing the server down: HUP INT QUIT PIPE TERM.
All POSIX, so no more macOS/BSD trap-install breakage.
In the engine, drop the SIGSTKFLT crash handler for the same reason: Linux
marks SIGSTKFLT unused (legacy x87 coprocessor fault) and never raises it;
a genuine stack overflow arrives as SIGSEGV, which sig_fatal already
catches. The SEGV/BUS/ILL/ABRT handlers stay -- those are the real crash
reporter.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Drop the leftover SIGSTKFLT comment
A comment documenting a signal we deliberately don't handle is noise.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit eac13ea24888a53d8c8d7ce3e3c10da9e98989c9
Author: Xavier Roche <roche@httrack.com>
Date: Sun Jul 12 13:55:40 2026 +0200
webhttrack: let a configured browser win over Darwin's open -W (#546)
The Darwin block pre-set BROWSEREXE="/usr/bin/open -W" before the browser
search loop, whose per-step "test -n $BROWSEREXE && break" then exited on the
first iteration. The search was dead on macOS: $BROWSER and any installed
browser were ignored and webhttrack always handed the URL to open -W, which
opens the system default and blocks until the whole app quits.
Keep open -W only as a fallback after the search, so a configured browser wins.
Closes #544
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit dc3e485f92cddf72936cad3ca537926a3cc41211
Author: Xavier Roche <roche@httrack.com>
Date: Sun Jul 12 13:54:42 2026 +0200
webhttrack: drop Linux-only SIGSTKFLT from the cleanup trap (#547)
SIGSTKFLT does not exist on macOS/BSD, so bash rejects the trap command there
and the signal cleanup may fail to install. Interrupting webhttrack (Ctrl-C)
could then leave htsserver running and the temp server file behind. Drop
STKFLT; the remaining signals are all POSIX.
Closes #545
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit fac84f1631df6aca0e2f4f7f7501d0a613dbb5d2
Author: Xavier Roche <roche@httrack.com>
Date: Sun Jul 12 13:52:31 2026 +0200
ci: harden the webhttrack smoke test and drop its leaky watchdog (#548)
Follow-up to #540. A narrow-charter review found two issues in the smoke test
as merged:
- The content check matched only the HTTrack brand string, which is a literal
in every page header, so a truncated or degraded template served 200 would
still pass. Also require the step-2 form action, which such a page lacks.
- The hard watchdog subshell was un-redirected, so its orphaned sleep kept the
CI step stdout open ~35s after every run. The poll loop and teardown already
bound the run, so drop the watchdog and SIGKILL webhttrack if it ignores TERM.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit c32d28693800e93056536c1287bb09f271991acb
Author: Xavier Roche <roche@httrack.com>
Date: Sun Jul 12 13:18:06 2026 +0200
ci: smoke-test webhttrack on macOS (#540)
* ci: smoke-test webhttrack on macOS
Nothing exercised the WebHTTrack GUI launcher or its htsserver backend, and
webhttrack has a Darwin-only browser path (open -W). Install into a temp prefix
on a macOS runner and assert webhttrack brings up htsserver and serves the UI
(fetched via a stub browser that shadows the first name in webhttrack's list).
The UI is served ISO-8859-1, so the content check uses grep -a.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* ci: reap htsserver in the webhttrack smoke test
webhttrack backgrounds htsserver and leaves it running; killing only webhttrack
orphaned the server, which held the macOS CI step open until timeout. Kill
htsserver explicitly (scoped to the test prefix) in teardown and on exit, and
dump webhttrack.log so a real failure is diagnosable.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* ci: bound and instrument the webhttrack smoke test
The smoke step hung on the macOS runner. Add a hard watchdog (macOS has no
timeout(1)), drop the blocking wait for a bounded liveness poll, and log each
step plus webhttrack.log so a macOS-specific stall is visible instead of
running to the job timeout.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* ci: route the webhttrack smoke past macOS open -W
On Darwin webhttrack hardcodes the browser to "open -W", which launches a real
GUI browser and blocks headless (the macOS smoke stalled here). Shadow uname so
webhttrack takes the generic path and uses the stub browser; htsserver and
webhttrack path resolution still run for real on macOS.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit 5a7ab7a653afa534acf175532a23254b408b2f60
Author: Xavier Roche <roche@httrack.com>
Date: Sun Jul 12 13:00:41 2026 +0200
ci: alias 127.0.0.2/.3 on macOS lo0 for the connect-fallback test (#543)
macOS configures only 127.0.0.1 as loopback, so 19_local-connect-fallback's
dead 127.0.0.2/.3 addresses have no fast-refuse path: the connect intermittently
stalls to the timeout and logs "Connect Time Out", which the zero-errors
assertion rejects (flaked ~1/6 on the macOS runner). #531's --timeout/--retries
bound the stall but not the spurious error. Aliasing the addresses onto lo0
makes them refuse instantly like Linux, so the fallback runs cleanly.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit 5756a9830cec0e1ca29c1f64f0908f06a6d22e0b
Author: Xavier Roche <roche@httrack.com>
Date: Sun Jul 12 12:59:19 2026 +0200
Bound strjoker recursion depth and work to stop OSS-Fuzz stack-overflow and timeout (#542)
The wildcard filter matcher recurses one frame per star group, so a 16000-star
pattern overflows the call stack. And strjokerfind re-runs the polynomial
matcher at every subject position, so a star-heavy pattern against a long
subject runs for minutes. The #501 failure memo made a single match polynomial
but bounds neither recursion depth nor the outer sweep.
Cap matcher input at HTS_URLMAXSIZE*2 (the length the engine already bounds
URLs to), which holds recursion depth far below the overflow point and leaves
real crawls unaffected. Add a work budget that strjokerfind shares across its
whole scan so the per-position cost no longer multiplies. Both limits fail the
match safely once exceeded. A filterbounds self-test covers each and fails
without the caps.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit 28c11d55eb19a04411eb0d2cd5274833ac00ec89
Author: Xavier Roche <roche@httrack.com>
Date: Sun Jul 12 12:59:05 2026 +0200
Fix one-byte fil[] overflow in ident_url_absolute on hostless ?-URLs (#541)
OSS-Fuzz caught an abort in ident_url_absolute: strncat_safe_'s bounds
check fired on a one-byte overflow of the 2048-byte fil[] path buffer.
URLs of 2048 bytes or more are rejected up front, so the path suffix is
at most 2047 bytes; but when it has no leading slash, one is prepended
before the copy, pushing the result to exactly 2048. A hostless URL
starting with '?' and 2047 bytes long reaches that copy before the
empty-host check at the end rejects it. Subtract the possible slash from
the limit and return -1 for those inputs, leaving valid URLs untouched.
The identabs self-test feeds the 2047-byte trigger and asserts a clean
-1, plus two valid URLs still parse.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit b5f28f6eb54bdc05e5fa5d0e36a7cb5ba4ce78d7
Author: Xavier Roche <roche@httrack.com>
Date: Sun Jul 12 12:24:42 2026 +0200
Un-isolate bigcrawl on macOS and bump it to -c16 (#539)
PR #538's larger listen backlog fixed the real cause of the macOS bigcrawl
file-count flake: Python's default 5-slot accept queue overflowed under parallel
CPU contention, and macOS drops SYNs on overflow. With that gone, macOS no
longer needs the second-pass "run bigcrawl alone" workaround, so it runs the
full suite in parallel like Linux.
The crawl can also push more connections now: -c8 to -c16 shaves ~14% off its
wall time (33.4s to 28.6s locally; -%c100 caps the rest) and stresses the engine
harder. 128 backlog slots leave ample room for 16 connections.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit 3d65bb2e3bbc677bc3627e074a635bef46ca6ba6
Author: Xavier Roche <roche@httrack.com>
Date: Sun Jul 12 11:38:20 2026 +0200
Raise the test server's listen backlog so -c8 bigcrawl stops flaking on macOS (#538)
local-server.py used ThreadingHTTPServer's default request_queue_size of 5.
macOS/BSD drop SYNs once the accept queue overflows; Linux is lenient. Under a
parallel make check the single-threaded accept loop is starved for CPU, drains
the 5-deep queue slowly, and an -c8 bigcrawl burst overflows it. The
36_local-bigcrawl file count then flaked (e.g. 361 -> 283) because --retries=0
turns each dropped connection into a lost file. Raise the backlog to 128 for
ample headroom; -c8 is unchanged.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit 68897b09ce6f0414cc9f8b7b1a53a5ab9567f923
Author: Xavier Roche <roche@httrack.com>
Date: Sun Jul 12 09:44:40 2026 +0200
Re-fetch a self-redirect cookie wall instead of dropping it (#537)
* Re-fetch a self-redirect cookie wall instead of dropping it (#15)
A page that 302-redirects to itself only to set a cookie (a consent or session
"cookie wall") was lost: the self-redirect guard treated the loop as a crazy
server and never re-issued the request, so the real content behind the cookie
was never fetched.
The redirect's Set-Cookie is already folded into the shared jar before the
guard runs, so a re-issue would carry it. In the delayed-type loop
(hts_wait_delayed) snapshot the jar before the request; when a self-redirect
changed it, evict the cached fast-header for that URL and re-fetch once with the
new cookie. Termination is bounded: the jar stops changing once the cookie is
satisfied (real walls resolve in two requests), and the existing loops<7 cap
backstops a server that mints a fresh cookie on every hit.
Under the default HARD delayed-type mode every URL routes through this loop, so
both unknown-ext (wall.php) and known-ext (wall.html) walls are covered. A wall
reached only under -%N0/1 still takes the hts_mirror_check_moved path, which
cannot see the Set-Cookie without an ABI change; left as a known limitation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Test the cookie-wall give-up and retry-cap paths (#15)
The happy-path test only drove a wall that sets its cookie on the first hit, so
an engine that dropped the jar-changed gate and retried every self-redirect would
still pass. Add two cases that pin the gate and the bound:
- cookiewall3: a self-redirect that sets no cookie must give up at once (asserts
the "loop to same filename" guard fires and the page is not mirrored). A
dropped gate would retry and skip that log line.
- cookiewall4: a self-redirect that mints a fresh cookie every hit must stop at
the loops<7 cap, not spin (asserts the crawl terminates, takes the retry path,
and does not mirror the wall).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Scope the cookie-wall retry to this URL's own cookies (#15)
The retry trigger compared the whole shared cookie jar before and after the
request. Because the jar is one process-wide store mutated by every in-flight
fetch, a concurrent slot's Set-Cookie landing mid-loop could flip the compare
and force a needless (though bounded) re-fetch of an unrelated self-redirect.
Compare instead the Cookie header THIS url would send, built for its own
host/path. A Set-Cookie from another host no longer trips the retry; a change
in this URL's own outgoing cookies (its self-redirect setting one) still does,
which is exactly the wall we want to re-fetch. No jar-wide snapshot, and the
same natural termination: once the cookie stabilizes the header stops changing.
Reuses append_cookie_header via http_cookie_header (renamed from the
_selftest-suffixed wrapper, now that engine code calls it too).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit e8d1bf9c1965470b05a4ea5fa89a2eb742f2e975
Author: Xavier Roche <roche@httrack.com>
Date: Sat Jul 11 23:21:50 2026 +0200
Bound the in-memory receive buffer against a hostile Content-Length (#536)
* Bound the in-memory receive buffer against a hostile Content-Length
http_xfread1 buffers a non-disk (hypertext) response whole in RAM, sized
straight from the server's Content-Length, or grown without limit for a
chunked/unknown-length stream. The reads there already assume the buffer fits a
32-bit index (the "no 4GB html" comment and the int offsets), but nothing
enforced it: a hostile text/html response with a multi-GB Content-Length drove
an unbounded allocation on the first fetch. Reject an in-memory size that would
exceed INT32_MAX and return a clean error. A resource httrack holds whole in RAM
is hypertext and small; binary content (video, archives, ...) streams to disk
and never reaches this branch, so nothing legitimate is refused.
This is the initial-fetch companion to #535, which bounded the same allocation
on the 206-resume path. Self-test -#test=xfread-limit + 01_engine-xfread.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Reject exactly INT32_MAX and cover the accept path
An adversarial review of the in-memory bound found an off-by-one: the guard used
`> INT32_MAX`, but at `size + bufl == INT32_MAX` the unknown-length reallocs
compute `(int) size + bufl + 1 == INT32_MAX + 1`, a signed-overflow UB (it fails
safe because the negative width becomes a huge size_t and realloc returns NULL,
but it still aborts under UBSan). Use `>= INT32_MAX`.
The self-test gains a boundary case (that exact size must be refused, the teeth
for the off-by-one) and an accept case (a normal small size must not be), so a
stray `>` or a refuse-everything guard both fail it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit ef40277c245b79b72a5113da1205fb4115a99ace
Author: Xavier Roche <roche@httrack.com>
Date: Sat Jul 11 21:03:07 2026 +0200
Bound the in-memory 206-resume allocation to a 32-bit size (#535)
CodeQL flagged the memory-branch resume buffer as an uncontrolled allocation
size: malloct(resume + 1 + totalsize) takes an attacker-controlled
Content-Length. #534 guarded the int64 add against overflow but left the
allocation itself unbounded, and the (size_t) cast still truncated on a 32-bit
build. Cap the buffer at INT32_MAX -- a resource held whole in RAM is far
smaller -- and route an over-large or overflowing size to the existing
drop-and-refetch path. Test 48_local-crange-memresume already exercises that
path (its INT64_MAX Content-Length now trips the tighter bound).
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit 797acecfd2db8dd50d8b3354997b8a5efe949c0e
Author: Xavier Roche <roche@httrack.com>
Date: Sat Jul 11 20:04:47 2026 +0200
Harden Content-Range integer handling against hostile and oversize values (#534)
* Harden Content-Range integer handling against hostile and oversize values
Two integer-safety defects in the Content-Range / size path, both reachable
from a hostile or buggy server and abort-worthy under the project's UBSan/
OSS-Fuzz CI (a sanitizer abort is a real bug, not flakiness).
C2 (signed-overflow UB). The signed int64 crange fields are parsed straight
from the header with no validation, so a crafted Content-Range drives the
crange +/- 1 range checks past the int64 extremes. Clamp any negative field to
the all-zero "no range" sentinel at parse time (this covers the parse-side
crange-1 fallback and the 200->206 fake-range check), and rewrite the 206
resume check crange_end + 1 == crange as crange_end == crange - 1, which is
guarded by the preceding crange > 0 so an INT64_MAX end cannot overflow.
C4 (off_t truncated to int). The sizehack read the on-disk size into a 32-bit
int, truncating it for files >= 2 GiB, so a same-size match could force a 304
that records the wrong size (cache/disk metadata desync, never a truncated
file). Widen to off_t like every sibling call site.
Tests: -#test=crange drives the parser (UBSan aborts pre-fix on the INT64_MIN
fallback); 47_local-crange-overflow drives the 206-resume path with an
INT64_MAX Content-Range (UBSan aborts pre-fix at the crange+1 check).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Also guard the in-memory 206-resume buffer size against a hostile Content-Length
An adversarial review of the crange hardening surfaced a third overflow in the
same 206-resume path: the in-memory branch sizes its buffer as
`resume + 1 + totalsize`, and `totalsize` (Content-Length) is attacker-
controlled and unclamped. A resume answered with a 206 that lies `text/html`
(so the resume buffers in memory) and a matching INT64_MAX Content-Length drives
the add to INT64_MAX + 1 — a signed-overflow UBSan abort, reachable from a single
hostile response. Reject a Content-Length that would overflow the size, drop the
partial, and refetch from scratch (the retry restarts clean).
Test 48_local-crange-memresume drives it end to end (UBSan aborts pre-fix at
`alloc_mem += totalsize`); 01_engine-crange gains a `-5-10/20` case exercising
the clamp's independent per-field negativity.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit 8237c3ed2b75dcbcbb1a6d9467f91c2e68b6bde4
Author: Xavier Roche <roche@httrack.com>
Date: Sat Jul 11 18:27:57 2026 +0200
Don't finalize a resume's partial as complete on an out-of-protocol 304 (#533)
* Don't finalize a resume's partial as complete on an out-of-protocol 304
A resume request carries Range plus If-Unmodified-Since/If-Match, to which a
conformant server answers 206/200/412, never 304. The 304 handler's
"if-unmodified-since hack" re-accepted any existing on-disk file and recorded
it complete at its current byte count, so a broken or hostile 304 finalized
the partial as the whole file. In the repro the partial was never committed,
leaving the mirror claiming a complete file that is absent.
Capture whether the server itself sent the 304 (the size-match hacks force
NOT_MODIFIED only after confirming completeness). When it did on a Range
resume, drop the partial and its temp-ref and refetch, mirroring the
unusable-206 restart. Non-resume validations (range_req_size == 0) and the
416/sizehack completions are unaffected.
Test 46_local-update-304-resume drives it end to end: a stalled first fetch
leaves a partial + temp-ref, the resume's Range gets a bogus 304, and httrack
must recover the whole file rather than trust the 304.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Test 46: assert the resume's Range actually drew the 304
The size and request-count checks alone could pass a wrong-fix that silently
switched to a fresh re-crawl instead of handling the 304 on the resume path.
Mark when the server 304s a Range request and assert it fired, so recovery is
proven to go through the resume rather than around it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit 75b815e1fb300ea05ef38c67ec0f3fd2e8689cd5
Author: Xavier Roche <roche@httrack.com>
Date: Sat Jul 11 17:57:34 2026 +0200
tests: drop GNU-only uname -o so make check is quiet on macOS (#532)
The crawl harness built the test user-agent with uname -omrs. BSD/macOS
uname has no -o flag, so every crawl printed "uname: illegal option -- o"
to stderr and emitted an empty OS field. -s already names the kernel
(Linux/Darwin), making -o redundant; drop it for a portable uname -mrs.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit 8d02e609075ad4d37d365f0f5e52f26410055dcb
Author: Xavier Roche <roche@httrack.com>
Date: Sat Jul 11 17:18:18 2026 +0200
Bound test 19's dead-address connects so it is fast on macOS (#531)
19_local-connect-fallback pins dead loopback IPs (127.0.0.2/.3) before the live
one to exercise address fallback. On Linux those refuse instantly (all of
127.0.0.0/8 is loopback); macOS has only 127.0.0.1, so the connects block to
--timeout, and default retries replay the whole crawl, so the exhaustion case
took ~2 minutes. It passed but dominated the macOS suite once the rest runs in
parallel. Drop --timeout to 5s (bounds the last candidate's stall; the earlier
ones already fall back at min(timeout,10)) and set --retries=0 (no ×3 replay).
Linux is unchanged: the refused connects never reach the timeout.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit e70607e70c1f7d9389413d91187db508d46e8164
Author: Xavier Roche <roche@httrack.com>
Date: Sat Jul 11 17:18:04 2026 +0200
Run the test suite in parallel in CI (make check -j) (#528)
* Run the test suite in parallel in CI (make check -j)
The suite already builds under automake's parallel test harness (no
serial-tests), but every make check in CI ran it serially, so the ~90 tests
executed one at a time. Each crawl test spawns its own Python server on an
ephemeral port into a private mktemp dir, so nothing is shared across tests and
-j never contends on a port or a fixture; the only cross-test file
(check-network_sh.cache) is idempotent and gated behind the online tests.
Pass -j (nproc, or hw.ncpu on macOS) to every make check. Measured 3:26 -> 0:37
on a 12-core box, stable across runs including 2x oversubscription. Past a few
cores the wall time floors on the single longest test (bigcrawl ~24s), so this
is the whole win short of trimming those long poles.
The Debian build already parallelizes its packaged test pass: debhelper's
dh_auto_test emits `make -jN check` from DEB_BUILD_OPTIONS parallel=N, which the
CI deb job sets to nproc and Debian buildds set from their job count. Nothing to
change in debian/rules (Policy requires honoring DEB_BUILD_OPTIONS, not
hard-coding -j).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* AGENTS.md: note new tests auto-spread across -j workers
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Oversubscribe make check to 2x cores (capped 16) in CI
The suite is sleep-heavy: bigcrawl self-paces over ~365 files, resume-overlap
polls and waits, and the trickle/delayed tests sleep server-side. On a
core-constrained runner those idle cores go to waste at -j=cores. Running twice
as many test jobs as cores lets a ready test use a core while another sleeps.
Measured on a 3-core cpu-set (the macOS/Linux runner regime): -j3 79.5s, -j6
(2x) 43.1s (1.85x), then diminishing returns to the ~34s wall-clock floor set
by the longest single test (-j9 39.9s, -j16 37.3s). Stable across repeats, all
tests pass. The cap keeps big machines from spawning 100+ server+httrack pairs
for no gain once floor-bound.
Only the make check steps change; the build stays at -j nproc (CPU-bound, where
oversubscription only hurts), and the Debian job keeps parallel=nproc since that
knob drives both its build and test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Keep macOS at one test job per core (loopback drops under 2x)
2x oversubscription overloads macOS's loopback: six httrack processes each at
-c8 push ~48 concurrent connections, the stack drops a couple, and
36_local-bigcrawl under-fetches (expected 361 files, got 359). This is the same
macOS-only flake #527 fixed for the error count, now surfaced on the exact file
count. Linux tolerates 2x (all Linux jobs green, stable in local sim); macOS
does not, so pin it to one job per core.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Run macOS test suite serially (parallel flakes bigcrawl on loopback)
Parallel make check on macOS runs 36_local-bigcrawl's -c8 crawl alongside the
other crawl tests; the aggregate load overloads macOS's loopback, which drops
fetches, so bigcrawl under-fetches and its exact file count fails (saw 359 and
341 vs 361, at both 2x and one-per-core). This is the macOS-only loopback drop
#527 handled for the error count, now on the file count. Linux does not exhibit
it. Pin macOS to a serial run until bigcrawl tolerates transient drops.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Isolate bigcrawl in a second pass on macOS to parallelize the rest
macOS was the CI bottleneck (~460s serial, the ~8min total): its loopback drops
fetches when bigcrawl's sustained -c8 crawl competes with other crawls, so a
parallel make check flaked bigcrawl's exact file count. Rather than keep the
whole macOS suite serial, run everything except bigcrawl in parallel (2x cores,
capped 16), then bigcrawl alone in a second pass, which is the serial-safe
condition green on master. Linux still runs the full suite in parallel.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit cada6a1ae3e724154ad4fddf55613dbe9a915829
Author: Xavier Roche <roche@httrack.com>
Date: Sat Jul 11 17:17:30 2026 +0200
Meter -M against received volume, not saved bytes (#530)
* Meter -M against received volume, not saved bytes (#520)
-M ("maximum overall size that can be uploaded/scanned") enforced its
cap against HTS_STAT.stat_bytes, which counts only saved HTTP 200 bodies
(minus robots.txt). On crawls dominated by redirects, error bodies, or
plugin-written files the counter lags the volume actually pulled off the
network, so the cap barely bit — #77's reporter saw ~12.5 MB on disk
while "Bytes saved" hovered near a 3 MB cap.
Enforce against HTS_STAT.HTS_TOTAL_RECV instead: the true received
volume, incremented on every recv()/fread and monotonic, matching the
option's "scanned" wording. The #519 smooth-stop/grace/hard-abort
machinery is unchanged; only the metered quantity moves, and it stays
monotonic so the sticky size-branch invariant still holds. Received is
credited live per-chunk rather than at transfer completion, so the cap
also bounds overshoot slightly tighter.
Test 45_local-maxsize-recv drives a fixture of large 404 bodies
(received but never saved): the cap fires though only the index lands on
disk. It fails against master (stat_bytes never reaches the cap, all
links fetched) and passes with the fix.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Test 45: bound saved bytes so the cap-meter differential is pinned
The single --log-found assertion tripped on "some counter crossed the
cap"; its master-fails differential silently depended on 404 bodies not
being saved. Add --max-mirror-bytes 100000 (observed 573 B) so the test
asserts the cap fired while almost nothing landed on disk, which only the
received-volume meter can do. A build that saved the 404 bodies would
push stat_bytes past the cap and the mirror past the bound, failing here.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit 145ea4769f5146d5ae63d7589e733a1a3b868837
Author: Xavier Roche <roche@httrack.com>
Date: Sat Jul 11 16:33:45 2026 +0200
Make test 31 fail loudly when the java plugin doesn't load (#529)
31_local-javaclass verifies the java plugin parses a .class constant pool
by asserting a resource named only inside Foo.class gets crawled. When the
plugin fails to load, that assertion just reports hello.gif "not found",
with no hint the plugin was the cause -- which is why an intermittent
local failure sat uninvestigated.
The failure mode is a shadowing system libhtsjava: on a box with httrack
installed, if the soname httrack dlopens is absent from the fresh build
but present in /usr/lib, the loader picks up the system plugin (a
mismatched, non-instrumented build) and it silently parses nothing. It
looked ASan-specific only because that was the build in use; clean CI
runners have no system plugin, so they never hit it. #475 aligned the
dlopen name with the build's own soname, closing the gap.
Assert the plugin's launched-banner suffix ("+libhtsjava") first, so a
load failure trips there with a clear message instead of the opaque
hello.gif miss. The "+libhtsjava" prefix survives future soname bumps.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit c5da5a2f2e035e283dd022d14857c1351cb46cd9
Author: Xavier Roche <roche@httrack.com>
Date: Sat Jul 11 15:25:59 2026 +0200
Stop test 36 flaking on macOS when loopback drops connections (#527)
* Stop test 36 flaking on macOS when loopback drops connections
The 36_local-bigcrawl crawl runs -c8 over 361 files and asserted its
error count exactly (--errors 4, the planted 404/410/500/gztrunc). On the
macOS CI runner the busy loopback intermittently resets a few in-flight
connections, so the count came back 7 and the job failed while Linux
passed.
Add a --errors-content harness assertion that counts every "Error:" line
minus the transient-network family (statuscodes -2..-7:
timeout/connect/reset), and switch test 36 to it. The four planted
content errors (404/410/500 and the -1 decompression failure) still count
exactly, so a real regression that adds a content/HTTP error still fails;
only loopback flakiness is tolerated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Tighten --errors-content: drop -7 from transient, match retried errors
Review follow-up. Two fixes to the transient-error filter:
- Narrow the excluded band from -2..-7 to -2..-6. STATUSCODE_TOO_BIG (-7)
is a deterministic size-cap rejection, not a network transient, so it
must count as a real error; excluding it could mask a -M regression in
a future test that reuses this option.
- Also match the "after N retries" error format. htsparse.c logs
"<msg>" (<code>) after N retries at link ... when retries are enabled,
which the old "(-code) at link" regex missed. Test 36 forces
--retries=0 so only the immediate format appears there, but the option
is a general harness primitive and should hold under default retries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit 6eaefe6233944781e4abf7eb7ce4af2ba184242b
Author: Xavier Roche <roche@httrack.com>
Date: Sat Jul 11 15:09:54 2026 +0200
Type LLint/TStamp as int64_t and drop the per-platform boilerplate (#526)
* Type LLint/TStamp as int64_t, drop the per-platform boilerplate
The wide-integer typedefs picked long/long long/__int64 per platform and
carried a matching printf-format ladder plus the HTS_LONGLONG capability
macro. That is what let x32 pick a 32-bit long for a "64-bit" LLint (#524).
<stdint.h>/<inttypes.h> answer both questions directly: int64_t is exactly
64-bit signed with defined wrap, and "%" PRId64 is its conversion. LLint
stays signed because -1 is a size/range sentinel across the engine (a
uint64_t would silently break every "< 0" check). The names LLint, TStamp,
and the LLintP macro are kept, so the ~70 call sites are untouched.
HTS_LONGLONG is now dead (a 64-bit type is a hard C99 dependency here:
md5.h includes <stdint.h> and the installed httrack-library.h includes
<inttypes.h>), so its detection blocks and htslib.c's #ifdef around the
GiB/TiB/PiB formatting go too; those branches are always valid now.
No ABI change: int64_t is `long` on LP64, so httrackp/htsblk stay
byte-identical and the exported symbol set is unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Tighten the LLint typedef comments
Two one-line blocks instead of a four-line header; drop the change-narration
("no per-platform ladder"). Keep the load-bearing note that LLintP carries its
own '%'.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit 33249be5a0c89f16ead1cbbb2b216448f6fe0aa1
Author: Xavier Roche <roche@httrack.com>
Date: Sat Jul 11 14:49:01 2026 +0200
Assert kept content in the update-error test, not just size (#176 follow-up) (#525)
44_local-update-errormask checked only keep.dat's size (>= 1024). The good
body carries a distinctive KEEP marker that nothing asserted, so a same-size
wrong-content overwrite would still pass. Also assert the content matches
^KEEP and lacks the 403 error body, so the test discriminates the fix from
an equal-size clobber the size check alone misses.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit 167ede32c094615c8e595d2f441cced7d2c412b5
Author: Xavier Roche <roche@httrack.com>
Date: Sat Jul 11 14:30:43 2026 +0200
Keep the good local copy when an update fetch returns an HTTP error (#176) (#523)
On --update, a URL that returned 200 on the first crawl but now answers a
transient error (403/404/5xx) had its good local copy destroyed: the error
body overwrote it (errpage is on by default) and/or the delete_old purge
removed it. The guard that masks such an error as a 304 to keep the cached
copy was gated on !opt->delete_old, but delete_old is on by default, so it
never ran for a normal update.
Drop that outer gate so the masking runs by default, but restrict it to the
complete-cached-copy case (range_req_size == 0): an error on a resume/range
fetch must still fall through, else a stale-partial 416 would be masked to
304 and the partial never re-fetched (regressing #206). The masked error
routes through the not-modified handler, which reloads and re-registers the
cached file, blocking both the overwrite and the purge. Unlinked-page purging
is unaffected: it targets pages not fetched this run, not a fetched page that
errored.
Test 44_local-update-errormask drives it over a local server whose keep.dat
200s on the first crawl and 403s on the conditional update fetch.
Closes #176
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit c8b7c63c9b37759e256be839e7dc1d3d6171c766
Author: Xavier Roche <roche@httrack.com>
Date: Sat Jul 11 14:21:27 2026 +0200
Keep LLint 64-bit on the x32 ABI (#524)
The wide-integer typedef keyed on __x86_64__, which the x32 ABI also
defines even though its long is 32-bit. That silently made LLint -- the
signed 64-bit type for byte counts and file sizes -- 32-bit on x32, so
any value past INT_MAX overflowed (a >2GB size wrapped negative and the
cache dropped the entry). Exclude __ILP32__ so x32 takes the long-long
path, plus a compile-time guard that trips wherever 64-bit support is
claimed but LLint isn't actually 64-bit.
Surfaced by the cache selftests on Debian x32: cache-writefail (oversize)
and cache-corrupt both exercise >2GB entries and failed.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit 04f37b2094116592ce1f5cd3dff9b58b136205cb
Author: Xavier Roche <roche@httrack.com>
Date: Sat Jul 11 11:20:29 2026 +0200
Don't destroy already-complete files when -M/-E aborts an --update (#522)
An --update run stopped by the -M size cap (or -E time cap) could wipe out
files that were already fully mirrored, via two paths:
- An incomplete re-fetch opens the real file "wb" (truncating the good copy)
before any body byte arrives; the hard-abort then leaves it truncated.
- A file that fully arrives is committed, but the aborted slot then reaches
the main-loop save with statuscode <= 0 and size 0, and filesave() writes a
zero-byte file over it -- leaving the file empty on disk while the cache
still records it 200 OK, so a later --update never re-fetches it (permanent
loss).
For the first path, a re-fetch over an existing file now moves the good copy
aside before truncating; back_finalize commits the fresh file on a full body and
restores the previous copy on an aborted/incomplete transfer. url_sav is still
written the normal way, so the file list, update purge and cache bookkeeping are
unchanged. For the second, an empty body only overwrites the file on a real
response (statuscode > 0), never on a failed transfer.
Adds 43_local-update-truncate.test with a stateful bigtrunc fixture that
mirrors, then re-fetches under --update -M and asserts both the completed and
the incomplete-refetch file keep their full size across the abort. The harness
gains --rerun-args (a second pass with extra args) and --file-min-bytes.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit c07b19bc3d76e980ca2d664b5ab9689d1119326e
Author: Xavier Roche <roche@httrack.com>
Date: Sat Jul 11 09:17:13 2026 +0200
-M overall-size cap never bounds the overshoot under a slow server (#519)
* -M byte cap only smooth-stopped, never bounded the overshoot (#77)
The overall-size limit (-M / maxsite) requested a smooth stop once
stat_bytes crossed the cap but back_checkmirror always returned 1, so a
slow or throttling server let the in-flight transfers drain at its own
pace and the wait loops starved on them. On a bot-throttled archive the
mirror ran for over an hour and blew far past the cap. This is the size
counterpart of the -E fix in #482, which left the maxsite branch alone.
Give -M the same grace-then-abort escape hatch: once saved bytes overrun
the cap by maxsite/10, back_checkmirror returns 0 and the existing #482
abort path tears down the in-flight HTTP transfers (FTP slots stay with
their thread; real-named partials survive for --continue). The generic
abort log is reworded from "time limit" to "mirror limit" since it now
covers both quotas.
42_local-maxsize-slow.test crawls a server whose files trickle for a
minute under -M400000: the fixed engine stops in about 8 seconds, the
unfixed binary waits the full 60s.
The -M meter itself (saved bytes, which undercounts redirect/plugin-heavy
crawls vs. bytes actually received) is unchanged here; that is a separate
semantic question tracked as a follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
* Name the abort cause (size vs time) instead of a generic "mirror limit"
The #482 abort log and per-slot status were reworded to a generic "mirror
limit" once the size cap could also drive them, which dropped the cause. Add
an hts_mirror_limit enum and a back_mirror_limit() helper that reports which
cap overran its grace; back_checkmirror() now delegates its hard-stop decision
to it (removing the duplicated grace conditions), and the back_wait abort loop
uses it to log "size limit reached" / "time limit reached" and stamp the slot
"Mirror Size Limit" / "Mirror Time Out". This restores the precise wording -E
had before and gives the size path its own.
back_maxsize_grace keeps no floor, unlike the time grace: a size overrun
should abort promptly rather than let more bytes through. Documented inline.
Dropped the `--found bigtrickle/p0.bin` check from the test: the #482 hard
abort truncates a large file that is being re-fetched when the cap trips
(reproducible on master's -E path too), so p0.bin is legitimately 0 bytes on
many runs. The wall-clock bound is the discriminating assertion; the "giving
up" line fires for the unfixed engine as well.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit 04bbb489cfc3e3fe427875b09b5589be8e942d8c
Author: Xavier Roche <roche@httrack.com>
Date: Sat Jul 11 00:13:40 2026 +0200
Drop the per-request pre-resolve that slowed crawls on non-default ports (#181) (#518)
back_solve() resolved the raw "host:port" authority once per backing
request through check_hostname_dns() — an uncached getaddrinfo whose
result was only logged (host_wait() is always synchronous now). The
connect path already strips the port and resolves through the DNS cache,
so this lookup was pure overhead. On resolvers that don't fail fast on
the malformed "host:port" name (search-domain expansion, DNS timeouts)
it added a per-request stall: the reported slowness. Mirrors on port 80,
or with the port omitted, avoided the bad name and stayed fast.
Remove back_solve() and its debug-only lookup; keep the synchronous
host_wait() gate. Add a DNS self-test pinning that the cache/connect
path resolves "host:port" as the bare host.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![[bug history graph]](https://qa.debian.org/data/bts/graphs/h/httrack.png){kind=link}