vcswatch reports that
this package seems to have new commits in its VCS but has
not yet updated debian/changelog. You should consider updating
the Debian changelog and uploading this new version into the archive.
Here are the relevant commit logs:
commit d607c5a441e5b6dba2c2e6df8d518f11e73098cf
Author: Xavier Roche <roche@httrack.com>
Date: Sat Nov 7 08:54:29 2026 +0100
Refuse a bad --warc-max-size or -%c value (#2081)
Rejects a bad `--warc-max-size` or `-%c` value with an error and a
non-zero exit before any crawl. Both were ignored silently, and a tiny
`-%c` reached an undefined float-to-integer cast. A rate of 0 still
means no limit.
The connect limiter also floors the rate at 0.000001 where it divides,
because the WinHTTrack options dialog sets it without going through the
command line.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 38ff068d939a3ae3dfa3c04e20bd827bef05d519
Author: Xavier Roche <roche@httrack.com>
Date: Sat Nov 7 08:32:50 2026 +0100
Cap the cookies one host can store (#2083)
One host can no longer fill the cookie jar. When the jar is full, a
cookie from any host evicts the oldest cookies of the largest host
holding over 50 records or 8 KB.
A cookies file still loads uncapped and nothing is evicted while the jar
has room. If no host is over its share the cookie is refused. "Oldest"
means the first record of the shortest path, because the jar keeps no
age.
This also fixes a prefix match in `cookie_same_domain`, where a
Set-Cookie from `google.com.au` replaced the cookie of `google.com`.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
commit c319b0bf0210ddeadfab283e2788ea74039eded5
Author: Xavier Roche <roche@httrack.com>
Date: Fri Nov 6 23:26:51 2026 +0100
Add --check-cert to verify https certificates and host names (#2084)
`--check-cert` (`-@c`) verifies each https server's certificate chain
against the system CA store, and checks that the certificate names the
host. A failed check refuses that connection and logs the host and the
OpenSSL reason. The option is off by default, so a plain crawl behaves
as before.
`htsopt.h` is an installed header, so the new `check_cert` field is
appended at the end of `struct httrackp`, like the fields before it. On
Windows the option is refused with an error message, because OpenSSL
there has no system CA store yet. When the option is turned on during a
crawl, a keep-alive connection opened without the check is not reused.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 0a8b1daf3650cfa7cdd4ce7c159511e73bea916b
Author: Xavier Roche <roche@httrack.com>
Date: Fri Nov 6 23:20:36 2026 +0100
Convert pages with a UTF-16 BOM to UTF-8, surrogate pairs included (#2085)
A page starting with a UTF-16 byte order mark is now converted to UTF-8
when it reads as text. It needs some ASCII, and at most 1% of its units
may be lone surrogates, private use, noncharacters or C0 controls. It is
also converted when many of its bytes are NUL, as before. Mostly-CJK
pages therefore keep their text and links, while a binary file behind a
stray BOM is saved untouched. A surrogate pair becomes one 4-byte UTF-8
sequence, and a lone surrogate or the dangling byte of an odd-length
body becomes U+FFFD. The warning #2071 added is gone.
The converted page keeps any meta charset or SVG `encoding=` it
declared, as before, because the UTF-8 BOM wins in browsers. The
parser-diff corpus holds no UTF-16 page, so its clean result only shows
that nothing else moved. Test 734 shows the change, and its UTF-8,
BOM-less and binary controls are byte-identical to what master saves.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit d960f63bb17915f01b36c5622a9207e565835220
Author: Xavier Roche <roche@httrack.com>
Date: Fri Nov 6 23:05:07 2026 +0100
Name --filterlist instead of --urllist in the man page (#2082)
The man page, its HTML copy and the guide now name `--filterlist` for
`-%S`. `--urllist`, `--fast-engine` and `--debug-overflows` still run,
and a new test checks that.
`-#C` already lists zip cache entries on master (test 427), and the two
no-op options were already out of help and man.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
commit cc131fb627f1cc51fd8bc0c163a4797b3b9fcac0
Author: Xavier Roche <roche@httrack.com>
Date: Fri Nov 6 23:04:30 2026 +0100
Cap a decoded body at 256 MiB when -m is not set (#2079)
Without `-m`, a compressed body now stops decoding at 256 MiB and is
refused as "File too big", as #2070 does at the `-m` limit. Before, a
1.5 MB gzip body decoded to 1.5 GiB on disk.
The default applies to each kind (HTML, non-HTML) that `-m` leaves
unlimited, so `-mN` alone now caps HTML at 256 MiB.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit a455cf26a6c8084ba5e091b3b92b9eddb6c8d712
Author: Xavier Roche <roche@httrack.com>
Date: Fri Nov 6 22:53:58 2026 +0100
Scope a cookie without Path to the directory of the page that set it (#2080)
A cookie set without a `Path` attribute now applies only to the
directory of the page that set it (RFC 6265 section 5.1.4). So cookies
without `Path` now reach fewer pages, as they do in browsers. A `Path=`
that does not start with "/" also gets this default path now (section
5.2.4), which is why one row of test 570 changes.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit fd95eaf6d6389008c56ce1bfbcbefe62513878c9
Author: Xavier Roche <roche@httrack.com>
Date: Fri Nov 6 22:48:02 2026 +0100
Collapse the double slash a blanked .. leaves in a save name (#2077)
A save name no longer keeps a `//` after a directory is blanked, so
`/d/%2e%2e/x.html` saves as `d/x.html`, the same name as `/d/x.html`.
This covers `..`, dots-only or spaces-only directories, and a directory
the long-name cut shortens to dots.
Three old rows in `01_engine-savename.test` and two in
`400_engine-savename-type-digits.test` pinned the old `//` names and now
expect the collapsed ones. Three rows are new.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
commit 84ae53bb804a57285fa32ddb30d598cf92a24068
Author: Xavier Roche <roche@httrack.com>
Date: Fri Nov 6 22:47:06 2026 +0100
Document trusted folders, -%X scope and the -C default (#2078)
The docs now say that a project folder or `./.httrackrc` can run a
command or load a plugin. They also say that `-%X` goes to every host,
and give the real `-C` default.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
commit 4fb9cc89dbd9c263c3c511b843561bf842c2437e
Author: Xavier Roche <roche@httrack.com>
Date: Fri Nov 6 04:34:55 2026 +0100
Stop the parser rescanning the page for unclosed meta, comment and url( (#2067)
Three parser scans no longer rescan the rest of the page for each
occurrence. They are a `<meta` with no `>`, each `>` in a comment with
no `-->`, and an unclosed CSS `url(`. A 1.5 MB hostile page now parses
in about 3 seconds instead of 24 to 300+.
The `<meta` attribute scan now stops after 4096 bytes. An unquoted
`url(` not closed within 2048 bytes is no longer a link. A rare long
link cut by `#` or `\` is lost. The parser differential shows no change
on the corpus.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 104a5d1abc7103c53108c9885e60f00670edf149
Author: Xavier Roche <roche@httrack.com>
Date: Fri Nov 6 04:34:54 2026 +0100
Connect FTP data to the control host, and fix two header parsers (#2074)
FTP passive mode now connects the data channel to the control
connection's address and ignores the IP in the 227 reply, as curl does.
NAT'd servers that name a private address keep working, but a server
that sends data to a third host (FXP) no longer does.
Two parser fixes ride along. A zone letter such as `N` is no longer read
as a month. An out-of-range port in a fetch origin keeps the scheme
default.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit a3cc0b03a6084c61e0058c68e9dfd5665542353f
Author: Xavier Roche <roche@httrack.com>
Date: Fri Nov 6 04:28:40 2026 +0100
Fail a connection on a socket number above FD_SETSIZE (#2076)
Fails a connection whose socket number is FD_SETSIZE or more, instead of
writing past the `fd_set` in `back_wait` and the `check_*` helpers. Also
fixes an unchecked `malloc` for the FTP worker and an `isspace` call on
a signed `char`.
Windows is not guarded, because its `fd_set` is a bounded list. A crawl
in a process with over 1000 open files now logs "Socket N is beyond
FD_SETSIZE" and fails that link.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 6055b18711ae196e2fad1ac02cba16ffecd1b949
Author: Xavier Roche <roche@httrack.com>
Date: Fri Nov 6 04:15:08 2026 +0100
Fix the keyword index on accented words and repeated words (#2075)
The keyword indexer (`-%I`) now folds UTF-8 accented letters without
reading past its table. It also lists each word once per page, counts in
64 bits, closes the last word's block, and truncates `index.txt` after
an "ignored" rewind. Latin-1 pages are still not folded, because the
indexer reads raw bytes.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 572d1b2b488e076d9514ef38c9ccd881c70b89b3
Author: Xavier Roche <roche@httrack.com>
Date: Fri Nov 6 04:00:40 2026 +0100
Fix ProxyTrack keep-alive hang, archive merge loss and bad status lines (#2072)
ProxyTrack fixes three audit findings: a hang on the request after a
body, lost URLs when merging archives, and the status line `HTTP/1.1
-1`. Clients now time out after 30 seconds and get a 503 past 256
connections, so a silent client cannot hold a thread.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 3bfb458f1aa4f626246a0f96077ca226d43c3d31
Author: Xavier Roche <roche@httrack.com>
Date: Fri Nov 6 03:59:26 2026 +0100
Stop plain http from changing Secure cookies and match cookie paths on a slash (#2073)
Plain http can no longer set a Secure cookie, or set one that would
hide, replace or erase a Secure cookie. A cookie path now matches only
on a "/" boundary, so `/foo` no longer reaches `/foobar`.
A query string still ends the path. Three findings are left alone,
because each changes what ordinary crawls send or load.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 8c10addf154c13809d32464640d8b9b7ecb4a402
Author: Xavier Roche <roche@httrack.com>
Date: Fri Nov 6 03:50:52 2026 +0100
Match robots.txt wildcards in linear time and warn about unconverted UTF-16 pages (#2071)
Match robots.txt wildcard rules in linear time. A hostile file used to
cost 49 s of CPU for 150 links, and now costs 0.9 s. A UTF-16 page with
a BOM but few NUL bytes is still saved unconverted, and now logs a
warning.
Every rule is still evaluated, so no robots.txt can make us skip its own
later rules.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 5a3bd0af7cd7d3201d565fa4a6d7d37b677113d2
Author: Xavier Roche <roche@httrack.com>
Date: Fri Nov 6 03:38:04 2026 +0100
Drop WebHTTrack clients that stall mid-request (#2068)
The WebHTTrack server now gives each request 30s in total, so a stalled
or slow-dripping client is dropped. A body cut short by that deadline
gets a 400 instead of being processed. Test 78 also covers an over-long
sid, a duplicate sid in both orders and a `..` static path.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 3c32caa9a894675a2d3f75f3836c4e30378b218e
Author: Xavier Roche <roche@httrack.com>
Date: Fri Nov 6 03:20:43 2026 +0100
Stop the interactive wizard from overrunning its argument array (#2069)
The interactive wizard now refuses a command line with too many
arguments instead of writing past its 256-slot array. The HTML progress
percentage is computed in 64 bits, and `GID=` in the progress file
prints the group id.
Test 424 drives the wizard on a pty with 250 arguments and fails with a
crash on the old code.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 50a7d00e858cbef707cd1041450c2a6359ef83b1
Author: Xavier Roche <roche@httrack.com>
Date: Fri Nov 6 03:12:27 2026 +0100
Stop decoding a compressed body at the -m limit (#2070)
Decoding any content coding (gzip, deflate, brotli, zstd) now stops at
the user's `-m` limit and reports "File too big", as an uncoded body
does. A failed temporary file now fails the item instead of saving the
compressed bytes as the page.
HTML needs `-m,N`, because `-mN` alone leaves HTML uncapped. The default
budget is unchanged, so a crawl without `-m` still allows up to 2 GiB
per body.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 798c1cffe868e409c363eb953d51bc2c0209cb41
Author: Xavier Roche <roche@httrack.com>
Date: Fri Nov 6 01:21:56 2026 +0100
Make the WARC archive owner-only and open cookie temp files exclusively (#2066)
Creates the `--warc` archive, its `.wacz` and its `.cdx` owner-only. It
also opens cookie and bundle temp files with `O_EXCL`, so a planted file
or link makes the open fail. The WARC content is not redacted.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 0df6908b88704eb2dbe0646d2a400028e04c1068
Author: Xavier Roche <roche@httrack.com>
Date: Fri Nov 6 00:58:12 2026 +0100
Stop save-name collisions and purge escapes from deleting files (#2064)
Stops a directory-in-the-way rename from overwriting an existing
`<name>.txt`, and makes the update purge skip old.lst lines that leave
the mirror. It also escapes the device names `com5`-`com9`,
`lpt5`-`lpt9`, `conin$` and `conout$`.
The first fix changes what a crawl saves only in the collision case.
There a page was lost silently, so this is a data-loss fix: the moved
file now lands at `<name>.1.txt` (then `.2`, up to 99). A page crawled
later under the moved name can still overwrite the moved file, as
before. The new device names also get a trailing `_`, like the existing
ones.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit d3ba2202e11d059ffe5a0194d019976ac69d1d01
Author: Xavier Roche <roche@httrack.com>
Date: Thu Nov 5 22:38:45 2026 +0100
Stop leaking URL credentials in TLS SNI and over plain http (#2065)
The TLS SNI is now the bare host name. A Basic login learned over https
is never sent on plain http. A long proxy credential is refused instead
of aborting.
An http login may still be sent on https to the same host and port, so a
redirect to https keeps working. Credentials up to 765 bytes are
accepted now, up from 255, the most a stored login holds.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 4c1db118b87b7244793e00c0dc67eeb32a76ba0d
Author: Xavier Roche <roche@httrack.com>
Date: Wed Nov 4 01:56:32 2026 +0100
Document when hts_bundle_from_paste_hosts skips hosts (#2063)
The header now says that hosts past 256 scope domains are skipped. It
also says the whole list is ignored when the address cannot have a scope
line, such as `intranet`.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 483d5d47a6243cf57d1495e473a09a09519f3dea
Author: Xavier Roche <roche@httrack.com>
Date: Wed Nov 4 00:17:59 2026 +0100
Keep the cookies of the sites a browser login went through (#2062)
A browser login that passes through a second site writes a bundle
holding only the start site's cookies. The new
`hts_bundle_from_paste_hosts()` takes the hosts the login went through.
It keeps their cookies under one `scope` line per registrable domain,
and they never become start URLs.
The start site's records still load first, and a new `dropped` field
counts the listed-host records left out. Two bundles that signed in
through one site no longer stop the mirror when their cookie or user
agent differ there. The first bundle wins and the mirror logs a warning.
A clash on a site that is one bundle's own start site still refuses.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 9148ca019193daafda51f56a6e30e427e032a56b
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date: Tue Nov 3 11:20:24 2026 +0000
Bump src/coucal from `d77caa7` to `031eaf1` (#2060)
commit 25a8dcafa36bad99265e19d557c9dc98d68c3f83
Author: Xavier Roche <roche@httrack.com>
Date: Mon Nov 2 16:04:45 2026 +0100
Time test 72's watchdog from the engine launch, not the test start (#2059)
Test 72 now times the crawl watchdog from the moment httrack starts, not
from the moment the test starts. On a loaded Hurd VM the test server
took most of the 25s budget to start. That failed Tier-2 run 41395642768
with `watchdog fired late (25s)`, although the watchdog fired on time.
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
commit 8d7be643af8dc29db6fbdbd7b2b0ad34da330cad
Author: Xavier Roche <roche@httrack.com>
Date: Sat Oct 31 17:55:18 2026 +0100
Skip collision suffixes an earlier walk found taken (#2057)
Each collision walk now remembers the suffix it reached, so the next
case variant jumps there instead of trying `-2`, `-3` and so on again.
Saved names do not change, and old and new builds named 144 input sets
identically. These cover the 8.3 and ISO9660 modes, existing `-N`
suffixes, and links named again with the host in another case.
This branch is based on #2055, which changes the same function, so that
one should merge first.
Closes #2056
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
commit 256b7e4322911f51e981f4cd23e5555b5feb81b5
Author: Xavier Roche <roche@httrack.com>
Date: Sat Oct 31 15:48:23 2026 +0100
Stop a huge collision suffix from overflowing the save-name counter (#2055)
The case-collision rename overflowed a signed int when it added one to a
suffix like `-2147483647`, and `sscanf` overflowed on a 20-digit suffix.
The suffix is now range-checked, and one too large to increment stays in
the name, which then gets a fresh `-2`. The fuzz-savename harness in PR
#2050 found it.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
commit a3a745c8efb204d113376fea3e7bbe61b8c042ab
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 29 10:57:24 2026 +0100
Stop a long URL port from overflowing in the WARC CDX index (#2054)
A long numeric port in a URL no longer overflows an int when the CDXJ
index builds its sort key. Such a port is now kept verbatim, where
before it could wrap to 80 and be dropped. The WARC fuzz harness
proposed in #2053 found it, and the `warc-surt` self-test now covers
both cases.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
commit d0d448e919358ba49fac2d2cbcf883318bc12721
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 29 10:13:28 2026 +0100
Stop pwsh crashing at startup in test 226 on macOS (#2051)
Test 226 now sets `__PWSH_LOGIN_CHECKED` so pwsh skips its macOS login
check. The argv sysctl behind that check sometimes fails and kills pwsh
at startup. A Linux control fails the test if pwsh stops honouring the
variable.
The CI failure print now runs `tools/ci-test-log.sh`. When a step
timeout left no `test-suite.log`, it prints the logs of the failed and
unfinished tests instead.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 60b374ccaffd483e39f316b476982736039083d9
Author: Xavier Roche <roche@httrack.com>
Date: Thu Oct 29 06:19:45 2026 +0100
Skip the y2038 file-date check on a filesystem that clips dates (#2046)
The y2038 self-test now skips its post-2038 file checks when the
filesystem itself clips dates. The sparc64 buildd failed
`659_engine-y2038` on 3.51.0-1 because its filesystem reads back any
date past 2038 as 2147483647. The test detects this with a raw `utime()`
call, so an engine that clips a 64-bit `time_t` still fails on every
other host.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit a8bf81fb00628f41f9f76817019c2ba9fb5daa1a
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 28 22:21:02 2026 +0100
Update the guide screenshots for 3.51 (#2045)
The guide screenshots of WinHTTrack, WebHTTrack and Android now come
from the 3.51 builds (httrack-works 8f87a01, 456702a and e7a4bc1).
Android added the Accept headers tab as its eleventh option tab, so its
later screenshots are renumbered in `tools/doc-images.py`. The guide
also gains two images: the WinHTTrack Insert URL dialog with the browser
login button, and the Android Accept headers tab.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
commit 0a7ee7f100ad833e97e9df9ead6a88078bb0076c
Author: Xavier Roche <roche@httrack.com>
Date: Wed Oct 28 21:40:12 2026 +0100
Document the browser login of WinHTTrack, WebHTTrack and Android (#2044)
The login page now starts with the "Log in with a browser" button of
each 3.51 interface. Exporting cookies by hand becomes the fallback. The
FAQ answers about logins, 403s and login forms point to that page. A new
FAQ entry covers a mirror that holds only a login, CAPTCHA or "checking
your browser" page.
---------
Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
![[bug history graph]](https://qa.debian.org/data/bts/graphs/h/httrack.png){kind=link}