diffoscope (240+deb12u1) stable; urgency=medium [ Chris Lamb ] * Backport a patch by FC (Fay) Stegerman to fix a FTBFS caused by a .zip-related security fix that was included in Debian's own upload of python3.11 3.11.2-6+deb12u2 (see #1070133). Diffoscope's testsuite deliberately excercises a Mozilla-style ZIP file that has its Central Directory secton at the beginning of the file, rather than at the end. This breaks the new overlap check in Python's built-in zipfile.py library as that checks that every entry ends before the Central Directory begins. Many thanks to Fay for both the patch and related guidance. (Closes: #1078883) * Do not call marshal.loads() on precompiled Python bytecode as it is inherently unsafe. The loads() method can easily cause the CPython process running diffoscope to irretrievably crash (e.g. when presented with a newer .pyc format), and potentially permit of arbitrary code execution. Replace, for now, with a brief textual summary of the code section of .pyc files instead. For more information, see: -- Chris Lamb Mon, 26 Aug 2024 11:43:37 +0100 diffoscope (240) unstable; urgency=medium [ Holger Levsen ] * Update Lintian override info format in debian/source/lintian-overrides. * Add Lintian overrides for some "very long lines" in test cases. * Update Lintian overrides for tests being tagged source-is-missing or prebuilt. * Add Lintian override for very long lines for debian/tests/control. * Re-add two Lintian overrides about (well-known) source-is-missing instances. [ Mattia Rizzolo ] * Drop the use of include_package_data=True in setup.py. -- Chris Lamb Fri, 31 Mar 2023 09:05:24 +0100 diffoscope (239) unstable; urgency=medium [ Chris Lamb ] * Fix compatibility with pypdf 3.x, and correctly restore test data. (Closes: reproducible-builds/diffoscope#335) * Rework PDF annotations processing into a separate method. -- Chris Lamb Fri, 17 Mar 2023 08:31:17 +0000 diffoscope (238) unstable; urgency=medium * autopkgtest: fix tool name in the skippable list. -- Mattia Rizzolo Wed, 01 Mar 2023 20:17:05 +0100 diffoscope (237) unstable; urgency=medium * autopkgtest: only install aapt and dexdump on architectures where they are available. (Closes: #1031297) * compartors/pdf: + Drop backward compatibility assignment. + Fix flake warnings, potentially reinstating PyPDF 1.x support (untested). -- Mattia Rizzolo Tue, 28 Feb 2023 18:16:09 +0100 diffoscope (236) unstable; urgency=medium [ FC Stegerman ] * Update code to match latest version of Black. (Closes: #1031433) [ Chris Lamb ] * Require at least Black version 23.1.0 to run the internal Black tests. * Update copyright years. -- Chris Lamb Fri, 17 Feb 2023 08:48:41 -0800 diffoscope (235) unstable; urgency=medium [ Akihiro Suda ] * Update .gitlab-ci.yml to push versioned tags to the container registry. (Closes: reproducible-builds/diffoscope!119) [ Chris Lamb ] * Fix compatibility with PyPDF2. (Closes: reproducible-builds/diffoscope#331) * Fix compatibility with ImageMagick 7.1. (Closes: reproducible-builds/diffoscope#330) [ Daniel Kahn Gillmor ] * Update from PyPDF2 to pypdf. (Closes: #1029741, #1029742) [ FC Stegerman ] * Add support for Android resources.arsc files. (Closes: reproducible-builds/diffoscope!116) * Add support for dexdump. (Closes: reproducible-builds/diffoscope#134) * Improve DexFile's FILE_TYPE_RE and add FILE_TYPE_HEADER_PREFIX, and remove "Dalvik dex file" from ApkFile's FILE_TYPE_RE as well. [ Efraim Flashner ] * Update external tool for isoinfo on guix. (Closes: reproducible-builds/diffoscope!124) -- Chris Lamb Fri, 10 Feb 2023 10:31:52 -0800 diffoscope (234) unstable; urgency=medium [ FC Stegerman ] * test_text_proper_indentation requires at least file version 5.44. (Closes: reproducible-builds/diffoscope#329) -- Chris Lamb Mon, 30 Jan 2023 09:29:37 -0800 diffoscope (233) unstable; urgency=medium [ FC Stegerman ] * Split packaging metadata into an extras_require.json file instead of using the pep517 and the pip modules directly. This was causing build failures if not using a virtualenv and/or building without internet access. (Closes: #1029066, reproducible-builds/diffoscope#325) [ Vagrant Cascadian ] * Add an external tool reference for GNU Guix (lzip). * Drop an external tool reference for GNU Guix (pedump). [ Chris Lamb ] * Split inline Python code in shell script to generate test dependencies to a separate Python script. * No need for "from __future__ import print_function" import in setup.py anymore. * Comment and tidy the new extras_require.json handling. -- Chris Lamb Fri, 20 Jan 2023 08:56:22 -0800 diffoscope (232) unstable; urgency=medium [ Chris Lamb ] * Allow ICC tests to (temporarily) fail. * Update debian/tests/control after the addition of PyPDF 3 support. [ FC Stegerman ] * Update regular expression for Android .APK files. [ Sam James ] * Support PyPDF version 3. -- Chris Lamb Fri, 13 Jan 2023 07:05:01 +0000 diffoscope (231) unstable; urgency=medium * Improve "[X] may produce better output" messages. Based on a patch by Helmut Grohne. (Closes: #1026982) -- Chris Lamb Fri, 06 Jan 2023 09:02:57 +0000 diffoscope (230) unstable; urgency=medium [ Chris Lamb ] * Fix compatibility with file(1) version 5.43; thanks, Christoph Biedl. [ Jelle van der Waa ] * Support Berkeley DB version 6. -- Chris Lamb Fri, 30 Dec 2022 06:35:04 +0000 diffoscope (229) unstable; urgency=medium [ Chris Lamb ] * Skip test_html.py::test_diff if html2text is not installed. (Closes: #1026034) [ Holger Levsen ] * Bump standards version to 4.6.2, no changes needed. -- Chris Lamb Tue, 20 Dec 2022 23:12:16 +0000 diffoscope (228) unstable; urgency=medium [ FC Stegerman ] * As an optimisation, don't run apktool if no differences are detected before the signing block. (Closes: reproducible-builds/diffoscope!105) [ Chris Lamb ] * Support both the python3-progressbar and python3-progressbar2 Debian packages, two modules providing the "progressbar" Python module. (Closes: reproducible-builds/diffoscope#323) * Ensure we recommend apksigcopier. (Re: reproducible-builds/diffoscope!105) * Make the code clearer around generating the Debian substvars and tidy generation of os_list. * Update copyright years. -- Chris Lamb Fri, 02 Dec 2022 08:10:44 +0000 diffoscope (227) unstable; urgency=medium [ Chris Lamb ] * Don't attempt to attach text-only differences notice if there are no differences to begin with. (Closes: #1024171, #1024349) * Don't run Python decompiling tests on Python bytecode that both file(1) cannot yet detect and Python 3.11 cannot demarshall. (Closes: #1024335) -- Chris Lamb Fri, 18 Nov 2022 09:38:34 +0000 diffoscope (226) unstable; urgency=medium [ Christopher Baines ] * Add an lzip comparator with tests. [ Chris Lamb ] * Add support for comparing the "text" content of HTML files using html2text. (Closes: #1022209, reproducible-builds/diffoscope#318) * Misc/test improvements: * Drop the ALLOWED_TEST_FILES test; it's mostly just annoying. * Drop other copyright notices from lzip.py and test_lzip.py. * Use assert_diff helper in test_lzip.py. * Pylint tests/test_source.py. [ Mattia Rizzolo ] * Add lzip to debian dependencies. -- Chris Lamb Fri, 04 Nov 2022 06:51:43 +0000 diffoscope (225) unstable; urgency=medium [ Chris Lamb ] * Add support for detecting ordering-only differences in XML files. (Closes: #1022146) * Fix an issue with detecting ordering differences. (Closes: #1022145) * Add support for ttx(1) from fonttools. (Re: reproducible-builds/diffoscope#315) * Test improvements: - Tidy up the JSON tests and use assert_diff over get_data and manual assert in XML tests. - Rename order1.diff to json_expected_ordering_diff for consistency. - Temporarily allow the stable-po pipeline to fail in the CI. * Use consistently capitalised "Ordering" everywhere we use the word in diffoscope's output. -- Chris Lamb Fri, 28 Oct 2022 08:29:53 -0700 diffoscope (224) unstable; urgency=medium [ Mattia Rizzolo ] * Fix rlib test failure with LLVM 15. Thanks to Gianfranco Costamagna (locutusofborg) for the patch. -- Chris Lamb Fri, 07 Oct 2022 09:35:04 -0700 diffoscope (223) unstable; urgency=medium [ Chris Lamb ] * The cbfstools utility is now provided in Debian via the coreboot-utils Debian package, so we can enable that functionality within Debian. (Closes: #1020630) [ Mattia Rizzolo ] * Also include coreboot-utils in Build-Depends and Test-Depends so it is available for the tests. [ Jelle van der Waa ] * Add support for file 5.43. -- Chris Lamb Fri, 30 Sep 2022 08:25:31 -0700 diffoscope (222) unstable; urgency=medium [ Mattia Rizzolo ] * Use pep517 and pip to load the requirements. (Closes: #1020091) * Remove old Breaks/Replaces in debian/control that have been obsoleted since bullseye -- Chris Lamb Fri, 23 Sep 2022 09:05:12 +0100 diffoscope (221) unstable; urgency=medium * Don't crash if we can open a PDF file with PyPDF but cannot parse the annotations within. (Closes: reproducible-builds/diffoscope#311) * Depend on the dedicated xxd package, not vim-common. * Update external_tools.py to reflect xxd/vim-common change. -- Chris Lamb Fri, 19 Aug 2022 07:32:23 -0700 diffoscope (220) unstable; urgency=medium * Support Haskell 9.x series files and update the test files to match. Thanks to Scott Talbert for the relevant info about the new format. (Closes: reproducible-builds/diffoscope#309) * Fix a regression introduced in diffoscope version 207 where diffoscope would crash if one directory contained a directory that wasn't in the other. Thanks to Alderico Gallo for the report and the testcase. (Closes: reproducible-builds/diffoscope#310) -- Chris Lamb Fri, 29 Jul 2022 11:03:24 -0700 diffoscope (219) unstable; urgency=medium * Don't traceback if we encounter an invalid Unicode character in Haskell versioning headers. (Closes: reproducible-builds/diffoscope#307) * Update various copyright years. -- Chris Lamb Fri, 15 Jul 2022 06:45:55 +0100 diffoscope (218) unstable; urgency=medium * Improve output of Markdown and reStructuredText to use code blocks with syntax highlighting. (Closes: reproducible-builds/diffoscope#306) -- Chris Lamb Fri, 08 Jul 2022 08:45:11 +0100 diffoscope (217) unstable; urgency=medium * Update test fixtures for GNU readelf 2.38 (now in Debian unstable). * Be more specific about the minimum required version of readelf (ie. binutils) as it appears that this "patch" level version change resulted in a change of output, not the "minor" version. (Closes: #1013348) * Don't leak the (likely-temporary) pathname when comparing PDF documents. -- Chris Lamb Thu, 23 Jun 2022 07:43:21 +0100 diffoscope (216) unstable; urgency=medium * Print profile output if we were called with --profile and we receive a TERM signal. * Emit a warning if/when we are handling a TERM signal. * Clarify in the code in what situations the main "finally" block gets called, especially in relation to handling TERM signals. * Clarify and tidy some unconditional control flow in diffoscope.profiling. -- Chris Lamb Fri, 10 Jun 2022 07:00:54 +0100 diffoscope (215) unstable; urgency=medium [ Chris Lamb ] * Bug fixes: - Also catch IndexError (in addition to ValueError) when parsing .pyc files. (Closes: #1012258) - Strip "sticky" etc. from "x.deb: sticky Debian binary package […]". Thanks to David Prévot for the report. (Closes: #1011635) - Correctly package diffoscope's scripts/ directory, fixing the extraction of vmlinuz kernel images. (Closes: reproducible-builds/diffoscope#305) - Correct the logic for supporting different versions of argcomplete in debian/rules. * New features: - Support both PyPDF 1.x and 2.x. * Codebase improvements: - Don't call re.compile and then call .sub on the result; just call re.sub directly. - Clarify the logic around the difference between --usage and --help. * Testsuite improvements: - Integrate test coverage with GitLab's concept of artifacts. - Re-enable Gnumeric tests as its now available again. - Test --help and --usage, and additionally test that --help includes the programmatically-generated file format list as well. [ Holger Levsen ] * Bump Standards Version field in debian/control to 4.6.1. -- Chris Lamb Fri, 03 Jun 2022 06:54:48 +0100 diffoscope (214) unstable; urgency=medium [ Chris Lamb ] * Support both python-argcomplete 1.x and 2.x. [ Vagrant Cascadian ] * Add external tool on GNU Guix for xb-tool. -- Chris Lamb Fri, 27 May 2022 06:53:04 +0100 diffoscope (213) unstable; urgency=medium * Don't mask differences in .zip/.jar central directory extra fields. * Don't show a binary comparison of .zip/.jar files if we have at least one observed nested difference. * Use assert_diff in test_zip over get_data and separate assert. -- Chris Lamb Fri, 20 May 2022 08:15:29 -0700 diffoscope (212) unstable; urgency=medium * Add support for extracting vmlinuz/vmlinux Linux kernel images. (Closes: reproducible-builds/diffoscope#304) * Some Python .pyc files report as "data", so support ".pyc" as a fallback extension. -- Chris Lamb Fri, 13 May 2022 08:19:57 -0700 diffoscope (211) unstable; urgency=medium [ Mattia Rizzolo ] * Drop mplayer from the Build-Depends, it was add likely by accident and it's not needed. * Disable gnumeric tests in Debian because it's not currently available. -- Chris Lamb Thu, 28 Apr 2022 08:20:23 -0700 diffoscope (210) unstable; urgency=medium [ Mattia Rizzolo ] * Make sure that PATH is properly mangled for all diffoscope actions, not just when running comparators. -- Chris Lamb Fri, 15 Apr 2022 09:02:25 +0100 diffoscope (209) unstable; urgency=medium * Update R test fixture for R 4.2.x series. (Closes: #1008446) * Update minimum version of Black to prevent test failure on Ubuntu jammy. -- Chris Lamb Sun, 27 Mar 2022 09:23:02 +0100 diffoscope (208) unstable; urgency=medium [ Brent Spillner ] * Add graceful handling for UNIX sockets and named pipes. (Closes: reproducible-builds/diffoscope#292) * Remove a superfluous log message and reformatt comment lines. [ Chris Lamb ] * Reformat various files to satisfy current version of Black. -- Chris Lamb Fri, 25 Mar 2022 08:24:33 +0000 diffoscope (207) unstable; urgency=medium * Fix a gnarly regression when comparing directories against non-directories. (Closes: reproducible-builds/diffoscope#292) * Use our assert_diff utility where we can within test_directory.py -- Chris Lamb Fri, 04 Mar 2022 12:55:26 +0000 diffoscope (206) unstable; urgency=medium * Also allow "Unicode text, UTF-8 text" as well as "UTF-8 Unicode text" to match for .buildinfo files too. * Add a test for recent file(1) issue regarding .changes files. (Re: reproducible-builds/diffoscope#291) * Drop "_PATH" suffix from some module-level globals that are not paths. -- Chris Lamb Fri, 25 Feb 2022 08:03:04 +0000 diffoscope (205) unstable; urgency=medium * Fix a file(1)-related regression where .changes files that contained non-ASCII text were not identified as being .changes files, resulting in seemingly arbitrary packages on tests.reproducible-builds.org and elswhere not comparing the package at all. The non-ASCII parts could have been in the Maintainer or in the upload changelog, so we were effectively penalising anyone outside of the Anglosphere. (Closes: reproducible-builds/diffoscope#291) * Don't print a warning to the console regarding NT_GNU_BUILD_ID changes in ELF binaries. -- Chris Lamb Sat, 19 Feb 2022 14:58:57 -0800 diffoscope (204) unstable; urgency=medium [ Chris Lamb ] * Don't run the binwalk comparator tests as root (or fakeroot) as the latest version of binwalk has some security protection against doing precisely this. * If we fail to scan a file using binwalk, return 'False' from BinwalkFile.recognizes rather than raise a traceback. * If we fail to import the Python "binwalk" module, don't accidentally report that we are missing the "rpm" module instead. [ Mattia Rizzolo ] * Use dependencies to ensure that "diffoscope" and "diffoscope-minimal" packages always have the precise same version. -- Chris Lamb Fri, 11 Feb 2022 10:39:27 -0800 diffoscope (203) unstable; urgency=medium [ Chris Lamb ] * Improve documentation for --timeout due to a few misconceptions. Add an allowed-to-fail test regarding a regression in directory handling. * Tidy control flow in Difference._reverse_self a little. [ Alyssa Ross ] * Fix diffing CBFS names that contain spaces. -- Chris Lamb Fri, 04 Feb 2022 08:13:13 -0800 diffoscope (202) unstable; urgency=medium [ Chris Lamb ] * Don't fail if comparing a nonexistent file with a .pyc file (and add test). (Closes: #1004312) * Drop a reference in the manual page which claims the ability to compare non-existent files on the command-line. This has not been possible since version 32 which was released in September 2015. (Closes: #1004182) * Add experimental support for incremental output support with a timeout. Passing, for example, --timeout=60 will mean that diffoscope will not recurse into any sub-archives after 60 seconds total execution time has elapsed and mark the diff as being incomplete. (Note that this is not a fixed/strict timeout due to implementation issues.) (Closes: reproducible-builds/diffoscope#301) * Don't return with an exit code of 0 if we encounter device file such as /dev/stdin with human-readable metadata that matches literal, non-device, file contents. (Closes: #1004198) * Correct a "recompile" typo. [ Sergei Trofimovich ] * Fix/update whitespace for Black 21.12. -- Chris Lamb Fri, 28 Jan 2022 08:17:21 -0800 diffoscope (201) unstable; urgency=medium [ Chris Lamb ] * If the debian.deb822 module raises any exception on import, re-raise it as an ImportError instead. This should fix diffoscope on some Fedora systems. Thanks to Mattia Rizzolo for suggesting this particular solution. (Closes: reproducible-builds/diffoscope#300) [ Zbigniew Jędrzejewski-Szmek ] * Fix json detection with file-5.41-3.fc36.x86_64. -- Chris Lamb Fri, 21 Jan 2022 08:35:36 -0800 diffoscope (200) unstable; urgency=medium * Even if a Sphinx .inv inventory file is labelled "The remainder of this file is compressed using zlib", it might not actually be. In this case, don't traceback, and simply return the original content. (Closes: reproducible-builds/diffoscope#299) * Update "X has been modified after NT_GNU_BUILD_ID has been applied" message to, for instance, not duplicating the full filename in the primary diffoscope's output. -- Chris Lamb Fri, 14 Jan 2022 09:15:19 +0000 diffoscope (199) unstable; urgency=medium [ Chris Lamb ] * Support both variants of "odt2txt", including the one provided by unoconv. (Closes: reproducible-builds/diffoscope#298) [ Jelle van der Waa ] * Add external tool reference on Arch Linux for xb-tool. -- Chris Lamb Fri, 07 Jan 2022 08:46:42 +0000 diffoscope (198) unstable; urgency=medium [ Chris Lamb ] * Support showing "Ordering differences only" within .dsc field values. (Closes: #1002002, reproducible-builds/diffoscope#297) * Support OCaml versions 4.11, 4.12 and 4.13. (Closes: #1002678) * Add support for XMLb files. (Closes: reproducible-builds/diffoscope#295) * Also add, for example, /usr/lib/x86_64-linux-gnu to our internal PATH. [ Mattia Rizzolo ] * Also recognize "GnuCash file" files as XML. -- Chris Lamb Fri, 31 Dec 2021 08:52:04 +0000 diffoscope (197) unstable; urgency=medium [ Chris Lamb ] * Drop unnecessary has_same_content_as logging calls. [ Mattia Rizzolo ] * Ignore the new "binary-with-bad-dynamic-table" Lintian tag. * Support pgpdump 0.34 in the tests. Thanks to Michael Weiss for reporting and testing the fix. -- Chris Lamb Fri, 17 Dec 2021 09:56:04 +0000 diffoscope (196) unstable; urgency=medium [ Roland Clobus ] * Add a comment/annotation when the GNU_BUILD_ID field has been modified. [ Brent Spillner ] * Fix the "Black" version detection. [ Chris Lamb ] * Replace "token" with anonymous variable "x" in order to remove extra lines. -- Chris Lamb Fri, 10 Dec 2021 08:05:10 -0800 diffoscope (195) unstable; urgency=medium [ Chris Lamb ] * Don't use the runtime platform's native endianness when unpacking .pyc files to fix test failures on big-endian machines. -- Chris Lamb Sun, 05 Dec 2021 09:30:08 -0800 diffoscope (194) unstable; urgency=medium [ Chris Lamb ] * Don't traceback when comparing nested directories with non-directories. (Closes: reproducible-builds/diffoscope#288) -- Chris Lamb Fri, 26 Nov 2021 10:01:05 -0800 diffoscope (193) unstable; urgency=medium [ Chris Lamb ] * Don't duplicate file lists at each directory level. (Closes: #989192, reproducible-builds/diffoscope#263) * When pretty-printing JSON, mark the difference as such, additionally avoiding including the full path. (Closes: reproducible-builds/diffoscope#205) * Codebase improvements: - Update a bunch of %-style string interpolations into f-strings or str.format. - Import itertools top-level directly. - Drop some unused imports. - Use isinstance(...) over type(...) == - Avoid aliasing variables if we aren't going to use them. [ Brandon Maier ] * Fix missing diff output on large diffs. [ Mattia Rizzolo ] * Ignore a Python warning coming from a dependent library (triggered by supporting Python 3.10) * Document that support both Python 3.9 and 3.10. -- Chris Lamb Fri, 19 Nov 2021 07:35:10 -0800 diffoscope (192) unstable; urgency=medium * Update .epub test methodology after improving XML file parsing. -- Chris Lamb Fri, 12 Nov 2021 08:17:14 -0800 diffoscope (191) unstable; urgency=medium [ Chris Lamb ] * Detect XML files as XML files if either file(1) claims if they are XML files, or if they are named .xml. (Closes: #999438, reproducible-builds/diffoscope#287) * Don't reject Debian .changes files if they contain non-printable characters. (Closes: reproducible-builds/diffoscope#286) * Continue loading a .changes file even if the referenced files inside it do not exist, but include a comment in the diff as a result. * Log the reason if we cannot load a Debian .changes file. [ Zbigniew Jędrzejewski-Szmek ] * Fix inverted logic in the assert_diff_startswith() utility. -- Chris Lamb Fri, 12 Nov 2021 06:43:57 -0800 diffoscope (190) unstable; urgency=medium [ Chris Lamb ] * Don't raise a traceback if we cannot de-marshal Python bytecode to support Python 3.7 loading newer .pyc files. (Closes: reproducible-builds/diffoscope#284) * Fix Python tests under Python 3.7 with file 5.39+. [ Vagrant Cascadian ] * Skip Python bytecode testing when "file" is older than 5.39. [ Roland Clobus ] * Detect whether the GNU_BUILD_ID field has been modified. -- Chris Lamb Fri, 05 Nov 2021 08:47:27 +0000 diffoscope (189) unstable; urgency=medium [ Chris Lamb ] * Try some alternative suffixes (eg. ".py") to support distributions that strip or retain them. (Closes: reproducible-builds/diffoscope#283) * Skip Python bytecode testing where we do not have an expected diff. (Closes: reproducible-builds/diffoscope#284) * Refactor the find_executable utility into an explicit method. * Split out a custom call to assert_diff to support a .startswith equivalent. * Use skipif instead of manual conditionals in some tests. [ Vagrant Cascadian ] * Add an external tool reference for Guix to support ppudump and dumppdf. [ Sergei Trofimovich ] * Update uImage test output for file(1) version 5.41. [ Jelle van der Waa ] * Add Arch Linux as CI test target. * Add external tools on Arch Linux for ffmpeg, openssl and ocalobjinfo. -- Chris Lamb Fri, 29 Oct 2021 09:47:04 +0100 diffoscope (188) unstable; urgency=medium * Add support for Python Sphinx inventory files, usually named objects.inv. * Fix Python bytecode decompilation tests with Python 3.10+. (Closes: reproducible-builds/diffoscope#278) -- Chris Lamb Fri, 22 Oct 2021 09:08:55 +0100 diffoscope (187) unstable; urgency=medium * Add support for comparing .pyc files. Thanks to Sergei Trofimovich. (Closes: reproducible-builds/diffoscope#278) -- Chris Lamb Fri, 08 Oct 2021 09:01:52 +0100 diffoscope (186) unstable; urgency=medium [ Chris Lamb ] * Don't call close_archive when garbage-collecting Archive instances unless open_archive returned successfully. This prevents, amongst others, an AttributeError traceback due to PGPContainer's cleanup routines assuming that its temporary directory had been created. (Closes: reproducible-builds/diffoscope#276) * Ensure that the string "RPM archives" exists in the package description, regardless of whether python3-rpm is installed or not at build time. [ Jean-Romain Garnier ] * Fix the LVM Macho comparator for non-x86-64 architectures. -- Chris Lamb Fri, 01 Oct 2021 09:02:48 +0100 diffoscope (185) unstable; urgency=medium [ Mattia Rizzolo ] * Fix the autopkgtest in order to fix testing migration: the androguard Python module is not in the python3-androguard Debian package * Ignore a warning in the tests from the h5py package that doesn't concern diffoscope. [ Chris Lamb ] * Bump Standards-Version to 4.6.0. -- Chris Lamb Tue, 21 Sep 2021 16:15:17 +0100 diffoscope (184) unstable; urgency=medium [ Chris Lamb ] * Fix the semantic comparison of R's .rdb files after a refactoring of temporary directory handling in a previous version. * Support a newer format version of R's .rds files. * Update tests for OCaml 4.12. (Closes: reproducible-builds/diffoscope#274) * Move diffoscope.versions to diffoscope.tests.utils.versions. * Use assert_diff in tests/comparators/test_rdata.py. * Reformat various modules with Black. [ Zbigniew Jędrzejewski-Szmek ] * Stop using the deprecated distutils module by adding a version comparison class based on the RPM version rules. * Update invocations of llvm-objdump for the latest version of LLVM. * Adjust a test with one-byte text file for file(1) version 5.40. * Improve the parsing of the version of OpenSSH. [ Benjamin Peterson ] * Add a --diff-context option to control the unified diff context size. (reproducible-builds/diffoscope!88) -- Chris Lamb Fri, 17 Sep 2021 09:10:38 +0100 diffoscope (183) unstable; urgency=medium [ Chris Lamb ] * Add support for extracting Android signing blocks. (Closes: reproducible-builds/diffoscope#246) * Format debug messages for elf sections using our diffoscope.utils.format_class utility. * Clarify a comment about the HUGE_TOOLS dict in diffoscope.external_tools. [ Felix C. Stegerman ] * Clarify output around APK Signing Blocks and remove an accidental duplicate "0x" prefix. -- Chris Lamb Fri, 03 Sep 2021 11:14:32 +0100 diffoscope (182) unstable; urgency=medium [ Chris Lamb ] * Also ignore, for example, spurious "fwGCC: (Debian ... )" lines in output from strings(1). * Only use "java -jar /path/to/apksigner.jar" if we have a .jar as newer versions of apksigner use a shell wrapper script which will obviously be rejected by the JVM. Also mention in the diff if apksigner is missing. * Pass "-f" to apktool to avoid creating a strangely-named subdirectory and to simplify code. * If we specify a suffix for temporary file or directory, ensure it starts with a "_" to make the generated filenames more human-readable. * Drop an unused File import. * Update the minimum version of the Black source code formatter. [ Santiago Torres Arias ] * Support parsing the return value of squashfs versions which discriminate between fatal and non-fatal errors. -- Chris Lamb Fri, 27 Aug 2021 09:11:44 +0100 diffoscope (181) unstable; urgency=medium [ Chris Lamb ] * New features and bug fixes: - Don't require apksigner in order to compare .apk files using apktool. - Add a special-case to squshfs image extraction to not fail if we aren't root/superuser. (Closes: #991059) - Reduce the maximum line length to avoid O(n^2) Wagner-Fischer algorithm, which meant that diff generation took an inordinate amount of time. (Closes: reproducible-builds/diffoscope#272) - Include profiling information in --debug output if --profile is not set. - Don't print an orphan newline when the Black source code formatter self-test passes. * Tests: - Update test to check specific contents of squashfs listing, otherwise it fails depending on the test systems uid-to-username mapping in passwd(5). - Assign "seen" and "expected" values to local variables to improve contextual information in/around failed tests. * Misc changes: - Print the size of generated HTML, text (etc.) reports. - Profile calls to specialize and diffoscope.diff.linediff. - Update various copyright years. -- Chris Lamb Fri, 20 Aug 2021 10:03:35 +0100 diffoscope (180) unstable; urgency=medium * Don't include specific ".debug"-like lines in the output, as it invariably a duplicate of the debug ID that exists in a better form in the readelf(1) differences for this file. * Also ignore include short "GCC" lines that differs on a single prefix byte too. These are distracting, not very useful and are simply the strings(1) command's idea of the build ID, which, again, is displayed nearby in the file's diff. * Update the invocation arguments and tests for the latest version of odt2txt. -- Chris Lamb Fri, 06 Aug 2021 13:57:31 +0100 diffoscope (179) unstable; urgency=medium * Ensure that various LLVM tools are installed, even when testing whether a MacOS binary has zero differences when compared to itself. (Closes: reproducible-builds/diffoscope#270) -- Chris Lamb Fri, 30 Jul 2021 09:05:01 +0100 diffoscope (178) unstable; urgency=medium [ Chris Lamb ] * Don't traceback on an broken symlink in a directory. (Closes: reproducible-builds/diffoscope#269) * Rewrite the calculation of a file's "fuzzy hash" to make the control flow cleaner. [ Balint Reczey ] * Support .deb package members compressed with the Zstandard algorithm. (LP: #1923845) [ Jean-Romain Garnier ] * Overhaul the Mach-O executable file comparator. * Implement tests for the Mach-O comparator. * Switch to new argument format for the LLVM compiler. * Fix test_libmix_differences in testsuite for the ELF format. * Improve macOS compatibility for the Mach-O comparator. * Add llvm-readobj and llvm-objdump to the internal EXTERNAL_TOOLS data structure. [ Mattia Rizzolo ] * Invoke gzip(1) with the short option variants to support Busybox's gzip. (Closes: reproducible-builds/diffoscope#268) -- Chris Lamb Fri, 16 Jul 2021 15:37:59 +0100 diffoscope (177) unstable; urgency=medium [ Keith Smiley ] * Improve support for Apple "provisioning profiles". * Fix ignoring objdump tests on MacOS. -- Chris Lamb Fri, 04 Jun 2021 10:03:04 +0100 diffoscope (176) unstable; urgency=medium * Update ffmpeg tests to work with ffmpeg 4.4. (Closes: reproducible-builds/diffoscope#258) -- Chris Lamb Fri, 28 May 2021 10:52:25 +0100 diffoscope (175) unstable; urgency=medium * Use the actual filesystem path name (instead of diffoscope's concept of the source name) to correct APK filename filtering when an APK file is in another container -- we need to filter the auto-generated "1.apk" instead of "original-filename.apk". (Closes: reproducible-builds/diffoscope#255) * Don't call os.path.basename twice. * Correct grammar in a fsimage.py debug message. * Add a comment about stripping filenames. -- Chris Lamb Fri, 14 May 2021 10:43:10 +0100 diffoscope (174) unstable; urgency=medium [ Chris Lamb ] * Check that we are parsing an actual Debian .buildinfo file, not just a file with that extension. (Closes: #987994, reproducible-builds/diffoscope#254) * Support signed .buildinfo files again -- file(1) reports them as "PGP signed message". [ Mattia Rizzolo ] * Make the testsuite pass with file(1) version 5.40. * Embed some short test fixtures in the test code itself. * Fix recognition of compressed .xz files with file(1) 5.40. -- Chris Lamb Fri, 07 May 2021 10:48:54 +0100 diffoscope (173) unstable; urgency=medium [ Chris Lamb ] * Add support for showing annotations in PDF files. (Closes: reproducible-builds/diffoscope#249) * Move to assert_diff in test_pdf.py. [ Zachary T Welch ] * Difference.__init__: Demote unified_diff argument to a Python "kwarg". -- Chris Lamb Fri, 30 Apr 2021 08:51:21 +0100 diffoscope (172) unstable; urgency=medium * If zipinfo(1) shows a difference but we cannot uncover a difference within the underlying .zip or .apk file, add a comment and show the binary comparison. (Closes: reproducible-builds/diffoscope#246) * Make "error extracting X, falling back to binary comparison E" error message nicer. -- Chris Lamb Fri, 02 Apr 2021 08:49:24 +0100 diffoscope (171) unstable; urgency=medium [ Mattia Rizzolo ] * Do not list as a "skipping reason" tools that do exist. * Drop the "compose" tool from the list of required tools for these tests, since it doesn't seem to be required. -- Chris Lamb Fri, 26 Mar 2021 09:35:37 +0000 diffoscope (170) unstable; urgency=medium [ Chris Lamb ] * Avoid frequent long lines in RPM header outputs that cause very very slow HTML outputs. (Closes: reproducible-builds/diffoscope#245) * Fix test_libmix_differences on openSUSE Tumbleweed. (Closes: reproducible-builds/diffoscope#244) * Move test_rpm to use the assert_diff utility helper. [ Hans-Christoph Steiner ] * Add a diffoscope.tools.get_tools() method to support programmatically fetching Diffoscope's config. [ Roland Clobus ] * Become tolerant of malformed Debian .changes files. -- Chris Lamb Fri, 19 Mar 2021 10:30:53 +0000 diffoscope (169) unstable; urgency=medium [ Chris Lamb ] * Optimisations: - Use larger buffer/block sizes when extracting files from libarchive- based archives. - Use a much-shorter CSS class (instead of "diffponct") to dramatically reduce uncompressed HTML output. * Logging improvements: - Don't emit "Unable to stat file" warning/debug messages; we have entirely-artificial directory entries such as ELF sections which, of course, never exist as filesystem files. - Don't emit a "Returning a FooContainer" logging message - we already emit "Instantiating a FooContainer" one and are unlikely to fail in the middle. - Format the report size logging messages when generating HTML reports. - Add the target directory when logging which directory we are extracting containers to. * Miscellaneous: - Ignore "--debug" and similar arguments when creating a (hopefully useful) temporary directory. - Ensure all internal temporary directories have useful names. - Clarify a comment regarding diffoscope not extracting excluded files. [ Vagrant Cascadian ] * Skip a DEX-related test if the "procyon" tool is unavailable. -- Chris Lamb Fri, 12 Mar 2021 15:33:54 +0000 diffoscope (168) unstable; urgency=medium * Don't call difflib.Differ.compare with very large inputs; it is at least O(n^2) and makes diffoscope appear to hang. (Closes: reproducible-builds/diffoscope#240) * Don't use "Inheriting PATH of X" in debug log message; use "PATH is X". * Correct the capitalisation of jQuery. -- Chris Lamb Sat, 27 Feb 2021 09:34:37 +0000 diffoscope (167) unstable; urgency=medium * Temporary directory handling: - Ensure we cleanup our temporary directory by avoiding confusion between the TemporaryDirectory instance and the underlying directory. (Closes: #981123) - Use a potentially-useful suffix to our temporary directory based on the command-line passed to diffoscope. - Fix some tempfile/weakref interaction in Python 3.7 (ie. Debian buster). (Closes: reproducible-builds/diffoscope#239) - If our temporary directory does not exist anymore (eg. it has been cleaned up in tests, signal handling or reference counting), make sure we recreate it. * Bug fixes: - Don't rely on magic.Magic(...) to have an identical API between file's magic.py and PyPI's "python-magic" library. (Closes: reproducible-builds/diffoscope#238) - Don't rely on dumpimage returning an appropriate exit code; check that the file actually exists after we call it. * Codebase changes: - Set a default Config.extended_filesystem_attributes. - Drop unused Config.acl and Config.xattr attributes. - Tidy imports in diffoscope/comparators/fit.py. * Tests: - Add u-boot-tools to test dependencies so that salsa.debian.org pipelines actually test the new FIT comparator. - Strip newlines when determining Black version to avoid "requires black >= 20.8b1 (18.9b0\n detected)" in test output (NB. embedded newline). - Gnumeric is back in testing so re-add to test dependencies. - Use assert_diff (over get_data, etc.) in the FIT and APK comparators. - Mark test_apk.py::test_android_manifest as being allowed to fail for now. - Fix the FIT tests in buster and unstable. -- Chris Lamb Fri, 19 Feb 2021 09:06:42 +0000 diffoscope (166) unstable; urgency=medium [ Chris Lamb ] * New features and bugfixes: - Explicitly remove our top-level temporary directory. (Closes: #981123, reproducible-builds/diffoscope#234) - Increase fuzzy matching threshold to 130 ensure that we show more differences. (Closes: reproducible-builds/diffoscope#232) - Save our sys.argv in our top-level temporary directory in case it helps debug current/errant temporary directories. - Prefer to use "magic.Magic" over the "magic.open" compatibility interface. (Closes: reproducible-builds/diffoscope#236) - Reduce fuzzy threshold to 110 to prevent some test failures. (Closes: reproducible-builds/diffoscope#233) * Output improvements: - Show fuzzyness amount in percentage terms, not out of the rather-arbitrary "400". - Improve the logging of fuzzy matching. - Print the free space in our temporary directory when we create it, not from within diffoscope.main. * Codebase improvements: - Tidy the diffoscopecomparators.utils.fuzzy module. - Update my copyright years. - Clarify the grammar of a comment. - Clarify in a comment that __del__ is not always called, so temporary directories are not neccessary removed the *moment* they go out of scope. [ Conrad Ratschan ] * Fix U-Boot Flattened Image Tree ("FIT") image detection for larger image files. (MR: reproducible-builds/diffoscope!75) -- Chris Lamb Sat, 30 Jan 2021 12:31:22 +0000 diffoscope (165) unstable; urgency=medium [ Dimitrios Apostolou ] * Introduce the --no-acl and --no-xattr arguments [later collapsed to --extended-filesystem-attributes] to improve performance. * Avoid calling the external stat command. [ Chris Lamb ] * Collapse --acl and --xattr into --extended-filesystem-attributes to cover all of these extended attributes, defaulting the new option to false (ie. to not check these very expensive external calls). [ Mattia Rizzolo ] * Override several lintian warnings regarding prebuilt binaries in the * source. * Add a pytest.ini file to explicitly use Junit's xunit2 format. * Ignore the Python DeprecationWarning message regarding the `imp` module deprecation as it comes from a third-party library. * debian/rules: filter the content of the d/*.substvars files -- Chris Lamb Sat, 23 Jan 2021 08:57:13 +0000 diffoscope (164) unstable; urgency=medium [ Chris Lamb ] * Truncate jsondiff differences at 512 bytes lest they consume the entire page. * Wrap our external call to cmp(1) with a profile (to match the internal profiling). * Add a note regarding the specific ordering of the new all_tools_are_listed test. [ Dimitrios Apostolou ] * Performance improvements: - Improve speed of has_same_content by spawning cmp(1) less frequently. - Log whenever the external cmp(1) command is spawn.ed - Avoid invoking external diff for identical, short outputs. * Rework handling of temporary files: - Clean up temporary directories as we go along, instead of at the end. - Delete FIFO files when the FIFO feeder's context manager exits. [ Mattia Rizzolo ] * Fix a number of potential crashes in --list-debian-substvars, including explicitly listing lipo and otool as external tools. - Remove redundant code and let object destructors clean up after themselves. [ Conrad Ratschan ] * Add a comparator for Flattened Image Trees (FIT) files, a boot image format used by U-Boot. -- Chris Lamb Fri, 08 Jan 2021 13:00:55 +0000 diffoscope (163) unstable; urgency=medium [ Chris Lamb ] * Bug fixes & new features: - Normalise "ret" to "retq" in objdump output to support multiple versions of obdump(1). (Closes: #976760, reproducible-builds/diffoscope#227) - Don't show progress indicators when running zstd(1). (Closes: reproducible-builds/diffoscope#226) - Move the slightly-confusing behaviour of loading an "existing diff" if a single file is passed to diffoscope to the new --load-existing-diff command. * Output improvements: - Use pprint.pformat in the JSON comparator to serialise the differences from jsondiff to make the output render better. - Correct tense in --debug log output. * Code quality: * Don't use an old-style "super" call. - Rewrite the filter routine for post-processing output from readelf(1). - Update my copyright years. - Remove unncessary PEP 263 encoding lines (replaced via PEP 3120). - Use "minimal" instead of "basic" as a variable name to match the underlying package name. - Add comment regarding Java tests for diffoscope contributors who are not using Debian. (Re: reproducible-builds/diffoscope!58) * Debian packaging: - Update debian/copyright to match copyright notices in source-tree. (Closes: reproducible-builds/diffoscope#224) - Ensure the new "diffoscope-minimal" package has a different short description from the main "diffoscope" one. [ Jean-Romain Garnier ] * Add tests for OpenJDK 14. [ Conrad Ratschan ] * Add comparator for "legacy" uboot uImage files. (MR: reproducible-builds/diffoscope!69) -- Chris Lamb Fri, 11 Dec 2020 10:07:28 +0000 diffoscope (162) unstable; urgency=medium [ Chris Lamb ] * Don't depends on radare2 in the Debian autopkgtests as it will not be in bullseye due to security considerations (#950372). (Closes: #975313) * Avoid "Command `s p a c e d o u t` failed" messages when creating an artificial CalledProcessError instance in our generic from_operation feeder creator. * Overhaul long and short descriptions. * Use the operation's full name so that "command failed" messages include its arguments. * Add a missing comma in a comment. [ Jelmer Vernooij ] * Add missing space to the error message when only one argument is passed to diffoscope. [ Holger Levsen ] * Update Standards-Version to 4.5.1. [ Mattia Rizzolo ] * Split the diffoscope package into a diffoscope-minimal package that excludes the larger packages from Recommends. (Closes: #975261) * Drop support for Python 3.6. -- Chris Lamb Fri, 27 Nov 2020 09:41:32 +0000 diffoscope (161) unstable; urgency=medium [ Chris Lamb ] * Fix failing testsuite: (Closes: #972518) - Update testsuite to support OCaml 4.11.1. (Closes: #972518) - Reapply Black and bump minimum version to 20.8b1. * Move the OCaml tests to the assert_diff helper. [ Jean-Romain Garnier ] * Add support for radare2 as a disassembler. [ Paul Spooren ] * Automatically deploy Docker images in the continuous integration pipeline. -- Chris Lamb Tue, 20 Oct 2020 15:52:28 +0100 diffoscope (160) unstable; urgency=medium * Check that pgpdump is actually installed before attempting to run it. Thanks to Gianfranco Costamagna (locutusofborg). (Closes: #969753) * Add some documentation for the EXTERNAL_TOOLS dictionary. * Ensure we check FALLBACK_FILE_EXTENSION_SUFFIX, otherwise we run pgpdump against all files that are recognised by file(1) as "data". -- Chris Lamb Fri, 11 Sep 2020 10:08:38 +0100 diffoscope (159) unstable; urgency=medium [ Chris Lamb ] * Show "ordering differences only" in strings(1) output. (Closes: reproducible-builds/diffoscope#216) * Don't alias output from "os.path.splitext" to variables that we do not end up using. * Don't raise exceptions when cleaning up after a guestfs cleanup failure. [ Jean-Romain Garnier ] * Make "Command" subclass a new generic Operation class. -- Chris Lamb Fri, 04 Sep 2020 11:12:52 +0100 diffoscope (158) unstable; urgency=medium * Improve PGP support: - Support extracting of files within PGP signed data. (Closes: reproducible-builds/diffoscope#214) - pgpdump(1) can successfully parse some unrelated, non-PGP binary files, so check that the parsed output contains something remotely sensible before identifying it as a PGP file. * Don't use Python's repr(...)-style output in "Calling external command" logging output. * Correct a typo of "output" in an internal comment. -- Chris Lamb Fri, 28 Aug 2020 11:53:10 +0100 diffoscope (157) unstable; urgency=medium [ Chris Lamb ] * Try obsensibly "data" files named .pgp against pgpdump to determine whether they are PGP files. (Closes: reproducible-builds/diffoscope#211) * Don't raise an exception when we encounter XML files with "" declarations inside the DTD, or when a DTD or entity references an external resource. (Closes: reproducible-builds/diffoscope#212) * Temporarily drop gnumeric from Build-Depends as it has been removed from testing due to Python 2.x deprecation. (Closes: #968742) * Codebase changes: - Add support for multiple file extension matching; we previously supported only a single extension to match. - Move generation of debian/tests/control.tmp to an external script. - Move to our assert_diff helper entirely in the PGP tests. - Drop some unnecessary control flow, unnecessary dictionary comprehensions and some unused imports found via pylint. * Include the filename in the "... not identified by any comparator" logging message. -- Chris Lamb Fri, 21 Aug 2020 12:24:25 +0100 diffoscope (156) unstable; urgency=medium [ Chris Lamb ] * Update PPU tests for compatibility with Free Pascal versions 3.2.0 or greater. (Closes: #968124) * Emit a debug-level logging message when our ppudump(1) version does not match file header. * Add and use an assert_diff helper that loads and compares a fixture output to avoid a bunch of test boilerplate. [ Frazer Clews ] * Apply some pylint suggestions to the codebase. -- Chris Lamb Fri, 14 Aug 2020 10:04:26 +0100 diffoscope (155) unstable; urgency=medium [ Chris Lamb ] * Bump Python requirement from 3.6 to 3.7 - most distributions are either shipping3.5 or 3.7, so supporting 3.6 is not somewhat unnecessary and also more difficult to test locally. * Improvements to setup.py: - Apply the Black source code reformatter. - Add some URLs for the site of PyPI.org. - Update "author" and author email. * Explicitly support Python 3.8. [ Frazer Clews ] * Move away from the deprecated logger.warn method logger.warning. [ Mattia Rizzolo ] * Document ("classify") on PyPI that this project works with Python 3.8. -- Chris Lamb Fri, 07 Aug 2020 12:09:50 +0100 diffoscope (154) unstable; urgency=medium [ Chris Lamb ] * Add support for F2FS filesystems. (Closes: reproducible-builds/diffoscope#207) * Allow "--profile" as a synonym for "--profile=-". * Add an add_comment helper method so don't mess with our _comments list directly. * Add missing bullet point in a previous changelog entry. * Use "human-readable" over unhyphenated version. * Add a bit more debugging around launching guestfs. * Profile the launch of guestfs filesystems. * Correct adding a comment when we cannot extract a filesystem due to missing guestfs module. -- Chris Lamb Fri, 31 Jul 2020 10:26:15 +0100 diffoscope (153) unstable; urgency=medium [ Chris Lamb ] * Drop some legacy argument styles; --exclude-directory-metadata and --no-exclude-directory-metadata have been replaced with --exclude-directory-metadata={yes,no}. * Code improvements: - Make it easier to navigate the main.py entry point. - Use a relative import for get_temporary_directory in diffoscope.diff. - Rename bail_if_non_existing to exit_if_paths_do_not_exist. - Rewrite exit_if_paths_do_not_exist to not check files multiple times. * Documentation improvements: - CONTRIBUTING.md: - Add a quick note about adding/suggesting new options. - Update and expand the release process documentation. - Add a reminder to regenerate debian/tests/control. - README.rst: - Correct URL to build job on Jenkins. - Clarify and correct contributing info to point to salsa.debian.org. -- Chris Lamb Fri, 24 Jul 2020 12:03:51 +0100 diffoscope (152) unstable; urgency=medium [ Chris Lamb ] * Bug fixes: - Don't require zipnote(1) to determine differences in a .zip file as we can use libarchive directly. * Reporting improvements: - Don't emit "javap not found in path" if it is available in the path but it did not result in any actual difference. - Fix "... not available in path" messages when looking for Java decompilers; we were using the Python class name (eg. "") over the actual command we looked for (eg. "javap"). * Code improvements: - Replace some simple usages of str.format with f-strings. - Tidy inline imports in diffoscope.logging. - In the RData comparator, always explicitly return a None value in the failure cases as we return a non-None value in the "success" one. [ Jean-Romain Garnier ] * Improve output of side-by-side diffs, detecting added lines better. (MR: reproducible-builds/diffoscope!64) * Allow passing file with list of arguments to ArgumentParser (eg. "diffoscope @args.txt"). (MR: reproducible-builds/diffoscope!62) -- Chris Lamb Sat, 18 Jul 2020 11:00:13 +0100 diffoscope (151) unstable; urgency=medium [ Chris Lamb] * Improvements and bug fixes: - Pass the absolute path when extracting members from SquashFS images as we run the command with our working directory set to the temporary directory. (Closes: #964365, reproducible-builds/diffoscope#189) - Increase the minimum length of the output from strings(1) to 8 characters to avoid unnecessary diff noise. (Re. reproducible-builds/diffoscope#148) * Logging improvements: - Fix the compare_files message when the file does not have a literal name. - Reduce potential log noise by truncating the has_some_content messages. * Codebase changes: - Clarify use of a "null" diff in order to remember an exit code. - Don't alias a variable when don't end up it; use "_" instead. - Use a "NullChanges" file to represent missing data in the Debian package comparator. - Update some miscellaneous terms. -- Chris Lamb Fri, 10 Jul 2020 10:52:56 +0100 diffoscope (150) unstable; urgency=medium [ Chris Lamb ] * Don't crash when listing entries in archives if they don't have a listed size (such as hardlinks in .ISO files). (Closes: reproducible-builds/diffoscope#188) * Dump PE32+ executables (including EFI applications) using objdump. (Closes: reproducible-builds/diffoscope#181) * Tidy detection of JSON files due to missing call to File.recognizes that checks against the output of file(1) which was also causing us to attempt to parse almost every file using json.loads. (Whoops.) * Drop accidentally-duplicated copy of the new --diff-mask tests. * Logging improvements: - Split out formatting of class names into a common method. - Clarify that we are generating presenter formats in the opening logs. [ Jean-Romain Garnier ] * Remove objdjump(1) offsets before instructions to reduce diff noise. (Closes: reproducible-builds/diffoscope!57) -- Chris Lamb Fri, 03 Jul 2020 11:04:46 +0100 diffoscope (149) unstable; urgency=medium [ Chris Lamb ] * Update tests for file 5.39. (Closes: reproducible-builds/diffoscope#179) * Downgrade the tlsh warning message to an "info" level warning. (Closes: #888237, reproducible-builds/diffoscope#29) * Use the CSS "word-break" property over manually adding U+200B zero-width spaces that make copy-pasting cumbersome. (Closes: reproducible-builds/diffoscope!53) * Codebase improvements: - Drop some unused imports from the previous commit. - Prevent an unnecessary .format() when rendering difference comments. - Use a semantic "AbstractMissingType" type instead of remembering to check for both "missing" files and missing containers. [ Jean-Romain Garnier ] * Allow user to mask/filter reader output via --diff-mask=REGEX. (MR: reproducible-builds/diffoscope!51) * Make --html-dir child pages open in new window to accommodate new web browser content security policies. * Fix the --new-file option when comparing directories by merging DirectoryContainer.compare and Container.compare. (Closes: reproducible-builds/diffoscope#180) * Fix zsh completion for --max-page-diff-block-lines. [ Mattia Rizzolo ] * Do not warn about missing tlsh during tests. -- Chris Lamb Fri, 26 Jun 2020 15:57:41 +0100 diffoscope (148) unstable; urgency=medium [ Daniel Fullmer ] * Fix a regression in the CBFS comparator due to changes in our_check_output. [ Chris Lamb ] * Add a remark in the deb822 handling re. potential security issue in the .changes, .dsc, .buildinfo comparator. -- Chris Lamb Fri, 19 Jun 2020 11:38:20 +0100 diffoscope (147) unstable; urgency=medium [ Chris Lamb ] * New features: - Add output from strings(1) to ELF binaries. It is intended this will expose expose build paths that are hidden somewhere within the objdump(1) output. (Closes: reproducible-builds/diffoscope#148) - Add basic zsh shell tab-completion support. (Closes: reproducible-builds/diffoscope#158) * Bug fixes: - Prevent a traceback when comparing a PDF document that does not contain any metadata, ie. it is missing a PDF "/Info" stanza. (Closes: reproducible-builds/diffoscope#150) - Fix compatibility with jsondiff 1.2.0 which was causing a traceback and log the version of jsondiff we are using to aid debugging in the future. (Closes: reproducible-builds/diffoscope#159 - Fix an issue in GnuPG keybox handling that left filenames in the diff. - Don't mask an existing test name; ie. ensure it is actually run. * Reporting: - Log all calls to subprocess.check_output by using our own wrapper utility. (Closes: reproducible-builds/diffoscope#151) * Code improvements: - Replace references to "WF" with "Wagner-Fischer" for clarity. - Drop a large number of unused imports (list_libarchive, ContainerExtractionError, etc.) - Don't assign exception to a variable that we do not use. - Compare string values with the equality operator, not via "is" identity. - Don't alias an open file to a variable when we don't use it. - Don't alias "filter" builtin. - Refactor many small parts of the HTML generation, dropping explicit u"unicode" strings, tidying the generation of the "Offset X, Y lines modified" messages, moving to PEP 498 f-strings where appropriate, etc. - Inline a number of single-used utility methods. -- Chris Lamb Thu, 11 Jun 2020 12:50:34 +0100 diffoscope (146) unstable; urgency=medium [ Chris Lamb ] * Refactor .changes and .buildinfo handling to show all details (including the GPG header and footer components), even when referenced files are not present. (Closes: reproducible-builds/diffoscope#122) * Normalise filesystem stat(2) "birth times" (ie. st_birthtime) in the same way we do with stat(1)'s "Access:" and "Change:" times to fix a nondetermistic build failure on GNU Guix. (Closes: reproducible-builds/diffoscope#74) * Drop the (default) subprocess.Popen(shell=False) keyword argument so that the more unsafe shell=True is more obvious. * Ignore lower vs. upper-case when ordering our file format descriptions. * Don't skip string normalisation in Black. [ Mattia Rizzolo ] * Add a "py3dist" override for the rpm-python module (Closes: #949598) * Bump the debhelper compat level to 13 and use the new execute_after_*/execture_before_* style rules. * Fix a spelling error in changelog. [ Daniel Fullmer ] * Mount GuestFS filesystem images readonly. [ Jean-Romain Garnier ] * Prevent an issue where (for example) LibarchiveMember's has_same_content method is called regardless of the actual type of file. -- Chris Lamb Sat, 30 May 2020 12:36:51 +0100 diffoscope (145) unstable; urgency=medium [ Chris Lamb ] * Improvements: - Add support for Apple Xcode mobile provisioning .mobilepovision files. (Closes: reproducible-builds/diffoscope#113) - Add support for printing the signatures via apksigner(1). (Closes: reproducible-builds/diffoscope#121) - Use SHA256 over MD5 when generating page names for the HTML directory presenter, validate checksums for files referenced in .changes files using SHA256 too, and move to using SHA256 in "Too much input for diff" output too. (Closes: reproducible-builds/diffoscope#124) - Don't leak the full path of the temporary directory in "Command [..] exited with 1". (Closes: reproducible-builds/diffoscope#126) - Identify "iOS App Zip archive data" files as .zip files. (Closes: reproducible-builds/diffoscope#116) * Bug fixes: - Correct "differences" typo in the ApkFile handler. (Closes: reproducible-builds/diffoscope#127) * Reporting/output improvements: - Never emit the same id="foo" TML anchor reference twice, otherwise identically-named parts will not be able to linked to via "#foo". (Closes: reproducible-builds/diffoscope#120) - Never emit HTML with empty "id" anchor lements as it is not possible to link to "#" (vs "#foo"). We use "#top" as a fallback value so it will work for the top-level parent container. - Clarify the message when we cannot find the "debian" Python module. - Clarify "Command [..] failed with exit code" to remove duplicate "exited with exit" but also to note that diffoscope is interpreting this as an error. - Add descriptions for the 'fallback' Debian module file types. - Rename the --debugger command-line argument to --pdb. * Testsuite improvements: - Prevent CI (and runtime) apksigner test failures due to lack of binfmt_misc on Salsa CI and elsewhere. * Codebase improvements: - Initially add a pair of comments to tidy up a slightly abstraction level violating code in diffoscope.comparators.mising_file and the .dsc/.buildinfo file handling, but replace this later by by inlining MissingFile's special handling of deb822 to prevent leaking through abstraction layers in the first place. - Use a BuildinfoFile (etc.) regardless of whether the associated files such as the orig.tar.gz and the .deb are present, but don't treat them as actual containers. (Re: reproducible-builds/diffoscope#122) - Rename the "Openssl" command class to "OpenSSLPKCS7" to accommodate other commands with this prefix. - Wrap a docstring across multiple lines, drop an inline pprint import and comment the HTMLPrintContext class, etc. [ Emanuel Bronshtein ] * Avoid build-cache in building the released Docker image. (Closes: reproducible-builds/diffoscope#123) [ Holger Levsen ] * Wrap long lines in older changelog entries. -- Chris Lamb Sat, 23 May 2020 09:31:26 +0100 diffoscope (144) unstable; urgency=medium [ Chris Lamb ] * Improvements: - Print the amount of free space that we have available in our temporary directory as a debugging message. - Remove (broken) fuzzy matching of JSON files as file 5.35 (in buster, released 2018-10-18) supports recognising JSON data. (Closes: reproducible-builds/diffoscope#106) - Don't pretty-print the JSON output by default as it will usually be so complicated to be unreadable by the human eye and it can be easily replaced by piping to "| jq". - Don't print a traceback if we pass a single, missing argument to diffoscope (eg. a JSON diff to re-load). * Reporting/output improvements: - Reduce the default number of maximum standard error lines printed from 50 to 25; usually the error is obvious by that point. - Clarify the message when we truncate the number of lines to standard error. - Clarify when an external command emits for both files, otherwise it can look like diffoscope is repeating itself when it is actually being run twice. - Don't repeat "stderr from {}" if both commands emit the same thing. * Dockerfile improvements: - Use ARG instead of ENV for DEBIAN_FRONTEND so we do not set this environment variable at runtime. (Closes: reproducible-builds/diffoscope#103) - Run diffoscope as a non-root user in the runtime container. (Closes: reproducible-builds/diffoscope#102) - Add a .dockerignore file to whitelist files we need in our container. Thanks to Emanuel Bronshtein for the original idea. (Closes: reproducible-builds/diffoscope#105) - Install/remove the build-essential package during build so we can install the recommended packages from Git. * Testsuite improvements: - Include the Black output in the assertion failure too. - Update the Black self-test; we don't care about the length of the black output, rather whether it has some or, preferably, not. * Codebase improvements: - Bump the officially required version of Python from 3.5 to 3.6. (Closes: reproducible-builds/diffoscope#117) - Drop an unused shlex import. - Instruct linters to pass over a bare try-except when cleaning up temporary files used to extract archives. - Format diffoscope/comparators/utils/command.py according to Black 19.10b0-3. - Drop entries from Uploaders that have not uploaded in over three years with esteemed thanks for their previous work. - Drop .travis.yml; we are using Salsa now and likely would not give support for running on Travis CI at this point. [ Jelle van der Waa ] * Update LLVM diff for LLVM version 10. [ Vagrant Cascadian ] * Add external tool reference on openssl for GNU Guix. -- Chris Lamb Thu, 14 May 2020 16:16:06 +0100 diffoscope (143) unstable; urgency=medium * Add support for .p7c and .p7b certificates. (Closes: reproducible-builds/diffoscope#94) * Add "pdftotext" as a requirement to run the PDF test_metadata text. Special thanks to Chocimier/Piotr for the debugging work. (Closes: reproducible-builds/diffoscope#99) * Improve the documentation of FALLBACK_FILE_TYPE_HEADER_PREFIX and FILE_TYPE_HEADER_PREFIX to note that only the first 16 bytes are used. -- Chris Lamb Mon, 27 Apr 2020 13:14:07 +0100 diffoscope (142) unstable; urgency=medium [ Chris Lamb ] * Render multiline difference comments in a way to show indentation. (Closes: reproducible-builds/diffoscope#101) * Correct parsing of "./setup.py test --pytest-args ...". * Capitalise "Ordering differences only" in text comparison comments. * Don't include the JSON similarity percentage if it is 0.0%. [ Ben Hutchings ] * Document how --exclude arguments are matched against filenames. [ Vagrant Cascadian ] * Add external tool reference for h5dump on Guix. -- Chris Lamb Wed, 22 Apr 2020 16:35:14 +0100 diffoscope (141) unstable; urgency=medium [ Chris Lamb ] * Dalvik .dex files can also serve as APK containers. Restrict the narrower identification of .dex files to files ending with this extension, and widen the identification of APK files to when file(1) discovers a Dalvik file. (Closes: #884095, reproducible-builds/diffoscope#28) * Explicitly list python3-h5py in debian/tests/control.in to ensure that we have this module installed during an autopkgtest run to generate the test fixture & regenerate debian/tests/control from debian/tests/control.in to match. * Drop unnecessary and unused assignment to "diff" variable. * Strip paths from the output of zipinfo(1) warnings. (re. reproducible-builds/diffoscope#97) [ Michael Osipov ] * Revert to using zipinfo(1) directly instead of piping input via /dev/stdin for BSD portability. (Closes: reproducible-builds/diffoscope#97) [ Jelle van der Waa ] * Add an external tool for h5dump on Arch. -- Chris Lamb Wed, 15 Apr 2020 19:15:42 +0100 diffoscope (140) unstable; urgency=medium * apktool 2.5.0 changed the handling of output of XML schemas, so update and restrict the corresponding test to match. (Closes: reproducible-builds/diffoscope#96) * Add support for Hierarchical Data Format (HD5) files. (Closes: reproducible-builds/diffoscope#95) -- Chris Lamb Sat, 11 Apr 2020 19:10:57 +0100 diffoscope (139) unstable; urgency=medium [ Mattia Rizzolo ] * Fix Build-Depends on python3-pdfminer. (Closes: #955645) -- Chris Lamb Fri, 03 Apr 2020 21:29:22 +0100 diffoscope (138) unstable; urgency=medium * New features & bug fixes: - Use dumppdf from python3-pdfminer if we do not see any other differences from pdftext, etc. (reproducible-builds/diffoscope#92) - Prevent a traceback when comparing two .rdx files directly as get_member will return a file even if the file is missing and not raise a KeyError exception; we therefore explicitly test for the existence of the file. * Don't allow errors within "Rscript" deserialisation to cause the entire operation to fail (for example if an external library cannot be loaded). (reproducible-builds/diffoscope#91) * Reporting improvements: - Inject the supported file formats into the package long description and remove any duplicate comparator descriptions when formatting. (#reproducible-builds/diffoscope#90) - Print a potentially-helpful message if the PyPDF2 module is not installed (#reproducible-builds/diffoscope#92) - Weaken "Install the X package to get a better output" claim to "... may produce a better output" as this is not guaranteed. * Code improvements: - Inline the RequiredToolNotFound.get_package method; only used once. - Drop deprecated "py36 = [..]" argument in pyproject.toml. - Factor out generation of comparator descriptions. - Add an upstream metadata file. -- Chris Lamb Mon, 30 Mar 2020 16:16:28 +0100 diffoscope (137) unstable; urgency=medium * Also extract classes2.dex, classes3.dex etc. from .apk files. (Closes: reproducible-builds/diffoscope#88) * Add generalised support for "ignoring" returncodes and move special-casing of returncodes in the zip comparator to this. * Accommodate sng returning with a UNIX exit code of 1 even if there minor errors in the file (discovered via #950806). -- Chris Lamb Tue, 18 Feb 2020 17:15:59 +0000 diffoscope (136) unstable; urgency=medium [ Chris Lamb ] * Improvements: - Support external build tools. (Closes: reproducible-builds/diffoscope#87) - Fallback to the regular .zip container format for .apk files if apktool is not available. - Clarify that "install X" in difference comment messages actually refer to a system/distribution package. - Drop the --max-report-size-child and --max-diff-block-lines-parent options; both deprecated and scheduled for removal in January 2018. * Bug fixes: - No longer raise a KeyError exception if we request an invalid member from a directory container. * Logging improvements: - Log a debug-level message if we cannot open a file as container due to a missing tool in order to assist diagnosing issues. - If we don't know the HTML output name, don't emit an enigmatic "html output for" message. - Add the current PATH environment variable to the "Normalising locale..." debug-level message. - Print the "Starting diffoscope $VERSION" line as the first line. - Correct a debug-level message for compare_meta calls to quote the arguments correctly. * Refactoring: - Add support for alternative container types for a file, allowing for runtime (vs import time) control of fallbacks such as adding comments. and append a comment to a difference if we fallback to an inferior container format due to missing a tool. - Factor-out the generation of "foo not available in path" difference comment messages as a helper method in the exception that represents them. * Code improvements: - Tidy diffoscope.main's configure method, factoring out the set of the Config() global out of the run_diffoscope method and inlining the functionality of maybe_set_limit, etc. - Rename diffoscope.locale module to diffoscope.environ as we are modifying things beyond just the locale (eg. calling tzset(), etc.) - Drop unused "Difference" import from the APK comparator. - Drop an assertion that is guaranteed by parallel "if" conditional. - Add a "noqa" line to avoid a false-positive flake8 "unused import" warning. - Turn down the "volume" for a recommendation in a comment. * Release/source-code management: - Add a .git-blame-ignore-revs file to improve the output of git-blame(1) by ignoring large changes when introducing the Black source code reformatter and update the CONTRIBUTING.md guide on how to optionally use it locally. - Convert CONTRIBUTING.rst to CONTRIBUTING.md and include it in the PyPI.org release. * Test improvements - Refresh and update the fixtures for the .ico tests to match the latest version of Imagemagick in Debian unstable. [ Holger Levsen ] * Bump Standards Version to 4.5.0, no changes needed. [ Marc Herbert ] * Search for expected keywords in the output of cbfstool tests and not a specific output. (Closes: reproducible-builds/diffoscope!42) -- Chris Lamb Fri, 24 Jan 2020 15:38:57 +0000 diffoscope (135) unstable; urgency=medium [ Chris Lamb ] * Extract resources.arsc files (as well as classes.dex) from Android .apk files to ensure that we at least show differences there. (Closes: #916359) * Always pass a filename with a ".zip" extension to zipnote otherwise it will return with an exit code of 9 and revert to a binary diff for the entire file. * Don't exhaustively output the entire HTML report when testing the regression for #875281; parsing the JSON and pruning the tree should be enough. * Factor out running all of our zipinfo variants into a new method and use this routine when comparing .apk files, thus now also displaying any differences exposed by bsdtar. * Testsuite improvements: - Always allow fixtures called "output*". - Actually test the output of the test_html_regression_875281 test. * Add a note to the "Contributing" page to suggest enable concurrency when running the tests locally. [ Marc Herbert ] * Fix a misplaced debug "Loading diff from stdin" logging message. * Add new "Testing" section to CONTRIBUTING.rst. -- Chris Lamb Tue, 14 Jan 2020 14:26:07 +0000 diffoscope (134) unstable; urgency=medium [ Chris Lamb ] * Ensure that autopkgtests are run with our pyproject.toml present for the correct "black" source code formatter settings. (Closes: #945993) * Tidy some unnecessary boolean logic in the ISO9660 tests. * Rename the "text_option_with_stdiout" to "text_option_with_stdout". * Include the libarchive file listing for ISO images to ensure that timestamps (not just dates) are visible in any difference. (Closes: reproducible-builds/diffoscope#81) [ Eli Schwartz ] * Fix an exception in the progressbar handler. [ Vagrant Cascadian ] * Add an external tool reference for zstd on GNU Guix. -- Chris Lamb Fri, 27 Dec 2019 19:17:16 +0000 diffoscope (133) unstable; urgency=medium * Correct the substitution/filtering of paths in ELF output to avoid unnecessary differences depending on the path name provided on the commandline. (Closes: #945572) * Silence/correct a SyntaxWarning message due to incorrectly comparing an integer by identity (is) over equality (==). (Closes: #945531) -- Chris Lamb Thu, 28 Nov 2019 10:03:33 +0000 diffoscope (132) unstable; urgency=medium * Allow all possible .zip variations to return with non-zero exit codes, not just known types we can explicitly identify (eg. Java .jmod and .jar files). (Closes: reproducible-builds/diffoscope#78) * Also permit UTF-8 encoded .dsc and .changes files. (Re: reproducible-builds/diffoscope#78) * Rework a long string of "or" statements into a loop with a "break". -- Chris Lamb Tue, 26 Nov 2019 10:31:44 +0000 diffoscope (131) unstable; urgency=medium * Clarify in the HTML and text outputs that the limits are per output format, not global. (Closes: #944882, reproducible-builds/diffoscope#76) - Bump the previous "max_page_size" limit from 400 kB to 4 MB. * Update an Android manifest test to reflect that parsed XML attributes are returned in a sorted or otherwise novel manner under Python 3.8. * Update code to reflect version 19.10b0 of the black source code reformatter and don't run our self-test for versions earlier than this as it will generate different results and thus fail. * Limit .dsc and .buildinfo matching; don't attempt to compare them as ELF sections or similar. (Closes: reproducible-builds/diffoscope#77) * Add a comment that the text_ascii{1,2} test fixture files are used in multiple places so are not trivial to generate on the fly. -- Chris Lamb Sat, 23 Nov 2019 15:22:51 -0500 diffoscope (130) unstable; urgency=medium [ Chris Lamb ] * debian/tests/basic-command-line: Move from deprecated ADTTMP to AUTOPKGTEST_TMP. * Correct the matching of R .rds files by also detecting newer versions of this file format. * Drop unused BASE_DIR global in the tests. * Try and ensure that new test data files are generated dynamically, ie. at least no new ones are added without "good" reasons. * Truncate the tcpdump expected diff to 8KB (from ~600KB). * Refresh OCaml test fixtures to support OCaml >= 4.08.1. Closes: #944709 * Correct reference to the ".rdx" extension in a comment. * Update XML test for Python 3.8+. * Don't use line-base dbuffering when communucating with subprocesses in binary mode. (Closes: reproducible-builds/diffoscope#75) [ Jelle van der Waa ] * Add support for comparing .zst files are created by zstd. (Closes: reproducible-builds/diffoscope!34) -- Mattia Rizzolo Thu, 14 Nov 2019 11:22:20 +0100 diffoscope (129) unstable; urgency=medium * Call R's "deparse" function to ensure that we do not error out and revert to a binary diff when processing .rdb files with internal "vector" types as they do not automatically coerce to strings. * Add the ability to pass Python bytestrings to external commands and pass our long script to parse R .rdb files using this new method over a long command-line argument * Use Rscript's --vanilla option over --no-environ as this also enables --no-save, --no-restore, --no-site-file and --no-init-file. * Improve command-line error messages: - Split out formatting into a separate utility function. - Truncate very long lines when displaying them as an external source of data. - When printing an error from a command, format the command for the user. * Use "exit code" over "return code" when referring to UNIX error codes in displayed differences. -- Chris Lamb Mon, 28 Oct 2019 11:15:18 +0000 diffoscope (128) unstable; urgency=medium * Query the container for the full path of the parallel R .rdx file for a .rdb file as well as looking in the same directory. This ensures that comparing two .deb/.changes files shows a varying path introduced in version 127. -- Chris Lamb Fri, 25 Oct 2019 09:21:40 +0100 diffoscope (127) unstable; urgency=medium [ Chris Lamb ] * Move build-dependency on python-argcomplete to the Python 3.x version to facilitate Python 2.x removal. (Closes: #942967) * Overhaul the handling of GNU R .rdb files: - Rework and refactor the handling of .rdb files specifically with respect to locating the parallel .rdx file prior to inspecting the file to ensure that we do not add files to the user's filesystem in the case of directly comparing two .rdb files or, worse, overwriting a file in is place. - Use a ("""-formatted) docstring for our internal R script to dump variables. - Mask/hide standard errors; R will often produce noisy output that is not useful to us. - Don't include a useless and misleading "NULL". - Include all R object names are displayed, including ones beginning with a fullstop ("."). - Sort package fields when dumping data for output stability. - Format package contents as "foo = bar" rather than using ugly and misleading brackets. - Include the object's type when dumping package contents. - Never read the site or user's R location environment configuration to ensure output stability. - Expose absolute paths in the semantic/human-readable output, preventing falling back to a useless hexdump. * Improve the formatting of command lines: - Ensure newlines and other metacharacters appear escaped as "\n", etc. - Use the newline (etc.) escaped version of the commandline being executed in logging/debug output. - When displaying standard error, ensure use the escaped version too. * Add support for easily masking the standard error of commands and use this in the ffprobe comparator. * To match the libarchive container, raise a KeyError exception if we request an invalid member from a directory container. * Correct string representation output in the traceback when we cannot locate a specific item in a container. [ Mattia Rizzolo ] * Run Debian autopkgtests against all Python versions. -- Chris Lamb Thu, 24 Oct 2019 15:59:34 +0100 diffoscope (126) unstable; urgency=medium [ Chris Lamb ] * Track and report on missing Python modules. (Closes: reproducible-builds/diffoscope#72) * Drop some unnecessary control flow. * Drop explicit inheriting from 'object' class; unnecessary in Python 3. * Bump Standards-Version to 4.4.1. [ Mattia Rizzolo ] * Exit with 2 instead of 1 in case of no disk space. -- Chris Lamb Mon, 14 Oct 2019 12:29:47 -0700 diffoscope (125) unstable; urgency=medium * The test_libmix_differences ELF test requires xxd. (Closes: #940645) -- Chris Lamb Wed, 18 Sep 2019 11:44:57 +0200 diffoscope (124) unstable; urgency=medium [ Chris Lamb ] * Also conditionally skip the identification and "no differences" tests as we require the Ocaml compiler to be present when building the test files themselves. (Closes: #940471) [ Mattia Rizzolo ] * Permit all sorts of version suffixes when checking the Debian package version matches setup.py, not just for backports. (Closes: #939387) [ Vagrant Cascadian ] * Add external tools on GNU Guix for odt2txt, sng and pgpdump. [ Marc Herbert ] * Remove StaticLibFile in the ELF comparator -- ArFile.compare_details() is superior. (Closes: reproducible-builds/diffoscope#64) -- Chris Lamb Tue, 17 Sep 2019 12:19:07 +0200 diffoscope (123) unstable; urgency=medium [ Chris Lamb ] * Build OCaml test input files on-demand rather than shipping them with the package in order to prevent test failures with OCaml 4.08. (Closes: #939386) * Update test skipping messages: - When skipping tests due to the lack of installed tool, print the package that may provide it. - Update "requires foo module" messages to clarify that they are regarding Python modules, not packages. * Rebuild the test squashfs images to exclude the character device as they requires root or fakeroot to extract. (reproducible-builds/diffoscope#65) * Remove accidentally-committed test fixture generation code from tests. * Set long_description_content_type to 'text/x-rst' in setup.py to appease the PyPI.org linter. [ Mattia Rizzolo ] * Don't crash when the progress is requested but the Python module is missing. (Closes: #939085) [ Vagrant Cascadian ] * Add external tools for gifbuild, javap and kbxutil in GNU Guix. [ Marc Herbert ] * In the cbfs tests, add a "Comp" column test to support Coreboot utils > 4.6. -- Chris Lamb Sat, 07 Sep 2019 14:16:31 +0100 diffoscope (122) unstable; urgency=medium [ Chris Lamb ] * Apply patch from László Böszörményi to update the squashfs test output and bump the required version for the test itself. (Closes: #935684) * Skip calls to unsquashfs when we are not root or fakeroot. (Re: reproducible-builds/diffoscope#63) * Include either standard error or standard output (and not just the latter) if/when an external command fails. * Fix a few unicode/bytes issues: - Avoid a possible traceback caused by a str/bytes confusion when handling the output of failing external commands. - Ensure that all of our artificially-created subprocess.CalledProcessError instances have `output` instances that are bytes objects, not str. * Improve debugging output: * Add the containing module name to the (eg.) "Using StaticLibFile for ..." * Improve and condense output when creating our Comparator object types. * Correct a reference to `parser.diff` as `diff` in this context is a Python function in the module, not the actual output returned from diff(1). * Add the "wabt" Debian package to the test dependencies so that we run the wasm tests. [ Mattia Rizzolo ] * Now that we test-require wabt, expect that its tools to be available during autopkgtests. -- Chris Lamb Fri, 30 Aug 2019 08:52:25 +0100 diffoscope (121) unstable; urgency=medium * Don't fallback to a (useless) raw hexdump when readelf(1) reports an minor issue in a section in an ELF binary. For example, when the "frames" section is of the "NOBITS" type, its contents are apparently "unreliable" and thus readelf(1) exits with a returncode of 1. (Closes: #849407, #931962) * Add support to Difference.from_command_exc and friends to optionally ignore specified returncodes from the called program and treat them as "no" difference. * Simplify the parsing of the optional "command_args" argument to the from_command and from_command_exc methods in the Difference class. -- Chris Lamb Fri, 16 Aug 2019 09:49:07 -0700 diffoscope (120) unstable; urgency=medium * No-change sourceful after accidentally uploading binaries in order to migration to testing. -- Chris Lamb Mon, 29 Jul 2019 17:17:46 -0300 diffoscope (119) unstable; urgency=medium [ Chris Lamb ] * If a command fails to execute but does not print anything to standard error, try and include the first line of standard output in the message we include in the diff. This was motivated by readelf(1) returning its error messages on stdout. (Closes: #931963) * Add support for Java ".jmod" modules. Not all versions of file(1) support detection of Jmod files yet, so we perform a manual comparison instead. (Closes: #933308) * Re-add "return code" noun to "Command `foo` exited with X" error messages. * Factor out the ability to ignore the exit codes of "zipinfo" and "zipinfo -v" in the presence of non-standard headers. * Only override the exit code from our special-cased calls to zipinfo(1) if they are 1 or 2 to avoid potentially masking real errors. * Also add missing textual description entries for ZipFile and MozillaZipFile file types. * Skip extra newline in "Output:\n". * Cease ignoring test failures in stable-backports. * Bump debhelper compat level to 12. [ Marc Herbert ] * Catch failures to disassemble and rescue all other differences. (Closes: reproducible-builds/diffoscope!29) -- Chris Lamb Mon, 29 Jul 2019 15:22:24 -0300 diffoscope (118) unstable; urgency=medium * Don't fail in autopkgtests when, for example, we do not have sufficiently newer or older version of file. (Closes: #931881) * Also include python3-tlsh in test dependencies. * Tidy handling of DIFFOSCOPE_FAIL_TESTS_ON_MISSING_TOOLS: - Merge two previously-overlapping environment variables. - Use repr(..)-style output when printing test status. - Add some explicit return values to appease pylint. -- Chris Lamb Sat, 13 Jul 2019 10:23:29 -0300 diffoscope (117) unstable; urgency=medium [ Chris Lamb ] * Add support for file 5.37. Thanks again to Christoph Biedl for the heads-up in advance. (Closes: reproducible-builds/diffoscope/#57) * Apply patch from Gianfranco Costamagna to fix autopkgtest failures in Debian. (Closes: #931709) [ Holger Levsen ] * debian/control: Bump Standards-Version to 4.4.0. -- Chris Lamb Tue, 09 Jul 2019 10:24:54 -0300 diffoscope (116) unstable; urgency=medium [ Chris Lamb ] * Upload to unstable after the release of Debian "buster". * README.rst & manual page generation: - Strip out manpage-only parts of the README rather than using "only" reStructuredText directives in order to support the demands of the latest PyPI website. - Use "real" reStructuredText comments instead of using the "raw" directive. * Dockerfile: - Build the Docker image from the current Git checkout, not the Debian archive. (reproducible-builds/diffoscope#56) - Use the ENTRYPOINT directive with the JSON syntax instead so we pass all arguments to the underlying diffoscope executable. * Document that run_diffoscope should not be considered a stable API. [ Mattia Rizzolo ] * Rename a test function to prevent shadowing a previous one with the same name. * Add ffmpeg to the list of Debian build-dependencies for the testsuite. [ Vagrant Cascadian ] * Add support for known external tools in GNU Guix. -- Chris Lamb Sun, 07 Jul 2019 11:54:29 -0300 diffoscope (115) experimental; urgency=medium [ Chris Lamb ] * Fix execution of symbolic links that point to the "bin/diffoscope" entry point by fully resolving the location. * Add a Dockerfile. * Update contributing messages with instructions regarding the Docker image. [ Mattia Rizzolo ] * tests: + Allow specifying which tools are known missing using DIFFOSCOPE_TESTS_MISSING_TOOLS, to override a _FAIL_ON_MISSING_TOOLS. + With DIFFOSCOPE_TESTS_FAIL_ON_MISSING_TOOLS=1, actually fail only tests that are missing the required tools. * debian: + Add black to the test dependencies. + Enforce the "fail on missing tools" only when testing with all the recommended packages. + Ack some missing tools during autopkgtest. + Reinstate oggvideotools and procyon-decompiler test dependencies, now that are not buggy anymore. -- Mattia Rizzolo Tue, 21 May 2019 16:02:02 +0200 diffoscope (114) experimental; urgency=medium [ Chris Lamb ] * Add support for GnuPG "keybox" files. Thanks to Daniel Kahn Gillmor for the suggestion. (Closes: #871244, reproducible-builds/diffoscope#23) * Always warn if tlsh module is not available (not just if a specific fuzziness threshold is specified) to match the epilog of the --help output. This prevents missing support for file rename detection. (Closes: #888237, reproducible-builds/diffoscope#29) * Treat missing tools on Debian autopkgtests as individual test failures by checking whether a new DIFFOSCOPE_TESTS_FAIL_ON_MISSING_TOOLS environment variable is exported. (Closes: #905885, reproducible-builds/diffoscope#35) * Require that "-" is explicitly specified to read a single diff from standard input to avoid non-intuitive behaviour when diffoscope is called without any arguments. (Closes: reproducible-builds/diffoscope#54) * Make --use-dbgsym a ternary operator to make it easier to totally disable. Thanks to Mattia Rizzolo for the suggestion. * Consolidate on "e" as the aliased exception name. [ Milena Boselli Rosa ] * Various fixes to the HTML markup to prevent validation warnings/errors: - Prevent empty values for the "name" attribute name on HTML anchor tags, and add an "id" to its parent "div" container. - Fix "table column x established by element 'col' has no cells beginning in it" warnings. - Fix "Text run is not in Unicode Normalization Form C". - Remove the "type" HTML attribute from