History log of /haiku/src/add-ons/kernel/file_systems/packagefs/package/Package.cpp
Revision Date Author Comments
# d230b5fd 08-Jul-2019 Augustin Cavalier <waddlesplash@gmail.com>

packagefs: Use object caches for the most used node classes.

Slab statistics from KDL show that on a stock nightly image
(i.e. no additional packages besides the standards installed)
there are 43848 *objects* (not bytes) in the PackageNodeAttribute
cache, and 25090 in the PackageFile cache, so this seems more than
worth it.

The last commit seems to reduce memory usage at boot by about 1%,
this commit seems to not affect it at all; but it is a significant
performance optimization and on systems with more packages installed
the effect may be very noticeable.

Change-Id: I676a642ed6003f82b14396e1f02684575d899362


# 11e3c6dd 01-Apr-2019 Augustin Cavalier <waddlesplash@gmail.com>

packagefs: Open HPKG files as O_NOCACHE.

We have not one but two internal caches in packagefs for *uncompressed* data;
we do not need the underlying filesystem to cache the *compressed* data
needlessly.

This seems to be a negligible memory and boot speed improvement at best,
although it will likely help in low-memory situations in reducing
the pressure on the file cache (... actually it doesn't look like
the file cache responds to low resource notifications at all, which
may be relevant here.)


# d1a0bc4f 01-Apr-2019 Augustin Cavalier <waddlesplash@gmail.com>

packagefs: Remove support for HPKGv1.

The current version is "v2", and it has been since the final PM branch
was merged back in 2013, so it's unclear if anyone actually uses this
file format. The "package" command can still of course read these
files, it's just that packagefs will now not be able to.

It's very unlikely anyone is impacted by this change.


# a6a6e3a8 19-Feb-2019 Adrien Destugues <pulkomandy@gmail.com>

Package kit: add likely-useful architectures

Required for bootstrapping them.

Change-Id: I179d8c73b08688946049bf18ff9151ef047e449b
Reviewed-on: https://review.haiku-os.org/c/1071
Reviewed-by: waddlesplash <waddlesplash@gmail.com>


# abf0c287 05-Aug-2018 Augustin Cavalier <waddlesplash@gmail.com>

packagefs: system_packages take precedence over non-system packages.

Affects *all* nodes, i.e., both directories and files. It is still possible
to override these, but it must be done in another system_package, not just by
mounting a package with newer datestamps on the conflicting files.

This is the pure version of axeld's proposal 2(d) from #10071. However,
as the haiku_secondary packages contain application mime_db entires and
are also 'system_package's, it does not fully solve that ticket.


# 53aae3db 29-Oct-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

packagefs: Package::Open(): Also log error code


# e527b796 12-Jul-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

Switch package file accessor classes to use BPositionIO

* PackageFileHeap{Reader,Writer} as well as Package{Reader,Writer} and
their implementation and super classes do now internally use a
BPositionIO instead of a FD to access the package file. This provides
more flexibility needed for features to come.
* BPackageReader has already grown a new Init() version with a
BPositionIO* parameter.


# 32832cbe 23-Jun-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

Remove BPackageKit::BDataOutput

Use BDataIO instead.


# d2d1af83 18-Jun-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

Revert "Move ZlibDecompressor to libshared"

This reverts commit 9af2105d36856be60528c361edf14a263f242338.

Conflicts:
src/kits/package/Jamfile


# dff8d2ea 17-Feb-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

packagefs: Package: Use PackagesDirectory directly

... instead of Volume. Necessary to support using packages from
different directories.


# 9af2105d 10-Feb-2014 Adrien Destugues <pulkomandy@pulkomandy.tk>

Move ZlibDecompressor to libshared

* This will be used to implement compressed http streams
* Remove the custom BDataOutput class, and use BDataIO instead, for
easier integration with existing code.


# 3a7e0b00 19-Nov-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

packagefs: Support blacklisting entries in packages

In each installation location, it is now possible to create a settings
file "packages" that allows to blacklist entries contained in packages.
The format is:

Package <package name> {
EntryBlacklist {
<entry path>
...
}
}
...

<package name> is the base name (no version) of the respective package
(e.g. "haiku"), <entry path> is an installation location relative path
(e.g. "add-ons/Translators/FooTranslator").

Blacklisted entries will be ignored by packagefs, i.e. they won't appear
in the file system. This addresses the issue that it may be necessary to
remove a problematic file (e.g. driver, add-on, or library), which would
otherwise require editing the containing package file.

The settings file is not not "live". Changes take effect only after
reboot (respectively when remounting the concerned packagefs volume).


# 415e374e 03-Nov-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

packagefs: Package: Add versionedName property


# 624ba548 29-Sep-2013 Fran├žois Revol <revol@free.fr>

Add missing architectures for packages

The list in packagefs wasn't in sync anyway.


# a11b0003 24-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

packagefs: CachingPackageReader: Don't delete cached heap reader

... in destructor. After we create it, it belongs to the super class and
is destroyed by it.


# 5dae1541 21-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

Add "source" package architecture


# 46122852 21-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

packagefs: Add caching for the package file heap reader

* ReaderImplBase:
- Add virtual CreateCachedHeapReader() which can create a cached
reader based on the given heap reader.
- Rename HeapReader() to RawHeapReader() and add HeapReader() for the
cached heap reader.
- Add DetachHeapReader() to allow a clients to remove the heap
reader(s) after deleting the ReaderImplBase object.
* packagefs:
- Add CachedDataReader class, which wraps a given
BAbstractBufferedDataReader and provides caching for it using a
VMCache. The implementation is based on the IOCache implementation.
- Use CachedDataReader to wrap the heap reader. For file data that
means they are cached twice -- in the heap reader cache and in the
file cache -- but due to the heap reader using a VMCache as well,
the pages will be recycled automatically anyway. For attribute data
the cache should be very helpful, since they weren't cached at all
before.


# d59e0feb 20-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

package: Suppress version mismatch errors where V1 is supported

* Add flags parameter to Init() of BPackageReader and friends.
* Introduce flag B_HPKG_READER_DONT_PRINT_VERSION_MISMATCH_MESSAGE and
don't print a version mismatch error when given.
* package extract/list: Use the new flag.


# 2a6f1725 20-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

packagefs: Resolve error output TODOs


# 1f633814 17-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

hpkg format: compress the whole heap

Instead of handling compression for individual file/attribute data we
do now compress the whole heap where they are stored. This
significantly improves compression ratios. We still divide the
uncompressed data into 64 KiB chunks and use a chunk offset array for
the compressed chunks to allow for quick random access without too much
overhead. The tradeoff is a limited possible compression ratio -- i.e.
we won't be as good as tar.gz (though surprisingly with my test
archives we did better than zip).

The other package file sections (package attributes and TOC) are no
longer compressed individually. Their uncompressed data are simply
pushed onto the heap where the usual compression strategy applies. To
simplify things the repository format has been changed in the same
manner although it doesn't otherwise use the heap, since it only stores
meta data.

Due to the data compression having been exposed in public and private
API, this change touches a lot of package kit using code, including
packagefs and the boot loader packagefs support. The latter two haven't
been tested yet. Moreover packagefs needs a new kind of cache so we
avoid re-reading the same heap chunk for two different data items it
contains.


# 11cecf98 12-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

packagefs: Add support for hpkg format V1

* Add union-like class PackageData which wraps the V1 and V2
BPackageData classes.
* GlobalFactory: Create a data reader depending on the data format
version.
* Package: Add a loader for V1 format and try that, if the other one
fails.


# d07c930c 11-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

packagefs: Share string instances globally

Introduce a class String which refers to shared private data that is
registered in a global hash table (in class StringPool) and use the
class consequently. This eliminates duplicate allocations for the same
string and also speeds up tests for equality. There's quite a bit
overhead for the management structures (apparently even more than for
the string data itself), but due to almost all strings being used
multiple times this still almost halves the memory usage for string
data.


# 1e7416d9 11-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

packagefs: Organize sources into subdirectories


# 53aae3db4d0f0e4dfe08506217a55fda0b2fa7c6 29-Oct-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

packagefs: Package::Open(): Also log error code


# e527b796319f21ca025f68e1964df140daa6de35 12-Jul-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

Switch package file accessor classes to use BPositionIO

* PackageFileHeap{Reader,Writer} as well as Package{Reader,Writer} and
their implementation and super classes do now internally use a
BPositionIO instead of a FD to access the package file. This provides
more flexibility needed for features to come.
* BPackageReader has already grown a new Init() version with a
BPositionIO* parameter.


# 32832cbe47f991cc6d2b29824903181d8baaaa63 23-Jun-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

Remove BPackageKit::BDataOutput

Use BDataIO instead.


# d2d1af830b3f144937ea5e4c3f0e5b74c4c3d737 18-Jun-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

Revert "Move ZlibDecompressor to libshared"

This reverts commit 9af2105d36856be60528c361edf14a263f242338.

Conflicts:
src/kits/package/Jamfile


# dff8d2ea22e081f2369f0e0967c812123d27cbad 17-Feb-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

packagefs: Package: Use PackagesDirectory directly

... instead of Volume. Necessary to support using packages from
different directories.


# 9af2105d36856be60528c361edf14a263f242338 10-Feb-2014 Adrien Destugues <pulkomandy@pulkomandy.tk>

Move ZlibDecompressor to libshared

* This will be used to implement compressed http streams
* Remove the custom BDataOutput class, and use BDataIO instead, for
easier integration with existing code.


# 3a7e0b00147f7a33bc52cb75a56bde8d9652d92a 19-Nov-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

packagefs: Support blacklisting entries in packages

In each installation location, it is now possible to create a settings
file "packages" that allows to blacklist entries contained in packages.
The format is:

Package <package name> {
EntryBlacklist {
<entry path>
...
}
}
...

<package name> is the base name (no version) of the respective package
(e.g. "haiku"), <entry path> is an installation location relative path
(e.g. "add-ons/Translators/FooTranslator").

Blacklisted entries will be ignored by packagefs, i.e. they won't appear
in the file system. This addresses the issue that it may be necessary to
remove a problematic file (e.g. driver, add-on, or library), which would
otherwise require editing the containing package file.

The settings file is not not "live". Changes take effect only after
reboot (respectively when remounting the concerned packagefs volume).


# 415e374ec5a5ebc87ba2dccde59308130aaf3f84 03-Nov-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

packagefs: Package: Add versionedName property


# 624ba5482b6c126c8c460b6c169592cb3afc4291 29-Sep-2013 Fran├žois Revol <revol@free.fr>

Add missing architectures for packages

The list in packagefs wasn't in sync anyway.


# a11b0003ecee864899a0b54f8d403e0026ac3783 24-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

packagefs: CachingPackageReader: Don't delete cached heap reader

... in destructor. After we create it, it belongs to the super class and
is destroyed by it.


# 5dae1541d63974a6e2011553e9de484a0495414a 21-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

Add "source" package architecture


# 46122852f1dfbbf4bac6f5a634452ff90c5635ce 21-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

packagefs: Add caching for the package file heap reader

* ReaderImplBase:
- Add virtual CreateCachedHeapReader() which can create a cached
reader based on the given heap reader.
- Rename HeapReader() to RawHeapReader() and add HeapReader() for the
cached heap reader.
- Add DetachHeapReader() to allow a clients to remove the heap
reader(s) after deleting the ReaderImplBase object.
* packagefs:
- Add CachedDataReader class, which wraps a given
BAbstractBufferedDataReader and provides caching for it using a
VMCache. The implementation is based on the IOCache implementation.
- Use CachedDataReader to wrap the heap reader. For file data that
means they are cached twice -- in the heap reader cache and in the
file cache -- but due to the heap reader using a VMCache as well,
the pages will be recycled automatically anyway. For attribute data
the cache should be very helpful, since they weren't cached at all
before.


# d59e0feb59b92ca19ecda2935369fd0fcc5a4f56 20-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

package: Suppress version mismatch errors where V1 is supported

* Add flags parameter to Init() of BPackageReader and friends.
* Introduce flag B_HPKG_READER_DONT_PRINT_VERSION_MISMATCH_MESSAGE and
don't print a version mismatch error when given.
* package extract/list: Use the new flag.


# 2a6f1725c648a566ce51c4894b902a13ed4c11eb 20-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

packagefs: Resolve error output TODOs


# 1f633814fa347b0bd822f683af731615df600347 17-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

hpkg format: compress the whole heap

Instead of handling compression for individual file/attribute data we
do now compress the whole heap where they are stored. This
significantly improves compression ratios. We still divide the
uncompressed data into 64 KiB chunks and use a chunk offset array for
the compressed chunks to allow for quick random access without too much
overhead. The tradeoff is a limited possible compression ratio -- i.e.
we won't be as good as tar.gz (though surprisingly with my test
archives we did better than zip).

The other package file sections (package attributes and TOC) are no
longer compressed individually. Their uncompressed data are simply
pushed onto the heap where the usual compression strategy applies. To
simplify things the repository format has been changed in the same
manner although it doesn't otherwise use the heap, since it only stores
meta data.

Due to the data compression having been exposed in public and private
API, this change touches a lot of package kit using code, including
packagefs and the boot loader packagefs support. The latter two haven't
been tested yet. Moreover packagefs needs a new kind of cache so we
avoid re-reading the same heap chunk for two different data items it
contains.


# 11cecf980b5ee6a21563a7e78d4f2238a8b4d90f 12-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

packagefs: Add support for hpkg format V1

* Add union-like class PackageData which wraps the V1 and V2
BPackageData classes.
* GlobalFactory: Create a data reader depending on the data format
version.
* Package: Add a loader for V1 format and try that, if the other one
fails.


# d07c930c1eb710289922edd3abf5244c259a86c6 11-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

packagefs: Share string instances globally

Introduce a class String which refers to shared private data that is
registered in a global hash table (in class StringPool) and use the
class consequently. This eliminates duplicate allocations for the same
string and also speeds up tests for equality. There's quite a bit
overhead for the management structures (apparently even more than for
the string data itself), but due to almost all strings being used
multiple times this still almost halves the memory usage for string
data.


# 1e7416d9b3846396e2f7d6552e37d72664cd6853 11-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

packagefs: Organize sources into subdirectories