* Add support for "files"-archive
i.e. write individual pbf-files to disk in the format <base>/z/x/y.pbf
in order to use that format it must be passed as "--ouput=/path/to/tiles?format=files"
Fixes#536
* default to files format
...if no explict format query param given, path ends with a slash, or no extension given
* output metadata.json in files-archive
and refactor TileArchiveMetadata
1. put zoom into center (CoordinateXY->Coordinate) - in sync with mbtiles-format
2. add (De-)Serializer for Coordinate+Envelop => avoid duplication and cleaner
3. change the json and proto output for TileArchiveMetadata to be (more) in sync with mbtiles-format
* add support for custom tile scheme in files-archive
{z}/{x}/{y}.pbf is the default and can be configured as needed - e.g.:
- different order: {x}/{y}/{z}.pbf
- with intermediate dirs: {x}/a/{y}/b/{z}.pbf
- with different extension: {z}/{y}/{y}.pbf.gz
instead of {x} and {y}, {xs} and {xy} can be used which breaks up
x and y into 2 directories each and ensures that each directory has <1000 children
* fix issues with multiple writers
1. call finish archive only once after all writers are finished
...and not every time a writer finishes
2. log "zoom-progress" for the first tile write only
(Finished z11 ... now starting z12)
3. remove file/dir-size progress logger bottleneck for files archive
=> each archive now reports the bytes written, which also fixes
the issues of stream-archives reporting the size incorrectly
4. introduce printStats-hook on archive-level
* add async file write support to files archive
...allow to use virtual threads ExecturService (bound only!) for tile writing
also add some benchmark for writing tiles to disk: fixed, bound virtual, async, unbound virtual
* Revert "add async file write support to files archive"
This reverts commit b8cfa56977d98520aa8b62252c3a2726d440afe0.
* few improvements
- extract TileSchemeEncoding
- use Counter.MultithreadCounter rather than LongAdder to count bytes written
- add some JavaDoc
* simplify files archive usage
1. allow to pass tile scheme directly via output: --output=tiles/{x}/{y}/{z}.pbf
2. auto-encode { (%7B) and } (%7D) => no need to encode it the URI on CLI
* few more adjustments according to PR feeback
1. use WriteableTileArchive#bytesWritten in summmary as well
2. call WriteableTileArchive#init in a safer manner
..and a few more adjustments
* more PR feedback
* Add --nodemap-type=array option for 2-3x faster osm pass 1 imports
* Add --nodemap-storage=direct option to experiment with direct (off-heap) memory usage
* Extract ResourceUsage and OsmPhaser utilities
* Use large memory-mapped file segments to avoid running out of space on smaller machines
* Add `--nodemap-madvise` argument to opt into madvise(random) for memory-mapped file access
Michael Barry