planetiler/planetiler-core/src/main/java/com/onthegomap/planetiler/util/CountingOutputStream.java

37 wiersze
929 B
Java
Czysty Zwykły widok Historia

Add support for "files"-archive (#761) * Add support for "files"-archive i.e. write individual pbf-files to disk in the format <base>/z/x/y.pbf in order to use that format it must be passed as "--ouput=/path/to/tiles?format=files" Fixes #536 * default to files format ...if no explict format query param given, path ends with a slash, or no extension given * output metadata.json in files-archive and refactor TileArchiveMetadata 1. put zoom into center (CoordinateXY->Coordinate) - in sync with mbtiles-format 2. add (De-)Serializer for Coordinate+Envelop => avoid duplication and cleaner 3. change the json and proto output for TileArchiveMetadata to be (more) in sync with mbtiles-format * add support for custom tile scheme in files-archive {z}/{x}/{y}.pbf is the default and can be configured as needed - e.g.: - different order: {x}/{y}/{z}.pbf - with intermediate dirs: {x}/a/{y}/b/{z}.pbf - with different extension: {z}/{y}/{y}.pbf.gz instead of {x} and {y}, {xs} and {xy} can be used which breaks up x and y into 2 directories each and ensures that each directory has <1000 children * fix issues with multiple writers 1. call finish archive only once after all writers are finished ...and not every time a writer finishes 2. log "zoom-progress" for the first tile write only (Finished z11 ... now starting z12) 3. remove file/dir-size progress logger bottleneck for files archive => each archive now reports the bytes written, which also fixes the issues of stream-archives reporting the size incorrectly 4. introduce printStats-hook on archive-level * add async file write support to files archive ...allow to use virtual threads ExecturService (bound only!) for tile writing also add some benchmark for writing tiles to disk: fixed, bound virtual, async, unbound virtual * Revert "add async file write support to files archive" This reverts commit b8cfa56977d98520aa8b62252c3a2726d440afe0. * few improvements - extract TileSchemeEncoding - use Counter.MultithreadCounter rather than LongAdder to count bytes written - add some JavaDoc * simplify files archive usage 1. allow to pass tile scheme directly via output: --output=tiles/{x}/{y}/{z}.pbf 2. auto-encode { (%7B) and } (%7D) => no need to encode it the URI on CLI * few more adjustments according to PR feeback 1. use WriteableTileArchive#bytesWritten in summmary as well 2. call WriteableTileArchive#init in a safer manner ..and a few more adjustments * more PR feedback
2024-01-03 01:37:49 +00:00
package com.onthegomap.planetiler.util;
import java.io.IOException;
import java.io.OutputStream;
import java.util.function.LongConsumer;
/**
* {@link OutputStream} decorator that notifies the callback about the written bytes.
*/
public class CountingOutputStream extends DelegatingOutputStream {
Add support for "files"-archive (#761) * Add support for "files"-archive i.e. write individual pbf-files to disk in the format <base>/z/x/y.pbf in order to use that format it must be passed as "--ouput=/path/to/tiles?format=files" Fixes #536 * default to files format ...if no explict format query param given, path ends with a slash, or no extension given * output metadata.json in files-archive and refactor TileArchiveMetadata 1. put zoom into center (CoordinateXY->Coordinate) - in sync with mbtiles-format 2. add (De-)Serializer for Coordinate+Envelop => avoid duplication and cleaner 3. change the json and proto output for TileArchiveMetadata to be (more) in sync with mbtiles-format * add support for custom tile scheme in files-archive {z}/{x}/{y}.pbf is the default and can be configured as needed - e.g.: - different order: {x}/{y}/{z}.pbf - with intermediate dirs: {x}/a/{y}/b/{z}.pbf - with different extension: {z}/{y}/{y}.pbf.gz instead of {x} and {y}, {xs} and {xy} can be used which breaks up x and y into 2 directories each and ensures that each directory has <1000 children * fix issues with multiple writers 1. call finish archive only once after all writers are finished ...and not every time a writer finishes 2. log "zoom-progress" for the first tile write only (Finished z11 ... now starting z12) 3. remove file/dir-size progress logger bottleneck for files archive => each archive now reports the bytes written, which also fixes the issues of stream-archives reporting the size incorrectly 4. introduce printStats-hook on archive-level * add async file write support to files archive ...allow to use virtual threads ExecturService (bound only!) for tile writing also add some benchmark for writing tiles to disk: fixed, bound virtual, async, unbound virtual * Revert "add async file write support to files archive" This reverts commit b8cfa56977d98520aa8b62252c3a2726d440afe0. * few improvements - extract TileSchemeEncoding - use Counter.MultithreadCounter rather than LongAdder to count bytes written - add some JavaDoc * simplify files archive usage 1. allow to pass tile scheme directly via output: --output=tiles/{x}/{y}/{z}.pbf 2. auto-encode { (%7B) and } (%7D) => no need to encode it the URI on CLI * few more adjustments according to PR feeback 1. use WriteableTileArchive#bytesWritten in summmary as well 2. call WriteableTileArchive#init in a safer manner ..and a few more adjustments * more PR feedback
2024-01-03 01:37:49 +00:00
private final LongConsumer writtenBytesConsumer;
public CountingOutputStream(OutputStream wrapped, LongConsumer writtenBytesConsumer) {
super(wrapped);
Add support for "files"-archive (#761) * Add support for "files"-archive i.e. write individual pbf-files to disk in the format <base>/z/x/y.pbf in order to use that format it must be passed as "--ouput=/path/to/tiles?format=files" Fixes #536 * default to files format ...if no explict format query param given, path ends with a slash, or no extension given * output metadata.json in files-archive and refactor TileArchiveMetadata 1. put zoom into center (CoordinateXY->Coordinate) - in sync with mbtiles-format 2. add (De-)Serializer for Coordinate+Envelop => avoid duplication and cleaner 3. change the json and proto output for TileArchiveMetadata to be (more) in sync with mbtiles-format * add support for custom tile scheme in files-archive {z}/{x}/{y}.pbf is the default and can be configured as needed - e.g.: - different order: {x}/{y}/{z}.pbf - with intermediate dirs: {x}/a/{y}/b/{z}.pbf - with different extension: {z}/{y}/{y}.pbf.gz instead of {x} and {y}, {xs} and {xy} can be used which breaks up x and y into 2 directories each and ensures that each directory has <1000 children * fix issues with multiple writers 1. call finish archive only once after all writers are finished ...and not every time a writer finishes 2. log "zoom-progress" for the first tile write only (Finished z11 ... now starting z12) 3. remove file/dir-size progress logger bottleneck for files archive => each archive now reports the bytes written, which also fixes the issues of stream-archives reporting the size incorrectly 4. introduce printStats-hook on archive-level * add async file write support to files archive ...allow to use virtual threads ExecturService (bound only!) for tile writing also add some benchmark for writing tiles to disk: fixed, bound virtual, async, unbound virtual * Revert "add async file write support to files archive" This reverts commit b8cfa56977d98520aa8b62252c3a2726d440afe0. * few improvements - extract TileSchemeEncoding - use Counter.MultithreadCounter rather than LongAdder to count bytes written - add some JavaDoc * simplify files archive usage 1. allow to pass tile scheme directly via output: --output=tiles/{x}/{y}/{z}.pbf 2. auto-encode { (%7B) and } (%7D) => no need to encode it the URI on CLI * few more adjustments according to PR feeback 1. use WriteableTileArchive#bytesWritten in summmary as well 2. call WriteableTileArchive#init in a safer manner ..and a few more adjustments * more PR feedback
2024-01-03 01:37:49 +00:00
this.writtenBytesConsumer = writtenBytesConsumer;
}
@Override
public void write(int i) throws IOException {
super.write(i);
Add support for "files"-archive (#761) * Add support for "files"-archive i.e. write individual pbf-files to disk in the format <base>/z/x/y.pbf in order to use that format it must be passed as "--ouput=/path/to/tiles?format=files" Fixes #536 * default to files format ...if no explict format query param given, path ends with a slash, or no extension given * output metadata.json in files-archive and refactor TileArchiveMetadata 1. put zoom into center (CoordinateXY->Coordinate) - in sync with mbtiles-format 2. add (De-)Serializer for Coordinate+Envelop => avoid duplication and cleaner 3. change the json and proto output for TileArchiveMetadata to be (more) in sync with mbtiles-format * add support for custom tile scheme in files-archive {z}/{x}/{y}.pbf is the default and can be configured as needed - e.g.: - different order: {x}/{y}/{z}.pbf - with intermediate dirs: {x}/a/{y}/b/{z}.pbf - with different extension: {z}/{y}/{y}.pbf.gz instead of {x} and {y}, {xs} and {xy} can be used which breaks up x and y into 2 directories each and ensures that each directory has <1000 children * fix issues with multiple writers 1. call finish archive only once after all writers are finished ...and not every time a writer finishes 2. log "zoom-progress" for the first tile write only (Finished z11 ... now starting z12) 3. remove file/dir-size progress logger bottleneck for files archive => each archive now reports the bytes written, which also fixes the issues of stream-archives reporting the size incorrectly 4. introduce printStats-hook on archive-level * add async file write support to files archive ...allow to use virtual threads ExecturService (bound only!) for tile writing also add some benchmark for writing tiles to disk: fixed, bound virtual, async, unbound virtual * Revert "add async file write support to files archive" This reverts commit b8cfa56977d98520aa8b62252c3a2726d440afe0. * few improvements - extract TileSchemeEncoding - use Counter.MultithreadCounter rather than LongAdder to count bytes written - add some JavaDoc * simplify files archive usage 1. allow to pass tile scheme directly via output: --output=tiles/{x}/{y}/{z}.pbf 2. auto-encode { (%7B) and } (%7D) => no need to encode it the URI on CLI * few more adjustments according to PR feeback 1. use WriteableTileArchive#bytesWritten in summmary as well 2. call WriteableTileArchive#init in a safer manner ..and a few more adjustments * more PR feedback
2024-01-03 01:37:49 +00:00
writtenBytesConsumer.accept(1L);
}
@Override
public void write(byte[] b) throws IOException {
super.write(b);
Add support for "files"-archive (#761) * Add support for "files"-archive i.e. write individual pbf-files to disk in the format <base>/z/x/y.pbf in order to use that format it must be passed as "--ouput=/path/to/tiles?format=files" Fixes #536 * default to files format ...if no explict format query param given, path ends with a slash, or no extension given * output metadata.json in files-archive and refactor TileArchiveMetadata 1. put zoom into center (CoordinateXY->Coordinate) - in sync with mbtiles-format 2. add (De-)Serializer for Coordinate+Envelop => avoid duplication and cleaner 3. change the json and proto output for TileArchiveMetadata to be (more) in sync with mbtiles-format * add support for custom tile scheme in files-archive {z}/{x}/{y}.pbf is the default and can be configured as needed - e.g.: - different order: {x}/{y}/{z}.pbf - with intermediate dirs: {x}/a/{y}/b/{z}.pbf - with different extension: {z}/{y}/{y}.pbf.gz instead of {x} and {y}, {xs} and {xy} can be used which breaks up x and y into 2 directories each and ensures that each directory has <1000 children * fix issues with multiple writers 1. call finish archive only once after all writers are finished ...and not every time a writer finishes 2. log "zoom-progress" for the first tile write only (Finished z11 ... now starting z12) 3. remove file/dir-size progress logger bottleneck for files archive => each archive now reports the bytes written, which also fixes the issues of stream-archives reporting the size incorrectly 4. introduce printStats-hook on archive-level * add async file write support to files archive ...allow to use virtual threads ExecturService (bound only!) for tile writing also add some benchmark for writing tiles to disk: fixed, bound virtual, async, unbound virtual * Revert "add async file write support to files archive" This reverts commit b8cfa56977d98520aa8b62252c3a2726d440afe0. * few improvements - extract TileSchemeEncoding - use Counter.MultithreadCounter rather than LongAdder to count bytes written - add some JavaDoc * simplify files archive usage 1. allow to pass tile scheme directly via output: --output=tiles/{x}/{y}/{z}.pbf 2. auto-encode { (%7B) and } (%7D) => no need to encode it the URI on CLI * few more adjustments according to PR feeback 1. use WriteableTileArchive#bytesWritten in summmary as well 2. call WriteableTileArchive#init in a safer manner ..and a few more adjustments * more PR feedback
2024-01-03 01:37:49 +00:00
writtenBytesConsumer.accept(b.length);
}
@Override
public void write(byte[] b, int off, int len) throws IOException {
super.write(b, off, len);
Add support for "files"-archive (#761) * Add support for "files"-archive i.e. write individual pbf-files to disk in the format <base>/z/x/y.pbf in order to use that format it must be passed as "--ouput=/path/to/tiles?format=files" Fixes #536 * default to files format ...if no explict format query param given, path ends with a slash, or no extension given * output metadata.json in files-archive and refactor TileArchiveMetadata 1. put zoom into center (CoordinateXY->Coordinate) - in sync with mbtiles-format 2. add (De-)Serializer for Coordinate+Envelop => avoid duplication and cleaner 3. change the json and proto output for TileArchiveMetadata to be (more) in sync with mbtiles-format * add support for custom tile scheme in files-archive {z}/{x}/{y}.pbf is the default and can be configured as needed - e.g.: - different order: {x}/{y}/{z}.pbf - with intermediate dirs: {x}/a/{y}/b/{z}.pbf - with different extension: {z}/{y}/{y}.pbf.gz instead of {x} and {y}, {xs} and {xy} can be used which breaks up x and y into 2 directories each and ensures that each directory has <1000 children * fix issues with multiple writers 1. call finish archive only once after all writers are finished ...and not every time a writer finishes 2. log "zoom-progress" for the first tile write only (Finished z11 ... now starting z12) 3. remove file/dir-size progress logger bottleneck for files archive => each archive now reports the bytes written, which also fixes the issues of stream-archives reporting the size incorrectly 4. introduce printStats-hook on archive-level * add async file write support to files archive ...allow to use virtual threads ExecturService (bound only!) for tile writing also add some benchmark for writing tiles to disk: fixed, bound virtual, async, unbound virtual * Revert "add async file write support to files archive" This reverts commit b8cfa56977d98520aa8b62252c3a2726d440afe0. * few improvements - extract TileSchemeEncoding - use Counter.MultithreadCounter rather than LongAdder to count bytes written - add some JavaDoc * simplify files archive usage 1. allow to pass tile scheme directly via output: --output=tiles/{x}/{y}/{z}.pbf 2. auto-encode { (%7B) and } (%7D) => no need to encode it the URI on CLI * few more adjustments according to PR feeback 1. use WriteableTileArchive#bytesWritten in summmary as well 2. call WriteableTileArchive#init in a safer manner ..and a few more adjustments * more PR feedback
2024-01-03 01:37:49 +00:00
writtenBytesConsumer.accept(len);
}
}