mirror
/
planetiler
kopia lustrzana https://github.com/onthegomap/planetiler
1
0
Forkuj 0
Kod Zgłoszenia Projekty Wydania Wiki Aktywność

Shortbread schema (#323)

pull/346/head
Michael Barry 2022-09-23 06:49:09 -04:00 zatwierdzone przez GitHub
rodzic b934f4ee89
commit 5296d1772e
Nie znaleziono w bazie danych klucza dla tego podpisu
ID klucza GPG: 4AEE18F83AFDEB23
93 zmienionych plików z 7861 dodań i 879 usunięć
Pokaż statystyki Ściągnij plik aktualizacji Ściągnij plik porównania Expand all files Collapse all files

1
.gitignore vendored
Wyświetl plik

@ -13,6 +13,7 @@ target/
!.idea/codeStyles
!.idea/vcs.xml
!.idea/eclipseCodeFormatter.xml
!.idea/jsonSchemas.xml
# eclipse
.classpath

5
.idea/codeStyles/Project.xml
Wyświetl plik

@ -114,6 +114,9 @@
<XML>
<option name="XML_ALIGN_ATTRIBUTES" value="false" />
</XML>
<yaml>
<option name="INDENT_SEQUENCE_VALUE" value="false" />
</yaml>
<codeStyleSettings language="CSS">
<indentOptions>
<option name="INDENT_SIZE" value="2" />
@ -595,4 +598,4 @@
</indentOptions>
</codeStyleSettings>
</code_scheme>
</component>
</component>

45
.idea/jsonSchemas.xml 100644
Wyświetl plik

@ -0,0 +1,45 @@
<?xml version="1.0" encoding="UTF-8"?>
<project version="4">
<component name="JsonSchemaMappingsProjectConfiguration">
<state>
<map>
<entry key="JSON Schema version 7">
<value>
<SchemaInfo>
<option name="name" value="JSON Schema version 7" />
<option name="relativePathToSchema" value="http://json-schema.org/draft-07/schema" />
<option name="schemaVersion" value="JSON Schema version 7" />
<option name="applicationDefined" value="true" />
<option name="patterns">
<list>
<Item>
<option name="path" value="planetiler.schema.yml" />
</Item>
</list>
</option>
</SchemaInfo>
</value>
</entry>
<entry key="Planetiler">
<value>
<SchemaInfo>
<option name="generatedName" value="New Schema" />
<option name="name" value="Planetiler" />
<option name="relativePathToSchema" value="planetiler-custommap/planetiler.schema.json" />
<option name="schemaVersion" value="JSON Schema version 7" />
<option name="patterns">
<list>
<Item>
<option name="directory" value="true" />
<option name="path" value="planetiler-custommap" />
<option name="mappingKind" value="Directory" />
</Item>
</list>
</option>
</SchemaInfo>
</value>
</entry>
</map>
</state>
</component>
</project>

10
.vscode/settings.json vendored
Wyświetl plik

@ -1,11 +1,11 @@
{
"java.format.settings.url": "eclipse-formatter.xml",
"java.format.settings.profile": "Planetiler",
"java.completion.importOrder": [
"#",
""
],
"java.completion.importOrder": ["#", ""],
"java.sources.organizeImports.staticStarThreshold": 5,
"java.sources.organizeImports.starThreshold": 999,
"java.saveActions.organizeImports": true
"java.saveActions.organizeImports": true,
"yaml.schemas": {
"./planetiler-custommap/planetiler.schema.json": "planetiler-custommap/**/*.yml"
}
}

30
ARCHITECTURE.md
Wyświetl plik

@ -4,13 +4,14 @@
Planetiler builds a map in 3 phases:
1. [Process Input Files](#1-process-input-files) according to
the [Profile](planetiler-core/src/main/java/com/onthegomap/planetiler/Profile.java) and write vector tile features
to intermediate files on disk
1. [Process Input Files](#1-process-input-files) according to the [Profile](#profiles) and write vector tile features to
intermediate files on disk
2. [Sort Features](#2-sort-features) by tile ID
3. [Emit Vector Tiles](#3-emit-vector-tiles) by iterating through sorted features to group by tile ID, encoding, and
writing to the output MBTiles file
User-defined [profiles](#profiles) customize the behavior of each part of this pipeline.
## 1) Process Input Files
First, Planetiler
@ -28,7 +29,8 @@ from each input source:
- nodes: store node latitude/longitude locations in-memory or on disk
using [LongLongMap](planetiler-core/src/main/java/com/onthegomap/planetiler/collection/LongLongMap.java)
- ways: nothing
- relations: call `preprocessOsmRelation` on the profile and store information returned for each relation of
- relations: call `preprocessOsmRelation` on the [profile](#profiles) and store information returned for each
relation of
interest, along with relation member IDs in-memory using
a [LongLongMultimap](planetiler-core/src/main/java/com/onthegomap/planetiler/collection/LongLongMultimap.java).
- pass 2:
@ -46,11 +48,9 @@ from each input source:
then emit a polygon source feature with the reconstructed geometry if successful
Then, for each [SourceFeature](planetiler-core/src/main/java/com/onthegomap/planetiler/reader/SourceFeature.java),
generate vector tile features according to
the [Profile](planetiler-core/src/main/java/com/onthegomap/planetiler/Profile.java) in a worker thread (default 1 per
core):
generate vector tile features according to the [profile](#profiles) in a worker thread (default 1 per core):
- Call `processFeature` method on the profile for each source feature
- Call `processFeature` method on the [profile](#profiles) for each source feature
- For every vector tile feature added to
the [FeatureCollector](planetiler-core/src/main/java/com/onthegomap/planetiler/FeatureCollector.java):
- Call [FeatureRenderer#accept](planetiler-core/src/main/java/com/onthegomap/planetiler/render/FeatureRenderer.java)
@ -122,3 +122,17 @@ Finally, a single-threaded writer writes encoded vector tiles to the output MBTi
- Iterate through finished vector tile batches until the prepared statement is full, flush to disk, then repeat
- Then flush any remaining tiles at the end
## Profiles
To customize the behavior of this pipeline, custom profiles implement
the [Profile](planetiler-core/src/main/java/com/onthegomap/planetiler/Profile.java) interface to override:
- what vector tile features to generate from an input feature
- what information from OpenStreetMap relations we need to save for later use
- how to post-process vector features grouped into a tile before emitting
A Java project can implement this interface and add arbitrarily complex processing when overriding the methods.
The [custommap](planetiler-custommap) project defines
a [ConfiguredProfile](planetiler-custommap/src/main/java/com/onthegomap/planetiler/custommap/ConfiguredProfile.java)
implementation that loads instructions from a YAML config file to dynamically control how schemas are generated without
needing to write or compile Java code.

2
NOTICE.md
Wyświetl plik

@ -24,6 +24,7 @@ The `planetiler-core` module includes the following software:
- com.carrotsearch:hppc (Apache license)
- com.github.jnr:jnr-ffi (Apache license)
- org.roaringbitmap:RoaringBitmap (Apache license)
- org.projectnessie.cel:cel-tools (Apache license)
- Adapted code:
- `DouglasPeuckerSimplifier` from [JTS](https://github.com/locationtech/jts) (EDL)
- `OsmMultipolygon` from [imposm3](https://github.com/omniscale/imposm3) (Apache license)
@ -41,6 +42,7 @@ The `planetiler-core` module includes the following software:
from [openstreetmap/OSM-binary](https://github.com/openstreetmap/OSM-binary/tree/master/osmpbf) (MIT License)
- Maven Dependencies:
- org.yaml:snakeyaml (Apache license)
- org.snakeyaml:snakeyaml-engine (Apache license)
- org.commonmark:commonmark (BSD 2-clause license)
- Code adapted from OpenMapTiles (BSD 3-Clause License):
- `generated` package generated from OpenMapTiles

45
README.md
Wyświetl plik

@ -138,9 +138,22 @@ Learn more about working with submodules [here](https://git-scm.com/book/en/v2/G
See [PLANET.md](PLANET.md).
## Examples
## Generating Custom Vector Tiles
See the [planetiler-examples](planetiler-examples) project.
If you want to customize the OpenMapTiles schema or generate an mbtiles file with OpenMapTiles + extra layers, then
fork https://github.com/openmaptiles/planetiler-openmaptiles make changes there, and run directly from that repo. It
is a standalone Java project with a dependency on Planetiler.
If you want to generate a separate mbtiles file with overlay layers or a full custom basemap, then:
- For simple schemas, run a recent planetiler jar or docker image with a custom schema defined in a yaml
configuration file. See [planetiler-custommap](planetiler-custommap) for details.
- For complex schemas (or if you prefer working in Java), create a new Java project
that [depends on Planetiler](#use-as-a-library). See the [planetiler-examples](planetiler-examples) project for a
working example.
If you want to customize how planetiler works internally, then fork this project, build from source, and
consider [contributing](#contributing) your change back for others to use!
## Benchmarks
@ -198,6 +211,7 @@ download regularly-updated tilesets.
- Java-based [Profile API](planetiler-core/src/main/java/com/onthegomap/planetiler/Profile.java) to customize how source
elements map to vector tile features, and post-process generated tiles
using [JTS geometry utilities](https://github.com/locationtech/jts)
- [YAML config file format](planetiler-custommap) that lets you create custom schemas without writing Java code
- Merge nearby lines or polygons with the same tags before emitting vector tiles
- Automatically fixes self-intersecting polygons
- Built-in OpenMapTiles profile based on [OpenMapTiles](https://openmaptiles.org/) v3.13.1
@ -224,7 +238,25 @@ Planetiler can be used as a maven-style dependency in a Java project using the s
### Maven
Add this dependency to your java project:
Add this repository block to your `pom.xml`:
```xml
<repositories>
<repository>
<id>osgeo</id>
<name>OSGeo Release Repository</name>
<url>https://repo.osgeo.org/repository/release/</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
<releases>
<enabled>true</enabled>
</releases>
</repository>
</repositories>
```
Then add the following dependency:
```xml
<dependency>
@ -236,7 +268,7 @@ Add this dependency to your java project:
### Gradle
Set up your repositories block as follows:
Set up your repositories block::
```groovy
mavenCentral()
@ -245,7 +277,7 @@ maven {
}
```
Set up your dependencies block as follows:
Set up your dependencies block:
```groovy
implementation 'com.onthegomap.planetiler:planetiler-core:<version>'
@ -292,6 +324,9 @@ Planetiler is made possible by these awesome open source projects:
- [Osmosis](https://wiki.openstreetmap.org/wiki/Osmosis) for Java utilities to parse OpenStreetMap data
- [JNR-FFI](https://github.com/jnr/jnr-ffi) for utilities to access low-level system utilities to improve memory-mapped
file performance.
- [cel-java](https://github.com/projectnessie/cel-java) for the Java implementation of
Google's [Common Expression Language](https://github.com/google/cel-spec) that powers dynamic expressions embedded in
schema config files.
See [NOTICE.md](NOTICE.md) for a full list and license details.

4
planetiler-core/src/main/java/com/onthegomap/planetiler/FeatureCollector.java
Wyświetl plik

@ -231,7 +231,7 @@ public class FeatureCollector implements Iterable<FeatureCollector.Feature> {
private Feature(String layer, Geometry geom, long sourceId) {
this.layer = layer;
this.geom = geom;
this.geometryType = GeometryType.valueOf(geom);
this.geometryType = GeometryType.typeOf(geom);
this.sourceId = sourceId;
}
@ -634,7 +634,7 @@ public class FeatureCollector implements Iterable<FeatureCollector.Feature> {
return attrs;
}
if (attrCache == null) {
attrCache = CacheByZoom.create(config, this::computeAttrsAtZoom);
attrCache = CacheByZoom.create(this::computeAttrsAtZoom);
}
return attrCache.get(zoom);
}

2
planetiler-core/src/main/java/com/onthegomap/planetiler/VectorTile.java
Wyświetl plik

@ -401,7 +401,7 @@ public class VectorTile {
}
public static VectorGeometry encodeGeometry(Geometry geometry, int scale) {
return new VectorGeometry(getCommands(geometry, scale), GeometryType.valueOf(geometry), scale);
return new VectorGeometry(getCommands(geometry, scale), GeometryType.typeOf(geometry), scale);
}
/**

54
planetiler-core/src/main/java/com/onthegomap/planetiler/config/Arguments.java
Wyświetl plik

@ -15,6 +15,7 @@ import java.util.Map;
import java.util.Properties;
import java.util.TreeMap;
import java.util.function.Function;
import java.util.function.UnaryOperator;
import java.util.stream.Stream;
import org.locationtech.jts.geom.Envelope;
import org.slf4j.Logger;
@ -29,8 +30,9 @@ public class Arguments {
private static final Logger LOGGER = LoggerFactory.getLogger(Arguments.class);
private final Function<String, String> provider;
private boolean silent = false;
private Arguments(Function<String, String> provider) {
private Arguments(UnaryOperator<String> provider) {
this.provider = provider;
}
@ -116,9 +118,7 @@ public class Arguments {
* @return arguments parsed from those sources
*/
public static Arguments fromArgsOrConfigFile(String... args) {
Arguments fromArgsOrEnv = fromArgs(args)
.orElse(fromJvmProperties())
.orElse(fromEnvironment());
Arguments fromArgsOrEnv = fromEnvOrArgs(args);
Path configFile = fromArgsOrEnv.file("config", "path to config file", null);
if (configFile != null) {
return fromArgsOrEnv.orElse(fromConfigFile(configFile));
@ -127,6 +127,25 @@ public class Arguments {
}
}
/**
* Returns arguments parsed from command-line arguments, JVM properties, environmental variables.
* <p>
* Priority order:
* <ol>
* <li>command-line arguments: {@code java ... key=value}</li>
* <li>jvm properties: {@code java -Dplanetiler.key=value ...}</li>
* <li>environmental variables: {@code PLANETILER_KEY=value java ...}</li>
* </ol>
*
* @param args command-line args provide to main entrypoint method
* @return arguments parsed from those sources
*/
public static Arguments fromEnvOrArgs(String... args) {
return fromArgs(args)
.orElse(fromJvmProperties())
.orElse(fromEnvironment());
}
public static Arguments of(Map<String, String> map) {
return new Arguments(map::get);
}
@ -200,7 +219,15 @@ public class Arguments {
}
private void logArgValue(String key, String description, Object result) {
LOGGER.debug("argument: " + key + "=" + result + " (" + description + ")");
if (!silent) {
LOGGER.debug("argument: {}={} ({})", key, result, description);
}
}
/** Stop logging argument values when they are read and return this instance. */
public Arguments silence() {
this.silent = true;
return this;
}
public String getString(String key, String description, String defaultValue) {
@ -209,6 +236,12 @@ public class Arguments {
return value;
}
public String getString(String key, String description) {
String value = getRequiredArg(key, description);
logArgValue(key, description, value);
return value;
}
/** Returns a {@link Path} parsed from {@code key} argument, or fall back to a default if the argument is not set. */
public Path file(String key, String description, Path defaultValue) {
String value = getArg(key);
@ -219,13 +252,18 @@ public class Arguments {
/** Returns a {@link Path} parsed from {@code key} argument which may or may not exist. */
public Path file(String key, String description) {
String value = getRequiredArg(key, description);
Path file = Path.of(value);
logArgValue(key, description, file);
return file;
}
private String getRequiredArg(String key, String description) {
String value = getArg(key);
if (value == null) {
throw new IllegalArgumentException("Missing required parameter: " + key + " (" + description + ")");
}
Path file = Path.of(value);
logArgValue(key, description, file);
return file;
return value;
}
/**

61
planetiler-core/src/main/java/com/onthegomap/planetiler/expression/DataType.java 100644
Wyświetl plik

@ -0,0 +1,61 @@
package com.onthegomap.planetiler.expression;
import com.onthegomap.planetiler.reader.WithTags;
import com.onthegomap.planetiler.util.Parse;
import java.util.Objects;
import java.util.function.BiFunction;
import java.util.function.UnaryOperator;
/**
* Destination data types for an attribute that link the type to functions that can parse the value from an input object
*/
public enum DataType implements BiFunction<WithTags, String, Object> {
GET_STRING("string", WithTags::getString, Objects::toString),
GET_BOOLEAN("boolean", WithTags::getBoolean, Parse::bool),
GET_DIRECTION("direction", WithTags::getDirection, Parse::direction),
GET_LONG("long", WithTags::getLong, Parse::parseLongOrNull),
GET_INT("integer", Parse::parseIntOrNull),
GET_DOUBLE("double", Parse::parseDoubleOrNull),
GET_TAG("get", WithTags::getTag, s -> s);
private final BiFunction<WithTags, String, Object> getter;
private final String id;
private final UnaryOperator<Object> parser;
DataType(String id, BiFunction<WithTags, String, Object> getter, UnaryOperator<Object> parser) {
this.id = id;
this.getter = getter;
this.parser = parser;
}
DataType(String id, UnaryOperator<Object> parser) {
this(id, (d, k) -> parser.apply(d.getTag(k)), parser);
}
@Override
public Object apply(WithTags withTags, String string) {
return this.getter.apply(withTags, string);
}
public Object convertFrom(Object value) {
return this.parser.apply(value);
}
/** Returns the data type associated with {@code id}, or {@link #GET_TAG} as a fallback. */
public static DataType from(String id) {
for (var value : values()) {
if (value.id.equals(id)) {
return value;
}
}
return GET_TAG;
}
public String id() {
return id;
}
public UnaryOperator<Object> parser() {
return parser;
}
}

270
planetiler-core/src/main/java/com/onthegomap/planetiler/expression/Expression.java
Wyświetl plik

@ -1,10 +1,11 @@
package com.onthegomap.planetiler.expression;
import com.onthegomap.planetiler.reader.SourceFeature;
import static com.onthegomap.planetiler.expression.DataType.GET_TAG;
import com.onthegomap.planetiler.reader.WithGeometryType;
import com.onthegomap.planetiler.reader.WithTags;
import com.onthegomap.planetiler.util.Format;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Objects;
import java.util.Set;
@ -13,6 +14,7 @@ import java.util.function.Predicate;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
import java.util.stream.Stream;
import org.apache.logging.log4j.util.Strings;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@ -28,18 +30,18 @@ import org.slf4j.LoggerFactory;
* }
* </pre>
*/
public interface Expression {
// TODO rename to BooleanExpression
public interface Expression extends Simplifiable<Expression> {
Logger LOGGER = LoggerFactory.getLogger(Expression.class);
String LINESTRING_TYPE = "linestring";
String POINT_TYPE = "point";
String POLYGON_TYPE = "polygon";
String RELATION_MEMBER_TYPE = "relation_member";
String UNKNOWN_GEOMETRY_TYPE = "unknown_type";
Set<String> supportedTypes = Set.of(LINESTRING_TYPE, POINT_TYPE, POLYGON_TYPE, RELATION_MEMBER_TYPE);
Set<String> supportedTypes = Set.of(LINESTRING_TYPE, POINT_TYPE, POLYGON_TYPE, UNKNOWN_GEOMETRY_TYPE);
Expression TRUE = new Constant(true, "TRUE");
Expression FALSE = new Constant(false, "FALSE");
BiFunction<WithTags, String, Object> GET_TAG = WithTags::getTag;
List<String> dummyList = new NoopList<>();
@ -78,7 +80,7 @@ public interface Expression {
* {@code values} can contain exact matches, "%text%" to match any value containing "text", or "" to match any value.
*/
static MatchAny matchAny(String field, List<?> values) {
return new MatchAny(field, GET_TAG, values);
return MatchAny.from(field, GET_TAG, values);
}
/**
@ -99,7 +101,7 @@ public interface Expression {
*/
static MatchAny matchAnyTyped(String field, BiFunction<WithTags, String, Object> typeGetter,
List<?> values) {
return new MatchAny(field, typeGetter, values);
return MatchAny.from(field, typeGetter, values);
}
/** Returns an expression that evaluates to true if the element has any value for tag {@code field}. */
@ -129,80 +131,6 @@ public interface Expression {
return items.stream().map(Expression::generateJavaCode).collect(Collectors.joining(", "));
}
private static Expression simplify(Expression initial) {
// iteratively simplify the expression until we reach a fixed point and start seeing
// an expression that's already been seen before
Expression simplified = initial;
Set<Expression> seen = new HashSet<>();
seen.add(simplified);
while (true) {
simplified = simplifyOnce(simplified);
if (seen.contains(simplified)) {
return simplified;
}
if (seen.size() > 1000) {
throw new IllegalStateException("Infinite loop while simplifying expression " + initial);
}
seen.add(simplified);
}
}
private static Expression simplifyOnce(Expression expression) {
if (expression instanceof Not not) {
if (not.child instanceof Or or) {
return and(or.children.stream().<Expression>map(Expression::not).toList());
} else if (not.child instanceof And and) {
return or(and.children.stream().<Expression>map(Expression::not).toList());
} else if (not.child instanceof Not not2) {
return not2.child;
} else if (not.child == TRUE) {
return FALSE;
} else if (not.child == FALSE) {
return TRUE;
} else if (not.child instanceof MatchAny any && any.values.equals(List.of(""))) {
return matchField(any.field);
}
return not;
} else if (expression instanceof Or or) {
if (or.children.isEmpty()) {
return FALSE;
}
if (or.children.size() == 1) {
return simplifyOnce(or.children.get(0));
}
if (or.children.contains(TRUE)) {
return TRUE;
}
return or(or.children.stream()
// hoist children
.flatMap(child -> child instanceof Or childOr ? childOr.children.stream() : Stream.of(child))
.filter(child -> child != FALSE) // or() == or(FALSE) == or(FALSE, FALSE) == FALSE, so safe to remove all here
.map(Expression::simplifyOnce).toList());
} else if (expression instanceof And and) {
if (and.children.isEmpty()) {
return TRUE;
}
if (and.children.size() == 1) {
return simplifyOnce(and.children.get(0));
}
if (and.children.contains(FALSE)) {
return FALSE;
}
return and(and.children.stream()
// hoist children
.flatMap(child -> child instanceof And childAnd ? childAnd.children.stream() : Stream.of(child))
.filter(child -> child != TRUE) // and() == and(TRUE) == and(TRUE, TRUE) == TRUE, so safe to remove all here
.map(Expression::simplifyOnce).toList());
} else {
return expression;
}
}
/** Returns an equivalent, simplified copy of this expression but does not modify {@code this}. */
default Expression simplify() {
return simplify(this);
}
/** Returns a copy of this expression where every nested instance of {@code a} is replaced with {@code b}. */
default Expression replace(Expression a, Expression b) {
return replace(a::equals, b);
@ -251,8 +179,6 @@ public interface Expression {
//A list that silently drops all additions
class NoopList<T> extends ArrayList<T> {
private static final long serialVersionUID = 1L;
@Override
public boolean add(T t) {
return true;
@ -302,6 +228,25 @@ public interface Expression {
}
return true;
}
@Override
public Expression simplifyOnce() {
if (children.isEmpty()) {
return TRUE;
}
if (children.size() == 1) {
return children.get(0).simplifyOnce();
}
if (children.contains(FALSE)) {
return FALSE;
}
return and(children.stream()
// hoist children
.flatMap(child -> child instanceof And childAnd ? childAnd.children.stream() : Stream.of(child))
.filter(child -> child != TRUE) // and() == and(TRUE) == and(TRUE, TRUE) == TRUE, so safe to remove all here
.distinct()
.map(Simplifiable::simplifyOnce).toList());
}
}
record Or(List<Expression> children) implements Expression {
@ -338,6 +283,24 @@ public interface Expression {
return Objects.hash(children);
}
@Override
public Expression simplifyOnce() {
if (children.isEmpty()) {
return FALSE;
}
if (children.size() == 1) {
return children.get(0).simplifyOnce();
}
if (children.contains(TRUE)) {
return TRUE;
}
return or(children.stream()
// hoist children
.flatMap(child -> child instanceof Or childOr ? childOr.children.stream() : Stream.of(child))
.filter(child -> child != FALSE) // or() == or(FALSE) == or(FALSE, FALSE) == FALSE, so safe to remove all here
.distinct()
.map(Simplifiable::simplifyOnce).toList());
}
}
record Not(Expression child) implements Expression {
@ -351,6 +314,24 @@ public interface Expression {
public boolean evaluate(WithTags input, List<String> matchKeys) {
return !child.evaluate(input, new ArrayList<>());
}
@Override
public Expression simplifyOnce() {
if (child instanceof Or or) {
return and(or.children.stream().<Expression>map(Expression::not).toList());
} else if (child instanceof And and) {
return or(and.children.stream().<Expression>map(Expression::not).toList());
} else if (child instanceof Not not2) {
return not2.child;
} else if (child == TRUE) {
return FALSE;
} else if (child == FALSE) {
return TRUE;
} else if (child instanceof MatchAny any && any.values.equals(List.of(""))) {
return matchField(any.field);
}
return this;
}
}
/**
@ -359,35 +340,78 @@ public interface Expression {
*
* @param values all raw string values that were initially provided
* @param exactMatches the input {@code values} that should be treated as exact matches
* @param wildcards the input {@code values} that should be treated as wildcards
* @param pattern regular expression that the value must match, or null
* @param matchWhenMissing if {@code values} contained ""
*/
record MatchAny(
String field, List<?> values, Set<String> exactMatches, List<String> wildcards, boolean matchWhenMissing,
String field, List<?> values, Set<String> exactMatches,
Pattern pattern,
boolean matchWhenMissing,
BiFunction<WithTags, String, Object> valueGetter
) implements Expression {
private static final Pattern containsPattern = Pattern.compile("^%(.*)%$");
static MatchAny from(String field, BiFunction<WithTags, String, Object> valueGetter, List<?> values) {
List<String> exactMatches = new ArrayList<>();
List<String> patterns = new ArrayList<>();
MatchAny(String field, BiFunction<WithTags, String, Object> valueGetter, List<?> values) {
this(field, values,
values.stream().map(Object::toString).filter(v -> !v.contains("%")).collect(Collectors.toSet()),
values.stream().map(Object::toString).filter(v -> v.contains("%")).map(val -> {
var matcher = containsPattern.matcher(val);
if (!matcher.matches()) {
throw new IllegalArgumentException("wildcards must start/end with %: " + val);
for (var value : values) {
if (value != null) {
String string = value.toString();
if (string.matches("^.*(?<!\\\\)%.*$")) {
patterns.add(wildcardToRegex(string));
} else {
exactMatches.add(unescape(string));
}
return matcher.group(1);
}).toList(),
values.contains(""),
}
}
boolean matchWhenMissing = values.stream().anyMatch(v -> v == null || "".equals(v));
return new MatchAny(field, values,
Set.copyOf(exactMatches),
patterns.isEmpty() ? null : Pattern.compile("(" + Strings.join(patterns, '|') + ")"),
matchWhenMissing,
valueGetter
);
}
private static String wildcardToRegex(String string) {
StringBuilder regex = new StringBuilder("^");
StringBuilder token = new StringBuilder();
while (!string.isEmpty()) {
if (string.startsWith("\\%")) {
if (!token.isEmpty()) {
regex.append(Pattern.quote(token.toString()));
}
token.setLength(0);
regex.append("%");
string = string.replaceFirst("^\\\\%", "");
} else if (string.startsWith("%")) {
if (!token.isEmpty()) {
regex.append(Pattern.quote(token.toString()));
}
token.setLength(0);
regex.append(".*");
string = string.replaceFirst("^%+", "");
} else {
token.append(string.charAt(0));
string = string.substring(1);
}
}
if (!token.isEmpty()) {
regex.append(Pattern.quote(token.toString()));
}
regex.append('$');
return regex.toString();
}
private static String unescape(String input) {
return input.replace("\\%", "%");
}
@Override
public boolean evaluate(WithTags input, List<String> matchKeys) {
Object value = valueGetter.apply(input, field);
if (value == null) {
if (value == null || "".equals(value)) {
return matchWhenMissing;
} else {
String str = value.toString();
@ -395,16 +419,19 @@ public interface Expression {
matchKeys.add(field);
return true;
}
for (String target : wildcards) {
if (str.contains(target)) {
matchKeys.add(field);
return true;
}
if (pattern != null && pattern.matcher(str).matches()) {
matchKeys.add(field);
return true;
}
return false;
}
}
@Override
public Expression simplifyOnce() {
return isMatchAnything() ? matchField(field) : this;
}
@Override
public String generateJavaCode() {
// java code generation only needed for the simple cases used by openmaptiles schema generation
@ -424,6 +451,31 @@ public interface Expression {
}
return "matchAny(" + Format.quote(field) + ", " + String.join(", ", valueStrings) + ")";
}
public boolean isMatchAnything() {
return !matchWhenMissing && exactMatches.isEmpty() && (pattern != null && pattern.toString().equals("(^.*$)"));
}
@Override
public boolean equals(Object o) {
return this == o || (o instanceof MatchAny matchAny &&
matchWhenMissing == matchAny.matchWhenMissing &&
Objects.equals(field, matchAny.field) &&
Objects.equals(values, matchAny.values) &&
Objects.equals(exactMatches, matchAny.exactMatches) &&
// Patterns for the same input string are not equal
Objects.equals(patternString(), matchAny.patternString()) &&
Objects.equals(valueGetter, matchAny.valueGetter));
}
private String patternString() {
return pattern == null ? null : pattern.pattern();
}
@Override
public int hashCode() {
return Objects.hash(field, values, exactMatches, patternString(), matchWhenMissing, valueGetter);
}
}
/** Evaluates to true if an input element contains any value for {@code field} tag. */
@ -436,7 +488,8 @@ public interface Expression {
@Override
public boolean evaluate(WithTags input, List<String> matchKeys) {
if (input.hasTag(field)) {
Object value = input.getTag(field);
if (value != null && !"".equals(value)) {
matchKeys.add(field);
return true;
}
@ -456,12 +509,11 @@ public interface Expression {
@Override
public boolean evaluate(WithTags input, List<String> matchKeys) {
if (input instanceof SourceFeature sourceFeature) {
if (input instanceof WithGeometryType withGeom) {
return switch (type) {
case LINESTRING_TYPE -> sourceFeature.canBeLine();
case POLYGON_TYPE -> sourceFeature.canBePolygon();
case POINT_TYPE -> sourceFeature.isPoint();
case RELATION_MEMBER_TYPE -> sourceFeature.hasRelationInfo();
case LINESTRING_TYPE -> withGeom.canBeLine();
case POLYGON_TYPE -> withGeom.canBePolygon();
case POINT_TYPE -> withGeom.isPoint();
default -> false;
};
} else {

116
planetiler-core/src/main/java/com/onthegomap/planetiler/expression/MultiExpression.java
Wyświetl plik

@ -3,10 +3,9 @@ package com.onthegomap.planetiler.expression;
import static com.onthegomap.planetiler.expression.Expression.FALSE;
import static com.onthegomap.planetiler.expression.Expression.TRUE;
import static com.onthegomap.planetiler.expression.Expression.matchType;
import static com.onthegomap.planetiler.geo.GeoUtils.EMPTY_GEOMETRY;
import com.onthegomap.planetiler.reader.SimpleFeature;
import com.onthegomap.planetiler.reader.SourceFeature;
import com.onthegomap.planetiler.reader.WithGeometryType;
import com.onthegomap.planetiler.reader.WithTags;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.HashMap;
@ -28,12 +27,12 @@ import org.slf4j.LoggerFactory;
* {@link #index()} returns an optimized {@link Index} that evaluates the minimal set of expressions on the keys present
* on the element.
* <p>
* {@link Index#getMatches(SourceFeature)} returns the data value associated with the expressions that match an input
* {@link Index#getMatches(WithTags)} )} returns the data value associated with the expressions that match an input
* element.
*
* @param <T> type of data value associated with each expression
*/
public record MultiExpression<T> (List<Entry<T>> expressions) {
public record MultiExpression<T> (List<Entry<T>> expressions) implements Simplifiable<MultiExpression<T>> {
private static final Logger LOGGER = LoggerFactory.getLogger(MultiExpression.class);
private static final Comparator<WithId> BY_ID = Comparator.comparingInt(WithId::id);
@ -46,25 +45,6 @@ public record MultiExpression<T> (List<Entry<T>> expressions) {
return new Entry<>(result, expression);
}
/**
* Evaluates a list of expressions on an input element, storing the matches into {@code result} and using {@code
* visited} to avoid evaluating an expression more than once.
*/
private static <T> void visitExpressions(SourceFeature input, List<Match<T>> result,
boolean[] visited, List<EntryWithId<T>> expressions) {
if (expressions != null) {
for (EntryWithId<T> expressionValue : expressions) {
if (!visited[expressionValue.id]) {
visited[expressionValue.id] = true;
List<String> matchKeys = new ArrayList<>();
if (expressionValue.expression().evaluate(input, matchKeys)) {
result.add(new Match<>(expressionValue.result, matchKeys, expressionValue.id));
}
}
}
}
}
/**
* Returns true if {@code expression} only contains "not filter" so we can't limit evaluating this expression to only
* when a particular key is present on the input.
@ -79,7 +59,9 @@ public record MultiExpression<T> (List<Entry<T>> expressions) {
} else if (expression instanceof Expression.MatchAny any && any.matchWhenMissing()) {
return true;
} else {
return TRUE.equals(expression);
return !(expression instanceof Expression.MatchAny) &&
!(expression instanceof Expression.MatchField) &&
!FALSE.equals(expression);
}
}
@ -136,8 +118,8 @@ public record MultiExpression<T> (List<Entry<T>> expressions) {
}
/**
* Returns a copy of this multi-expression that replaces every sub-expression that matches {@code test} with {@code
* b}.
* Returns a copy of this multi-expression that replaces every sub-expression that matches {@code test} with
* {@code b}.
*/
public MultiExpression<T> replace(Predicate<Expression> test, Expression b) {
return map(e -> e.replace(test, b));
@ -151,8 +133,9 @@ public record MultiExpression<T> (List<Entry<T>> expressions) {
}
/** Returns a copy of this multi-expression with each expression simplified. */
public MultiExpression<T> simplify() {
return map(e -> e.simplify());
@Override
public MultiExpression<T> simplifyOnce() {
return map(Simplifiable::simplify);
}
/** Returns a copy of this multi-expression, filtering-out the entry for each data value matching {@code accept}. */
@ -176,37 +159,36 @@ public record MultiExpression<T> (List<Entry<T>> expressions) {
/**
* An optimized index for finding which expressions match an input element.
*
* @param <T> type of data value associated with each expression
* @param <O> type of data value associated with each expression
*/
public interface Index<T> {
public interface Index<O> {
List<Match<T>> getMatchesWithTriggers(SourceFeature input);
List<Match<O>> getMatchesWithTriggers(WithTags input);
/** Returns all data values associated with expressions that match an input element. */
default List<T> getMatches(SourceFeature input) {
List<Match<T>> matches = getMatchesWithTriggers(input);
return matches.stream().sorted(BY_ID).map(d -> d.match).toList();
default List<O> getMatches(WithTags input) {
return getMatchesWithTriggers(input).stream().map(d -> d.match).toList();
}
/**
* Returns the data value associated with the first expression that match an input element, or {@code defaultValue}
* if none match.
*/
default T getOrElse(SourceFeature input, T defaultValue) {
List<T> matches = getMatches(input);
default O getOrElse(WithTags input, O defaultValue) {
List<O> matches = getMatches(input);
return matches.isEmpty() ? defaultValue : matches.get(0);
}
/**
* Returns the data value associated with expressions matching a feature with {@code tags}.
*/
default T getOrElse(Map<String, Object> tags, T defaultValue) {
List<T> matches = getMatches(SimpleFeature.create(EMPTY_GEOMETRY, tags));
default O getOrElse(Map<String, Object> tags, O defaultValue) {
List<O> matches = getMatches(WithTags.from(tags));
return matches.isEmpty() ? defaultValue : matches.get(0);
}
/** Returns true if any expression matches that tags from an input element. */
default boolean matches(SourceFeature input) {
default boolean matches(WithTags input) {
return !getMatchesWithTriggers(input).isEmpty();
}
@ -216,13 +198,14 @@ public record MultiExpression<T> (List<Entry<T>> expressions) {
}
private interface WithId {
int id();
}
private static class EmptyIndex<T> implements Index<T> {
@Override
public List<Match<T>> getMatchesWithTriggers(SourceFeature input) {
public List<Match<T>> getMatchesWithTriggers(WithTags input) {
return List.of();
}
@ -277,9 +260,28 @@ public record MultiExpression<T> (List<Entry<T>> expressions) {
numExpressions = id;
}
/**
* Evaluates a list of expressions on an input element, storing the matches into {@code result} and using
* {@code visited} to avoid evaluating an expression more than once.
*/
private static <T> void visitExpressions(WithTags input, List<Match<T>> result,
boolean[] visited, List<EntryWithId<T>> expressions) {
if (expressions != null) {
for (EntryWithId<T> expressionValue : expressions) {
if (!visited[expressionValue.id]) {
visited[expressionValue.id] = true;
List<String> matchKeys = new ArrayList<>();
if (expressionValue.expression().evaluate(input, matchKeys)) {
result.add(new Match<>(expressionValue.result, matchKeys, expressionValue.id));
}
}
}
}
}
/** Lookup matches in this index for expressions that match a certain type. */
@Override
public List<Match<T>> getMatchesWithTriggers(SourceFeature input) {
public List<Match<T>> getMatchesWithTriggers(WithTags input) {
List<Match<T>> result = new ArrayList<>();
boolean[] visited = new boolean[numExpressions];
visitExpressions(input, result, visited, alwaysEvaluateExpressionList);
@ -295,6 +297,7 @@ public record MultiExpression<T> (List<Entry<T>> expressions) {
}
}
}
result.sort(BY_ID);
return result;
}
}
@ -305,6 +308,7 @@ public record MultiExpression<T> (List<Entry<T>> expressions) {
private final KeyIndex<T> pointIndex;
private final KeyIndex<T> lineIndex;
private final KeyIndex<T> polygonIndex;
private final KeyIndex<T> otherIndex;
private GeometryTypeIndex(MultiExpression<T> expressions, boolean warn) {
// build an index per type then search in each of those indexes based on the geometry type of each input element
@ -312,6 +316,7 @@ public record MultiExpression<T> (List<Entry<T>> expressions) {
pointIndex = indexForType(expressions, Expression.POINT_TYPE, warn);
lineIndex = indexForType(expressions, Expression.LINESTRING_TYPE, warn);
polygonIndex = indexForType(expressions, Expression.POLYGON_TYPE, warn);
otherIndex = indexForType(expressions, Expression.UNKNOWN_GEOMETRY_TYPE, warn);
}
private KeyIndex<T> indexForType(MultiExpression<T> expressions, String type, boolean warn) {
@ -328,21 +333,26 @@ public record MultiExpression<T> (List<Entry<T>> expressions) {
* Returns all data values associated with expressions that match an input element, along with the tag keys that
* caused the match.
*/
public List<Match<T>> getMatchesWithTriggers(SourceFeature input) {
public List<Match<T>> getMatchesWithTriggers(WithTags input) {
List<Match<T>> result;
if (input.isPoint()) {
result = pointIndex.getMatchesWithTriggers(input);
} else if (input.canBeLine()) {
result = lineIndex.getMatchesWithTriggers(input);
// closed ways can be lines or polygons, unless area=yes or no
if (input.canBePolygon()) {
result.addAll(polygonIndex.getMatchesWithTriggers(input));
if (input instanceof WithGeometryType withGeometryType) {
if (withGeometryType.isPoint()) {
result = pointIndex.getMatchesWithTriggers(input);
} else if (withGeometryType.canBeLine()) {
result = lineIndex.getMatchesWithTriggers(input);
// closed ways can be lines or polygons, unless area=yes or no
if (withGeometryType.canBePolygon()) {
result.addAll(polygonIndex.getMatchesWithTriggers(input));
}
} else if (withGeometryType.canBePolygon()) {
result = polygonIndex.getMatchesWithTriggers(input);
} else {
result = otherIndex.getMatchesWithTriggers(input);
}
} else if (input.canBePolygon()) {
result = polygonIndex.getMatchesWithTriggers(input);
} else {
result = pointIndex.getMatchesWithTriggers(input);
result = otherIndex.getMatchesWithTriggers(input);
}
result.sort(BY_ID);
return result;
}
}

53
planetiler-core/src/main/java/com/onthegomap/planetiler/expression/Simplifiable.java 100644
Wyświetl plik

@ -0,0 +1,53 @@
package com.onthegomap.planetiler.expression;
import java.util.HashSet;
import java.util.Set;
/**
* An expression that can be simplified to an equivalent, but cheaper to evaluate expression.
* <p>
* Implementers should only override {@link #simplifyOnce()} which applies all the rules that can be used to simplify
* this expression, and {@link #simplify()} will take care of applying it repeatedly until the output settles to a fixed
* point.
* <p>
* Implementers must also ensure {@code equals} and {@code hashCode} reflect equivalence between expressions so that
* {@link #simplify()} can know when to stop.
*/
public interface Simplifiable<T extends Simplifiable<T>> {
/**
* Returns a copy of this expression, with all simplification rules applied once.
* <p>
* {@link #simplify()} will take care of applying it repeatedly until the output settles.
*/
default T simplifyOnce() {
return self();
}
default T self() {
@SuppressWarnings("unchecked") T self = (T) this;
return self;
}
/**
* Returns an equivalent, simplified copy of this expression but does not modify {@code this} by repeatedly running
* {@link #simplifyOnce()}.
*/
default T simplify() {
// iteratively simplify the expression until we reach a fixed point and start seeing
// an expression that's already been seen before
T simplified = self();
Set<T> seen = new HashSet<>();
seen.add(simplified);
while (true) {
simplified = simplified.simplifyOnce();
if (seen.contains(simplified)) {
return simplified;
}
if (seen.size() > 1000) {
throw new IllegalStateException("Infinite loop while simplifying " + this);
}
seen.add(simplified);
}
}
}

32
planetiler-core/src/main/java/com/onthegomap/planetiler/geo/GeometryType.java
Wyświetl plik

@ -1,11 +1,7 @@
package com.onthegomap.planetiler.geo;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.onthegomap.planetiler.FeatureCollector;
import com.onthegomap.planetiler.FeatureCollector.Feature;
import com.onthegomap.planetiler.expression.Expression;
import java.util.function.BiFunction;
import java.util.function.Function;
import org.locationtech.jts.geom.Geometry;
import org.locationtech.jts.geom.Lineal;
import org.locationtech.jts.geom.Polygonal;
@ -13,30 +9,25 @@ import org.locationtech.jts.geom.Puntal;
import vector_tile.VectorTileProto;
public enum GeometryType {
UNKNOWN(VectorTileProto.Tile.GeomType.UNKNOWN, 0, (f, l) -> {
throw new UnsupportedOperationException();
}, "unknown"),
UNKNOWN(VectorTileProto.Tile.GeomType.UNKNOWN, 0, "unknown"),
@JsonProperty("point")
POINT(VectorTileProto.Tile.GeomType.POINT, 1, FeatureCollector::point, "point"),
POINT(VectorTileProto.Tile.GeomType.POINT, 1, "point"),
@JsonProperty("line")
LINE(VectorTileProto.Tile.GeomType.LINESTRING, 2, FeatureCollector::line, "linestring"),
LINE(VectorTileProto.Tile.GeomType.LINESTRING, 2, "linestring"),
@JsonProperty("polygon")
POLYGON(VectorTileProto.Tile.GeomType.POLYGON, 4, FeatureCollector::polygon, "polygon");
POLYGON(VectorTileProto.Tile.GeomType.POLYGON, 4, "polygon");
private final VectorTileProto.Tile.GeomType protobufType;
private final int minPoints;
private final BiFunction<FeatureCollector, String, Feature> geometryFactory;
private final String matchTypeString;
GeometryType(VectorTileProto.Tile.GeomType protobufType, int minPoints,
BiFunction<FeatureCollector, String, Feature> geometryFactory, String matchTypeString) {
GeometryType(VectorTileProto.Tile.GeomType protobufType, int minPoints, String matchTypeString) {
this.protobufType = protobufType;
this.minPoints = minPoints;
this.geometryFactory = geometryFactory;
this.matchTypeString = matchTypeString;
}
public static GeometryType valueOf(Geometry geom) {
public static GeometryType typeOf(Geometry geom) {
return geom instanceof Puntal ? POINT : geom instanceof Lineal ? LINE : geom instanceof Polygonal ? POLYGON :
UNKNOWN;
}
@ -66,17 +57,6 @@ public enum GeometryType {
return minPoints;
}
/**
* Generates a factory method which creates a {@link Feature} from a {@link FeatureCollector} of the appropriate
* geometry type.
*
* @param layerName - name of the layer
* @return geometry factory method
*/
public Function<FeatureCollector, Feature> geometryFactory(String layerName) {
return features -> geometryFactory.apply(features, layerName);
}
/**
* Generates a test for whether a source feature is of the correct geometry to be included in the tile.
*

24
planetiler-core/src/main/java/com/onthegomap/planetiler/reader/SourceFeature.java
Wyświetl plik

@ -11,9 +11,7 @@ import org.locationtech.jts.geom.Geometry;
import org.locationtech.jts.geom.LineString;
import org.locationtech.jts.geom.Lineal;
import org.locationtech.jts.geom.MultiLineString;
import org.locationtech.jts.geom.MultiPoint;
import org.locationtech.jts.geom.MultiPolygon;
import org.locationtech.jts.geom.Point;
import org.locationtech.jts.geom.Polygon;
/**
@ -26,7 +24,7 @@ import org.locationtech.jts.geom.Polygon;
* All geometries except for {@link #latLonGeometry()} return elements in world web mercator coordinates where (0,0) is
* the northwest corner and (1,1) is the southeast corner of the planet.
*/
public abstract class SourceFeature implements WithTags {
public abstract class SourceFeature implements WithTags, WithGeometryType {
private final Map<String, Object> tags;
private final String source;
@ -247,25 +245,6 @@ public abstract class SourceFeature implements WithTags {
(isPoint() || canBePolygon() || canBeLine()) ? worldGeometry().getLength() : 0) : length;
}
/** Returns true if this feature can be interpreted as a {@link Point} or {@link MultiPoint}. */
public abstract boolean isPoint();
/**
* Returns true if this feature can be interpreted as a {@link Polygon} or {@link MultiPolygon}.
* <p>
* A closed ring can either be a polygon or linestring, so return false to not allow this closed ring to be treated as
* a polygon.
*/
public abstract boolean canBePolygon();
/**
* Returns true if this feature can be interpreted as a {@link LineString} or {@link MultiLineString}.
* <p>
* A closed ring can either be a polygon or linestring, so return false to not allow this closed ring to be treated as
* a linestring.
*/
public abstract boolean canBeLine();
/** Returns the ID of the source that this feature came from. */
public String getSource() {
return source;
@ -308,6 +287,7 @@ public abstract class SourceFeature implements WithTags {
return id;
}
/** Returns true if this element has any OSM relation info. */
public boolean hasRelationInfo() {
return relationInfos != null && !relationInfos.isEmpty();

34
planetiler-core/src/main/java/com/onthegomap/planetiler/reader/WithGeometryType.java 100644
Wyświetl plik

@ -0,0 +1,34 @@
package com.onthegomap.planetiler.reader;
import org.locationtech.jts.geom.LineString;
import org.locationtech.jts.geom.MultiLineString;
import org.locationtech.jts.geom.MultiPoint;
import org.locationtech.jts.geom.MultiPolygon;
import org.locationtech.jts.geom.Point;
import org.locationtech.jts.geom.Polygon;
/**
* Something attached to a geometry that can be matched using a
* {@link com.onthegomap.planetiler.expression.Expression.MatchType} geometry type filter expression.
*/
public interface WithGeometryType {
/** Returns true if this feature can be interpreted as a {@link Point} or {@link MultiPoint}. */
boolean isPoint();
/**
* Returns true if this feature can be interpreted as a {@link Polygon} or {@link MultiPolygon}.
* <p>
* A closed ring can either be a polygon or linestring, so return false to not allow this closed ring to be treated as
* a polygon.
*/
boolean canBePolygon();
/**
* Returns true if this feature can be interpreted as a {@link LineString} or {@link MultiLineString}.
* <p>
* A closed ring can either be a polygon or linestring, so return false to not allow this closed ring to be treated as
* a linestring.
*/
boolean canBeLine();
}

5
planetiler-core/src/main/java/com/onthegomap/planetiler/reader/WithTags.java
Wyświetl plik

@ -6,6 +6,9 @@ import java.util.Map;
/** An input element with a set of string key/object value pairs. */
public interface WithTags {
static WithTags from(Map<String, Object> tags) {
return new OfMap(tags);
}
/** The key/value pairs on this element. */
Map<String, Object> tags();
@ -108,4 +111,6 @@ public interface WithTags {
default void setTag(String key, Object value) {
tags().put(key, value);
}
record OfMap(@Override Map<String, Object> tags) implements WithTags {}
}

30
planetiler-core/src/main/java/com/onthegomap/planetiler/stats/ProgressLoggers.java
Wyświetl plik

@ -1,5 +1,9 @@
package com.onthegomap.planetiler.stats;
import static com.onthegomap.planetiler.util.AnsiColors.blue;
import static com.onthegomap.planetiler.util.AnsiColors.green;
import static com.onthegomap.planetiler.util.AnsiColors.red;
import static com.onthegomap.planetiler.util.AnsiColors.yellow;
import static com.onthegomap.planetiler.util.Exceptions.throwFatalException;
import static com.onthegomap.planetiler.util.Format.padLeft;
import static com.onthegomap.planetiler.util.Format.padRight;
@ -39,36 +43,10 @@ import org.slf4j.LoggerFactory;
*/
@SuppressWarnings({"UnusedReturnValue", "unused"})
public class ProgressLoggers {
private static final String COLOR_RESET = "\u001B[0m";
private static final String FG_RED = "\u001B[31m";
private static final String FG_GREEN = "\u001B[32m";
private static final String FG_YELLOW = "\u001B[33m";
private static final String FG_BLUE = "\u001B[34m";
private static final Logger LOGGER = LoggerFactory.getLogger(ProgressLoggers.class);
private final List<Object> loggers = new ArrayList<>();
private final Format format;
private static String fg(String fg, String string) {
return fg + string + COLOR_RESET;
}
private static String red(String string) {
return fg(FG_RED, string);
}
private static String green(String string) {
return fg(FG_GREEN, string);
}
private static String yellow(String string) {
return fg(FG_YELLOW, string);
}
private static String blue(String string) {
return fg(FG_BLUE, string);
}
private ProgressLoggers(Locale locale) {
this.format = Format.forLocale(locale);
}

50
planetiler-core/src/main/java/com/onthegomap/planetiler/util/AnsiColors.java 100644
Wyświetl plik

@ -0,0 +1,50 @@
package com.onthegomap.planetiler.util;
/** Utilities for styling terminal output. */
public class AnsiColors {
private AnsiColors() {}
private static final String COLOR_RESET = "\u001B[0m";
private static final String FG_RED = "\u001B[31m";
private static final String FG_GREEN = "\u001B[32m";
private static final String FG_YELLOW = "\u001B[33m";
private static final String FG_BLUE = "\u001B[34m";
private static final String REVERSE = "\u001B[7m";
private static final String BOLD = "\u001B[1m";
private static String color(String fg, String string) {
return fg + string + COLOR_RESET;
}
public static String red(String string) {
return color(FG_RED, string);
}
public static String green(String string) {
return color(FG_GREEN, string);
}
public static String yellow(String string) {
return color(FG_YELLOW, string);
}
public static String blue(String string) {
return color(FG_BLUE, string);
}
public static String redBackground(String string) {
return color(REVERSE + BOLD + FG_RED, string);
}
public static String greenBackground(String string) {
return color(REVERSE + BOLD + FG_GREEN, string);
}
public static String redBold(String string) {
return color(BOLD + FG_RED, string);
}
public static String greenBold(String string) {
return color(BOLD + FG_GREEN, string);
}
}

5
planetiler-core/src/main/java/com/onthegomap/planetiler/util/CacheByZoom.java
Wyświetl plik

@ -23,13 +23,12 @@ public class CacheByZoom<T> {
/**
* Returns a cache for {@code supplier} that can handle a min/max zoom range specified in {@code config}.
*
* @param config min/max zoom range this can handle
* @param supplier function that will be called with each zoom-level to get the value
* @param <T> return type of the function
* @return a cache for {@code supplier} by zom
*/
public static <T> CacheByZoom<T> create(PlanetilerConfig config, IntFunction<T> supplier) {
return new CacheByZoom<>(config.minzoom(), config.maxzoomForRendering(), supplier);
public static <T> CacheByZoom<T> create(IntFunction<T> supplier) {
return new CacheByZoom<>(0, PlanetilerConfig.MAX_MAXZOOM, supplier);
}
public T get(int zoom) {

122
planetiler-core/src/main/java/com/onthegomap/planetiler/util/FileWatcher.java 100644
Wyświetl plik

@ -0,0 +1,122 @@
package com.onthegomap.planetiler.util;
import static com.onthegomap.planetiler.util.Exceptions.throwFatalException;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Duration;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.TreeMap;
import java.util.TreeSet;
import java.util.stream.Collectors;
/**
* Watches a set of paths so that each time you call {@link #poll()} it returns the set of paths that have been modified
* since the last call to {@link #poll()}.
*/
public class FileWatcher {
private final Map<Path, Long> modificationTimes = new TreeMap<>();
/** Returns the canonical form of {@code path}. */
static Path normalize(Path path) {
return path.toAbsolutePath().normalize();
}
/** Returns a new file watcher watching a set of files for modifications. */
public static FileWatcher newWatcher(Path... paths) {
var watcher = new FileWatcher();
for (var path : paths) {
watcher.watch(path);
}
return watcher;
}
/** Returns the (normalized) paths modified since the last call to poll. */
public Set<Path> poll() {
Set<Path> result = new TreeSet<>();
for (var path : List.copyOf(modificationTimes.keySet())) {
if (watch(path)) {
result.add(path);
}
}
return result;
}
/** Adds {@code path} to the set of paths to check on each call to {@link #poll()}. */
public boolean watch(Path path) {
path = normalize(path);
Long modifiedTime;
try {
modifiedTime = Files.getLastModifiedTime(path).toMillis();
} catch (IOException e) {
modifiedTime = null;
}
var last = modificationTimes.put(path, modifiedTime);
return !Objects.equals(modifiedTime, last);
}
/** Removes {@code path} from the set of paths to check on each call to {@link #poll()}. */
public void unwatch(Path path) {
modificationTimes.remove(normalize(path));
}
/** Returns true if we are currently watching {@code path} for changes. */
public boolean watching(Path path) {
return modificationTimes.containsKey(normalize(path));
}
/** Ensures we are only watching {@code paths} provided. */
public void setWatched(Set<Path> paths) {
if (paths == null || paths.isEmpty()) {
return;
}
paths = paths.stream().map(FileWatcher::normalize).collect(Collectors.toSet());
for (var toWatch : paths) {
if (!watching(toWatch)) {
watch(toWatch);
}
}
for (var watching : Set.copyOf(modificationTimes.keySet())) {
if (!paths.contains(watching)) {
unwatch(watching);
}
}
}
/**
* Blocks and invokes {@code action} every time one of the watched files changes, checking every {@code delay}
* interval.
*/
public void pollForChanges(Duration delay, FunctionThatThrows<Set<Path>, Set<Path>> action) {
while (!Thread.currentThread().isInterrupted()) {
var changes = poll();
if (!changes.isEmpty()) {
setWatched(action.runAndWrapException(changes));
}
try {
Thread.sleep(delay.toMillis());
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
}
@FunctionalInterface
public interface FunctionThatThrows<I, O> {
@SuppressWarnings("java:S112")
O apply(I value) throws Exception;
default O runAndWrapException(I value) {
try {
return apply(value);
} catch (Exception e) {
return throwFatalException(e);
}
}
}
}

8
planetiler-core/src/main/java/com/onthegomap/planetiler/util/Imposm3Parsers.java
Wyświetl plik

@ -71,10 +71,12 @@ public class Imposm3Parsers {
*
* @see <a href="https://wiki.openstreetmap.org/wiki/Key:oneway">OSM one-way</a>
*/
public static int direction(Object string) {
if (string == null) {
public static int direction(Object obj) {
if (obj == null) {
return 0;
} else if (forwardDirections.contains(string(string))) {
}
String string = string(obj);
if (forwardDirections.contains(string)) {
return 1;
} else if ("-1".equals(string)) {
return -1;

64
planetiler-core/src/main/java/com/onthegomap/planetiler/util/Parse.java
Wyświetl plik

@ -1,6 +1,10 @@
package com.onthegomap.planetiler.util;
import java.text.NumberFormat;
import java.text.ParseException;
import java.util.Locale;
import java.util.Map;
import java.util.function.Function;
import java.util.regex.Pattern;
/**
@ -8,8 +12,6 @@ import java.util.regex.Pattern;
*/
public class Parse {
private Parse() {}
private static final Pattern INT_SUBSTRING_PATTERN = Pattern.compile("^(-?\\d+)(\\D|$)");
private static final Pattern TO_ROUND_INT_SUBSTRING_PATTERN = Pattern.compile("^(-?[\\d.]+)(\\D|$)");
// See https://wiki.openstreetmap.org/wiki/Map_features/Units
@ -17,13 +19,31 @@ public class Parse {
Pattern.compile(
"(?<value>-?[\\d.]+)\\s*((?<mi>mi)|(?<m>m|$)|(?<km>km|kilom)|(?<ft>ft|')|(?<in>in|\")|(?<nmi>nmi|international nautical mile|nautical))",
Pattern.CASE_INSENSITIVE);
private static final NumberFormat PARSER = NumberFormat.getNumberInstance(Locale.ROOT);
private Parse() {}
/** Returns {@code tag} as a long or null if invalid. */
public static Long parseLongOrNull(Object tag) {
return tag == null ? null : tag instanceof Number number ? Long.valueOf(number.longValue()) :
parseLongOrNull(tag.toString());
}
/** Returns {@code tag} as a long or null if invalid. */
public static Long parseLongOrNull(String tag) {
try {
return tag == null ? null : tag instanceof Number number ? number.longValue() : Long.parseLong(tag.toString());
return tag == null ? null : Long.parseLong(tag);
} catch (NumberFormatException e) {
return null;
return retryParseNumber(tag, Number::longValue, null);
}
}
private static <T> T retryParseNumber(Object obj, Function<Number, T> getter, T backup) {
// more expensive parser in case simple valueOf parse fails
try {
return getter.apply(PARSER.parse(obj.toString()));
} catch (ParseException e) {
return backup;
}
}
@ -32,7 +52,7 @@ public class Parse {
try {
return tag == null ? 0 : tag instanceof Number number ? number.longValue() : Long.parseLong(tag.toString());
} catch (NumberFormatException e) {
return 0;
return retryParseNumber(tag, Number::longValue, 0L);
}
}
@ -64,16 +84,16 @@ public class Parse {
/** Returns {@code tag} as an integer or null if invalid. */
public static Integer parseIntOrNull(Object tag) {
if (tag instanceof Number num) {
return num.intValue();
}
if (!(tag instanceof String)) {
return null;
}
return tag == null ? null : tag instanceof Number number ? Integer.valueOf(number.intValue()) :
parseIntOrNull(tag.toString());
}
/** Returns {@code tag} as an integer or null if invalid. */
public static Integer parseIntOrNull(String tag) {
try {
return Integer.parseInt(tag.toString());
return tag == null ? null : Integer.parseInt(tag);
} catch (NumberFormatException e) {
return null;
return retryParseNumber(tag, Number::intValue, null);
}
}
@ -101,17 +121,17 @@ public class Parse {
}
/** Returns {@code tag} as a double or null if invalid. */
public static Double parseDoubleOrNull(Object value) {
if (value instanceof Number num) {
return num.doubleValue();
}
if (value == null) {
return null;
}
public static Double parseDoubleOrNull(Object tag) {
return tag == null ? null : tag instanceof Number number ? Double.valueOf(number.doubleValue()) :
parseDoubleOrNull(tag.toString());
}
/** Returns {@code tag} as a double or null if invalid. */
public static Double parseDoubleOrNull(String tag) {
try {
return Double.parseDouble(value.toString());
return tag == null ? null : Double.parseDouble(tag);
} catch (NumberFormatException e) {
return null;
return retryParseNumber(tag, Number::doubleValue, null);
}
}

62
planetiler-core/src/main/java/com/onthegomap/planetiler/util/Try.java 100644
Wyświetl plik

@ -0,0 +1,62 @@
package com.onthegomap.planetiler.util;
/**
* A container for the result of an operation that may succeed or fail.
*
* @param <T> Type of the result value, if success
*/
public interface Try<T> {
/**
* Calls {@code supplier} and wraps the result in {@link Success} if successful, or {@link Failure} if it throws an
* exception.
*/
static <T> Try<T> apply(SupplierThatThrows<T> supplier) {
try {
return success(supplier.get());
} catch (Exception e) {
return failure(e);
}
}
static <T> Success<T> success(T item) {
return new Success<>(item);
}
static <T> Failure<T> failure(Exception throwable) {
return new Failure<>(throwable);
}
/**
* Returns the result if success, or throws an exception if failure.
*
* @throws IllegalStateException wrapping the exception on failure
*/
T get();
default boolean isSuccess() {
return !isFailure();
}
default boolean isFailure() {
return exception() != null;
}
default Exception exception() {
return null;
}
record Success<T> (T get) implements Try<T> {}
record Failure<T> (@Override Exception exception) implements Try<T> {
@Override
public T get() {
throw new IllegalStateException(exception);
}
}
@FunctionalInterface
interface SupplierThatThrows<T> {
@SuppressWarnings("java:S112")
T get() throws Exception;
}
}

121
planetiler-core/src/test/java/com/onthegomap/planetiler/expression/ExpressionTest.java
Wyświetl plik

@ -2,13 +2,11 @@ package com.onthegomap.planetiler.expression;
import static com.onthegomap.planetiler.expression.Expression.*;
import static com.onthegomap.planetiler.expression.ExpressionTestUtil.featureWithTags;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.junit.jupiter.api.Assertions.*;
import com.onthegomap.planetiler.reader.WithTags;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;
import org.junit.jupiter.api.Test;
@ -30,6 +28,17 @@ class ExpressionTest {
);
}
@Test
void testSimplifyDuplicates() {
assertEquals(matchAB, or(or(matchAB), or(matchAB)).simplify());
assertEquals(matchAB, and(matchAB, matchAB).simplify());
}
@Test
void testMatchAnyEquals() {
assertEquals(matchAny("a", "b%"), matchAny("a", "b%"));
}
@Test
void testSimplifyOrWithOneChild() {
assertEquals(matchAB, or(matchAB).simplify());
@ -133,12 +142,82 @@ class ExpressionTest {
@Test
void testContains() {
assertNull(matchCD.pattern());
assertTrue(matchCD.contains(e -> e.equals(matchCD)));
assertTrue(or(not(matchCD)).contains(e -> e.equals(matchCD)));
assertFalse(matchCD.contains(e -> e.equals(matchAB)));
assertFalse(or(not(matchCD)).contains(e -> e.equals(matchAB)));
}
@Test
void testWildcardStartsWith() {
var matcher = matchAny("key", "a%");
assertEquals(Set.of(), matcher.exactMatches());
assertNotNull(matcher.pattern());
assertTrue(matcher.evaluate(featureWithTags("key", "abc")));
assertTrue(matcher.evaluate(featureWithTags("key", "a")));
assertFalse(matcher.evaluate(featureWithTags("key", "cba")));
}
@Test
void testWildcardEndsWith() {
var matcher = matchAny("key", "%a");
assertEquals(Set.of(), matcher.exactMatches());
assertNotNull(matcher.pattern());
assertTrue(matcher.evaluate(featureWithTags("key", "cba")));
assertTrue(matcher.evaluate(featureWithTags("key", "a")));
assertFalse(matcher.evaluate(featureWithTags("key", "abc")));
}
@Test
void testWildcardContains() {
var matcher = matchAny("key", "%a%");
assertEquals(Set.of(), matcher.exactMatches());
assertNotNull(matcher.pattern());
assertTrue(matcher.evaluate(featureWithTags("key", "bab")));
assertTrue(matcher.evaluate(featureWithTags("key", "a")));
assertFalse(matcher.evaluate(featureWithTags("key", "c")));
}
@Test
void testWildcardAny() {
var matcher = matchAny("key", "%");
assertEquals(Set.of(), matcher.exactMatches());
assertNotNull(matcher.pattern());
assertEquals(matchField("key"), matcher.simplify());
assertTrue(matcher.evaluate(featureWithTags("key", "abc")));
assertFalse(matcher.evaluate(featureWithTags("key", "")));
}
@Test
void testWildcardMiddle() {
var matcher = matchAny("key", "a%c");
assertEquals(Set.of(), matcher.exactMatches());
assertNotNull(matcher.pattern());
assertTrue(matcher.evaluate(featureWithTags("key", "abc")));
assertTrue(matcher.evaluate(featureWithTags("key", "ac")));
assertFalse(matcher.evaluate(featureWithTags("key", "ab")));
}
@Test
void testWildcardEscape() {
assertTrue(matchAny("key", "a\\%").evaluate(featureWithTags("key", "a%")));
assertFalse(matchAny("key", "a\\%").evaluate(featureWithTags("key", "ab")));
assertTrue(matchAny("key", "a\\%b").evaluate(featureWithTags("key", "a%b")));
assertTrue(matchAny("key", "%a\\%b%").evaluate(featureWithTags("key", "dda%b")));
assertTrue(matchAny("key", "\\%%").evaluate(featureWithTags("key", "%abc")));
assertTrue(matchAny("key", "%\\%").evaluate(featureWithTags("key", "abc%")));
assertTrue(matchAny("key", "%\\%%").evaluate(featureWithTags("key", "a%c")));
assertTrue(matchAny("key", "%\\%%").evaluate(featureWithTags("key", "%")));
assertFalse(matchAny("key", "\\%%").evaluate(featureWithTags("key", "abc%")));
}
@Test
void testStringifyExpression() {
//Ensure Expression.toString() returns valid Java code
@ -150,7 +229,7 @@ class ExpressionTest {
@Test
void testEvaluate() {
WithTags feature = featureWithTags("key1", "value1", "key2", "value2");
WithTags feature = featureWithTags("key1", "value1", "key2", "value2", "key3", "");
//And
assertTrue(and(matchAny("key1", "value1"), matchAny("key2", "value2")).evaluate(feature));
@ -173,9 +252,41 @@ class ExpressionTest {
assertFalse(matchField("wrong").evaluate(feature));
assertTrue(not(matchAny("key1", "")).evaluate(feature));
assertTrue(matchAny("wrong", "").evaluate(feature));
assertTrue(matchAny("key3", "").evaluate(feature));
//Constants
assertTrue(TRUE.evaluate(feature));
assertFalse(FALSE.evaluate(feature));
}
@Test
void testCustomExpression() {
Expression custom = new Expression() {
@Override
public boolean evaluate(WithTags input, List<String> matchKeys) {
return input.hasTag("abc");
}
@Override
public String generateJavaCode() {
return null;
}
};
WithTags matching = featureWithTags("abc", "123");
WithTags notMatching = featureWithTags("abcd", "123");
assertTrue(custom.evaluate(matching));
assertTrue(and(custom).evaluate(matching));
assertTrue(and(custom, custom).evaluate(matching));
assertTrue(or(custom, custom).evaluate(matching));
assertTrue(and(TRUE, custom).evaluate(matching));
assertTrue(or(FALSE, custom).evaluate(matching));
assertFalse(custom.evaluate(notMatching));
assertFalse(and(custom).evaluate(notMatching));
assertFalse(and(custom, custom).evaluate(notMatching));
assertFalse(or(custom, custom).evaluate(notMatching));
assertFalse(and(TRUE, custom).evaluate(notMatching));
assertFalse(or(FALSE, custom).evaluate(notMatching));
}
}

84
planetiler-core/src/test/java/com/onthegomap/planetiler/expression/MultiExpressionTest.java
Wyświetl plik

@ -8,6 +8,7 @@ import static com.onthegomap.planetiler.expression.ExpressionTestUtil.featureWit
import static com.onthegomap.planetiler.expression.MultiExpression.entry;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import com.onthegomap.planetiler.expression.MultiExpression.Index;
@ -131,8 +132,9 @@ class MultiExpressionTest {
private void matchFieldCheck(Index<String> index) {
assertSameElements(List.of("a"), index.getMatches(featureWithTags("key", "value")));
assertSameElements(List.of("a"), index.getMatches(featureWithTags("key", "")));
assertSameElements(List.of("a"), index.getMatches(featureWithTags("key", "value2", "otherkey", "othervalue")));
assertSameElements(List.of(), index.getMatches(featureWithTags("key", "")));
assertSameElements(List.of(), index.getMatches(featureWithTags("key", null)));
assertSameElements(List.of(), index.getMatches(featureWithTags("key2", "value", "key3", "value")));
assertSameElements(List.of(), index.getMatches(featureWithTags("key2", "value")));
assertSameElements(List.of(), index.getMatches(featureWithTags("key2", "no")));
@ -145,7 +147,8 @@ class MultiExpressionTest {
entry("a", not(matchField("key")))
)).index();
assertSameElements(List.of(), index.getMatches(featureWithTags("key", "value")));
assertSameElements(List.of(), index.getMatches(featureWithTags("key", "")));
assertSameElements(List.of("a"), index.getMatches(featureWithTags("key", "")));
assertSameElements(List.of("a"), index.getMatches(featureWithTags("key", null)));
assertSameElements(List.of(), index.getMatches(featureWithTags("key", "value2", "otherkey", "othervalue")));
assertSameElements(List.of("a"), index.getMatches(featureWithTags("key2", "value", "key3", "value")));
assertSameElements(List.of("a"), index.getMatches(featureWithTags("key2", "value")));
@ -207,6 +210,38 @@ class MultiExpressionTest {
assertSameElements(List.of(), index.getMatches(featureWithTags()));
}
@Test
void testStartsWith() {
var index = MultiExpression.of(List.of(
entry("a", matchAny("key", "value%"))
)).index();
assertSameElements(List.of("a"), index.getMatches(featureWithTags("key", "value")));
assertSameElements(List.of("a"), index.getMatches(featureWithTags("key", "value1")));
assertSameElements(List.of(), index.getMatches(featureWithTags("key", "1value")));
assertSameElements(List.of(), index.getMatches(featureWithTags("key", "1value1")));
assertSameElements(List.of(), index.getMatches(featureWithTags("key", "1value1", "otherkey", "othervalue")));
assertSameElements(List.of(), index.getMatches(featureWithTags("key2", "value", "key3", "value")));
assertSameElements(List.of(), index.getMatches(featureWithTags("key", "no")));
assertSameElements(List.of(), index.getMatches(featureWithTags("key2", "value")));
assertSameElements(List.of(), index.getMatches(featureWithTags()));
}
@Test
void testEndsWith() {
var index = MultiExpression.of(List.of(
entry("a", matchAny("key", "%value"))
)).index();
assertSameElements(List.of("a"), index.getMatches(featureWithTags("key", "value")));
assertSameElements(List.of(), index.getMatches(featureWithTags("key", "value1")));
assertSameElements(List.of("a"), index.getMatches(featureWithTags("key", "1value")));
assertSameElements(List.of(), index.getMatches(featureWithTags("key", "1value1")));
assertSameElements(List.of(), index.getMatches(featureWithTags("key", "1value1", "otherkey", "othervalue")));
assertSameElements(List.of(), index.getMatches(featureWithTags("key2", "value", "key3", "value")));
assertSameElements(List.of(), index.getMatches(featureWithTags("key", "no")));
assertSameElements(List.of(), index.getMatches(featureWithTags("key2", "value")));
assertSameElements(List.of(), index.getMatches(featureWithTags()));
}
@Test
void testMultipleMatches() {
var feature = featureWithTags("a", "b", "c", "d");
@ -520,21 +555,26 @@ class MultiExpressionTest {
Expression polygonExpression = and(matchType("polygon"), matchField("field"));
Expression linestringExpression = and(matchType("linestring"), matchField("field"));
Expression pointExpression = and(matchType("point"), matchField("field"));
Expression otherExpression = matchField("field");
Map<String, Object> map = Map.of("field", "value");
SourceFeature point = SimpleFeature.create(newPoint(0, 0), map);
SourceFeature linestring = SimpleFeature.create(newLineString(0, 0, 1, 1), map);
SourceFeature polygon = SimpleFeature.create(rectangle(0, 1), map);
WithTags other = WithTags.from(Map.of("field", "value"));
var index = MultiExpression.of(List.of(
entry("polygon", polygonExpression),
entry("linestring", linestringExpression),
entry("point", pointExpression)
entry("point", pointExpression),
entry("other", otherExpression)
)).index();
assertTrue(pointExpression.evaluate(point, new ArrayList<>()));
assertTrue(linestringExpression.evaluate(linestring, new ArrayList<>()));
assertTrue(polygonExpression.evaluate(polygon, new ArrayList<>()));
assertTrue(otherExpression.evaluate(other, new ArrayList<>()));
assertEquals("point", index.getOrElse(point, null));
assertEquals("linestring", index.getOrElse(linestring, null));
assertEquals("polygon", index.getOrElse(polygon, null));
assertEquals("other", index.getOrElse(other, null));
}
@Test
@ -583,6 +623,44 @@ class MultiExpressionTest {
});
}
@Test
void testCustomExpression() {
Expression dontEvaluate = new Expression() {
@Override
public boolean evaluate(WithTags input, List<String> matchKeys) {
throw new AssertionError("should not evaluate");
}
@Override
public String generateJavaCode() {
return null;
}
};
Expression matchAbc = new Expression() {
@Override
public boolean evaluate(WithTags input, List<String> matchKeys) {
return input.hasTag("abc");
}
@Override
public String generateJavaCode() {
return null;
}
};
var index = MultiExpression.of(List.of(
entry("a", matchAbc),
entry("b", and(matchField("def"), dontEvaluate)),
entry("c", or(matchField("abc"), matchAbc))
)).index();
assertSameElements(List.of(), index.getMatches(featureWithTags()));
assertSameElements(List.of(), index.getMatches(featureWithTags("a", "1")));
assertSameElements(List.of("a", "c"), index.getMatches(featureWithTags("abc", "123")));
var bad = featureWithTags("def", "123");
assertThrows(AssertionError.class, () -> index.getMatches(bad));
}
@Test
void testAndOrMatch() {
var expr = and(
or(

2
planetiler-core/src/test/java/com/onthegomap/planetiler/geo/GeometryTypeTest.java
Wyświetl plik

@ -11,7 +11,7 @@ import org.junit.jupiter.api.Test;
class GeometryTypeTest {
@Test
void testGeometryFactory() throws Exception {
void testGeometryFactory() {
Map<String, Object> tags = Map.of("key1", "value1");
var line =

7
planetiler-core/src/test/java/com/onthegomap/planetiler/util/CacheByZoomTest.java
Wyświetl plik

@ -2,8 +2,6 @@ package com.onthegomap.planetiler.util;
import static org.junit.jupiter.api.Assertions.assertEquals;
import com.onthegomap.planetiler.config.Arguments;
import com.onthegomap.planetiler.config.PlanetilerConfig;
import java.util.ArrayList;
import java.util.List;
import org.junit.jupiter.api.Test;
@ -13,10 +11,7 @@ class CacheByZoomTest {
@Test
void testCacheZoom() {
List<Integer> calls = new ArrayList<>();
CacheByZoom<Integer> cached = CacheByZoom.create(PlanetilerConfig.from(Arguments.of(
"minzoom", "1",
"maxzoom", "10"
)), i -> {
CacheByZoom<Integer> cached = CacheByZoom.create(i -> {
calls.add(i);
return i + 1;
});

88
planetiler-core/src/test/java/com/onthegomap/planetiler/util/FileWatcherTest.java 100644
Wyświetl plik

@ -0,0 +1,88 @@
package com.onthegomap.planetiler.util;
import static com.onthegomap.planetiler.util.FileWatcher.normalize;
import static org.junit.jupiter.api.Assertions.assertEquals;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.attribute.FileTime;
import java.util.Set;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
class FileWatcherTest {
@TempDir
static Path tempDir;
Path a = normalize(tempDir.resolve("a"));
Path b = normalize(tempDir.resolve("b"));
Path c = normalize(tempDir.resolve("c"));
long time = 1;
private void touch(Path... paths) throws IOException {
for (var path : paths) {
Files.write(path, new byte[0]);
Files.setLastModifiedTime(path, FileTime.fromMillis(time++));
}
}
@Test
void testWatch() throws IOException {
touch(a);
var watcher = FileWatcher.newWatcher(a, b);
assertEquals(Set.of(), watcher.poll());
assertEquals(Set.of(), watcher.poll());
touch(a);
assertEquals(Set.of(a), watcher.poll());
touch(b);
assertEquals(Set.of(b), watcher.poll());
touch(a, b);
assertEquals(Set.of(a, b), watcher.poll());
}
@Test
void testRemoveWatch() throws IOException {
touch(a, b);
var watcher = FileWatcher.newWatcher(a, b);
assertEquals(Set.of(), watcher.poll());
watcher.unwatch(a);
touch(a, b);
assertEquals(Set.of(b), watcher.poll());
watcher.unwatch(b);
touch(a, b);
assertEquals(Set.of(), watcher.poll());
}
@Test
void testReturnWatched() throws IOException {
touch(a);
var watcher = FileWatcher.newWatcher(a, b);
assertEquals(Set.of(), watcher.poll());
assertEquals(Set.of(), watcher.poll());
touch(a);
assertEquals(Set.of(a), watcher.poll());
watcher.setWatched(Set.of(b, c));
touch(b);
assertEquals(Set.of(b), watcher.poll());
touch(a);
assertEquals(Set.of(), watcher.poll());
touch(a, b, c);
assertEquals(Set.of(b, c), watcher.poll());
watcher.setWatched(Set.of(a));
touch(b);
assertEquals(Set.of(), watcher.poll());
touch(a);
assertEquals(Set.of(a), watcher.poll());
touch(a, b, c);
assertEquals(Set.of(a), watcher.poll());
watcher.setWatched(Set.of());
touch(a, b, c);
assertEquals(Set.of(a), watcher.poll());
watcher.setWatched(null);
touch(a, b, c);
assertEquals(Set.of(a), watcher.poll());
}
}

3
planetiler-core/src/test/java/com/onthegomap/planetiler/util/ParseTest.java
Wyświetl plik

@ -33,7 +33,8 @@ class ParseTest {
@CsvSource(value = {
"0, 0, 0",
"false, 0, null",
"123, 123, 123"
"123, 123, 123",
"123.123, 123, 123",
}, nullValues = {"null"})
void testLong(String in, long out, Long obj) {
assertEquals(out, Parse.parseLong(in));

25
planetiler-core/src/test/java/com/onthegomap/planetiler/util/TryTest.java 100644
Wyświetl plik

@ -0,0 +1,25 @@
package com.onthegomap.planetiler.util;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;
import org.junit.jupiter.api.Test;
class TryTest {
@Test
void success() {
var result = Try.apply(() -> 1);
assertEquals(Try.success(1), result);
assertEquals(1, result.get());
}
@Test
void failure() {
var exception = new IllegalStateException();
var result = Try.apply(() -> {
throw exception;
});
assertEquals(Try.failure(exception), result);
assertThrows(IllegalStateException.class, result::get);
}
}

625
planetiler-custommap/README.md
Wyświetl plik

@ -1,113 +1,586 @@
# Configurable Planetiler Schema
It is possible to customize planetiler's output from configuration files. This is done using the parameter:
`--schema=schema_file.yml`
You can define how planetiler turns input sources into vector tiles by running planetiler with a YAML configuration
file as the first argument:
The schema file provides information to planetiler about how to construct the tiles and which layers, features, and
attributes will be posted to the file. Schema files are in [YAML](https://yaml.org) format.
```bash
# from a java build
java -jar planetiler.jar schema.yml
# or with docker (put the schema in data/schema.yml to include in the attached volume)
docker run -v "$(pwd)/data":/data ghcr.io/onthegomap/planetiler:latest /data/schema.yml
```
NOTE: The configuration schema is under active development so the format may change between releases. Feedback is
welcome to help shape the final product!
Schema files are in [YAML 1.2](https://yaml.org) format and this page and
accompanying [JSON schema](planetiler.schema.json) describe the required format and available
options. See the [samples](src/main/resources/samples) directory for working examples.
For examples, see [samples](src/main/resources/samples) or [test cases](src/test/resources/validSchema).
:construction: The configuration schema is under active development so the format may change between releases.
Only a subset of the Java API is currently exposed so for more complex schemas you should switch to the Java API (see
the [examples project](../planetiler-examples)). Feedback is welcome to help shape the final product!
## Schema file definition
## Root
The root of the schema has the following attributes:
* `schema_name` - A descriptive name for the schema
* `schema_description` - A longer description of the schema
* `attribution` - An attribution statement, which may include HTML such as links
* `sources` - A list of sources from which features should be extracted, specified as a list of names.
- `schema_name` - A descriptive name for the schema
- `schema_description` - A longer description of the schema
- `attribution` - An attribution string, which may include HTML such as links
- `sources` - An object where key is the source ID and object is the [Source](#source) definition that points to a file
containing geographic features to process
- `tag_mappings` - Specifies that certain tag key should have their values treated as a certain data type.
See [Tag Mappings](#tag-mappings).
* `dataTypes` - A map of tag keys that should be treated as a certain data type, with strings being the default.
See [Tag Mappings](#tag-mappings).
* `layers` - A list of vector tile layers and their definitions. See [Layers](#layers)
- `layers` - A list of vector tile [Layers](#layer) to emit and their definitions
- `examples` - A list of [Test Case](#test-case) input features and the vector tile features they should map to, or a
relative path to a file with those examples in it. Run planetiler with `verify schema_file.yml` to see
if they work as expected.
- `definitions` - An unparsed spot where you can
define [anchor labels](https://en.wikipedia.org/wiki/YAML#Advanced_components) to be used in other parts of the
schema
### Data Sources
For example:
A data source contains geospatial objects with tags that are consumed by planetiler. The configured data sources in the
schema provide complete information on how to access those data sources.
```yaml
schema_name: Power Lines
schema_description: A map of power lines from OpenStreetMap
attribution: <a href="https://www.openstreetmap.org/copyright" target="_blank">&copy; OpenStreetMap contributors</a>
sources: { ... }
tag_mappings: { ... }
layers: [...]
examples: [...]
definitions: # anything ...
```
* `type` - Either `shapefile` or `osm`
* `url` - Location to download the shapefile from. For geofabrik named areas, use `geofabrik:` prefixes, for
example `geofabrik:rhode-island`
## Source
### Layers
A description that tells planetiler how to read geospatial objects with tags from an input file.
A layer contains a thematically-related set of features.
- `type` - Enum representing the file format of the data source, one
of [`osm`](https://wiki.openstreetmap.org/wiki/PBF_Format) or [`shapefile`](https://en.wikipedia.org/wiki/Shapefile)
- `local_path` - Local path to the file to use, inferred from `url` if missing
- `url` - Location to download the file from if not present at `local_path`.
For [geofabrik](https://download.geofabrik.de/) named areas, use `geofabrik:`
prefixes, for example `geofabrik:rhode-island`.
* `name` - Name of this layer
* `features` - A list of features contained in this layer. See [Features](#features)
For example:
### Features
```yaml
sources:
osm:
type: osm
url: geofabrik:switzerland
```
A feature is a defined set of objects that meet specified filter criteria.
## Tag Mappings
* `geometry` - Include objects of a certain geometry type. Options are `polygon`, `line`, or `point`.
* `min_tile_cover_size` - include objects of a certain geometry size, where 1.0 means "is the same size as a tile at
this zoom".
* `include_when` - A tag specification which determines which features to include. If unspecified, all features from the
specified sources are included. See [Tag Filters](#tag-filters)
* `exclude_when` - A tag specification which determines which features to exclude. This rule is applied
after `includeWhen`. If unspecified, no exclusion filter is applied. See [Tag Filters](#tag-filters)
* `min_zoom` - Minimum zoom to show the feature that matches the filter specifications.
* `zoom_override` - List of rules that overrides the `min_zoom` for this feature if certain tags are present. If
multiple rules match, the first matching rule will be applied. See [Feature Zoom Overrides](#feature-zoom-override)
* `attributes` - Specifies the attributes that should be rendered into the tiles for this feature, and how they are
constructed. See [Attributes](#attributes)
Specifies that certain tags should have their values parsed to a certain data type. This can be specified as an object
where key is the tag name and value is the [data type](#data-type), for example:
### Tag Mappings
```yaml
tag_mappings:
population: integer
```
Specifies that certain tag key should have their values treated as being a certain data type.
If you still want to be able to access the original value, then you can remap the parsed value into a new tag
using `type` and `input` fields:
* `<key>: data_type` - A key, along with one of `boolean`, `string`, `direction`, or `long`
* `<key>: mapping` - A mapping which produces a new attribute by retrieving from a different key.
See [Tag Input and Output Mappings](#tag-input-and-output-mappings)
```yaml
tag_mappings:
population_as_int:
input: population
type: integer
```
### Tag Input and Output Mappings
## Layer
* `type`: One of `boolean`, `string`, `direction`, or `long`
* `output`: The name of the typed key that will be presented to the attribute logic
A layer contains a thematically-related set of features from one or more input sources.
### Feature Zoom Override
- `id` - Unique name of this layer
- `features` - A list of features contained in this layer. See [Layer Features](#layer-feature)
Specifies a zoom-based inclusion rules for this feature.
For example:
* `min` - Minimum zoom to render a feature matching this rule
* `tag` - List of tags for which this rule applies. Tags are specified as a list of key/value pairs
```yaml
layers:
- id: power
features:
- { ... }
- { ... }
```
### Attributes
## Layer Feature
* `key` - Name of this attribute in the tile.
* `constant_value` - Value of the attribute in the tile, as a constant
* `tag_value` - Value of the attribute in the tile, as copied from the value of the specified tag key. If neither
constantValue nor tagValue are specified, the default behavior is to set the tag value equal to the input value (
pass-through)
* `include_when` - A filter specification which determines whether to include this attribute. If unspecified, the
attribute will be included unless excluded by `excludeWhen`. See [Tag Filters](#tag-filters)
* `exclude_when` - A filter specification which determines whether to exclude this attribute. This rule is applied
after `includeWhen`. If unspecified, no exclusion filter is applied. See [Tag Filters](#tag-filters)
* `min_zoom` - The minimum zoom at which to render this attribute.
* `min_zoom_by_value` - Minimum zoom to render this attribute depending on the value. Contains a map of `value: zoom`
entries that indicate the minimum zoom for each possible value.
A feature is a defined set of objects that meet a specified filter criteria.
### Tag Filters
- `source` - A string [source](#source) ID, or list of source IDs from which features should be extracted
- `geometry` - A string enum that indicates which geometry types to include, and how to transform them. Can be one
of:
- `point` `line` or `polygon` to pass the original feature through
- `polygon_centroid` to match on polygons, and emit a point at the center
- `polygon_point_on_surface` to match on polygons, and emit an interior point
- `polygon_centroid_if_convex` to match on polygons, and if the polygon is convex emit the centroid, otherwise emit an
interior point
- `min_tile_cover_size` - Include objects of a certain geometry size, where 1.0 means "is
the same size as a tile at this zoom"
- `include_when` - A [Boolean Expression](#boolean-expression) which determines the features to include.
If unspecified, all features from the specified sources are included.
- `exclude_when` - A [Boolean Expression](#boolean-expression) which determines if a feature that matched the include
expression should be skipped. If unspecified, no exclusion filter is applied.
- `min_zoom` - An [Expression](#expression) that returns the minimum zoom to render this feature at.
- `attributes` - An array of [Feature Attribute](#feature-attribute) objects that specify the attributes to be included
on this output feature.
A tag filter matches an object based on its tagging. Multiple key entries may be specified:
For example:
* `<key>:` - Match objects that contain this key.
* ` <value>` - A single value or a list of values. Match objects in the specified key that contains one of these
values. If no values are specified, this will match any value tagged with the specified key.
```yaml
source: osm
geometry: line
min_zoom: 7
include_when:
power:
- line
attributes:
- { ... }
- { ... }
```
Example: match all `natural=water`:
## Feature Attribute
natural: water
Defines an attribute to include on an output vector tile feature and how to compute its value.
Example: match residential, commercial, and industrial land use:
- `key` - ID of this attribute in the tile
- `include_when` - A [Boolean Expression](#boolean-expression) which determines whether to include
this attribute. If unspecified, the attribute will be included unless
excluded by `excludeWhen`.
- `exclude_when` - A [Boolean Expression](#boolean-expression) which determines whether to exclude
this attribute. This rule is applied after `include_when`. If unspecified,
no exclusion filter is applied.
- `min_zoom` - The minimum zoom at which to render this attribute
- `min_zoom_by_value` - Minimum zoom to render this attribute depending on the
value. Contains an object with `<value>: zoom` entries that indicate the
minimum zoom for each output value.
- `type` - The [Data Type](#data-type) to coerce the value to, or `match_key` to set this attribute to the key that
triggered the match in the include expression, or `match_value` to set it to the value for the matching key.
landuse:
- residential
- commercial
- industrial
To define the value, use one of:
- `value` - A constant string/number/boolean value, or an [Expression](#expression) that computes the value for this key
for each input element.
- `coalesce` - A [Coalesce Expression](#coalesce-expression) that sets this attribute to the first non-null match from a
list of expressions.
- `tag_value` - A [Tag Value Expression](#tag-value-expression) that sets this attribute to the value for a tag.
For example:
```yaml
key: voltage
min_zoom: 10
include_when: "${ double(feature.tags.voltage) > 1000 }"
tag_value: voltage
type: integer
```
## Data Type
A string enum that defines how to map from an input. Allowed values:
- `boolean` - Map 0, "no", or "false" to false and everything else to true
- `string` - Returns the string representation of the input value
- `direction` - Maps "-1" to -1, "1" "yes" or "true" to 1, and everything else to 0.
See [Key:oneway](https://wiki.openstreetmap.org/wiki/Key:oneway#Data_consumers).
- `long` - Parses an input as a 64-bit signed number
- `integer` - Parses an input as a 32-bit signed number
- `double` - Parses an input as a floating point number
## Expression
Expressions let you define how to dynamically compute a value (attribute value, min zoom, etc.) at runtime. You can
structure data-heavy expressions in YAML (ie. [match](#match-expression) or [coalesce](#coalesce-expression)) or
simpler expressions that require more flexibility as an [inline script](#inline-script) using `${ expression }` syntax.
### Constant Value Expression
The simplest expression just returns a constant value from a string, number or boolean, for example:
```yaml
value: 1
value: 'string'
value: true
```
### Tag Value Expression
Use `tag_value:` to return the value for each feature's tag at runtime:
```yaml
# return value for "natural" tag
value:
tag_value: natural
```
### Coalesce Expression
Use `coalesce: [expression, expression, ...]` to make the expression evaluate to the first non-null result of a list of
expressions at runtime:
```yaml
value:
coalesce:
- tag_value: highway
- tag_value: aerialway
- tag_value: railway
- "fallback value"
```
### Match Expression
Use `{ value1: condition1, value2: condition2, ... }` to make the expression evaluate to the value associated
with the first matching [boolean expression](#boolean-expression) at runtime:
```yaml
value:
# returns "farmland" if subclass is farmland, farm, or orchard
farmland:
subclass:
- farmland
- farm
- orchard
ice:
subclass:
- glacier
- ice_shelf
# "otherwise" keyword means this is the fallback value
water: otherwise
```
If the values are not simple strings, then you can use an array of objects with `if` / `value` / `else` conditions:
```yaml
value:
- value: 100000
if:
place: city
- value: 5000
if:
place: town
- value: 100
if:
place: [village, neighborhood]
# fallback value
- else: 0
```
In some cases it is more straightforward to express match logic as a `default_value` with `overrides`, for example:
```yaml
min_zoom:
default_value: 13
overrides:
5:
# match motorway or motorway_link
highway: motorway%
6:
highway: trunk%
8:
highway: primary%
```
Default values, and values associated with conditions can themselves be an [Expression](#expression).
### Type
Add the `type` property to any expression to coerce the result to a particular [data type](#data-type):
```yaml
value:
tag_value: oneway
type: direction
```
### Inline Script
Use `${ expression }` syntax to compute a value dynamically at runtime using an
embedded [Common Expression Language (CEL)](https://github.com/google/cel-spec) script.
For example, to normalize highway values like "motorway_link" to "motorway":
```yaml
value: '${ feature.tags.highway.replace("_link", "") }'
```
If a script's value will never change, planetiler evaluates it once ahead of time, so you can also use this to
compute a complex value with no runtime overhead:
```yaml
value: "${ 8 * 24 - 2 }"
```
#### Inline Script Contexts
Scripts are parsed and evaluated inside a "context" that defines the variables available to that script. Contexts are
nested, so each child context can also access the variables from its parent.
> ##### root context
>
> defines no variables
>
>> ##### process feature context
>>
>> Context available when processing an input feature, for example testing whether to include it from `include_when`.
>> Available variables:
>>
>> - `feature.tags` - map with key/value tags from the input feature
>> - `feature.id` - numeric ID of the input feature
>> - `feature.source` - string source ID this feature came from
>> - `feature.source_layer` - optional layer within the source the feature came from
>>
>>> ##### post-match context
>>>
>>> Context available after a feature has matched, for example computing an attribute value. Adds variables:
>>>
>>> - `match_key` - string tag that triggered a match to include the feature in this layer
>>> - `match_value` - the tag value associated with that key
>>>
>>>> ##### configure attribute context
>>>>
>>>> Context available after the value of an attribute has been computed, for example: set min zoom to render an
>>>> attribute. Adds variables:
>>>>
>>>> - `value` the value that was computed for this key
For example:
```yaml
# return the value associated with the matching tag, converted to lower case:
value: '${ match_value.lowerAscii() }'
```
#### Built-In Functions
Inline scripts can use
the [standard CEL built-in functions](https://github.com/google/cel-spec/blob/master/doc/langdef.md#list-of-standard-definitions)
plus the following added by planetiler (defined
in [PlanetilerStdLib](src/main/java/com/onthegomap/planetiler/custommap/expression/stdlib/PlanetilerStdLib.java)).
- `coalesce(any, any, ...)` returns the first non-null argument
- `nullif(arg1, arg2)` returns null if arg1 is the same as arg2, otherwise arg1
- `min(list<number>)` returns the minimum value from a list
- `max(list<number>)` returns the maximum value from a list
- map extensions:
- `<map>.has(key)` returns true if the map contains a key
- `<map>.has(key, value)` returns true if the map contains a key and the value for that key is value
- `<map>.has(key, value1, value2, ...)` returns true if the map contains a key and the value for that key is in the
list provided
- `<map>.get(key)` similar to `map[key]` except it returns null instead of throwing an error if the map is missing
that key
- `<map>.getOrDefault(key, default)` returns the value for key if it is present, otherwise default
- string extensions:
- `<string>.charAt(number)` returns the character at an index from a string
- `<string>.indexOf(string)` returns the first index of a substring or -1 if not found
- `<string>.lastIndexOf(string)` returns the last index of a substring or -1 if not found
- `<list>.join(separator)` returns a string that joins elements together separated by the provided string
- `<string>.lowerAscii()` returns the input string transformed to lower-case
- `<string>.upperAscii()` returns the input string transformed to upper-case
- `<string>.replace(from, to)` returns the input string with all occurrences of from replaced by to
- `<string>.replace(from, to, limit)` returns the input string with the first N occurrences of from replaced by to
- `<string>.replaceRegex(pattern, value)` replaces every occurrence of regular expression with value from the string
it was called on using java's
built-in [replaceAll](<https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/util/regex/Matcher.html#replaceAll(java.lang.String)>)
behavior
- `<string>.split(separator)` returns a list of strings split from the input by a separator
- `<string>.split(separator, limit)` splits the list into up to N parts
- `<string>.substring(n)` returns a copy of the string with first N characters omitted
- `<string>.substring(a, b)` returns a substring from index [a, b)
- `<string>.trim()` trims leading and trailing whitespace
## Boolean Expression
A boolean expression evaluates to true or false for a given input feature. It can be specified as
a [tag-based boolean expression](#tag-based-boolean-expressions),
a [complex boolean expression](#complex-boolean-expressions), or
an [inline script](#inline-boolean-expression-script).
### Tag-Based Boolean Expressions
Boolean expressions can be specified as a map from key to value or list of values. For example:
```yaml
# match features where natural=glacier, waterway=riverbank, OR waterway=canal
include_when:
natural: water
waterway:
- riverbank
- canal
```
Planetiler optimizes runtime performance by pre-processing all of the `include_when` boolean expressions in
each [match expression](#match-expression) and `include_when` block in order to evaluate the minimum set of them at
runtime based on the tags present on the feature.
To match when a tag is present, use the `__any__` keyword:
```yaml
# match when the feature has a building tag
include_when:
building: __any__
```
To match when a feature does _not_ have a tag use `''` as the value:
```yaml
# exclude features without a name tag
exclude_when:
name: ""
```
To match when the value for a key matches a pattern, use the `%` wildcard character:
```yaml
# include features where highway tag ends in "_link"
include_when:
highway: "%_link"
```
When a feature matches a boolean expression in the `include_when` field, the first key that triggered the match is
available to other expressions as `match_key` and its value is available as `match_value`
(See [Post-Match Context](#post-match-context)):
```yaml
include_when:
highway:
- motorway%
- trunk%
- primary%
railway: rail
attributes:
# set "kind" attribute to the value for highway or railway, with trailing "_link" stripped off
- key: kind
value: '${ match_value.replace("_link", ") }'
```
### Complex Boolean Expressions
The [tag-based boolean expressions](#tag-based-boolean-expressions) above match when _any_ of the tag conditions are
true, but to match only when all of them are true, you can nest them under an `__all__` key:
```yaml
# match when highway=pedestrian or highway=service AND area=yes
__all__:
highway:
- pedestrian
- service
area: yes
```
`__all__` can take an array as well. By default, each array item matches if _any_ of its children match, and you can
make that explicit with the `__any__` keyword:
```yaml
# match when highway=pedestrian OR foot=yes, and area=yes
__all__:
- highway: pedestrian
foot: yes
- area: yes
# equivalent to:
__all__:
- __any__:
highway: pedestrian
foot: yes
- area: yes
```
You can also match when the subexpression is false using the `__not__` keyword:
```yaml
# match when place=city AND capital is not 'yes' or '4'
__all__:
place: city
__not__:
capital: [yes, "4"]
```
### Inline Boolean Expression Script
You can also specify boolean logic with an [inline script](#inline-script) that evaluates to `true` or `false` using
the `${ expression }` syntax. For example:
```yaml
# set the `min_zoom` attribute to:
# 2 if area > 20 million, 3 if > 7 million, 4 if > 1 million, or 5 otherwise
min_zoom:
default_value: 5
overrides:
2: "${ double(feature.tags.area) >= 2e8 }"
3: "${ double(feature.tags.area) >= 7e7 }"
4: "${ double(feature.tags.area) >= 1e7 }"
```
:warning: If you use an expression script in `include_when`, it will get evaluated against every input element
and will not set the `match_key` or `match_value` variables. When possible,
use [structured tag expressions](#tag-based-boolean-expressions) which are optimized for runtime matching performance.
You can, however combine a post-filter in an `__all__` block which will only get evaluated if
the [structured tag expressions](#tag-based-boolean-expressions) matches first:
```yaml
# Include a feature when place=city or place=town
# AND it has a population tag
# AND the population value is greater than 10000
include_when:
__all__:
- place: [city, town]
- population: __any__
# only evaluated if previous conditions are true
- "${ double(feature.tags.population) > 10000 }"
```
## Test Case
An example input source feature, and the expected vector tile features that it produces. Run planetiler
with `verify schema.yml` to test your schema against each of the examples. Or you can add the `--watch` argument watch
the input file(s) for changes and validate the test cases on each change:
```yaml
# from a java build
java -jar planetiler.jar verify schema.yml --watch
# or with docker (put the schema in data/schema.yml to include in the attached volume)
docker run -v "$(pwd)/data":/data ghcr.io/onthegomap/planetiler:latest verify /data/schema.yml --watch
```
- `name` - Unique name for this test case.
- `input` - The input feature from a source, with the following attributes:
- `source` - ID of the source this feature comes from.
- `geometry` - Geometry type of the input feature, one of `point` `line` `polygon` or
a [WKT](https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry) encoding of a specific geometry.
- `tags` - Key/value attributes on the source feature.
- `output` - The output vector tile feature(s) this map to, or `[]` for no features. Allowed attributes:
- `layer` - Vector tile layer of the expected output feature.
- `geometry` - Geometry type of the expected output feature.
- `min_zoom` - Min zoom level that the output feature appears in.
- `max_zoom` - Max zoom level that the output feature appears in.
- `tags` - Attributes expected on the output vector tile feature, or `null` if the attribute should not be set. Use
`allow_extra_tags: true` to fail if any other tags appear besides the ones specified here.
- `allow_extra_tags` - If `true`, then fail when extra attributes besides tags appear on the output feature.
If `false` or unset then ignore them.
- `at_zoom` - Some attributes change by zoom level, so get values at this zoom level for comparison.
For example:
```yaml
name: Example power=line
input:
geometry: line
source: osm
tags:
power: line
voltage: "1200"
output:
- layer: power
geometry: line
min_zoom: 7
tags:
power: line
voltage: 1200
```
See [shortbread.spec.yml](src/main/resources/samples/shortbread.spec.yml) for more examples.

451
planetiler-custommap/planetiler.schema.json 100644
Wyświetl plik

@ -0,0 +1,451 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://raw.githubusercontent.com/onthegomap/planetiler/main/planetiler-custommap/planetiler.schema.json",
"title": "Planetiler",
"description": "Planetiler schema definition",
"type": "object",
"properties": {
"schema_name": {
"description": "A descriptive name for the schema",
"type": "string"
},
"schema_description": {
"description": "A longer description of the schema",
"type": "string"
},
"attribution": {
"description": "An attribution statement, which may include HTML such as links",
"type": "string"
},
"definitions": {
"description": "An unparsed spot where you can define anchors and aliases to be used in other parts of the schema",
"type": "object",
"properties": {
"attributes": {
"description": "An unparsed array of attribute fragments to be used below.",
"type": "array",
"items": {
"$ref": "#/$defs/attribute"
}
}
}
},
"sources": {
"description": "An object where key is the source ID and value is the definition of where the features should be extracted from",
"type": "object",
"additionalProperties": {
"type": "object",
"properties": {
"type": {
"description": "File format of the data source",
"enum": [
"osm",
"shapefile"
]
},
"url": {
"description": "Location to download the file from. For geofabrik named areas, use `geofabrik:` prefixes, for example `geofabrik:rhode-island`.",
"type": "string"
},
"local_path": {
"description": "Local path to the file to use, inferred from `url` if missing"
}
},
"anyOf": [
{
"required": [
"url"
]
},
{
"required": [
"local_path"
]
}
]
}
},
"tag_mappings": {
"description": "Specifies that certain tag key should have their values treated as being a certain data type",
"type": "object",
"additionalProperties": {
"anyOf": [
{
"$ref": "#/$defs/datatype"
},
{
"type": "object",
"properties": {
"type": {
"$ref": "#/$defs/datatype"
},
"input": {
"description": "The name of the key that this attribute is parsed from",
"type": "string"
}
}
}
]
}
},
"layers": {
"description": "A list of vector tile layers and their definitions",
"type": "array",
"items": {
"type": "object",
"required": [
"id"
],
"properties": {
"id": {
"description": "Unique layer name",
"type": "string"
},
"features": {
"description": "A list of features contained in this layer",
"type": "array",
"items": {
"$ref": "#/$defs/feature"
}
}
}
}
},
"examples": {
"description": "Example input features and the vector tile features they map to, or a relative path to a file with those examples in it.",
"oneOf": [
{
"$ref": "#/$defs/include"
},
{
"$ref": "planetilerspec.schema.json#/properties/examples"
}
]
}
},
"$defs": {
"datatype": {
"type": "string",
"enum": [
"boolean",
"string",
"direction",
"long",
"integer",
"double"
]
},
"feature": {
"type": "object",
"required": [
"geometry"
],
"properties": {
"geometry": {
"description": "Include objects of a certain geometry type",
"type": "string",
"enum": [
"point",
"line",
"polygon",
"polygon_centroid",
"polygon_centroid_if_convex",
"polygon_point_on_surface"
]
},
"source": {
"description": "A source ID or list of source IDs from which features should be extracted",
"oneOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
]
},
"min_tile_cover_size": {
"description": "include objects of a certain geometry size, where 1.0 means \"is the same size as a tile at this zoom\"",
"type": "number",
"minimum": 0,
"maximum": 1
},
"include_when": {
"description": "A tag specification which determines the features to include. If unspecified, all features from the specified sources are included",
"$ref": "#/$defs/boolean_expression"
},
"exclude_when": {
"description": "A tag specification which determines the features to exclude. This rule is applied after `includeWhen`. If unspecified, no exclusion filter is applied.",
"$ref": "#/$defs/boolean_expression"
},
"min_zoom": {
"description": "An expression that returns the minimum zoom to render this feature at.",
"$ref": "#/$defs/expression"
},
"attributes": {
"description": "Specifies the attributes that should be rendered into the tiles for this feature, and how they are constructed",
"type": "array",
"items": {
"$ref": "#/$defs/attribute"
}
}
}
},
"zoom_level": {
"type": "integer",
"minimum": 0,
"maximum": 15
},
"attribute": {
"type": "object",
"anyOf": [
{
"$ref": "#/$defs/expression_coalesce"
},
{
"$ref": "#/$defs/expression_tag_value"
},
{
"$ref": "#/$defs/expression_value"
},
{
"$ref": "#/$defs/expression_with_type_or_match_key_value"
},
{
"type": "object",
"properties": {
"key": {
"description": "ID of this attribute in the tile",
"type": "string"
},
"include_when": {
"description": "A filter specification which determines whether to include this attribute. If unspecified, the attribute will be included unless excluded by `excludeWhen`",
"$ref": "#/$defs/boolean_expression"
},
"exclude_when": {
"description": "A filter specification which determines whether to exclude this attribute. This rule is applied after `includeWhen`. If unspecified, no exclusion filter is applied.",
"$ref": "#/$defs/boolean_expression"
},
"min_zoom": {
"description": "The minimum zoom at which to render this attribute",
"$ref": "#/$defs/zoom_level"
},
"min_zoom_by_value": {
"description": "Minimum zoom to render this attribute depending on the value. Contains a map of `value: zoom` entries that indicate the minimum zoom for each possible value",
"type": "object",
"additionalProperties": {
"$ref": "#/$defs/zoom_level"
}
}
}
}
]
},
"boolean_expression": {
"anyOf": [
{
"type": "string"
},
{
"$ref": "#/$defs/single_boolean_expression"
},
{
"type": "array",
"items": {
"$ref": "#/$defs/single_boolean_expression"
}
}
]
},
"single_boolean_expression": {
"type": "object",
"properties": {
"__all__": {
"$ref": "#/$defs/boolean_expression"
},
"__any__": {
"$ref": "#/$defs/boolean_expression"
},
"__not__": {
"$ref": "#/$defs/boolean_expression"
}
},
"additionalProperties": {
"anyOf": [
{
"description": "Matches any value for this key",
"const": "__any__"
},
{
"description": "Matches when this key is missing or empty",
"const": ""
},
{
"type": "array",
"description": "A list of possible values for the key",
"items": {
"description": "One of the possible values for the key"
}
},
{
"description": "A single value for the key"
}
]
}
},
"expression": {
"anyOf": [
{
"type": "string"
},
{
"type": "number"
},
{
"$ref": "#/$defs/expression_default_overrides"
},
{
"$ref": "#/$defs/expression_match"
},
{
"$ref": "#/$defs/expression_coalesce"
},
{
"$ref": "#/$defs/expression_tag_value"
},
{
"$ref": "#/$defs/expression_value"
},
{
"$ref": "#/$defs/expression_with_type"
},
{
"$ref": "#/$defs/multiexpression"
}
]
},
"expression_with_type_or_match_key_value": {
"type": "object",
"properties": {
"type": {
"description": "Type of the attribute to map to",
"oneOf": [
{
"type": "string",
"enum": [
"match_key",
"match_value"
]
},
{
"$ref": "#/$defs/datatype"
}
]
}
}
},
"expression_with_type": {
"type": "object",
"properties": {
"type": {
"description": "Type of the attribute to map to",
"$ref": "#/$defs/datatype"
}
}
},
"expression_tag_value": {
"type": "object",
"properties": {
"tag_value": {
"description": "Value of the attribute in the tile, as copied from the value of the specified tag key. If neither constantValue nor tagValue are specified, the default behavior is to set the tag value equal to the input value (pass-through)",
"$ref": "#/$defs/expression"
}
}
},
"expression_value": {
"type": "object",
"properties": {
"value": {
"description": "An expression that computes the value for this key for each input element",
"$ref": "#/$defs/expression"
}
}
},
"expression_coalesce": {
"type": "object",
"properties": {
"coalesce": {
"type": "array",
"items": {
"$ref": "#/$defs/expression"
}
}
}
},
"expression_match": {
"type": "object",
"properties": {
"match": {
"$ref": "#/$defs/multiexpression"
}
}
},
"expression_default_overrides": {
"type": "object",
"properties": {
"default_value": {
"$ref": "#/$defs/expression"
},
"overrides": {
"$ref": "#/$defs/multiexpression"
}
}
},
"multiexpression": {
"oneOf": [
{
"$ref": "#/$defs/multiexpression_object"
},
{
"$ref": "#/$defs/multiexpression_array"
}
]
},
"multiexpression_object": {
"type": "object",
"additionalProperties": {
"oneOf": [
{
"$ref": "#/$defs/boolean_expression"
},
{
"const": "otherwise"
}
]
}
},
"multiexpression_array": {
"type": "array",
"items": {
"type": "object",
"properties": {
"value": {
"$ref": "#/$defs/expression"
},
"if": {
"$ref": "#/$defs/boolean_expression"
},
"else": {
"$ref": "#/$defs/expression"
}
}
}
},
"include": {
"type": "string"
}
}
}

116
planetiler-custommap/planetilerspec.schema.json 100644
Wyświetl plik

@ -0,0 +1,116 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://raw.githubusercontent.com/onthegomap/planetiler/main/planetiler-custommap/planetilerspec.schema.json",
"title": "Planetiler Specification",
"description": "Planetiler schema specification with example input features and the expected output features",
"type": "object",
"properties": {
"examples": {
"description": "A list of example input features, and the output vector tile features they should map to",
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {
"description": "Unique name for this test case",
"type": "string"
},
"input": {
"description": "The input feature from a source",
"type": "object",
"properties": {
"source": {
"description": "ID of the source this feature comes from",
"type": "string"
},
"geometry": {
"description": "Geometry type of the input feature",
"type": "string",
"enum": [
"polygon",
"line",
"point"
]
},
"tags": {
"description": "Key/value attributes on the source feature",
"oneOf": [
{
"type": "object"
},
{
"type": "null"
}
]
}
}
},
"output": {
"description": "The output vector tile feature(s) this map to, or [] for no features",
"oneOf": [
{
"$ref": "#/$defs/output"
},
{
"type": "array",
"items": {
"$ref": "#/$defs/output"
}
}
]
}
}
}
}
},
"$defs": {
"output": {
"type": "object",
"properties": {
"layer": {
"description": "Vector tile layer of the expected output feature",
"type": "string"
},
"geometry": {
"description": "Geometry type of the expected output feature",
"$ref": "#/$defs/geometry"
},
"min_zoom": {
"description": "Min zoom level that the output feature appears in",
"type": "integer"
},
"max_zoom": {
"description": "Max zoom level that the output feature appears in",
"type": "integer"
},
"tags": {
"description": "Attributes expected on the output vector tile feature, or null if the attribute should not be set. Use allow_extra_tags: true to fail if any other tags appear besides the ones specified here",
"oneOf": [
{
"type": "object"
},
{
"type": "null"
}
]
},
"allow_extra_tags": {
"description": "If true, then fail when extra attributes besides tags appear on the output feature. If false or unset then ignore them.",
"type": "boolean"
},
"at_zoom": {
"description": "Some attributes change by zoom level, so get values at this zoom level for comparison",
"type": "integer"
}
}
},
"geometry": {
"type": "string",
"enum": [
"polygon",
"line",
"point"
]
}
}
}

20
planetiler-custommap/pom.xml
Wyświetl plik

@ -19,13 +19,17 @@
<version>${project.parent.version}</version>
</dependency>
<dependency>
<groupId>org.yaml</groupId>
<artifactId>snakeyaml</artifactId>
<groupId>org.snakeyaml</groupId>
<artifactId>snakeyaml-engine</artifactId>
</dependency>
<dependency>
<groupId>org.commonmark</groupId>
<artifactId>commonmark</artifactId>
</dependency>
<dependency>
<groupId>org.projectnessie.cel</groupId>
<artifactId>cel-tools</artifactId>
</dependency>
<dependency>
<groupId>com.onthegomap.planetiler</groupId>
@ -36,6 +40,18 @@
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.projectnessie.cel</groupId>
<artifactId>cel-bom</artifactId>
<version>0.3.10</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<build>
<plugins>
<plugin>

134
planetiler-custommap/src/main/java/com/onthegomap/planetiler/custommap/BooleanExpressionParser.java 100644
Wyświetl plik

@ -0,0 +1,134 @@
package com.onthegomap.planetiler.custommap;
import static com.onthegomap.planetiler.expression.Expression.matchAnyTyped;
import static com.onthegomap.planetiler.expression.Expression.matchField;
import static com.onthegomap.planetiler.expression.Expression.not;
import com.onthegomap.planetiler.custommap.expression.BooleanExpressionScript;
import com.onthegomap.planetiler.custommap.expression.ConfigExpressionScript;
import com.onthegomap.planetiler.custommap.expression.ParseException;
import com.onthegomap.planetiler.custommap.expression.ScriptContext;
import com.onthegomap.planetiler.custommap.expression.ScriptEnvironment;
import com.onthegomap.planetiler.expression.Expression;
import java.util.Collection;
import java.util.List;
import java.util.Map;
import java.util.function.Function;
import java.util.function.Predicate;
import java.util.regex.Pattern;
/**
* Parses user-defined YAML into boolean {@link Expression expressions} that can be evaluated against an input feature.
*
* @param <T> Input type of the expression
*/
public class BooleanExpressionParser<T extends ScriptContext> {
private static final Pattern ESCAPED =
Pattern.compile("^([\\s\\\\]*)\\\\(__any__|__all__)", Pattern.CASE_INSENSITIVE);
private static final Predicate<String> IS_ANY =
Pattern.compile("^\\s*__any__\\s*$", Pattern.CASE_INSENSITIVE).asMatchPredicate();
private static final Predicate<String> IS_ALL =
Pattern.compile("^\\s*__all__\\s*$", Pattern.CASE_INSENSITIVE).asMatchPredicate();
private static final Predicate<String> IS_NOT =
Pattern.compile("^\\s*__not__\\s*$", Pattern.CASE_INSENSITIVE).asMatchPredicate();
private final TagValueProducer tagValueProducer;
private final ScriptEnvironment<T> context;
private BooleanExpressionParser(TagValueProducer tagValueProducer, ScriptEnvironment<T> context) {
this.tagValueProducer = tagValueProducer;
this.context = context;
}
/**
* Returns a boolean expression that determines whether a source feature matches a criteria defined in yaml config.
*
* @param <T> Type of input the expression takes
* @param object a map or list of tag criteria
* @param tagValueProducer a TagValueProducer
* @return a predicate which returns true if this criteria matches
*/
public static <T extends ScriptContext> Expression parse(Object object, TagValueProducer tagValueProducer,
ScriptEnvironment<T> context) {
return new BooleanExpressionParser<>(tagValueProducer, context).parse(object);
}
private static boolean isListOrMap(Object object) {
return object instanceof Map<?, ?> || object instanceof Collection<?>;
}
private static String unescape(String s) {
var matcher = ESCAPED.matcher(s);
if (matcher.matches()) {
return matcher.replaceFirst("$1$2");
}
return s;
}
private static Object unescape(Object o) {
if (o instanceof String s) {
return unescape(s);
}
return o;
}
private Expression parse(Object object) {
return parse(object, Expression::or);
}
private Expression parse(Object object, Function<List<Expression>, Expression> collector) {
if (object == null) {
return Expression.FALSE;
} else if (object instanceof String s && s.trim().equalsIgnoreCase("__any__")) {
return Expression.TRUE;
} else if (ConfigExpressionScript.isScript(object)) {
return BooleanExpressionScript.script(ConfigExpressionScript.extractScript(object), context);
} else if (object instanceof Map<?, ?> map) {
return parseMapMatch(map, collector);
} else if (object instanceof Collection<?> list) {
return collector.apply(list.stream().map(this::parse).toList());
} else {
throw new ParseException("Unsupported object for matcher input: " + object);
}
}
private Expression parseMapMatch(Map<?, ?> map, Function<List<Expression>, Expression> collector) {
return collector.apply(map.entrySet()
.stream()
.map(entry -> tagCriterionToExpression(entry.getKey().toString(), entry.getValue()))
.toList());
}
private Expression tagCriterionToExpression(String key, Object value) {
if (IS_ANY.test(key) && isListOrMap(value)) {
// __any__ ors together its children
return parse(value, Expression::or);
} else if (IS_ALL.test(key) && isListOrMap(value)) {
// __all__ ands together its children
return parse(value, Expression::and);
} else if (IS_NOT.test(key)) {
// __not__ negates its children
return not(parse(value));
} else if (value == null || IS_ANY.test(value.toString()) ||
(value instanceof Collection<?> values &&
values.stream().anyMatch(d -> d != null && IS_ANY.test(d.toString().trim())))) {
//If only a key is provided, with no value, match any object tagged with that key.
return matchField(unescape(key));
} else if (value instanceof Collection<?> values) {
//If a collection is provided, match any of these values.
return matchAnyTyped(
unescape(key),
tagValueProducer.valueGetterForKey(key),
values.stream().map(BooleanExpressionParser::unescape).toList());
} else {
//Otherwise, a key and single value were passed, so match that exact tag
return matchAnyTyped(
unescape(key),
tagValueProducer.valueGetterForKey(key),
unescape(value));
}
}
}

137
planetiler-custommap/src/main/java/com/onthegomap/planetiler/custommap/ConfigExpressionParser.java 100644
Wyświetl plik

@ -0,0 +1,137 @@
package com.onthegomap.planetiler.custommap;
import static com.onthegomap.planetiler.custommap.expression.ConfigExpression.*;
import static com.onthegomap.planetiler.custommap.expression.ConfigExpressionScript.unescape;
import com.onthegomap.planetiler.custommap.expression.ConfigExpression;
import com.onthegomap.planetiler.custommap.expression.ConfigExpressionScript;
import com.onthegomap.planetiler.custommap.expression.ParseException;
import com.onthegomap.planetiler.custommap.expression.ScriptContext;
import com.onthegomap.planetiler.custommap.expression.ScriptEnvironment;
import com.onthegomap.planetiler.expression.DataType;
import com.onthegomap.planetiler.expression.MultiExpression;
import java.util.ArrayList;
import java.util.Collection;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.stream.Collectors;
import java.util.stream.Stream;
/**
* Parses user-defined YAML into an {@link ConfigExpressionParser expression} that can be evaluated against an input
* feature.
*
* @param <I> Input type of the expression
*/
public class ConfigExpressionParser<I extends ScriptContext> {
private final TagValueProducer tagValueProducer;
private final ScriptEnvironment<I> input;
public ConfigExpressionParser(TagValueProducer tagValueProducer, ScriptEnvironment<I> input) {
this.tagValueProducer = tagValueProducer;
this.input = input;
}
/**
* Returns an expression parsed from user-defined YAML that can be evaluated against an input of type {@code <I>} and
* returns output of type {@code <O>}.
*
* @param object a map or list of tag criteria
* @param tagValueProducer a TagValueProducer
* @param <I> Input type of the expression
* @param <O> Return type of the expression
*/
public static <I extends ScriptContext, O> ConfigExpression<I, O> parse(Object object,
TagValueProducer tagValueProducer, ScriptEnvironment<I> context, Class<O> outputClass) {
return new ConfigExpressionParser<>(tagValueProducer, context).parse(object, outputClass).simplify();
}
private <O> ConfigExpression<I, O> parse(Object object, Class<O> output) {
if (object == null) {
return ConfigExpression.constOf(null);
} else if (ConfigExpressionScript.isScript(object)) {
return ConfigExpression.script(signature(output), ConfigExpressionScript.extractScript(object));
} else if (object instanceof Collection<?> collection) {
return parseMatch(collection, true, output);
} else if (object instanceof Map<?, ?> map) {
if (map.get("type") != null) {
var map2 = new HashMap<>(map);
var type = map2.remove("type");
DataType dataType = DataType.from(Objects.toString(type));
if (!dataType.id().equals(type)) {
throw new ParseException("Unrecognized datatype '" + type + "' supported values: " +
Stream.of(DataType.values()).map(DataType::id).collect(
Collectors.joining(", ")));
}
var child = parse(map2, Object.class);
return cast(signature(output), child, dataType);
} else {
var keys = map.keySet();
if (keys.equals(Set.of("coalesce")) && map.get("coalesce")instanceof Collection<?> cases) {
return coalesce(cases.stream().map(item -> parse(item, output)).toList());
} else if (keys.equals(Set.of("match"))) {
return parseMatch(map.get("match"), true, output);
} else if (keys.equals(Set.of("default_value", "overrides"))) {
var match = parseMatch(map.get("overrides"), false, output);
var defaultValue = parse(map.get("default_value"), output);
return match.withDefaultValue(defaultValue);
} else if (keys.equals(Set.of("tag_value"))) {
var tagProducer = parse(map.get("tag_value"), String.class);
return getTag(signature(output), tagProducer);
} else if (keys.equals(Set.of("value"))) {
return parse(map.get("value"), output);
}
try {
return parseMatch(map, true, output);
} catch (ParseException e) {
throw new ParseException("Failed to parse: " + map);
}
}
} else {
object = unescape(object);
return constOf(TypeConversion.convert(object, output));
}
}
private <O> ConfigExpression.Match<I, O> parseMatch(Object match, boolean allowElse, Class<O> output) {
List<MultiExpression.Entry<ConfigExpression<I, O>>> conditions = new ArrayList<>();
ConfigExpression<I, O> fallback = constOf(null);
if (match instanceof Collection<?> items) {
for (var item : items) {
if (item instanceof Map<?, ?> map) {
if (map.keySet().equals(Set.of("if", "value"))) {
conditions.add(MultiExpression.entry(parse(map.get("value"), output),
BooleanExpressionParser.parse(map.get("if"), tagValueProducer, input)));
} else if (allowElse && map.keySet().equals(Set.of("else"))) {
fallback = parse(map.get("else"), output);
break;
} else {
throw new ParseException(
"Invalid match case. Expected if/then" + (allowElse ? " or else" : "") + ", got: " + match);
}
}
}
} else if (match instanceof Map<?, ?> map) {
for (var entry : map.entrySet()) {
String value = Objects.toString(entry.getValue());
if (value.matches("^_*(default_value|otherwise|default)_*$")) {
fallback = parse(entry.getKey(), output);
} else {
conditions.add(MultiExpression.entry(parse(entry.getKey(), output),
BooleanExpressionParser.parse(entry.getValue(), tagValueProducer, input)));
}
}
} else {
throw new ParseException("Invalid match block. Expected a list or map, but got: " + match);
}
return ConfigExpression.match(signature(output), MultiExpression.of(List.copyOf(conditions)), fallback);
}
private <O> Signature<I, O> signature(Class<O> outputClass) {
return new Signature<>(input, outputClass);
}
}

322
planetiler-custommap/src/main/java/com/onthegomap/planetiler/custommap/ConfiguredFeature.java
Wyświetl plik

@ -1,58 +1,45 @@
package com.onthegomap.planetiler.custommap;
import static com.onthegomap.planetiler.custommap.TagCriteria.matcher;
import static com.onthegomap.planetiler.custommap.expression.ConfigExpression.constOf;
import static com.onthegomap.planetiler.expression.Expression.not;
import com.onthegomap.planetiler.FeatureCollector;
import com.onthegomap.planetiler.FeatureCollector.Feature;
import com.onthegomap.planetiler.custommap.configschema.AttributeDefinition;
import com.onthegomap.planetiler.custommap.configschema.FeatureGeometry;
import com.onthegomap.planetiler.custommap.configschema.FeatureItem;
import com.onthegomap.planetiler.custommap.configschema.ZoomOverride;
import com.onthegomap.planetiler.expression.Expression;
import com.onthegomap.planetiler.expression.MultiExpression;
import com.onthegomap.planetiler.expression.MultiExpression.Entry;
import com.onthegomap.planetiler.expression.MultiExpression.Index;
import com.onthegomap.planetiler.geo.GeometryException;
import com.onthegomap.planetiler.geo.GeometryType;
import com.onthegomap.planetiler.reader.SourceFeature;
import com.onthegomap.planetiler.reader.WithTags;
import java.util.Collection;
import java.util.HashSet;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.function.BiConsumer;
import java.util.function.BiFunction;
import java.util.function.Function;
import java.util.function.ToIntFunction;
/**
* A map feature, configured from a YML configuration file.
*
* {@link #matchExpression()} returns a filtering expression to limit input elements to ones this feature cares about,
* and {@link #processFeature(SourceFeature, FeatureCollector)} processes matching elements.
* and {@link #processFeature(Contexts.FeaturePostMatch, FeatureCollector)} processes matching elements.
*/
public class ConfiguredFeature {
private final Set<String> sources;
private static final double LOG4 = Math.log(4);
private final Expression geometryTest;
private final Function<FeatureCollector, Feature> geometryFactory;
private final Expression tagTest;
private final Index<Integer> zoomOverride;
private final Integer featureMinZoom;
private final Integer featureMaxZoom;
private final TagValueProducer tagValueProducer;
private final List<BiConsumer<Contexts.FeaturePostMatch, Feature>> featureProcessors;
private final Set<String> sources;
private static final double LOG4 = Math.log(4);
private static final Index<Integer> NO_ZOOM_OVERRIDE = MultiExpression.<Integer>of(List.of()).index();
private static final Integer DEFAULT_MAX_ZOOM = 14;
private final List<BiConsumer<SourceFeature, Feature>> attributeProcessors;
public ConfiguredFeature(String layer, TagValueProducer tagValueProducer, FeatureItem feature) {
sources = Set.copyOf(feature.source());
public ConfiguredFeature(String layerName, TagValueProducer tagValueProducer, FeatureItem feature) {
sources = new HashSet<>(feature.sources());
GeometryType geometryType = feature.geometry();
FeatureGeometry geometryType = feature.geometry();
//Test to determine whether this type of geometry is included
geometryTest = geometryType.featureTest();
@ -61,130 +48,56 @@ public class ConfiguredFeature {
this.tagValueProducer = tagValueProducer;
//Test to determine whether this feature is included based on tagging
Expression filter;
if (feature.includeWhen() == null) {
tagTest = Expression.TRUE;
filter = Expression.TRUE;
} else {
tagTest = matcher(feature.includeWhen(), tagValueProducer);
filter =
BooleanExpressionParser.parse(feature.includeWhen(), tagValueProducer, Contexts.ProcessFeature.DESCRIPTION);
}
//Index of zoom ranges for a feature based on what tags are present.
zoomOverride = zoomOverride(feature.zoom());
//Test to determine at which zooms to include this feature based on tagging
featureMinZoom = feature.minZoom() == null ? 0 : feature.minZoom();
featureMaxZoom = feature.maxZoom() == null ? DEFAULT_MAX_ZOOM : feature.maxZoom();
if (feature.excludeWhen() != null) {
filter = Expression.and(
filter,
Expression.not(
BooleanExpressionParser.parse(feature.excludeWhen(), tagValueProducer, Contexts.ProcessFeature.DESCRIPTION))
);
}
tagTest = filter;
//Factory to generate the right feature type from FeatureCollector
geometryFactory = geometryType.geometryFactory(layerName);
geometryFactory = geometryType.newGeometryFactory(layer);
//Configure logic for each attribute in the output tile
attributeProcessors = feature.attributes()
.stream()
.map(this::attributeProcessor)
.toList();
}
/**
* Produce an index that matches tags from configuration and returns a minimum zoom level
*
* @param zoom the configured zoom overrides
* @return an index
*/
private Index<Integer> zoomOverride(Collection<ZoomOverride> zoom) {
if (zoom == null || zoom.isEmpty()) {
return NO_ZOOM_OVERRIDE;
List<BiConsumer<Contexts.FeaturePostMatch, Feature>> processors = new ArrayList<>();
for (var attribute : feature.attributes()) {
processors.add(attributeProcessor(attribute));
}
processors.add(makeFeatureProcessor(feature.minZoom(), Integer.class, Feature::setMinZoom));
processors.add(makeFeatureProcessor(feature.maxZoom(), Integer.class, Feature::setMaxZoom));
return MultiExpression.of(
zoom.stream()
.map(this::generateOverrideExpression)
.toList())
.index();
featureProcessors = processors.stream().filter(Objects::nonNull).toList();
}
/**
* Takes the zoom override configuration for a single zoom level and returns an expression that matches tags for that
* level.
*
* @param config zoom override for a single level
* @return matching expression
*/
private Entry<Integer> generateOverrideExpression(ZoomOverride config) {
return MultiExpression.entry(config.min(),
Expression.or(
config.tag()
.entrySet()
.stream()
.map(this::generateKeyExpression)
.toList()));
}
/**
* Returns an expression that matches against single key with one or more values
*
* @param keyExpression a map containing a key and one or more values
* @return a matching expression
*/
private Expression generateKeyExpression(Map.Entry<String, Object> keyExpression) {
// Values are either a single value, or a collection
String key = keyExpression.getKey();
Object rawVal = keyExpression.getValue();
if (rawVal instanceof List<?> tagValues) {
return Expression.matchAnyTyped(key, tagValueProducer.valueGetterForKey(key), tagValues);
}
return Expression.matchAnyTyped(key, tagValueProducer.valueGetterForKey(key), rawVal);
}
/**
* Produces logic that generates attribute values based on configuration and input data. If both a constantValue
* configuration and a tagValue configuration are set, this is likely a mistake, and the constantValue will take
* precedence.
*
* @param attribute - attribute definition configured from YML
* @return a function that generates an attribute value from a {@link SourceFeature} based on an attribute
* configuration.
*/
private Function<WithTags, Object> attributeValueProducer(AttributeDefinition attribute) {
Object constVal = attribute.constantValue();
if (constVal != null) {
return sf -> constVal;
}
String tagVal = attribute.tagValue();
if (tagVal != null) {
return tagValueProducer.valueProducerForKey(tagVal);
}
//Default to producing a tag identical to the input
return tagValueProducer.valueProducerForKey(attribute.key());
}
/**
* Generate logic which determines the minimum zoom level for a feature based on a configured pixel size limit.
*
* @param minTilePercent - minimum percentage of a tile that a feature must cover to be shown
* @param minZoom - global minimum zoom for this feature
* @param minZoomByValue - map of tag values to zoom level
* @return minimum zoom function
*/
private static BiFunction<SourceFeature, Object, Integer> attributeZoomThreshold(Double minTilePercent, int minZoom,
Map<Object, Integer> minZoomByValue) {
if (minZoom == 0 && minZoomByValue.isEmpty()) {
private <T> BiConsumer<Contexts.FeaturePostMatch, Feature> makeFeatureProcessor(Object input, Class<T> clazz,
BiConsumer<Feature, T> consumer) {
if (input == null) {
return null;
}
ToIntFunction<SourceFeature> staticZooms = sf -> Math.max(minZoom, minZoomFromTilePercent(sf, minTilePercent));
if (minZoomByValue.isEmpty()) {
return (sf, key) -> staticZooms.applyAsInt(sf);
var expression = ConfigExpressionParser.parse(
input,
tagValueProducer,
Contexts.FeaturePostMatch.DESCRIPTION,
clazz
);
if (expression.equals(constOf(null))) {
return null;
}
//Attribute value-specific zooms override static zooms
return (sourceFeature, key) -> minZoomByValue.getOrDefault(key, staticZooms.applyAsInt(sourceFeature));
return (context, feature) -> {
var result = expression.apply(context);
if (result != null) {
consumer.accept(feature, result);
}
};
}
private static int minZoomFromTilePercent(SourceFeature sf, Double minTilePercent) {
@ -198,17 +111,85 @@ public class ConfiguredFeature {
}
}
/**
* Produces logic that generates attribute values based on configuration and input data. If both a constantValue
* configuration and a tagValue configuration are set, this is likely a mistake, and the constantValue will take
* precedence.
*
* @param attribute - attribute definition configured from YML
* @return a function that generates an attribute value from a {@link SourceFeature} based on an attribute
* configuration.
*/
private Function<Contexts.FeaturePostMatch, Object> attributeValueProducer(AttributeDefinition attribute) {
Object type = attribute.type();
// some expression features are hoisted to the top-level for attribute values for brevity,
// so just map them to what the equivalent expression syntax would be and parse as an expression.
Map<String, Object> value = new HashMap<>();
if ("match_key".equals(type)) {
value.put("value", "${match_key}");
} else if ("match_value".equals(type)) {
value.put("value", "${match_value}");
} else {
if (type != null) {
value.put("type", type);
}
if (attribute.coalesce() != null) {
value.put("coalesce", attribute.coalesce());
} else if (attribute.value() != null) {
value.put("value", attribute.value());
} else if (attribute.tagValue() != null) {
value.put("tag_value", attribute.tagValue());
} else {
value.put("tag_value", attribute.key());
}
}
return ConfigExpressionParser.parse(value, tagValueProducer, Contexts.FeaturePostMatch.DESCRIPTION, Object.class);
}
/**
* Generate logic which determines the minimum zoom level for a feature based on a configured pixel size limit.
*
* @param minTilePercent - minimum percentage of a tile that a feature must cover to be shown
* @param rawMinZoom - global minimum zoom for this feature, or an expression providing the min zoom dynamically
* @param minZoomByValue - map of tag values to zoom level
* @return minimum zoom function
*/
private Function<Contexts.FeatureAttribute, Integer> attributeZoomThreshold(
Double minTilePercent, Object rawMinZoom, Map<Object, Integer> minZoomByValue) {
var result = ConfigExpressionParser.parse(rawMinZoom, tagValueProducer,
Contexts.FeatureAttribute.DESCRIPTION, Integer.class);
if ((result.equals(constOf(0)) ||
result.equals(constOf(null))) && minZoomByValue.isEmpty()) {
return null;
}
if (minZoomByValue.isEmpty()) {
return context -> Math.max(result.apply(context), minZoomFromTilePercent(context.feature(), minTilePercent));
}
//Attribute value-specific zooms override static zooms
return context -> {
var value = minZoomByValue.get(context.value());
return value != null ? value :
Math.max(result.apply(context), minZoomFromTilePercent(context.feature(), minTilePercent));
};
}
/**
* Generates a function which produces a fully-configured attribute for a feature.
*
* @param attribute - configuration for this attribute
* @return processing logic
*/
private BiConsumer<SourceFeature, Feature> attributeProcessor(AttributeDefinition attribute) {
private BiConsumer<Contexts.FeaturePostMatch, Feature> attributeProcessor(AttributeDefinition attribute) {
var tagKey = attribute.key();
var attributeMinZoom = attribute.minZoom();
attributeMinZoom = attributeMinZoom == null ? 0 : attributeMinZoom;
Object attributeMinZoom = attribute.minZoom();
attributeMinZoom = attributeMinZoom == null ? "0" : attributeMinZoom;
var minZoomByValue = attribute.minZoomByValue();
minZoomByValue = minZoomByValue == null ? Map.of() : minZoomByValue;
@ -217,33 +198,46 @@ public class ConfiguredFeature {
minZoomByValue = tagValueProducer.remapKeysByType(tagKey, minZoomByValue);
var attributeValueProducer = attributeValueProducer(attribute);
var fallback = attribute.fallback();
var attrIncludeWhen = attribute.includeWhen();
var attrExcludeWhen = attribute.excludeWhen();
var attributeTest =
Expression.and(
attrIncludeWhen == null ? Expression.TRUE : matcher(attrIncludeWhen, tagValueProducer),
attrExcludeWhen == null ? Expression.TRUE : not(matcher(attrExcludeWhen, tagValueProducer))
attrIncludeWhen == null ? Expression.TRUE :
BooleanExpressionParser.parse(attrIncludeWhen, tagValueProducer, Contexts.FeaturePostMatch.DESCRIPTION),
attrExcludeWhen == null ? Expression.TRUE :
not(BooleanExpressionParser.parse(attrExcludeWhen, tagValueProducer, Contexts.FeaturePostMatch.DESCRIPTION))
).simplify();
var minTileCoverage = attrIncludeWhen == null ? null : attribute.minTileCoverSize();
BiFunction<SourceFeature, Object, Integer> attributeZoomProducer =
Function<Contexts.FeatureAttribute, Integer> attributeZoomProducer =
attributeZoomThreshold(minTileCoverage, attributeMinZoom, minZoomByValue);
if (attributeZoomProducer != null) {
return (sf, f) -> {
if (attributeTest.evaluate(sf)) {
Object value = attributeValueProducer.apply(sf);
f.setAttrWithMinzoom(tagKey, value, attributeZoomProducer.apply(sf, value));
return (context, f) -> {
Object value = null;
if (attributeTest.evaluate(context)) {
value = attributeValueProducer.apply(context);
if ("".equals(value)) {
value = null;
}
}
if (value == null) {
value = fallback;
}
if (value != null) {
if (attributeZoomProducer != null) {
Integer minzoom = attributeZoomProducer.apply(context.createAttrZoomContext(value));
if (minzoom != null) {
f.setAttrWithMinzoom(tagKey, value, minzoom);
} else {
f.setAttr(tagKey, value);
}
} else {
f.setAttr(tagKey, value);
}
};
}
return (sf, f) -> {
if (attributeTest.evaluate(sf)) {
f.setAttr(tagKey, attributeValueProducer.apply(sf));
}
};
}
@ -258,24 +252,18 @@ public class ConfiguredFeature {
/**
* Generates a tile feature based on a source feature.
*
* @param sourceFeature - input source feature
* @param features - output rendered feature collector
* @param context The evaluation context containing the source feature
* @param features output rendered feature collector
*/
public void processFeature(SourceFeature sourceFeature, FeatureCollector features) {
public void processFeature(Contexts.FeaturePostMatch context, FeatureCollector features) {
var sourceFeature = context.feature();
//Ensure that this feature is from the correct source
if (!sources.contains(sourceFeature.getSource())) {
return;
}
// Ensure that this feature is from the correct source (index should enforce this, so just check when assertions enabled)
assert sources.contains(sourceFeature.getSource());
var minZoom = zoomOverride.getOrElse(sourceFeature, featureMinZoom);
var f = geometryFactory.apply(features)
.setMinZoom(minZoom)
.setMaxZoom(featureMaxZoom);
for (var processor : attributeProcessors) {
processor.accept(sourceFeature, f);
var f = geometryFactory.apply(features);
for (var processor : featureProcessors) {
processor.accept(context, f);
}
}
}

33
planetiler-custommap/src/main/java/com/onthegomap/planetiler/custommap/ConfiguredMapMain.java
Wyświetl plik

@ -1,19 +1,15 @@
package com.onthegomap.planetiler.custommap;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.onthegomap.planetiler.Planetiler;
import com.onthegomap.planetiler.config.Arguments;
import com.onthegomap.planetiler.custommap.configschema.DataSource;
import com.onthegomap.planetiler.custommap.configschema.DataSourceType;
import com.onthegomap.planetiler.custommap.configschema.SchemaConfig;
import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Map;
import org.yaml.snakeyaml.Yaml;
/**
* Main driver to create maps configured by a YAML file.
@ -23,9 +19,6 @@ import org.yaml.snakeyaml.Yaml;
*/
public class ConfiguredMapMain {
private static final Yaml yaml = new Yaml();
private static final ObjectMapper mapper = new ObjectMapper();
/*
* Main entrypoint
*/
@ -37,11 +30,24 @@ public class ConfiguredMapMain {
var dataDir = Path.of("data");
var sourcesDir = dataDir.resolve("sources");
var schemaFile = args.inputFile(
var schemaFile = args.getString(
"schema",
"Location of YML-format schema definition file");
"Location of YML-format schema definition file"
);
var config = loadConfig(schemaFile);
var path = Path.of(schemaFile);
SchemaConfig config;
if (Files.exists(path)) {
config = SchemaConfig.load(path);
} else {
// if the file doesn't exist, check if it's bundled in the jar
schemaFile = schemaFile.startsWith("/samples/") ? schemaFile : "/samples/" + schemaFile;
if (ConfiguredMapMain.class.getResource(schemaFile) != null) {
config = YAML.loadResource(schemaFile, SchemaConfig.class);
} else {
throw new IllegalArgumentException("Schema file not found: " + schemaFile);
}
}
var planetiler = Planetiler.create(args)
.setProfile(new ConfiguredProfile(config));
@ -55,13 +61,6 @@ public class ConfiguredMapMain {
.run();
}
static SchemaConfig loadConfig(Path schemaFile) throws IOException {
try (var schemaStream = Files.newInputStream(schemaFile)) {
Map<String, Object> parsed = yaml.load(schemaStream);
return mapper.convertValue(parsed, SchemaConfig.class);
}
}
private static void configureSource(Planetiler planetiler, Path sourcesDir, String sourceName, DataSource source)
throws URISyntaxException {

39
planetiler-custommap/src/main/java/com/onthegomap/planetiler/custommap/ConfiguredProfile.java
Wyświetl plik

@ -1,6 +1,7 @@
package com.onthegomap.planetiler.custommap;
import static com.onthegomap.planetiler.expression.MultiExpression.Entry;
import static java.util.Map.entry;
import com.onthegomap.planetiler.FeatureCollector;
import com.onthegomap.planetiler.Profile;
@ -11,7 +12,10 @@ import com.onthegomap.planetiler.expression.MultiExpression.Index;
import com.onthegomap.planetiler.reader.SourceFeature;
import java.util.ArrayList;
import java.util.Collection;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;
/**
* A profile configured from a yml file.
@ -20,7 +24,8 @@ public class ConfiguredProfile implements Profile {
private final SchemaConfig schemaConfig;
private final Index<ConfiguredFeature> featureLayerMatcher;
private final Map<String, Index<ConfiguredFeature>> featureLayerMatcher;
private final TagValueProducer tagValueProducer;
public ConfiguredProfile(SchemaConfig schemaConfig) {
this.schemaConfig = schemaConfig;
@ -30,20 +35,25 @@ public class ConfiguredProfile implements Profile {
throw new IllegalArgumentException("No layers defined");
}
TagValueProducer tagValueProducer = new TagValueProducer(schemaConfig.inputMappings());
tagValueProducer = new TagValueProducer(schemaConfig.inputMappings());
List<MultiExpression.Entry<ConfiguredFeature>> configuredFeatureEntries = new ArrayList<>();
Map<String, List<MultiExpression.Entry<ConfiguredFeature>>> configuredFeatureEntries = new HashMap<>();
for (var layer : layers) {
String layerName = layer.name();
String layerId = layer.id();
for (var feature : layer.features()) {
var configuredFeature = new ConfiguredFeature(layerName, tagValueProducer, feature);
configuredFeatureEntries.add(
new Entry<>(configuredFeature, configuredFeature.matchExpression()));
var configuredFeature = new ConfiguredFeature(layerId, tagValueProducer, feature);
var entry = new Entry<>(configuredFeature, configuredFeature.matchExpression());
for (var source : feature.source()) {
var list = configuredFeatureEntries.computeIfAbsent(source, s -> new ArrayList<>());
list.add(entry);
}
}
}
featureLayerMatcher = MultiExpression.of(configuredFeatureEntries).index();
featureLayerMatcher = configuredFeatureEntries.entrySet().stream()
.map(entry -> entry(entry.getKey(), MultiExpression.of(entry.getValue()).index()))
.collect(Collectors.toUnmodifiableMap(Map.Entry::getKey, Map.Entry::getValue));
}
@Override
@ -58,8 +68,17 @@ public class ConfiguredProfile implements Profile {
@Override
public void processFeature(SourceFeature sourceFeature, FeatureCollector featureCollector) {
featureLayerMatcher.getMatches(sourceFeature)
.forEach(configuredFeature -> configuredFeature.processFeature(sourceFeature, featureCollector));
var context = new Contexts.ProcessFeature(sourceFeature, tagValueProducer);
var index = featureLayerMatcher.get(sourceFeature.getSource());
if (index != null) {
var matches = index.getMatchesWithTriggers(context);
for (var configuredFeature : matches) {
configuredFeature.match().processFeature(
context.createPostMatchContext(configuredFeature.keys()),
featureCollector
);
}
}
}
@Override

195
planetiler-custommap/src/main/java/com/onthegomap/planetiler/custommap/Contexts.java 100644
Wyświetl plik

@ -0,0 +1,195 @@
package com.onthegomap.planetiler.custommap;
import com.onthegomap.planetiler.custommap.expression.ScriptContext;
import com.onthegomap.planetiler.custommap.expression.ScriptEnvironment;
import com.onthegomap.planetiler.reader.SourceFeature;
import com.onthegomap.planetiler.reader.WithGeometryType;
import com.onthegomap.planetiler.reader.WithTags;
import java.util.List;
import java.util.Map;
import org.projectnessie.cel.checker.Decls;
import org.projectnessie.cel.common.types.NullT;
/**
* Wrapper objects that provide all available inputs to different parts of planetiler schema configs at runtime.
* <p>
* Contexts provide inputs to java code, and also global variable definitions to CEL expressions. Contexts are nested so
* that all global variables from a parent context are also available to its child context.
*/
public class Contexts {
private static Object wrapNullable(Object nullable) {
return nullable == null ? NullT.NullValue : nullable;
}
public static Root root() {
return new Root();
}
/**
* Root context available everywhere in a planetiler schema config.
*/
public record Root() implements ScriptContext {
// TODO add argument parsing
public static final ScriptEnvironment<Root> DESCRIPTION =
ScriptEnvironment.root().forInput(Root.class);
@Override
public Object apply(String input) {
return null;
}
}
/**
* Makes nested contexts adhere to {@link WithTags} and {@link WithGeometryType} by recursively fetching source
* feature from the root context.
*/
private interface FeatureContext extends ScriptContext, WithTags, WithGeometryType {
default FeatureContext parent() {
return null;
}
default SourceFeature feature() {
return parent().feature();
}
@Override
default Map<String, Object> tags() {
return feature().tags();
}
@Override
default TagValueProducer tagValueProducer() {
var parent = parent();
return parent == null ? TagValueProducer.EMPTY : parent.tagValueProducer();
}
@Override
default boolean isPoint() {
return feature().isPoint();
}
@Override
default boolean canBeLine() {
return feature().canBeLine();
}
@Override
default boolean canBePolygon() {
return feature().canBePolygon();
}
}
/**
* Context available when processing an input feature.
*
* @param feature The input feature being processed
* @param tagValueProducer Common parsing for input feature tags
*/
public record ProcessFeature(@Override SourceFeature feature, @Override TagValueProducer tagValueProducer)
implements FeatureContext {
private static final String FEATURE_TAGS = "feature.tags";
private static final String FEATURE_ID = "feature.id";
private static final String FEATURE_SOURCE = "feature.source";
private static final String FEATURE_SOURCE_LAYER = "feature.source_layer";
public static final ScriptEnvironment<ProcessFeature> DESCRIPTION = ScriptEnvironment.root()
.forInput(ProcessFeature.class)
.withDeclarations(
Decls.newVar(FEATURE_TAGS, Decls.newMapType(Decls.String, Decls.Any)),
Decls.newVar(FEATURE_ID, Decls.Int),
Decls.newVar(FEATURE_SOURCE, Decls.String),
Decls.newVar(FEATURE_SOURCE_LAYER, Decls.String)
);
@Override
public Object apply(String key) {
if (key != null) {
return switch (key) {
case FEATURE_TAGS -> tagValueProducer.mapTags(feature);
case FEATURE_ID -> feature.id();
case FEATURE_SOURCE -> feature.getSource();
case FEATURE_SOURCE_LAYER -> wrapNullable(feature.getSourceLayer());
default -> null;
};
} else {
return null;
}
}
public FeaturePostMatch createPostMatchContext(List<String> matchKeys) {
return new FeaturePostMatch(this, matchKeys);
}
}
/**
* Context available after a feature has been matched.
*
* Adds {@code match_key} and {@code match_value} variables that capture which tag key/value caused the feature to be
* included.
*
* @param parent The parent context
* @param matchKeys Keys that triggered the match
*/
public record FeaturePostMatch(@Override ProcessFeature parent, List<String> matchKeys) implements FeatureContext {
private static final String MATCH_KEY = "match_key";
private static final String MATCH_VALUE = "match_value";
public static final ScriptEnvironment<FeaturePostMatch> DESCRIPTION = ProcessFeature.DESCRIPTION
.forInput(FeaturePostMatch.class)
.withDeclarations(
Decls.newVar(MATCH_KEY, Decls.String),
Decls.newVar(MATCH_VALUE, Decls.Any)
);
@Override
public Object apply(String key) {
if (key != null) {
return switch (key) {
case MATCH_KEY -> wrapNullable(matchKey());
case MATCH_VALUE -> wrapNullable(matchValue());
default -> parent.apply(key);
};
} else {
return null;
}
}
public String matchKey() {
return matchKeys().isEmpty() ? null : matchKeys().get(0);
}
public Object matchValue() {
String matchKey = matchKey();
return matchKey == null ? null : parent.tagValueProducer.valueForKey(parent().feature(), matchKey);
}
public FeatureAttribute createAttrZoomContext(Object value) {
return new FeatureAttribute(this, value);
}
}
/**
* Context available when configuring an attribute on an output feature after its value has been assigned (for example
* setting min/max zoom).
*
* @param parent The parent context
* @param value Value of the attribute
*/
public record FeatureAttribute(@Override FeaturePostMatch parent, Object value) implements FeatureContext {
private static final String VALUE = "value";
public static final ScriptEnvironment<FeatureAttribute> DESCRIPTION = FeaturePostMatch.DESCRIPTION
.forInput(FeatureAttribute.class)
.withDeclarations(Decls.newVar(VALUE, Decls.Any));
@Override
public Object apply(String key) {
return VALUE.equals(key) ? wrapNullable(value) : parent.apply(key);
}
}
}

55
planetiler-custommap/src/main/java/com/onthegomap/planetiler/custommap/TagCriteria.java
Wyświetl plik

@ -1,55 +0,0 @@
package com.onthegomap.planetiler.custommap;
import static com.onthegomap.planetiler.expression.Expression.matchAnyTyped;
import static com.onthegomap.planetiler.expression.Expression.matchField;
import com.onthegomap.planetiler.expression.Expression;
import java.util.Collection;
import java.util.Map;
/**
* Utility that maps expressions in YAML format to {@link Expression Expressions}.
*/
public class TagCriteria {
private TagCriteria() {
//Hide implicit public constructor
}
/**
* Returns a function that determines whether a source feature matches any of the entries in this specification
*
* @param map a map of tag criteria
* @param tagValueProducer a TagValueProducer
* @return a predicate which returns true if this criteria matches
*/
public static Expression matcher(Map<String, Object> map, TagValueProducer tagValueProducer) {
return map.entrySet()
.stream()
.map(entry -> tagCriterionToExpression(tagValueProducer, entry.getKey(), entry.getValue()))
.reduce(Expression::or)
.orElse(Expression.TRUE);
}
private static Expression tagCriterionToExpression(TagValueProducer tagValueProducer, String key, Object value) {
//If only a key is provided, with no value, match any object tagged with that key.
if (value == null) {
return matchField(key);
//If a collection is provided, match any of these values.
} else if (value instanceof Collection<?> values) {
return matchAnyTyped(
key,
tagValueProducer.valueGetterForKey(key),
values.stream().toList());
//Otherwise, a key and single value were passed, so match that exact tag
} else {
return matchAnyTyped(
key,
tagValueProducer.valueGetterForKey(key),
value);
}
}
}

71
planetiler-custommap/src/main/java/com/onthegomap/planetiler/custommap/TagValueProducer.java
Wyświetl plik

@ -1,7 +1,9 @@
package com.onthegomap.planetiler.custommap;
import static com.onthegomap.planetiler.expression.DataType.GET_TAG;
import com.onthegomap.planetiler.expression.DataType;
import com.onthegomap.planetiler.reader.WithTags;
import com.onthegomap.planetiler.util.Parse;
import java.util.HashMap;
import java.util.LinkedHashMap;
import java.util.Map;
@ -13,34 +15,12 @@ import java.util.function.UnaryOperator;
* Utility that parses attribute values from source features, based on YAML config.
*/
public class TagValueProducer {
private static final String STRING_DATATYPE = "string";
private static final String BOOLEAN_DATATYPE = "boolean";
private static final String DIRECTION_DATATYPE = "direction";
private static final String LONG_DATATYPE = "long";
private static final BiFunction<WithTags, String, Object> DEFAULT_GETTER = WithTags::getTag;
public static final TagValueProducer EMPTY = new TagValueProducer(null);
private final Map<String, BiFunction<WithTags, String, Object>> valueRetriever = new HashMap<>();
private final Map<String, String> keyType = new HashMap<>();
private static final Map<String, BiFunction<WithTags, String, Object>> inputGetter =
Map.of(
STRING_DATATYPE, WithTags::getString,
BOOLEAN_DATATYPE, WithTags::getBoolean,
DIRECTION_DATATYPE, WithTags::getDirection,
LONG_DATATYPE, WithTags::getLong
);
private static final Map<String, UnaryOperator<Object>> inputParse =
Map.of(
STRING_DATATYPE, s -> s,
BOOLEAN_DATATYPE, Parse::bool,
DIRECTION_DATATYPE, Parse::direction,
LONG_DATATYPE, Parse::parseLong
);
public TagValueProducer(Map<String, Object> map) {
if (map == null) {
return;
@ -48,17 +28,20 @@ public class TagValueProducer {
map.forEach((key, value) -> {
if (value instanceof String stringType) {
valueRetriever.put(key, inputGetter.get(stringType));
valueRetriever.put(key, DataType.from(stringType));
keyType.put(key, stringType);
} else if (value instanceof Map<?, ?> renameMap) {
String output = renameMap.containsKey("output") ? renameMap.get("output").toString() : key;
BiFunction<WithTags, String, Object> getter =
renameMap.containsKey("type") ? inputGetter.get(renameMap.get("type").toString()) : DEFAULT_GETTER;
String inputKey = renameMap.containsKey("input") ? renameMap.get("input").toString() : key;
var getter =
renameMap.containsKey("type") ? DataType.from(renameMap.get("type").toString()) : DataType.GET_TAG;
//When requesting the output value, actually retrieve the input key with the desired getter
valueRetriever.put(output,
(withTags, requestedKey) -> getter.apply(withTags, key));
if (inputKey.equals(key)) {
valueRetriever.put(key, getter);
} else {
valueRetriever.put(key, (withTags, requestedKey) -> getter.convertFrom(valueForKey(withTags, inputKey)));
}
if (renameMap.containsKey("type")) {
keyType.put(output, renameMap.get("type").toString());
keyType.put(key, renameMap.get("type").toString());
}
}
});
@ -68,15 +51,22 @@ public class TagValueProducer {
* Returns a function that extracts the value for {@code key} from a {@link WithTags} instance.
*/
public BiFunction<WithTags, String, Object> valueGetterForKey(String key) {
return valueRetriever.getOrDefault(key, DEFAULT_GETTER);
return valueRetriever.getOrDefault(key, GET_TAG);
}
/**
* Returns a function that extracts the value for {@code key} from a {@link WithTags} instance.
*/
public Function<WithTags, Object> valueProducerForKey(String key) {
public Function<Contexts.FeaturePostMatch, Object> valueProducerForKey(String key) {
var getter = valueGetterForKey(key);
return withTags -> getter.apply(withTags, key);
return context -> getter.apply(context.parent().feature(), key);
}
/**
* Returns the mapped value for a key where the key is not known ahead of time.
*/
public Object valueForKey(WithTags feature, String key) {
return valueGetterForKey(key).apply(feature, key);
}
/**
@ -88,7 +78,7 @@ public class TagValueProducer {
String dataType = keyType.get(key);
UnaryOperator<Object> parser;
if (dataType == null || (parser = inputParse.get(dataType)) == null) {
if (dataType == null || (parser = DataType.from(dataType).parser()) == null) {
newMap.putAll(keyedMap);
} else {
keyedMap.forEach((mapKey, value) -> newMap.put(parser.apply(mapKey), value));
@ -96,4 +86,15 @@ public class TagValueProducer {
return newMap;
}
/** Returns a new map where every tag has been transformed (or inferred) by the registered conversions. */
public Map<String, Object> mapTags(WithTags feature) {
if (valueRetriever.isEmpty()) {
return feature.tags();
} else {
Map<String, Object> result = new HashMap<>(feature.tags());
valueRetriever.forEach((key, retriever) -> result.put(key, retriever.apply(feature, key)));
return result;
}
}
}

91
planetiler-custommap/src/main/java/com/onthegomap/planetiler/custommap/TypeConversion.java 100644
Wyświetl plik

@ -0,0 +1,91 @@
package com.onthegomap.planetiler.custommap;
import com.onthegomap.planetiler.util.Parse;
import java.util.List;
import java.util.function.Function;
/**
* Utility for convert between types in a forgiving way (parse strings to get a number, call toString to get a string,
* etc.).
*/
public class TypeConversion {
// convert() uses the first conversion from this list where:
// - the input is a subclass of the first argument
// - and expected output is equal to, or a superclass of the second argument
// so put more specific conversions first, and general fallbacks last
// NOTE: only does single-hop conversions, does NOT attempt to chain together multiple conversions
private static final List<Converter<?, ?>> CONVERTERS = List.of(
// implicit initial conversion returns the input if it is null, or already a subclass of the output type
converter(Number.class, Double.class, Number::doubleValue),
converter(Number.class, Integer.class, Number::intValue),
converter(Number.class, Long.class, Number::longValue),
converter(String.class, Double.class, Parse::parseDoubleOrNull),
converter(String.class, Integer.class, Parse::parseIntOrNull),
converter(String.class, Long.class, Parse::parseLongOrNull),
converter(Integer.class, Boolean.class, n -> n != 0),
converter(Long.class, Boolean.class, n -> n != 0),
converter(Number.class, Boolean.class, n -> Math.abs(n.doubleValue()) > 2 * Double.MIN_VALUE),
converter(String.class, Boolean.class, s -> Parse.bool(s.toLowerCase())),
converter(Object.class, Boolean.class, Parse::bool),
converter(Double.class, String.class, TypeConversion::doubleToString),
converter(Object.class, String.class, Object::toString)
);
private TypeConversion() {}
private static <I, O> Converter<I, O> converter(Class<I> in, Class<O> out, Function<I, O> fn) {
return new Converter<>(in, out, fn);
}
/**
* Attempts to coerce {@code in} to an instance {@code out} using the first registered conversion functions that
* applies.
*
* @throws IllegalArgumentException if there is no available conversion
*/
@SuppressWarnings("unchecked")
public static <O> O convert(Object in, Class<O> out) {
if (in == null || out.isInstance(in)) {
return (O) in;
}
for (var converter : CONVERTERS) {
if (converter.canConvertBetween(in.getClass(), out)) {
return (O) converter.apply(in);
}
}
throw new IllegalArgumentException(
"No conversion from " + in.getClass().getSimpleName() + " to " + out.getSimpleName());
}
private static String doubleToString(Double d) {
return d % 1 == 0 ? Long.toString(d.longValue()) : d.toString();
}
private record Converter<I, O> (Class<I> in, Class<O> out, Function<I, O> fn) implements Function<Object, O> {
@Override
public O apply(Object in) {
@SuppressWarnings("unchecked") I converted = (I) in;
try {
return fn.apply(converted);
} catch (NumberFormatException e) {
return null;
}
}
boolean canConvertTo(Class<?> clazz) {
return clazz.isAssignableFrom(out);
}
boolean canConvertFrom(Class<?> clazz) {
return in.isAssignableFrom(clazz);
}
boolean canConvertBetween(Class<?> from, Class<?> to) {
return canConvertFrom(from) && canConvertTo(to);
}
}
}

61
planetiler-custommap/src/main/java/com/onthegomap/planetiler/custommap/YAML.java 100644
Wyświetl plik

@ -0,0 +1,61 @@
package com.onthegomap.planetiler.custommap;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.UncheckedIOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import org.snakeyaml.engine.v2.api.Load;
import org.snakeyaml.engine.v2.api.LoadSettings;
/**
* Utility for parsing YAML files into java objects using snakeyaml to handle aliases and anchors and jackson to map
* into java model objects.
*/
public class YAML {
private YAML() {}
private static final Load snakeYaml = new Load(LoadSettings.builder().build());
public static final ObjectMapper jackson = new ObjectMapper();
public static <T> T load(Path file, Class<T> clazz) {
try (var schemaStream = Files.newInputStream(file)) {
return load(schemaStream, clazz);
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
public static <T> T load(InputStream stream, Class<T> clazz) {
try (stream) {
Object parsed = snakeYaml.loadFromInputStream(stream);
return convertValue(parsed, clazz);
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
public static <T> T load(String config, Class<T> clazz) {
try (var stream = new ByteArrayInputStream(config.getBytes(StandardCharsets.UTF_8))) {
return load(stream, clazz);
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
public static <T> T loadResource(String resourceName, Class<T> clazz) {
try (var stream = YAML.class.getResourceAsStream(resourceName)) {
return load(stream, clazz);
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
public static <T> T convertValue(Object parsed, Class<T> clazz) {
return jackson.convertValue(parsed, clazz);
}
}

16
planetiler-custommap/src/main/java/com/onthegomap/planetiler/custommap/configschema/AttributeDefinition.java
Wyświetl plik

@ -5,11 +5,15 @@ import java.util.Map;
public record AttributeDefinition(
String key,
@JsonProperty("constant_value") Object constantValue,
@JsonProperty("tag_value") String tagValue,
@JsonProperty("include_when") Map<String, Object> includeWhen,
@JsonProperty("exclude_when") Map<String, Object> excludeWhen,
@JsonProperty("min_zoom") Integer minZoom,
@JsonProperty("include_when") Object includeWhen,
@JsonProperty("exclude_when") Object excludeWhen,
@JsonProperty("min_zoom") Object minZoom,
@JsonProperty("min_zoom_by_value") Map<Object, Integer> minZoomByValue,
@JsonProperty("min_tile_cover_size") Double minTileCoverSize
@JsonProperty("min_tile_cover_size") Double minTileCoverSize,
@JsonProperty("else") Object fallback,
// pass-through to value expression
@JsonProperty("value") Object value,
@JsonProperty("tag_value") String tagValue,
Object type,
Object coalesce
) {}

52
planetiler-custommap/src/main/java/com/onthegomap/planetiler/custommap/configschema/FeatureGeometry.java 100644
Wyświetl plik

@ -0,0 +1,52 @@
package com.onthegomap.planetiler.custommap.configschema;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.onthegomap.planetiler.FeatureCollector;
import com.onthegomap.planetiler.expression.Expression;
import com.onthegomap.planetiler.geo.GeometryType;
import java.util.function.BiFunction;
import java.util.function.Function;
public enum FeatureGeometry {
@JsonProperty("point")
POINT(GeometryType.POINT, FeatureCollector::point),
@JsonProperty("line")
LINE(GeometryType.LINE, FeatureCollector::line),
@JsonProperty("polygon")
POLYGON(GeometryType.POLYGON, FeatureCollector::polygon),
@JsonProperty("polygon_centroid")
POLYGON_CENTROID(GeometryType.POLYGON, FeatureCollector::centroid),
@JsonProperty("polygon_centroid_if_convex")
POLYGON_CENTROID_IF_CONVEX(GeometryType.POLYGON, FeatureCollector::centroidIfConvex),
@JsonProperty("polygon_point_on_surface")
POLYGON_POINT_ON_SURFACE(GeometryType.POLYGON, FeatureCollector::pointOnSurface);
public final GeometryType geometryType;
public final BiFunction<FeatureCollector, String, FeatureCollector.Feature> geometryFactory;
FeatureGeometry(GeometryType type, BiFunction<FeatureCollector, String, FeatureCollector.Feature> geometryFactory) {
this.geometryType = type;
this.geometryFactory = geometryFactory;
}
/**
* Generates a test for whether a source feature is of the correct geometry to be included in the tile.
*
* @return geometry test method
*/
public Expression featureTest() {
return geometryType.featureTest();
}
/**
* Generates a factory method which creates a {@link FeatureCollector.Feature} from a {@link FeatureCollector} of the
* appropriate geometry type.
*
* @param layerName - name of the layer
* @return geometry factory method
*/
public Function<FeatureCollector, FeatureCollector.Feature> newGeometryFactory(String layerName) {
return features -> geometryFactory.apply(features, layerName);
}
}

25
planetiler-custommap/src/main/java/com/onthegomap/planetiler/custommap/configschema/FeatureItem.java
Wyświetl plik

@ -1,17 +1,22 @@
package com.onthegomap.planetiler.custommap.configschema;
import com.fasterxml.jackson.annotation.JsonFormat;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.onthegomap.planetiler.geo.GeometryType;
import java.util.Collection;
import java.util.Map;
import java.util.List;
public record FeatureItem(
Collection<String> sources,
@JsonProperty("min_zoom") Integer minZoom,
@JsonProperty("max_zoom") Integer maxZoom,
GeometryType geometry,
@JsonProperty("zoom_override") Collection<ZoomOverride> zoom,
@JsonProperty("include_when") Map<String, Object> includeWhen,
@JsonProperty("exclude_when") Map<String, Object> excludeWhen,
@JsonFormat(with = JsonFormat.Feature.ACCEPT_SINGLE_VALUE_AS_ARRAY) List<String> source,
@JsonProperty("min_zoom") Object minZoom,
@JsonProperty("max_zoom") Object maxZoom,
@JsonProperty(required = true) FeatureGeometry geometry,
@JsonProperty("include_when") Object includeWhen,
@JsonProperty("exclude_when") Object excludeWhen,
Collection<AttributeDefinition> attributes
) {}
) {
@Override
public Collection<AttributeDefinition> attributes() {
return attributes == null ? List.of() : attributes;
}
}

2
planetiler-custommap/src/main/java/com/onthegomap/planetiler/custommap/configschema/FeatureLayer.java
Wyświetl plik

@ -3,6 +3,6 @@ package com.onthegomap.planetiler.custommap.configschema;
import java.util.Collection;
public record FeatureLayer(
String name,
String id,
Collection<FeatureItem> features
) {}

14
planetiler-custommap/src/main/java/com/onthegomap/planetiler/custommap/configschema/SchemaConfig.java
Wyświetl plik

@ -1,6 +1,8 @@
package com.onthegomap.planetiler.custommap.configschema;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.onthegomap.planetiler.custommap.YAML;
import java.nio.file.Path;
import java.util.Collection;
import java.util.Map;
@ -12,8 +14,10 @@ public record SchemaConfig(
@JsonProperty("schema_description") String schemaDescription,
String attribution,
Map<String, DataSource> sources,
Object definitions,
@JsonProperty("tag_mappings") Map<String, Object> inputMappings,
Collection<FeatureLayer> layers
Collection<FeatureLayer> layers,
Object examples
) {
private static final String DEFAULT_ATTRIBUTION = """
@ -24,4 +28,12 @@ public record SchemaConfig(
public String attribution() {
return attribution == null ? DEFAULT_ATTRIBUTION : attribution;
}
public static SchemaConfig load(Path path) {
return YAML.load(path, SchemaConfig.class);
}
public static SchemaConfig load(String string) {
return YAML.load(string, SchemaConfig.class);
}
}

4
planetiler-custommap/src/main/java/com/onthegomap/planetiler/custommap/configschema/ZoomOverride.java
Wyświetl plik

@ -1,7 +1,5 @@
package com.onthegomap.planetiler.custommap.configschema;
import java.util.Map;
/**
* Configuration item that instructs the renderer to override the default zoom range for features which contain specific
* tag combinations.
@ -9,4 +7,4 @@ import java.util.Map;
public record ZoomOverride(
Integer min,
Integer max,
Map<String, Object> tag) {}
Object tag) {}

62
planetiler-custommap/src/main/java/com/onthegomap/planetiler/custommap/expression/BooleanExpressionScript.java 100644
Wyświetl plik

@ -0,0 +1,62 @@
package com.onthegomap.planetiler.custommap.expression;
import com.onthegomap.planetiler.expression.Expression;
import com.onthegomap.planetiler.reader.WithTags;
import com.onthegomap.planetiler.util.Format;
import java.util.List;
import java.util.Objects;
/**
* A boolean {@link Expression} based off of a dynamic expression script parsed from a string.
*
* @param expression The parsed CEL script
* @param expressionText The original CEL script string to evaluate
* @param inputClass Type of the context that the script is expecting.
*
* @param <T> Type of the expression context
*/
public record BooleanExpressionScript<T extends ScriptContext> (
String expressionText,
ConfigExpressionScript<T, Boolean> expression,
Class<T> inputClass
) implements Expression {
/** Creates a new boolean expression from {@code script} where {@code context} defines the available variables. */
public static <T extends ScriptContext> BooleanExpressionScript<T> script(String script,
ScriptEnvironment<T> context) {
var parsed = ConfigExpressionScript.parse(script, context, Boolean.class);
return new BooleanExpressionScript<>(script, parsed, context.clazz());
}
@Override
public boolean evaluate(WithTags input, List<String> matchKeys) {
return inputClass.isInstance(input) && expression.apply(inputClass.cast(input));
}
@Override
public String generateJavaCode() {
return "script(" + Format.quote("${ " + expressionText + " }") + ")";
}
@Override
public boolean equals(Object o) {
return o == this ||
(o instanceof BooleanExpressionScript<?> e && Objects.equals(e.expressionText, expressionText) &&
Objects.equals(e.inputClass, inputClass));
}
@Override
public int hashCode() {
return Objects.hash(expressionText, inputClass);
}
@Override
public Expression simplifyOnce() {
var result = expression.tryStaticEvaluate();
if (result.isSuccess()) {
return Boolean.TRUE.equals(result.get()) ? Expression.TRUE : Expression.FALSE;
} else {
return this;
}
}
}

246
planetiler-custommap/src/main/java/com/onthegomap/planetiler/custommap/expression/ConfigExpression.java 100644
Wyświetl plik

@ -0,0 +1,246 @@
package com.onthegomap.planetiler.custommap.expression;
import com.onthegomap.planetiler.custommap.TypeConversion;
import com.onthegomap.planetiler.expression.DataType;
import com.onthegomap.planetiler.expression.Expression;
import com.onthegomap.planetiler.expression.MultiExpression;
import com.onthegomap.planetiler.expression.Simplifiable;
import java.util.List;
import java.util.Objects;
import java.util.function.Function;
import java.util.stream.Stream;
/**
* A function defined in part of a schema config that produces an output value (min zoom, attribute value, etc.) for a
* feature at runtime.
* <p>
* This can be parsed from a structured object that lists combinations of tag key/values, an embedded script, or a
* combination of the two.
*
* @param <I> Type of the input context that expressions can pull values from at runtime.
* @param <O> Output type
*/
public interface ConfigExpression<I extends ScriptContext, O>
extends Function<I, O>, Simplifiable<ConfigExpression<I, O>> {
static <I extends ScriptContext, O> ConfigExpression<I, O> script(Signature<I, O> signature, String script) {
return ConfigExpressionScript.parse(script, signature.in(), signature.out());
}
static <I extends ScriptContext, O> ConfigExpression<I, O> variable(Signature<I, O> signature, String text) {
return new Variable<>(signature, text);
}
static <I extends ScriptContext, O> ConfigExpression<I, O> constOf(O value) {
return new Const<>(value);
}
static <I extends ScriptContext, O> ConfigExpression<I, O> coalesce(
List<ConfigExpression<I, O>> values) {
return new Coalesce<>(values);
}
static <I extends ScriptContext, O> ConfigExpression<I, O> getTag(Signature<I, O> signature,
ConfigExpression<I, String> tag) {
return new GetTag<>(signature, tag);
}
static <I extends ScriptContext, O> ConfigExpression<I, O> cast(Signature<I, O> signature,
ConfigExpression<I, ?> input, DataType dataType) {
return new Cast<>(signature, input, dataType);
}
static <I extends ScriptContext, O> Match<I, O> match(Signature<I, O> description,
MultiExpression<ConfigExpression<I, O>> multiExpression) {
return new Match<>(description, multiExpression, constOf(null));
}
static <I extends ScriptContext, O> Match<I, O> match(Signature<I, O> description,
MultiExpression<ConfigExpression<I, O>> multiExpression, ConfigExpression<I, O> fallback) {
return new Match<>(description, multiExpression, fallback);
}
static <I extends ScriptContext, O> Signature<I, O> signature(ScriptEnvironment<I> in, Class<O> out) {
return new Signature<>(in, out);
}
/** An expression that always returns {@code value}. */
record Const<I extends ScriptContext, O> (O value) implements ConfigExpression<I, O> {
@Override
public O apply(I i) {
return value;
}
}
/** An expression that returns the value associated with the first matching boolean expression. */
record Match<I extends ScriptContext, O> (
Signature<I, O> signature,
MultiExpression<ConfigExpression<I, O>> multiExpression,
ConfigExpression<I, O> fallback,
MultiExpression.Index<ConfigExpression<I, O>> indexed
) implements ConfigExpression<I, O> {
public Match(
Signature<I, O> signature,
MultiExpression<ConfigExpression<I, O>> multiExpression,
ConfigExpression<I, O> fallback
) {
this(signature, multiExpression, fallback, multiExpression.index());
}
@Override
public boolean equals(Object o) {
// ignore the indexed expression
return this == o ||
(o instanceof Match<?, ?> match &&
Objects.equals(signature, match.signature) &&
Objects.equals(multiExpression, match.multiExpression) &&
Objects.equals(fallback, match.fallback));
}
@Override
public int hashCode() {
// ignore the indexed expression
return Objects.hash(signature, multiExpression, fallback);
}
@Override
public O apply(I i) {
var resultFunction = indexed.getOrElse(i, fallback);
return resultFunction == null ? null : resultFunction.apply(i);
}
@Override
public ConfigExpression<I, O> simplifyOnce() {
var newMultiExpression = multiExpression
.mapResults(Simplifiable::simplifyOnce)
.simplify();
var newFallback = fallback.simplifyOnce();
if (newMultiExpression.expressions().isEmpty()) {
return newFallback;
}
var expressions = newMultiExpression.expressions();
for (int i = 0; i < expressions.size(); i++) {
var expression = expressions.get(i);
// if one of the cases is always true, then ignore the cases after it and make this value the fallback
if (Expression.TRUE.equals(expression.expression())) {
return new Match<>(
signature,
MultiExpression.of(expressions.stream().limit(i).toList()),
expression.result()
);
}
}
return new Match<>(signature, newMultiExpression, newFallback);
}
public Match<I, O> withDefaultValue(ConfigExpression<I, O> newFallback) {
return new Match<>(signature, multiExpression, newFallback);
}
}
/** An expression that returns the first non-null result of evaluating each child expression. */
record Coalesce<I extends ScriptContext, O> (List<? extends ConfigExpression<I, O>> children)
implements ConfigExpression<I, O> {
@Override
public O apply(I i) {
for (var condition : children) {
var result = condition.apply(i);
if (result != null) {
return result;
}
}
return null;
}
@Override
public ConfigExpression<I, O> simplifyOnce() {
return switch (children.size()) {
case 0 -> constOf(null);
case 1 -> children.get(0);
default -> {
var result = children.stream()
.flatMap(
child -> child instanceof Coalesce<I, O> childCoalesce ? childCoalesce.children.stream() :
Stream.of(child))
.filter(child -> !child.equals(constOf(null)))
.distinct()
.toList();
var indexOfFirstConst = result.stream().takeWhile(d -> !(d instanceof ConfigExpression.Const<I, O>)).count();
yield coalesce(result.stream().limit(indexOfFirstConst + 1).toList());
}
};
}
}
/** An expression that returns the value associated a given variable name at runtime. */
record Variable<I extends ScriptContext, O> (
Signature<I, O> signature,
String name
) implements ConfigExpression<I, O> {
public Variable {
if (!signature.in.containsVariable(name)) {
throw new ParseException("Variable not available: " + name);
}
}
@Override
public O apply(I i) {
return TypeConversion.convert(i.apply(name), signature.out);
}
}
/** An expression that returns the value associated a given tag of the input feature at runtime. */
record GetTag<I extends ScriptContext, O> (
Signature<I, O> signature,
ConfigExpression<I, String> tag
) implements ConfigExpression<I, O> {
@Override
public O apply(I i) {
return TypeConversion.convert(i.tagValueProducer().valueForKey(i, tag.apply(i)), signature.out);
}
@Override
public ConfigExpression<I, O> simplifyOnce() {
return new GetTag<>(signature, tag.simplifyOnce());
}
}
/** An expression that converts the input to a desired output {@link DataType} at runtime. */
record Cast<I extends ScriptContext, O> (
Signature<I, O> signature,
ConfigExpression<I, ?> input,
DataType output
) implements ConfigExpression<I, O> {
@Override
public O apply(I i) {
return TypeConversion.convert(output.convertFrom(input.apply(i)), signature.out);
}
@Override
public ConfigExpression<I, O> simplifyOnce() {
var in = input.simplifyOnce();
if (in instanceof ConfigExpression.Const<?, ?> inConst) {
return constOf(TypeConversion.convert(output.convertFrom(inConst.value), signature.out));
} else if (in instanceof ConfigExpression.Cast<?, ?> cast && cast.output == output) {
@SuppressWarnings("unchecked") ConfigExpression<I, ?> newIn = (ConfigExpression<I, ?>) cast.input;
return cast(signature, newIn, output);
} else {
return new Cast<>(signature, input.simplifyOnce(), output);
}
}
}
record Signature<I extends ScriptContext, O> (ScriptEnvironment<I> in, Class<O> out) {
public <O2> Signature<I, O2> withOutput(Class<O2> newOut) {
return new Signature<>(in, newOut);
}
}
}

187
planetiler-custommap/src/main/java/com/onthegomap/planetiler/custommap/expression/ConfigExpressionScript.java 100644
Wyświetl plik

@ -0,0 +1,187 @@
package com.onthegomap.planetiler.custommap.expression;
import com.onthegomap.planetiler.custommap.Contexts;
import com.onthegomap.planetiler.custommap.TypeConversion;
import com.onthegomap.planetiler.custommap.expression.stdlib.PlanetilerStdLib;
import com.onthegomap.planetiler.util.Try;
import java.util.Map;
import java.util.Objects;
import java.util.concurrent.ConcurrentHashMap;
import java.util.regex.Pattern;
import org.projectnessie.cel.extension.StringsLib;
import org.projectnessie.cel.tools.Script;
import org.projectnessie.cel.tools.ScriptCreateException;
import org.projectnessie.cel.tools.ScriptException;
import org.projectnessie.cel.tools.ScriptHost;
/**
* An expression that returns the result of evaluating a user-defined string script on the input environment context.
*
* @param <I> Type of the context that the script is expecting
* @param <O> Result type of the script
*/
public class ConfigExpressionScript<I extends ScriptContext, O> implements ConfigExpression<I, O> {
private static final Pattern EXPRESSION_PATTERN = Pattern.compile("^\\s*\\$\\{(.*)}\\s*$");
private static final Pattern ESCAPED_EXPRESSION_PATTERN = Pattern.compile("^\\s*\\\\+\\$\\{(.*)}\\s*$");
private final Script script;
private final Class<O> returnType;
private final String scriptText;
private final ScriptEnvironment<I> descriptor;
private ConfigExpressionScript(String scriptText, Script script, ScriptEnvironment<I> descriptor,
Class<O> returnType) {
this.scriptText = scriptText;
this.script = script;
this.returnType = returnType;
this.descriptor = descriptor;
}
/** Returns true if this is a string expression like {@code "${ ... }"} */
public static boolean isScript(Object obj) {
if (obj instanceof String string) {
var matcher = EXPRESSION_PATTERN.matcher(string);
return matcher.matches();
}
return false;
}
/**
* Returns true if this is an escaped string expression that should just be treated as a string like {@code "\${ ...
* }"}
*/
public static boolean isEscapedScript(Object obj) {
if (obj instanceof String string) {
var matcher = ESCAPED_EXPRESSION_PATTERN.matcher(string);
return matcher.matches();
}
return false;
}
/**
* Removes script escape character from a string {@code "\${ ... }"} becomes {@code "${ ... }"}
*/
public static Object unescape(Object obj) {
if (isEscapedScript(obj)) {
return obj.toString().replaceFirst("\\\\\\$", "\\$");
} else {
return obj;
}
}
/**
* Returns the script text between the {@code "${ ... }"} characters.
*/
public static String extractScript(Object obj) {
if (obj instanceof String string) {
var matcher = EXPRESSION_PATTERN.matcher(string);
if (matcher.matches()) {
return matcher.group(1);
}
}
return null;
}
/**
* Returns an expression parsed from a user-supplied script string.
*
* @throws ParseException if the script failes to compile or type-check
*/
public static <I extends ScriptContext> ConfigExpressionScript<I, Object> parse(String string,
ScriptEnvironment<I> description) {
return parse(string, description, Object.class);
}
/**
* Returns an expression parsed from a user-supplied script string that coerces the result to {@code O}.
*
* @throws ParseException if the script failes to compile or type-check
*/
public static <I extends ScriptContext, O> ConfigExpressionScript<I, O> parse(String string,
ScriptEnvironment<I> description, Class<O> expected) {
ScriptHost scriptHost = ScriptHost.newBuilder().build();
try {
var scriptBuilder = scriptHost.buildScript(string).withLibraries(
new StringsLib(),
new PlanetilerStdLib()
);
if (!description.declarations().isEmpty()) {
scriptBuilder.withDeclarations(description.declarations());
}
if (!description.types().isEmpty()) {
scriptBuilder.withTypes(description.types());
}
var script = scriptBuilder.build();
return new ConfigExpressionScript<>(string, script, description, expected);
} catch (ScriptCreateException e) {
throw new ParseException(string, e);
}
}
@Override
public O apply(I input) {
try {
return TypeConversion.convert(script.execute(Object.class, input), returnType);
} catch (ScriptException e) {
throw new EvaluationException("Error evaluating script '%s'".formatted(scriptText), e);
}
}
@Override
public boolean equals(Object o) {
// ignore the parsed script object
return this == o || (o instanceof ConfigExpressionScript<?, ?> config &&
returnType.equals(config.returnType) &&
scriptText.equals(config.scriptText));
}
@Override
public int hashCode() {
// ignore the parsed script object
return Objects.hash(returnType, scriptText);
}
private static final Map<ConfigExpressionScript<?, ?>, Boolean> staticEvaluationCache = new ConcurrentHashMap<>();
/**
* Attempts to parse and evaluate this script in an environment with no variables.
* <p>
* If this returns {@link Try.Success} then it means this script will always return the same constant value and we can
* avoid evaluating it at runtime.
*/
public Try<O> tryStaticEvaluate() {
// type checking can be expensive when run hundreds of times simplifying expressions iteratively and it never
// changes for a given script and input environment, so cache results between calls.
boolean canStaticEvaluate =
staticEvaluationCache.computeIfAbsent(this, config -> config.doTryStaticEvaluate().isSuccess());
if (canStaticEvaluate) {
return doTryStaticEvaluate();
} else {
return Try.failure(new IllegalStateException());
}
}
private Try<O> doTryStaticEvaluate() {
return Try
.apply(
() -> ConfigExpressionScript.parse(scriptText, Contexts.Root.DESCRIPTION, returnType).apply(Contexts.root()));
}
@Override
public String toString() {
return "ConfigExpression[returnType=" + returnType +
", scriptText='" + scriptText + '\'' +
']';
}
@Override
public ConfigExpression<I, O> simplifyOnce() {
var result = tryStaticEvaluate();
if (result.isSuccess()) {
return ConfigExpression.constOf(result.get());
} else if (descriptor.containsVariable(scriptText.strip())) {
return ConfigExpression.variable(ConfigExpression.signature(descriptor, returnType), scriptText.strip());
}
return this;
}
}

11
planetiler-custommap/src/main/java/com/onthegomap/planetiler/custommap/expression/EvaluationException.java 100644
Wyświetl plik

@ -0,0 +1,11 @@
package com.onthegomap.planetiler.custommap.expression;
/**
* Exception that occurs at runtime when evaluating a {@link ConfigExpressionScript}.
*/
public class EvaluationException extends RuntimeException {
public EvaluationException(String script, Exception cause) {
super("Error evaluating script: %s".formatted(script), cause);
}
}

15
planetiler-custommap/src/main/java/com/onthegomap/planetiler/custommap/expression/ParseException.java 100644
Wyświetl plik

@ -0,0 +1,15 @@
package com.onthegomap.planetiler.custommap.expression;
/**
* Exception that occurs at compile-time when preparing an embedded expression.
*/
public class ParseException extends RuntimeException {
public ParseException(String script, Exception cause) {
super("Error parsing: %s".formatted(script), cause);
}
public ParseException(String message) {
super(message);
}
}

25
planetiler-custommap/src/main/java/com/onthegomap/planetiler/custommap/expression/ScriptContext.java 100644
Wyświetl plik

@ -0,0 +1,25 @@
package com.onthegomap.planetiler.custommap.expression;
import com.google.common.base.Function;
import com.onthegomap.planetiler.custommap.TagValueProducer;
import com.onthegomap.planetiler.reader.WithTags;
import java.util.Map;
/**
* The runtime environment of an executing expression script that returns variables by their name.
*/
public interface ScriptContext extends Function<String, Object>, WithTags {
static ScriptContext empty() {
return key -> null;
}
@Override
default Map<String, Object> tags() {
// TODO remove this when MultiExpression can take any object
return Map.of();
}
default TagValueProducer tagValueProducer() {
return TagValueProducer.EMPTY;
}
}

47
planetiler-custommap/src/main/java/com/onthegomap/planetiler/custommap/expression/ScriptEnvironment.java 100644
Wyświetl plik

@ -0,0 +1,47 @@
package com.onthegomap.planetiler.custommap.expression;
import com.google.api.expr.v1alpha1.Decl;
import java.util.List;
import java.util.stream.Stream;
/**
* Type definitions for the environment that a script expression runs in.
*
* @param types Additional types available.
* @param declarations Global variable types
* @param clazz Class of the input context type
* @param <T> The runtime expression context type
*/
public record ScriptEnvironment<T extends ScriptContext> (List<Object> types, List<Decl> declarations, Class<T> clazz) {
private static <T> List<T> concat(List<T> a, T[] b) {
return Stream.concat(a.stream(), Stream.of(b)).toList();
}
/** Returns a copy of this environment with a new input type {@code U}. */
public <U extends ScriptContext> ScriptEnvironment<U> forInput(Class<U> newClazz) {
return new ScriptEnvironment<>(types, declarations, newClazz);
}
/** Returns a copy of this environment with a list of variable declarations appended to the global environment. */
public ScriptEnvironment<T> withDeclarations(Decl... others) {
return new ScriptEnvironment<>(types, concat(declarations, others), clazz);
}
/** Returns an empty environment with no variables defined. */
public static ScriptEnvironment<ScriptContext> root() {
return new ScriptEnvironment<>(List.of(), List.of(), ScriptContext.class);
}
/** Returns true if this contains a variable declaration for {@code variable}. */
public boolean containsVariable(String variable) {
return declarations().stream().anyMatch(decl -> decl.getName().equals(variable));
}
@Override
public String toString() {
return "ScriptContextDescription{" +
"declarations=" + declarations.stream().map(Decl::getName).toList() +
", clazz=" + clazz +
'}';
}
}

14
planetiler-custommap/src/main/java/com/onthegomap/planetiler/custommap/expression/stdlib/BuiltInFunction.java 100644
Wyświetl plik

@ -0,0 +1,14 @@
package com.onthegomap.planetiler.custommap.expression.stdlib;
import com.google.api.expr.v1alpha1.Decl;
import java.util.List;
import org.projectnessie.cel.interpreter.functions.Overload;
/**
* Groups together a built-in function's type signature and implementation that is available to dynamic expressions.
*/
record BuiltInFunction(Decl signature, List<Overload> implementations) {
BuiltInFunction(Decl signature, Overload... implementations) {
this(signature, List.of(implementations));
}
}

31
planetiler-custommap/src/main/java/com/onthegomap/planetiler/custommap/expression/stdlib/PlanetilerLib.java 100644
Wyświetl plik

@ -0,0 +1,31 @@
package com.onthegomap.planetiler.custommap.expression.stdlib;
import java.util.List;
import org.projectnessie.cel.EnvOption;
import org.projectnessie.cel.Library;
import org.projectnessie.cel.ProgramOption;
import org.projectnessie.cel.interpreter.functions.Overload;
/** Creates a {@link Library} of built-in functions that can be made available to dynamic expressions. */
class PlanetilerLib implements Library {
private final List<BuiltInFunction> builtInFunctions;
PlanetilerLib(List<BuiltInFunction> builtInFunctions) {
this.builtInFunctions = builtInFunctions;
}
@Override
public List<EnvOption> getCompileOptions() {
return List.of(EnvOption.declarations(
builtInFunctions.stream().map(BuiltInFunction::signature).toList()
));
}
@Override
public List<ProgramOption> getProgramOptions() {
return List.of(ProgramOption.functions(
builtInFunctions.stream().flatMap(b -> b.implementations().stream()).toArray(Overload[]::new)
));
}
}

199
planetiler-custommap/src/main/java/com/onthegomap/planetiler/custommap/expression/stdlib/PlanetilerStdLib.java 100644
Wyświetl plik

@ -0,0 +1,199 @@
package com.onthegomap.planetiler.custommap.expression.stdlib;
import static org.projectnessie.cel.checker.Decls.newOverload;
import com.google.api.expr.v1alpha1.Type;
import java.util.List;
import java.util.Objects;
import java.util.function.DoubleBinaryOperator;
import java.util.function.LongBinaryOperator;
import java.util.stream.IntStream;
import java.util.stream.Stream;
import org.projectnessie.cel.checker.Decls;
import org.projectnessie.cel.common.types.BoolT;
import org.projectnessie.cel.common.types.DoubleT;
import org.projectnessie.cel.common.types.Err;
import org.projectnessie.cel.common.types.IntT;
import org.projectnessie.cel.common.types.NullT;
import org.projectnessie.cel.common.types.StringT;
import org.projectnessie.cel.common.types.ref.Val;
import org.projectnessie.cel.common.types.traits.Lister;
import org.projectnessie.cel.common.types.traits.Mapper;
import org.projectnessie.cel.interpreter.functions.Overload;
/**
* Built-in functions to expose to all dynamic expression used in planetiler configs.
*/
public class PlanetilerStdLib extends PlanetilerLib {
private static final int VARARG_LIMIT = 32;
private static final Type T = Decls.newTypeParamType("T");
private static final Type K = Decls.newTypeParamType("K");
private static final Type V = Decls.newTypeParamType("V");
public PlanetilerStdLib() {
super(List.of(
// coalesce(a, b, c...) -> first non-null value
new BuiltInFunction(
Decls.newFunction("coalesce",
IntStream.range(0, VARARG_LIMIT)
.mapToObj(
i -> newOverload("coalesce_" + i, IntStream.range(0, i).mapToObj(d -> Decls.Any).toList(), Decls.Any))
.toList()
),
Overload.overload("coalesce",
null,
null,
(a, b) -> a == null || a instanceof NullT ? b : a,
args -> {
for (var arg : args) {
if (!(arg instanceof NullT)) {
return arg;
}
}
return NullT.NullValue;
})
),
// nullif(a, b) -> null if a == b, otherwise a
new BuiltInFunction(
Decls.newFunction("nullif",
Decls.newOverload("nullif", List.of(T, T), T)
),
Overload.binary("nullif", (a, b) -> Objects.equals(a, b) ? NullT.NullValue : a)
),
// string.replaceRegex(regex, replacement) -> replaces all matches for regex in string with replacement
new BuiltInFunction(
Decls.newFunction("replaceRegex",
Decls.newInstanceOverload("replaceRegex", List.of(Decls.String, Decls.String, Decls.String), Decls.String)
),
Overload.function("replaceRegex", values -> {
try {
String string = ((String) values[0].value());
String regexp = ((String) values[1].value());
String replace = ((String) values[2].value());
return StringT.stringOf(string.replaceAll(regexp, replace));
} catch (RuntimeException e) {
return Err.newErr(e, "%s", e.getMessage());
}
})
),
// map.has(key) -> true if key is present in map
// map.has(key, value...) true if the value for key is in the list of values provided
new BuiltInFunction(
Decls.newFunction("has",
IntStream.range(0, VARARG_LIMIT)
.mapToObj(
i -> Decls.newInstanceOverload("map_has_" + i, Stream.concat(
Stream.of(Decls.newMapType(K, V), K),
IntStream.range(0, i).mapToObj(n -> V)
).toList(), Decls.Bool)
).toList()
),
Overload.overload("has",
null,
null,
(map, key) -> {
try {
return getFromMap(map, key) != null ? BoolT.True : BoolT.False;
} catch (RuntimeException e) {
return Err.newErr(e, "%s", e.getMessage());
}
},
args -> {
try {
Val elem = getFromMap(args[0], args[1]);
if (elem == null) {
return BoolT.False;
}
for (int i = 2; i < args.length; i++) {
if (args[i].equals(elem)) {
return BoolT.True;
}
}
return BoolT.False;
} catch (RuntimeException e) {
return Err.newErr(e, "%s", e.getMessage());
}
})
),
// map.get(key) -> the value for key, or null if missing
new BuiltInFunction(
Decls.newFunction("get", Decls.newInstanceOverload("get", List.of(Decls.newMapType(K, V), K), V)),
Overload.binary("get", (map, key) -> {
try {
var value = getFromMap(map, key);
return value == null ? NullT.NullValue : value;
} catch (RuntimeException e) {
return Err.newErr(e, "%s", e.getMessage());
}
})
),
// map.getOrDefault(key, default) -> the value for key, or default if missing
new BuiltInFunction(
Decls.newFunction("getOrDefault",
Decls.newInstanceOverload("getOrDefault", List.of(Decls.newMapType(K, V), K, V), V)),
Overload.function("getOrDefault", args -> {
try {
var value = getFromMap(args[0], args[1]);
return value == null ? args[2] : value;
} catch (RuntimeException e) {
return Err.newErr(e, "%s", e.getMessage());
}
})
),
// min(list) -> the minimum value from the list, or null if empty
new BuiltInFunction(
Decls.newFunction("min",
Decls.newOverload("min_int", List.of(Decls.newListType(Decls.Int)), Decls.Int),
Decls.newOverload("min_double", List.of(Decls.newListType(Decls.Double)), Decls.Double)
),
Overload.unary("min", list -> reduceNumeric(list, Math::min, Math::min))
),
// max(list) -> the maximum value from the list, or null if empty
new BuiltInFunction(
Decls.newFunction("max",
Decls.newOverload("max_int", List.of(Decls.newListType(Decls.Int)), Decls.Int),
Decls.newOverload("max_double", List.of(Decls.newListType(Decls.Double)), Decls.Double)
),
Overload.unary("max", list -> reduceNumeric(list, Math::max, Math::max))
)
));
}
private static Val getFromMap(Val map, Val key) {
return map instanceof Mapper mapper ? mapper.find(key) : null;
}
private static Val reduceNumeric(Val list, LongBinaryOperator intFn, DoubleBinaryOperator doubleFn) {
try {
var iterator = ((Lister) list).iterator();
if (!iterator.hasNext().booleanValue()) {
return NullT.NullValue;
}
var next = iterator.next();
if (next instanceof IntT intT) {
long acc = intT.intValue();
while (iterator.hasNext().booleanValue()) {
acc = intFn.applyAsLong(iterator.next().convertToNative(Long.class), acc);
}
return IntT.intOf(acc);
} else if (next instanceof DoubleT doubleT) {
double acc = doubleT.convertToNative(Double.class);
while (iterator.hasNext().booleanValue()) {
acc = doubleFn.applyAsDouble(iterator.next().convertToNative(Double.class), acc);
}
return DoubleT.doubleOf(acc);
} else {
return Err.newErr("Bad element of list for min(): %s", next);
}
} catch (RuntimeException e) {
return Err.newErr(e, "%s", e.getMessage());
}
}
}

78
planetiler-custommap/src/main/java/com/onthegomap/planetiler/custommap/validator/SchemaSpecification.java 100644
Wyświetl plik

@ -0,0 +1,78 @@
package com.onthegomap.planetiler.custommap.validator;
import static com.onthegomap.planetiler.config.PlanetilerConfig.MAX_MAXZOOM;
import com.fasterxml.jackson.annotation.JsonFormat;
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.onthegomap.planetiler.custommap.YAML;
import com.onthegomap.planetiler.geo.GeometryType;
import java.nio.file.Path;
import java.util.List;
import java.util.Map;
/** A model of example input source features and expected output vector tile features that a schema should produce. */
@JsonIgnoreProperties(ignoreUnknown = true)
public record SchemaSpecification(List<Example> examples) {
public static SchemaSpecification load(Path path) {
return YAML.load(path, SchemaSpecification.class);
}
public static SchemaSpecification load(String string) {
return YAML.load(string, SchemaSpecification.class);
}
/** An individual test case */
public record Example(
String name,
InputFeature input,
@JsonFormat(with = JsonFormat.Feature.ACCEPT_SINGLE_VALUE_AS_ARRAY) List<OutputFeature> output
) {
@Override
public List<OutputFeature> output() {
return output == null ? List.of() : output;
}
}
/** Description of an input feature from a source that the schema will process. */
public record InputFeature(
String source,
String geometry,
Map<String, Object> tags
) {
@Override
public Map<String, Object> tags() {
return tags == null ? Map.of() : tags;
}
}
/** Description of an expected vector tile feature that the schema should produce. */
public record OutputFeature(
String layer,
GeometryType geometry,
@JsonProperty("min_zoom") Integer minZoom,
@JsonProperty("max_zoom") Integer maxZoom,
@JsonProperty("at_zoom") Integer atZoom,
@JsonProperty("allow_extra_tags") Boolean allowExtraTags,
@JsonProperty("tags") Map<String, Object> tags
) {
@Override
public Map<String, Object> tags() {
return tags == null ? Map.of() : tags;
}
@Override
public Boolean allowExtraTags() {
return allowExtraTags == null || allowExtraTags;
}
@Override
public Integer atZoom() {
return atZoom == null ? MAX_MAXZOOM : atZoom;
}
}
}

273
planetiler-custommap/src/main/java/com/onthegomap/planetiler/custommap/validator/SchemaValidator.java 100644
Wyświetl plik

@ -0,0 +1,273 @@
package com.onthegomap.planetiler.custommap.validator;
import com.fasterxml.jackson.core.JacksonException;
import com.onthegomap.planetiler.FeatureCollector;
import com.onthegomap.planetiler.Profile;
import com.onthegomap.planetiler.config.Arguments;
import com.onthegomap.planetiler.config.PlanetilerConfig;
import com.onthegomap.planetiler.custommap.ConfiguredProfile;
import com.onthegomap.planetiler.custommap.YAML;
import com.onthegomap.planetiler.custommap.configschema.SchemaConfig;
import com.onthegomap.planetiler.geo.GeometryType;
import com.onthegomap.planetiler.reader.SimpleFeature;
import com.onthegomap.planetiler.stats.Stats;
import com.onthegomap.planetiler.util.AnsiColors;
import com.onthegomap.planetiler.util.FileWatcher;
import com.onthegomap.planetiler.util.Format;
import com.onthegomap.planetiler.util.Try;
import java.io.PrintStream;
import java.nio.file.Path;
import java.time.Duration;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.HashSet;
import java.util.List;
import java.util.Locale;
import java.util.Objects;
import java.util.Set;
import java.util.TreeSet;
import java.util.stream.Stream;
import org.apache.commons.lang3.exception.ExceptionUtils;
import org.geotools.geometry.jts.WKTReader2;
import org.locationtech.jts.geom.Geometry;
import org.locationtech.jts.io.ParseException;
import org.snakeyaml.engine.v2.exceptions.YamlEngineException;
/** Verifies that a profile maps input elements map to expected output vector tile features. */
public class SchemaValidator {
private static final String PASS_BADGE = AnsiColors.greenBackground(" PASS ");
private static final String FAIL_BADGE = AnsiColors.redBackground(" FAIL ");
public static void main(String[] args) {
// let users run `verify schema.yml` as a shortcut
String schemaFile = null;
if (args.length > 0 && args[0].endsWith(".yml") && !args[0].startsWith("-")) {
schemaFile = args[0];
args = Stream.of(args).skip(1).toArray(String[]::new);
}
var arguments = Arguments.fromEnvOrArgs(args);
var schema = schemaFile == null ? arguments.inputFile("schema", "Schema file") :
arguments.inputFile("schema", "Schema file", Path.of(schemaFile));
var watch =
arguments.getBoolean("watch", "Watch files for changes and re-run validation when schema or spec changes", false);
PrintStream output = System.out;
output.println("OK");
var paths = validateFromCli(schema, arguments, output);
if (watch) {
output.println();
output.println("Watching filesystem for changes...");
var watcher = FileWatcher.newWatcher(paths.toArray(Path[]::new));
watcher.pollForChanges(Duration.ofMillis(300), changed -> validateFromCli(schema, arguments, output));
}
}
private static boolean hasCause(Throwable t, Class<?> cause) {
return t != null && (cause.isInstance(t) || hasCause(t.getCause(), cause));
}
static Set<Path> validateFromCli(Path schema, Arguments args, PrintStream output) {
Set<Path> pathsToWatch = new HashSet<>();
pathsToWatch.add(schema);
output.println();
output.println("Validating...");
output.println();
SchemaValidator.Result result;
try {
var parsedSchema = SchemaConfig.load(schema);
var examples = parsedSchema.examples();
// examples can either be embedded in the yaml file, or referenced
SchemaSpecification spec;
if (examples instanceof String s) {
var path = Path.of(s);
if (!path.isAbsolute()) {
path = schema.resolveSibling(path);
}
// if referenced, make sure we watch that file for changes
pathsToWatch.add(path);
spec = SchemaSpecification.load(path);
} else if (examples != null) {
spec = YAML.convertValue(parsedSchema, SchemaSpecification.class);
} else {
spec = new SchemaSpecification(List.of());
}
result = validate(parsedSchema, spec, args);
} catch (Exception exception) {
Throwable rootCause = ExceptionUtils.getRootCause(exception);
if (hasCause(exception, com.onthegomap.planetiler.custommap.expression.ParseException.class)) {
output.println(AnsiColors.red("Malformed expression:\n\n" + rootCause.toString().indent(4)));
} else if (hasCause(exception, YamlEngineException.class) || hasCause(exception, JacksonException.class)) {
output.println(AnsiColors.red("Malformed yaml input:\n\n" + rootCause.toString().indent(4)));
} else {
output.println(AnsiColors.red(
"Unexpected exception thrown:\n" + rootCause.toString().indent(4) + "\n" +
String.join("\n", ExceptionUtils.getStackTrace(rootCause)))
.indent(4));
}
return pathsToWatch;
}
int failed = 0, passed = 0;
List<ExampleResult> failures = new ArrayList<>();
for (var example : result.results) {
if (example.ok()) {
passed++;
output.printf("%s %s%n", PASS_BADGE, example.example().name());
} else {
failed++;
printFailure(example, output);
failures.add(example);
}
}
if (!failures.isEmpty()) {
output.println();
output.println("Summary of failures:");
for (var failure : failures) {
printFailure(failure, output);
}
}
List<String> summary = new ArrayList<>();
boolean none = (passed + failed) == 0;
if (none || failed > 0) {
summary.add(AnsiColors.redBold(failed + " failed"));
}
if (none || passed > 0) {
summary.add(AnsiColors.greenBold(passed + " passed"));
}
if (none || passed > 0 && failed > 0) {
summary.add((failed + passed) + " total");
}
output.println();
output.println(String.join(", ", summary));
return pathsToWatch;
}
private static void printFailure(ExampleResult example, PrintStream output) {
output.printf("%s %s%n", FAIL_BADGE, example.example().name());
if (example.issues.isFailure()) {
output.println(ExceptionUtils.getStackTrace(example.issues.exception()).indent(4).stripTrailing());
} else {
for (var issue : example.issues().get()) {
output.println(" ● " + issue.indent(4).strip());
}
}
}
private static Geometry parseGeometry(String geometry) {
String wkt = switch (geometry.toLowerCase(Locale.ROOT).trim()) {
case "point" -> "POINT (0 0)";
case "line" -> "LINESTRING (0 0, 1 1)";
case "polygon" -> "POLYGON ((0 0, 1 0, 1 1, 0 1, 0 0))";
default -> geometry;
};
try {
return new WKTReader2().read(wkt);
} catch (ParseException e) {
throw new IllegalArgumentException("""
Bad geometry: "%s", must be "point" "line" "polygon" or a valid WKT string.
""".formatted(geometry));
}
}
/**
* Returns the result of validating the profile defined by {@code schema} against the examples in
* {@code specification}.
*/
public static Result validate(SchemaConfig schema, SchemaSpecification specification, Arguments args) {
return validate(new ConfiguredProfile(schema), specification, args);
}
/** Returns the result of validating {@code profile} against the examples in {@code specification}. */
public static Result validate(Profile profile, SchemaSpecification specification, Arguments args) {
var featureCollectorFactory = new FeatureCollector.Factory(PlanetilerConfig.from(args.silence()), Stats.inMemory());
return new Result(specification.examples().stream().map(example -> new ExampleResult(example, Try.apply(() -> {
List<String> issues = new ArrayList<>();
var input = example.input();
var expectedFeatures = example.output();
var geometry = parseGeometry(input.geometry());
var feature = SimpleFeature.create(geometry, input.tags(), input.source(), null, 0);
var collector = featureCollectorFactory.get(feature);
profile.processFeature(feature, collector);
List<FeatureCollector.Feature> result = new ArrayList<>();
collector.forEach(result::add);
if (result.size() != expectedFeatures.size()) {
issues.add(
"Different number of elements, expected=%s actual=%s".formatted(expectedFeatures.size(), result.size()));
} else {
// TODO print a diff of the input and output feature YAML representations
for (int i = 0; i < expectedFeatures.size(); i++) {
var expected = expectedFeatures.get(i);
var actual = result.stream().max(proximityTo(expected)).orElseThrow();
result.remove(actual);
var actualTags = actual.getAttrsAtZoom(expected.atZoom());
String prefix = "feature[%d]".formatted(i);
validate(prefix + ".layer", issues, expected.layer(), actual.getLayer());
validate(prefix + ".minzoom", issues, expected.minZoom(), actual.getMinZoom());
validate(prefix + ".maxzoom", issues, expected.maxZoom(), actual.getMaxZoom());
validate(prefix + ".geometry", issues, expected.geometry(), GeometryType.typeOf(actual.getGeometry()));
Set<String> tags = new TreeSet<>(actualTags.keySet());
expected.tags().forEach((tag, value) -> {
validate(prefix + ".tags[\"%s\"]".formatted(tag), issues, value, actualTags.get(tag), false);
tags.remove(tag);
});
if (Boolean.FALSE.equals(expected.allowExtraTags())) {
for (var tag : tags) {
validate(prefix + ".tags[\"%s\"]".formatted(tag), issues, null, actualTags.get(tag), false);
}
}
}
}
return issues;
}))).toList());
}
private static Comparator<FeatureCollector.Feature> proximityTo(SchemaSpecification.OutputFeature expected) {
return Comparator.comparingInt(item -> (Objects.equals(item.getLayer(), expected.layer()) ? 2 : 0) +
(Objects.equals(GeometryType.typeOf(item.getGeometry()), expected.geometry()) ? 1 : 0));
}
private static <T> void validate(String field, List<String> issues, T expected, T actual, boolean ignoreWhenNull) {
if ((!ignoreWhenNull || expected != null) && !Objects.equals(expected, actual)) {
// handle when expected and actual are int/long or long/int
if (expected instanceof Number && actual instanceof Number && expected.toString().equals(actual.toString())) {
return;
}
issues.add("%s: expected <%s> actual <%s>".formatted(field, format(expected), format(actual)));
}
}
private static String format(Object o) {
if (o == null) {
return "null";
} else if (o instanceof String s) {
return Format.quote(s);
} else {
return o.toString();
}
}
private static <T> void validate(String field, List<String> issues, T expected, T actual) {
validate(field, issues, expected, actual, true);
}
/** Result of comparing the output vector tile feature to what was expected. */
public record ExampleResult(
SchemaSpecification.Example example,
// TODO include a symmetric diff so we can pretty-print the expected/actual output diff
Try<List<String>> issues
) {
public boolean ok() {
return issues.isSuccess() && issues.get().isEmpty();
}
}
public record Result(List<ExampleResult> results) {
public boolean ok() {
return results.stream().allMatch(ExampleResult::ok);
}
}
}

12
planetiler-custommap/src/main/resources/samples/highway_areas.yml
Wyświetl plik

@ -10,10 +10,9 @@ tag_mappings:
bridge: boolean
layer: long
layers:
- name: highway_area
- id: highway_area
features:
- sources:
- osm
- source: osm
geometry: polygon
min_zoom: 14
include_when:
@ -24,14 +23,13 @@ layers:
- key: layer
- key: surface
- key: bridge
- sources:
- osm
- source: osm
geometry: polygon
min_zoom: 14
include_when:
man_made: bridge
attributes:
- key: man_made
constant_value: bridge
value: bridge
- key: layer
- key: surface
- key: surface

6
planetiler-custommap/src/main/resources/samples/manholes.yml
Wyświetl plik

@ -7,9 +7,9 @@ sources:
type: osm
url: geofabrik:rhode-island
layers:
- name: manhole
- id: manhole
features:
- sources:
- source:
- osm
geometry: point
min_zoom: 14
@ -19,4 +19,4 @@ layers:
- key: man_made
- key: manhole
- key: operator
- key: ref
- key: ref

78
planetiler-custommap/src/main/resources/samples/owg_simple.yml
Wyświetl plik

@ -15,10 +15,9 @@ tag_mappings:
layer: long
tunnel: boolean
layers:
- name: water
- id: water
features:
- sources:
- osm
- source: osm
geometry: polygon
include_when:
natural: water
@ -29,7 +28,6 @@ layers:
intermittent: true
- key: name
min_tile_cover_size: 0.01
include_when:
exclude_when:
tag:
key: water
@ -37,15 +35,12 @@ layers:
- river
- canal
- stream
- sources:
- water_polygons
- source: water_polygons
geometry: polygon
include_when:
attributes:
- key: natural
constant_value: water
- sources:
- osm
value: water
- source: osm
min_zoom: 7
geometry: line
include_when:
@ -62,10 +57,9 @@ layers:
intermittent: true
- key: name
min_zoom: 12
- name: road
- id: road
features:
- sources:
- osm
- source: osm
geometry: line
include_when:
highway:
@ -84,37 +78,31 @@ layers:
- living_street
- service
- track
min_zoom: 4
zoom_override:
- min: 5
tag:
highway: trunk
- min: 7
tag:
highway: primary
- min: 8
tag:
highway: secondary
- min: 9
tag:
highway:
- tertiary
- motorway_link
- trunk_link
- primary_link
- secondary_link
- tertiary_link
- min: 11
tag:
highway:
- unclassified
- residential
- living_street
- min: 12
tag:
highway: track
- min: 13
tag:
highway: service
min_zoom:
default_value: 4
overrides:
5:
highway: trunk
7:
highway: primary
8:
highway: secondary
9:
highway:
- tertiary
- motorway_link
- trunk_link
- primary_link
- secondary_link
- tertiary_link
11:
highway:
- unclassified
- residential
- living_street
12:
highway: track
13:
highway: service
attributes:
- key: highway

10
planetiler-custommap/src/main/resources/samples/power.yml
Wyświetl plik

@ -7,10 +7,9 @@ sources:
type: osm
url: geofabrik:new-jersey
layers:
- name: power
- id: power
features:
- sources:
- osm
- source: osm
geometry: point
min_zoom: 13
include_when:
@ -21,8 +20,7 @@ layers:
- key: ref
- key: height
- key: operator
- sources:
- osm
- source: osm
geometry: line
min_zoom: 7
include_when:
@ -32,4 +30,4 @@ layers:
- key: power
- key: voltage
- key: cables
- key: operator
- key: operator

816
planetiler-custommap/src/main/resources/samples/shortbread.spec.yml 100644
Wyświetl plik

@ -0,0 +1,816 @@
examples:
- name: ocean polygons
input:
source: ocean
geometry: polygon
tags:
x: 1
y: 2
output:
layer: ocean
geometry: polygon
min_zoom: 0
tags: # no tags
- name: natural=glacier
input:
source: osm
geometry: polygon
tags:
natural: glacier
name: The glacier
name:en: The glacier (en)
name:de: The glacier (de)
output:
- layer: water_polygons
geometry: polygon
min_zoom: 4
tags:
kind: glacier
- layer: water_polygons_labels
geometry: point
min_zoom: 14
tags:
kind: glacier
name: The glacier
name_en: The glacier (en)
name_de: The glacier (de)
- name: waterway=dock
input:
source: osm
geometry: polygon
tags:
waterway: dock
output:
layer: water_polygons
geometry: polygon
min_zoom: 10
tags:
kind: dock
- name: waterway=canal linestring
input:
source: osm
geometry: line
tags:
waterway: canal
name: The Canal
name:en: The Canal (en)
name:de: The Canal (de)
output:
- layer: water_lines
geometry: line
min_zoom: 9
tags:
kind: canal
- layer: water_line_labels
geometry: line
min_zoom: 12
tags:
kind: canal
name: The Canal
name_en: The Canal (en)
name_de: The Canal (de)
- name: waterway=stream linestring
input:
source: osm
geometry: line
tags:
waterway: stream
name: The Stream
name:en: The Stream (en)
name:de: The Stream (de)
output:
- layer: water_lines
geometry: line
min_zoom: 14
tags:
kind: stream
- layer: water_line_labels
geometry: line
min_zoom: 14
tags:
kind: stream
name: The Stream
name_en: The Stream (en)
name_de: The Stream (de)
- name: landuse=grass
input:
source: osm
geometry: polygon
tags:
landuse: grass
output:
- layer: land
geometry: polygon
min_zoom: 11
tags:
kind: grass
- name: natural=wood
input:
source: osm
geometry: polygon
tags:
natural: wood
output:
- layer: land
geometry: polygon
min_zoom: 7
tags:
kind: wood
- name: landuse=forest override kind to wood
input:
source: osm
geometry: polygon
tags:
landuse: forest
output:
- layer: land
geometry: polygon
min_zoom: 7
tags:
kind: wood
- name: amenity=parking
input:
source: osm
geometry: polygon
tags:
amenity: parking
output:
- layer: sites
geometry: polygon
min_zoom: 14
tags:
kind: parking
- name: building=yes
input:
source: osm
geometry: polygon
tags:
building: yes
output:
- layer: buildings
geometry: polygon
min_zoom: 14
- name: building=house
input:
source: osm
geometry: polygon
tags:
building: house
output:
- layer: buildings
geometry: polygon
min_zoom: 14
- name: address polygon with house number
input:
source: osm
geometry: polygon
tags:
addr:housenumber: 123
output:
- layer: addresses
geometry: point
min_zoom: 14
tags:
number: 123
- name: address point with house name
input:
source: osm
geometry: point
tags:
addr:housename: the 123 house
output:
- layer: addresses
geometry: point
min_zoom: 14
tags:
name: the 123 house
- name: B27 highway
input:
source: osm
geometry: line
tags:
change:lanes:backward: no
change:lanes:forward: not_left|not_right
embankment: yes
highway: primary
lanes: 3
lanes:backward: 1
lanes:forward: 2
maxspeed: 100
overtaking:backward: no
priority_road: designated
ref: B 27
sidewalk: no
source:maxspeed: DE:rural
surface: asphalt
zone:traffic: DE:rural
output:
- layer: streets
geometry: line
min_zoom: 8
tags:
bridge: false
kind: primary
link: false
rail: false
surface: asphalt
tunnel: false
- layer: street_labels
geometry: line
min_zoom: 12
allow_extra_tags: false
tags:
kind: primary
ref: B 27
ref_rows: 1
ref_cols: 4
- name: B39A link
input:
source: osm
geometry: line
tags:
cycleway:right: no
destination: Löwenstein;Obersulm;Ellhofen;Breitenauer See
destination:colour: ;;;brown
foot: no
hazard: traffic_signals
highway: primary_link
lanes: 1
lit: no
maxspeed: 50
oneway: yes
ref: B 39;B 39A
sidewalk: no
surface: asphalt
turn: right
output:
- layer: streets
geometry: line
min_zoom: 8
allow_extra_tags: false
tags:
bridge: false
kind: primary
link: true
rail: false
surface: asphalt
tunnel: false
- layer: street_labels
geometry: line
min_zoom: 13
allow_extra_tags: false
tags:
kind: primary
ref: "B 39\nB 39A"
ref_rows: 2
ref_cols: 5
- name: rail with service
input:
source: osm
geometry: line
tags:
railway: rail
service: service_value
output:
layer: streets
geometry: line
min_zoom: 8
allow_extra_tags: false
tags:
bridge: false
kind: rail
link: false
rail: true
tunnel: false
service: service_value
- name: narrow_gauge without service
input:
source: osm
geometry: line
tags:
railway: narrow_gauge
output:
layer: streets
geometry: line
min_zoom: 10
allow_extra_tags: false
tags:
bridge: false
kind: narrow_gauge
link: false
rail: true
tunnel: false
- name: 'track with grade'
input:
source: osm
geometry: line
tags:
highway: track
tracktype: grade2
output:
layer: streets
geometry: line
min_zoom: 13
at_zoom: 11
allow_extra_tags: false
tags:
bridge: false
kind: track
link: false
rail: false
tracktype: grade2
tunnel: false
- name: 'named path'
input:
source: osm
geometry: line
tags:
highway: path
name: Name
name:en: English Name
name:de: German Name
output:
- layer: streets
geometry: line
tags:
kind: path
- layer: street_labels
geometry: line
tags:
kind: path
name: Name
name_en: English Name
name_de: German Name
- name: 'motorway attributes drop below z11'
input:
source: osm
geometry: line
tags:
highway: motorway
tracktype: grade2
output:
layer: streets
geometry: line
min_zoom: 5
at_zoom: 10
allow_extra_tags: false
tags:
kind: motorway
- name: 'rail attributes drop below z11'
input:
source: osm
geometry: line
tags:
railway: rail
service: primary
output:
layer: streets
geometry: line
min_zoom: 8
at_zoom: 10
allow_extra_tags: false
tags:
kind: rail
- name: 'path bridge'
input:
source: osm
geometry: line
tags:
highway: path
bridge: yes
output:
layer: streets
geometry: line
min_zoom: 13
tags:
bridge: true
kind: path
tunnel: false
- name: 'pedestrian tunnel'
input:
source: osm
geometry: line
tags:
highway: pedestrian
tunnel: yes
output:
layer: streets
geometry: line
tags:
bridge: false
kind: pedestrian
tunnel: true
- name: 'horse'
input:
source: osm
geometry: line
tags:
highway: track
horse: definitely
output:
layer: streets
geometry: line
tags:
kind: track
horse: definitely
- name: 'bicycle'
input:
source: osm
geometry: line
tags:
highway: track
bicycle: definitely
output:
layer: streets
geometry: line
tags:
kind: track
bicycle: definitely
- name: 'aeroway=taxiway'
input:
source: osm
geometry: line
tags:
aeroway: taxiway
ref: N
output:
layer: streets
geometry: line
min_zoom: 13
tags:
kind: taxiway
- name: 'aeroway=runway'
input:
source: osm
geometry: line
tags:
aeroway: runway
ref: 07/25
surface: concrete:lanes
output:
layer: streets
geometry: line
min_zoom: 11
tags:
kind: runway
surface: concrete:lanes
- name: 'pedestrian polygon'
input:
source: osm
geometry: polygon
tags:
area: yes
highway: pedestrian
lit: yes
surface: paving_stones
name: 'Name'
name:en: 'Name (en)'
output:
- layer: street_polygons
geometry: polygon
min_zoom: 14
allow_extra_tags: false
tags:
bridge: false
kind: pedestrian
rail: false
surface: paving_stones
tunnel: false
- layer: street_polygons_labels
geometry: point
min_zoom: 14
allow_extra_tags: false
tags:
kind: pedestrian
name: 'Name'
name_en: 'Name (en)'
- name: 'ignore pedestrian polygon without area=yes'
input:
source: osm
geometry: polygon
tags:
highway: pedestrian
name: 'Name'
name:en: 'Name (en)'
output: [ ]
- name: 'pedestrian polygon bridge'
input:
source: osm
geometry: polygon
tags:
area: yes
highway: service
bridge: yes
output:
layer: street_polygons
geometry: polygon
min_zoom: 14
tags:
kind: service
bridge: true
- name: 'motorway junction'
input:
source: osm
geometry: point
tags:
highway: motorway_junction
name: 'Name'
ref: 'ref'
output:
layer: street_labels_points
geometry: point
min_zoom: 12
tags:
kind: motorway_junction
name: 'Name'
ref: 'ref'
- name: 'gondola'
input:
source: osm
geometry: line
tags:
aerialway: gondola
name: 'Name'
output:
layer: aerialways
geometry: line
min_zoom: 12
tags:
kind: gondola
- name: 'train station point'
input:
source: osm
geometry: point
tags:
railway: station
name: 'Name'
output:
layer: public_transport
geometry: point
allow_extra_tags: false
tags:
kind: station
name: 'Name'
- name: 'airport polygon'
input:
source: osm
geometry: polygon
tags:
aeroway: aerodrome
name: 'Name'
iata: eye eight a
output:
layer: public_transport
geometry: point
allow_extra_tags: false
tags:
kind: aerodrome
name: 'Name'
iata: eye eight a
- name: 'unnamed hamlet'
input:
source: osm
geometry: point
tags:
place: hamlet
output: [ ]
- name: 'boundary_labels'
input:
source: admin_points
geometry: point
tags:
WAY_AREA: 10.5
ADMIN_LEVEL: 2
NAME: Name
NAME_EN: ''
NAME_DE: Name (de)
output:
layer: boundary_labels
geometry: point
allow_extra_tags: false
min_zoom: 5
tags:
way_area: 10.5
admin_level: 2
name: Name
name_de: Name (de)
- name: 'boundary_labels z4'
input:
source: admin_points
geometry: point
tags:
WAY_AREA: 1e7
ADMIN_LEVEL: 2
NAME: name
output:
layer: boundary_labels
geometry: point
min_zoom: 4
tags:
way_area: 1e7
admin_level: 2
- name: 'boundary_labels z3'
input:
source: admin_points
geometry: point
tags:
WAY_AREA: 7e7
ADMIN_LEVEL: 2
NAME: name
output:
layer: boundary_labels
geometry: point
min_zoom: 3
tags:
way_area: 7e7
admin_level: 2
- name: 'boundary_labels z2'
input:
source: admin_points
geometry: point
tags:
WAY_AREA: 2e8
ADMIN_LEVEL: '2'
NAME: name
output:
layer: boundary_labels
geometry: point
min_zoom: 2
tags:
way_area: 2e8
admin_level: 2
- name: 'boundary_labels admin_level=4 z3'
input:
source: admin_points
geometry: point
tags:
WAY_AREA: 7e7
ADMIN_LEVEL: 4
NAME: name
output:
layer: boundary_labels
geometry: point
min_zoom: 3
tags:
way_area: 7e7
admin_level: 4
- name: 'country boundary'
input:
source: osm
geometry: line
# TODO from relation
tags:
boundary: administrative
admin_level: '2'
maritime: yes
output:
layer: boundaries
geometry: line
min_zoom: 0
tags:
maritime: true
admin_level: 2
- name: 'state boundary'
input:
source: osm
geometry: line
# TODO from relation
tags:
boundary: administrative
admin_level: 4
output:
layer: boundaries
geometry: line
min_zoom: 7
tags:
maritime: false
admin_level: 4
# TODO take min admin level
- name: 'hamlet'
input:
source: osm
geometry: point
tags:
place: hamlet
name: 'Name'
output:
layer: place_labels
geometry: point
allow_extra_tags: false
min_zoom: 10
tags:
kind: hamlet
name: 'Name'
population: 50
- name: 'city with population'
input:
source: osm
geometry: point
tags:
place: city
name: 'Name'
population: '1300'
output:
layer: place_labels
geometry: point
allow_extra_tags: false
min_zoom: 6
tags:
kind: city
name: 'Name'
population: 1300
- name: 'state capital'
input:
source: osm
geometry: point
tags:
place: city
capital: '4'
name: 'Name'
output:
layer: place_labels
geometry: point
allow_extra_tags: false
min_zoom: 4
tags:
kind: state_capital
name: 'Name'
population: 100000
- name: 'capital'
input:
source: osm
geometry: point
tags:
place: city
capital: yes
name: 'Name'
output:
layer: place_labels
geometry: point
allow_extra_tags: false
min_zoom: 4
tags:
kind: capital
name: 'Name'
population: 100000
- name: 'population with comma'
input:
source: osm
geometry: point
tags:
place: city
population: 123,123
name: 'Name'
output:
layer: place_labels
geometry: point
tags:
kind: city
population: 123123

631
planetiler-custommap/src/main/resources/samples/shortbread.yml 100644
Wyświetl plik

@ -0,0 +1,631 @@
schema_name: Shortbread
schema_description: A basic, lean, general-purpose vector tile schema for OpenStreetMap data. See https://shortbread.geofabrik.de/
attribution: <a href="https://www.openstreetmap.org/copyright" target="_blank">&copy; OpenStreetMap contributors</a>
examples: shortbread.spec.yml
sources:
ocean:
type: shapefile
url: https://osmdata.openstreetmap.de/download/water-polygons-split-3857.zip
admin_points:
type: shapefile
url: https://shortbread.geofabrik.de/shapefiles/admin-points-4326.zip
osm:
type: osm
url: geofabrik:massachusetts
definitions:
# TODO let attribute definitions set multiple keys so you can just use `- *names`
attributes:
- &name
key: name
- &name_en
key: name_en
tag_value: name:en
- &name_de
key: name_de
tag_value: name:de
layers:
# Water
- id: ocean
features:
- source: ocean
geometry: polygon
- id: water_polygons
features:
- source: osm
geometry: polygon
min_zoom:
default_value: 4
overrides: &water_zoom_overrides
10:
waterway: [ dock, canal ]
include_when: &water_filter
natural:
- glacier
- water
waterway:
- riverbank
- dock
- canal
landuse:
- reservoir
- basin
attributes:
- key: kind
type: match_value
- id: water_polygons_labels
features:
- source: osm
geometry: polygon_centroid
min_zoom:
default_value: 14
overrides: *water_zoom_overrides
include_when: *water_filter
exclude_when:
name: ''
attributes:
- key: kind
type: match_value
- *name
- *name_en
- *name_de
- id: water_lines
features:
- source: osm
geometry: line
min_zoom:
default_value: 9
overrides:
14:
waterway: [ stream, ditch ]
# TODO rivers and canals min length=0.25px
include_when:
waterway:
- canal
- river
- stream
- ditch
attributes:
- key: kind
type: match_value
- id: water_line_labels
features:
- source: osm
geometry: line
min_zoom:
default_value: 12
overrides:
14:
waterway: [ stream, ditch ]
# TODO rivers and canals min length=0.25px
include_when:
waterway:
- canal
- river
- stream
- ditch
exclude_when:
name: ''
attributes:
- key: kind
type: match_value
- *name
- *name_en
- *name_de
## Countries, States, Cities
- id: boundaries
features:
- source: osm
geometry: line
# TODO get min admin level from relations
min_zoom:
default_value: 7
overrides:
0:
admin_level: 2
include_when:
__all__:
- boundary: administrative
- admin_level: [ 2, 4 ]
attributes:
- key: maritime
type: boolean
- key: admin_level
type: integer
- id: boundary_labels
features:
- source: admin_points
geometry: point
min_zoom:
default_value: 5
overrides:
2: '${ feature.tags.has("ADMIN_LEVEL", "2") && double(feature.tags.WAY_AREA) >= 2e8 }'
3: '${ double(feature.tags.WAY_AREA) >= 7e7 }'
4: '${ double(feature.tags.WAY_AREA) >= 1e7 }'
# TODO sort by WAY_AREA descending
attributes:
- key: way_area
tag_value: WAY_AREA
type: double
- key: admin_level
tag_value: ADMIN_LEVEL
type: integer
- key: name
tag_value: NAME
- key: name_en
tag_value: NAME_EN
- key: name_de
tag_value: NAME_DE
- id: place_labels
features:
- source: osm
geometry: point
include_when:
place:
- city
- town
- village
- hamlet
- suburb
- neighbourhood
- isolated_dwelling
- farm
- island
- locality
exclude_when:
name: ''
min_zoom:
default_value: 10
overrides:
4:
__all__:
place: [ city, town, village, hamlet ]
capital: [ yes, '4' ]
6:
__all__:
place: city
__not__:
capital: [ yes, '4' ]
7:
__all__:
place: town
__not__:
capital: [ yes, '4' ]
# TODO z-order
attributes:
- key: kind
value:
default_value: '${ match_value }'
overrides:
capital:
__all__:
place: [ city, town, village, hamlet ]
capital: yes
state_capital:
__all__:
place: [ city, town, village, hamlet ]
capital: '4'
- *name
- *name_en
- *name_de
- key: population
type: integer
value:
match:
- value: '${ feature.tags.get("population") }'
if: { population: __any__ }
- value: 100000
if: { place: city }
- value: 5000
if: { place: town }
- value: 1000
if: { place: suburb }
- value: 100
if: { place: [ village, neighborhood ] }
- value: 50
if: { place: hamlet }
- value: 5
if: { place: [ isolated_dwelling, farm ] }
- else: 0
# Land Use, Land Cover, Buildings
- id: land
features:
- source: osm
geometry: polygon
include_when:
amenity:
- grave_yard
landuse:
- allotments
- brownfield
- cemetery
- commercial
- farmland
- farmyard
- forest
- grass
- greenfield
- greenhouse_horticulture
- industrial
- landfill
- meadow
- orchard
- plant_nursery
- quarry
- railway
- recreation_ground
- residential
- retail
- village_green
- vineyard
leisure:
- garden
- golf_course
- miniature_golf
- park
- playground
natural:
- bare_rock
- beach
- grassland
- heath
- sand
- scree
- scrub
- shingle
- wood
wetland:
- bog
- marsh
- string_bog
- swamp
- wet_meadow
min_zoom:
default_value: 11
overrides:
7:
natural: wood
landuse: forest
10:
landuse:
- brownfield
- commercial
- farmland
- farmyard
- greenfield
- industrial
- landfill
- railway
- residential
- retail
natural:
- beach
- sand
13:
amenity: grave_yard
natural: wood
landuse: cemetery
attributes:
- key: kind
value: '${match_value == "forest" ? "wood": match_value}'
- id: sites
features:
- source: osm
geometry: polygon
min_zoom: 14
include_when:
military: danger_area
leisure: sports_center
landuse: construction
amenity:
- university
- hospital
- prison
- parking
- bicycle_parking
attributes:
- key: kind
type: match_value
- id: buildings
features:
- source: osm
geometry: polygon
min_zoom: 14
include_when:
building: __any__
exclude_when:
building: no
- id: addresses
features:
- source: osm
geometry: polygon_centroid_if_convex
min_zoom: 14
include_when: &address_filter
addr:housenumber: __any__
addr:housename: __any__
attributes: &address_attributes
- key: name
tag_value: addr:housename
- key: number
tag_value: addr:housenumber
- source: osm
geometry: point
min_zoom: 14
include_when: *address_filter
attributes: *address_attributes
## Streets and Transport
- id: streets
features:
- source: osm
geometry: line
min_zoom:
default_value: 13
overrides:
5:
highway: motorway%
6:
highway: trunk%
8:
highway: primary%
__all__:
railway: [ rail, narrow_gauge ]
service: __any__
9:
highway: secondary%
10:
__all__:
railway: [ rail, narrow_gauge ]
service: ''
railway:
- funicular
- light_rail
- monorail
- subway
- tram
highway: tertiary
11:
aeroway: runway
12:
highway: [ residential, unclassified ]
# TODO min_tile_cover_size: 0
# TODO z-order
include_when:
highway:
- motorway
- motorway_link
- trunk
- trunk_link
- primary
- primary_link
- secondary
- secondary_link
- tertiary
- tertiary_link
- unclassified
- residential
- living_street
- service
- pedestrian
- track
- footway
- steps
- path
- cycleway
aeroway: # TODO update shortbread spec
- runway
- taxiway
railway:
- rail
- narrow_gauge
- tram
- light_rail
- funicular
- subway
- monorail
attributes:
- key: kind
value: '${ match_value.replace("_link", "") }'
- key: link
min_zoom: 11
value: true
include_when:
highway: '%_link'
else: false
- key: rail
min_zoom: 11
value: true
include_when:
railway: __any__
else: false
- &tunnel_attr
key: tunnel
min_zoom: 11
value: true
include_when:
tunnel: [ yes, building_passage ]
covered: yes
else: false
- &bridge_attr
key: bridge
min_zoom: 11
value: true
include_when:
bridge: yes
else: false
- key: tracktype
min_zoom: 11
- key: surface # TODO canonicalize?
min_zoom: 11
- key: service
min_zoom: 11
- key: bicycle
min_zoom: 14
- key: horse
min_zoom: 14
- id: street_labels
features:
- source: osm
geometry: line
min_zoom:
default_value: 14
overrides:
10:
highway: motorway
12:
highway: [ trunk, primary ]
13:
highway:
- motorway_link
- trunk_link
- primary_link
- secondary
- secondary_link
- tertiary
include_when:
highway:
- motorway
- motorway_link
- trunk
- trunk_link
- primary
- primary_link
- secondary
- secondary_link
- tertiary
- tertiary_link
- unclassified
- residential
- living_street
- service
- pedestrian
- track
- footway
- steps
- path
- cycleway
exclude_when:
__all__:
name: ''
ref: ''
attributes:
- key: kind
value: '${ match_value.replace("_link", "") }'
- *name
- *name_en
- *name_de
# TODO use ref var to avoid duplicating logic
- key: ref
exclude_when: &missing_ref
ref: ''
value: '${ feature.tags["ref"].replace(";", "\n") }'
- key: ref_cols
value: '${ max(feature.tags["ref"].split(";").map(r, size(r))) }'
exclude_when: *missing_ref
- key: ref_rows
value: '${ size(feature.tags["ref"].split(";")) }'
exclude_when: *missing_ref
- id: street_polygons
features:
- source: osm
geometry: polygon
min_zoom: 14
include_when:
__all__:
- highway: [ pedestrian, service ]
- area: yes
attributes:
- key: kind
type: match_value
- *bridge_attr
- *tunnel_attr
- key: surface
- key: rail
value: false # TODO omit?
- id: street_polygons_labels
features:
- source: osm
geometry: polygon_point_on_surface
min_zoom: 14
include_when:
__all__:
highway: [ pedestrian, service ]
area: yes
name: __any__
attributes:
- key: kind
type: match_value
- *name
- *name_en
- *name_de
- id: street_labels_points # TODO update documentation (streetS_labels_points)
features:
- source: osm
geometry: point
min_zoom: 12
include_when:
highway: motorway_junction
attributes:
- key: kind
type: match_value
- key: ref
- *name
- *name_en
- *name_de
- id: aerialways
features:
- source: osm
geometry: line
min_zoom: 12
include_when:
aerialway: __any__
attributes:
- key: kind
type: match_value
- id: public_transport
features:
- source: osm
geometry: point
min_zoom: &public_transport_zoom
default_value: 14
overrides:
11:
aeroway: aerodrome
13:
railway: [ station, halt ]
aerialway: station
include_when: &public_transport_filter
railway: [ station, halt, tram_stop ]
aeroway: aerodrome
aerialway: station
attributes: &public_transport_attrs
- key: kind
type: match_value
- *name
- *name_en
- *name_de
- key: iata
- source: osm
geometry: polygon_point_on_surface
min_zoom: *public_transport_zoom
include_when: *public_transport_filter
attributes: *public_transport_attrs

275
planetiler-custommap/src/test/java/com/onthegomap/planetiler/custommap/BooleanExpressionParserTest.java 100644
Wyświetl plik

@ -0,0 +1,275 @@
package com.onthegomap.planetiler.custommap;
import static com.onthegomap.planetiler.expression.Expression.*;
import static org.junit.jupiter.api.Assertions.assertEquals;
import com.onthegomap.planetiler.custommap.expression.ScriptEnvironment;
import com.onthegomap.planetiler.expression.Expression;
import java.util.Map;
import org.junit.jupiter.api.Test;
class BooleanExpressionParserTest {
private static final TagValueProducer TVP = new TagValueProducer(Map.of());
private static void assertParse(String yaml, Expression parsed) {
Object expression = YAML.load(yaml, Object.class);
var actual = BooleanExpressionParser.parse(expression, TVP, ScriptEnvironment.root());
assertEquals(
parsed.simplify().generateJavaCode(),
actual.simplify().generateJavaCode()
);
assertEquals(
parsed.simplify(),
actual.simplify()
);
}
@Test
void testEmpty() {
assertParse("""
""",
Expression.FALSE);
}
@Test
void testSingleValue() {
assertParse("""
a: b
""",
matchAny("a", "b")
);
}
@Test
void testMultivalue() {
assertParse("""
a:
- b
- c
""",
or(matchAny("a", "b", "c"))
);
}
@Test
void testMultiKey() {
assertParse("""
a: b
c: [d, e]
""",
or(
matchAny("a", "b"),
matchAny("c", "d", "e")
)
);
}
@Test
void testAnyValue() {
assertParse("""
a: __any__
""",
matchField("a")
);
assertParse("""
a: __ANY__
""",
matchField("a")
);
assertParse("""
a: [b, __any__]
""",
matchField("a")
);
}
@Test
void testEscapeAny() {
assertParse("""
a: \\__any__
b: [\\__any__]
""",
or(
matchAny("a", "__any__"),
matchAny("b", "__any__")
)
);
assertParse("""
a: \\__ANY__
""",
matchAny("a", "__ANY__")
);
assertParse("""
a: \\\\__any__
""",
matchAny("a", "\\__any__")
);
assertParse("""
a: \\\\__ANY__
""",
matchAny("a", "\\__ANY__")
);
}
@Test
void testMatchAnything() {
assertParse("__any__", Expression.TRUE);
}
@Test
void testAnyWrapper() {
assertParse("""
__any__:
a: b
c: d
""",
or(
matchAny("a", "b"),
matchAny("c", "d")
)
);
}
@Test
void testAllWrapper() {
assertParse("""
__all__:
a: b
c: d
""",
and(
matchAny("a", "b"),
matchAny("c", "d")
)
);
}
@Test
void testNestedNot() {
assertParse("""
__all__:
a: b
__not__:
c: d
""",
and(
matchAny("a", "b"),
not(
matchAny("c", "d")
)
)
);
}
@Test
void testNestedAnd() {
assertParse("""
a: b
__ALL__:
c: d
e: f
""",
or(
matchAny("a", "b"),
and(
matchAny("c", "d"),
matchAny("e", "f")
)
)
);
}
@Test
void testNestedAndOrNot() {
assertParse("""
a: b
__ALL__:
c: d
__NOT__:
e: f
g: h
""",
or(
matchAny("a", "b"),
and(
matchAny("c", "d"),
not(or(
matchAny("e", "f"),
matchAny("g", "h")
))
)
)
);
}
@Test
void testActualAnyAllUnescaped() {
assertParse("""
a: b
__any__: d
__all__: d
""",
or(
matchAny("a", "b"),
matchAny("__any__", "d"),
matchAny("__all__", "d")
)
);
}
@Test
void testActualAnyAll() {
assertParse("""
a: b
\\__any__: d
\\__all__: d
""",
or(
matchAny("a", "b"),
matchAny("__any__", "d"),
matchAny("__all__", "d")
)
);
}
@Test
void testActualAnyAllList() {
assertParse("""
a: b
\\__any__: [d1, d2]
\\__all__: [e1, e2]
""",
or(
matchAny("a", "b"),
matchAny("__any__", "d1", "d2"),
matchAny("__all__", "e1", "e2")
)
);
}
@Test
void testList() {
assertParse("""
__all__:
- a: b
- __any__:
- c: d
- e: f
- __any__:
- g: h
- j: i
""",
and(
matchAny("a", "b"),
or(
matchAny("c", "d"),
matchAny("e", "f")
),
or(
matchAny("g", "h"),
matchAny("j", "i")
)
)
);
}
}

167
planetiler-custommap/src/test/java/com/onthegomap/planetiler/custommap/ConfigExpressionParserTest.java 100644
Wyświetl plik

@ -0,0 +1,167 @@
package com.onthegomap.planetiler.custommap;
import static com.onthegomap.planetiler.custommap.expression.ConfigExpression.*;
import static org.junit.jupiter.api.Assertions.assertEquals;
import com.onthegomap.planetiler.custommap.expression.ConfigExpression;
import com.onthegomap.planetiler.expression.DataType;
import com.onthegomap.planetiler.expression.Expression;
import com.onthegomap.planetiler.expression.MultiExpression;
import java.util.List;
import java.util.Map;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
class ConfigExpressionParserTest {
private static final TagValueProducer TVP = new TagValueProducer(Map.of());
private static final ConfigExpression.Signature<Contexts.ProcessFeature, Object> FEATURE_SIGNATURE =
signature(Contexts.ProcessFeature.DESCRIPTION, Object.class);
private static <O> void assertParse(String yaml, ConfigExpression<?, ?> parsed, Class<O> clazz) {
Object expression = YAML.load(yaml, Object.class);
var actual = ConfigExpressionParser.parse(expression, TVP, FEATURE_SIGNATURE.in(), clazz);
assertEquals(
parsed.simplify(),
actual.simplify()
);
}
@Test
void testEmpty() {
assertParse("", constOf(null), String.class);
}
@ParameterizedTest
@ValueSource(strings = {"1", "'1'"})
void testConst(String input) {
assertParse(input, constOf(1), Integer.class);
assertParse(input, constOf(1L), Long.class);
assertParse(input, constOf(1d), Double.class);
}
@Test
void testVar() {
assertParse("${feature.id}", variable(FEATURE_SIGNATURE.withOutput(Integer.class), "feature.id"), Integer.class);
assertParse("${feature.id}", variable(FEATURE_SIGNATURE.withOutput(Long.class), "feature.id"), Long.class);
assertParse("${feature.id}", variable(FEATURE_SIGNATURE.withOutput(Double.class), "feature.id"), Double.class);
}
@Test
void testStaticExpression() {
assertParse("${1+2}", constOf(3), Integer.class);
assertParse("${1+2}", constOf(3L), Long.class);
}
@Test
void testDynamicExpression() {
assertParse("${feature.tags.a}", script(FEATURE_SIGNATURE, "feature.tags.a"), Object.class);
}
@Test
void testCoalesceStatic() {
assertParse("""
coalesce:
- 1
- 2
""", constOf(1), Integer.class);
}
@Test
void testCoalesceDynamic() {
assertParse("""
coalesce:
- ${feature.tags.get('a')}
- ${feature.tags.get('b')}
""", coalesce(List.of(
script(FEATURE_SIGNATURE, "feature.tags.get('a')"),
script(FEATURE_SIGNATURE, "feature.tags.get('b')")
)), Object.class);
}
@Test
void testMatch() {
assertParse("""
match:
- if:
natural: water
value: 1
- if:
natural: lake
value: 2
- else: 3
""", match(FEATURE_SIGNATURE.withOutput(Integer.class), MultiExpression.of(List.of(
MultiExpression.entry(constOf(1), Expression.matchAny("natural", "water")),
MultiExpression.entry(constOf(2), Expression.matchAny("natural", "lake"))
)), constOf(3)), Integer.class);
}
@Test
void testMatchMap() {
assertParse("""
match:
1:
natural: water
2:
natural: lake
3: default_value
""", match(FEATURE_SIGNATURE.withOutput(Integer.class), MultiExpression.of(List.of(
MultiExpression.entry(constOf(1), Expression.matchAny("natural", "water")),
MultiExpression.entry(constOf(2), Expression.matchAny("natural", "lake"))
)), constOf(3)), Integer.class);
}
@Test
void testOverrides() {
assertParse("""
default_value: 3
overrides:
- if: {natural: water}
value: 1
- if: {natural: lake}
value: 2
""", match(FEATURE_SIGNATURE.withOutput(Integer.class), MultiExpression.of(List.of(
MultiExpression.entry(constOf(1), Expression.matchAny("natural", "water")),
MultiExpression.entry(constOf(2), Expression.matchAny("natural", "lake"))
)), constOf(3)), Integer.class);
}
@Test
void testCast() {
assertParse("""
tag_value: abc
type: integer
""",
cast(
FEATURE_SIGNATURE.withOutput(Integer.class),
getTag(FEATURE_SIGNATURE.withOutput(Object.class), constOf("abc")),
DataType.GET_INT
),
Integer.class
);
}
@Test
void testCoalesceWithType() {
assertParse("""
type: double
coalesce:
- '1'
- '2'
""", constOf(1d), Double.class);
}
@Test
void testCastAValue() {
assertParse("""
type: double
value: '${feature.tags.a}'
""",
cast(
FEATURE_SIGNATURE.withOutput(Double.class),
script(FEATURE_SIGNATURE.withOutput(Object.class), "feature.tags.a"),
DataType.GET_DOUBLE
),
Double.class);
}
}

600
planetiler-custommap/src/test/java/com/onthegomap/planetiler/custommap/ConfiguredFeatureTest.java
Wyświetl plik

@ -1,6 +1,7 @@
package com.onthegomap.planetiler.custommap;
import static com.onthegomap.planetiler.TestUtils.newLineString;
import static com.onthegomap.planetiler.TestUtils.newPoint;
import static com.onthegomap.planetiler.TestUtils.newPolygon;
import static java.util.Collections.emptyList;
import static org.junit.jupiter.api.Assertions.*;
@ -8,21 +9,21 @@ import static org.junit.jupiter.api.Assertions.*;
import com.onthegomap.planetiler.FeatureCollector;
import com.onthegomap.planetiler.FeatureCollector.Feature;
import com.onthegomap.planetiler.Profile;
import com.onthegomap.planetiler.TestUtils;
import com.onthegomap.planetiler.config.PlanetilerConfig;
import com.onthegomap.planetiler.custommap.configschema.SchemaConfig;
import com.onthegomap.planetiler.custommap.util.TestConfigurableUtils;
import com.onthegomap.planetiler.reader.SimpleFeature;
import com.onthegomap.planetiler.reader.SourceFeature;
import com.onthegomap.planetiler.stats.Stats;
import java.io.IOException;
import java.nio.file.Path;
import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.function.Consumer;
import java.util.function.Function;
import java.util.function.Supplier;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.CsvSource;
import org.junit.jupiter.params.provider.ValueSource;
class ConfiguredFeatureTest {
@ -37,65 +38,69 @@ class ConfiguredFeatureTest {
"test_zoom_tag", "test_zoom_value"
);
private static Map<String, Object> motorwayTags = Map.of(
private static final Map<String, Object> motorwayTags = Map.of(
"highway", "motorway",
"layer", "1",
"bridge", "yes",
"tunnel", "yes"
);
private static Map<String, Object> trunkTags = Map.of(
private static final Map<String, Object> trunkTags = Map.of(
"highway", "trunk",
"toll", "yes"
);
private static Map<String, Object> primaryTags = Map.of(
private static final Map<String, Object> primaryTags = Map.of(
"highway", "primary",
"lanes", "2"
);
private static Map<String, Object> highwayAreaTags = Map.of(
private static final Map<String, Object> highwayAreaTags = Map.of(
"area:highway", "motorway",
"layer", "1",
"bridge", "yes",
"surface", "asphalt"
);
private static Map<String, Object> inputMappingTags = Map.of(
private static final Map<String, Object> inputMappingTags = Map.of(
"s_type", "string_val",
"l_type", "1",
"i_type", "1",
"double_type", "1.5",
"b_type", "yes",
"d_type", "yes",
"intermittent", "yes",
"bridge", "yes"
);
private static FeatureCollector polygonFeatureCollector() {
var config = PlanetilerConfig.defaults();
var factory = new FeatureCollector.Factory(config, Stats.inMemory());
return factory.get(SimpleFeature.create(TestUtils.newPolygon(0, 0, 0.1, 0, 0.1, 0.1, 0, 0), new HashMap<>()));
}
private static FeatureCollector linestringFeatureCollector() {
var config = PlanetilerConfig.defaults();
var factory = new FeatureCollector.Factory(config, Stats.inMemory());
return factory.get(SimpleFeature.create(TestUtils.newLineString(0, 0, 0.1, 0, 0.1, 0.1, 0, 0), new HashMap<>()));
}
private static Profile loadConfig(Function<String, Path> pathFunction, String filename) throws IOException {
private static Profile loadConfig(Function<String, Path> pathFunction, String filename) {
var staticAttributeConfig = pathFunction.apply(filename);
var schema = ConfiguredMapMain.loadConfig(staticAttributeConfig);
var schema = SchemaConfig.load(staticAttributeConfig);
return new ConfiguredProfile(schema);
}
private static Profile loadConfig(String config) {
var schema = SchemaConfig.load(config);
return new ConfiguredProfile(schema);
}
private static void testFeature(Function<String, Path> pathFunction, String schemaFilename, SourceFeature sf,
Supplier<FeatureCollector> fcFactory,
Consumer<Feature> test, int expectedMatchCount)
throws Exception {
Consumer<Feature> test, int expectedMatchCount) {
var profile = loadConfig(pathFunction, schemaFilename);
var fc = fcFactory.get();
testFeature(sf, test, expectedMatchCount, profile);
}
private static void testFeature(String config, SourceFeature sf, Consumer<Feature> test, int expectedMatchCount) {
var profile = loadConfig(config);
testFeature(sf, test, expectedMatchCount, profile);
}
private static void testFeature(SourceFeature sf, Consumer<Feature> test, int expectedMatchCount, Profile profile) {
var config = PlanetilerConfig.defaults();
var factory = new FeatureCollector.Factory(config, Stats.inMemory());
var fc = factory.get(sf);
profile.processFeature(sf, fc);
@ -109,26 +114,44 @@ class ConfiguredFeatureTest {
assertEquals(expectedMatchCount, length.get(), "Wrong number of features generated");
}
private static void testPolygon(Function<String, Path> pathFunction, String schemaFilename, Map<String, Object> tags,
Consumer<Feature> test, int expectedMatchCount)
throws Exception {
private static void testPolygon(String config, Map<String, Object> tags,
Consumer<Feature> test, int expectedMatchCount) {
var sf =
SimpleFeature.createFakeOsmFeature(newPolygon(0, 0, 1, 0, 1, 1, 0, 0), tags, "osm", null, 1, emptyList());
testFeature(pathFunction, schemaFilename, sf,
ConfiguredFeatureTest::polygonFeatureCollector, test, expectedMatchCount);
testFeature(config, sf, test, expectedMatchCount);
}
private static void testPoint(String config, Map<String, Object> tags,
Consumer<Feature> test, int expectedMatchCount) {
var sf =
SimpleFeature.createFakeOsmFeature(newPoint(0, 0), tags, "osm", null, 1, emptyList());
testFeature(config, sf, test, expectedMatchCount);
}
private static void testLinestring(String config,
Map<String, Object> tags, Consumer<Feature> test, int expectedMatchCount) {
var sf =
SimpleFeature.createFakeOsmFeature(newLineString(0, 0, 1, 0, 1, 1), tags, "osm", null, 1, emptyList());
testFeature(config, sf, test, expectedMatchCount);
}
private static void testPolygon(Function<String, Path> pathFunction, String schemaFilename, Map<String, Object> tags,
Consumer<Feature> test, int expectedMatchCount) {
var sf =
SimpleFeature.createFakeOsmFeature(newPolygon(0, 0, 1, 0, 1, 1, 0, 0), tags, "osm", null, 1, emptyList());
testFeature(pathFunction, schemaFilename, sf, test, expectedMatchCount);
}
private static void testLinestring(Function<String, Path> pathFunction, String schemaFilename,
Map<String, Object> tags, Consumer<Feature> test, int expectedMatchCount)
throws Exception {
Map<String, Object> tags, Consumer<Feature> test, int expectedMatchCount) {
var sf =
SimpleFeature.createFakeOsmFeature(newLineString(0, 0, 1, 0, 1, 1), tags, "osm", null, 1, emptyList());
testFeature(pathFunction, schemaFilename, sf,
ConfiguredFeatureTest::linestringFeatureCollector, test, expectedMatchCount);
testFeature(pathFunction, schemaFilename, sf, test, expectedMatchCount);
}
@Test
void testStaticAttributeTest() throws Exception {
void testStaticAttributeTest() {
testPolygon(TEST_RESOURCE, "static_attribute.yml", waterTags, f -> {
var attr = f.getAttrsAtZoom(14);
assertEquals("aTestConstantValue", attr.get("natural"));
@ -136,7 +159,7 @@ class ConfiguredFeatureTest {
}
@Test
void testTagValueAttributeTest() throws Exception {
void testTagValueAttributeTest() {
testPolygon(TEST_RESOURCE, "tag_attribute.yml", waterTags, f -> {
var attr = f.getAttrsAtZoom(14);
assertEquals("water", attr.get("natural"));
@ -144,7 +167,7 @@ class ConfiguredFeatureTest {
}
@Test
void testTagIncludeAttributeTest() throws Exception {
void testTagIncludeAttributeTest() {
testPolygon(TEST_RESOURCE, "tag_include.yml", waterTags, f -> {
var attr = f.getAttrsAtZoom(14);
assertEquals("ok", attr.get("test_include"));
@ -153,7 +176,7 @@ class ConfiguredFeatureTest {
}
@Test
void testZoomAttributeTest() throws Exception {
void testZoomAttributeTest() {
testPolygon(TEST_RESOURCE, "tag_include.yml", waterTags, f -> {
var attr = f.getAttrsAtZoom(14);
assertEquals("test_zoom_value", attr.get("test_zoom_tag"));
@ -167,7 +190,7 @@ class ConfiguredFeatureTest {
}
@Test
void testTagHighwayLinestringTest() throws Exception {
void testTagHighwayLinestringTest() {
testLinestring(TEST_RESOURCE, "road_motorway.yml", motorwayTags, f -> {
var attr = f.getAttrsAtZoom(14);
assertEquals("motorway", attr.get("highway"));
@ -175,7 +198,7 @@ class ConfiguredFeatureTest {
}
@Test
void testTagTypeConversionTest() throws Exception {
void testTagTypeConversionTest() {
testLinestring(TEST_RESOURCE, "road_motorway.yml", motorwayTags, f -> {
var attr = f.getAttrsAtZoom(14);
@ -190,7 +213,7 @@ class ConfiguredFeatureTest {
}
@Test
void testZoomFilterAttributeTest() throws Exception {
void testZoomFilterAttributeTest() {
testLinestring(TEST_RESOURCE, "road_motorway.yml", motorwayTags, f -> {
var attr = f.getAttrsAtZoom(14);
assertTrue(attr.containsKey("bridge"), "Produce attribute bridge at z14");
@ -201,7 +224,7 @@ class ConfiguredFeatureTest {
}
@Test
void testZoomFilterConditionalTest() throws Exception {
void testZoomFilterConditionalTest() {
testLinestring(TEST_RESOURCE, "zoom_filter.yml", motorwayTags, f -> {
var attr = f.getAttrsAtZoom(4);
assertEquals("motorway", attr.get("highway"), "Produce attribute highway at z4");
@ -236,7 +259,7 @@ class ConfiguredFeatureTest {
}
@Test
void testAllValuesInKey() throws Exception {
void testAllValuesInKey() {
//Show that a key in includeWhen with no values matches all values
testPolygon(SAMPLE_RESOURCE, "highway_areas.yml", highwayAreaTags, f -> {
var attr = f.getAttrsAtZoom(14);
@ -248,7 +271,7 @@ class ConfiguredFeatureTest {
}
@Test
void testInputMapping() throws Exception {
void testInputMapping() {
//Show that a key in includeWhen with no values matches all values
testLinestring(TEST_RESOURCE, "data_type_attributes.yml", inputMappingTags, f -> {
var attr = f.getAttrsAtZoom(14);
@ -256,6 +279,8 @@ class ConfiguredFeatureTest {
assertEquals("string_val", attr.get("s_type"), "Produce string");
assertEquals(1, attr.get("d_type"), "Produce direction");
assertEquals(1L, attr.get("l_type"), "Produce long");
assertEquals(1, attr.get("i_type"), "Produce integer");
assertEquals(1.5, attr.get("double_type"), "Produce double");
assertEquals("yes", attr.get("intermittent"), "Produce raw attribute");
assertEquals(true, attr.get("is_intermittent"), "Produce and rename boolean");
@ -263,32 +288,495 @@ class ConfiguredFeatureTest {
}, 1);
}
@ParameterizedTest
@ValueSource(strings = {"natural:", "natural: [__any__]", "natural: __any__"})
void testMatchAny(String filter) {
testPolygon("""
sources:
osm:
type: osm
url: geofabrik:rhode-island
local_path: data/rhode-island.osm.pbf
layers:
- id: testLayer
features:
- source: osm
geometry: polygon
include_when:
%s
""".formatted(filter), Map.of(
"natural", "water"
), feature -> {
}, 1);
}
@Test
void testGeometryTypeMismatch() throws Exception {
void testExcludeValue() {
var config = """
sources:
osm:
type: osm
url: geofabrik:rhode-island
local_path: data/rhode-island.osm.pbf
layers:
- id: testLayer
features:
- source: osm
geometry: polygon
include_when:
natural: water
exclude_when:
name: excluded
""";
testPolygon(config, Map.of(
"natural", "water",
"name", "name"
), feature -> {
}, 1);
testPolygon(config, Map.of(
"natural", "water",
"name", "excluded"
), feature -> {
}, 0);
}
@ParameterizedTest
@ValueSource(strings = {"''", "['']", "[null]"})
void testRequireValue(String matchString) {
var config = """
sources:
osm:
type: osm
url: geofabrik:rhode-island
local_path: data/rhode-island.osm.pbf
layers:
- id: testLayer
features:
- source: osm
geometry: polygon
include_when:
natural: water
exclude_when:
name: %s
""".formatted(matchString);
testPolygon(config, Map.of(
"natural", "water",
"name", "name"
), feature -> {
}, 1);
testPolygon(config, Map.of(
"natural", "water"
), feature -> {
}, 0);
testPolygon(config, Map.of(
"natural", "water",
"name", ""
), feature -> {
}, 0);
}
@Test
void testMappingKeyValue() {
testPolygon("""
sources:
osm:
type: osm
url: geofabrik:rhode-island
local_path: data/rhode-island.osm.pbf
layers:
- id: testLayer
features:
- source: osm
geometry: polygon
include_when:
natural: water
attributes:
- key: key
type: match_key
- key: value
type: match_value
""", Map.of(
"natural", "water"
), feature -> {
assertEquals(Map.of(
"key", "natural",
"value", "water"
), feature.getAttrsAtZoom(14));
}, 1);
}
@Test
void testCoerceAttributeValue() {
testPolygon("""
sources:
osm:
type: osm
url: geofabrik:rhode-island
local_path: data/rhode-island.osm.pbf
layers:
- id: testLayer
features:
- source: osm
geometry: polygon
attributes:
- key: int
type: integer
- key: long
type: long
- key: double
type: double
""", Map.of(
"int", "1",
"long", "-1",
"double", "1.5"
), feature -> {
assertEquals(Map.of(
"int", 1,
"long", -1L,
"double", 1.5
), feature.getAttrsAtZoom(14));
}, 1);
}
@ParameterizedTest
@CsvSource(value = {
"1| 1",
"1+1| 1+1",
"${1+1}| 2",
"${match_key + '=' + match_value}| natural=water",
"${match_value.replace('ter', 'wa')}| wawa",
"${feature.tags.natural}| water",
"${feature.id}|1",
"\\${feature.id}|${feature.id}",
"\\\\${feature.id}|\\${feature.id}",
"${feature.source}|osm",
"${feature.source_layer}|null",
"${coalesce(feature.source_layer, 'missing')}|missing",
"{match: {test: {natural: water}}}|test",
"{match: {test: {natural: not_water}}}|null",
}, delimiter = '|')
void testExpressionValue(String expression, Object value) {
testPoint("""
sources:
osm:
type: osm
url: geofabrik:rhode-island
local_path: data/rhode-island.osm.pbf
layers:
- id: testLayer
features:
- source: osm
geometry: point
include_when:
natural: water
attributes:
- key: key
value: %s
""".formatted(expression), Map.of(
"natural", "water"
), feature -> {
var result = feature.getAttrsAtZoom(14).get("key");
String resultString = result == null ? "null" : result.toString();
assertEquals(value, resultString);
}, 1);
}
@Test
void testGetTag() {
testPoint("""
sources:
osm:
type: osm
url: geofabrik:rhode-island
local_path: data/rhode-island.osm.pbf
layers:
- id: testLayer
features:
- source: osm
geometry: point
include_when:
natural: water
attributes:
- key: key
value:
tag_value: natural
- key: key2
value:
tag_value: intval
type: integer
""", Map.of(
"natural", "water",
"intval", "1"
), feature -> {
assertEquals("water", feature.getAttrsAtZoom(14).get("key"));
assertEquals(1, feature.getAttrsAtZoom(14).get("key2"));
}, 1);
}
@ParameterizedTest
@ValueSource(strings = {
"",
"tag_value: depth",
"value: '${feature.tags[\"depth\"]}'",
"value: '${feature.tags.get(\"depth\")}'"
})
void testGetInExpressionUsesTagMapping(String getter) {
testPoint("""
sources:
osm:
type: osm
url: geofabrik:rhode-island
local_path: data/rhode-island.osm.pbf
tag_mappings:
depth: long
layers:
- id: testLayer
features:
- source: osm
geometry: point
attributes:
- key: depth
%s
""".formatted(getter), Map.of(
"depth", "35"
), feature -> {
assertEquals(35L, feature.getAttrsAtZoom(14).get("depth"));
}, 1);
}
@ParameterizedTest
@CsvSource(value = {
"12|12",
"${5+5}|10",
"${match_key.size()}|7",
"${value.size()}|5",
"{default_value: 4, overrides: {3: {natural: water}}}|3",
"{default_value: 4, overrides: {3: {natural: not_water}}}|4",
}, delimiter = '|')
void testAttributeMinZoomExpression(String expression, int minZoom) {
testPoint("""
sources:
osm:
type: osm
url: geofabrik:rhode-island
local_path: data/rhode-island.osm.pbf
layers:
- id: testLayer
features:
- source: osm
geometry: point
include_when:
natural: water
attributes:
- key: key
value: value
min_zoom: %s
""".formatted(expression), Map.of(
"natural", "water"
), feature -> {
assertNull(feature.getAttrsAtZoom(minZoom - 1).get("key"));
assertEquals("value", feature.getAttrsAtZoom(minZoom).get("key"));
}, 1);
}
@Test
void testMinZoomExpression() {
var config = """
sources:
osm:
type: osm
url: geofabrik:rhode-island
local_path: data/rhode-island.osm.pbf
layers:
- id: testLayer
features:
- source: osm
geometry: point
min_zoom:
default_value: 4
overrides:
- if: '${feature.tags.has("a", "b")}'
value: 5
include_when:
natural: water
""";
testPoint(config, Map.of(
"natural", "water"
), feature -> {
assertEquals(4, feature.getMinZoom());
}, 1);
testPoint(config, Map.of(
"natural", "water",
"a", "b"
), feature -> {
assertEquals(5, feature.getMinZoom());
}, 1);
}
@Test
void testFallbackValue() {
var config = """
sources:
osm:
type: osm
url: geofabrik:rhode-island
local_path: data/rhode-island.osm.pbf
layers:
- id: testLayer
features:
- source: osm
geometry: polygon
include_when:
natural: water
attributes:
- key: key
value: 1
include_when:
otherkey: value
else: 0
""";
testPolygon(config, Map.of(
"natural", "water"
), feature -> {
assertEquals(Map.of("key", 0), feature.getAttrsAtZoom(14));
}, 1);
testPolygon(config, Map.of(
"natural", "water",
"otherkey", "othervalue"
), feature -> {
assertEquals(Map.of("key", 0), feature.getAttrsAtZoom(14));
}, 1);
testPolygon(config, Map.of(
"natural", "water",
"otherkey", "value"
), feature -> {
assertEquals(Map.of("key", 1), feature.getAttrsAtZoom(14));
}, 1);
}
@ParameterizedTest
@CsvSource(value = {
"\"${feature.tags.has('natural', 'water')}\"",
"{__all__: [\"${feature.tags.has('natural', 'water')}\"]}",
}, delimiter = '|')
void testExpressionInMatch(String filter) {
var config = """
sources:
osm:
type: osm
url: geofabrik:rhode-island
local_path: data/rhode-island.osm.pbf
layers:
- id: testLayer
features:
- source: osm
geometry: polygon
include_when: %s
""".formatted(filter);
testPolygon(config, Map.of(
"natural", "water"
), feature -> {
}, 1);
testPolygon(config, Map.of(
"natural", "other"
), feature -> {
}, 0);
testPolygon(config, Map.of(
), feature -> {
}, 0);
}
@Test
void testExpressionAttrFilter() {
var config = """
sources:
osm:
type: osm
url: geofabrik:rhode-island
local_path: data/rhode-island.osm.pbf
layers:
- id: testLayer
features:
- source: osm
geometry: polygon
include_when:
natural: water
highway: motorway
attributes:
- key: key
value: true
include_when: ${ match_value.startsWith("wa") }
else: false
""";
testPolygon(config, Map.of(
"natural", "water"
), feature -> {
assertEquals(true, feature.getAttrsAtZoom(14).get("key"));
}, 1);
testPolygon(config, Map.of(
"highway", "motorway"
), feature -> {
assertEquals(false, feature.getAttrsAtZoom(14).get("key"));
}, 1);
}
@Test
void testExpressionAttrFilterNoMatchingKey() {
var config = """
sources:
osm:
type: osm
url: geofabrik:rhode-island
local_path: data/rhode-island.osm.pbf
layers:
- id: testLayer
features:
- source: osm
geometry: polygon
include_when: ${ feature.tags.has("natural", "water") }
attributes:
- key: key
value: true
include_when: ${ coalesce(match_value, '').startsWith("wa") }
else: false
""";
testPolygon(config, Map.of(
"natural", "water"
), feature -> {
assertEquals(false, feature.getAttrsAtZoom(14).get("key"));
}, 1);
}
@Test
void testGeometryTypeMismatch() {
//Validate that a schema that filters on lines does not match on a polygon feature
var sf =
SimpleFeature.createFakeOsmFeature(newPolygon(0, 0, 1, 0, 1, 1, 0, 0), motorwayTags, "osm", null, 1,
emptyList());
testFeature(TEST_RESOURCE, "road_motorway.yml", sf,
ConfiguredFeatureTest::linestringFeatureCollector, f -> {
}, 0);
testFeature(TEST_RESOURCE, "road_motorway.yml", sf, f -> {
}, 0);
}
@Test
void testSourceTypeMismatch() throws Exception {
void testSourceTypeMismatch() {
//Validate that a schema only matches on the specified data source
var sf =
SimpleFeature.createFakeOsmFeature(newLineString(0, 0, 1, 0, 1, 1, 0, 0), highwayAreaTags, "not_osm", null, 1,
emptyList());
testFeature(SAMPLE_RESOURCE, "highway_areas.yml", sf,
ConfiguredFeatureTest::linestringFeatureCollector, f -> {
}, 0);
testFeature(SAMPLE_RESOURCE, "highway_areas.yml", sf, f -> {
}, 0);
}
@Test
void testInvalidSchemas() throws Exception {
void testInvalidSchemas() {
testInvalidSchema("bad_geometry_type.yml", "Profile defined with invalid geometry type");
testInvalidSchema("no_layers.yml", "Profile defined with no layers");
}

37
planetiler-custommap/src/test/java/com/onthegomap/planetiler/custommap/SchemaTests.java 100644
Wyświetl plik

@ -0,0 +1,37 @@
package com.onthegomap.planetiler.custommap;
import static org.junit.jupiter.api.DynamicTest.dynamicTest;
import com.onthegomap.planetiler.config.Arguments;
import com.onthegomap.planetiler.custommap.configschema.SchemaConfig;
import com.onthegomap.planetiler.custommap.validator.SchemaSpecification;
import com.onthegomap.planetiler.custommap.validator.SchemaValidator;
import java.nio.file.Path;
import java.util.List;
import org.junit.jupiter.api.DynamicTest;
import org.junit.jupiter.api.TestFactory;
class SchemaTests {
@TestFactory
List<DynamicTest> shortbread() {
return testSchema("shortbread.yml", "shortbread.spec.yml");
}
private List<DynamicTest> testSchema(String schema, String spec) {
var base = Path.of("src", "main", "resources", "samples");
var result = SchemaValidator.validate(
SchemaConfig.load(base.resolve(schema)),
SchemaSpecification.load(base.resolve(spec)),
Arguments.of()
);
return result.results().stream()
.map(test -> dynamicTest(test.example().name(), () -> {
if (test.issues().isFailure()) {
throw test.issues().exception();
}
if (!test.issues().get().isEmpty()) {
throw new AssertionError("Validation failed:\n" + String.join("\n", test.issues().get()));
}
})).toList();
}
}

6
planetiler-custommap/src/test/java/com/onthegomap/planetiler/custommap/SchemaYAMLLoadTest.java
Wyświetl plik

@ -3,6 +3,7 @@ package com.onthegomap.planetiler.custommap;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import com.onthegomap.planetiler.custommap.configschema.SchemaConfig;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
@ -13,7 +14,7 @@ class SchemaYAMLLoadTest {
/**
* Test to ensure that all bundled schemas load to POJOs.
*
*
* @throws Exception
*/
@Test
@ -25,12 +26,13 @@ class SchemaYAMLLoadTest {
private void testSchemasInFolder(Path path) throws IOException {
var schemaFiles = Files.walk(path)
.filter(p -> p.getFileName().toString().endsWith(".yml"))
.filter(p -> !p.getFileName().toString().endsWith("spec.yml"))
.toList();
assertFalse(schemaFiles.isEmpty(), "No files found");
for (Path schemaFile : schemaFiles) {
var schemaConfig = ConfiguredMapMain.loadConfig(schemaFile);
var schemaConfig = SchemaConfig.load(schemaFile);
assertNotNull(schemaConfig, () -> "Failed to unmarshall " + schemaFile.toString());
assertNotNull(new ConfiguredProfile(schemaConfig), () -> "Failed to load profile from " + schemaFile.toString());
}

68
planetiler-custommap/src/test/java/com/onthegomap/planetiler/custommap/TagValueProducerTest.java 100644
Wyświetl plik

@ -0,0 +1,68 @@
package com.onthegomap.planetiler.custommap;
import static org.junit.jupiter.api.Assertions.assertEquals;
import com.onthegomap.planetiler.geo.GeoUtils;
import com.onthegomap.planetiler.reader.SimpleFeature;
import com.onthegomap.planetiler.reader.WithTags;
import java.util.List;
import java.util.Map;
import org.junit.jupiter.api.Test;
class TagValueProducerTest {
private static void testGet(TagValueProducer tvp, Map<String, Object> tags, String key, Object expected) {
var wrapped = WithTags.from(tags);
assertEquals(expected, tvp.mapTags(wrapped).get(key));
assertEquals(expected, tvp.valueForKey(wrapped, key));
assertEquals(expected, tvp.valueGetterForKey(key).apply(wrapped, key));
assertEquals(expected, tvp.valueProducerForKey(key)
.apply(new Contexts.ProcessFeature(SimpleFeature.create(GeoUtils.EMPTY_GEOMETRY, tags), tvp)
.createPostMatchContext(List.of())));
}
@Test
void testEmptyTagValueProducer() {
var tvp = new TagValueProducer(Map.of());
testGet(tvp, Map.of(), "key", null);
testGet(tvp, Map.of("key", 1), "key", 1);
testGet(tvp, Map.of("key", 1), "other", null);
}
@Test
void testNullTagValueProducer() {
var tvp = new TagValueProducer(null);
testGet(tvp, Map.of(), "key", null);
}
@Test
void testParseTypes() {
var tvp = new TagValueProducer(Map.of(
"int", "integer",
"double", Map.of("type", "double"),
"direction", Map.of("type", "direction")
));
testGet(tvp, Map.of(), "int", null);
testGet(tvp, Map.of(), "double", null);
testGet(tvp, Map.of(), "direction", 0);
testGet(tvp, Map.of("int", 1), "int", 1);
testGet(tvp, Map.of("int", "1"), "int", 1);
testGet(tvp, Map.of("direction", "-1"), "direction", -1);
}
@Test
void testRemapKeys() {
var tvp = new TagValueProducer(Map.of(
"int2", Map.of("type", "integer", "input", "int"),
"int3", Map.of("type", "integer", "input", "int2")
));
testGet(tvp, Map.of("int", "1"), "int", "1");
testGet(tvp, Map.of("int", "1"), "int2", 1);
testGet(tvp, Map.of("int", "1"), "int3", 1);
testGet(tvp, Map.of(), "int", null);
testGet(tvp, Map.of(), "int2", null);
testGet(tvp, Map.of(), "int3", null);
}
}

53
planetiler-custommap/src/test/java/com/onthegomap/planetiler/custommap/TypeConversionTest.java 100644
Wyświetl plik

@ -0,0 +1,53 @@
package com.onthegomap.planetiler.custommap;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertInstanceOf;
import java.util.List;
import java.util.stream.Stream;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.MethodSource;
class TypeConversionTest {
record Case(Object in, Class<?> clazz, Object out) {
Case {
if (out != null) {
assertInstanceOf(clazz, out);
}
}
}
private static Stream<Case> testTo(Class<?> clazz, Object out, Object... in) {
return Stream.of(in).map(i -> new Case(i, clazz, out));
}
private static Stream<Case> testConvertTo(Object out, Object... in) {
return testTo(out.getClass(), out, in);
}
static List<Case> cases() {
return Stream.of(
testConvertTo(1, "1", 1L, 1.1),
testConvertTo(1L, "1", 1L, 1.1),
testConvertTo(1d, "1", "1.0", "1e0", 1L, 1d),
testConvertTo(1.1, "1.1", 1.1),
testConvertTo("1", "1", 1, 1L, 1d),
testConvertTo("1.1", "1.1", 1.1d, 1.1f),
testConvertTo("1000", 1000, 1000d),
testConvertTo("NaN", Double.NaN),
testConvertTo(true, 1, 1L, 1d, "true", "TRUE"),
testConvertTo(false, 0, 0L, 0d, "false", "FALSE", "no"),
testConvertTo("true", true),
testConvertTo("false", false),
testTo(String.class, null, (String) null),
testTo(Integer.class, null, (String) null)
).flatMap(d -> d).toList();
}
@ParameterizedTest
@MethodSource("cases")
void testConversion(Case testCase) {
Object out = TypeConversion.convert(testCase.in, testCase.clazz);
assertEquals(testCase.out, out);
}
}

22
planetiler-custommap/src/test/java/com/onthegomap/planetiler/custommap/expression/BooleanExpressionScriptTest.java 100644
Wyświetl plik

@ -0,0 +1,22 @@
package com.onthegomap.planetiler.custommap.expression;
import static com.onthegomap.planetiler.expression.Expression.and;
import static com.onthegomap.planetiler.expression.Expression.or;
import static org.junit.jupiter.api.Assertions.assertEquals;
import com.onthegomap.planetiler.custommap.Contexts;
import com.onthegomap.planetiler.expression.Expression;
import org.junit.jupiter.api.Test;
class BooleanExpressionScriptTest {
@Test
void testSimplify() {
assertEquals(Expression.TRUE,
and(or(BooleanExpressionScript.script("1+1<3", Contexts.Root.DESCRIPTION))).simplify());
assertEquals(Expression.FALSE,
and(or(BooleanExpressionScript.script("1+1>3", Contexts.Root.DESCRIPTION))).simplify());
var other = BooleanExpressionScript.script("feature.tags.natural", Contexts.ProcessFeature.DESCRIPTION);
assertEquals(other, and(or(other)).simplify());
}
}

344
planetiler-custommap/src/test/java/com/onthegomap/planetiler/custommap/expression/ConfigExpressionTest.java 100644
Wyświetl plik

@ -0,0 +1,344 @@
package com.onthegomap.planetiler.custommap.expression;
import static com.onthegomap.planetiler.TestUtils.newPoint;
import static com.onthegomap.planetiler.custommap.expression.ConfigExpression.*;
import static com.onthegomap.planetiler.expression.Expression.matchAny;
import static com.onthegomap.planetiler.expression.Expression.or;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
import com.onthegomap.planetiler.custommap.Contexts;
import com.onthegomap.planetiler.custommap.TagValueProducer;
import com.onthegomap.planetiler.expression.DataType;
import com.onthegomap.planetiler.expression.Expression;
import com.onthegomap.planetiler.expression.MultiExpression;
import com.onthegomap.planetiler.reader.SimpleFeature;
import java.util.List;
import java.util.Map;
import org.junit.jupiter.api.Test;
class ConfigExpressionTest {
private static final ConfigExpression.Signature<Contexts.Root, Integer> ROOT =
signature(Contexts.Root.DESCRIPTION, Integer.class);
private static final ConfigExpression.Signature<Contexts.ProcessFeature, Integer> FEATURE_SIGNATURE =
signature(Contexts.ProcessFeature.DESCRIPTION, Integer.class);
@Test
void testConst() {
assertEquals(1, constOf(1).apply(ScriptContext.empty()));
}
@Test
void testVariable() {
var feature = SimpleFeature.create(newPoint(0, 0), Map.of("a", "b", "c", 1), "source", "source_layer", 99);
var context = new Contexts.ProcessFeature(feature, new TagValueProducer(Map.of()));
// simple match
assertEquals("source", variable(FEATURE_SIGNATURE.withOutput(String.class), "feature.source").apply(context));
assertEquals("source_layer",
variable(FEATURE_SIGNATURE.withOutput(String.class), "feature.source_layer").apply(context));
assertEquals(99L, variable(FEATURE_SIGNATURE.withOutput(Long.class), "feature.id").apply(context));
assertEquals(99, variable(FEATURE_SIGNATURE.withOutput(Integer.class), "feature.id").apply(context));
assertEquals(99d, variable(FEATURE_SIGNATURE.withOutput(Double.class), "feature.id").apply(context));
assertThrows(ParseException.class, () -> variable(FEATURE_SIGNATURE, "missing"));
}
@Test
void testCoalesce() {
assertNull(coalesce(List.of()).apply(Contexts.root()));
assertNull(coalesce(
List.of(
constOf(null)
)).apply(Contexts.root()));
assertEquals(2, coalesce(
List.of(
constOf(null),
constOf(2)
)).apply(Contexts.root()));
assertEquals(1, coalesce(
List.of(
constOf(1),
constOf(2)
)).apply(Contexts.root()));
}
@Test
void testDynamic() {
assertEquals(1, script(ROOT, "5 - 4").apply(Contexts.root()));
}
@Test
void testMatch() {
var feature = SimpleFeature.create(newPoint(0, 0), Map.of("a", "b", "c", 1));
var context = new Contexts.ProcessFeature(feature, new TagValueProducer(Map.of()));
// simple match
assertEquals(2, match(FEATURE_SIGNATURE, MultiExpression.of(List.of(
MultiExpression.entry(constOf(1),
BooleanExpressionScript.script("feature.tags.has('a', 'c')", FEATURE_SIGNATURE.in())),
MultiExpression.entry(constOf(2),
BooleanExpressionScript.script("feature.tags.has('a', 'b')", FEATURE_SIGNATURE.in()))
))).apply(context));
// dynamic fallback
assertEquals(5, match(FEATURE_SIGNATURE, MultiExpression.of(List.of(
MultiExpression.entry(constOf(1),
BooleanExpressionScript.script("feature.tags.has('a', 'c')", FEATURE_SIGNATURE.in())),
MultiExpression.entry(constOf(2),
BooleanExpressionScript.script("feature.tags.has('a', 'd')", FEATURE_SIGNATURE.in()))
)), ConfigExpression.script(FEATURE_SIGNATURE, "feature.tags.c + 4")).apply(context));
// no fallback
assertNull(match(FEATURE_SIGNATURE, MultiExpression.of(List.of(
MultiExpression.entry(constOf(1),
BooleanExpressionScript.script("feature.tags.has('a', 'd')", FEATURE_SIGNATURE.in())),
MultiExpression.entry(constOf(2),
BooleanExpressionScript.script("feature.tags.has('a', 'e')", FEATURE_SIGNATURE.in()))
))).apply(context));
// dynamic value
assertEquals(2, match(
FEATURE_SIGNATURE,
MultiExpression.of(List.of(
MultiExpression.entry(script(FEATURE_SIGNATURE, "1 + size(feature.tags.a)"),
Expression.matchAny("a", "b")),
MultiExpression.entry(constOf(1), Expression.matchAny("a", "c"))
))
).apply(context));
}
@Test
void testSimplifyCelFunction() {
assertEquals(
constOf(3),
script(FEATURE_SIGNATURE, "1+2").simplify()
);
}
@Test
void testSimplifyCelFunctionThatJustAccessesVar() {
assertEquals(
variable(FEATURE_SIGNATURE, "feature.id"),
script(FEATURE_SIGNATURE, "feature.id").simplify()
);
assertEquals(
script(FEATURE_SIGNATURE, "feature.tags.a"),
script(FEATURE_SIGNATURE, "feature.tags.a").simplify()
);
}
@Test
void testSimplifyCoalesce() {
assertEquals(
constOf(null),
coalesce(List.of()).simplify()
);
assertEquals(
constOf(null),
coalesce(List.of(constOf(null))).simplify()
);
assertEquals(
constOf(1),
coalesce(List.of(constOf(1))).simplify()
);
assertEquals(
constOf(1),
coalesce(List.of(constOf(1), constOf(2))).simplify()
);
assertEquals(
constOf(1),
coalesce(List.of(constOf(1), constOf(1))).simplify()
);
assertEquals(
constOf(1),
coalesce(List.of(constOf(1), constOf(2), constOf(1))).simplify()
);
}
@Test
void testSimplifyMatchAllFalse() {
assertEquals(
constOf(null),
match(FEATURE_SIGNATURE, MultiExpression.of(List.of(
MultiExpression.entry(constOf(1),
BooleanExpressionScript.script("1 > 2", FEATURE_SIGNATURE.in())),
MultiExpression.entry(constOf(2),
BooleanExpressionScript.script("1 > 3", FEATURE_SIGNATURE.in()))
))).simplify()
);
}
@Test
void testSimplifyMatchAllFalseWithFallback() {
assertEquals(
constOf(3),
match(FEATURE_SIGNATURE, MultiExpression.of(List.of(
MultiExpression.entry(constOf(1),
BooleanExpressionScript.script("1 > 2", FEATURE_SIGNATURE.in())),
MultiExpression.entry(constOf(2),
BooleanExpressionScript.script("1 > 3", FEATURE_SIGNATURE.in()))
)), script(FEATURE_SIGNATURE, "1+2")).simplify()
);
}
@Test
void testSimplifyRemoveCasesAfterTrueAndReplaceFallback() {
assertEquals(
match(FEATURE_SIGNATURE, MultiExpression.of(List.of(
MultiExpression.entry(constOf(0),
BooleanExpressionScript.script("feature.tags.has('a', 'b')", FEATURE_SIGNATURE.in()))
)), constOf(1)),
match(FEATURE_SIGNATURE, MultiExpression.of(List.of(
MultiExpression.entry(constOf(0),
BooleanExpressionScript.script("feature.tags.has('a', 'b')", FEATURE_SIGNATURE.in())),
MultiExpression.entry(constOf(1),
BooleanExpressionScript.script("1 < 2", FEATURE_SIGNATURE.in())),
MultiExpression.entry(constOf(2),
BooleanExpressionScript.script("feature.tags.has('c', 'd')", FEATURE_SIGNATURE.in()))
)), constOf(2)).simplify()
);
}
@Test
void testSimplifyRemoveFalseCases() {
assertEquals(
match(FEATURE_SIGNATURE, MultiExpression.of(List.of(
MultiExpression.entry(constOf(2),
BooleanExpressionScript.script("feature.tags.has('a', 'b')", FEATURE_SIGNATURE.in()))
)), script(FEATURE_SIGNATURE, "size(feature.tags.a)")),
match(FEATURE_SIGNATURE, MultiExpression.of(List.of(
MultiExpression.entry(constOf(1),
BooleanExpressionScript.script("1 > 2", FEATURE_SIGNATURE.in())),
MultiExpression.entry(constOf(2),
BooleanExpressionScript.script("feature.tags.has('a', 'b')", FEATURE_SIGNATURE.in()))
)), script(FEATURE_SIGNATURE, "size(feature.tags.a)")).simplify()
);
}
@Test
void testSimplifyMatchCondition() {
assertEquals(
match(FEATURE_SIGNATURE, MultiExpression.of(List.of(
MultiExpression.entry(constOf(2), matchAny("a", "b")))
), script(FEATURE_SIGNATURE, "size(feature.tags.a)")),
match(FEATURE_SIGNATURE, MultiExpression.of(List.of(
MultiExpression.entry(constOf(2), or(or(matchAny("a", "b"))))
)), script(FEATURE_SIGNATURE, "size(feature.tags.a)")).simplify()
);
}
@Test
void testSimplifyMatchResultFunction() {
assertEquals(
match(FEATURE_SIGNATURE, MultiExpression.of(List.of(
MultiExpression.entry(constOf(2), matchAny("a", "b")))
), script(FEATURE_SIGNATURE, "size(feature.tags.a)")),
match(FEATURE_SIGNATURE, MultiExpression.of(List.of(
MultiExpression.entry(coalesce(List.of(constOf(2))), matchAny("a", "b"))
)), script(FEATURE_SIGNATURE, "size(feature.tags.a)")).simplify()
);
}
@Test
void testSimplifyFallbackFunction() {
assertEquals(
match(FEATURE_SIGNATURE, MultiExpression.of(List.of(
MultiExpression.entry(constOf(2), matchAny("a", "b")))
), constOf(3)),
match(FEATURE_SIGNATURE, MultiExpression.of(List.of(
MultiExpression.entry(constOf(2), matchAny("a", "b"))
)), script(FEATURE_SIGNATURE, "1+2")).simplify()
);
}
@Test
void testSimplifyFirstTrue() {
assertEquals(
constOf(1),
match(FEATURE_SIGNATURE, MultiExpression.of(List.of(
MultiExpression.entry(constOf(1),
BooleanExpressionScript.script("1 < 2", FEATURE_SIGNATURE.in())),
MultiExpression.entry(constOf(2),
BooleanExpressionScript.script("1 > 3", FEATURE_SIGNATURE.in()))
))).simplify()
);
}
@Test
void testGetTag() {
var feature = SimpleFeature.create(newPoint(0, 0), Map.of("abc", "123"), "source", "source_layer", 99);
assertEquals(
"123",
getTag(FEATURE_SIGNATURE.withOutput(Object.class), constOf("abc")).apply(
new Contexts.ProcessFeature(feature, new TagValueProducer(Map.of())))
);
assertEquals(
123,
getTag(FEATURE_SIGNATURE.withOutput(Object.class), constOf("abc"))
.apply(new Contexts.ProcessFeature(feature, new TagValueProducer(Map.of("abc", "integer"))))
);
assertEquals(
123,
getTag(signature(Contexts.FeaturePostMatch.DESCRIPTION, Object.class), constOf("abc"))
.apply(new Contexts.ProcessFeature(feature, new TagValueProducer(Map.of("abc", "integer")))
.createPostMatchContext(List.of()))
);
assertEquals(
null,
getTag(signature(Contexts.Root.DESCRIPTION, Object.class), constOf("abc"))
.apply(Contexts.root())
);
}
@Test
void testCastGetTag() {
var feature = SimpleFeature.create(newPoint(0, 0), Map.of("abc", "123"), "source", "source_layer", 99);
var context = new Contexts.ProcessFeature(feature, new TagValueProducer(Map.of()));
var expression = cast(
FEATURE_SIGNATURE.withOutput(Integer.class),
getTag(FEATURE_SIGNATURE.withOutput(Object.class), constOf("abc")),
DataType.GET_INT
);
assertEquals(123, expression.apply(context));
assertEquals(123d, cast(
FEATURE_SIGNATURE.withOutput(Double.class),
getTag(FEATURE_SIGNATURE.withOutput(Object.class), constOf("abc")),
DataType.GET_INT
).apply(context));
}
@Test
void testCast() {
var expression = cast(
ROOT.withOutput(Integer.class),
constOf("123"),
DataType.GET_INT
);
assertEquals(123, expression.apply(Contexts.root()));
}
@Test
void testSimplifyCast() {
assertEquals(constOf(123),
cast(
ROOT.withOutput(Integer.class),
constOf("123"),
DataType.GET_INT
).simplify()
);
}
}

66
planetiler-custommap/src/test/java/com/onthegomap/planetiler/custommap/expression/DataTypeTest.java 100644
Wyświetl plik

@ -0,0 +1,66 @@
package com.onthegomap.planetiler.custommap.expression;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNull;
import com.onthegomap.planetiler.expression.DataType;
import org.junit.jupiter.api.Test;
class DataTypeTest {
@Test
void testLong() {
assertEquals(1L, DataType.from("long").convertFrom("1"));
assertEquals(1L, DataType.from("long").convertFrom(1));
assertNull(DataType.from("long").convertFrom("garbage"));
}
@Test
void testInteger() {
assertEquals(1, DataType.from("integer").convertFrom("1"));
assertNull(DataType.from("integer").convertFrom("garbage"));
assertEquals(1, DataType.from("integer").convertFrom("1.5"));
}
@Test
void testDouble() {
assertEquals(1.5, DataType.from("double").convertFrom("1.5"));
assertEquals(1.0, DataType.from("double").convertFrom(1));
assertNull(DataType.from("double").convertFrom("garbage"));
}
@Test
void testString() {
assertEquals("1.5", DataType.from("string").convertFrom("1.5"));
assertEquals("1.5", DataType.from("string").convertFrom(1.5));
}
@Test
void testRaw() {
assertEquals("1.5", DataType.from("raw").convertFrom("1.5"));
assertEquals(1.5, DataType.from("raw").convertFrom(1.5));
}
@Test
void testBoolean() {
assertEquals(true, DataType.from("boolean").convertFrom("1"));
assertEquals(true, DataType.from("boolean").convertFrom("true"));
assertEquals(true, DataType.from("boolean").convertFrom("yes"));
assertEquals(true, DataType.from("boolean").convertFrom(1));
assertEquals(false, DataType.from("boolean").convertFrom(0));
assertEquals(false, DataType.from("boolean").convertFrom("false"));
assertEquals(false, DataType.from("boolean").convertFrom("no"));
}
@Test
void testDirection() {
assertEquals(1, DataType.from("direction").convertFrom("1"));
assertEquals(1, DataType.from("direction").convertFrom(1));
assertEquals(1, DataType.from("direction").convertFrom("true"));
assertEquals(1, DataType.from("direction").convertFrom("yes"));
assertEquals(-1, DataType.from("direction").convertFrom(-1));
assertEquals(-1, DataType.from("direction").convertFrom("-1"));
assertEquals(0, DataType.from("direction").convertFrom(0));
assertEquals(0, DataType.from("direction").convertFrom("no"));
assertEquals(0, DataType.from("direction").convertFrom("false"));
}
}

68
planetiler-custommap/src/test/java/com/onthegomap/planetiler/custommap/expression/ExpressionTests.java 100644
Wyświetl plik

@ -0,0 +1,68 @@
package com.onthegomap.planetiler.custommap.expression;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNull;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.CsvSource;
class ExpressionTests {
@ParameterizedTest
@CsvSource(value = {
"1|1|long",
"1+1|2|long",
"'1' + string(1)|11|string",
"coalesce(null, null)||null",
"coalesce(null, 1)|1|long",
"coalesce(1, null)|1|long",
"coalesce(1, 2)|1|long",
"coalesce(null, null, 1)|1|long",
"coalesce(null, 1, 2)|1|long",
"coalesce(1, 2, null)+2|3|long",
"nullif('abc', '')|'abc'|string",
"nullif('', '')|null|null",
"nullif(1, 1)|null|null",
"nullif(1, 12)|1|long",
"'123'.replace('12', 'X')|X3|string",
"'123'.replaceRegex('1(.)', '$1')|23|string",
"string(123)|123|string",
"string({1:2,3:'4'}[1])|2|string",
"'abc'.matches('a.c')|true|boolean",
"'abc'.matches('a.d')|false|boolean",
"{'a': 1}.has('a')|true|boolean",
"{'a': 1}.has('a', 1)|true|boolean",
"{'a': 1}.has('a', 1, 2)|true|boolean",
"{'a': 2}.has('a', 1, 2)|true|boolean",
"{'a': 2}.has('a', 3)|false|boolean",
"{'a': 1}.has('b')|false|boolean",
"coalesce({'a': 1}.get('a'), 2)|1|long",
"coalesce({'a': 1}.get('b'), 2)|2|long",
"{'a': 1}.getOrDefault('a', 2)|1|long",
"{'a': 1}.getOrDefault('b', 2)|2|long",
"max([1, 2, 3])|3|long",
"max([1.1, 2.2, 3.3])|3.3|double",
"min([1, 2, 3])|1|long",
"min([1.1, 2.2, 3.3])|1.1|double",
"max([1])|1|long",
"min([1])|1|long",
}, delimiter = '|')
void testExpression(String in, String expected, String type) {
var expression = ConfigExpressionScript.parse(in, ScriptEnvironment.root());
var result = expression.apply(ScriptContext.empty());
switch (type) {
case "long" -> assertEquals(Long.valueOf(expected), result);
case "double" -> assertEquals(Double.valueOf(expected), result);
case "string" -> assertEquals(expected, result);
case "boolean" -> assertEquals(Boolean.valueOf(expected), result);
case "null" -> assertNull(result);
default -> throw new IllegalArgumentException(type);
}
}
}

183
planetiler-custommap/src/test/java/com/onthegomap/planetiler/custommap/validator/SchemaValidatorTest.java 100644
Wyświetl plik

@ -0,0 +1,183 @@
package com.onthegomap.planetiler.custommap.validator;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
import com.onthegomap.planetiler.config.Arguments;
import com.onthegomap.planetiler.custommap.configschema.SchemaConfig;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.PrintStream;
import java.io.UncheckedIOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.CsvSource;
class SchemaValidatorTest {
@TempDir
Path tmpDir;
record Result(SchemaValidator.Result output, String cliOutput) {}
Result validate(String schema, String spec) throws IOException {
var args = Arguments.of();
var result = SchemaValidator.validate(
SchemaConfig.load(schema),
SchemaSpecification.load(spec),
args
);
for (var example : result.results()) {
if (example.issues().isFailure()) {
assertNotNull(example.issues().get());
}
}
// also exercise the cli writer and return what it would have printed to stdout
var cliOutput = validateCli(Files.writeString(tmpDir.resolve("schema"),
schema + "\nexamples: " + Files.writeString(tmpDir.resolve("spec.yml"), spec)), args);
// also test the case where the examples are embedded in the schema itself
assertEquals(
cliOutput,
validateCli(Files.writeString(tmpDir.resolve("schema"), schema + "\n" + spec), args)
);
// also test where examples points to a relative path (written in previous step)
assertEquals(
cliOutput,
validateCli(Files.writeString(tmpDir.resolve("schema"), schema + "\nexamples: spec.yml"), args)
);
return new Result(result, cliOutput);
}
private String validateCli(Path path, Arguments args) {
try (
var baos = new ByteArrayOutputStream();
var printStream = new PrintStream(baos, true, StandardCharsets.UTF_8)
) {
SchemaValidator.validateFromCli(
path,
args,
printStream
);
return baos.toString(StandardCharsets.UTF_8);
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
String waterSchema = """
sources:
osm:
type: osm
url: geofabrik:rhode-island
layers:
- id: water
features:
- source: osm
geometry: polygon
include_when:
natural: water
attributes:
- key: natural
""";
private Result validateWater(String layer, String geometry, String tags, String allowExtraTags) throws IOException {
return validate(
waterSchema,
"""
examples:
- name: test output
input:
source: osm
geometry: polygon
tags:
natural: water
output:
layer: %s
geometry: %s
%s
tags:
%s
""".formatted(layer, geometry, allowExtraTags == null ? "" : allowExtraTags,
tags == null ? "" : tags.indent(6).strip())
);
}
@ParameterizedTest
@CsvSource(value = {
"true,water,polygon,natural: water,",
"true,water,polygon,,",
"true,water,polygon,'natural: water\nother: null',",
"false,water,polygon,natural: null,",
"false,water2,polygon,natural: water,",
"false,water,line,natural: water,",
"false,water,line,natural: water,",
"false,water,polygon,natural: water2,",
"false,water,polygon,'natural: water\nother: value',",
"true,water,polygon,natural: water,allow_extra_tags: true",
"true,water,polygon,natural: water,allow_extra_tags: false",
"true,water,polygon,,allow_extra_tags: true",
"false,water,polygon,,allow_extra_tags: false",
})
void testValidateWaterPolygon(boolean shouldBeOk, String layer, String geometry, String tags, String allowExtraTags)
throws IOException {
var results = validateWater(layer, geometry, tags, allowExtraTags);
assertEquals(1, results.output.results().size());
assertEquals("test output", results.output.results().get(0).example().name());
if (shouldBeOk) {
assertTrue(results.output.ok(), results.toString());
assertFalse(results.cliOutput.contains("FAIL"), "contained FAIL but should not have: " + results.cliOutput);
} else {
assertFalse(results.output.ok(), "Expected an issue, but there were none");
assertTrue(results.cliOutput.contains("FAIL"), "did not contain FAIL but should have: " + results.cliOutput);
}
}
@Test
void testValidationFailsWrongNumberOfFeatures() throws IOException {
var results = validate(
waterSchema,
"""
examples:
- name: test output
input:
source: osm
geometry: polygon
tags:
natural: water
output:
"""
);
assertFalse(results.output.ok(), results.toString());
results = validate(
waterSchema,
"""
examples:
- name: test output
input:
source: osm
geometry: polygon
tags:
natural: water
output:
- layer: water
geometry: polygon
tags:
natural: water
- layer: water2
geometry: polygon
tags:
natural: water2
"""
);
assertFalse(results.output.ok(), results.toString());
}
}

6
planetiler-custommap/src/test/resources/invalidSchema/bad_geometry_type.yml
Wyświetl plik

@ -6,13 +6,13 @@ sources:
type: osm
url: geofabrik:rhode-island
layers:
- name: testLayer
- id: testLayer
features:
- sources:
- source:
- osm
geometry: smurf
include_when:
natural: water
attributes:
- key: water
- constant_value: wet
- value: wet

12
planetiler-custommap/src/test/resources/validSchema/data_type_attributes.yml
Wyświetl plik

@ -8,22 +8,26 @@ sources:
tag_mappings:
b_type: boolean
l_type: long
i_type: integer
d_type: direction
s_type: string
intermittent:
output: is_intermittent
double_type: double
is_intermittent:
input: intermittent
type: boolean
bridge:
type: boolean
layers:
- name: testLayer
- id: testLayer
features:
- sources:
- source:
- osm
geometry: line
attributes:
- key: b_type
- key: l_type
- key: i_type
- key: double_type
- key: d_type
- key: s_type
- key: intermittent

4
planetiler-custommap/src/test/resources/validSchema/local_path.yml
Wyświetl plik

@ -7,9 +7,9 @@ sources:
url: geofabrik:rhode-island
local_path: data/rhode-island.osm.pbf
layers:
- name: testLayer
- id: testLayer
features:
- sources:
- source:
- osm
geometry: polygon
include_when:

6
planetiler-custommap/src/test/resources/validSchema/road_motorway.yml
Wyświetl plik

@ -14,9 +14,9 @@ tag_mappings:
layer: long
tunnel: boolean
layers:
- name: road
- id: road
features:
- sources:
- source:
- osm
min_zoom: 4
geometry: line
@ -29,7 +29,7 @@ layers:
bridge: true
min_zoom: 11
- key: tunnel
constant_value: true
value: true
include_when:
tunnel: true
min_zoom: 11

6
planetiler-custommap/src/test/resources/validSchema/static_attribute.yml
Wyświetl plik

@ -6,13 +6,13 @@ sources:
type: osm
url: geofabrik:rhode-island
layers:
- name: testLayer
- id: testLayer
features:
- sources:
- source:
- osm
geometry: polygon
include_when:
natural: water
attributes:
- key: natural
constant_value: aTestConstantValue
value: aTestConstantValue

4
planetiler-custommap/src/test/resources/validSchema/tag_attribute.yml
Wyświetl plik

@ -6,9 +6,9 @@ sources:
type: osm
url: geofabrik:rhode-island
layers:
- name: testLayer
- id: testLayer
features:
- sources:
- source:
- osm
geometry: polygon
include_when:

10
planetiler-custommap/src/test/resources/validSchema/tag_include.yml
Wyświetl plik

@ -6,9 +6,9 @@ sources:
type: osm
url: geofabrik:rhode-island
layers:
- name: testLayer
- id: testLayer
features:
- sources:
- source:
- osm
min_zoom: 10
geometry: polygon
@ -16,12 +16,12 @@ layers:
natural: water
attributes:
- key: test_include
constant_value: ok
value: ok
include_when:
natural: water
- key: test_exclude
constant_value: bad
value: bad
include_when:
natural: mud
- key: test_zoom_tag
min_zoom: 12
min_zoom: 12

23
planetiler-custommap/src/test/resources/validSchema/zoom_filter.yml
Wyświetl plik

@ -8,19 +8,20 @@ sources:
tag_mappings:
lanes: long
layers:
- name: testLayer
- id: testLayer
features:
- sources:
- source:
- osm
geometry: line
min_zoom: 4
zoom_override:
- min: 5
tag:
highway: trunk
- min: 7
tag:
highway: primary
min_zoom:
default_value: 4
overrides:
- value: 5
if:
highway: trunk
- value: 7
if:
highway: primary
include_when:
highway:
attributes:
@ -34,4 +35,4 @@ layers:
3: 9
2: 10
- key: toll
min_zoom: 8
min_zoom: 8

45
planetiler-dist/src/main/java/com/onthegomap/planetiler/Main.java
Wyświetl plik

@ -5,6 +5,7 @@ import static java.util.Map.entry;
import com.onthegomap.planetiler.benchmarks.LongLongMapBench;
import com.onthegomap.planetiler.benchmarks.OpenMapTilesMapping;
import com.onthegomap.planetiler.custommap.ConfiguredMapMain;
import com.onthegomap.planetiler.custommap.validator.SchemaValidator;
import com.onthegomap.planetiler.examples.BikeRouteOverlay;
import com.onthegomap.planetiler.examples.OsmQaTiles;
import com.onthegomap.planetiler.examples.ToiletsOverlay;
@ -13,6 +14,7 @@ import com.onthegomap.planetiler.mbtiles.Verify;
import java.util.Arrays;
import java.util.Locale;
import java.util.Map;
import java.util.stream.Stream;
import org.openmaptiles.OpenMapTilesMain;
import org.openmaptiles.util.VerifyMonaco;
@ -25,32 +27,57 @@ public class Main {
private static final EntryPoint DEFAULT_TASK = OpenMapTilesMain::main;
private static final Map<String, EntryPoint> ENTRY_POINTS = Map.ofEntries(
entry("generate-openmaptiles", OpenMapTilesMain::main),
entry("generate-custom", ConfiguredMapMain::main),
entry("openmaptiles", OpenMapTilesMain::main),
entry("generate-custom", ConfiguredMapMain::main),
entry("custom", ConfiguredMapMain::main),
entry("generate-shortbread", bundledSchema("shortbread.yml")),
entry("shortbread", bundledSchema("shortbread.yml")),
entry("verify", SchemaValidator::main),
entry("verify-custom", SchemaValidator::main),
entry("verify-schema", SchemaValidator::main),
entry("example-bikeroutes", BikeRouteOverlay::main),
entry("example-toilets", ToiletsOverlay::main),
entry("example-toilets-lowlevel", ToiletsOverlayLowLevelApi::main),
entry("example-qa", OsmQaTiles::main),
entry("osm-qa", OsmQaTiles::main),
entry("benchmark-mapping", OpenMapTilesMapping::main),
entry("benchmark-longlongmap", LongLongMapBench::main),
entry("verify-mbtiles", Verify::main),
entry("verify-monaco", VerifyMonaco::main)
);
private static EntryPoint bundledSchema(String path) {
return args -> ConfiguredMapMain.main(Stream.concat(
Stream.of("--schema=" + path),
Stream.of(args)
).toArray(String[]::new));
}
public static void main(String[] args) throws Exception {
EntryPoint task = DEFAULT_TASK;
if (args.length > 0) {
String maybeTask = args[0].trim().toLowerCase(Locale.ROOT);
EntryPoint taskFromArg0 = ENTRY_POINTS.get(maybeTask);
if (taskFromArg0 != null) {
args = Arrays.copyOfRange(args, 1, args.length);
task = taskFromArg0;
} else if (!maybeTask.contains("=") && !maybeTask.startsWith("-")) {
System.err.println("Unrecognized task: " + maybeTask);
System.err.println("possibilities: " + ENTRY_POINTS.keySet());
System.exit(1);
if (maybeTask.matches("^.*\\.ya?ml$")) {
task = ConfiguredMapMain::main;
args[0] = "--schema=" + args[0];
} else {
EntryPoint taskFromArg0 = ENTRY_POINTS.get(maybeTask);
if (taskFromArg0 != null) {
args = Arrays.copyOfRange(args, 1, args.length);
task = taskFromArg0;
} else if (!maybeTask.contains("=") && !maybeTask.startsWith("-")) {
System.err.println("Unrecognized task: " + maybeTask);
System.err.println("possibilities: " + ENTRY_POINTS.keySet());
System.exit(1);
}
}
}

5
pom.xml
Wyświetl plik

@ -98,6 +98,11 @@
<artifactId>snakeyaml</artifactId>
<version>1.32</version>
</dependency>
<dependency>
<groupId>org.snakeyaml</groupId>
<artifactId>snakeyaml-engine</artifactId>
<version>2.4</version>
</dependency>
<dependency>
<groupId>org.commonmark</groupId>
<artifactId>commonmark</artifactId>

62
quickstart.sh
Wyświetl plik

@ -10,17 +10,12 @@ JAVA="${JAVA:-java}"
METHOD="build"
AREA="monaco"
STORAGE="mmap"
PLANETILER_ARGS=("--download" "--force")
PLANETILER_ARGS=()
MEMORY=""
DRY_RUN=""
VERSION="latest"
DOCKER_DIR="$(pwd)/data"
# Handle quickstart.sh planet or quickstart.sh monaco
case $1 in
-*) ;;
*) AREA="$1"; shift ;;
esac
TASK="openmaptiles"
# Parse args into env vars
while [[ $# -gt 0 ]]; do
@ -43,17 +38,36 @@ while [[ $# -gt 0 ]]; do
--dry-run) DRY_RUN="true" ;;
*) PLANETILER_ARGS+=("$1") ;;
*)
# on the first passthrough arg, check if it's instructions to do something besides openmaptiles
if (( ${#PLANETILER_ARGS[@]} == 0 )); then
case $1 in
*openmaptiles*) PLANETILER_ARGS+=("$1") ;;
-*) PLANETILER_ARGS+=("$1") ;;
*.yml|*shortbread*|*generate*|*-qa|*example*|*verify*|*custom*|*benchmark*)
TASK="$1"
PLANETILER_ARGS+=("$1")
;;
*) AREA="$1" ;;
esac
else
PLANETILER_ARGS+=("$1")
fi
;;
esac
shift
done
PLANETILER_ARGS+=("--area=$AREA")
PLANETILER_ARGS+=("--storage=$STORAGE")
PLANETILER_ARGS+=("--download")
PLANETILER_ARGS+=("--force")
# Configure memory settings based on the area being built
PLANETILER_ARGS+=("--area=$AREA")
case $AREA in
planet)
# For extracts, use default nodemap type (sortedtable) and -Xmx (25% of RAM up to 25GB) and hope for the best.
# You can set --memory=5g if you want to change it.
PLANETILER_ARGS+=("--nodemap-type=array" "--download-threads=20" "--download-chunk-size-mb=500")
case "$STORAGE" in
ram) MEMORY="${MEMORY:-"-Xmx150g"}" ;;
@ -61,43 +75,39 @@ case $AREA in
esac
;;
monaco)
# Use mini extracts for monaco
PLANETILER_ARGS+=("--water-polygons-url=https://github.com/onthegomap/planetiler/raw/main/planetiler-core/src/test/resources/water-polygons-split-3857.zip")
PLANETILER_ARGS+=("--water-polygons-path=data/sources/monaco-water.zip")
PLANETILER_ARGS+=("--natural-earth-url=https://github.com/onthegomap/planetiler/raw/main/planetiler-core/src/test/resources/natural_earth_vector.sqlite.zip")
PLANETILER_ARGS+=("--natural-earth-path=data/sources/monaco-natural_earth_vector.sqlite.zip")
if [ "$TASK" == "openmaptiles" ]; then
# Use mini extracts for monaco
PLANETILER_ARGS+=("--water-polygons-url=https://github.com/onthegomap/planetiler/raw/main/planetiler-core/src/test/resources/water-polygons-split-3857.zip")
PLANETILER_ARGS+=("--water-polygons-path=data/sources/monaco-water.zip")
PLANETILER_ARGS+=("--natural-earth-url=https://github.com/onthegomap/planetiler/raw/main/planetiler-core/src/test/resources/natural_earth_vector.sqlite.zip")
PLANETILER_ARGS+=("--natural-earth-path=data/sources/monaco-natural_earth_vector.sqlite.zip")
fi
;;
esac
# For extracts, use default nodemap type (sortedtable) and -Xmx (25% of RAM up to 25GB) and hope for the best.
# You can set --memory=5g if you want to change it.
JVM_ARGS="-XX:+UseParallelGC $MEMORY"
echo "Running planetiler with:"
echo " METHOD=\"$METHOD\" (change with --docker --jar or --build)"
echo " JVM_ARGS=\"${JVM_ARGS}\" (change with --memory=Xg)"
echo " TASK=\"${TASK}\""
echo " PLANETILER_ARGS=\"${PLANETILER_ARGS[*]}\""
echo " DRY_RUN=\"${DRY_RUN:-false}\""
echo ""
if [ "$DRY_RUN" == "true" ]
then
if [ "$DRY_RUN" == "true" ]; then
echo "Without --dry-run, will run commands:"
else
sleep 3
fi
function run() {
echo "$ $*"
if [ "$DRY_RUN" != "true" ]
then
if [ "$DRY_RUN" != "true" ]; then
eval "$*"
fi
}
function check_java_version() {
if [ "$DRY_RUN" != "true" ]
then
if [ "$DRY_RUN" != "true" ]; then
if [ -z "$(which java)" ]; then
echo "java not found on path"
exit 1
@ -118,12 +128,14 @@ case $METHOD in
run docker run -e JAVA_TOOL_OPTIONS=\'"${JVM_ARGS}"\' -v "$DOCKER_DIR":/data "ghcr.io/onthegomap/planetiler:${VERSION}" "${PLANETILER_ARGS[@]}"
;;
jar)
echo "Downloading latest planetiler release..."
run wget -nc "https://github.com/onthegomap/planetiler/releases/${VERSION}/download/planetiler.jar"
check_java_version planetiler.jar
run "$JAVA" "${JVM_ARGS}" -jar planetiler.jar "${PLANETILER_ARGS[@]}"
;;
build)
run ./mvnw -DskipTests --projects planetiler-dist -am clean package
echo "Building planetiler..."
run ./mvnw -q -DskipTests --projects planetiler-dist -am clean package
run "$JAVA" "${JVM_ARGS}" -jar planetiler-dist/target/*with-deps.jar "${PLANETILER_ARGS[@]}"
;;
esac