Terria Archives - Matthew Gove Blog https://blog.matthewgove.com/tag/terria/ Travel the World through Maps, Data, and Photography Wed, 26 Jan 2022 03:34:35 +0000 en-US hourly 1 https://wordpress.org/?v=6.1.5 https://blog.matthewgove.com/wp-content/uploads/2021/03/cropped-android-chrome-512x512-1-32x32.png Terria Archives - Matthew Gove Blog https://blog.matthewgove.com/tag/terria/ 32 32 Travel Bloggers: How to Stand Out with Powerful Interactive Maps https://blog.matthewgove.com/2022/02/04/travel-bloggers-how-to-stand-out-with-powerful-interactive-maps/ Fri, 04 Feb 2022 16:00:00 +0000 https://blog.matthewgove.com/?p=3818 Interactive maps are one of the most powerful tools available to travel bloggers today. Unfortunately, far too many travel bloggers today are either using interactive maps incorrectly or not using them to their full potential. As a GIS expert, data scientist, and travel blogger myself, it pains me greatly when […]

The post Travel Bloggers: How to Stand Out with Powerful Interactive Maps appeared first on Matthew Gove Blog.

]]>
Interactive maps are one of the most powerful tools available to travel bloggers today. Unfortunately, far too many travel bloggers today are either using interactive maps incorrectly or not using them to their full potential. As a GIS expert, data scientist, and travel blogger myself, it pains me greatly when I see so many crappy maps on travel blogs. There is so much potential going to waste. As a result, I want to help you, the travel blogger, realize the full potential of your interactive maps. At the very minimum, you’ll see more traffic, retain more visitors, and make your website easier to navigate.

What Makes a Good Interactive Map?

Regardless of what industry you’re in, a good interactive map is easy to understand, offers an intuitive user experience, and draws the user’s focus to the data on the map. One of the best examples of an interactive map is the RadarScope application, which we covered in detail last month in 6 Powerful Weather Applications for Stunning Landscape Photography. If you’re not familiar with it, RadarScope is an application that plots weather radar data, severe weather warnings, and much more on a map.

So how exactly does RadarScope do it so well? When I look at the maps in the screenshots above, I make a few key observations about what makes it such a powerful interactive map, even without having access to its interactivity.

  1. Your eye is immediately drawn to the radar data on the screen.
  2. The basemap is simple enough that you instantly know where the storms are, while at the same time fading into the background and not distracting you from the data on the map.
  3. RadarScope’s chosen color scheme is easy to understand because it’s the industry standard for the weather and meteorology field.
  4. The design of RadarScope’s user interface is timeless. The last two screenshots in the above gallery are from the El Reno (31 May) and Moore, Oklahoma (20 May) tornadoes in 2013. The remaining screenshots were taken between 2017 and 2021.

You Can Bring That Same User Experience to Your Travel Blog Without Breaking the Bank

Thankfully, there are plenty of tools and applications available today to bring a similar mapping experience to your travel blog. Best of all, many are free and open source, so you don’t have to spend your hard-earned cash on expensive licensing fees. However, before we look at solutions, let’s have a look at the problem.

Too Many Travel Bloggers’ Interactive Maps are Not Actually Fully Interactive

In order to fully understand the problem so many travel bloggers run into, let’s first look at the definition of interactive.

Interactive: allowing a two-way flow of information between a computer and a computer-user; responding to a user’s input.

Oxford Dictionary

Armed with that definition, have a look at what you’ll find on far too many travel blogs. From the home page, you click on a link or button to the interactive map. You see a simple map that looks something like this screenshot. Some of the countries may be shaded to indicate that the blogger has traveled there.

A Map of world with countries shaded is not an interactive map
You’ll find maps that look like this on a lot of travel blogs

If you hover over a country, it’ll often show the country’s name and maybe a count of the number of blog posts, photos, or videos the travel blogger has created. However, if you click on a country, you’ll just be brought to another page with a list of post titles. In the best case scenario, you’ll also see a featured image and the first 20-40 words of the post, much like our All Posts page.

However, this still leaves me asking one question. If you have a map whose only purpose is to redirect visitors off of said map, why even have the map at all?

How Can Travel Bloggers Make Their Maps Interactive?

First and foremost, if you’re going to add a feature to your blog or website, it should serve more of a purpose than just redirecting visitors off of it. Your visitors should be able to gather all of the information they need without leaving the map. There is one exception, however. If you’re trying to display lengthy content, such as an entire blog post, don’t try to put the entire post in a pop-up window on the map. Nobody in their right mind is going to scroll through all of that.

Instead, you want to include key details and a link to the full blog post to make a fully interactive map. For a blog post, you’ll want to include at least four things.

  • Title of the post
  • A summary of the post or the first 20-25 words of it
  • The post’s feature image
  • A link to the full post
  • Anything else that’s relevant and important, such as the date, author, or location

Example: The Matt Gove Photo Visual Media Map

To best demonstrate a fully interactive map, let’s have a look at the Matt Gove Photo Visual Media Map. Instead of blog posts, the map includes datasets to display all of our photos and videos on a map. Notice how it includes all five requirements for full interactivity. As a user, you can easily explore our photos and videos on a map without having to leave the map. In addition, you can click on the link in the pop-up window to view the full album or video.

Best GIS Solutions for Travel Bloggers to Create Interactive Maps

As recently as 10 years ago, getting professional-quality interactive maps meant shelling out hundreds, if not thousands of dollars every year in GIS software licensing fees. Even worse, there were very few online GIS programs to choose from back then.

Thankfully, that has all changed. Today, you there are an plenty of free online mapping programs available. Many of these programs are both incredibly powerful and easy to set up. While it’s certainly not a requirement, I highly recommend investing in a developer to install, connect, and integrate the maps on your website. They’ll be able to connect your maps directly to your database. As a result, new content will automatically be added to your maps, allowing them to effortlessly scale with your business. It’ll cost more up front, but with a good developer, you’ll save money in the long run.

We’d love to help you get set up with your maps. If you want to discuss your project further or get a free quote to add maps to your travel blog, please get in touch with us today. Or if you’re still just kicking tires, please feel free to browse our catalog of GIS products and services.

Google Maps or Bing Maps

If you’re just starting out or are looking for a simple solution, Google or Bing Maps are great options! Both platforms allow you to create high-quality maps and embed them on your website free of charge. And best of all, you don’t need a developer. Instead, you’ll just copy and paste a short block of code into your website or blog.

Unfortunately, the simplicity of both Google and Bing Maps leave them with some downsides. Most notably, if you have a dynamic dataset or high traffic volumes, you’ll run into issues. Neither platform is built to display large datasets. As a result, you’ll struggle to scale your application and will run into high API fees to generate the basemaps as your organization grows.

Mapbox

Mapbox is a direct competitor of both Google and Bing Maps, but it offers so much more. As a result, I typically recommend that my clients use Mapbox over Google or Bing. Mapbox gives you much finer control over your mapping application, allowing you to scale it both up and down as your business evolves. And best of all, it’s completely free to use, unless you have more than 50,000 map loads per month.

Mapbox map of Boston, in a light grey color scheme
Screenshot of a Simple Mapbox Map of Boston, Massachusetts

However, my favorite Mapbox feature is its use of vector tiles, which allows you to display huge amounts (read: gigabytes) of data that load and respond extremely fast.

  • Fully customizable maps
  • Much friendlier pricing than Google or Bing
  • Supports both proprietary and open source data formats
  • Offers a software development kit to add your own functionalities and customizations
  • Developer optional to use maps or embed them on your website

Leaflet

Leaflet is a simple, yet powerful open source JavaScript library that displays two-dimensional maps on your website or blog. Because it’s hosted on your server, you don’t need to worry about API fees, regardless of how much or how little traffic you get. It’s lightweight, fast, powerful, and completely customizable. Furthermore, it has an extensive library of plugins and extensions if you need additional functionality. If you don’t know JavaScript, you’ll need a developer for the initial set up of your maps. You can easily connect Leaflet to nearly any type of database or data repository. As a result, Leaflet maps will easily scale with your business once it’s set up.

A popup box on a sample Leaflet interactive map
Screenshot of a Data Point on a Leaflet Map

Thankfully, Leaflet requires very little maintenance once you get it up and running. In fact, I tell my clients that it’s often more cost-effective to pay for Leaflet maintenance as you need it instead of paying a recurring monthly maintenance fee. Yes, there are obviously exceptions to that rule. However, for the vast majority of people, Leaflet is an extremely cost-effective way to add high-quality maps to your website or blog.

Cesium

Like Leaflet, Cesium is an open source JavaScript library that creates powerful 3D maps to “unleash the power of 3D data”. Their maps are engineered to maximize performance, precision, and experience. With Cesium, it often feels like there is no limit when it comes to what you can do with a map. In fact, Cesium also includes a timeline, so you could make the argument that its maps are four dimensional instead of three.

Furthermore, they’ve even created their own 3D vector tile format that lets you load enormous datasets in seconds. For example, check out Cesium’s demo that loads 3D models of nearly every building in New York City. It’s fast, fluid, and responsive. For additional demos, have a look at Cesium’s Use Cases. You’ll find examples from many different industries, applications, and regions.

Screenshot of a cockpit from a flight simulator built on Cesium's 3D mapping technology
Someone even built a flight simulator on the Cesium platform

You can get an incredible amount of power and functionality out of Cesium’s free base functionality. For the average travel blog, the free functionality is probably more than you need. However, if you want to harness its full potential, you should at least look into some of the paid add-ons for Cesium. Those paid add-ons will streamline your workflow and optimize your data. As a result, your users will ultimately have a better experience.

Terria

If you’re trying to decide between Leaflet or Cesium, why not use both? Originally developed as an open source platform for the Australian Government, Terria lets your users choose whether they want to view a two or three dimensional map. And you can probably see where this is going. Leaflet powers Terria’s two dimensional maps, which Cesium is behind its three dimensional maps.

The best feature of Terria, however, is its user interface. Easily organize and browse through a huge number of datasets. It uses Cesium’s 3D data optimization to ensure your map remains fast and responsive, even if your datasets are massive. Use Terria’s slider to compare datasets side-by-side. It even includes a feature for you to build stories with your data and share them with your audience.

I use Terria for all of my mapping needs, and also recommend it for most of my clients. Its power and responsiveness across all devices, including mobile phones, coupled with its flexibility and minimal programming required to set it up make it the optimal platform for me. My users and clients have never complained about being confused using Terria, and are often impressed at how easy it is to analyze huge amounts of data. And best of all, I can set it up so it scales up and down as I need without needing hardly any maintenance.

If you want your travel blog to stand out from the rest, adding fully interactive maps with Terria is one of the easiest and most cost-effective way to do so. To learn more or get started, please get in touch with us or browse our online resources.

ESRI ArcGIS Online

If you have a complex dataset, but would prefer not to hire a developer, ESRI’s ArcGIS Online may be the best solution for you. Yes, it does have licensing fees, but you’ll get much of the functionality of the other applications we’ve discussed without needing a developer to set them up for you. Like the other platforms, ArcGIS online can easily handle large numbers of complex datasets, and plot those data on maps that need just a copy and paste to embed in your website. Plus ESRI is widely considered to be the industry standard for anything related to GIS and maps. If anything goes wrong for you, they have excellent documentation and support.

If you’re looking for a real-world example of ArcGIS Online in action, you’ve probably seen them already. Since the COVID-19 pandemic began, most dashboards that display maps of COVID-19 data use ArcGIS online.

Interactive map on Johns Hopkins University's COVID-19 dashboard
ESRI ArcGIS Online powers many COVID-19 Dashboards, including the one by John’s Hopkins University

Summary

PlatformFreeAPI FeesDimensionsDeveloperDynamic Data
Google/Bing MapsYesOptional2D OnlyNot RequiredNo
MapboxYes> 50K loads/mo2D OnlyOptionalYes
LeafletYesNot Required2D OnlyRequiredYes
CesiumYesNot Required3D OnlyRequiredYes
TerriaYesNot Required2D & 3DRequiredYes
ArcGIS OnlineNoN/A2D OnlyNot RequiredYes

Travel Bloggers, Take Your Interactive Maps to the Next Level

Once you have your new interactive map set up on your website or blog, simply adding a link to it is not enough. In addition, you should strategically embed them on different pages of your website to give your users the most immersive experience. For example, you’ll find the Matt Gove Photo Visual Media map embedded both on the home page of this blog and the main photo gallery page on the Matt Gove Photo website, in addition to being embedded above. I’ll continue to add maps as we go forward, too.

Interactive map embedded on the Matt Gove Photo website
You’ll find the Matt Gove Photo Visual Media Map strategically embedded across our websites

To figure out where to embed your interactive maps, have a look at your website’s analytics. Are there pages that have a lot of page views? Is there a specific page people are navigating to that the map could benefit? Are your visitors getting confused and leaving your website instead of navigating somewhere the map could help? Is there a logical place for your maps in your navigation or sales funnel?

Finally, as a travel blogger, you shouldn’t plot just your blog posts on the interactive map. Geotag photo albums, videos, social media posts, guides, fun activities, scenic drives, and much more. Don’t be afraid to make multiple maps, either. Try to use a platform like Terria or ArcGIS Online that organizes your datasets in a logical manner and makes it easy to both add and remove data from the map. If that’s not an option, don’t overwhelm your users with too much data on a single map. That’s one of the best ways to drive visitors off of your website and directly into the arms of your competitors.

Conclusion

Fast, professional quality interactive maps are one of the best ways travel bloggers can stand out from the crowd. Interactive maps are easy and cost-effective to implement and maintain. They’re also incredibly effective at retaining your visitors’ engagement and keeping them on your website. It boggles my mind why so many travel bloggers haven’t taken full advantage of the incredible potential interactive maps present to both grow your audience and keep your existing followers coming back for more.

Are you ready to take the next steps with interactive maps and bring your website or travel blog to the next level? As avid travelers and data science experts who specialize in online GIS and mapping applications, we’d love to help you take that next step in your journey. I invite you to please browse our catalog of GIS and mapping services. Then, get in touch or book a free info session with us to discuss your specific project. We can’t wait to hear from you.

Top Photo: Chapman’s Peak Drive on the Matt Gove Photo Scenic Drives Map
Cape Town, Western Cape, South Africa

The post Travel Bloggers: How to Stand Out with Powerful Interactive Maps appeared first on Matthew Gove Blog.

]]>
How to Automate Region Mapping in TerriaJS with 49 Lines of Python https://blog.matthewgove.com/2021/10/15/how-to-automate-region-mapping-in-terriajs-with-49-lines-of-python/ https://blog.matthewgove.com/2021/10/15/how-to-automate-region-mapping-in-terriajs-with-49-lines-of-python/#comments Fri, 15 Oct 2021 16:00:00 +0000 https://blog.matthewgove.com/?p=3334 A little over a month ago, we examined the benefits of using region mapping in your TerriaJS applications. Region mapping allows you to reduce your GIS application’s data usage by over 99%, permitting you to display massive datasets on two and three-dimensional maps that load quickly and are highly responsive. […]

The post How to Automate Region Mapping in TerriaJS with 49 Lines of Python appeared first on Matthew Gove Blog.

]]>
A little over a month ago, we examined the benefits of using region mapping in your TerriaJS applications. Region mapping allows you to reduce your GIS application’s data usage by over 99%, permitting you to display massive datasets on two and three-dimensional maps that load quickly and are highly responsive.

Example region mapping in TerriaJS
Our COVID-19 Dashboard Map Makes Extensive Use of Region Mapping in TerriaJS

Despite the power and benefits region mapping offers, setting it up can be tedious, time consuming, and rife with type-o’s if you attempt to do it manually. This can be particularly frustrating if you have a large number of regions you want to map. Thankfully, there’s a much easier way. With just 49 lines of Python code, you can generate region maps of any size in just seconds, freeing up valuable time for you to focus on more important tasks.

What You’ll Need

You’ll need a few items to get started with your TerriaJS region mapping automation.

  • The ESRI Shapefile or GeoJSON file that you used to generate your Mapbox Vector Tiles (MVTs) for the region mapping
  • The Python script you’ll write in this tutorial
  • A Terminal or Command Prompt

What is a Region Mapping File

In TerriaJS, a region map consists of two JSON files.

  1. The actual region map
  2. The region mapping configuration file

The Python script we’re writing in this tutorial generates the actual region map. The region map tells TerriaJS the order that each polygon appears in the vector tiles. The configuration file instructs TerriaJS which vector tile parameters contain the region’s unique identifier, name, and more. You can easily write a short Python script to generate the configuration. However, unless you have a lot of region maps you’re generating, I find that the configuration is so short, it’s easier to do manually.

Convert all Shapefiles to GeoJSON Before Automating Region Mapping

If you have ESRI Shapefiles, convert them to GeoJSON before automating your region mapping. We do this for two reasons.

  1. Python can read and parse GeoJSON files natively. To parse shapefiles, you’ll need a third-party library such as GeoPandas.
  2. Mapbox’s Tippecanoe program, which generates the Mapbox Vector Tiles we use for region mapping, requires files to be input in GeoJSON format.

The GeoJSON should contain at least three properties for each feature. We covered these properties in detail in our previous article about region mapping, so I won’t repeat them here. You’re welcome to add as many properties as you want, but it should at the bare minimum contain the following three features.

  • A Feature ID, or FID.
  • A unique identifier you’ll use to identify the feature in the CSV data files you load into Terria.
  • The name of the feature

Most of our map tiles actually contain several unique identifiers that can be used. For example, countries actually have two sets of ISO (International Organization for Standardization) standards, plus a United Nations code that can be used to uniquely identify them. We can use any of these three, plus our own unique identifier to map them to our Mapbox Vector Tiles.

CountryISO Alpha 2 CodeISO Alpha 3 CodeUN Numeric Code
AustraliaAUAUS036
BrazilBRBRA076
CanadaCACAN124
FranceFRFRA250
MexicoMXMEX484
New ZealandNZNZL554
United StatesUSUSA840
Unique Country ID Examples

All right, let’s dive into the Python code.

First, Input Python’s Built-In json Library

The real magic of region mapping in TerriaJS is that all of the files are just JSON files, which are compatible with every popular programming language today. As a result, all we need to do the automation is Python’s json library. The json library comes standard with every installation of Python. We just need to import it. We will also be using Python’s os library to ensure the correct paths exist to output our region mapping files.

import json
import os

First, Write a Python Function to Extract the Feature ID from each Feature in the GeoJSON

This is because you need to go down several layers into the GeoJSON and we need this functionality a lot. It’s best practice to avoid string literals in programming, so we’ll use a function to extract the Feature ID from any given feature in the GeoJSON file.

def fid(f):
    return f["properties"]["FID"]

Define Your Input Parameters

We define the GeoJSON file name, the layer name in the TerriaJS region mapping configuration (regionMapping.json), and the property we’ll use as the unique identifier in our region mapping. Even if your MVT tiles have multiple unique identifier, you may only choose one for each region map. You’ll need to generate a second region map if you want to use multiple unique identifiers.

GEOJSON = "world-countries.geojson"
LAYER = "WORLD_COUNTRIES"
MAPPING_ID_PROPERTY = "alpha3code"
OUTPUT_FOLDER = "regionIds"

If you prefer not to have to manually edit the Python script every time you want to change files, you can easily update the Python script to extract this information from the filename or receive it through command line arguments.

Define the Output Path for Your Region Map File

I like to do this right away because it uses the input parameters. This both gets it out of the way so we don’t need to deal with it later and eliminates the need to scroll up and down looking for variable names if we were to define these at the end. We’ll structure our output paths so they can just be copied and pasted into Terria when we’re done. No renaming or reshuffling of files required.

geojson_filename = GEOJSON.replace(".geojson", "")
output_json_filename = "region_map-{}.json".format(geojson_filename)
output_fpath = "{}/{}".format(OUTPUT_FOLDER, output_json_filename)

In this example, the output filepath or our region mapping file will be regionIds/region_map-world-countries.json.

Read the GeoJSON into Python

Because GeoJSONs are just a specific type of JSON file, we can use Python’s built in json module to read and parse the GeoJSON file.

with open(GEOJSON, "r") as infile:
    raw_json = json.load(infile)

Sort the GeoJSON features by Feature ID

When you convert the GeoJSON to MVT tiles, Tippecanoe sorts the GeoJSON by Feature ID. For the region mapping to work correctly, the feature in the region map must appear in the exact same order as they appear in the Mapbox Vector Tiles. If they’re not in the exact same order, your data will be mapped to the wrong features in TerriaJS. What’s particularly insidious about this issue, is that it will appear as if your data is mapped to the correct county, even though it’s not.

When I first tried to set up region mapping for the World Countries vector tiles, I did not have the features in the right order. Data for the United States was mapped to Ukraine and labeled as Ukraine’s data. France’s data showed up in Fiji, and New Zealand’s data appeared to be Nepal’s. And that’s just to name a few. Except for a few countries at the top of the alphabet that start with “A”, this issue plagued every country. Once I sorted the GeoJSON features by Feature ID, the problem magically went away.

raw_features = raw_json["features"]
features = sorted(raw_features, key=fid)

Note here that we use the fid function we defined above to identify the Feature ID in the sorted() function.

TerriaJS Region Maps are just Arrays of Sorted Feature ID’s

To generate the region map for TerriaJS, all we need to do is just loop through the sorted features and create an array of the unique ID for each feature. Remember that the unique ID is different from the Feature ID. For world countries, TerriaJS comes with the Alpha 2 Codes built into it. For this tutorial, we’ll use the ISO Alpha 3 code as our unique identifier, which we defined in the MAPPING_ID_PROPERTY variable in the use input. If you’ve forgotten, the alpha 3 code is just a 3-letter code that identifies each country. For example, use “USA” for the United States, “CAN” for Canada, “JPN” for Japan, and so forth.

We’ll generate the region map with a simple for loop.

region_mapping_ids = []

for f in features:
    properties = f["properties"]
    region_map_id = properties[MAPPING_ID_PROPERTY]
    region_mapping_ids.append(region_map_id)

Assemble the full TerriaJS Region Mapping JSON in a Python Dictionary

TerrisJS region mapping files are breathtakingly simple. They require three parameters.

  • layer: The name of the layer in the TerriaJS Region Mapping configuration (regionMapping.json). We defined it above with the LAYER variable in the user input.
  • property: The name of the unique identifier in your vector tiles that you’ll use in the CSV files you load into Terria. For the world countries, we’re using the alpha2code identifier. We defined this in the user input using the MAPPING_ID_PROPERTY variable.
  • values: This is the the array of sorted feature ID’s we created in the previous section

It’s important to note that Python does not actually support JSON format. Instead, when you read a JSON file into Python, it actually converts the JSON into a Python dictionary. When we export the dictionary to JSON format, Python just simply does the conversion in the opposite direction.

Anyway, the TerriaJS region mapping JSON should look like this as a Python dictionary.

output_json = {
    "layer": LAYER,
    "property": MAPPING_ID_PROPERTY,
    "values": region_mapping_ids,
}

Also, don’t forget to create the output directory if it doesn’t exist. Your Python script will crash without it.

if not os.path.isdir(OUTPUT_FOLDER):
    os.makedirs(OUTPUT_FOLDER)

Finally, Write the TerriaJS Region Mapping JSON to a .json File

Thanks to Python’s json module, we can output the region mapping file with just a couple lines of code.

with open(output_fpath, "w") as ofile:
    json.dump(output_json, ofile, indent=4)

If you did everything correctly, the region mapping JSON you just created should look like this.

{
    "layer": "WORLD_COUNTRIES",
    "property": "alpha3code",
    "values": [
        "AFG",
        "ALB",
        "DZA",
        "AND",
        "AGO",
        "ATG",
        ...
    ]
}

But we’re not quite done, yet!

Add Your New TerriaJS Region Mapping to the Configuration File

The final step is to add your new region map to the TerriaJS Region Mapping configuration file. By default, the configuration file is located at data/regionMapping.json. However, if you can’t find it, it’s defined in the config.json file at the root of the Terria application. You are more than welcome to automate this, too, but I find that it’s so simple, it’s often easier to just do manually.

The configuration file instructs TerriaJS how to interpret each region map. You’ll need to include several parameters.

ParameterDescription
layerNameThe name of the layer in the Mapbox Vector Tiles. If you can’t remember it, check your vector tiles’ metadata.
serverThe URL from where your Mapbox Vector Tiles are served
serverTypeThe type of files the server is serving. For this tutorial, use MVT, which stands for “Mapbox Vector Tiles”.
serverMinZoomThe minimum zoom level of the vector tiles
serverMaxNativeZoomThe maximum zoom level the vector tiles support natively
serverMaxZoomThe maximum zoom level the vector tiles support
regionIdsFileThe path to the TerriaJS Region Mapping JSON we created in this tutorial
regionPropThe name of the property in the vector tiles that contains the unique identifier we’re using in the region map.
aliasesAny column headers in the CSV data file that TerriaJS should interpret as the unique identifier of your region map
descriptionA description of the region map
bboxThe bounding box of your vector tiles, in the format [west, south, east, north]
namePropThe name of the property in the vector tiles that contains the name of each feature

Using the layer we defined in the LAYER variable as the JSON key, your regionMapping.json configuration file should look like the following.

{
    "regionWmsMap": {
        ...
        "WORLD_COUNTRIES": {
            "layerName": "world-countries",
            "server": "https://yourvectortileserver.com/worldcountries/{z}/{x}/{y}.pbf,
            "serverType": "MVT",
            "serverMinZoom": 0,
            "serverMaxNativeZoom": 3,
            "serverMaxZoom": 8,
            "regionIdsFile": "data/regionIds/region_map-world-countries.json",
            "regionProp": "alpha3code",
            "aliases": ["alpha3code", "country_id", "COUNTRY_ID"],
            "description": "World Countries",
            "bbox": [
                -179.99999999999999999999999,
                -85,
                179.999999999999999999999999,
                85
            ],
            "nameProp": "name"
        },
        ...
    }
}

Create a Dummy CSV File to Test it Out

The funnest part of any project is seeing it all come to life once you’re done. To do this, create a dummy CSV file to test that your region mapping actually works. Pick a bunch of countries at random, assign them some random values as a dummy parameter, and see if they show up on the map.

alpha3codeValue
USA7
CAN4
DEU15
THA12
AUS9
BRA3

To load the file into TerriaJS, just click on the upload button at the top of the workbench, next to the “Browse Data” or “Explore Map Data” button. If the region mapping is working properly, you should see your data appear appear on the map.

Our dummy CSV data displayed on a choropleth world map
The Dummy CSV Data on a World Map

Conclusion

Region mapping is one of the most powerful and efficient ways to display enormous amounts of data in TerriaJS. Automating the region mapping process only saves you valuable time and makes your application even more powerful.

While manually region mapping feature sets such as the 50 US states or the roughly 200 world countries may initially seem manageable, it rapidly becomes a nightmare once you try to scale it up. Sticking just within the United States, what if instead of the 50 states, you were mapping America’s more than 3,000 counties? Our COVID-19 Dashboard map does just that. Or even worse, the United States has over 41,000 postal codes and more than 73,000 census tracts. Can you imagine having to assemble those region maps manually, or the opportunity for type-o’s manually entering tens of thousands of data points?

Instead, save yourself the time, money, and hassle. We’ve made the Python script available for free on Bitbucket so you can configure region maps of any size in TerriaJS in just seconds. And if you ever run into issues, we’re always here to help you with any questions you may have. Happy mapping!

Top Photo: Snow-Capped Sierra Nevada Provide a Spectacular Backdrop for Lake Tahoe’s Brilliant Turquoise Waters
Meeks Bay, California – February, 2020

The post How to Automate Region Mapping in TerriaJS with 49 Lines of Python appeared first on Matthew Gove Blog.

]]>
https://blog.matthewgove.com/2021/10/15/how-to-automate-region-mapping-in-terriajs-with-49-lines-of-python/feed/ 1
How to Set Up and Automate Region Mapping in Terria https://blog.matthewgove.com/2021/09/10/how-to-set-up-and-automate-region-mapping-in-terria/ Fri, 10 Sep 2021 16:00:00 +0000 https://blog.matthewgove.com/?p=3222 In today’s media-rich internet, load times are critical to your web application’s success. There is no industry for which that rings true more than the online mapping and GIS community. In the past, even small to medium-sized datasets were notorious for crippling web-based GIS applications and crashing web browsers. Thankfully, […]

The post How to Set Up and Automate Region Mapping in Terria appeared first on Matthew Gove Blog.

]]>
In today’s media-rich internet, load times are critical to your web application’s success. There is no industry for which that rings true more than the online mapping and GIS community. In the past, even small to medium-sized datasets were notorious for crippling web-based GIS applications and crashing web browsers. Thankfully, the leaders of the online GIS community have come up with a solution that I used to reduce data usage in my COVID-19 Map by 99.92%. And that’s just for one country. Today, you’re going to learn how to achieve similar results by setting up region mapping in Terria.

What is Region Mapping?

Region mapping is the process of using unique identifiers to map a dataset to a separate map geometry. While this geometry can be any polygon, line, or point on a map, it is most often some kind of physical or political boundary, ranging from state and country borders down to census tracts, neighborhoods, and property lines. The benefits of region mapping are profound for web-based GIS applications such as Terria and Cesium.

  • Load map geometries once in region mapped datasets vs every data point in datasets that do not use region mapping
  • In Terria, your dataset is stored in CSV format, which can opened with everyday tools such as Microsoft Excel
  • Vector tiles used in region mapping are typically between 1% and 10% the size of most other GIS formats, including Shapefiles, GeoJSON, and KML/KMZ.
  • Reuse vector tilesets over multiple datasets to easily scale your application up and down. The geometries I use to track COVID-19 can be reused to track weather, elections, Census data, population, and more.
  • Region mapping only loads the visible part of the map geometry in Terria. Without vector tiles, you must load the entire geometry, regardless of whether or not it’s visible.

Region Mapping Performance in Terria

So just how much of a performance boost do you get using region mapping? Let’s look at the actual performance of loading the geographies of the Canadian provinces that we use in our COVID-19 map. Keep in mind that these numbers are for loading the geometry only. They do not load any COVID-19 data.

ParameterGeoJSONRegion Mapping/Vector Tile
Entire Geometry Size141 MB26 MB
Whole Country Overview on Map141 MB643 kB
Whole Country Overview Load Time (100 mbit/sec)11.28 sec0.05 sec
Provincial Overview: Ontario141 MB281 kB
Ontario Overview Load Time (100 mbit/sec)11.28 sec0.02 sec

As you can see, the region mapping wins hands down. This effect only gets compounded once you add the COVID-19 data. If you’re not using region mapping, the map geometry needs to be loaded for every data point you have. When you take that 141 MB GeoJSON and multiply it by 550-plus days of the pandemic, you’re looking at about 70 GB of data for just the geometry alone! And that number gets bigger with each day that passes. With region mapping, you only load the 643 kB of geometry to see the overview of Canada, regardless of how many days the pandemic drags on for.

If it weren’t for region mapping, we would not be able to publish our COVID-19 maps because the large datasets would crash your browser.

I do want to point out that GeoJSON currently does not support time-series data. However, if you convert it to a format that does, such as CZML, you’re still looking at similar sizes and load times for the geometry.

What You’ll Need to Set Up Region Mapping in Terria

The Terria developers have thankfully made it very easy to set up region mapping. You’ll need a few tools to do it.

  • Your map geometry, in either GeoJSON format or a format that can be easily converted to GeoJSON, such as an Esri Shapefile.
  • A means to convert your geometry to GeoJSON. I recommend either a GIS program or Python GeoPandas.
  • Your favorite text editor
  • [Optional] Python, if you want to automate the process. We’ll cover automating this process in detail in a future tutorial.

Step 1: Configure Your GeoJSON or Shapefile

Before we convert our GeoJSON into vector tile format, we need to make sure each feature has the necessary parameters for region mapping. At the very minimum, each feature needs to have a feature ID and a unique ID that can easily identify it.

Set a Numeric ID for Each Feature in the GeoJSON

The feature ID is simply an incremental number starting at zero or one. We’ll use the feature ID to tell Terria the correct order that each feature appears in the vector tileset. Without this feature ID, your features will not display the correct dataset on the map unless they are in numerical or alphabetical order based on your ID. For example, let’s stick with the COVID-19 data and look at US States. Your GeoJSON is in alphabetical order by state name, and you’re using the 2-letter postal abbreviation as the unique ID.

Here’s where things go awry without a feature ID. If your GeoJSON is in alphabetical order by state name, Maine comes before Maryland. Likewise, Maryland comes before Massachusetts. However, using the postal abbreviations, that order changes. Massachusetts (MA) comes first, followed by Maryland (MD), and then Maine (ME). Because you specified the postal abbreviations as the unique ID for region mapping, Terria assumes that they are in alphabetical order by postal abbreviation. However, you have actually set them in alphabetical order by state name.

As a result, Terria will incorrectly determine that Maine’s data belongs to Massachsetts and vice-versa. If you clicked on Maine on the Map, the data table would be labeled as Maine, but it would actually display Massachusetts’ data. With a feature ID, you can actually tell Terria that the correct order of postal abbreviations is ME, MD, and then MA. We’ll circle back to this shortly.

Once you get your GeoJSON into the order that you want it to be in, go through an assign an incremental Feature ID to each feature in the GeoJSON. If you’re using the US States in alphabetical order by name, Alabama would be 1, Alaska would be 2, Arizona would be 3, and so on.

Include a Unique Identifier You’ll Use in Your Datasets

You’re more than welcome to use your Feature ID in the datasets. However, I really don’t recommend doing that because you won’t be able to remember them. Instead, assign them a standardized ID, such as an ISO, FIPS, or postal code. Let’s say you want to plot some data for the New England States. If you had to update the dataset several years down the road, which identifiers would be easier to recognize when you open the data file?

  • ME, NH, VT, MA, RI, CT
  • 21, 30, 46, 19, 39, 7

Interestingly, there’s still a problem using just the postal abbreviations as your unique identifier. What happens if you’re working across multiple countries? You’re bound to run into a conflict at some point where two countries use the same abbreviation. Take Massachusetts as an example. There are 23 countries that use “MA” as a state or provincial abbreviation.

To avoid those conflicts, simply use the ISO 3166-2 codes to identify each state. The ISO codes are standardized worldwide and prepend the 2-letter country abbreviation to the state or province abbreviation to make it a unique identifier worldwide. Even if you only plan to operate in just one country, you should still use the ISO codes. You never know when you may need to expand.

ISO 3166-2 CodeState or ProvinceCountry
US-MAMassachusettsUnited States
ES-MAMálagaSpain
BR-MAMaranhãoBrazil
CD-MAManiemaDem. Republic of Congo
JO-MAAl MafraqJordan

Our Recommended GeoJSON Configuration

Our region mapping files consist of four properties.

  • Feature ID
  • ISO Code or other standardized unique identifier
  • Unique numerical identifier that’s tied to our master geodatabase
  • The feature or entity’s name

We include two unique identifiers because it gives us flexibility in how we structure our datasets. While we primarily use the standardized identifier, there are certainly instances where it’s more beneficial to use the once that’s directly linked to our master geodatabase. If you aren’t using a master database, by all means omit that property.

When you put it all together, here’s an example of what the properties for Massachusetts look like in our GeoJSON.

{
    "FID": 19,
    "iso3166_2": "US-MA",
    "state_id": 4239,
    "name": "Massachusetts"
}

Step 2: Convert the GeoJSON to Mapbox Vector Tile Format

To convert the GeoJSON file to vector tile format so you can use it for region mapping in Terria, you’ll need to use a tool called Tippecanoe. Mapbox created Tippecanoe specifically for converting GeoJSON files to vector tiles, but it comes with a catch. It’s Unix based, so it only runs on Mac OS or Linux. It does not work on Windows. You can install Tippecanoe for free from GitHub.

Once you have Tippecanoe installed, it only takes one command from your command line to generate the vector tiles. We use the following command to generate vector tiles of state and provincial boundaries:

tippecanoe -z10 -e states --drop-densest-as-needed mexico-states.geojson -l mxstates

So what does this all mean? Let’s break it down. Please consult the Tippecanoe documentation or reach out to us for further details. We strongly recommend you standardize your naming conventions across all vector tile properties so they’re consistent and easy to reference in the future.

CommandMeaning
tippecanoeUse the Tippecanoe program
-z10Include zoom levels from 0 to 10 in the vector tile set
-e statesOutput raw vector tile files to a folder called “states”
--drop-densest-as-neededDrop the least visible features at each zoom level as needed to reduce file size
mexico-states.geojsonThe path to the GeoJSON we want to convert to vector tiles
-l mxstatesThe name of the layer you’ll reference in the region mapping in Terria

Upload Your Vector Tiles to a Mapping Server

Finally, you’ll want to upload your newly created vector tiles to a mapping server. However, be aware that they will not work “out-of-the-box” if you upload them to a standard web host. However, you can configure standard web hosts to serve vector tiles. We’ll cover setting up a mapping server in a future tutorial. In the meantime, if you want to just upload your vector tiles and be done with them, put them in an Amazon S3 bucket, which is configured to serve map tiles by default.

Step 3: Create Your Region Mapping File For Terria

In this step we will create the actual map that tells Terria which unique identifiers correspond to which geometries. The region map is stored in a JSON file. Creating the region map is very prone to type-o’s and human error, so I highly recommend automating this step with Python. We’ll cover automation in a future tutorial. Before we begin, you’ll need a few items from previous steps.

  • The name of the layer in your vector tiles from Step 2
  • Your unique identifier from Step 1 (not the Feature ID)
  • The order that each feature appears in the GeoJSON from Step 1

Remember the Feature ID we discussed in Step 1? If you followed our advice, you put the feature in the GeoJSON in numerical order by that Feature ID. To generate your region map, all you have to do is sort your GeoJSON by Feature ID and pull your unique identifiers out of the GeoJSON in that same order. It’s that simple.

Recall in Step 1 that we had a GeoJSON of US States sorted Alphabetically by state name. We used the ISO 3166-2 codes, which are based on postal abbreviations, as our unique identifier. To generate the region map, simply list the ISO 3166-2 codes for each state in alphabetical order by name. Here’s our region map showing the preserved alphabetical order of Maine, Maryland, and Massachusetts by name. I omitted the rest of the states to save space.

Example Using US State Geometries

{
    "layer": "usstates",
    "property": "iso3166_2",
    "values": [
        ...
        "US-KY",
        "US-LA",
        "US-ME",
        "US-MD",
        "US-MA",
        "US-MI",
        "US-MN",
        ...
    ]
}
ParameterDefinitionDefined In
layerLayer Name of the Vector Tileset, defined with the -l flag in TippecanoeStep 2
propertyThe key of your unique identifier you defined in the GeoJSONStep 1
valuesYour unique identifiers, in the exact order in which their features appear in the GeoJSONStep 1

If you haven’t already, create a folder called data in the root of your Terria application. Inside that folder, create another folder called regionIds. Save your region map in that regionIds folder. You can call it whatever you want, but I recommend naming it something that you can easily identify. For the US States name it something like region_map-us-states.json.

Step 4: Create or Update Your Terria Region Mapping Configuration

Now that we’ve created our region map, we need to tell Terria both to use it and how to use it. We’ll use a JSON file to provide Terria with our region mapping configuration. Terria actually comes with a default region mapping configuration for Australia, but I find it’s much easier if we just define our own instead of trying to add to Terria’s.

First, open the data folder you created in the root directory of Terria at the end of Step 3. Inside, create a file called regionMapping.json. This file will store the region mapping configuration for our Terria application. Open the regionMapping.json file in your favorite text editor. You’ll paste a key/value pair into it for each vector tileset you wish to use for region mapping. Set the keys so they’re easy to identify. In our COVID-19 map, we use keys such as US_STATES, US_COUNTIES, CA_PROVINCES, and WORLD_COUNTRIES.

Example Using US State Geometries

{
    "US_STATES": {
        "layerName": "usstates",
        "server": "https://yourmaptiledomain.com/us-states/{z}{x}{y}.pbf",
        "serverType": "MVT",
        "serverMinZoom": 0,
        "serverMaxNativeZoom": 3,
        "serverMaxZoom": 10,
        "regionIdsFile": "data/regionIds/region_map-us-states.json",
        "regionProp": "iso3166_2",
        "aliases": ["ISO3166_2", "US_STATE_ID"],
        "description": "US States",
        "bbox": [
            -179.999999999999,
            18,
            179.999999999999,
            72
        ],
        "nameProp": "name"
    }
}
ParameterDefinitionDefined In
layerNameLayer Name of the Vector Tileset, defined with the -l flag in TippecanoeStep 2
serverThe URL to your map tile serverStep 2
serverTypeSet to “MVT”, which stands for Mapbox Vector TilesN/A
serverMinZoomThe minimum zoom we defined when we created the vector tileset in TippecanoeStep 2
serverMaxNativeZoomThe maximum zoom level the server supports nativelyN/A
serverMaxZoomThe maximum zoom we defined when we created the vector tileset in TippecanoeStep 2
regionIdsFilePath to the region map JSON fileStep 3
regionPropThe key of your unique identifier that you defined in the GeoJSONStep 1
aliasesAdditional column headers in your dataset that Terria will recognize as your unique identifier. If you want the Terria’s drag and drop feature to work, one of these must be set to a unique value.Step 1
descriptionDescription of your region mapN/A
bboxBounding box that contains the map geometry, in degrees of latitude and longitude. It has the format [west, south, east, north].N/A
namePropThe key of the GeoJSON feature’s property that contains the feature’s nameStep 1

Step 5: Link Your CSV Data to the Region Map in the Terria Initialization File

The final step completes the link between your dataset and the vector tiles. We’ll update your initialization file to tell Terria which region map to use for each dataset in your catalog. You can find your initialization files in Terria’s init directory. All you have to do is add your CSV dataset to your initialization file. At the very minimum, you’ll need the following.

{
    "name": "United States: COVID-19 by State",
    "type": "csv",
    "url": "data/us-covid19-by-state.csv"
    "tableStyle": {
        "regionVariable": "iso3166_2",
        "regionType": "US_STATES",
        "dataVariable": "New Cases",
        "columns": Outside the Scope of this tutorial; Refer to Terria Documentation
    }
}
ParameterDefinitionDefined In
nameThe name under which your dataset will appear in Terria’s data catalogN/A
typeSet to “csv” because your data is in CSV formatN/A
urlThe URL where the CSV file containing your data is locatedN/A
tableStyle.regionVariableThe key of your unique identifier that you defined in the GeoJSONStep 1
tableStyle.regionTypeThe key of our region map in the Terria Region Mapping configurationStep 4
tableStyle.dataVariableThe header of the column that contains the data you want Terria to display when it initially loads your datasetN/A
tableStyle.columnsData formatting that’s outside the scope of this tutorial. Please consult the Terria documentation for details.N/A

If you’ve set everything up right, you’ll see each state get colored in when you use the ISO 3166-2 codes in your CSV file.

Example CSV Snippet of COVID-19 Data

ISO3166_2DateConfirmedDeadNew CasesNew Deaths
US-CA2020-07-01238,6816,1697,44987
US-CA2020-07-02246,5506,2657,86996
US-FL2020-07-01158,9973,5506,56345
US-FL2020-07-02169,1063,61710,10967
US-TX2020-07-01172,3862,5089,25550
US-TX2020-07-02179,1112,5476,72539
US-HI2020-07-019261890
US-HI2020-07-0194618200
US-NY2020-07-01394,07932,04362511
US-NY2020-07-02394,95432,06487521

Conclusion

Region mapping with vector tiles is the most powerful and efficient way to display massive datasets in web-based GIS platforms such as Terria and Cesium. Region mapping can seem daunting and even a little intimidating the first time you try it. Trust me, it took me a long time to figure it out, too, when I first started working with it.

If you’re struggling with region, please don’t hesitate to reach out to me personally or leave a comment below. The Terria developers are generally pretty responsive as well. Just a few small fractions of a second can be the difference between your mapping application taking off and failing. Are you willing to take that chance with slow, bloated file formats?

Top Photo: Region Mapping for the US, Canada, and Mexico on our COVID-19 Map
September, 2021

The post How to Set Up and Automate Region Mapping in Terria appeared first on Matthew Gove Blog.

]]>