The GeoStack Project

A Self-Study Beginner Course in Open Source Geospatial Programming for Data Scientists

Logo - The GeoStack Project

Introduction

Welcome to the Open Source Geospatial Programming Course!

So, now Go, Enjoy and… Learn things!

The repositories with software and content will be published here somewhere between July 25 and August 1, 2020.

Introduction video

See the introduction video ‘Presenting The GeoStack Project’ on the project’s YouTube Channel - The GeoStack Project.

Tip: visit our YouTube channel for lots of additional tutorial videos and use the playlists to view them.

Reader’s Guide

  1. Read this project’s home page first!
  2. Follow the advice in the section ‘Skill Levels’ below to check and get the required skill levels for this course.
  3. Follow the steps in the section ‘Getting Started’ below to get the course on your pc or laptop.
  4. Do the 1-day GeoStack Workshop to explore the world of Open Source Geospatial Programming by building the Workshop Virtual Machine using the installation scripts and then by working through the programming assignments to build the simplified 2D Map Viewer webapplication.
  5. Explore the overview videos and tutorial clips on the YouTube channel for the different parts of the full course.
  6. Do the full 10-day GeoStack Course by first building the Course VM the easy way with installation scripts to explore and then building the Course VM the manual way and get the 3 webapplications up and running.
  7. Finally experiment with loading your own datasets and extra digital topographical maps from OpenStreetMap and of course by modifying and extending the webapplications to learn how to build your own dataset visualizations.

Learning Goals

The learning goals of this course are to create and run a free and light-weight Open Source Geospatial Software Stack, the GeoStack for short, in a Virtual Machine (VM) with Ubuntu Linux on a stand-alone pc or laptop and to program webapplications to visualize geospatial datasets on locally hosted topographical digital maps from OpenStreetMap and OpenSeaMap.

So this course is about downloading some manuals, software and data sets from the internet, adjust it so it can run on the pc in ‘airplaine mode’ without an internet connection and then learn how to install and configure the GeoStack and program Geospatial Web Applications by using 100% Open Source Software, Open Standards and Open Content!

There are 3 learning goals:

  1. End goal: Programming Webapplications in TypeScript
  2. Intermediate goal: Programming Middleware in Python
  3. Starting goal: Programming a Client / Server Software Architecture

The end goal is to learn to program three end-user webapplications in TypeScript using the Angular framework in combination with the geospatial JavaScript frameworks OpenLayers and Cesium to plot data from datasets on digital topographical maps that the end-user can run from a webbrowser:

  1. a Dataset Dashboard to visualize datasets of GPS track logs that are stored as JSON documents in MongoDB;
  2. a 2D Map Viewer with OpenLayers to visualize the datasets on 2D digital topographical maps from OpenStreetMap (OSM);
  3. a 3D Map Viewer with Cesium to visualize the datasets on 3D maps with Digital Elevation Models (DEM height files).

The intermediate goal is to learn how to build three web services with an API for the web applications to use.

The starting goal is to learn about the approximately 50 software components required in the Client / Server software architecture you will need to download, install, configure, program and run to be able to use the end-user web applications in your web browser.

Workflow

In order to achieve these learning goals the activities are of course in the reverse order of the learning goals:

  1. Building an Ubuntu Linux Virtual Machine: first you will learn to download all the required datasets and software packages to be stored and installed on your pc or laptop in a Virtual Machine running Ubuntu Linux in VirtualBox.
  2. Data Storage: then you will learn to store the digital topographical maps in the relational spatial database PostgreSQL + spatial PostGIS extension and convert the GPS track log datasets from CSV and GPX files to the JSON dataformat to store the data as JSON documents in the MongoDB datastore.
  3. Programming: finally you will learn how to program in Python and TypeScript in the Atom programming editor and run the web services and web applications locally in your web browser, without the further need for an internet connection (running in ‘airplane mode’).

Study time

The course consists of a 1-day Workshop followed by a 10-day Course when you have some basic programming skills in Python and preferably in TypeScript or JavaScript too.

Don’t worry if it takes you a little longer because you will get there with the user friendly installation and programming manuals with lot’s of screenshots and the Python source code files in Jupyter Notebooks and example scripts in Python and TypeScript with lot’s of inline comments!

If you are on the other hand a real beginner in programming, you might want to invest some extra well spent study time first as a newbie to get to the required skill levels to follow this course as per the advice below in the section ‘Skill levels’!

Skill levels

1 - Basic skills in Python programming are a must!

2 - Basic knowledge in TypeScript / JavaScript + Angular is an advantage

3 - Some knowledge of Virtual Box, Ubuntu Linux, MongoDB, Docker, OpenLayers and Cesium comes in handy of course!

Course overview

Course structure To get fast situational awareness there is a 1-day GeoStack Workshop to build the Workshop VM with installation scripts which is documented in both a manual and in a series of presentation files to carefully read to get step-by-step to a working GeoStack and basic end-user webapplication.

About the 1-day GeoStack Workshop To learn most of the components of the Development Environment, which is described below, to install the required server software for the GeoStack and to program a webapplication this workshop will take you through the whole process to get familiar with the server and client software architecture for geospatial programming.

The steps will include installing a spatial database, importing topographic maps from OpenStreetMap, install MongoDB as a datastore for JSON documents with geospatial data, prepare and import geospatial datasets in MongoDB, install a Python webserver with a middleware Flask webapplication for data handling, install a NGINX webserver and finally program an end-user webapplication to visualize the dataset on a digital topographical map.

If you are not yet familiar with the components of the Development Environment then just take a day extra to read a little more.

If you’re in a hurry (not recommended!) or just curious then simply run the installation scripts to get to a working GeoStack and webapplication more quickly.

Building the Workshop VM is documented in the cookbook ‘Creating the GeoStack Workshop VM’. The cookbook ‘Workshop Assignments’ introduces the programming assignments with lots of background information for beginners! The programming assignments themselves are of course in the Jupyter Notebooks, source code files and configuration files.

About the 10-day GeoStack Course In the GeoStack Course there are many manuals, cookbooks, installation scripts and programming examples in Python to learn the details of installing, configuring, using and programming all components in the entire GeoStack. The same goes for using the datasets en programming the end-user webapplications in TypeScript + Angular and JavaScript.

This will take a serious time investment of 10 - 20 or even 30 days of study time to read and experiment with the software and datasets but after that you will have build a GeoStack VM from scratch and you know how geospatial web applications work!

You will also learn to use Docker Containers to run applications and Docker Volumes to host data. For topographical sea maps you will learn how to add the OpenSeaMap data of nautical objects to your spatial database PostgreSQL/PostGIS and visualise these objects with OpenLayers in a data layer overlay to create a digital nautical map.

At the end of the course you will have 3 working end-user web applications that run in your web browser to visualize geospatial datasets as basic programming examples for future programming projects and knowledge building:

  1. a Dataset Dashboard with graphs plotted with ChartJS to visualize information on the datasets stored in MongoDB;
  2. a 2D map viewer web application with OpenLayers to plot geospatial data on OpenStreetMap and OpenSeaMap maps;
  3. a 3D map viewer web application with Cesium to plot data in a 3D map projection with the map showing a terrain elevation model to get a nice visualization when plotting datapoints with a height coordinate. So it’s all about plotting (x,y,z) coordinates instead of just (x,y) location positions.

Building the Course VM is documented in the cookbook ‘Creating the GeoStack Course VM’.

The full GeoStack Course is documented in the following set of detailed Cookbooks and Programming manuals:

  1. Cookbook-ETL-Process-With-Datasets
  2. Cookbook-Data-Modeling-in-MongoDB
  3. Cookbook-A-Secure-Nginx-Webserver-With-ModSecurity
  4. Manual-Creating-the-Python-Flask-API-Middleware
  5. Manual-Creating-a-Base-application-using-AngularJS
  6. Manual-Creating-a-Dataset-Dashboard
  7. Manual-Creating-an-2-Dimensional-Map-Viewer
  8. Manual-Creating-an-3-Dimensional-Map-Viewer

Environment

The Development Environment in this course will learn you about the following subjects:

1) Virtual Machine hosting: Virtualbox (Installation and Use).

2) The Linux Operating System: Ubuntu LTS (Long-Term Support version).

3) The programming languages used are Python and TypeScript/JavaScript.

4) Declarative languages: SQL for relational databases and for web pages of the end-user web applications HTML5 and CSS3 are used.

5) Compartmentalize the GeoStack components using Docker:

6) The ETL-Process (Extract, Transform and Load):

7) Working with different datasets:

8) Working with different file formats to store geospatial data:

9) Explore and Transform datasets:

10) Storing datasets in a NoSQL datastore: learn to use the JSON document datastore MongoDB for…

11) Storing datasets in a SQL datastore: learn to use the RDBMS PostgreSQL in combination with the spatial PostGIS extensions for storing RAW OpenStreetMap data and storing geospatial data from ShapeFiles:

12) Creating a Tile Server using TileStache and Gunicorn:

13) Creating a middleware web service with an API using the Python micro web services framework Flask and a Gunicorn webserver:

14) Creating a Secured NGINX web server with the ModSecurity web application firewall and the OWASP provided basic ruleset.

15) Creating 3 end-user webapplications using the JavaScript web application framework Angular in combination with TypeScript:

Getting started

Download instructions

  1. Go to the top of the web page and click the blue button ‘Download Manuals’.
  2. Save the ZIP file that contains the repository GeoStack-Manuals.
  3. Unzip the ZIP file which will give you the folder GeoStack-Manuals.
  4. In the folder you will find 2 PDF files: Creating-the-Workshop-VM.pdf and Creating-the-Course-VM.pdf.

Work instructions

  1. Start with the 1-day GeoStack Workshop by following the document Creating-the-GeoStack-Workshop-VM.pdf.
    • Here you will learn if you need to build some extra skills first.
    • After building the VM start the file manager Nautilus by clicking the file cabinet icon on the left in the Favorites menu.
    • Then in the Home folder of the GeoStack account on the VM, go to the folder GeoStack-Workshop.
    • Follow the programming assignments in the file Cookbook-Workshop-Assignments.pdf.
  2. Work through the full 10-day GeoStack Course by following the document Creating-the-GeoStack-Course-VM.pdf.
    • There are two ways of building the Course VM:
      • 1 - Automatic installation by using the supplied set of installation scripts.
      • 2 - Manual installation by working your way through the large set of 8 detailed course cookbooks and manuals.

Advice: first build the Course VM the easy way with the installation scripts to explore and then do it the hard way by really learning some serious Systems & Network Management in building the GeoStack and programming the web applications!

Note: in the manuals on how to create the VM’s you will find detailed instructions about:

  1. cloning the git repositories GeoStack-Workshop and GeoStack-Course;
  2. the download instructions for the GPS track log datasets;
  3. the instructions to export, download and import the map and terrain elevation datasets required for visualizing digital topographical maps in OpenLayers and Cesium.

Programming tips

Lowering the Learning Curve

1) First of all: read the weblinks on this web page! They paint the picture! Then read the cookbooks, manuals etc.

2) Second: study the installation scripts and example scripts!

3) Third: remember in programming ‘a point is not a comma’!

4) Finally: make copies of files before you change them!!!

–> Make it a habit to use a file copy and you’ll be safe and save yourself a lot of time!

Scope limitations

1) Educational goal: the software and content of this project are only intended to learn about programming geospatial dataset processing in Python and programming webapplications in TypeScript for the visualization of this geospatial data on digital topographical maps.

2) Software selection: this course focuses on creating an Open Source geospatial software stack, the GeoStack, to develop and run webapplications for visualization of geospatial datasets. There are more Open Source Software products that can provide the same functionality as the products chosen in this course.

3) Complexity reduction: some well known software products are left out of scope in order not to complicate things more than necessary for an already fairly complex beginner course, like:

History

Development

This programming course was originally created by computer science student Pieter Lems of the Rotterdam University of Applied Science (Hogeschool Rotterdam) as the result of his 3rd year Bachelor internship assignment (201909-202001) in practical programming for data scientists.

Motivation

The motivation to create this course is to collect and combine all the very much fragmented information from many manuals, (video)tutorials, applications, programming languages and frameworks, file formats etc. into one documentation set for beginners in data science, programming and systems managment to learn to download, install, program and use all the software components and datasets in one logical workflow to get:

-----> 'Working Software at the End of the Day!' <-----


Licenses and Copyrights

This project is published by the MoD-NL as an Open Source Software educational project with Open Content for easy re-use.

The software of The GeoStack Project is released under the European Union Public License (EUPL), Version 1.2.

The content of The GeoStack Project is released under the Creative Commmons Attribution License (CC BY), Version 4.0.

Terms of Service

Take also careful notice of the following Terms of Service applicable to the use of The GeoStack Project software and content available in:

Third Party Licenses

Any and all datasets, software and online services used in this course from third parties are (re-)used under their own respective licenses.

Attribution guidelines

Please refer for attribution to this project as: The GeoStack Project (https://The-GeoStack-Project.github.io)

If you refer to specific documents or videos of the project please follow these Creative Commons guidelines on ‘Best practices for attribution’ to compose your weblinks correctly.

Disclaimer

You are responsible for your use of the software and content of The GeoStack Project and any datasets, software and online services from third parties used in The GeoStack Project.

Some legal points of special interest, in short terms:

1) The software and content of The GeoStack Project is provided ”as is,” and ”as available”.

2) You agree that your use of The GeoStack Project software and content is your own responsibility and any use is at your own risk.

3) Any other application or use of the software and content other than for educational purposes is not intended and not advised!

Contributing

Please take special notice of the GitHub Terms of Service for contributions under the repository licenses, including contributions made to The GeoStack Project software and content in any of its repositories.

If you contribute software or content to The GeoStack Project in other ways than by means of GitHub, you also agree that the same software and content licenses (EUPL Version 1.2 and CC BY Version 4.0) will be applicable to your contributions as well.

Resources

For the project we use two resources: this GitHub account and a Google account.

GitHub account - the home of the project

Click the blue buttons at the top of this webpage to view the project’s public repositories and check the Issue Tracker, the Wiki pages or the latest News.

Overview GitHub Repositories - The project has 5 repositories and for two repositories there is a convenient blue download button at the top of this webpage.

1) GeoStack-Manuals: the blue download button offers a ZIP file download of the GeoStack-Manuals repository with the two manuals to start building the Virtual Machines for the Workshop and the Course.

2) GeoStack-Project-Files: the blue download button offers a ZIP file download of the GeoStack-Project-Files repository with the text editor source documents, document images, document cover design and project logo for all the PDF files of the workshop and course. Just to save you a lot of time in case you want to re-use content!

3) GeoStack-Workshop: the repository for the 1-day Workshop that you learn to download in your Linux Virtual Machine when following the cookbook ‘Creating the Workshop VM’.

4) GeoStack-Course: the repository for the 10-day Course that you learn to download in your Linux Virtual Machine when following the cookbook ‘Creating the Course VM’.

5) The-GeoStack-Project.github.io: the repository for this home page.

Google account - for videos and project team support

The project uses two main services:

1) YouTube Channel

2) Drive: for backup of the project

Participation

Contact

The project team only uses the Issue Tracker for contact:

Support

Currently we are only accepting tickets in the Issue Tracker but with no promises to response times or realisation.

Pull requests

Currently we are not accepting pull requests from GitHub users other than the current team members.

New Team Members

The project is currently published as-is with a project team of only a few volunteers for very, very low maintenance without making any promises.

New team members are needed to make this project an active Open Source project because, with around 50 software and content components that will be updated over time, the project is expected to have a limited shelf life of around 12 months and without maintenance the project will need to be archived then.

Currently the project has not yet a way to welcome new volunteers but the information for that will be published here.


Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.