Point clouds are rendered as individual points by default. Depending on the zoom level and density of the points, you might not be able to get a full picture of the map.
Rendering points as surface enables you to have a better understanding of the data before trying to do any analysis or processing. This has been possible in 3D map views for a couple of QGIS releases, now we have added the functionality also in 2D map views.
The feature generates a surface using triangulation on-the-fly from the points using the same symbology. Below you can see the difference between a point cloud rendered as individual points and rendered as a surface:
The good news is that rendering as a surface also works well with global map shading, allowing users to get a nice hillshade:
To enable this feature, you need to check the option for Render as a Surface (Triangulate) under the Layer Styling panel.
Pro-tip: if the on-the-fly rendering as a surface takes too long to render, try increasing the Maximum error: for example 0.6 mm instead of the default 0.3 mm.
Previously, point cloud data visualisation in QGIS was limited to rendering all points with a uniform size and opacity. This made it difficult to differentiate between different point classes and highlight specific features of interest. To address this issue, we have introduced a new feature that allows users to customise the point size and opacity for each point cloud data class. This provides a flexible way for visualising point cloud data, allowing users to highlight specific point classes, e.g. by increasing the point size.
Effectively navigating and visualising large-scale 3D datasets can be challenging on PCs with limited resources. To address this issue, we introduced a new feature that allows users to interactively limit the 3D scene, enabling them to focus on specific areas of interest. This new feature, conveniently accessible from the toolbar, eliminates the need for tedious manual entry of coordinates for bounding boxes. Instead, users can simply drag and draw a box around the desired area, instantly restricting the 3D scene to that specific extent. This interactive approach significantly enhances the user experience and streamlines the process of exploring and analysing 3D data:
Creating and viewing 3D maps in QGIS with the correct camera location and angle, scene tilt, light, and other parameters can be a time-consuming and error-prone process. This is because it requires users to manually adjust these settings, often through trial and error. However, with the introduction of the new 3D map view API in QGIS, Python plugins can now automate this process, making it much easier, consistent and more efficient to create high-quality 3D maps that are visually appealing and informative.
# List all open 3D map canvases
iface.mapCanvases3D()
# [<qgis._3d.Qgs3DMapCanvas object at 0x7f23491b5e10>]
canvas = iface.mapCanvases3D()[0]
Now let’s try something more complicated:
# Let's change some settings!
ms = canvas.mapSettings()
ms.setEyeDomeLightingEnabled(True)
ms.setBackgroundColor(QColor('beige'))
ms.setTerrainMapTheme('dtm')
ms.setFieldOfView(110)
# Move the camera to look at specific map coordinates in layer's CRS
cam = canvas.cameraController()
mapPoint = QgsVector3D(-498175.92, -1205400.58, 210)
worldPoint = ms.mapToWorldCoordinates(mapPoint)
cam.setLookingAtPoint(worldPoint, 60, 45, 100)
# Create four new 3D map views
c1 = iface.createNewMapCanvas3D('South View')
c2 = iface.createNewMapCanvas3D('West View')
c3 = iface.createNewMapCanvas3D('North View')
c4 = iface.createNewMapCanvas3D('East View')
# Apply settings to all open 3D map views
for canvas in iface.mapCanvases3D():
canvas.mapSettings().setEyeDomeLightingEnabled(True)
# Define a camera pose to update the views' cameras
pose = QgsCameraPose()
pose.setCenterPoint(QgsVector3D(0, 210, 0)) # This is in 3D world coordinates
pose.setDistanceFromCenterPoint(100)
pose.setPitchAngle(75) # Tilt the camera by 75 degrees
pose.setHeadingAngle(0) # Looking towards North
c1.cameraController().setCameraPose(pose)
pose.setHeadingAngle(90) # Towards East
c2.cameraController().setCameraPose(pose)
pose.setHeadingAngle(180) # Towards South
c3.cameraController().setCameraPose(pose)
pose.setHeadingAngle(270) # Towards West
c4.cameraController().setCameraPose(pose)
# We can set the 3D map views 2D extent to always match the main 2D canvas one
# Our 3D views get updated while zooming/panning the main 2D canvas
canvas = iface.mapCanvas()
canvas.extentsChanged.connect(lambda :c1.mapSettings().setExtent(canvas.extent()))
canvas.extentsChanged.connect(lambda :c2.mapSettings().setExtent(canvas.extent()))
canvas.extentsChanged.connect(lambda :c3.mapSettings().setExtent(canvas.extent()))
canvas.extentsChanged.connect(lambda :c4.mapSettings().setExtent(canvas.extent()))
LAS/LAZ/COPC point clouds have a classificationFlags
attribute that stores four types of information (Synthetic, Keypoint, Withheld, and Overlap) in a single value. This saves space, but it makes it difficult to use the information for styling or filtering, as you need to write complex expressions.
To make it easier to use, we are following the approach introduced in PDAL 2.6: the classificationFlags
attribute gets replaced with four separate attributes: Synthetic, Keypoint, Withheld, and Overlap. This will make it easier to include these attributes in styling and filtering expressions.
To improve the performance of point cloud rendering in QGIS, we introduced a new caching system to minimise the need for repeated decompression of LAS node data while panning or zooming. This caching mechanism efficiently stores decompressed points for each unique combination of layer URI, node, requested attributes, filter extent, and filter expression. This enables rapid rendering of previously cached nodes, significantly enhancing the overall rendering performance in 2D and 3D maps.
Performance can vary depending on actual data, but on a local sample COPC file, it renders 7 times faster with this change.
Viewing mesh data has been possible through styling, plotting or using the Identify tool. But now you can also create labels on mesh surfaces or vertices similar to vector layers.
To display labels for your mesh data, simply open the styling panel and enable labels for:
Below is an example of mesh Z values at vertices (yellow) and mesh areas at cell centre (purple):
Do you want to see more improvements and new features in QGIS? Do not hesitate to contact us to discuss your requiremnets.
]]>In this blog post, you can read more about how to work with this feature, where to get data and how to display your maps in 2D and 3D. For a video demo of this feature, you can watch Nyall Dawson’s presentation on Youtube.
3D tiles are a specification for streaming and rendering large-scale 3D geospatial datasets. They use a hierarchical structure to efficiently manage and display 3D content, optimising performance by dynamically loading appropriate levels of detail. This technology is widely used in urban planning, architecture, simulation, gaming, and virtual reality, providing a standardised and interoperable solution for visualising complex geographical data.
Examples of 3D tiles:
To be able to use 3D tiles in QGIS, you need to have QGIS 3.34 or later. You can add a new connection to a 3D tile service from within the Data Source Manager under Scene:
Alternatively, you can add the service from your Browser Panel:
To test the feature, you can use the following 3D tiles service:
Name: 3D Tiles example
URL: https://pelican-public.s3.amazonaws.com/3dtiles/agi-hq/tileset.json
You can then add the map from the newly generated connection to QGIS:
By default, the layer is styled using texture, but you can change it to see the wireframe mesh behind the scene:
You can change the mesh fill and line symbols similar to the vector polygons. Alternatively, you can use texture colors. This will render each mesh element with the average value of the full texture. This is ideal when dealing with a large dataset and want to get a quick overview of the data:
To view the data in 3D, you can open a new 3D map. Similar to 2D map, by zooming in/out, finer resolution tiles will be fetched and displayed:
Cesium ion is a cloud-based platform for managing and streaming 3D geospatial data. It simplifies data management, visualisation, and sharing.
To add 3D tiles from Cesium ion, you need to first sign up to their service here: https://ion.cesium.com/tokens
Under Asset Depot, you will see a catalogue of publicly available datasets. You can also upload your own 3D models (such as OBJ or PLY), georeference them and get them converted to 3D tiles.
You can also add one of the existing tile service under https://ion.cesium.com/assetdepot and select the tile service and then click on Add to my assets:
You can use the excellent Cesium ion plugin by North Road from the QGIS repository to add the data to QGIS:
In addition to accessing Google Photorealistic 3D tiles from Cesium ion, you can also add the tiles directly in QGIS. First you will need to follow the instructions below and obtain API keys for 3D tiles: https://developers.google.com/maps/documentation/tile/cloud-setup
During the registration process, you will be asked to add your credit card details. Currently (November 2023), they do not charge you for using the service.
Once you have obtained the API key, you can add Google tiles using the following connection details:
When dealing with large scenes, map extents should be set to a smaller area to be able to view it in 3D. This is the current limitation of QGIS 3D maps as it cannot handle scenes larger than 500 x 500 km.
To change the map extent, you can open Project Properties and under View Settings change the extent. In the example below, the map extent has been limited only to a part of London, so we can view Google Photorealistic tiles in the 3D map without rendering issues.
If you are handling a large dataset, it is recommended to increase network cache size to 1 GB or more. The default value in QGIS is much lower and it results in slower rendering of the data.
When you try to overlay other data sets on top of a global 3D tiles, the vertical datum might not match and hence you will see the data in the wrong place in a 3D map. To fix the issue, you may need to use elevation offsetting to shift the data along the Z axis under Layer Properties:
This is the first implementation of the 3D tiles in QGIS. For the future, we would like to add more features for handling and creation of the 3D tiles. Our wishlist in no particular order is:
If you would like to see those features in QGIS and want to fund the efforts, do not hesitate to contact us.
]]>Date: Monday, June 26, 2023 at 14:00 GMT
Duration: 30 minutes + 15 minutes Q&A session
Speaker: Martin Dobias, CTO at Lutra Consulting with more than 15 years of QGIS development experience
Point clouds are an increasingly popular data type thanks to the decreasing cost of their acquisition through lidar surveys and photogrammetry. On top of that, more and more national mapping agencies release high resolution point cloud data (spanning large areas and consisting of billions of points), unlocking many new use cases.
This webinar will summarize the latest QGIS release 3.32 and the addition of tools for point cloud analysis right from QGIS Processing toolbox: clip, filter, merge, export to raster, extract boundaries and more - all backed by PDAL library that already ships with QGIS, without having to rely on third party proprietary software.
This work was made possible by the generous donations to our crowdfunding.
]]>Lidar surveys of larger areas are often multi-terabyte datasets with many billions of points. Having such large datasets represented as a single point cloud file is not practical due to the difficulties of storage, transfer, display and analysis. Point cloud data are therefore typically stored and distributed split into square tiles (e.g. 1km x 1km), each tile having a more manageable file size (e.g. ~200 MB when compressed).
Tiling of data solves the problems with size of data, but it introduces issues when processing or viewing an area of interest that does not fit entirely into a single tile. Users need to develop workflows that take into account multiple tiles and special care needs to be taken to deal with data near edges of tiles to avoid unwanted artefacts in outputs. Similarly, when viewing point cloud data, it becomes cumbersome to load many individual files and apply the same symbology.
Here is an example of several point cloud tiles loaded in QGIS. Each tile is styled based on min/max Z values of the tile, creating visible artefacts on tile edges. The styling has to be adjusted for each layer separately:
In the GIS world, many users are familiar with the concept of virtual rasters. A virtual raster is a file that simply references other raster files with actual data. In this way, GIS software then treats the whole dataset comprising many files as a single raster layer, making the display and analysis of all the rasters listed in the virtual file much easier.
Borrowing the concept of virtual rasters from GDAL, we have introduced a new file format that references other point cloud files - and we started to call it virtual point cloud (VPC). Software supporting virtual point clouds handles the whole tiled dataset as a single data source.
At the core, a virtual point cloud file is a simple JSON file with .vpc extension, containing references to actual data files (e.g. LAS/LAZ or COPC files) and additional metadata extracted from the files. Even though it is possible to write VPC files by hand, it is strongly recommended to create them using an automated tool as described later in this post.
On a more technical level, a virtual point cloud file is based on the increasingly popular STAC specification (the whole file is a STAC API ItemCollection). For more details, please refer to the VPC specification that also contains best practices and optional extensions (such as overviews).
We have added support for virtual point clouds in QGIS 3.32 (released in June 2023) thanks to the many organisations and individuals who contributed to our last year’s joint crowdfunding with North Road and Hobu. The support in QGIS consists of three parts:
Those who prefer using command line tools, PDAL wrench includes a build_vpc command to create virtual point clouds, and all the other PDAL wrench commands support virtual point clouds as the input.
In this tutorial, we are going to generate a VPC using the new Processing algorithm, load it in QGIS and then generate a DTM from terrain class. You will need QGIS 3.32 or later for this. For the purpose of this example, we are using the LiDAR data provided by the IGN France data hub.
In QGIS, open the Processing toolbox panel, search for the Build virtual point cloud (VPC) algorithm ((located in the Point cloud data management group):
In the algorithm’s window, you can add point cloud layers already loaded in QGIS or alternatively point it to a folder containing your LAZ/LAS files. It is recommended to also check the optional parameters:
Calculate boundary polygons - QGIS will be able to show the exact boundaries of data (rather than just rectangular extent)
Calculate statistics - will help QGIS to understand ranges of values of various attributes
Build overview point cloud - will also generate a single “thinned” point cloud of all your input data (using only every 1000th point from original data). The overview point cloud will be created next to the VPC file - for example, for mydata.vpc, the overview point cloud would be named mydata-overview.copc.laz
After you set the output file and start the process, you should end up with a single VPC file referencing all your data. If you leave the optional parameters unchecked, the VPC file will be built very quickly as the algorithm will only read metadata of input files. With any of the optional parameters set, the algorithm will read all points which can take some time.
Now you can load the VPC file in QGIS as any other layer - using QGIS browser, Data sources dialog in QGIS or by doing drag&drop from a file browser. After loading a VPC in QGIS, the 2D canvas will show boundaries of individual files - and as you zoom in, the actual point cloud data will be shown. Here, a VPC loaded together with the overview point cloud:
Zooming in QGIS in 2D map with elevation shading - initially showing just the overview point, later replaced by the actual dense point cloud:
In addition to 2D maps, you can view the VPC in a 3D map windows too:
If the input files for VPCs are not COPC files, QGIS will currently only show their boundaries in 2D and 3D views, but processing algorithms will work fine. It is however possible to use the Create COPC algorithm to batch convert LAS/LAZ files to COPC files, and then load VPC with COPC files.
It is also worth noting that VPCs also work with input data that is not tiled - for example, in some cases the data are distributed as flightlines (with lots of overlaps between files). While this is handled fine by QGIS, for the best performance it is generally recommended to first tile such datasets (using the Tile algorithm) before doing further display and analysis.
Now that we have the VPC generated, we can run other processing algorithms. For this example, we are going to convert the ground class of the point cloud to a digital terrain model (DTM) raster. In the QGIS Processing toolbox, search for Export to raster algorithm (in the Point cloud conversion group):
This will use the Z values from the VPC layer and generate a terrain raster based on a user defined resolution. The algorithm will process the tiles in parallel, taking care of edge artefacts (at the edges, it will read data also from the neighbouring tiles). The output of this algorithm will look like this:
The output raster contains holes where there were no points classified as ground. If needed for your use case, you can fill the holes using Fill nodata algorithm from GDAL in the Processing toolbox and create a smooth terrain model for your input Virtual Point Cloud layer:
Virtual point clouds can be used also for any other algorithms in the point cloud processing toolbox. For more information about the newly introduced algorithms, please see our previous blog post.
All of the point cloud algorithms also allow setting filtering extent, so even with a very large VPC, it is possible to run algorithms directly on a small region of interest without having to create temporary point cloud files. Our recommendation is to have input data ready in COPC format, as this format provides more efficient access to data when spatial filtering is used.
One of the very useful features of VPCs is that they work not only with local files, but they can also reference data hosted on remote HTTP servers. Paired with COPCs, point cloud data can be streamed to QGIS for viewing and/or processing - that means QGIS will only download small portions of data of a virtual point cloud, rather than having to download all data before they could be viewed or analysed.
Using IGN’s lidar data provided as COPC files, we have built a small virtual point cloud ign-chambery.vpc referencing 16 km2 of data (nearly 700 million points). This VPC file can be loaded in QGIS and used for 2D/3D visualisation, elevation profiles and processing, with QGIS handling data requests to the server as necessary. Processing algorithms only take a couple of seconds if the selected area of interest is small (make sure to set the “Cropping extent” parameter of algorithms).
All this greatly simplifies data access to point clouds:
Data producers can use very simple infrastructure - a server hosting static COPC files together with a single VPC file referencing those COPC files.
Users can use QGIS to view and process point cloud data as a single map layer, with no need to download large amounts of data, QGIS (and PDAL) taking care of streaming data as needed.
We are very excited about the opportunities that virtual point clouds are bringing to users, especially when combined with COPC format and access from remote servers!
Thanks again to all contributors to our crowdfunding campaign - without their generous support, this work would not have been possible.
Contact us if you would like to add more features in QGIS to handle, analyse or visualise lidar data.
]]>There are several 3rd party QGIS plugins (either proprietary or not fully open source) which allow users to interrogate and analyse lidar data. But with our latest work, we have introduced powerful point cloud algorithms to the QGIS Processing framework. All the algorithms are available out of the box in QGIS 3.32, with no need to install plugins.
In this blog post, we summarise the initial point cloud algorithms for QGIS Processing toolbox which will be available in QGIS 3.32 (to be released at the end of June 2023). This work was made possible by the generous donations to our crowdfunding.
First off a quick look at the new algorithms as shown in the Processing toolbox in three groups:
LAS 1.4
point format 6
count 56736130
scale 0.001 0.001 0.001
offset 431749.999 5440919.999 968.898
extent 431250 5440420 424.266
432249.999 5441419.999 1513.531
crs ETRS89 / UTM zone 34N (N-E) (EPSG:3046) (vertical CRS missing!)
units horizontal=metre vertical=unknown
Attributes:
- X floating 8
- Y floating 8
- Z floating 8
- Intensity unsigned 2
- ReturnNumber unsigned 1
- NumberOfReturns unsigned 1
- ScanDirectionFlag unsigned 1
- EdgeOfFlightLine unsigned 1
- Classification unsigned 1
- ScanAngleRank floating 4
- UserData unsigned 1
- PointSourceId unsigned 2
- GpsTime floating 8
- ScanChannel unsigned 1
- ClassFlags unsigned 1
Output from point cloud information algorithm
All the heavy lifting of the point cloud processing is done by PDAL - a state of the art open source library for processing point clouds. PDAL provides a wide range of “readers”, “filters” and “writers” to build complex pipelines to process point clouds.
We have built a new standalone command line tool pdal_wrench on top of PDAL. It addresses two major issues that non-expert users typically face when working with PDAL:
The command line tool provides a simple set of commands that take care of everything. For example, to export a raster layer with elevations (DEM) with 1 meter resolution:
pdal_wrench to_raster --output=raster.tif --resolution=1 --attribute=Z data.las
The pdal_wrench tool does not depend on QGIS, so it can be easily used separately.
The commands are designed to run in parallel when there are multiple input files or when the input file is in COPC format. Depending on the algorithm, the work gets split spatially into square tiles (1000x1000 map units by default) for parallel processing, or individual files are processed in parallel. With a single ordinary LAS/LAZ file on input, there is currently no parallelism going on.
For commands that are sensitive to edge artifacts (such as export to raster), we take care of processing extra points outside of the extent of each tile (referred to as collar or buffer) to make sure the results are correct as if no tiling would be happening (see Martin Isenburg’s article for more details: https://rapidlasso.com/2015/08/07/use-buffers-when-processing-lidar-in-tiles/).
The current list of point cloud algorithms already allows users to do plenty of work. But more could be added to the toolbox - algorithms that are already supported by PDAL, but not exposed in QGIS: classification, noise removal, surface reconstruction, clustering, height above ground, colorizing and many more. If you are interested in more point cloud processing algorithms in QGIS, please contact us and we will be happy to add them to future QGIS releases.
]]>First and foremost, thanks to the generous support from the community to fund our work. Here is the list of our contributors in no particular order:
IGN (INSTITUT NATIONAL DE L’INFORMATION GEOGRAPHIQUE ET FORESTIERE), National Land Survey of Finland, Danish Agency for Data Supply and Infrastructure, K2 Engineering GmbH, ProScape Consulting, Västra Götalandsregionen, Kristianstads kommun, IGN FI, L’Arrière Guichet, Septima, QWAST-GIS, ATAPEX s.r.o., REDcatch GmbH, F.A.R.M. Facilitazioni Agroecologiche Regionali Mobili, EPYMA TERRITORIO Y MEDIO AMBIENTE, SL, GEO EXPLORATION LTD, Bohannan Huston, Inc., Lidar Guys, Neuchâtel- Service de la géomatique, Wooding Geospatial Solutions, Ville de Vevey, QGIS User Group Switzerland, Ecophylla Consulting, Refactor, Locate Press, Alta ehf, Oester Messtechnik GmbH, RUDAZ+PARTNER AG, BayesMap Solutions LLC, GEOACE, Natalie Gyles, Andreas Neumann, Dougal Munro, Spatial Thoughts, Cicada Systems GIS Consulting, Cori Hermle, Powell Asset Mapping LLC, Darren Farmer, Greg Hall, Ecothought Pty Ltd, Gabriel Diosan, Bhutan QGIS Group, Ultimatum Finesse, Balanced Risk Strategies, Ltd, Concordia University, Burmis Studio Inc., Nicholas Hadaller, Angello Villatoro, Yoichi Kayama, Hennessy Amor Becerra Ayala, Flow Design Limited, BNHR.XYZ, Roberto Moyano, Benjamin Kuster, Goldspot, North River Geographic Systems, Inc, David W. Wormuth, Victor Graphics, Valley Spatial, Stephen Mather, SANTIAGO AURELIO MOTA, Kelly Crowell, Brian Duhan, Paddy Fisher, OSGEO:UK, Christian Gugl, GIP CRAIG - Centre Régional Auvergne-Rhône-Alpes de l’Infromation Géographqiue, Raphael Mabit, Tibor Lieskovský, Kerstin Fohlert, Zhan Li, Bernd Vogelgesang, Marlin Müller, Johannes Bonekamp, Stefan Giese, Fabian Faßnacht, QGIS Sweden user group, Falo, DAVID GARCIA HERNANDEZ, Lint Data and Geospatial, Cliff Sin Wai Lau, Grzegorz Sapijaszko, Łukasz Rapa, Alessandro Pintucci and Maarten Pronk.
Our gratitude also goes to those who want to remain anonymous.
To see this feature in action, you can open QGIS project properties and under Terrain, there should be an option for Global Map Shading. You will need to first add a raster as your DEM under the Terrain section.
Elevations of all these layers are combined and the considered elevation is chosen depending on one of two methods:
Depending on the context and the use of the map the user can choose the more appropriate method.
For now, the shading methods implemented are the Eye Dome Lighting and the hill-shade. More methods could be added in the future - such as ambient occlusion.
The user can choose the elevation shading settings in a specific UI widget that can be found in two places:
This work was carried out by our collaborator North Road. The profile tool can embed elevation profiles within print layouts. It is possible to add beautifully styled profiles in your print outputs.
To use elevation profiles in print layouts, simply click the Add Elevation Profile button in the toolbar, and then to initialize it, copy profile configuration from an existing elevation profile from QGIS main window using the “Copy From Profile” button.
With the new improvements, you can:
The terrain’s and point cloud layers’ elevation range are taken into account so that the camera is not positioned below the scene’s contents, which was the case when using the terrain’s vertical scale setting to exaggerate the elevation differences.
3D Views can now be limited to a specific 2D extent. The terrain is clipped and no 3D features beyond that extent are loaded, making it easy to render specific areas of big QGIS projects. The project’s 2D extent is used by default which can then be adjusted in each 3D view separately using the new General tab in 3D configuration.
There will be more features planned for QGIS 3.32. We are finalising the Processing framework for point cloud data and it should be available on QGIS master in coming weeks.
Please do not hesitate to contact us if you have any suggestions to improve QGIS 3D and point cloud support.
]]>Martin Dobias, our CTO and the lead developer of 3D and point clouds integration in QGIS sat down with Daniel O’Donohue from Mapscaping to talk about point clouds and QGIS.
Martin discusses his early involvment with QGIS back in 2005 and how he started his journey to become a QGIS developer.
]]>Have you ever been in the situation when, after making a lot of changes in your Mergin Maps project, you hesitate to press Sync button because you are not sure that all required changes are made or afraid that some unwanted edits were introduced? Or maybe you need to review the work done and see what actually have changed between two versions? If the answer to any of these questions is “yes” then you will like the changes visualisation functionality we introduced in the 2022.4 version of the Mergin Maps plugin for QGIS.
Changes visualisation functionality comes handy in two use-cases: revising local changes made in the Mergin Maps project before syncing them with the server and getting a list of changes between two versions of the project. Let’s take a closer look at this feature.
While working with Mergin Maps project, the user can at any time revise their current changes made locally. First, make sure that all your layer’s edits are saved (committed) as currently viewing of the unsaved changes is not supported. Then right-click on any vector layer and select “Show Local Changes” entry in the context menu.
Accessing local changes from context menu
This will open the Changes Viewer dialog. Each vector layer with local changes has its own tab in the Changes Viewer dialog, the name of the tab matches the layer name and also contains information about the number of changes in this specific layer. Local changes are shown on the map and in the tabular form, to distinguish different types of edits a following color codes are used: inserts (new features) are green, edits orange and deletions red. It is possible to enlarge or reduce the size of the map and table by dragging the splitter between them, splitter position is applied to all tabs and will be saved and reused on the further dialog calls.
Features added, deleted and modified in map and tabular views
Map canvas in the Changes Viewer dialog supports basic operations like panning as well as zooming in and out. By default, all project layers are shown on the map to provide better context, but it is possible to toggle their visibility by unchecking the “Toggle Project Layers” button in the toolbar above the map. When this button is unchecked, only changes from the current vector layer are shown.
If, after some panning/zooming, you need to return to the extent where all changes are visible — press “Zoom Full” button. Also, it is possible to select a specific feature(s) in the table below map and zoom to them by clicking the “Zoom To Selection” button. Finally, changes can be added as a new memory layer to the current project. To do so, click “Add to project” button and choose one of the options: add changes from the current layer or add all changes from all layers. For each changed layer, a new memory layer will be added to the current project. These changes layers will preserve the same color coding for features and attribute table as used in the Changes Viewer dialog. Please note, that these layers should be manually removed from the project before the sync, unless it is your intention to make them a part of your Mergin Maps project. Another way to revise local changes is to open Changes Viewer from the Project Status dialog by clicking “View Changes” button.
Sometimes one may want to export local changes as a vector layer and save that file for further usage. Of course, this can be done with the help of Changes Viewer dialog, but it is time-consuming, especially when the Mergin Maps project has many layers or if there is a need to check local changes in several projects. To cover this use-case, we also provide “Extract local changes” tool. This tool is a part of the Mergin Maps QGIS plugin and can be found under the “Mergin Maps” group in the Processing Toolbox.
Mergin Maps Processing tools to create changeset
In the tool dialog you need to specify a directory with your Mergin Maps project, select a layer of interest either choosing from available layer or selecting a GeoPackage file in the project directory and layer in this file.
Processing tool to extract local changes
An output layer containing local changes will be created as a temporary or regular layer and added to the current project. This layer will have the same styling (both for features and attribute table) as the layers produced by Changes Viewer dialog.
Result of the local change processing tool
The “Create diff” tool comes handy when you need to revise the changes between two versions of the layer in the Mergin Maps project. This tool is also a part of the Mergin Maps QGIS plugin, and it is implemented as a Processing algorithm. The “Create diff” tool can be found under the “Mergin Maps” group in the Processing Toolbox.
The tool dialog is quite similar to the “Extract local changes” tool dialog. Fill in input values: directory of your Mergin Maps project, layer of interest, start and end version numbers. Finally, specify location of the output vector layer or leave the field empty if you want it as a temporary layer in your current project. After clicking “Run” the tool will query the server for information and generate a vector layer containing all changes made between specified layer versions. For example, if some field value was changed in one version and then the same field was changed again in another version, then only the last change will be shown in the output changes file.
This feature is an another step in our ongoing efforts to create an easy-to-use tool for collaborative data collection and data management. If you need help or want to share your experience with Mergin Maps QGIS plugin, please join us in the community chatroom, and we will be happy to hear your thoughts.
]]>The dumping of municipal waste is a global threat to our environment and all life forms.
Currently, there is a distinct trend of less landfilling, as countries move steadily towards alternative ways of recycling and incineration, where material use is not possible.
Dr Martin Pavlas, Associate Professor at the Institute of Process Engineering in the Faculty of Mechanical Engineering at Brno University of Technology in the Czech Republic, is doing important research as part of an EU project regarding municipal waste sampling.
Bins to be sampled (Photo: M. Pavlas)
With the aid of Mergin Maps and QGIS, he is carrying out an extensive municipal waste composition survey in the Czech Republic. Together with Peter Petrik of Lutra Consulting, a unique GIS-based tool was developed for the waste sampling. This includes a prototype mobile application based on Mergin Maps for waste sampling in the field.
The project will introduce point cloud processing to QGIS and further enhance profile tool and 3D maps. The new processing tools will allow you to create terrain/contours from your point cloud data, handle and manage large datasets and several other processing algorithms. In addition, we intend to allow you to embed profiles in your print layouts, export to other formats (e.g. DXF, CSV) and more improvements to the elevation profile tool. For more details see the crowdfunding page.
The work will start soon in collaboration with the excellent teams from North Road and Hobu. To stay up to date with progress, you can follow this blog or monitor QGIS code repository. If you would like to test the new features and provide us with your feedback, you can install QGIS nightly/master.
]]>Point cloud data and raster shading in QGIS.
In the last couple of years, we added point cloud data to QGIS, developed profile tool, improved 3D map navigation and many more features related to 3D data.
The new processing tools will allow you to create terrain/contours from your point cloud data, handle and manage large datasets and several other processing algorithms. In addition, we intend to allow you to embed profiles in your print layouts, export to other formats (e.g. DXF, CSV) and more improvements to the elevation profile tool. For more details see the crowdfunding page.
To pledge to this crowdfunding campaign, simply fill in the form. We will start the work as soon as the target fund is raised.
]]>Peter Petrik was a guest in the episode of QGIS Offline And In The Field. He talked with Daniel O’Donohue about collection of spatial data in the field.
Mergin Maps is a field data collection app based on QGIS. It makes field work easy with its simple interface and cloud-based sync. Available on Android, iOS and Windows.
With the new integrated profile tool, you can generate cross sections of point clouds, raster, vector and mesh data. For more information on this tool, you can see the excellent video introduction by North Road who implemented this part of the project.
To be able to view profiles from different data types, there is now a dedicated Elevation settings under layer properties. Users can set the elevation source, style and some other configurations. You can then enable elevation profile widget window by going to the main menu in QGIS, View > Elevation Profile.
Cloud Optimized Point Cloud (COPC) is a new format for point cloud data and QGIS 3.26 comes with support for it (for both local files and data hosted on remote servers).
COPC is a very exciting addition to the ecosystem, because it is “just” a LAZ file (a format well established in the industry) that brings some interesting extra features. This means all software supporting LAZ file format will also be able to read COPC files without any extra development. If you are familiar with Cloud Optimized GeoTIFF (COG) for rasters, COPC is an extension of the same concept for point cloud data. Read more at https://copc.io/
Ordinary LAS/LAZ files have an issue that it is not possible to efficiently read a subset of data without reading the entire file. This is less of an issue when processing point cloud data, but much more important for point cloud viewers, which typically show only a small portion of the data (e.g. zoomed in to a particular object or zoomed out to show the entire dataset). For that reason, viewers need to index (pre-process) the data before being able to show it - QGIS also needs to do the indexing when a point cloud file is first loaded. The new feature that COPC brings is that data is re-organized in a way that reading just some parts of data is efficient and easy. Therefore when loading COPC files, QGIS can immediately show them without any indexing (that takes time and extra storage).
In addition to that, COPC files can be efficiently used also directly from remote servers - clients such as QGIS can only request small portions of data needed, without the need to download the entire file (that can have size of many gigabytes). This makes dissemination of point cloud data easier than before - just make COPC files available through a static server and clients are ready to stream the data.
A small note: until now, QGIS indexed point cloud files to EPT format upon first load. From QGIS 3.26 we have switched to indexing to COPC - it has the advantage of being just a single file rather than lots of small files in a directory. If you have point cloud data indexed in EPT format already, QGIS will keep using EPT index (rather than indexing also to COPC).
Classified renderer for point clouds has been improved to:
This improvement is not part of the crowdfunding campaign and was exclusively funded by the Swedish QGIS user group, but it is somehow relevant to the audience of this blog post!
With this feature, you can set polygon transparency in 3D scenes.
We are trying to improve QGIS to handle point clouds for visualisation and analysis. If you would like certain features to be added to QGIS, do not hesitate to contact us on info@lutraconsulting.co.uk with your idea(s).
]]>With this feature, you can filter the point cloud data based on the classes or any other attributes. This is very similar to filtering available for vector layers.
Filtering data allows you superimpose for example building on top of the raster representation of your point cloud data:
Examples of filtering you can use:
Classification = 2
- only show ground points
ReturnNumber = 1
- only show the first return or ReturnNumber = NumberOfReturns
for the last return
Z >= 10 AND Z <= 50
- only show a slice from the range of elevations
The filtering window also displays statistics of some of the parameters.
When you navigate in the 2D map, you often want to see the 3D map view also updated and vice versa. This feature also allows you to view the extent and camera angle of your 3D map view on 2D map.
There is a new 3D styling mode for point cloud which follows the 2D styling. This means you do not need to apply the same styling, e.g. Classification twice: once for the 2D map view and another for the 3D map view. Once you set the 3D style to follow 2D map, any changes in 2D map style will be automatically displayed in 3D map.
This feature was funded by QGIS.org to improve 3D map navigation. Users can now better move, rotate both the map and camera. The 3D map navigation is now more inline with other applications like Google Earth.
This work was made possible with generous donations and support by the individuals and organisations below (not in a particular order):
Stuart Smith, BayesMap Solutions, Tibor Lieskovský, Balanced Risk Strategies, Yoichi Kayama, Basel Land Registry and Surveying Office (GVA), Rudaz + Partner, Jakub Fuska, Richard Barnes, Spatial Thoughts, Hans van der Kwast, António Pestana, Richard Lorion, Eagle Resources, Suresh Muthukrishnan, 12P Consulting, Alta, JCIS Consultants, Brenna Hughes, Amt für Geoinformation Basel-Landschaft, Darren Farmer, F.A.R.M. Facilitazioni Agroecologiche Regionali Mobili, Ali Nayeri, Land Vorarlberg, Landesamt für Vermessung und Geoinformation, QGIS User Group Switzerland, Robert Thunen, Twomile Heavy Industries, Inc., Roberto Moyano, Jens Grehl, Pēteris Daknis, Rob Willson (Ecophylla Consulting), Daniel Löwenborg, Ville de Vevey, Alfredo Toledo (Suriyaco), QTIBIA Engineering, Ian Burrows (FAS), Pascal Obstetar, Lidar Guys, Mapping Automation, LLC, Featherstone Survey and Civil, Peter Schmitz, Fernando Michel Tuesta Chichipe, Hugo Sørensen, Bernie Connors, Watershed Research and Training Center, MBS Environmental, Andreas Neumann, Adrian Matter, Mapfly, Enso, João Gaspar, Eric van Dijk, City of Uster, Switzerland, QGIS Usergroup Denmark, STAEREA, Ostschweizerische Gesellschaft für Höhlenforschung, Department of Environment, Land, Water and Planning (Victoria), IGN FI, Travis Flohr, Amt für Wald beider Basel, Matthew Bodnar, Surface libre, OSGeo:UK, National Land Survey of Finland,Natural Resources Canada, Fonds Brukhalter, Arbeitsgemeinschaft Höllochforachung AGH, gis experts, BNHR, Rogue Geoscience Ltd., USACE CRREL and Ian Huitson.
In addition to the list above, we thank several anonymous donors who chose not to be listed.
If you have made a donation towards this work and your name or your organisation name does not appear here, please contact us (info@lutraconsulting.co.uk).
Previously, if you closed a project with a 3D map view, the 3D map view and all its settings were lost when you reopen that project. So in QGIS 3.24 we’ve added a “3D map view manager” that takes care of listing, removing, renaming and duplicating 3D map views in your projects! We’ve also added a new “3D Map Views” menu, which contains all your created 3D map views for easy access.
To summarise, these are the advantages of this new feature:
3D map canvas panel was difficult to move, resize and often resulting in unwanted docking. With QGIS 3.24 we added the ability to switch 3D maps from a dockable widget to a top-level window (and back to a dock widget), so that these map views can now be managed, resized and moved just like a standard application window. In addition, you can now use 3D map view in full screen mode.
We’ve added an option to render point clouds according to their Z-order in 2D map views. With the new bottom-to-top ordering option enabled, points with larger Z values will cover lower points – resulting in the appearance of a true orthographic photo. There’s also an option for reverse sorting (top-to-bottom), where the scene appears as if viewed from below. This feature is available in QGIS 3.24
The image below displays the default Z ordering of a LAS file when loaded in QGIS:
The same layer with the ordering of Z switched to bottom-to-top:
With this feature you can render point cloud layer in the 3D view as solid surfaces generated by triangulation. The triangulation is available for all the 3D point cloud renderers: unique color, ramp color, classification and RGB. This feature will be available in QGIS 3.26 and you can try it in the current QGIS nightly/master.
“Input can in theory handle everything you’d ever want for a mapping tool!”
Greg Parry is President of the Swan Bay Environment Association in the Borough of Queenscliff, Victoria, Australia. This semi-retired marine ecologist runs a one-man ecological consulting business, Marine Ecological Solutions.
Bare sand and seagrass meadows off the beach at Edward’s Point.
Swan Bay is a significant marine wetland with an area of 30 km2, situated near the entrance to Port Phillip Bay on the south-central coast of the Province of Victoria. This is an important bird habitat, supporting the life of about 200 different bird species. It has been recognised as an area of international importance under the Ramsar Convention. Every October, Swan Bay is visited by thousands of migratory shorebirds. More than 3 000 Black Swans can be seen in the bay in Summer and Autumn. The seagrass in Swan Bay is also an important habitat for a variety of fish and other marine life.
Drone image of Swan Bay, Australia
The natural flora and fauna in the area are under threat by environmental weeds. Among the 300 different plant species, about 50 % are alien. Invasive environmental weeds harm native plants and animals, the natural landscape and threaten the biodiversity of indigenous species. The three main invasive weeds in Swan Bay are Italian Buckthorn, African Boxthorn and Polygala myrtifolia (myrtle-leaf milkwort from South Africa).
African Boxthorn
A team of volunteers from among the 120 members of the Environment Association will assist with the removal of invasive weeds. Some of these are so large or spiny they need to be cleared by contractors. Others require spraying, while many will be cleared manually by a team of volunteers.
Pre- and post-weeding of the same area
This is where Dr Parry’s expertise comes in – together with Mergin and the Input App. He realises the urgent need for effective mapping and planning, to co-ordinate the work of volunteers. If this is not done, there is “inadequate follow-up, so it is a waste of their time and energy”. He emphasises, “We need better records of when places are weeded and how they are weeded.”
The role of accurate mapping is two-fold: firstly to create a historical record of past weeding and to assess weed density, and, secondly, to identify the areas for future weeding.
Dr Parry has classified the following categories for weeding as:
Classification of 4 different weeding categories
Dr Parry realises the need to store the information accurately (how much work has been done, the man-hours and number of people, etc.) and then to analyse it, so as to achieve a more effective job by volunteers in future.
Map showing all areas where weeds have been removed between 2015 and mid-2021. Underlying these ‘weed density’ polygons are polygons showing areas that were weeded in different years between 2015-2021.
Dr Parry finds that Input has all the features required for this project. It will be very useful for volunteers in the field, especially as it is usable by both iOS and Android. He thinks it is “remarkably cheap”, compared to the software he had previously been using.
He has some experience with GIS and, after watching a YouTube instructional video, has managed to incorporate QGIS and set it up without many problems. Once he has fine-tuned the set-up, he is sure that he will be able to enlist many more willing volunteers for this important undertaking.
Back in 2005, Dr Parry had used a Magellan Mobile Mapper hand-held device for mapping, which was bulky and cost about $7 500. “Now all of the capacity of that system is available on your phone!” He is motivated to incorporate the user-friendly features of the Input App in order to achieve his main long-term objective:
“This information, I think, will be very helpful in improving co-ordination of weeding efforts within the borough and ensuring that resources are used efficiently. Over a few years, we will get a much better concept of the resources required to do the job in total. I’d summarise it to say we should be a bit more strategic about it, so as to be more effective.”
Nick was searching for a way for his remote team to safely edit the GIS layers of their fibre network designs at the same time. Solutions he’d tried were either unsafe for concurrent editing, not feature-rich enough or had prohibitive licence costs. Nick now uses Mergin to collaborate with his colleagues around the world in near real time.
Nick Whittaker is Director of Highbeech Consultancy, a company providing specialised fibre optic network design services. Highbeach is based in England and operates in markets around the world.
Around 9 years ago, Highbeech started helping their clients develop city-wide fibre-to-the-home network design strategies. Nick’s team developed network simulation models which they used to understand the performance and likely construction costs of different designs fed by both public and municipality GIS data. The models proved a success as several clients commissioned construction of the resulting designs.
Nick’s team were now tasked with acting as technical liaison to the engineering companies carrying out the build in the US.
Nick Whittaker
Nick now needed a way for the different project partners to safely view and edit the design at the same time: “This was working sequentially, not collaboratively - we needed to be able to work in parallel!”
“One of the engineering companies we were advising had requirements to have staff in different parts of the US and at present, everything we were doing with the designs ran on-premises. People could use VPNs but working on designs was a nightmare” Nick explained.
Nick found working on designs via VPN was too slow to be workable and instead looked for a solution which would bring designs closer to the staff needing to work on them.
Cloud storage solutions such as OneDrive, Google Drive or Dropbox are unsuitable for collaboration on GIS data as they allow files that would normally be locked and accessed one at a time to be modified simultaneously by multiple users. This situation commonly results in data loss.
“We were having to limit ourselves to one person working on the design at a time and having to notify each other when it was safe for the next person to take over - file locking by email. This was working sequentially, not collaboratively - we needed to be able to work in parallel!” he added.
Nick searched for better solutions for allowing the various teams to work together quickly and safely at the same time.
“In the past we’d used ArcGIS Server which worked well as a collaboration platform when everyone was in the same office but wouldn’t be suitable in this case. We’d also tried ArcGIS Online but found it frustrating as it was too cut-down compared with ArcGIS Server.” Nick said.
Licencing was also an important factor for Nick: “Some solutions we looked at were plugins to other suites, for example, AutoCAD. If I tell my client that we’ll do a project in AutoCAD, they may tell me they can’t afford an AutoCAD licence.”
“While searching for collaborative GIS platforms I came across the Mergin plugin for QGIS.”
QGIS is open source GIS software with a large number of extensions called plugins.
“GIS is widely used in fibre network design and I used ArcGIS primarily. I first heard about QGIS when using FiberPlanIT as it’s implemented as a QGIS plugin. Within about 12 months I was doing the majority of my GIS tasks in QGIS and now consider myself a QGIS convert, purely because I see the power, potential and capability of its community.”
A fibre network design shown in QGIS
After taking time to evaluate Mergin within Highbeech, Nick proposed it as a collaboration platform to the US-based engineering company.
“Initially they were unsure how to set it up so I offered to do that and to administer it for them. Within 2-3 weeks they had 20 guys using it and within 3 months I’d migrated control of it over to them.” Nick said.
“The value is all about collaborating in near real time rather than the days it used to take to do things”
Nick’s team can now see the changes their US-based client makes to the design as they make them and jump in/out as required to perform validations and make corrections and changes as required.
“The time zone difference with the US now works to our advantage - we even tell our clients we can turn their 8 hour working day into a 16 hour working day.”
“Once our client’s fielding team has finished for the day, we work through their data, performing checks and validations and everything’s done by the time they’re back in the office the next day. This forms much of what Highbeech does on the project nowadays and this wouldn’t be possible without Mergin.”
When asked what worked particularly well about Mergin, Nick said: “the seamless way you can be working with someone on a call, make a change to a design and have it appear in front of them within seconds. This is incredible and something that happens on a regular basis.”
“The value is all about collaborating in near real time rather than the days it used to take to do things. That’s the greatest strength of both Mergin and Input.”
Input is a mobile app that allows GIS projects to be viewed and edited in the field.
When asked about how collaborative working might change in the future, Nick said: “We live in a world where an ever increasing number of people choose to work from home. For those working in fibre network design, that’s only possible with collaborative platforms such as Mergin.”
Nick has since introduced other clients in the US and UK to Mergin which supports their collaborative GIS efforts together.
We are very excited and looking forward to developing those features in the upcoming QGIS releases in collaboration with North Road and Hobu.
Thanks again to all those who have contributed to the campaign. Without your support, these major developments would have not been possible. We will publish a blog post with the list of contributors in due course.
To stay tuned with the latest development, you can visit QGIS code repository or visit our blog for news and updates.
“QGIS is my Number One tool for all my work. Through a podcast, I heard about Mergin and the Input App - I immediately installed it and have been playing around with it ever since. It has worked wonderfully and flawlessly. We haven’t had any problem with it so far. It was the perfect solution for this specific project!”
Konan Pruiksma, born in the Netherlands, is an archaeologist and GIS (Geographic Information System) Specialist experienced in Data Analysis, Relational Databases and Geostatistics, who is making a career in exploring our cultural heritage – in particular the vast wealth buried in the fields of the Netherlands. As an employee of Tijdlab, he was approached by Heritage Quest (Erfgoed Gezocht) and Leiden University/Erfgoed Gelderland to assist in their citizen science project, in which volunteers participate in archaeological research on the Veluwe and Utrechtse Heuvelrug.
Area of already well-known burial mounds created in QGIS
The Dutch landscape is dotted with ancient burial mounds, Celtic fields and cart tracks, some dating back to the 3rd and 2nd millennium BC. Bodies were first cremated and then the ashes were buried in these mounds. In the Middle Ages, mounds were also used for navigational purposes and even for gallows. The ashes of important people were probably honoured with a solitary mound, whereas other mounds contained the ashes of whole families or of many individuals.
Two burial mounds
The aims of the Heritage Quest project are two-fold, focusing on two views of the concept of citizen science:
The public viewpoint - to introduce as many citizens as possible to an awareness of the unique archaeological heritage in the Netherlands literally under their feet. This encourages better protection and conservation of this ancient and fragile heritage. Citizen science lessons are even planned for the classroom, to make children aware of their archaeological heritage and how to conduct scientific research.
The scientific viewpoint - to collect as much information as possible about the as yet unknown archaeological treasure trove. With the aid of LiDAR map data, huge areas become visible, which were previously hidden by vegetation and thus undiscovered. Later, with the participation of citizen volunteers, a large amount of field data can be collected, which archaeologists simply do not have the manpower to gather.
Volunteers using Input App in the field
LiDAR maps – high-resolution models of ground elevation created by a laser scanner, GPS and INS systems mounted on a small aircraft – are made accessible to the project by the Dutch government. Over the past 2 years, these LiDAR maps of areas of suspected burial mounds were analysed by over 6 500 volunteers, even by children, who searched over 600 000 maps and identified many thousands of possible new discoveries. In addition to burial mounds, Celtic fields (agricultural fields about 2 500 years old) and deep linear depressions left by the wheels of carts or wagons are clearly visible in the sandy soil of the Utrechtse Heuvelrug and Veluwe on LiDAR maps.
This manual work of many volunteers has another great benefit. The identified objects from LiDAR maps were used as a teaching dataset for a neural network that could potentially do a similar task automatically in the near future for different sites.
LiDAR image showing burial mounds and tracks
As every map was inspected by at least 15 different participants, the difference in the probability of potential barrows become clear immediately; some hills are identified by all participants, while others only by a few. It is probable that hills recognised by more people have a higher chance of being burial mounds and not natural hillocks. Following this reasoning, there are about 6 000 hills that have a high potential of being ancient burial mounds. However, this needs to be checked in the field.
In summer 2021, the field work began and this is where Mergin and Input App came into play.
“For this project, I immediately thought of Mergin and Input App and it worked from the get-go,”
says Konan Pruiksma regarding the ideal tools he chose for data collection in the field. Volunteers install the Input App to their phones and see the potential burial mounds or Celtic fields on the map. They navigate to a point of interest, digitise the point and fill in the required information, such as photos and notes, in a form. Such field work will continue to be carried out over the next year or more.
Burial mounds to be verified and form to fill in by volunteers
Once they have all the information about the location in Input App, they synchronise the data back to the Mergin Cloud. Konan, as a field manager, sees that data are synchronised on Mergin Cloud Dashboard, as well as who made the changes and when. If he needs to update anything, he can do it even from his office and let volunteers refresh the map. The collected data are stored in the PostGIS database via a docker container for further analysis. Konan uses QGIS Plugin for Mergin to download the field data gathered and to analyse them. When the point has been confirmed as a possible burial mound in the field a certain number of times, it is removed from the volunteers’ maps on Input App and reported to the professional team of archaeologists.
“Input App is user-friendly and can be used on any smartphone. This makes it possible to be used by volunteers with only minimal instruction. Which is great, because with over 6 000 locations to be inspected, we need all the help we can get!” explains Konan. “The App furthermore enables volunteers and archaeologists alike to locate the barrows which are usually hidden beneath vegetation. In the field, these low rises are often poorly visible, which is the reason they have not been identified until now. Before we used Input App, it often took us a very long time just to be able to locate the hill highlighted on the LiDAR map.”
Konan doing analysis of collected field data on his laptop
This team further investigates the mound by borehole surveying and the removal of soil samples. The mounds are generally not excavated, only if there is a danger of them being destroyed. Archaeologists prefer to keep these archaeological remains intact.
By coring with a 7 cm auger, a thin soil profile can be extracted from the barrow. In this way, archaeologists can get a small glimpse of the different layers that are present beneath the soil without destroying them by excavation. This provides invaluable information about our prehistoric ancestors. Radiocarbon (or Carbon-14) dating can provide accurate dating of the contents of these prehistoric mounds.
Konan says, “We previously did not know that there were so many burial mounds in the Netherlands!” He explains how information is obtained from the soil, without the need for excavation: “If we find charcoal, it is almost 100 % sure that it is a burial mound. We can learn what the burial rituals were of our ancestors, how they lived and what they ate.”
There are inestimable benefits of collecting and interpreting data accurately. The vast number of burial mounds in the Netherlands would not be able to be detected were it not for the assistance of volunteers, combined with LiDAR maps and tools such as QGIS or Input App. At present, there is a team of 20 volunteers at work on the Veluwe project, but this number should increase in the future, as more citizens become interested in doing citizen science in their environment. As Input App is user-friendly and very intuitive, a minimum amount of training is needed for volunteers, many of whom are students or senior citizens not au fait with digital technology.
Buildings extracted from lidar, overlaid on top of a cadastral map from 1890. Made by Tibor Lieskovsky, data UGKK SR.
Last year, we added point cloud data to QGIS, thanks to the great support from the community. Based on numerous feedback and suggestions we had over the past year, we have decided to run another campaign to introduce some new tools and enhance user experience when dealing with elevation and point cloud data. Below is a list of those features we intend to introduce. For more details see the crowdfunding page.
A tool for drawing profiles of elevations is essential when working with 3D data, yet it has long been missing from QGIS. While there are several plugins for QGIS that add this missing functionality, unfortunately they all have limitations when it comes to ease of use, performance, integration with the rest of the user interface and wealth of functionality. The goal is to add a profile tool that would cover a wide variety of use cases, and be available immediately out-of-the-box for all QGIS users.
An example of a cross-section: view from the top including the profile line with a buffer, and the profile with two different ways of styling (classification / elevation-based).
A technique that is extremely useful for visualization of point clouds in the 3D map view is the Eye Dome Lighting effect, which adds subtle shading and silhouettes (when there is a greater change of elevation) making it much easier to identify various features of interest that could be otherwise hard to see. To date this feature is only available in 3D QGIS map views, but we would like to introduce it to 2D map view as well! Many GIS workflows and products are still entirely based around 2D views of data (such as digitizing of vector data), so having the extra visual clues which Eye Dome adds is also highly desirable in 2D views.
Top-down 2D view of a classified point cloud. Left: rendering without Eye Dome Lighting (EDL) effect. Right: rendering with EDL.
A common requirement of analysts when working with point clouds to work with just a subset of the dataset. This could be as simple as showing only one class of a classified dataset, using only a limited range of elevations, viewing just a particular flight line, or looking at just the first/last returns. Currently, QGIS only provides basic class-based filtering when using rendering based on classification.
One of the limitations is the 3D map view’s window: when it gets closed the configuration of the 3D map view is discarded and any newly opened 3D map view needs to be re-configured again. This is frustrating for users and results in a lot of wasted time! We would like to fix this by keeping 3D views stored in QGIS projects, even when they are not currently being shown. Another issue related to 3D map views is the fact that they are currently only “dock widgets” that can be embedded within the main window. However, often it is preferable to be able to see the 3D view content in a big window, which is quite tricky to do in current QGIS versions (especially when the user does not have multiple displays).
The new specification of cloud-optimized point clouds (COPC) is current being finalized (October 2021) and it is a very exciting innovation! Inspired by the hugely successful cloud-optimized GeoTIFF (COG) format, the idea is to reuse the existing LAZ format (widely used by the lidar community) and add the improvements which make the format more useful.
While LAS and LAZ files (currently supported by QGIS) are the most common formats for point clouds nowadays, they are not the only ones. In order to support more formats in QGIS we do not need to worry about technical details of each format thanks to the PDAL library. It comes with a multitude of drivers for various formats and services - see https://pdal.io/stages/readers.html . In QGIS we need to provide a user interface to allow people to get other additional formats to work. Some drivers are always available in PDAL, others are optional and may need extra third-party libraries in order to work. We welcome feedback from the community regarding drivers they consider important.
Please visit the crowdfunding page for more details about the campaign!
In summary, apps need to use a specific part of folders on Android devices to store app related data.
Currently, Input stores your QGIS project and some other settings (e.g. grid shift projection) on /Internal storage/INPUT
. With the new Android requirements, the app related data should be stored on /Internal storage/Android/data/uk.co.lutraconsulting
.
For the 1.1 release of Input on Android, there will be an extra process. This process will be a on-off action. When you launch the app after the upgrade, it will copy the data from /Internal storage/INPUT
to /Internal storage/Android/data/uk.co.lutraconsulting
. Depending on the size of your projects, this can take a couple of minutes. During the process you will see a screen similar to this one:
The upgrade and copy process should work smoothly. But we suggest to take the following actions to ensure you will not lose any data during the process:
Sync all you changes: before upgrading the app, open Input and sync all your local changes to Mergin.
Storage space: if you work with several projects and large volume of data, make sure you have enough storage. The process will make a copy of your existing /Internal storage/INPUT
without deleting it. So, you need at least the size of /Internal storage/INPUT
storage available.
In case you have encountered any issues, you can take the following steps to fix the problem manually:
/Internal storage/INPUT
on your Android phone/tablet/Internal storage/Android/data/uk.co.lutraconsulting
Missing data: the migration process does not delete /Internal storage/INPUT
folder. It will rename it /Internal storage/INPUT_migrated
. Similar to step above, you can copy the data to the PC and move them to /Internal storage/Android/data/uk.co.lutraconsulting
. Alternatively, you can use a file browser app on your device to copy files around.
/Internal storage/INPUT
and marks the folder as /Internal storage/INPUT_migrated
. Therefore, when the next user starts up the app, no data will be present. To fix the issue, you need to manually move the data from /Internal storage/INPUT_migrated
to /Internal storage/Android/data/uk.co.lutraconsulting
(as described above) for the other users on the device.If you need further help, please join us on the community chatroom and we will be able to help you with the upgrade issue (or other Input/Mergin related problems.)
]]>Making a sustainable profit from farming can be something like tossing a coin. However, today the advancements in digital technology provide the incredible benefit of reducing the unpredictability of farming in general. In pecan farming in particular, the proper know-how of a combination of multiple factors such as soil composition, environmental impact, climate, irrigation and sunlight management are vital for success. Hard data is essential for innovating key decisions. Riaan Burger, a South African farmer, successfully uses open source GIS software to tackle the issues of collecting and utilising such vital data.
Riaan Burger on his pecan farm.
Riaan Burger’s journey started back in 2017 when, after 20 years’ working as an electrical engineer, he decided to embark on his own business in pecan nut farming. He is a self-taught farmer and co-owner of one of the biggest pecan farms in the district of Weenen, South Africa. At first, Riaan had only a rough idea of how to improve his newly bought 24 hectares of planted orchard.
After reading books on soil structure, type, depth and chemical analyses, he had soil samples taken to create a soil sample map. In his orchard, the trees were planted in 10x10 metres geometric layouts. After 20 years, the trees were already creating shade. Riaan knew that sunlight penetration is important and that pecan trees are susceptible to fungus, so he decided to take a closer look.
“The previous owner of the farm had already started chopping out every second tree. I was in the position where I could compare the yield from areas where trees had been chopped down, to areas where they had not been felled,” explains Riaan.
A photo of the farm taken by drone and initial stage of farming - soil maps.
Riaan realised that understanding the ambient conditions and mapping them to yield results was the right way to go. This was the beginning of his exciting journey in precision farming.
It was no easy task for Riaan to get a complete picture. For two years, he only recorded the yield per day, and the relevant orchard block. Later, he used the services of a local consulting company which provided him with surveying applications to help him to set up a basic workflow.
“I record the weight of pecan nuts from each individual tree and I plot it on the map,” Riaan says. “And then afterwards I can see the different cultivars of trees, the weight of nuts they produced and the area in which they were produced. I previously had soil samples taken, so I’ve got soil sample maps. Now I can overlay a yield map over the soil sample map and correlate them”
Despite working well with the surveying app, Riaan was still in need of assistance to prepare data tables and then to export them to Excel sheets and to request map creation. After the consulting company moved from a third-party app to the in-house build solution, things got worse for Riaan: “It was a generic tool and a difficult app to work with. Displaying data was cumbersome. I wanted something that was quick and easy again. Ideally, I would like to capture data on the iPad and have it directly linked to my laptop – that is how I think it should work.”
So Riaan discovered the Input and Mergin suite, a surveying app based on QGIS and a cloud service for data synchronisation, respectively. Despite only a basic knowledge of QGIS and the necessity to learn and experiment, the effort immediately paid off!
“I’ve got Input, I’ve got a Mergin account and I’ve got QGIS. I now have full control of the process. I don’t need to fund a consultant and find that by 8 o’clock at night he has not yet responded to my query, and so I have to back it up with a WhatsApp call,” Riaan explains. “This is something I created entirely myself. I don’t need to ask someone to change this map, or add that icon, modify the legend or change the styling, I am in control of all of it! And I think there is a sense of satisfaction when one gets things going for oneself.”
A farming project in QGIS desktop and Input app UI.
With the Input app, Riaan has gained control of the workflow – something not before experienced. He has gained the freedom to set the project according to his own individual needs. Without much effort, not only the data but the styling, an important factor for him, are synchronised.
“In QGIS I created a style where each bubble represents the kilograms of the yield and the colour represents the variety,” says Riaan. “Then I saw in Input that there is suddenly an extra page on my iPad. So now I have the bubbles and bubble colours on the iPad instantly, which I did not have in previous apps. Previously, I had to create it the long way around, but now it is suddenly on the iPad automatically. I enjoy having that available! In the past, I only had that information by the end of the harvest.”
The energy invested in creating workflow, capturing and analysing data paid off handsomely for Riaan. All his initial doubts were suddenly eliminated. He has all the information essential for making any important decision right at his fingertips.
“The first time you fell trees, you have nightmares! How on earth you can be cutting your income and destroying 20 years of growth? Now, with the necessary proof at hand, it is easier in your mind to motivate that tree felling is actually your source of income, that it is the right thing to do,” says Riaan. “You can see that the yield from the remaining trees is more than double than before. The trees are healthier, because of allowing sunlight in. Moreover, you also get less pressure from fungus, because pecan trees are easily susceptible to fungus. Now that you have good ventilation, because you have a draught through your orchard, you can see on the map what your exposure to fungus is.”
Orchard with higher density of trees (top) and lower density (bottom).
Riaan continues: “I am not a sole owner, I have a partner. Now I have the data available, so that when we have a meeting I can explain to him this is what I did, this is what happened and this is the result. If you go to a bank manager for a loan, I mean, he really frowns at you when you tell him you are cutting down your trees which produce your income. Then you can explain to him why it is for the better.”
Increase in yield per tree after reduction of tree density.
The benefits of collecting and interpreting data carefully are limitless. Cutting down trees is just one example of the many advantages of careful data gathering and interpretation.
“Now I see that certain cultivars perform better than others this year,” adds Riaan. “So I can see if they consistently perform better or worse. I can decide, based on these results, what to do: if I need to add more nitrogen or improve soil pH, or whatever the case may be.”
User-friendly tools tailor-made to specific needs, such as the Input and Mergin suite, provided Riaan a way of collecting and analysing his specific type of farming data. His insights for necessary improvements also developed over the past few years.
“At first I did not have the distinction between individual trees and I even said to my partner” continues Riaan, “if only I’d started doing this in the very first season, I would have had 5 years of data to compare – but I didn’t. I will definitely continue to use this suite. It is valuable and I have been using it only for harvest. However, I see now that I can start recording where I have cut down a tree, so I have a record of when and where the tree was cut. I can also record my irrigation data.”
In spite of initial teething problems and the difficult decisions he had to take, Riaan’s success is proof that he has found the right way to go for his own specific pecan farming needs. He has even become an evangelist within the local farming community.
“I am not the only pecan farmer in our district. I am the biggest pecan farmer, but not the only one. I am trying to spread the word around. For example, my neighbour still does not want to cut down his trees. But I am getting there, showing him photos, proving that you have to be aggressive to open up the orchard to get sunlight in – and here is the proof that it has been working for me!”
When you work in a team with several collaborators adding data and photos to the project, the size of the project can get really big. Every time you try to sync your changes, photos from all users will be transferred to your phone. This can take a long while when there are several hundreds of photos collected by other collaborators.
Image below illustrates the difference between having selective sync or the default behaviour (selective sync is disabled):
The configuration file for enabling the selective sync is stored in mergin-config.json
which should be placed in the root of your folder. By opening the file in a text editor and adding the following option, the sync will be enabled for photos within your project root folder:
{
"input-selective-sync": true
}
Currently, the editing of the file is manual and through the text editor, but we plan to incorporate it within the Mergin plugin for QGIS.
To learn more about how to set up selective sync for your project, you can see the example project. Read more on our help pages for extra configuration options.
It is often the case that you have a set of spatial features and you want to record some parameters every now and then. For example, there is a GIS layer representing the manholes and the surveyors carry out regular inspections of the manholes. Instead of duplicating the manhole layer and recording each inspection, you can create a non-spatial table and store each inspection as a new line.
Another use-case for such a feature is that you’d like to attach multiple photos to a single feature.
To learn more about how to configure these types of projects in QGIS you can see the example projects (manhole example and multiple photos example). The documentation pages describes the logic and process in QGIS in more details.
In addition to the display of the accuracy bubble in the app, we have recently added a whole set of new variables to capture the GPS accuracy, e.g. horizontal and vertical accuracy, ground speed and many more. See the help pages to find out how you can set up those variable within your form. Alternatively, you can clone the example project on Mergin website.
In addition to capturing GPS metadata, this feature can be used for geo-fencing: for example, you can only allow users to edit/capture data when they are physically (i.e. their GPS location) within a certain area:
An exciting new feature in the upcoming release of https://t.co/KXCwKEzzE3: capturing GPS accuracy, speed, position, etc. With this new feature, combined with #QGIS, you can set geofencing rules. pic.twitter.com/zoKsrI29bc
— Lutra Consulting (@lutraconsulting) April 30, 2021
If you have any questions, would like to interact with the rest of community or want to give us your feedback, you can join the Slack community channel.
If you would like to add a new feature or have suggestions to improve the app, do not hesitate to contact us on info@lutraconsulting.co.uk
]]>In this year’s Google Summer of Code (GSoC), I decided to work on the native QGIS raster calculator. Martin Dobias and Peter Petrik volunteered to mentor my work. I’ve been studying Civil Engineering and GeoInformatics at the University of Padua; here I had the opportunity to work both with a lot of GIS software including QGIS. I enjoyed working with QGIS almost immediately because of the possibility to perform complex analysis with a few clicks or with few python commands. Being passionate about programming and enthusiastic about Open Source, I realized that having the possibility to work together with some experienced developers and with an active community was really a great and unique opportunity, so I decided to apply to the GSoC.
The existing raster calculator is a powerful tool to perform map algebra that outputs a raster layer, before this work it was possible to take advantage of this tool only by saving the output of this operation as a file. The aim of this year GSoC was to allow users to perform their analysis without creating a new derived raster and taking up disk space and therefore have the result as an on-the-fly computational layer.
Let’s jump to an example and let’s say I want to compute the Chanopy Height Model (CHM), subtracting the Digital Terrain Model (DTM) from the Digital Surface Model (DSM).
I also want to perform some other analysis on the DTM since I want to compute the ideal elevation value for a particular tree planting (disclaimer: the elevation value used is example purposes only, moreover when planting trees you should take into account a lot of factors like slope, aspect, latitude. QGIS, by the way, can really be helpful in this kind of analysis). To do so I will start from the same data and I will create different on-the-fly layers for each calculation, in order to avoid the creation of different files I can take advantage of the new checkbox added to the raster calculator dialog. The computation of CHM is performed in the next screencast and the output layer name is, of course, CHM.
I’ll end up with a new raster layer (CHM) that can be styled as a normal raster and that is not written as an output file to the disk.
For some further analysis, from the DTM, I want to obtain the portion of the area with an elevation between 150 and 350 metres above the datum. By applying the following expression to DTM I’ll end up with a raster that has a value of 1 where conditions specified by the expression is TRUE
and it will have value of 0 otherwise.
("dtm@1" > 150) AND ("dtm@1" < 350)
I did not select the output layer name intentionally. The resulting layer will be named after the expression used to generate the layer.
I also had the opportunity to improve the raster calculator capabilities by adding the possibility to write expressions that involve conditional statements. Taking the already used example, let’s imagine I want to compute the CHM only for the areas of the DTM that are between 150 and 350 metres above the datum. It’s now possible to write an expression as the following one:
if ( ("dtm@1" > 150) AND ("dtm@1" < 350), CHM, -10)
This expression will output a raster with values of the CHM where the conditions are met and value of -10 if the conditions are not met. Since this is a final result of our analysis I’ll store this output as a file to the disk in the form of a GeoTIFF. I’d like to outline that the CHM used in the expression above and in the next screencast is an onn-the-fly computed raster, so it is possible to:
I had fun and I struggled working with QGIS, but I learned a lot of new and interesting things. My pull requests were met with several constructive comments, suggestions and feedback. Some suggestions can be a starting point for future improvements.
This case study was originally written in Czech. The Czech version can be found here.
Every year, teams of volunteers walk door-to-door through the Czech town of Litomyšl collecting charitable donations. Event organisers define routes for the various volunteer teams by marking-up paper maps with pens. The process has a number of issues both in the making and usage of the maps which organisers worked to overcome by making the maps digital using open source GIS software.
Maps were developed using QGIS and made available on volunteers’ phones using the Input app. Volunteers are now able to easily orientate themselves on maps which clearly show their routes. Organisers have reduced the time it takes to update routes and distribute these to volunteers.
Veronika Peterková works for the Litomyšl Parish Charity, a non-profit organisation providing health and social services to people in need since 1993.
Veronika describes the charity’s activities: “We provide home medical services and nursing care to the residents of Litomyšl and its surrounding villages. This includes helping families where the healthy development of a child is at risk and providing respite stays for clients who are otherwise cared for by their families at home. We provide care for about 1000 clients a year.”
She added: “We also coordinate the activities of volunteers who visit the elderly, help with tutoring children and with various leisure and cultural activities.”
One of the parish charity’s biggest fundraising events is the “Tříkrálová sbírka” (Three Kings Collection), a door-to-door carol-singing collection taking part around the 6th of January each year.
Volunteers participating in the Three Kings Collection.
“The Three Kings Collection is the largest national volunteer event in the Czech Republic. In the Litomyšl region alone, nearly 300 volunteers are involved each year with the carol-singers collecting over 500,000 Czech crowns (~20,000 EUR) in sealed boxes. The proceeds are intended to help the sick, the disabled, the elderly, mothers with children in need and other in-need groups in the local area.” Veronika explains.
The Three Kings Collection is organised by Caritas Czech Republic and at least 10% of its proceeds are allocated for humanitarian aid abroad.
Veronika is responsible for planning routes for the carol-singers so they efficiently visit households in the Litomyšl area. Singers are split into groups and paper maps are provided which show groups which households to visit.
An example of previous paper maps, image courtesy of Farní charita Litomyšl.
The above maps were produced by printing screenshots from a national web mapping provider and marking-up printouts for each of the 50 teams using marker pens.
This method proved to have a number of issues as Veronika describes: “On maps of larger areas, house numbers were not always visible due to the scale. This made it even harder for coordinators not familiar with the area to orient themselves, leading to confusion. Coordinators also found it hard to keep the maps dry and undamaged during unfavourable weather. If new groups signed-up afterwards or others opted-out, we’d have to redo/redivide the areas which would be very time-consuming as the maps would need to be marked-up manually once again.”
Veronika wanted to try a new solution for organising the 2021 Three Kings Collection with the goal of making volunteer tasks clearer and less reliant on paper maps. She wanted the new solution to allow her to:
In addition, Veronika wanted the solution to be affordable and work offline without volunteers needing internet connectivity in the field.
Peter Petrík, a regular participant of the Litomyšl Three Kings Collection suggested Veronika try using the Input app for coordinating the collection in 2021. Peter works for Lutra Consulting, the company behind Input and Mergin.
He showed Veronika how to create the maps in QGIS, a free and open source mapping software. Using map data from OpenStreetMap, they created a project showing the buildings to be visited, coloured by their associated volunteer group number.
Houses grouped by team in QGIS, image courtesy of Farní charita Litomyšl.
The styled map was uploaded to Mergin, a collaborative mapping platform, making it readily available for viewing interactively on volunteer’s phones using the Input mobile app. Both QGIS and Input integrate closely with Mergin which meant that maps could be adjusted in QGIS with the resulting changes being visible to volunteers shortly thereafter.
Veronika reflects on the solution: “The solution met all our requirements and the maps we’ve prepared can easily be reused in upcoming events, saving us time. The fact that the new maps were made publicly accessible means volunteers can just download them using Input which makes distributing and updating them very easy.”
Volunteer routes and position information shown in Input, screenshot courtesy of Farní charita Litomyšl.
She adds: “All the districts we wanted to visit were distinguished from each other by colour and we were also pleased to be able to clearly mark the areas not to be visited like industrial areas by colouring them in grey.”
Unfortunately COVID meant that Veronika’s plans changed as she explains: “Using these new methods we were able to prepare for the 2021 Three Kings Collection in a short time. Unfortunately however, the COVID situation meant we could not go out on the streets to use the new maps as intended. We hope that in 2022 we’ll be able to more closely evaluate the positives and negatives of the field aspect of the project.”
She adds: “We already see it’s now much easier to allocate areas of the town to our volunteers in a clear and fair manner using QGIS. Producing printed maps for those who prefer them is also now easy and the maps look much more professional. Those who only wanted to use the Input app could see the same information as on the paper maps, but had the advantage of being able to pinpoint their exact location and clearly see the house numbers of each building.”
Example printed map created for volunteers wanting also paper maps, image courtesy of Farní charita Litomyšl.
She concludes: “Overall we found the solution user-friendly, and appreciated being able to discuss the process with Lutra Consulting who helped us solve issues as required. About a third of our volunteers are interested in using Input, which I consider positive.”
The Litomyšl Parish Charity are on Facebook and Instagram.
TUFLOW is a suite of advanced 1D/2D/3D computer simulation software for flooding, urban drainage, coastal hydraulics, sediment transport, particle tracking and water quality. With over 30 years of continuous development, TUFLOW is internationally recognised as one of the industry leaders for hydraulic modelling accuracy, speed and workflow efficiency.
Lutra Consulting Ltd is a leader in software development for pre- and post-processing of hydraulic and meteorological results in open-source QGIS. We also work on mobile data collection Input App and GIS data synchronization service Mergin
In 2019 the TUFLOW team commissioned us to develop post-processing support for their TUFLOW Flexible Mesh format for QGIS 3.12. The format is 3D stacked mesh, which consists of multiple stacked 2D unstructured meshes each extruded in the vertical direction (levels) by means of a vertical coordinate.
At that time QGIS only supported 2D meshes that defined results on vertices and faces. We had been keen to extend the capabilities of the software stack to support 3D mesh data for a long time so this was an exciting opportunity. Part of the task was also to include rendering support for TUFLOW model results on the QGIS 3D view. The delivery of the project was within one QGIS release cycle (less than 6 months time for users to use it on their projects!)
Contact us at info@lutraconsulting.co.uk if you’d like to discuss the benefits of integrating your flood modelling software more tightly with QGIS or you have some custom QGIS development in mind.
When doing a substantial change in the QGIS codebase, the developer needs to write a technical specification of the QGIS changes for community discussion. QGIS Core Developers (which Lutra is a part of) can give valuable feedback to the overall technical approach and the wider community can raise some usability issues or enhancement proposals. Most importantly, each part of the QGIS code has its lead maintainers, for example Martin Dobias, our CTO, is the maintainer of QGIS 3D code and Peter Petrik is the maintainer of the Mesh layer code. It is a good practice to address the maintainers’, users’ and other developers’ concerns and feedback to ensure the feature can be implemented in QGIS.
So after a thorough discussion about the requirements with the TUFLOW team and analysis of the existing tools for post-processing and display of the TUFLOW FV format we came up with the QGIS Enhancement: Support of 3D layered meshes
The community reaction was very positive and supportive. Time to start coding!
Mesh Data Abstraction Library MDAL is a C++ library for handling unstructured mesh data. It provides a single data model for multiple supported data formats. MDAL is used by QGIS for data access for mesh layers. If you want QGIS to support your data format, you need to have a driver in MDAL that implements it.
We added support for 3D stacked meshes and the TUFLOW FV format in MDAL. When we develop features in MDAL, we focus on quality code, so
To implement the TUFLOW FV driver for 3D stacked meshes, we added a new API/interface in MDAL, so we needed to follow up with the QGIS changes in QgsMeshLayer
and MDAL data-provider.
The implementation of large feature changes is best to split into smaller but self-consistent parts. For example the first pull request added the basic support for the new 3D stacked meshes. Each pull request we do has a screenshot or gif/video with the new functionality preview, follows QGIS Coding Standards, has unit tests where necessary and includes the documentation for the functions/classes added in the public interface. Once the request is merged, the features are next day available in nightly builds on all platforms for testing!
When all the features were in QGIS master, the TUFLOW team used windows nightly builds to test the new features and provide feedback. After a small number of iterations, all issues were resolved and implementation signed.
Shortly the new official QGIS release was published and we started promotion of the new features on our social media channels. Also, the features developed under this contract were promoted in the visual QGIS changelog.
Key benefits made available to TUFLOW users include:
Do you have any questions or would like to see a demo of the QGIS Mesh Layer? Contact us at info@lutraconsulting.co.uk or schedule a demo call calendly.com/saber-razmjooei/15min
QGIS, migration, optimised, speed up, fast, hydraulic modelling, water, 2D, 3D, open-source, cost reduction, software development, TUFLOW, TUFLOW FV
We would like to share the excellent news that the OSGeo incubation committee kindly accepted the MDAL project into its Community Program. MDAL is an MIT Licensed C++ Library which integrates into your QGIS installation and is backed by the Lutra Consulting team and the wider community.
MDAL is a low-level library which allows reading unstructured mesh data from various formats. If you’ve never seen a QGIS Mesh Layer, then check out my presentation at FOSS4G 2019.
We kicked-off the MDAL project at the time we were migrating the Crayfish plugin to QGIS version 3. The Crayfish project started in late 2012, has more than 120,000 downloads and opened the way engineers and data scientists to visualize their hydro and meteo-data in QGIS.
QGIS uses MDAL to visualize, analyze and even modify data on unstructured meshes which are used by many hydraulic / numercial modelling solvers. In addition to providing abstraction for complex mesh data structures, MDAL also allows vectoral (e.g. wind speed) and temporal data to be visualised effectively and easily in QGIS.
The MDAL project (and therefore the wider community) has benefitted from generous sponsors such as TUFLOW, 3Di, Hydrotech, DHI, FLO-2D, Artelia, Federal Ministry of Austria for Agriculture, Regions and Tourism and many others. If you’d like to see your formats or workflows supported by QGIS then please consider becoming a sponsor or contact Peter at peter.petrik@lutraconsulting.co.uk
Right now we’re working on some new mesh frame editing tools for QGIS 3.22 (due for release in autumn 2021) and of course support for additional result file formats.
In this post we’ll talk both about Mergin the online managed service (Mergin Cloud) and also about the software stack that powers it (the Mergin Software Stack).
The Mergin Software Stack has been developed and maintained at Lutra Consulting over the past 3 years to power our Mergin Cloud service and has been maturing nicely in production.
We believe a sync service for supporting field-based GIS activities has been a missing piece of the open geospatial puzzle, and as strong advocates of open source software, we’re now sharing the Mergin Software Stack with the community. Therefore, on the 14th of June we released Mergin CE on GitHub. The CE stands for Community Edition.
The release means a fully open solution for field data collection and synchronisation is now possible using QGIS, Input app and Mergin CE.
Mergin CE is released under the AGPL licence and we are open to contributions from others. Looking forward to seeing what the open source community has to offer!
Mergin CE gives you the freedom to deploy, host and manage your own Mergin server on your own infrastructure, giving you complete control over your data. Mergin CE comes without commercial support.
Our main efforts are still very much focussed on the continuous improvement of Mergin Cloud, making it an awesome fully managed service for our customers.
We also now provide Mergin EE (Enterprise Edition) for those who want an on-premises deployment but with extra features like Active Directory integration, commercial support, and/or prefer a licence other than AGPL.
Releasing Mergin CE got us taking another look at Mergin Cloud’s Community (free of charge) tier. That’s why from today we’re updating Mergin Cloud’s Terms of Service so its free tier can no longer be used to store projects for commercial use. Mergin accounts storing projects for commercial use should now purchase a paid subscription after their initial 14 day evaluation period.
A common surveying setup is a single paid account (providing extra storage for projects) and a handful of free tier accounts used by surveyors for collecting field data. This is still possible because the new Terms only require that the Mergin account hosting/storing/owning the commercial project has a paid subscription.
For example, consider two users: Fred (who uses a free account) and Penny (who uses a paid account). Penny is permitted to use the project Penny/Survey for commercial purposes as it resides on her paid account. Fred (who uses a free account) is permitted to collaborate on the commercial project Penny/Survey as it resides on Penny’s paid account. However, Fred may not use the project Fred/Survey2 for commercial use as it resides on his free account.
In light of the release of Mergin CE, we feel it is fair to encourage those using Mergin Cloud’s free tier for commercial gain to support us with one of our affordable subscriptions. The link also has a number of frequently asked questions relating to this change.
At the core, geodiff library provides functionality to:
Thanks to the above low-level operations, any changes to data stored in spatial/non-spatial tables in GeoPackages can be easily transferred to others and applied. And thanks to the “rebase” functionality - inspired by source code management systems like git - we can automatically merge changes from multiple users capturing data offline in Input/Mergin (see our recent blog post that covers rebasing for more).
The library is written in C++, providing stable C API and offering Python bindings as well (look for pygeodiff
package in pip
). It also comes with a command line interface tool geodiff
covering all major features. The whole package has a very permissive MIT license.
Initially, geodiff library only worked with SQLite / GeoPackage files. This has changed with the version 1.0 - geodiff supports drivers, allowing use of different database backends to compare and apply diffs. In the 1.0 release we have added PostGIS driver in addition to SQLite/GeoPackage driver.
This means that users can compare tables or apply diffs in PostGIS databases using the same APIs as with GeoPackages. And not only that - diff files are compatible across different drivers. That means it is possible to take a diff file from a GeoPackage and apply it to PostGIS database!
Using the PostGIS driver we were able to create mergin-db-sync tool as a companion to Mergin platform. With DB sync, one can keep a local PostGIS database always in sync with a project in Mergin, supporting automatic transfer of changes from Mergin to PostGIS and the other way round as well - from PostGIS to back Mergin.
The library is hosted on GitHub in lutraconsulting/geodiff repository. We would love to hear your feedback!
As announced earlier, next week we will be open sourcing Mergin, our platform for easy sharing of spatial data in teams (whether they are in office or in the field). If you have not heard about Mergin platform yet, please have a look at the Mergin website, try Mergin plugin for QGIS and Input app, a mobile app based on QGIS for iPhone/iPad and Android devices. Since the initial release in early 2019, Mergin and Input have been used by thousands of users around the world.
At Lutra Consulting, we are dedicated to improving free and open source software for geospatial. We will be releasing Mergin as open source to solve another missing piece in the puzzle, providing open source end-to-end solution for mobile data capture for QGIS users. Watch our blog and Twitter for further updates!
]]>They happily find out that the answer is yes - it is one of the core features that greatly simplifies life of our users. In this post we will shed some light on how things work behind the scenes.
Let’s think of a simple project directory that needs to be synchronised between multiple desktop or mobile users, containing just two files:
my-project.qgz
that sets up map layers, styling, …my-data.gpkg
containing all GIS dataOur sample GIS data will contain a tree survey table, containing location, species and age of various trees:
When users edit data in my-data.gpkg
, the traditional cloud storage solutions (such as Dropbox, Google Drive, Microsoft OneDrive and others) simply copy the modified files there. They do not understand the file content though - so if two people modify the same file, they have no way of knowing how to merge changes together. In the worse case, when two versions of the same file are uploaded, they keep just the version which was synchronised last. Or slightly better, they resort to creation of conflicting copies which need to be manually merged later. As one can imagine, merging and consolidating modifications from multiple GeoPackages back to one copy is a slow, tedious and error-prone job.
the Mergin service has been designed to understand spatial data, especially GeoPackages that are becoming the most popular format to store vector & attribute data. This is thanks to the open source geodiff library that we have developed while working on Mergin.
The first trick is that synchronisation of GeoPackage files between Mergin server and clients (Input app, QGIS or other apps) only transfers actual changes in tables (“diffs” in technical jargon).
Our Mergin project with the tree survey has been prepared and downloaded by users. Jack did a field survey and he added or updated some rows in the survey table (changes highlighted in yellow and green):
After pressing sync button, his changes are detected and uploaded to Mergin, encoded as a list of changes to the survey table:
Another user, Jill, also downloaded the tree survey project to her mobile device prior to Jack’s changes. When Jill synchronises the project to get the latest version, the changes as uploaded by Jack are downloaded and applied to her local copy of the project, getting the same data as seen by Jack.
At this point, the advantage of uploading/download only changes in tables may not seem obvious besides saving some network bandwidth… Read on to learn how this is used to support multi-user editing.
So far we have expected that Jill does not have any pending changes to sync, so that was easy. Now let’s assume that Jill has also done some changes on her device:
Here comes the more tricky part - how do we merge changes from Jack and Jill back to a single table:
In Mergin, cases that require merging changes from multiple users are handled by the “rebase” operation, a concept we have borrowed from version control systems for source code.
Let’s assume that Jack has synchronised his changes first. Later, when Jill synchronises her changes, a couple of things happen on her device before uploading the changes: Jill’s changes will be temporarily undone, Jack’s changes get applied, and finally Jill’s changes are re-applied after being rebased on top of Jack’s changes.
What does it mean to rebase someone’s changes? There are a couple of possible edit conflicts that could happen between rows of a database table with matching IDs (insert/insert, update/delete, delete/delete, update/update). These edit conflicts need to be resolved in order to be able to merge changes from multiple users.
In our example, both Jack and Jill have added a row with ID = 4. This is not allowed, and therefore Jill’s new row ID will get changed to ID = 5 (any unused ID would do). As a result, here’s how the merged table will look at the end - combining changes of both users:
If both Jack and Jill modified the same row (the update/update edit conflict), we can only accept one edit automatically. The conflicting edit of the other user is written to a special conflict file and uploaded to Mergin, so no data gets lost, and the conflict can be later inspected by the project admin. Fortunately, this kind of conflict does not happen often if the survey work is well planned to avoid users simultaneously modifying the same features within the GeoPackage data.
There are some cases when automatic merging is not supported. In those cases, Mergin is unable to find out details about changes in the data file(s) and has to resort to creation of a conflicting copy which gets uploaded to Mergin project along the original data file(s). In particular the problems may appear when:
In the future, these limitation may be removed, but at this point it is good to keep them in mind.
If you plan to change structure of the survey tables and the project is already being used on multiple devices, it may be a good idea to create a new Mergin project with the modified database structure and instruct users to switch to the new project. Otherwise conflict files may start popping up as long as some users have older version of the project, adding more manual work to collate data.
This article follows the help document about the projections and coordinate reference system handling in QGIS.
The accuracy of the captured points is affected by two main factors
Lets take a closer look at both problems in context of field surveys.
GPS accuracy depends on the quality of your GPS receiver and the number of visible GPS satellites your receiver can use at the moment of capturing the point.
When you load the map in Input app, the bottom bar contains the GPS marker with a coloured dot. The dot can have the following three colours:
For different use cases, acceptable accuracy is different. So the threshold for the colour scheme could be adjusted in the Input app settings. Always adjust the settings based on your project needs and check the GPS accuracy when capturing data.
The GPS receiver itself can be either improved by usage of device with a better internal hardware or usage of powerful external GPS receivers If you want to use the external GPS receiver, read the following help article for setup.
First of all, there could be multiple problems with setup of coordinate reference systems and transformations. To solve it, you need to load the project in QGIS, using Mergin Plugin and check for various situations as described in this help article.
Input app checks for missing datum shift files or other transformation problems and notify you once at the project load by following message.
To be able to use the datum grid shift in your mobile app:
proj
folder from your QGIS working directory (see help page for different platforms)For our example with the British national grid on Windows, the grid file is called
OSTN15_NTv2_OSGBtoETRS.gsb
and if the Mergin project is in
C:\users\qgis\myproject
then it should be copied to
C:\users\qgis\myproject\proj
directory and synchronised by the Mergin service to the Input app.
Notes:
You can find a short summary of this article on the Input app help pages
]]>When users take photos in the field, they often would like to record extra information which can help convey a better understanding and awareness of the surroundings. Recent mobile devices allow users to record photo direction, location, time stamp, camera settings, etc when taking photos. This information is often optional or restricted by default (due to privacy) within the mobile settings.
For surveying and data collection, this information can add extra dimension and context. For example, if you are surveying a bat nesting, it is useful to know the directions of the photos you have taken. This will help identifying the site easily in the subsequent site visit.
Geotag information is also useful metadata to have attached to your photos. There are other GIS or non-GIS applications which can read and interoperate the information.
In the recent version of the Input app, we have added a feature which allows you record and display the geotag information. Combined with QGIS styling, you can create a very informative symbology to display the information while you are capturing it in the field.
To be able to capture geotag information, you need to enable your Camera app to use location. For that:
in Android, from Settings > Apps and notifications > (see all) Camera and under Permission ensure you allow Location
in iOS, from Settings > Privacy > Location Services > Camera and the Precise Location is enabled.
If you want to record photos with directions as shown below, you can follow the following steps:
Download and install the Input app
Select My projects and log in using your Mergin credentials
Download the newly created project from the earlier step
Start recording points and add photos, you should see photo direction
The project configuration is done in QGIS. The form has been set up to allow you take photos using Attachment widget. To learn more about different form edit tools in Input app and how to set the up in QGIS, see the Input app help pages.
There are multiple fields to extract geotag information from the photo, as soon as you add the photo either using the camera or the gallery:
read_exif_img_direction(@project_home + '/' + "photo")
read_exif_latitude(@project_home + '/' + "photo")
read_exif_longitude(@project_home + '/' + "photo")
As noted, all the above functions take the path to the photo (@project_home + '/' + "photo"
) and return different metadata related to the image.
In addition to the form, the layer styling has been modified to resemble the direction of the camera and field of view.
To set the camera icon and direction of the camera:
To add field of view:
wedge_buffer(center:=$geometry,azimuth:= "direction",width:=45,outer_radius:=0.0008)
Input, a field data collection app based on QGIS. Input makes field work easy with its simple interface and cloud-based sync. Available on Android and iOS.
If you would like to add a new feature or suggestions to improve the app, do not hesitate to contact us on info@lutraconsulting.co.uk
]]>We have received a lot of feedback from users and we would like to summarise the most common problems people have faced, and add a couple of tips on useful features that are otherwise easy to overlook.
1. I am unable to load LAS/LAZ files
If your QGIS installation does not recognize .las or .laz files (which are the two point cloud formats that we currently support in QGIS), the most likely source of the problem will be that the PDAL library needed for reading LAS/LAZ is missing in your installation. Unfortunately not all installers include it at this point. Particularly on Windows, there are several options how to install QGIS, and only one choice is the right one for PDAL support. At the time of writing (April 2021), you need to download installer from section called “Standalone installers from OSGeo4W testing packages (MSI)”, here’s a screenshot of what to look for:
On macOS the official all-in-one installers include point cloud support. On Linux, PDAL support depends on the particular distribution/packages, but probably most of them include point PDAL library.
2. Point cloud is completely flat in 3D
It may happen that you load a point cloud layer, then open the 3D map view and the point is displayed as a flat surface like on this screenshot:
The reason for this is that the 3D renderer is not enabled for your point cloud layer. To enable 3D renderer, open the Layer Styling panel (there’s F7 shortcut to open it!), then switch to the second tab (“3D View” - with a cube icon) and change the “No Rendering” option to some other option - for example “Classification” in case your point cloud is classified. You should then see your point cloud in 3D.
3. Point cloud is rendered twice - in 3D and in 2D (“flat”)
Commonly when people open 3D view with a point cloud, they may see the point cloud rendered twice, like in the following screenshot:
The reason is that both 3D rendering and 2D rendering of point cloud is enabled, and therefore the layer is also rendered as a 2D map texture on top of terrain (which is flat by default). An easy way how to fix this is to set 2D rendering of the point cloud layer to “Extent Only” in Layer Styling panel (in the first tab):
If the dashed rectangle marking the extent is still bothering you, it is possible to change the line symbol to use white (or transparent) colour.
Hopefully in near future we would address unexpected behaviour and layers with a 3D renderer defined would not be rendered as 2D.
4. I still can’t see my point cloud in 3D view
It could happen that if your point cloud is for a small area, yet the elevation of points is relatively high: when you first open 3D view or when you click “Zoom Full” button, the view may get zoomed too close and the actual point cloud data may be behind the camera. Try zooming out a bit to see if it helps. (This is a bug in QGIS - at this point “zoom full” ignores extra entities and only takes into account terrain.)
5. Enable “Eye Dome Lighting” in 3D view
For a much better 3D perception of your point cloud, try clicking the “Options” button (with a wrench icon) in the toolbar of 3D view and enable “Show Eye Dome Lighting” in the pop-up menu. This will apply extra post-processing that adds slight shading based on the positions of nearby points, and adds silhouettes when there is a sudden change in depth:
As soon as you zoom into your point cloud more, to the point when individual points can be seen, the eye dome lighting effect will start to disappear. You can try experimenting with the point size (in Layer Panel, 3D View tab) - increasing the point size will help.
- A view with point size 2.0 pixels
- The same view with point size 6.0 pixels
6. Try the new camera navigation mode in 3D view
In QGIS 3.18 we have added a new “Walk Mode” camera navigation mode that is much better suited for inspection of point clouds (compared to the default “Terrain Based” camera navigation mode). Open 3D map view configuration dialogue, pick “Camera & Skybox” tab and set it here:
Control | Action |
---|---|
Mouse move | Rotate camera |
Mouse wheel | Change movement speed |
W / Up | Move forward |
S / Down | Move backward |
A / Left | Move left |
D / right | Move right |
Q / Page up | Move up |
E / Page dn | Move down |
7. Use elevation scaling and offset to your advantage
Sometimes it is useful to modify offset and/or scale of the elevation of points (Z values). For example, if the point elevations do not match your other 3D data, or maybe you have source data where X,Y coordinates are in meters and Z coordinate is in feet!
Another case when this can be useful, is when your point cloud data are further away from the terrain and the default “Terrain Based” navigation mode does not work nicely - it expects that data are near the terrain, and the camera rotates around a point terrain, which may feel strange when browsing point clouds. A workaround is to apply offset to the point cloud layer to move the points near the terrain. For example, this is a point cloud which is roughly at 200 meters elevation (the grey plane is the terrain):
When an offset of -200 is applied to the point cloud in Layer Styling panel, data show up much closer to the terrain and camera navigation feels more natural:
8. Try circular points in 2D maps
By default QGIS draws points as squares as this is the fastest option. But for higher quality output you may want to change point symbol style to circle in Layer Styling panel, which makes things look a little less jagged:
– using squares – using circles
At this point we always use square points in 3D views - in the future we will likely offer circular points in 3D views as well.
9. Give us your feedback
We would love to hear from you about your experience with point clouds in QGIS so far, what features you are missing or what problems you have encountered - feel free to drop us a mail at info@lutraconsulting.co.uk. If you think you have found a bug, best to file an issue for it in the QGIS GitHub repository.
With the QGIS 3.18 release, we know we are only at the beginning of a long journey to provide great point cloud support to the users. The decreasing cost of laser scanning (LIDAR) hardware and increasing availability of photogrammetric methods means that we will be be seeing point cloud data in GIS more and more often. There is still a lot of functionality missing for efficient work with raw point clouds, for easy processing of data and we are still using only a fraction of what the PDAL point cloud library offers. We are dedicated to provide first class support for point clouds in QGIS - however this cannot be done without funding. Therefore, if you would like to help us to push point clouds in QGIS to the next level, please do not hesitate to contact us at info@lutraconsulting.co.uk.
With the new release of the Mergin plugin for QGIS, we have introduced a project wizard, which can help users to package and upload the project to the Mergin service. There is also a new toolbar to simplify your workflow.
The plugin now comes with a new toolbar. The common tasks (syncing the project/data and checking the status of the project) are now available from the toolbar.
To create a new project, you can start from a blank project or alternatively open an existing project. You can then select Create Mergin Project from the toolbar:
A new window will appear which should give you three options (the last two options are available only if you have an existing project open):
New basic QGIS project: if you are new to QGIS, this is a good starting point. With this option, a project will be created. Within the project there will be a survey layer (a point layer) and background map (OpenStreetMap).
Package current QGIS project: this option will create a copy of your project and copies all the files to a single folder. The wizard tries to guess each format and offers users three options to package the layer, keep as is (i.e. the layer will be referenced as is in the new project) or ignore (the layer will not be included in the new project). The default for each layer type is as: the web services (e.g. WMTS, XYZ tiles, vector tile layers) will be referenced in the new project as they are. Vector layers will be all written to Geopackage format (each vector in one Geopackage database). Raster layers will be copied as they are. The layers will be referenced in the new project accordingly.
In the next window, you will be prompted to assign a project name and select a path where your project folder and associated files will be generated (this option is only available for the first two choices):
After the wizard, the new project will be created locally and on the Mergin server.
It is recommended to run the project status after changing your layers and project. This will help getting a list of pending changes and also see any warnings or validations of your project. The warnings are related to restructuring of a Geopackage layer (adding/removing a field or addding/removing a layer in a Geopackage database). Validations can be linked to missing layer or availability of a layer when working offline:
If you have any issues or suggestion to improve the plugin, you can file a ticket on the Github project repository.
Your project is ready to be used in Input. Get the app for your Android or iOS device. Log into the Mergin service and download the project(s) created in the above section on your device.
]]>(Data from UGKK SR, made by Tibor Lieskovsky)
When we announced the campaign in August 2020, the response was overwhelming and within weeks, we managed to reach and then exceed the amount required.
Below is the list of contributors in no particular order:
Mapfly, Ujaval Gandhi from Spatial Thoughts, National Land Survey of Finland, Daniel Löwenborg, BNHR, Imapct GIS, Andreas Neumann, Kanton Solothurn, Switzerland, City of Vevey, Mapping Automation, Service de la géomatique du canton de Neuchâtel, Hans van der Kwast, 3DGeoCloud, Rudaz+Partner AG, Leonard Gouzin, TileDB, Inc.
We wish to also thank the many anonymous contributors who do not appear in this list.
Note: if you have contributed to the campaign but your name does not appear here, you may not have selected the option to be listed in the campaign form. Please contact us if you’d like to be listed.
In addition to financial contributions, we’d like to extend our gratitude to all those who helped spread the word, helped with the testing and provided feedback and sample data.
This is a brief summary of new features from our joint work with North Road and Hobu:
ept.json
file (currently only supporting datasets on the local drive)Please note that as this is the initial release (with over 10 thousand lines of new code related to point clouds), there may be still some rough edges here and there, or some data may not load or display correctly. In case you encounter any issues with the new functionality, please let us know - do not hesitate to create a new QGIS issue
(Data from Helsinki City)
This has been the start of a larger effort to bring full support for point cloud data into QGIS. We, in collaboration with North Road and Hobu are developing requirements for integrating point cloud data processing and analysis, more data formats, better visualisation, profile tools etc. in future releases of QGIS.
If you’re interested in helping shape those requirements or funding such features, please contact us at info@lutraconsulting.co.uk.
To be able to use this feature, you need extra packages and also the latest OSGeo4W installer.
Since the original post, Jürgen Fischer has created a stand-alone MSI installer. You can download the new installer from the QGIS website. The installer is only for 64-bit platforms and does not support MS Windows 7.
Note that there have been several regressions and bugs with the first release of QGIS 3.18.0. The issues are being addressed and soon there will be an updated version available. The above link is only for those who are eager to test the point cloud data in QGIS.
To be able to use this feature, you need extra packages and also the latest OSGeo4W installer.
Note: This is a completely revamped and different packaging system than the current OSGeo4W installer. To avoid any clash with your current installation, it is recommended to use different paths for temporary download files and installation of the new packages. The new packaging only supports 64-bit platform.
1- Download and run the NEW OSGeo4W installer
2- Select the Advanced install and pick qgis-dev from the list of packages
Special thanks to Jürgen Fischer for his hard work on preparing the new packages for Windows.
Once installation is completed, try to run QGIS from the installation path (e.g. C:\OSGeo4W64\bin\qgis-dev.bat). You should be able to load LAS\LAZ file to your map from the Browser panel or the Data Source Manager.
The point cloud data can be visualised in 2D and 3D map canvas.
Please test and let us know if you encounter any problems when loading, viewing or styling point cloud data. The best way to do that is to create a new issue on GitHub: https://github.com/qgis/QGIS/issues
QGIS 3.18 will be released later this week (February 19), so grab your copy of QGIS today and give it a try, so that we can fix any remaining issues before the release!
Problem: I am unable to add any LAS/LAZ point cloud file
Solution: Ensure you have used the correct installer linked above. Development builds of QGIS in the ordinary OSGeo4W installer DO NOT include support for point clouds.
FLO-2D is one of the most widely used commercially available flood models. FLO-2D is capable of simulating urban flooding in high resolution including storm drain systems.
In 2016 the FLO-2D team invited us to develop a set of tools for optimising the flood model build process in QGIS. The resulting plugin allows hydraulic models to be built quickly by leveraging the wide range of tools available in the QGIS ecosystem.
The plugin can be downloaded by following the guidance in the plugin documentation
As the use of open source GIS grows within the water engineering sector, Lutra Consulting develops and maintains MDAL which makes the visualisation and post-processing of time-varying numerical model results possible in QGIS.
Contact us at info@lutraconsulting.co.uk if you’d like to discuss the benefits of integrating your flood modelling software more tightly with QGIS.
FLO-2D’s focus on urban modeling requires large datasets that may include several million grid cells. Each cell has between 4 and 10 attributes so datasets can often be in the range of several gigabytes.
Previously, Grid Developer System (GDS) was used to pre-process the spatial data used by FLO-2D. Being a 32-bit application, the GDS was only able to load up to 4 gigabytes of data and its programming framework was also no longer maintained by Microsoft. Modellers could only be built on Windows PCs.
All development and maintenance of the GDS was carried out by the FLO-2D team.
We assessed the tools used at the time and in close collaboration with the FLO-2D team, optimised the workflow from the users’ point of view with emphasis on speed and simplicity.
To keep things simple for the users, we aimed to provide a level of abstraction so users would no longer need to be involved with the internal structure of FLO-2D solver input files. They could instead focus on real-world aspects affecting their models.
Together with the FLO-2D team we designed a solution that was based on 3 core ideas:
Some of the key features of the solution are listed here:
Benefits for the developers of FLO-2D include:
Some of the benefits realised by FLO-2D customers include:
Do you have any questions or would like to see demo of QGIS Mesh Layer? Contact us at info@lutraconsulting.co.uk or schedule a demo call calendly.com/saber-razmjooei/15min
QGIS, plugin, python, migration, optimised, speed up, fast, hydraulic modelling, water, 2D, open-source, cost reduction, software development
Nowadays, it’s really easy to take georeferenced photos on site visits. Tools like Input can capture photos, descriptions and location information and bringing it all into GIS is straight-forward. However.. I recently discovered that people still spend significant amounts of time organising their photos into reports outside of GIS. This small plugin automates the process of getting the photos and their descriptions into Microsoft Word.
For this blog post I knocked-up a simple survey project based on the Field notes Mergin template. Points can have a photo, title and description amongst other fields. No comments on my gardening skills please!
When I sync the data back to QGIS, the attribute table looks something like this:
We’ll now use the HTML Table Exporter plugin to export the layer as an HTML table (this will let us get it into Word).
First install the plugin.
It’s an experimental plugin so you’ll first need to check the Show also experimental plugins option under Plugins > Manage and Install Plugins… > Settings.
When you have installed the plugin, open it using this button or via Plugins > HTML Table Exporter > Export table as HTML.
You should now see this:
Set Table to the layer you want to export. The image scaling option is described later, leave it at a small setting for the time being. Click OK to export and tell the plugin where to save the HTML file.
Now the data should be out of QGIS.. the next steps are in.. MS Word :o
Right click on the exported html file and open it with Microsoft Word:
Let’s see what it looks like:
OK.. we’re getting somewhere! At this point you’ll want to:
If playing with rotation, just focus on rotating a single image, we’ll batch rotate later as required. For now you want to get a feel of whether you want to adjust the scale factor in QGIS (to make the images smaller or larger) to save you having to resize them individually in Word.
I’ve decided to increase the scale factor from 10% to 15% so will now re-export.
Beware that Word has an exclusive lock on the html file when it’s open so you need to close it in Word before you can export it again from QGIS.
I settled for 10% in the end so I could get multiple images on each page in portrait mode. After removing the columns I didn’t want, the table in Word now looks like this:
To rapidly rotate images by 90 degrees, rotate the first one using Right click on photo > Size and position > Rotation then select subsequent photos and press the F4 key. This method is described in more detail here as well as other Word batch image rotation methods.
My document is almost finished. There are just a few small issues to iron out. Currently, the images are referenced by the Word document, not embedded. This means if I email the document to someone, the images will be missing. Let’s fix that by embedding the images in the word document.
First save the document as a Word document in Word’s native format (e.g. *.docx).
Next, locate the Edit Links to Files option:
Select all the linked images (the shift and arrow keys help here) and check the Save picture in document option and click OK:
Save the document, your photos should now be embedded within the document.
If you find your word document gets huge, you can use the method here to quickly batch compress all images in the document.
Input is a free and open source field data collection and mobile GIS app based on QGIS.
If this guide saved you some time and you feel like doing something awesome for us, a review of Input on the Apple App Store or Google Play Store would be really well appreciated.
]]>The releases of QGIS 3.16 LTR/QGIS 3.18, MDAL 0.8.0 and Crayfish 3.5.0 are planned for 19 February 2021. We are delighted to present the following improvements for the upcoming releases:
If you’d like try the latest features, you can always install QGIS nightlies/master, which comes with all the latest features described in this blog post.
If you want to learn more about Mesh Layer in QGIS, read more here…
The Virtual dataset groups for mesh layer and In-memory mesh datasets with persistence improvements greatly improves the workflows when using the Mesh Calculator in QGIS. Users can store the intermediate results into virtual layers that are recalculated on the fly (similarly to QGIS expressions for vector layers). The layers can be later persisted to any supported MDAL formats with write capabilities.
Multi identify tool for mesh feature allows to browse the temporal mesh data in more intuitive way and includes the extra information about the Mesh datasets loaded.
These features were sponsored by Artelia Group.
MDAL 0.8.0 supports loading of the external drivers. A first driver, available on Windows QGIS 3.18 only, is popular DFSU format by DHI, which is used to store MIKE 21 output results.
You can see how to configure and use QGIS to work with DFSU format on the DHI’s YouTube channel
Special thanks to the sponsor DHI this feature.
We have added the following new features to QGIS to convert between mesh and vector/raster:
A more cumbersome way around this, is to convert your tables from PostGIS to a file based GIS layer (e.g. GeoPackage) and take the files with you to the field. This will create a new problem: keeping all the tables (from multiple surveyors) and the PostGIS table up-to-date.
During a survey to assess water access for villages in Limpopo province, South Africa, our friends at Kartoza have commissioned us to extend the Mergin service to support PostGIS. The Mergin service already supports file-based data synchronisation. The aim was to bridge the gap between the Mergin service and PostGIS so that the changes from Mergin immediately appear on PostGIS and vice versa.
To facilitate that, we further developed the Geodiff library to support Postgres driver. In addition, we developed mergin-db-sync tool to sync the tables from Postgres database with the Mergin service. The mergin-db-sync tool runs as a service (daemon) that keeps an eye on a particular Mergin project, and if there is a new version of the project, it will fetch the most recent changes and apply them to database tables in PostgreSQL. It works also the other way around at the same time: it looks for changes in the configured PostgreSQL schema and upload them in a new version of the Mergin project if any changes were detected. This service can be easily started on the local Intranet (where the PostgreSQL database is run) and therefore it does not need any adjustments to the firewall to allow access to the local database from public Internet.
The above diagram details how Postgres/PostGIS synchronisation works with the Mergin service via the DB-Sync tool.
From the surveyors’ point of view, the extra set up to sync with the Postgres/PostGIS does not affect their workflow. In fact, mergin-db-sync tool acts as another client syncing data to the Mergin project, therefore it is possible to see all the changes in the project log originated from mergin-db-sync tool.
The tool is available on GitHub with a permissive open source license (MIT). At this point it supports PostgreSQL, but the mechanism is fairly generic and support for other database engines may be added in the future without great effort. All the heavy lifting is done by the Geodiff library which has been significantly enhanced during the development of mergin-db-sync tool.
To try the tool, please follow the instructions on the project’s readme on GitHub. The easiest way is to use it in a Docker container.
If you have any issues or feedback to enhance the tool, you can file a ticket on the project repo.
If you’d like to set up DB-Sync tool with your Mergin survey projects, you can contact us on info@lutraconsulting.co.uk
]]>In the previous version Serval allowed for setting a constant value to a single raster cell at a time. The latest version brings in new raster cells selection tools and new ways of manipulating cells value:
Multi-bands rasters are fully supported - each band can be edited separately. Serval is meant to provide a convenient way to modify selected parts of a raster layer. It is not optimized to process entire images - use Raster Calculator for that.
In order to modify raster cells values users need to select them first with a line or polygon selection tool. The line selection tool has a configurable width and unit. Selections can also be easily imported/exported from/to a vector map layer.
Multiple selections are allowed and users can add or remove from existing selection using CTRL and SHIFT keys modifiers, respectively.
In addition to setting a constant value, new cells value might be evaluated using QGIS expressions.
Users can build their expressions using cells’ row
and column
variables, or x
and y
of cell center point.
In the Serval
group of the builder users will find several expression functions for:
With the recent QGIS additions it is very easy to create mesh layers. Users can use a mesh layer for raster cell value interpolation.
For example, interpolate_from_mesh()
function will identify mesh dataset value and return it as a new raster value, optionally only when the value is higher than existing cell value.
It is a convenient way for finding an embankment shape on a raster DTM - see pictures below.
Another useful function finds the nearest point on a 3D line feature and interpolate its z-value. It could be used for fine-tuning terrain elevation from a 3D profile line.
For example, if the existing embankment crown needs to be raised by an amount at one of its ends, a user could create a profile line, adjust the vertex elevation in the Vertex Editor and use it for the interpolation.
In the Expression builder users can define their own functions using other types of vector and raster layers.
For eliminating peak values or local cell values averaging, users can use 3x3 low-pass filter.
For more details and examples, refer to the plugin’s Manual.
Now it is possible to add textures to vector layer data (buildings) in the 3D viewer: To do that you need to go the vector layer properties and select Realistic Textured (Phong) shading mode and select the image to be displayed on the surfaces by providing an image file to the diffuse texture field. You can adjust the scale of the image and its rotation from the texture scale and texture fields respectively. There is a workaround to seperate roofs from walls using rule based rendering by adding 2 seperate rules and selecting which facet (roof or a wall) to display and select which rendered facade to be used and the shading setting for each rule. Results for seperating roofs and walls
QGIS 3.16 adds support for directional lights that you can think of as the sun. The default light in QGIS 3.16 is now directional which makes the lighting of the whole scene better. To use directional lights, open the 3D configuration dialog and from the lights tab you can configure how many lights you use. If you select a directional light, you will see the new improved way to provide the direction of the light (Thanks to Vincent Cloarec for the UI improvements).
In the 3D viewer there is a new tool to export the scene as a .obj file and visualize or edit it in other software like Blender or MeshLab. To export the scene, from the 3D viewer select the 3D object icon and you will see the following dialog: You can select the name of the .obj file and where to save it (you need to select a folder because you will have multiple files if you export textures or color informations). You can specify the resolution of the terrain in case you’re using a DEM as well as the resolution of the exported textures. Here is a view of what an exported scene that has multiple rendering rules looks like in MeshLab:
You can add a skybox to your scene from the 3D configuration dialog and selecting from the 2 skybox formats provided:
Althought it might not work for everyone, shadow rendering was added as a technical preview in QGIS 3.16. To try shadows, you need to open the 3D configuration dialog and enable shadows from the Shadows tab. You can experiment with the shadow rendering parameters if you want to decrease the visual artifacts and make the shadows look better in the scene. Here is how the shadows look like in our scene:
If you want to try the new packages, download and install the QGIS from nightly builds. There are still few weeks before QGIS 3.16 release for testing and bug-fixing, so make sure you report all your issues before the 23rd October 2020. Multiple packages now can be installed side-by-side, just rename or move the installed QGIS.app!
The earlier packages were based on Homebrew, but we didn’t have control over the versions of dependencies. We switched to the new system where we have full control, which is important for good quality releases.
The package/installer is not yet notarized by Apple, so you need to right-click on the QGIS.app icon and open it to overcome the security control of your macOS (only for 10.15+).
If you want to join the effort in testing and/or development of macOS packages, please drop me a mail on peter.petrik@lutraconsulting.co.uk We have a dedicated Slack channel to discuss the maintenance of the macOS packages.
The goal is to have all advanced functionality of QGIS prepared and ready to use after simple one-click installation.
QGIS Desktop, of course, but also
/Applications/QGIS.app/Contents/MacOS/bin/qgis_mapserver
)/Applications/QGIS.app/Contents/MacOS/bin/qgis_process
)/Applications/QGIS.app/Contents/MacOS/bin/designer
)with pip, so you can install the missing packages with command
/Applications/QGIS.app/Contents/MacOS/bin/pip3 install <your package>
but, many packages are already preinstalled for you!
All basic providers
But also:
In Spring 2020, we prototyped the building of FOSS macOS libraries in completely controlled environment. Few weeks ago we have successfully finished the QGIS 2020 Grant “QGIS macOS Package Improvements”. This wouldn’t be possible without support from QGIS.org and its sponsors. And without proper testing and reporting of issues from our macOS power-users.
Do you want to see your QGIS projects and data from your iPhone and iPad? Check InputApp
With the proposed changes, you will be able to load, style and visualise your point cloud data in QGIS in 2D and 3D map views.
The work will be carried out by the trusted and highly skilled developers across PDAL and QGIS community (Lutra Consulting, North Road and Hobu) who have been at the forefront of some of the exciting features in Open Source projects.
If you or your organisation are point cloud data users, this is your chance to bring native support for your data in QGIS. With this work, you will be able to overlay your point cloud data to your other data (vector/raster). The addition of native support for point cloud data in QGIS will pave the way to support analytical tools for point cloud data in future.
The target amount is 49,000 € and the campaign will be active until 15 October 2020.
Please have a look at the dedicated page point cloud data support in QGIS for further details and help us spread the word!
The releases of QGIS 3.14, MDAL 0.6.1 and Crayfish 3.4.4 are planned for this Friday We are delighted to have made improvements for the upcoming release:
If you are eager to try the latest features, you can always install QGIS nightlies/master.
Special thanks to all contributors and the sponsors of these features
Do you want to use QGIS Mesh Layers in your projects? Read more…
The long awaited native temporal support in QGIS is finally materialised. We managed to port all QGIS Mesh Layer code to the new infrastructure and remove the custom time slider in the Mesh Layer properties/styling dialog. The native temporal support will greatly benefit users in the long run. We can now share the code components for time handing with raster, vector and WMS-T and other temporal data types. Also user can load multiple temporal layers and change time-domain for all of them at the same time.
Unfortunately, the new temporal framework changed the API, so if you use QgsMeshLayer in your plugins, consult the QGIS documentation for required changes.
We have started writing the proper documentation for all supported formats and also reference documentation for developers. We welcome all contributions to improve the documentations. Just go to https://www.mdal.xyz and click “Edit on GitHub”!
MDAL and QGIS now supports 1D meshes, currently for UGRID and 3Di formats. Other formats should be easy to add. If you are interested in supporting a format, drop us a line.
You can read the full technical description in the following QGIS Enhancement Proposal.
The 1D meshes can be styled with the various renderers, for example use of pressure data to calculate line width:
Furthermore, Crayfish plugin has been extended to support such mesh data frames. For example, 1D plots in Crayfish with the snapping and routing along the network can be used:
2D meshes streamlines/particles/arrows can be now styled by the colour ramp. You can better display streamlines and mesh vectoral data varaition:
The mesh layer is now fully supported in the 3D map view. You can visualise your terrain and overlay vector arrows (e.g. velocity) and other mesh quantities (e.g. water surface) in the 3D map view.
In this blog post, we are exploring different methods to add your vector tiles and style them.
If you have an MBTiles file containing vector tiles, you can simply drag and drop the layer in QGIS. Alternatively, you can connect to a local vector tile file(s) from the Data Source Manager or the Browser Panel. As an example, you can use this QGIS project which contains Switzerland vector tiles (maximum zoom level =14).
You can also test this project on your mobile device. Download Input app (for iOS you need the TestFlight version of the app), head to Projects > Explore and download saber\blogpost-vectortile.
You can access vector tiles served through the web. You need to have the URL and possibly API key to add them in your QGIS. In the example below, vector tiles from MapTiler are added to QGIS:
To use MapTiler data, we recommend using their dedicated QGIS plugin, which comes with vector tiles in different styles. The plugin should be available on the QGIS plugin repository.
Vector tiles contain points, line and polygon geometries. You can define a set of rules for features and apply style and label. Style and labelling can be dependent on the zoom level. Similar to vectors and rasters, styles for vector tiles can be stored in QML format. To set up filters, you can use the identify tool in QGIS to inspect geometries of your vector tile:
Currently, the most common format to store vector tiles styles are Mapbox GL (json). The OpenMapTiles repo contains some excellent Mapbox GL styles. We initially developed a tool to convert the json files to QGIS style. The plugin has been further extended and developed by MapTiler to better handle the OpenMapTiles data schema. This will ensure future changes to the data schema will be addressed by the plugin.
Example of styles converted from Mapbox GL to QGIS from Maptiler QGIS plugin
In the next post, we will explore different options within QGIS or other 3rd party tools to generate vector tiles.
All the data used in this blog post was from OpenStreetMap server by Geofabrik. OpenMapTiles conversion tool was used to generate the tiles from OpenStreetMap to vector tiles. See software license for more details.
The following OpenMapTiles Mapbox GL styles Bright, Positron, Basic and Fiord were converted using mapbox2qgis and QGIS MapTiler plugin.
Overview of the changes
In 2019, we upgraded the QGIS macOS packaging to address several issues at the time. This was made possible with the help of the QGIS project and donors who supported our work.
The upgraded system had a more transparent and automated packaging approach. But it came with its own limitation. We have been using Homebrew to fetch the QGIS dependencies and to compile only QGIS for the LTR, PR and nightly releases. The main problems with this approach are:
Homebrew can only support one version of a package. This will limit QGIS to be built against multiple versions of, for example GDAL or Proj libraries. In one hand, we want QGIS nightlies to use more bleeding edge versions of the dependencies (e.g. for plugin or core developers) . On the other hand, using a more stable and tested versions of the dependencies for PR or LTR are not recommended (e.g. users and organisations). OSGeo4W offers the same approach for different versions of QGIS in Windows. Unfortunately, Homebrew does not offer such an option.
Continuous integration has been a major part of QGIS infrastructure to ensure pull requests by developers do not break workflows in certain Operating Systems. Microsoft Windows and Linux have been well supported in the CI. But with the macOS packages, it was left to user to test and report back any bugs after Pull Requests were merged to QGIS source code.
With the recent work, we have addressed both of the above issues. QGIS and almost all of its dependencies are built from sources. It will be also possible to easily integrate the code with the current CI for macOS with GitHub Workflows. This will bring more stability and control over the way we build and package QGIS for our users, whilst helping developers to identify issues with their pull requests for macOS platform, before merging them to the QGIS source code.
If you want to check the latest QGIS master on macOS or tweak the code to see some new exciting features, now it should be easier with the following steps, to compile QGIS from source code an a clean machine:
sudo xcode-select --install
install_qgis_deps.bash
script to install Qt in /opt/Qt/5.14.1/clang_64
and QGIS-Deps to /opt/QGIS/qgis-deps-0.3.0
folder. You may need to use root privileges or create those folders under /opt/
and assign the right permission beforehand.You may use MacPorts, Conda or other packaging system, but by the end you need cmake
, git
, astyle
and other useful tools to be on the system PATH
~/Projects/
), clone QGIS with git clone git@github.com:qgis/QGIS.git
mkdir -p ~/Projects/build-QGIS
cd ~/Projects/build-QGIS
QGIS_DEPS_VERSION=0.3.0;\
QT_VERSION=5.14.1;\
PATH=/opt/QGIS/qgis-deps-${QGIS_DEPS_VERSION}/stage/bin:$PATH;\
cmake -DQGIS_MAC_DEPS_DIR=/opt/QGIS/qgis-deps-${QGIS_DEPS_VERSION}/stage \
-DCMAKE_PREFIX_PATH=/opt/Qt/${QT_VERSION}/clang_64 \
../QGIS
Note that all libraries are picked from qgis-deps and not from system /usr/lib
or Homebrew’s /usr/local/
or system Frameworks /Library/Frameworks/
. Especially check Proj, GDAL, sqlite3 and Python paths.
You should see output similar to this:
-- QGIS version: 3.13.0 Master (31300)
-- Found OpenCL: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/OpenCL.framework (found version "1.2")
-- Found OpenCL C++ headers: /Users/peter/Projects/mesh/QGIS/external/opencl-clhpp/include
-- Found GRASS 7: /opt/QGIS/qgis-deps-0.3.0/stage/grass78 (7.8.2, off_t size = 8)
-- Looking for openpty
-- Looking for openpty - found
-- Found Proj: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libproj.dylib version 6 (6.3.1)
-- Found GEOS: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libgeos_c.dylib (3.8.1)
-- Found GDAL: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libgdal.dylib (3.0.4)
-- Found Expat: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libexpat.dylib
-- Found Spatialindex: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libspatialindex.dylib
-- Found Qwt: /opt/QGIS/qgis-deps-0.3.0/stage/lib/qwt.framework (6.1.4)
-- Found LibZip: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libzip.dylib
-- Found libzip: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libzip.dylib
-- Found Sqlite3: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libsqlite3.dylib
-- Found Protobuf: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libprotobuf.dylib (found version "3.11.4")
-- Found Protobuf: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libprotobuf.dylib
-- Found ZLIB: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libz.dylib (found version "1.2.11")
-- Found zlib: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libz.dylib
-- Found PostgreSQL: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libpq.dylib
-- Found SpatiaLite: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libspatialite.dylib
-- Qt WebKit support enabled
-- Found Qt version: 5.14.1
-- Found QScintilla2: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libqscintilla2_qt5.dylib (2.11.4)
-- Found QtKeychain: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libqt5keychain.dylib
-- Found QCA: /opt/QGIS/qgis-deps-0.3.0/stage/lib/qca-qt5.framework (2.3.0)
-- Found QCA OpenSSL plugin
-- Found Libtasn1: /opt/QGIS/qgis-deps-0.3.0/stage/include
-- Pedantic compiler settings enabled
-- Found PythonInterp: /opt/QGIS/qgis-deps-0.3.0/stage/bin/python3 (found suitable version "3.7.7", minimum required is "3")
-- Found Python executable: /opt/QGIS/qgis-deps-0.3.0/stage/bin/python3
-- Found Python version: 3.7.7
-- Found Python library: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libpython3.7m.dylib
-- Found Python site-packages: /opt/QGIS/qgis-deps-0.3.0/stage/lib/python3.7/site-packages
-- Found PyQt5 version: 5.14.1
-- Found SIP version: 4.19.21
-- Found QScintilla2 PyQt module: 2.11.4
-- txt2tags not found - disabled
-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY
-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY - Success
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY - Success
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR - Success
-- Found exiv2: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libexiv2.dylib
-- HDF5: Using hdf5 compiler wrapper to determine C configuration
-- Found HDF5: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libhdf5.dylib;/opt/QGIS/qgis-deps-0.3.0/stage/lib/libz.dylib;/usr/lib/libdl.dylib;/usr/lib/libm.dylib (found version "1.10.0")
-- Found PkgConfig: /usr/local/bin/pkg-config (found version "0.29.2")
-- Found NetCDF: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libnetcdf.dylib
-- Found LibXml2: /opt/QGIS/qgis-deps-0.3.0/stage/lib/libxml2.dylib (found version "2.9.10")
-- Looking for updwtmpx
-- Looking for updwtmpx - not found
-- Found GSL: -L/opt/QGIS/qgis-deps-0.3.0/stage/lib -lgsl -lgslcblas
-- Using PROJ 6 srs database.
-- Ctest Binary Directory set to: /Users/peter/Projects/mesh/build-QGIS/output/bin
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/peter/Projects/mesh/build-QGIS
make -j4
and wait for [100%]
mark :)./output/bin/QGIS.app/Contents/MacOS/QGIS
or open ./output/bin/QGIS.app
Notes:
QGIS_MAC_DEPS_DIR
and you must ensure it contains valid qgis-deps installationIf you want to help with development of the qgis packages on macOS, please feel free to contact peter.petrik@lutraconsulting.co.uk. The development is taking place on public QGIS-Mac-Packager repository.
Currently, we are carrying out the following tasks:
to move all-in-one QGIS bundle for LTR/PR/nighlies to use this new package of dependencies. Check out Twitter for for any news on the packages. We are aiming to have the new packaging in place for the QGIS 3.14 release, which will bring GDAL 3 and Proj 6 to macOS users
to sign the qgis-deps packages, so that developers do not need to tweak Privacy settings to use it for compiling QGIS and its dependencies.
Here is the list of donors who have supported us and the wider QGIS user by funding this feature.
(in no particular order)
Also many thanks to those who spread the word and offered us moral support!
In the previous tutorial, we set up a simple QGIS project and transferred it to our devices using the Mergin service.
In this tutorial we are going to set up a new survey project with more advanced features:
To start with, you will need to install the following software applications:
In addition, you will need to register with the Mergin service. The Mergin service allows you to transfer data between your PC/laptop and mobile/table via the cloud. Note that after signing up to the Mergin service, you have to activate the account by clicking on the link sent to your email.
Similar to the previous tutorial, we are going to start with a blank QGIS project. All the data and the project will be stored locally on a folder called recording tracks.
First, start a QGIS project and add the OpenStreetMap layer under XYZ Tiles. You can add also add aerial imagery as an XYZ layer to your QGIS.
Save your project as tracks in the recording tracks folder.
In addition, we need to create a Geopackage survey layer. Below are the attribute columns for your survey layer. Also note that need to select Line as geometry type.
To create a survey layer, in QGIS, from the main menu select Layer > Create Layer > New Geopackage Layer …. A new window will appear:
For Database click on … and select the recording tracks folder then type survey.gpkg for the name of your database.
For Table name, type tracks.
For Geometry type, select Line.
You can select the option to Include Z dimension. The upcoming version of Input app supports capturing altitude from your device (or external GPS device tethered with your phone/tablet).
For Coordinate Reference System (CRS), select WGS 84 / Pseudo-Mercator EPSG:3857.
Add the following Fields. Note, different field types:
Click OK to create and add the layer to your QGIS session.
We are going to style the tracks layer based on Type field and use four categories:
In the Layer Panels right-click on tracks layer and select Properties. A new window will appear. From the left panel, select Symbology. Change the styling from Single to Categorized on the top left section of the window. For the Value, select Type from the drop-down menu.
In the bottom-left section of the window, click on Classify. Then add 4 classes (by adding the green plus adjacent to Classify) for track types. In addition, we can add another class for Other types. Adjust the colours and line style accordingly:
To set up the form, select Attribute forms from the left panel of the layer properties window. Adjust the Widget type as below:
Input app allows you to have a custom pop-up drawer window when you tap on feature. To customise that, you can change the Display settings under the layer properties window. For Display name select Type and in the HTML Map Tip section type:
# image
file:///[%@project_folder%]/[% "Photo" %]
We can set up two different map themes: one with aerial imagery and one with street map. To do that, ensure you have only the following layers are set to visible in your layer panel:
On top of the layers panel, select Manage Map Themes > Add Theme …
A new window will appear. Type Aerial photo for the name of your map theme and press OK.
Now, turn off your Aerial images layer in the layer panel and turn on the OpenStreetMap layer:
You can now add a new map theme and call it Street map.
Save your project and transfer your data and project to your phone through the Mergin service as described in the previous tutorial.
If you missed a step, you can see the final version of the project here. You can also skip the above section by cloning the project in the above link.
In Input app, download and open Tracks project.
You can easily change between different map themes in Input, by going to (… More) > Map Themes:
To capture tracks, press Record button in the lower panel in Input. A green cross will appear, centred on your GPS location. You can capture a line in two different methods:
You can change the frequency of adding points when streaming GPS location, under Settings in Input. Also note that the GPS streaming can be done for polygon survey layer.
Once you finished capturing the track, press Done and you will be presented with a form:
You can easily fill the form using the drop-down menu for Type and checkbox option for Public.
By tapping on tracks on the map, you will should be able to see a small preview window presenting Type and Photo:
After completion of the survey, you can synchronise your data back through the Mergin and see the results in QGIS desktop.
]]>In this post, we will explore 3d vectors and how to view buildings and trees.
For the purpose of this blog post, we will be using
You can download all the dataset and final QGIS project from our Mergin service. Note that the DTM has been resized from 1m resolution to 5m. All of the datasets have been clipped to Luxembourg commune extent.
After styling the layers you should be able to see an image like the one below in your map view.
You can open a new 3D map window and set the DTM layer as your terrain as shown in the earlier blog post.
The other layer we want to view in 3D canvas is the building layer. To adjust the settings, you need to change the layer styling (by pressing F7 to open the styling panel):
The buildings layer contain height information:
In case, your building layer is not a 3D dataset, you can apply height and/or extrusion to be able to see them in 3D map view.
Culling mode helps rendering your scenes faster by not drawing certain features facing the camera. You can read more about culling modes here and here.
You can change Diffuse, Ambient, Specular and Shininess of the polygons using relevant widgets. These parameters are based on Phong reflection model.
Finally to better display the buildings, you can define the Edges.
You should be able to see a 3d scene similar to the one below:
Points can be displayed in 3D map view as different type of objects. In this example, we are going to display trees as combination of predefined shapes. You can extract tree points from the OpenStreetMap data (the osm_pois layer with filter: "fclass" LIKE 'tree'
).
For tree trunks, we can set the 3D properties as follows:
The tree trunks are represented by a Cylinder shape. As you have noted, the Altitude clamping method for this layer is Relative.
For the tree top, we can duplicate the same tree layer in the layer panel and change the styling to be a Sphere shape. You will also need to set the Transformation for Y to e.g. 10, so that the spheres sit on top of the cylinders. For the example below, we have set the styling to be a mix of spheres and cones (Rule-based instead of Single symbol).
3D point renderer allows you to import 3d models supported by Open Asset Import Library.
You can create animations of 3d scenes, by defining key frames. Click on the Animation icon from the 3D map view window. A new toolbar will appear at the bottom of your window.
Click on the green plus sign to add the timing of the key frames and then move the camera to a different location. You can then play the fly-over using different interpolation methods. Alternatively, you can export them as images and generate an animation outside QGIS.
This will be a great step for QGIS to support an increasingly popular technology used in web mapping to deliver maps that are faster and more flexible at the same time.
With native support for Vector Tiles in QGIS, you will be able to add more resources to your QGIS map:
The target amount is 8,000 € and the campaign will be active until 20 March 2020.
Please have a look at the dedicated page Vector Tiles support in QGIS for further details and help us spread the word!
To start with, you will need to install the following software applications:
In addition, you will need to register with the Mergin service. The Mergin service allows you to transfer data between your PC/laptop and mobile/table via the cloud. Note that after signing up to the Mergin service, you have to activate the account by clicking on the link sent to your email.
To be able to survey data, we need to set up a project in QGIS. Usually, you will need some data for your background layer (so that you can locate yourself!). In addition, you need to set up a table (or layer), to store your survey information.
For background data, we are going to use Open Street Map. For survey table, we need to decide on a form structure and the type of feature you want to survey (e.g. point of interest, tracks or parcel of land). In this case, we want to survey potholes. Also, it would be good to attach some notes for each pothole, take a photo of it and add a date for survey. The GIS format best suited to store spatial information, is Geopackage.
Let’s start by opening QGIS and add the above layers to our project. To simplify things, we can create a folder on Desktop (referred to in this tutorial as data collection folder) and store everything there.
Open QGIS from your PC/laptop. From the Browser panel (usually located on the top left side), expand XYZ Tiles and double-click on OpenStreetMap to add it to QGIS:
You should see the OSM layer:
Save your project as pothole survey in the data collection folder.
To create a survey layer, in QGIS, from the main menu select Layer > Create Layer > New Geopackage Layer …. Note that Geopackage is a file based database where you can store multiple tables (spatial or non-spatial). A new window will appear:
For Database click on … and select the data collection folder on your Desktop and then type survey-db.gpkg for the name of your database.
For Table name, type Potholes.
For Geometry type, select Point.
For Coordinate Reference System (CRS), click on the icon to the right of EPSG:4326 - WGS84. A new window will appear. Under Filter section on the top of the window, type: 3857 and under Predefined Coordinate Reference Systems, select WGS 84 / Pseudo-Mercator EPSG:3857. Then click OK.
We can now create the column headers for our table under New Field section. For this form, we want to create the following columns to store data: Date, Notes, Photo
For Name, type Date
For Type, select Date
Click on Add to Field lists to add your column.
Repeat the same process for Notes and Photos columns, but make sure to change the Type for those columns to Text. At this stage, you should see an image similar to the one below:
Go ahead and click OK to create the layer and add it to QGIS.
The default style applied to Potholes layer is not very visible probably. To change it:
In the Layer Panels right-click on Potholes layer and select Properties. A new window will appear. From the left panel, select Symbology. Try to change the style to something shown in the image below:
Click on Apply.
We can also change the way user fills in the form. By default, you have to type in the values. But by using different widgets, we can simplify filling the form in the field.
In the Properties window, from the left panel, select Attribute forms.
We are going to change the Widget Type for each of the Fields.
fid is an auto-increment field and we can keep it hidden from users. So, highlight the fid field under Field section and then from the Widget Type select Hidden
For Data, it should have automatically selected the correct widget type:
For Notes, you can also leave the Widget Type as Text Edit.
For Photos, we need to change the Widget Type to Attachment. Also make sure to select the option for Relative paths. This will allow us to attach photos using mobile camera or gallery folder to the pothole point.
Tip: You can scroll further down and under Integrated Document Viewer and select Type as Image. This will show the image in QGIS forms too.
Project set up is completed and we can save the project.
You have 2 options to transfer your data to the mobile through the Mergin service: through website or through Mergin plugin in QGIS. In this tutorial we are going to use the plugin from within QGIS.
In QGIS, from the main menu, select Plugins > Manage and Install Plugins …. A new window will appear. From the left panel, select All and then in the search section (on the top) search for Mergin. Select the plugin from the list and click on Install plugin. After installation, you need to restart your QGIS.
After the restart, you should be able to see the Mergin icon in your Browser Panel:
In the Browser Panel, right click on the Mergin and select Configure. Type in your username (or email address) and password that you have registered with the Mergin service.
Click on Test Connection and you should see a green OK.
If you have selected to Save credentials (so you do not need to type in the username and password again) and you have not configured QGIS password manager, you will be prompted to set a password for your QGIS password manager.
After clicking OK, you should see a list of folders on your Mergin connection in your browser panel:
We can know upload the data:
Right click on the Mergin and select Create new project. A new window will appear:
For Project name type Potholes survey
Select Initialize from local drive
Click on … and and select data collection folder
Once click OK, the project will be created and content of the data collection folder will be uploaded there.
The project is now ready to be downloaded on your mobile device.
The project can now be accessed from Input app. Open your Input app and for the first time you should see a screen similar to the image below:
To log in to the Mergin service, you can select My projects or the green and white icon on the top right.
Type your Mergin username (or email address) and password and then select Sign in.
Once signed in, select My projects and you will see Potholes survey project in the lists
Select the download icon on the right side of Potholes survey to download your project on the phone and make it ready for survey.
After downloading is completed, select Home and you should be able to see Potholes survey.
Select Potholes survey and it will open the map:
To record a feature, select Record button and the pointer changes to a cross-hair.
By default, the cross-hair centres to your location (the orange point) on the map. You can move the map and adjust the location. To recentre the map to your location, you can select GPS button. Once you are happy with the location, you can select Add point and the form for your point will appear:
Fill in the form and press Save. You should see the map with the newly captured pothole:
The data you have captured on your phone can be synchronised through the Mergin service.
In Input app, select Projects and then My projects. You should see a double arrow on the right side of the Potholes survey.
Select the double arrow to sync your project. You can also open QGIS from your PC/laptop and synchronise changes back to your desktop:
In QGIS, from the Browser Panel under Mergin > My projects right-click on Potholes survey and select Synchronize
After synchronising is completed, you should be able to see the point and its associated form on your QGIS.
Input app’s manual can be found here.
With the Mergin service, multiple users can collect data on the same project. For more information, see this the blog post.
For more information on how to set up complex forms and map themes see QGIS documentation.
]]>The releases of QGIS 3.12, MDAL 0.5.0 and Crayfish 3.2.1 are planned for end of February 2020. We are proud to present you few of upcoming features we implemented for this release:
If you are hesitant to wait till end of February, feel free to get nightly build and test it out!
Do you want to use QGIS Mesh Layers in your projects? Read more…
Last feature from QGIS 2.x/Crayfish 2.x series that was not ported to QGIS 3 is finally available. You would be able to visualize streamlines and particles for vector datasets in mesh layers. In QGIS main menu, under Mesh>Crayfish>Export Trace you are also able to export animation with the particle traces to various video formats
This feature was funded by TUFLOW
MDAL and QGIS now supports 3D Stacked Meshes, particularly for TUFLOW-FV format. For this release, you need to choose appropriate averaging method in the QGIS interface and you are able to browse the data similarly to any other 2D dataset.
In Crayfish 3.2.1, you can create plots of the profile showing the variation along Z-axis.
The technical description can be found in the following QEP
This feature was funded by TUFLOW
For datasets defined on faces, one can choose to interpolate data to vertices with neighbour average method. When no data interpolation method is chosen, each pixel on a single face has a single value/color. With data on vertices, the rendering for each pixel is interpolated from the values on the vertices, making smoother figures.
Use mesh contours styling panel to switch between the data interpolation methods.
This feature was funded by Austrian Ministry of Agriculture, Forestry, Environment and Water Management
We have implemented a new algorithm in QGIS’s analysis library to export directly contour lines and polygons. The method is not based on GDAL as it was in the Crayfish 2.x releases. It is both faster and with smoother shapes, matching rendered images from QGIS. You can find the new processing algorithm in Crayfish processing toolbox.
This feature was funded by Austrian Ministry of Agriculture, Forestry, Environment and Water Management
From QGIS 3.12 you can use mesh calculator for all datasets, both defined on faces and vertices. Additionally, it allows users to store the result of mesh calculator under different name or format. This allows for example to work with FLO-2D or HEC-RAS data in the QGIS mesh calculator
This feature was funded by Austrian Ministry of Agriculture, Forestry, Environment and Water Management
For various dataset type, for example GRIB and NetCDF, the reference time in QGIS time settings dialog is prepopulated from the raw data and does not need to be set manually. Also we fixed various bugs related to time parsing, so in QGIS 3.12 it should be possible to format and show your time in plots/animations in proper way.
This feature was funded by TUFLOW
MDAL library now has a new utility: mdal_translate. For now, use can use the utility to convert text-based 2dm mesh definition files to UGRID NetCDF/HDF5 binary-based format and save up to 80% disk and speed up loading of your mesh by similar amount.
This feature was funded by TUFLOW
With Crayfish 3.2.1 you can export your time series or cross section raw dat to CSV format for further processing.
This feature was funded by Lutra Consulting
It is often the case, where multiple users need to make changes to a single vector layer. If you work in office, this issue is usually addressed by having a central geodatabase (e.g. Postgres/PostGIS). If you want extra information (e.g. audit trails, versioning, latest changes, etc), you can modify your database, to keep track of it.
The problem arises when you want to collect data, without having access to the central geodatabase. You can do some manual work to handle this scenario, but it can easily lead into data management nightmare.
To simplify the workflow, we have developed Geodiff, a multi-platform library to keep track of changes, calculate the differences, merge and consolidate the differences.
Geodiff has been integrated into the beta version of Input. This will allow you to share a project with your team and edit a single layer (geopackage format) all at the same time, even when you are offline.
To start with, you need to create a project and upload it to Mergin. You can then share the project (with write access) with your colleagues:
The project contains a survey layer (trees.gpkg). There is only one feature present within the layer:
The project is shared with two users. Both users download the project and take their devices to the field:
User 1, carried out a survey (using iPhone!), by adding a tree and editing the attribute table of the existing one. The changes were synchronised back through the Mergin:
Meanwhile, User 2 added a new feature to the survey layer. Once User 2 tries to synchronise the changes, Input automatically detects the changes not only made through User 2, but also the ones uploaded to the Mergin. The layer will be patched both locally and on the server to take all the changes into account:
The data administrator can now pull all the changes from both users in QGIS:
You can also see the history of changes to the project and the survey layer on the Mergin website. Below shows the changes from different users:
To see changes from each user, you can click on the version and it lists the changes. In this example, User 1 (jack) added a new feature and modified an existing feature:
You can also see the extended history and see where the changes have been made:
You can use Beta version of Input app in Android or TestFlight in iOS:
For any issues or feedback, please file a ticket on Input repository
]]>To be able to work with Input, you will need the following:
For the purpose of this workshop, we have prepared a QGIS project. Let’s use that as a starting point:
The project you have copied in Mergin, is a QGIS project with various map layers. To see the content of the project in QGIS:
In Mergin, in the top menu, select Projects > My projects
Select foss4guk_YOURUSERNAME (or the name you assigned when copying the project).
In the top menu, click on the icon to download the project
Extract the content of the zip file
Alternatively:
The above process can be done through the Mergin plugin for QGIS. To do that:
Install the Mergin plugin in QGIS
Restart QGIS
In the QGIS Browser panel, right-click on Mergin and select Configure
Enter your Mergin username and password
Under My Project, right-click on foss4guk_YOURUSERNAME and select Download
Select a location under which the project will be downloaded to
Once downloaded, select Open to open the project.
Input is based on QGIS, therefore, any layer symbology / styles you set in QGIS, will be displayed in Input. If you are using SVGs (e.g. OS MasterMap), you need to embed these in the QGIS project.
Input also supports most of the edit widgets from QGIS. Edit widgets allow you to simplify filling-in forms with drop-down options, TRUE/FALSE switches, sliders, calendar and time, default values, attachments, value relations and more. To see some of those settings:
From the Layers panel (in QGIS), right-click on the listed buildings (points layer) and open the Properties window.
From the left-hand panel, select Attributes Form. Explore the various widgets assigned to different fields.
For this layer, we have set the Photo field to use an Attachment widget. This will allow Input to make use of your mobile camera to attach photos to features.
For the Surveyor field, we have linked it to an external CSV table, to populate a drop-down option with the names of surveyors.
Input can also use a pop-up window (similar to Google Maps) to display basic information about a single feature:
To customise this pop window’s content:
Open the properties table, and select the Display tab
You can see the title is set to ENT_TITLE and there is an image tag referencing the Photo field:
# image
file:///[%@project_folder%]/[% "Photo" %]
To simplify handling layer visibility, Input makes use of map themes defined in your QGIS project. In this project, there is a map theme for aerial photo (using a Bing aerial layer) and OpenStreetMap (geopackage).
[]{#anchor-5}Survey layer
In Input, any vector layer (point, line, polygon) can be edited (as long as editing that format is supported in QGIS). This could be very confusing when dealing with large numbers of vector layers in a single project (trying to figure out which one to edit).
Luckily you can set background layers (or those you don’t want to be editable in Input) to read-only:
In QGIS, from the main menu, select Project > Properties
In the new window, select the Data Sources tab from the left-hand panel
Below is the list of layers and their capability settings for the project. Layers not marked as read-only will be shown as survey layers (editable) in Input.
By default, the file paths to layers are relative. You can change that under the General tab of this window.
To use Input, open the app on your device. On its first run, Input will show the Projects page.
When you open the project, you may not see all layers. This is because some of the layers have zoom-dependant visibility settings (again configured in QGIS).
To switch map themes:
Tap More on the bottom-right side of the screen
Tap Map themes > aerial photo
You can also display feature details simply by tapping on them.
To capture data:
Tap Record
You can then choose the layer in which you want to record your feature, by tapping on the light green band, in the lower part of the screen, above the Input menu.
If you are capturing a point, by default, the suggested point to capture will be on your GPS location. You can drag the map to adjust the location of the new point. To switch back to the current GPS location, tap the GPS icon on the bottom-left of your screen.
After adding a point, you will be prompted to fill-in the form.
If you are recording a line or a polygon, you can either add points to define the shape of your feature or press and hold the GPS icon when in Record mode to generate a shape from your GPS track.
You can edit the existing features on your map. For point layers, you can edit geometry and form data. For lines and polygons, you can edit only the form data.
Let’s get out and capture some data for the Path layer!
Once you have made changes to your data, you can upload them back to Mergin:
In Input, tap Projects
Select My projects
Click on the sync/refresh icon to the right of your project
You can now download the project again to your desktop and see the changes in QGIS. Alternatively, you can synchronise the changes you made back to QGIS by using the Mergin plugin for QGIS (described earlier).
]]>You can also learn more about this GSoC project here.
Previously, user could already navigate the 3D world by using mouse and keyboard. Unfortunately, for a new user it is not easy to start using them. 3D On-Screen Navigation will help navigating the 3D world. There are buttons to do zoom in/out, tilt up/down, pan up/down/left/right, and rotate the 3D map view. This feature can be activated from the 3D map view toolbar. See how to use it in this video:
Now you can measure distance in 3D map view with considering the z-value. This tool is available in the 3D map view toolbar. It has the same UI as in 2D measurement tool with the same configuration (rubber band color, unit, decimal place, and keeping the base unit). It also has the same behavior (left-click to add a new point, middle-click to delete the last point, and right-click to restart the measurement). Now you can measure the distance between two building’s top or length of a river in a mountain. See the 3D measurement tool in action:
A new kind of rendering style has been added for point layers. It allows you to show the point with QGIS symbol (e.g. marker, SVG, etc) that always face to the user and always has the same size. You can see sample usage:
Update: Input app is now available through Apple App Store:
We are pleased to announce the Beta release of Input on iOS TestFlight. To install the app, simply click on this link from your iOS device: https://testflight.apple.com/join/JO5EIywn. This will open a window to first install TestFlight app. After that, you should be able to install Input on your device.
Input is the first QGIS based app to be released for iOS. Using Input, you can open, view and edit your QGIS projects and data on your iPhone/iPad.
For setting QGIS projects, transferring data/projects and capturing data, you can see the documentation here.
In addition to the great works of the QGIS community in the past to port QGIS to Android devices, we had to do major changes to be able to have Input on iOS. Below are the steps we had taken in the past couple of years, to pave the way:
As a first step, we decided to create a new library in QGIS based on Qt Quick for QGIS. This allowed us to easily create platform independent apps for touch devices.
The library has been built on components ported from QField project. In addition, we have been improving the library and added support for new types of edit form widgets. Details of the QGIS Enhancement Proposal for QGIS Quick can be found here.
By providing static data providers, it was possible to compile code of data providers directly into qgis_core library. This was a major step, as iOS does not support dynamic libraries. Details of this QGIS enhancement can be found here.
We will be delighted to hear your suggestions and feedback, mainly critical ones so that we can improve the application and user experience. After ironing out any issues reported during the Beta testing in the TestFlight, we will publish the app to the App Store.
]]>The event started with the QGIS Developers meeting. We joined the meeting to discuss the upcoming release of QGIS (3.10) and some of the new features and bugs.
We had a workshop on Mesh layer. Participants created animation of the Hurricane Michael in QGIS. Details of the workshop can be found here.
Our first talk was on Input: a QGIS based app for mobile/tablet. During the talk, Saber demonstrated the workflow for setting up a survey project in QGIS and transfer it through the Mergin service to mobile device. Input was then used in a live demo to collect data and synchronise the information back to the server.
You can find the video recording from the talk here
Martin Dobias presented the current state of QGIS 3D and future plans. Martin and Peter Petrik have been mentoring Ismail Sunni for his Google Summer of Code to implement 3D billboard in QGIS. The work was completed and merged to QGIS project recently. Martin demonstrated the new features as the result of the GSoC.
Full presentation can be found here
Mesh layer has been a part of the QGIS for the past year. Peter discussed the improvements made over the recent months. He also gave an overview of the upcoming works to extend the mesh layer to handle 1D and 3D data.
Peter’s talk on mesh layer can be found here
The event was not all about work! We had fun times catching up with friends, partying and running around!
A lot of effort, sleepless nights and sweat went into organising this event. We thank the organisers to make this happen.
Data for this tutorial can be found here.
In this tutorial, we are going to work with a mesh layer in QGIS. Throughout this tutorial you will learn to:
Load a mesh layer
Change symbology
Working with time
Creating time series/cross section plots
Export mesh
Creating animation
For the purpose of this tutorial, we are going to use the ERA5 dataset from ECMWF:
(the data in the first link should contain what you need to proceed with this tutorial).
Before loading the mesh layer in QGIS, we are going to add the world map as a background layer.
In QGIS, from the Browser panel, browse to the downloaded data > Working with mesh layer > vector_data.gpkg and add world_map layer.
You can change the project background layer to blue to have an image similar to the one below:
To add the mesh layer:
In QGIS, from the Browser panel, browse to the downloaded data > Working with mesh layer and add Hurricane Michael data from Copernicus ECMWF.nc as a mesh layer (not raster).
Depending on your QGIS settings, the CRS setting window might appear. Ensure you assign EPSG:4326 to the mesh layer.
To change the mesh layer style:
In QGIS, from the layers panel, select the mesh layer and press F7
The layer styling panel will appear on the right of your QGIS window
Within this panel, you can switch on/off quantities, vectors, style the layer and browse through time.
Below, we are going to switch on the wind data and style it:
On the Style panel, click on Symbology tab
wind to switch on the quantity**
Under color ramp section:
Set Min to 0
Set Max to 20
Interpolation to Linear
Color ramp to Blues (Inverted)
Mode to Equal Interval
Classes to 11
You can change the blending mode to Darken and you will see an image similar to the one below:
To style vector component of the wind data:
metre wind to switch on the vector**
Click on the vector settings section:
Enable the option for Display Vectors on User Grid
X Spacing: 10 px
**Y Spacing: 10 px **
For Arrow Length, select Scaled to Magnitude
If your mesh layer has time dimension, you should be able to browse through time using the slider provided under the settings tab:
Note that the time reference does not always parse correctly. To change the time (if you know the correct format and starting date/time):
Click on the setting in front of time
In the new window:
Use absolute time
Reference date/time: 29.09.2018 04:00:00
To plot time series, you will need to install Crayfish plugin from QGIS plugin repository. Once the plugin is installed:
In QGIS, from the main menu > Mesh > Crayfish > Plot
An empty plot appears at the bottom of your QGIS window. To generate time series for multiple points on the map:
Make sure you have the following settings
Layer: Hurricane Michale data from Copernicus ECMWF
Plot: Time series
Group: [current]
And then click on From Map: Point
Hold Ctrl key on your keyboard and click on the locations you want to plot time series:
A series of graphs will be plotted for each point with matching colours.
You can also create long profile (including aggregated long plot) for a specific time step.
To export mesh to a raster or vector, you can use the processing toolbox:
In QGIS, from the main menu > Processing > Toolbox
Under Crayfish algorithm, double click on Rasterize
A new window will appear:
For Input mesh layer select Hurricane Michale data from Copernicus ECMWF
For Minimum extent click on … and select Use Layer Extent. In the new pop-up, select Hurricane Michale data from Copernicus ECMWF
For Map units per pixel, type: 0.500
For Dataset group, select 2 metre temperature
For Timestep, select 29 days, 6:00:00
Click Run
Similarly, you can export your mesh to points or polygon for each time step.
To export to animation, you can set up a print layout template.
Ensure you have selected 10 m wind quantity from the mesh layer properties panel
Right-click on Hurricane Michael data from Copernicus ECMWF and select Export to animation
Set the correct parameters for start time/end time
Set the values for time, legend, title, etc
Set the filename for the animation file
Mesh calculator is similar to the Raster calculator with the following added functionalities:
Aggregate functions, e.g. calculate maximum values over time
Time filter
Try to calculate the maximum precipitation values for Hurricane Michale data from Copernicus ECMWF dataset
]]>There have been many new features and enhancements in the new release. You can download the latest version from here:
Slider widget has been added as a method to insert values in the forms. The widget configuration follows the QGIS form settings. In addition, the default value settings in QGIS are also recognised in Input forms.
An indicator will appear on the map, while the app is trying to render the content.
Download and upload to the Mergin service have been improved substantially. There is a progress bar indicator for the download/upload. You can also cancel the process.
With the new Google Play Store requirement, we are also now shipping the 64-bit version of the application. Our initial tests showed faster loading and rendering of the maps.
We will be in Bucharest during FOSS4G 2019. Come and talk to us about Input, QGIS, web mapping, etc during the conference!
]]>For uploading/downloading your project, the only method was to use Mergin web interface. But with this plugin, users can interact with their projects and data directly from within QGIS desktop.
You can install the Mergin plugin from QGIS plugin repository. Once it’s installed, you will see Mergin provider from your QGIS browser panel.
To use the plugin, you need to sign into your Mergin account. You can then easily:
For more information about this plugin, see user documentation.
Once you synchronise your changes to Mergin, the data will be available from your Input mobile app. To download the app for your Android device, click below:
]]>Collecting data is often an essential part of a geo-data management workflow. The problems we have tried to resolve with Input:
A mobile app: to collect the data. The app has an intuitive interface. Similar to Google Maps, you don’t need to read a whole bunch of documentation to be able to use the app.
Data synchronisation: a collaborative way of managing the data. We provide Mergin, a central storage, where data administrators can upload their data/projects for their users to change in the field. Changes made to the data through Input will be tracked and can be uploaded to Mergin. Likewise, data administrators can propagate changes to their users by updating the files on Mergin.
You can set up your projects in QGIS. Input is based on QGIS, so it can read all data and services (e.g. Shapefile, Geopackage, TIF, WMS, XYZ tiles, WFS, etc) available in QGIS.
You can customise your forms and identify panel for the layers to be surveyed. Input uses Display tags and Map Themes to better view the data. For more information, see QGIS project configuration of the user documentations.
You can then upload your data and projects to Mergin and share them with your users. Permissions to users to read/write the project can be granted through Mergin.
Input can be installed from Google Play Store. Users will be able to log in to the their Mergin account and download the projects.
Within the app, users can capture geometries (e.g. points, lines and polygons). Capturing of data can be free-hand, or it can be achieved by streaming the GPS locations and generating vertices.
Forms can be easily edited based on the widgets users have set up in the QGIS project. For example, you can add photos, notes, dates, etc to the digitised features.
Input keeps track of changes and you can upload the changes back to Mergin from within the app.
If you’d like to use Input and Mergin within your enterprise environment, please contact us so that we can provide you with a dedicated hosting and no data limit.
]]>Thanks to the response from the QGIS community, we have finalised the packages, ready for your day-to-day use. Below are the list of improvements we have carried out:
The main benefits of the packages:
Unfortunately, we did not raise enough funds to resolve issues with macOS 10.11 and 10.12. Python (and hence plugins) are not supported for those OSes. In addition, it was not possible to polish the code and move the infrastructure to QGIS.org.
Our work was made possible with the help of the supporters of our campaign. Below is the list of individuals and organisations who supported this campaign:
In addition, there were individuals who donated directly to QGIS.org for this work.
The packages are proved to be popular and the QGIS PSC has accepted to take over the infrastructure and publish them eventually as the official QGIS packages for macOS.
Finally got the #equalearthprojection work in @qgis 3.4. Thanks, @lutraconsulting for your compilation. Thanks, @BojanSavric , @mappingbernie , and @MtnMapper for the projection. :) pic.twitter.com/MkQWWJLQUk
— David Garcia (@mapmakerdavid) December 6, 2018
To polish the work and sort out some of the issues, we will need extra funds. We are hoping QGIS macOS users will be able to help the crowdfunding campaign. The target amount is 8,500 € and the campaign will be active until 31 January 2019.
Please have a look at the dedicated page QGIS for macOS for further details and help us spread the word!
Example of those datasets are:
In the past, we introduced Crayfish plugin to handle unstructured grids, usually with temporal and other components. The main issues with Crayfish plugin were:
For more details of the rational behind introducing MDAL, see the QGIS Enhancement Proposal with great suggestions and feedback from the developers.
TL;DR: we needed a new abstraction library to handle the mesh data formats similar to GDAL/OGR dealing with rasters and vectors.
After the feedback from the QGIS community, we have introduced MDAL library earlier this year. It took us a bit of time to put the infrastructure in place. There are already some formats available from MDAL. The library was integrated to QGIS from 3.2. But extensive improvements and new features were added during 3.4.
There are still more formats to support. In QGIS, we will also need to enhance the spatial indexing and performance of the driver in general. For more information visit MDAL website.
Here is a mesh layer you can use to load in QGIS. For more data, you can visit the ECMWF(or Copernicus) or NASA websites.
In QGIS, from the main menu > Layer > Data Source Manager. A new window will appear. From the left panel, click on Mesh and point to the downloaded file.
You should be able to see a figure similar to the one below:
To view quantities within the mesh, you can open the Layer Styling Panel. Click on Styling tab:
We have recently released a new version of Crayfish plugin to work directly with mesh layer in QGIS. Crayfish is now based on Python only and can be installed on all main platforms (including MacOS).
We are planning to add mesh calculator (either as a core QGIS feature or Crayfish/Processing plugin). There will be more formats in the pipeline to support in MDAL too.
If you’d like to add support for your mesh layer to MDAL, you can follow the examples on the github repository. We are always happy to help.
Here are the highlights of the features:
With this new feature, you can embed a 3D scene to the print layout. In addition, there are camera and view setting tools available to adjust the scene from within the map frame. This will allow you to generate high resolution outputs similar to the other map frames.
You can create an animation based on a set of keyframes - camera positions at particular times. QGIS 3D then interpolates the camera positions/rotations in between the keyframes.
To create the keyframes, first set the scene for your map, by rotating, zooming or moving the camera. Then assign a time to the frame. There are several methods for interpolations between keyframes.
A new Identify Tool was introduced to 3D map. Using this tool, you can inspect features from the 3D scene. Thanks to our friends from Faunalia for funding this feature.
If you have a 3D linestring, you can now use the elevation of vertices to display it in 3D map.
There are more control on how you can pan, rotate and zoom the camera within the scene:
In addition, the point towards which the camera is looking is dynamically adjusted based on the terrain. The rotation in scenes where a DEM used for terrain has been enhanced significantly. This is more pronounced with presence of higher hills in the terrain.
There are still some exciting features from the crowdfunding which will be incorporated in QGIS 3.6. You can check the QGIS development or follow us on Twitter.
This blog post and the follow-up ones will discuss a range of topics: data sources, 3D canvas navigation, configuration, working with various types of layers, styling and more!
To work with QGIS 3D, you need data: rasters and vectors. We will use digital terrain model rasters for 3D visualisation purpose. You can download SRTM data from the SRTM Tile Downloader.
For vectors, you can use any point, line and polygon data. There are different methods of creating 3D objects from each data type in the 3D canvas. If you want true 3D data representing buildings, you can download CityGML data from the list of open CityGML datasets.
For the purpose of these blog posts, we will use SRTM data for Mont Blanc and CityGML data for Berlin.
Note: You need to use a projected coordinate reference system in metres (or feet in case you belong to one of these countries) for your data and canvas to be able to use QGIS 3D.
To start with, we are going to add the terrain model for Mont Blanc to the QGIS canvas. Bing Aerial photo (as XYZ tiles layer) was also loaded in QGIS.
To view the 3D canvas, in the main menu select View > New 3D Map View
A floating QGIS panel will appear. You can drag the panel to the bottom part of your canvas to dock it.
To start with, the 3D view shows the same extent and view as seen in the 2D canvas. Also note that there is no dedicated toolbar for navigation in the 3D canvas. You can zoom in/out and pan in the 3D canvas in the same way as in the main 2D canvas:
The following additional options allow you to explore the map in 3D:
To reset the camera view, click button in the 3D canvas panel.
You can use a terrain raster to represent 3D elevation in your canvas. It is expected that such raster layer contains one band where each raster cell represents elevation. To do that, click button to open a new window with 3D view configuration. After selecting your raster layer for Elevation and clicking OK, you should be able to see Mont Blanc in the 3D view:
In the configuration window there are various other options to fine-tune the 3D scene - let’s have a closer look at their meaning. Before diving into the details, it is worth noting that terrain in 3D view is represented by a hierarchy of terrain tiles and as the camera moves closer to the terrain, existing tiles that do not have sufficient detail are replaced by smaller tiles with more details. Each tile has mesh geometry derived from the elevation raster layer and texture created by rendering 2D map for the extent of the tile.
Here is the complete list of the configuration options and their meaning:
Here is a brief summary of what to expect if the campaign will be successful:
The target amount is 12,500 € and the campaign will be active until 16 March 2018.
Please have a look at the dedicated page More QGIS 3D for further details and help us spread the word!
Note: You can download the processed data here. and skip to Viewing data in QGIS.
There are several resources available to obtain climate data. Due to the large volume of data and number of variables, the providers usually offer APIs to interact with dataset repository. But, you can download some datasets directly from their websites.
ECMWF and NASA data portals offering such services.
In this tutorial, we are using NASA portal to download NLDAS Secondary Forcing data for September 2017. Follow the instructions to batch download all the GRIB files using the script provided on their website.
Each grb file downloaded in the previous step is only for one time step containing multiple variables. You can extract certain variable and also merge the files using Climate Data Operators:
cdo mergetime *.grb september.grb
cdo select,name=var61,var33,var34,var11 september.grb september_prec_wind_temp.grb
The first command will merge all the times and generates a single GRIB file containing all the time step. The second command, extracts temperature, precipitation and wind data.
To view the data, you will need Crayfish plugin. Download and install the plugin from the QGIS plugin repository.
Add the GRIB file as a Crayfish layer (under Plugins > Crayfish > Add Crayfish Layer). You should be able to see precipitation, temperature and wind data from the Crayfish panel. Note that the wind data contains vector in addition to grid.
You can use the slider to move the time. Read more on how to use Crayfish plugin.
To view wind data as trace animation, move the slider time to 12.09.2017 07:00 (or 319:00:16.04 if your time is set to relative) and select Vector Options. Set the parameters according to the dialog window below:
You should now be able to see live traces of the wind from your QGIS canvas:
You can plot the time series by clicking on the plot tool from Crayfish panel:
To generate an animation of precipitation and wind, you need to first click on from Crayfish panel and then enable wind vectors and precipitation grid. From Plugins > Crayfish > Export to animation you can generate animation of your time series.
If you are interested in using Crayfish in QGIS 3, help with this crowd-funding.
Crayfish has grown over the past few years and now handles several file formats. There are various functions within the plugin to analyse and process the data too. Instead of porting the plugin directly to QGIS 3, we decided to refactor the code with the possibility of incorporating the renderer directly in QGIS core. We have started a crowd-funding campaign and hoping the software vendors and organisations who are benefiting from the plugin will fund the effort:
https://www.lutraconsulting.co.uk/crowdfunding/qgis-crayfish-3/
We have implemented a new grid calculator for Crayfish layer. Users can now perform various operations on one or multiple layers. The output can be filtered by time or spatially. In addition, we have added logical and aggregating operators. The result is currently saved as a binary XMS file. Read Crayfish wiki page for more details.
Here is a list of enhancements for this release:
This release was funded by Austrian Ministry of Agriculture, Forestry, Environment and Water Management
If you have any problem with Crayfish, please do not email us directly and consider filing a bug here: https://github.com/lutraconsulting/qgis-crayfish-plugin/issues
You can use CityGML or ESRI Multipatch, where the height of buildings are stored within the feature. For the purpose of this example, we are going to view New York buildings using ESRI Multipatch format. You can download the data from here:
http://maps.nyc.gov/download/3dmodel/DA_WISE_Multipatch.zip
After unziping the file, you will have several gdb files. You can use GDAL/OGR to convert gdb to Geopackage (or ESRI Shapefile). We should also convert the geometry type from multipatch to multipolygon. In Microsoft Windows gdal/ogr commands are available from the OSGEO4W command line.
ogr2ogr -f GPKG ny_buildings3d.gpkg DA1_3D_Buildings_Multipatch.gdb -nlt multipolygon
You can append the remaining gdb files to the existing Geopackage:
ogr2ogr -append -f GPKG ny_buildings3d.gpkg DA2_3D_Buildings_Multipatch.gdb -nlt multipolygon
Alternatively, you can write a simple batch script to loop through the files.
Note: we used gdal/ogr from trunk (2.3.0dev).
If you know of any other data sources, please send us an email so we can compile a full list of supported formats.
To be able to use QGIS3D, you need to install the latest version of QGIS using OSGEO4W or other installer for your platform.
Add the Geopackage containing the buildings to your QGIS. In QGIS, from the main menu, click on **View > New 3D Map View **
A new view, similar to your 2D canvas will be added to the bottom of your canvas. To be able to extrude the buildings, we need to enable 3D styling of the building layer.
Ensure your Style panel is enabled (this is usually located on the right hand side of the canvas). Select 3D View tab and tick the box for Enable 3D renderer for building layer.
To navigate in 3D canvas, you can use Shift key + the wheel button on your mouse device.
Continue reading for more detail on how to build QGIS with the latest Qt on Ubuntu …
The default Qt (from official repositories) on (K)Ubuntu 16.04 is too old and does not include the new Qt 3D framework.
We build QGIS with Qt 5.9.1. We are going to install QT to /opt/Qt5.9.1/
and QGIS dependencies built with Qt5.9 to /opt/qt59_libs
,
so make sure you have these folders created and ready to use.
To add Qt 5.9.1, we can use a ppa:
sudo add-apt-repository ppa:beineri/opt-qt591-xenial
sudo apt-get update
sudo apt-get install qt59-meta-full
This will install Qt 5.9.1 side-by-side your current system Qt under /opt folder. You can later remove the package without affecting dependencies in your system.
alternatively you can download QT 5.9.1 installer from http://download.qt.io/official_releases/qt/5.9/5.9.1/qt-opensource-linux-x64-5.9.1.run and install it to the same location.
Another dependency is Qwt. You can download the package and build it with Qt 5.9.1. To download the package, click here: https://sourceforge.net/projects/qwt/files/qwt/6.1.3/ Make a new folder and move the zip file there:
mkdir /tmp/qgis_deps
mv ~/Downloads/qwt-6.1.3.zip /tmp/qgis_deps
cd /tmp/qgis_deps
unzip qwt-6.1.3.zip
cd qwt-6.1.3
We need to define the prefix path. To do that, open qwtconfig.pri
in a text editor and change the prefix path:
nano qwtconfig.pri
change QWT_INSTALL_PREFIX = /opt/qt59_libs/qwt-6.1.3
(more occurrences in the file!)
You can now compile the project:
/opt/qt59/bin/qmake qwt.pro
make -j4
make install
Check if the library has been installed correctly:
ls /opt/qt59_libs/qwt-6.1.3
Use the compressed file from here: https://www.riverbankcomputing.com/software/qscintilla/download
Download and copy to /tmp/qgis_deps
mv ~/Downloads/QScintilla_gpl-2.10.1.tar.gz /tmp/qgis_deps/
cd /tmp/qgis_deps
tar xvzf QScintilla_gpl-2.10.1.tar.gz
cd QScintilla_gpl-2.10.1/Qt4Qt5/
/opt/qt59/bin/qmake qscintilla.pro
make -j4
sudo make install
You should now have the compiled qscintilla in the following path:
ls /opt/qt59/lib/libqscintilla2_qt5.so
First clone (or add as another remote) QGIS fork wonder-sk/QGIS
and change branch to 3d
git clone git@github.com:wonder-sk/QGIS.git
cd QGIS
git checkout 3d
Now you can follow standard instructions on QGIS repo for building the applications: https://raw.githubusercontent.com/qgis/QGIS/master/INSTALL
Once you have created the build directory (after step https://github.com/qgis/QGIS/blob/master/INSTALL#L266) you need to configure the cmake with the following options:
CMAKE_PREFIX_PATH=/opt/qt59/lib/cmake cmake \
-GNinja \
-DCMAKE_BUILD_TYPE=Debug \
-DCMAKE_INSTALL_PREFIX=${HOME}/apps \
-DWITH_3D=TRUE \
-DWITH_QTWEBKIT=FALSE \
-DENABLE_TESTS=FALSE \
-DWITH_QWTPOLAR=FALSE \
-DWITH_BINDINGS=FALSE \
-DQWT_LIBRARY=/opt/qt59_libs/qwt-6.1.3/lib/libqwt.so \
-DQWT_INCLUDE_DIR=/opt/qt59_libs/qwt-6.1.3/include \
-DQSCINTILLA_LIBRARY=/opt/qt59/lib/libqscintilla2_qt5.so \
-QSCINTILLA_INCLUDE_DIR=/opt/qt59/include \
..
The new flag is WITH_3D=TRUE.
In the output, make sure it has found built libraries (NOT Qt 5.7 line)
-- Found Qt version: 5.9
-- Found Qwt: /opt/qt59_libs/qwt-6.1.3/lib/libqwt.so (6.1.3)
-- Found QScintilla2: /opt/Qt5.9.1/5.9.1/gcc_64/lib/libqscintilla2_qt5.so (2.10.1)
Note that if you are using your own compiled version of GDAL, you need to define it using this flag: -DGDAL_CONFIG=/PATH/TO/bin/gdal-config
If all dependencies are detected properly, you should be able to build QGIS using ninja:
ninja
To run QGIS from your build folder:
cd output/bin
./qgis
To verify that you are using the right version of QGIS, you can go to Help > About and check which version of Qt your application has been built against.
Now in QGIS, open 3D Canvas in menu: View->New 3D Map View
. For 3D styling of vector layers, open Layer Styling dock widget and enable 3D Renderer in the newly added tab with 3D cube icon.
Here are the new features in more detail…
We have added support for live trace animation on QGIS canvas:
Please note that export to trace animation (as an avi file) is not supported yet. To change vector style to traces, please refer to the Crayfish manual.
UGRID format for Deltares modelling packages are now supported in Crayfish. In addition to 2D mesh, UGRID also supports 1D mesh. Support for 1D mesh is experimental in this release.
In addition to ASCII FLO-2D files, Crayfish can handle binary HDF output from FLO-2D. Several bugs related to FLO-2D file format were also resolved.
We are pleased to merge changes from Furgo GeoConsulting to support XDMF format. The new format supports on-the-fly loading of the data.
The new and improved support for file formats for HDF5 and UGRID have been kindly sponsored by FLO-2D and Deltares. We developed trace animation for fun.
If you have any problem with Crayfish, please do not email us directly and consider filing a bug here: https://github.com/lutraconsulting/qgis-crayfish-plugin/issues
The WMTS provider had not been benefiting from the the QGIS multi-threaded rendering we did earlier in QGIS 2.4.
In previous versions of QGIS, users had to wait until download of all tiles of a layer has finished in order to view the resulting map. This has now been fixed and the tiles show up in map canvas immediately as they get downloaded, improving the user experience by greatly lowering the time until something is shown.
Moreover, previously downloaded tiles from lower or higher resolutions may be used for the preview functionality in the areas where the tiles with correct resolution have not been downloaded yet.
The screencast here shows fetching and rendering a WMTS layer in QGIS 2.14 (left) and the same layer in QGIS 2.18 (right):
There are a couple of python plugins allowing users to add XYZ tiles (e.g. Bing maps) to QGIS. The plugins only allow certain web services and it is often tricky for supporting the private ones with API keys.
In addition, there are other QGIS applications without python support (e.g. QGIS for Android devices) where they can leverage from having a native support.
Currently, you can only add XYZ tile services from the Browser panel. The video below demonstrates how to add the current precipitation and OpenStreetMap xyz tiles to your QGIS:
WMTS enhancements was sponsored by Land Information New Zealand. Support for XYZ tiles was funded internally.
Serval is available from QGIS plugin repository. Note that you will need to restart QGIS if you upgrade Serval from an earlier version.
Once installed, Serval functions and settings will be available from the toolbar.
Serval supports Undo/Redo for editing values of raster. But it is recommended to make a copy of your raster.
Currently, the following functionalities are available:
We’d like to add support to edit values using spatial and expression selection tools.
For any problems or feedback, please consider to file a ticket here.
We added support for FLO-2D result files.
Weather Research and Forecasting Model (WRF) outputs are now fully supported in Crayfish.
Users can now directly generate vector contours from a Crayfish layer. The export-to-contour feature allows you to select type of contour (e.g. line or area contour) and contour intervals. A very handy option is that the current colour ramp can be also used for your contour intervals.
We have incorporated many algorithms from Crayfish to processing toolbox (Export grid, Export mesh elements, …). To activate the Crayfish module, from the main menu, select Processing > Options and in the new window, under Providers > Crayfish algorithms select the option for Activate.
With the processing toolbox, user can create batch geo-processing algorithms to automate their work. For example, you can create a batch process using this module, to export several Crayfish grids to rasters.
Export to animation using the Processing Toolbox is not supported yet.
We’d like to add support for mesh generation and also mesh calculator. If these are something of you or your organisation interest and would like to contribute financially, feel free to get in touch.
For any problems or feedback, please consider to file a ticket here.
Because we also write QGIS plugins ourselves, we were thinking of how to ease the pain of getting rid of bugs – and so we ended up creating the First Aid plugin, now available in the official QGIS plugin repository.
It is meant to be a Swiss army knife for QGIS plugin developers, a tool that allows easy inspection of any Python code running within QGIS. This is important because it can potentially save developers a lot of their valuable time.
How many times did you end up adding “print” statements into the code to find out what was going wrong in your code? With First Aid plugin this should be no longer necessary.
So let me explain what does it do. First of all, it comes with an improved Python error handler. What this means is that whenever an error occurs in code of a plugin, a window with all the details comes up.
Previously in QGIS you could only find out what was the exception’s type, message and stack trace. First Aid plugin adds source code view, variables view and even an embedded Python console where you can further inspect the state of the plugin at the time of the error. Here is how it looks like in action:
Plugins however do not always come up with errors that can be caught and handled by QGIS. More often plugins simply do not behave as one would expect them to. Here is when people usually resort to using debuggers. There are IDEs like PyDev or PyCharm - or even standalone tools like Winpdb - that allow developers to do remote debugging. Basically they can connect to the Python environment within QGIS and debug the code there. Personally, I have never been a big fan of this approach and found remote debugging cumbersome to set up and use. And trying to debug something on a client’s computer is even a greater challenge.
First Aid fortunately integrates a debugger into QGIS environment. This allows developers to simply open the debugger window, load some Python files, set breakpoints and everything is ready. Once QGIS reaches a line with a breakpoint, the debugger window will be activated and it is possible to step through the code and inspect the variables to understand what is going on in the code.
The great thing is that once the execution of Python code is stopped, it is possible to step into code, step over, step out or run to cursor, just like in any other debugger. It is also possible to run custom scripts from within debugger window – they will be also run in debug mode.
Debugging is active only while the debugger window is still open. While debugging, there is some extra overhead when running any Python code (even for code that you do not intend to debug), so it is better to close the debugger when not needed.
The plugin has already helped us various times to quickly identify problems in plugins. Having said that, please note that the plugin is still quite young and may not work perfectly in all cases. We would be happy to hear your feedback. Any issues or pull requests on GitHub would be greatly appreciated!
The Open Source community in the UK will get together for another great FOSS4G meeting from 14 to 16 of June in Southampton.
QGIS has already established itself as one of the major Open Source GIS applications. To help the community to contribute back to the project, we are running a code sprint.
The code sprint session will not require prior knowledge of coding, so all participants are welcome to join. Make sure you bring your laptop!
There will be several power users and plugin developers to help with the code sprint. There will be 3 topics covered during the code sprint:
If you are interested in helping with the QGIS documentation, we will run a short presentation to introduce you to the workflow.
Participants are required to have installed and configured the following software packages:
For those who are interested in findings bugs, testing existing bugs and leaning up the bug queue, you will need:
If you are familiar with coding (in C++ and python) and would like to fix some bugs, you will need:
##Contributing to qgis2web plugin##
Help improve qgis2web by joining any of these activities:
Details of some specific bugs and features are listed in a dedicated Github issue. If you are not coming to FOSS4GUK 2016, we’d still really value any help you can give. We’ll keep an eye on Github, or try the qgis2web Gitter.
This is a great opportunity to take your first steps in contributing to an open-source project. Come and help - we’ll tell you everything you need to know, and help you learn the skills you don’t yet have. Rest assured that we will be at least as grateful as you for the help you give.
If you are a beginner, @archaeogeek is also running a great workshop at FOSS4GUK 2016 on getting started in open-source: Don’t be afraid to commit
The AddressBase data will be loaded into a PostGIS table for Discovery to query. At this stage we should already have a functioning PostgreSQL / PostGIS installation.
A previous blog post describes how to quickly set up such an environment.
Let’s now create a table for storing the addressbase data. In the example below we’ll create a table called addressbase
in the os_address
schema.
The script below can be executed through pgAdminIII.
To run the script:
When the query has finished you should see Query returned successfully with no result in … seconds. in the Messages panel:
At this point we should be able to locate the new addressbase
table within the os_address
schema:
If you can’t see the schema / table you probably need to refresh the schemas / tables views in pgAdminIII’s Object browser panel by hitting F5.
-- Create the destination schema if required
CREATE SCHEMA IF NOT EXISTS os_address;
-- Create a function which will populate the full_address and geom columns as
-- data are imported
CREATE OR REPLACE FUNCTION create_geom_and_address()
RETURNS trigger AS $$
BEGIN
-- The geometry
-- Set it based on the x_coord and y_coord fields
NEW.geom = ST_SetSRID(ST_MakePoint(NEW.x_coordinate, NEW.y_coordinate), 27700);
-- The full address
-- Initialise it
NEW.full_address = '';
-- Build the full address by only including optional address components if they
-- exist
IF NEW.organisation_name IS NOT NULL AND length(NEW.organisation_name) > 0 THEN
NEW.full_address = NEW.full_address || NEW.organisation_name || ', ';
END IF;
IF NEW.department_name IS NOT NULL AND length(NEW.department_name) > 0 THEN
NEW.full_address = NEW.full_address || NEW.department_name || ', ';
END IF;
IF NEW.po_box_number IS NOT NULL AND length(NEW.po_box_number) > 0 THEN
NEW.full_address = NEW.full_address || NEW.po_box_number || ', ';
END IF;
IF NEW.sub_building_name IS NOT NULL AND length(NEW.sub_building_name) > 0 THEN
NEW.full_address = NEW.full_address || NEW.sub_building_name || ', ';
END IF;
IF NEW.building_name IS NOT NULL AND length(NEW.building_name) > 0 THEN
NEW.full_address = NEW.full_address || NEW.building_name || ', ';
END IF;
IF NEW.building_number IS NOT NULL THEN
NEW.full_address = NEW.full_address || NEW.building_number || ', ';
END IF;
IF NEW.dependent_thoroughfare IS NOT NULL AND length(NEW.dependent_thoroughfare) > 0 THEN
NEW.full_address = NEW.full_address || NEW.dependent_thoroughfare || ', ';
END IF;
IF NEW.thoroughfare IS NOT NULL AND length(NEW.thoroughfare) > 0 THEN
NEW.full_address = NEW.full_address || NEW.thoroughfare || ', ';
END IF;
NEW.full_address = NEW.full_address || NEW.post_town || ', ';
IF NEW.double_dependent_locality IS NOT NULL AND length(NEW.double_dependent_locality) > 0 THEN
NEW.full_address = NEW.full_address || NEW.double_dependent_locality || ', ';
END IF;
IF NEW.dependent_locality IS NOT NULL AND length(NEW.dependent_locality) > 0 THEN
NEW.full_address = NEW.full_address || NEW.dependent_locality || ', ';
END IF;
NEW.full_address = NEW.full_address || NEW.postcode;
RETURN NEW;
END;
$$ LANGUAGE 'plpgsql';
-- Drop any existing addressbase table
DROP TABLE IF EXISTS os_address.addressbase CASCADE;
CREATE TABLE os_address.addressbase
(
-- id will be the primary key, populated automatically
id serial NOT NULL,
uprn bigint NOT NULL,
os_address_toid varchar(24) NOT NULL,
-- os_address_toid bigint NOT NULL,
udprn integer NOT NULL,
organisation_name varchar(60),
department_name varchar(60),
po_box_number varchar(6),
sub_building_name varchar(30),
building_name varchar(50),
building_number smallint,
dependent_thoroughfare varchar(80),
thoroughfare varchar(80),
post_town varchar(30) NOT NULL,
double_dependent_locality varchar(35),
dependent_locality varchar(35),
postcode varchar(8) NOT NULL,
postcode_type char(1) NOT NULL,
x_coordinate numeric(8,2) NOT NULL,
y_coordinate numeric(9,2) NOT NULL,
latitude numeric(9,7) NOT NULL,
longitude numeric(8,7) NOT NULL,
rpc char(1) NOT NULL,
country char(1) NOT NULL,
change_type char(1) NOT NULL,
la_start_date date NOT NULL,
rm_start_date date NOT NULL,
last_update_date date NOT NULL,
class char(1) NOT NULL,
-- the next two fields are populated automatically on insert
full_address text NOT NULL,
geom geometry(Point,27700) NOT NULL,
CONSTRAINT addressbase_pkey PRIMARY KEY (id)
)
WITH (
OIDS=FALSE
);
-- Create a pg_trgm index on the full_address column
-- This will allow super-fast, case-insensitive search on the column
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE INDEX addressbase_full_address_gin_trgm
ON os_address.addressbase
USING gin
("full_address" gin_trgm_ops);
-- Spatial index for the geometry column
CREATE INDEX addressbase_geom_gist
ON os_address.addressbase
USING gist
(geom);
-- trigger to create points and addresses
-- This trigger will be executed on each row inserted, calling the function defined above
CREATE TRIGGER tr_create_geom_and_address BEFORE INSERT
ON os_address.addressbase
FOR EACH ROW
EXECUTE PROCEDURE create_geom_and_address();
The script above has:
full_address will be used to store various address components into a sensible, human readable address. geom will be used to store point geometry based on address eastings/northings.
See the script comments for more information / detail.
At this point we have an empty table ready to accept our AddressBase data. We will now import the data using pgAdminIII. Extract the CSV files for the addresses, you should end up seeing one or more CSV files, for example AddressBase_FULL_2016-03-19_001.csv
In pgAdminIII:
An import dialog should appear. Select the first CSV file and set the settings in the File Options tab as shown here:
Uncheck the id, full_address and geom columns in the Columns tab as shown here:
Click Import. After a few seconds the dialog may report (Not Responding). This is nothing to worry about, be patient.
When the import process completes, close the import dialog and repeat the above steps with any remaining CSV files.
At this stage the data has been imported and the full_address field should contain sensible, human-readable addresses.
With the data loaded in QGIS, we can now configure Discovery to make use of it.
Congratulations! QGIS should now be set up to search your AddressBase data.
We develop several public and private plugins for QGIS users and our clients. During the testing phase, they often come across some bugs. To enable them to report the issues and include the right type of information, we have developed The Report plugin to simplify the process Report plugin.
The Report plugin helps developers get better bug reports from users, by auto-populating the relevant information. It also simplifies the bug reporting process for users.
The Report plugin can automatically extract information from the plugin crash as well as other important information for developers. With just one click you can report the issue to the official plugin issue tracker. No need of searching for a plugin’s tracker on the internet or copy-pasting long tracebacks to your browser.
When/if QGIS issue tracker moves to GitHub, a similar tool can be added to the core to automatically report bugs and crashes.
In this post, we will import the OS Open Names and configure the Discovery plugin to use the data.
In the following sections, we are going to set up a PostGIS database and import the OS Open Names, freely available from here.
To use the data with the Discovery plugin, we need to first set up a PostGIS database and load the data inside the geo-databases.
If you have an existing Postgresql/PostGIS server, you can skip the next section.
For MS Windows users, download and install Postgresql from here (32-bit) or here (64-bit) install the software.
During the installation, select the StackBuilder to install PostGIS, under Spatial Extensions.
If your StackBuilder fails to download PostGIS (in case your proxy server blocks it), you can download and install it manually.
Now that installation is successful, create a new database as osdata. Make sure you add PostGIS extension to your database. You can do that by simply running the following command in the Query editor:
CREATE EXTENSION postgis;
Create osopennames as a new schema under osdata.
In this example, we are focusing on the OS Open Names comes (for Great Britain only) which comes in a zip file containing several CSV files.
If you’d like to use address data for your part of the world, you can visit the OpenAddress website for the global coverage. Note that the CSV files for this dataset come with the vrt files, so you can skip the vrt creation and directly use ogr2ogr.
Once the zip file extracted, there are 2 folders, one containing the header file (DOC) and the other containing the csv files (DATA).
To be able to import all the csv files in PostGIS, we can merge all the files including the header file.
You can move the header file (OS_Open_Names_Header.csv) from the DOC folder. To ensure, the file will appear first during the merge process you can rename it to 1_OS_Open_Names_Header.csv.
You can use Windows command prompt to merge the files. The following command merges all csv files to all_open_names.csv:
copy /b \*.csv all_open_names.csv
There are several methods to import all_open_names.csv in PostGIS:
In the example below, we explore the third option.
To create a virtual vector layer from your csv file, open a text editor, copy and paste the following lines and save it as all_open_names.vrt under DATA folder along with your csv file.
<OGRVRTDataSource>
<OGRVRTLayer name="all_open_names">
<SrcDataSource relativeToVRT="1">all_open_names.csv</SrcDataSource>
<GeometryType>wkbPoint</GeometryType>
<LayerSRS>EPSG:27700</LayerSRS>
<GeometryField encoding="PointFromColumns" x="GEOMETRY_X" y="GEOMETRY_y"/>
</OGRVRTLayer>
</OGRVRTDataSource>
In fact, you can use the virtual vector layer to merge all your csv files and skip the previous section!
You can then use ogr2ogr command from the OSGeo4W shell to import it to your PostGIS:
ogr2ogr -append -a_srs EPSG:27700 -f "PostgreSQL" PG:"host=127.0.0.1 user=postgres dbname=osdata password=postgres active_schema=osopennames" -nln osnames all_open_names.vrt
Note: if you are using the OpenAdress data make sure you assign the right EPSG (4326) in the above command.
First you need to install the plugin from the QGIS plugin repository.
Once the plugin installed, you should have a new toolbar. Click on from the toolbar to open the configuration:
CASE
WHEN "MBR_XMIN" IS NOT NULL
THEN
MBR_XMIN || ',' ||
MBR_YMIN || ',' ||
MBR_XMAX || ',' ||
MBR_YMAX
END
And you should be able to start using the gazetteer plugin after pressing OK.
For performance enhancement and other tips visit this page.
Profile tool plugin deemed not to be suitable for plotting time series. It lacks several features for visualising plots over s specific grid and generally to user interface is not very intuitive.
With the new plot options, user can plot time series at a point from the map or using an existing layer to generate plots. The tool is very flexible and allows user to export the layer to another format. In addition to time series, you can also generate cross section plots. Video below demonstrates the new feature in action:
As you can see, in the above video, more points can be added to the plot. To do so, you need to hold down CTRL key.
We have decided to expand the user base for Crayfish and allow meteorologists and oceanographers to view their temporal unstructured GRIB or NetCDF directly in Crayfish. Examples of those datasets can be found here and here
We also added suppord for HEC RAS 2D and TELEMAC. Our initial benchmark shows that Crayfish is much faster than an other TELEMAC plugins we used. Plus, Crayfish is truly open source!
You can now filter vectors to display only values above/below a certain threshold.
Contour label and values in Crayfish are now similar to raster styling in QGIS. This will allow users to generate better legend directly from Crayfish for print or animation outputs.
For debugging, we have added option to allow users to label mesh and change symbology.
We’d like to thank The Laboratory of Hydraulics, Hydrology and Glaciology (VAW) of ETH Zurich for sponsoring plotting feature.
Having all the tools, some freely available data and a free weekend, I have decided to carry out my own analysis.
For the Digital Elevation Model (DEM), I used the data from JAXA Earth Observation Center (EORC).
The resolution of the DEM is approximately 30 metres. For more detailed analysis, a better dataset (e.g. LIDAR) would be ideal.
I set up a 2-dimensional model with the following assumptions and parameters:
Using Crayfish plugin for QGIS, the results can be animated to see the on-set of the flooding.
Note, that the first hour of the model run was used to initialise the water level within the dam. Hence, for the actual flood travel time, one hour should be deducted from the timing shown towards the bottom right corner of the video.
Below, you can see the flood extent generated based on my calculations and the extent shown here (red line in the main map) for Mosul.
Disclaimer: The calculation carried out for this demo was very approximate. Despite obtaining very similar results for Mosul compared to the original study, further checks and revisions are required.
This blog is more of a note to myself for future reference of how to install third-party modules in QGIS’ Python environment.
Installing third-party modules in Windows is usually quite straight-forward, either download an installer (which will find the Python environment from the registry) or use easy_install from the Python setuptools.
The problem is that QGIS ships with its own Python installation that these methods cannot easily add to.
(Don’t get me wrong - I like the fact that each QGIS instance has its own Python environment as it keeps each version very self-contained).
So here’s the method I successfully used to install SQLAlchemy (which I’ll use here to install the lxml module):
Wait for the QGIS version / instance you want to modify to appear:
from subprocess import call
# Replace the path below to the location of ez_setup.py that
# you just downloaded
call(['python', r'C:\Users\Pete\Downloads\ez_setup.py'])
# This will install *setuptools* which is a package manager
# The previous command should return 0 on success
# Replace lxml with the package you wish to install
call(['easy_install', 'lxml'])
# Again this will return 0 on success
The new module should now be available, we can test this (again on the Python console in QGIS):
import lxml
That’s it!
Funding for the work was generously raised with the help of the community in 2015 and the feature landed in QGIS 2.14
Check out the Trace Digitising in QGIS for detailed instructions on how to use the tool.
The trace tool:
The following video demonstrates how to use the tracing tool:
This video shows how you can make use of tracing while reshaping a polygon:
The project was kindly sponsored by the QGIS community. Special thanks to all those who contributed towards the development of this tool:
Trainees will learn how to
QGIS’ Print Composer allows users to create professional maps for printing. It supports legends, frames, logos and all the other features you’d expect to see in a print quality map.
In this example, we’ll use some of the most common features required to generate a map for printing.
We’ll first load the electricity usage data created in another tutorial, style it using a simple colour ramp, then create a new map composer complete with title and legend.
The styling of a vector layer can be performed using a single style, or a varying style, based on the value of one or more attributes. We are going to first load the London energy usage layer prepared in an another tutorial by joining electricity usage with the LSOAs vector layer.
Start a new QGIS project
Add the MSOA_GL_ElecUsage.shp layer under extracted
downloaded folder
The vector layer should now look like the figure below.
Energy consumption in London
Next, we are going to add a shapefile of London Boroughs to identify areas of high and low usage.
Add London_boroughs.shp from extracted downloaded
folder
Open layer properties and under Style:
Set the style of the polygons to "No Brush" with a
"Dash Line".
Your map now should look like the figure below:
Energy consumption in London
In QGIS, you can apply multiple styles to a single layer. Using this method, we do not need to load the same layer multiple times. In this case, we need two styles for London boroughs layer:
Without labels (as created above); to be used as an overview map
With labels (next section); for the main map
We can save the current style (without label):
Right-click on London_boroughs layer from the layer
tree and select Styles > Add
For the new style, type 'no label' and click OK
We can save the new styles (with labels) for the layer:
Right-click on London_boroughs layer from the layer
tree and select Styles > Add
For the new style, type 'with label' and click OK
Changes in the next section (adding label) will be saved in with label style.
We’ll now add labels to each of the boroughs. The name of each borough is stored as the value of the NAME attribute.
Click on the London_boroughs layer in the ToC
Form the main menu, select Layer > Labelling
This window will allow us the set an attribute for label and also change font, size, rotation, buffer, etc.
Setting up labels on the London boroughs layer
Enable the Label this layer with option
Set Field with labels to NAME
Under Text section, set the font to 7 pt
Under Buffer section:
Enable Draw text buffer
Set the Size to 0.5 mm
Click OK
You should now be able to see a label for each borough.
London boroughs with labels
In QGIS, you can create presets for layer visibility and styles. In this example we are going to create the following presets:
Map of LSOAs with the energy usage and borough boundaries
A separate preset for the London boroughs (without labels) which will be used as an “overview” for the print composer
To create the first preset:
Ensure both layers are visible
Right-click on London_boroughs and select
Style > with label
On top of the legend tree in QGIS, you can see the icon for Manage Layer Visibility
Click on the Manage Layer Visibility icon
From the drop-down menu, select Add preset...
For the Name of the new preset insert Energy map
Click OK
To ensure the preset has been correctly set, you can change the visibility of the layer, or change the style, and select the preset from the Manage Layer Visibility menu.
To create the second preset:
Hide All Layers from Manage Layer Visibility or simply
press Ctrl+Shift+H
Turn on the visibility for London_boroughs
Right-click on London_boroughs and select
Style > no label
On top of the legend tree in QGIS, click on
Manage Layer Visibility
From the drop-down menu, select Add preset...
For the Name of the new preset insert overview map
London boroughs
Now you should be able to switch between those two presets, and later use them as print composer frames.
The map is almost ready to be printed. Before printing, we’ll need to add a border, logos, legends, scale, etc.
There are quite a few steps involved in configuring the print composer so it’s recommended to regularly save your progress by saving the QGIS project.
From the main menu, select File > New Print Composer
A new window will appear for Composer title
Type Energy usage in London
The print composer title window
The print composer will appear. See Figure.
The print composer window
The print composer is split into two sections. Section 1 shows the map to be outputted and section 2 shows the settings for selected items in section 1.
The map we are going to produce is an A4 landscape PNG file.
In section 2, under Composition, for Presets, select
A4(210 x 297 mm)
Now we need to add frames for the map extent and legend sections. You can add a rectangular shape by clicking on from the main toolbar.
From the main toolbar, add a new rectangle
Click and drag a shape in section 1
Now let’s change the position and size of the rectangle:
Ensure the rectangle shape is selected in section 1
Click on Item properties in section 2
Under Shape, select Position and size
Set the option as shown in Figure
Shape options
Add second, third and fourth rectangles with the
following information
Rectangle 2:
X: 270
Y: 145
Width: 43.5
Height: 60
Reference Point: Centre
Rectangle 3:
X: 270
Y: 95
Width: 43.5
Height: 60
Reference Point: Centre
Rectangle 4:
X: 270
Y: 105
Width: 43.5
Height: 200
Reference Point: Centre
Lock the position of each window by right-clicking
on each of them
Section 1 of the print composer should look like this:
The print composer with placeholders layout
Let’s add the logos.
From the main menu, select Layout > Add Image
Click within the middle rectangle on the right
An empty box will appear
In the right-hand panel (section 2), under the
Item properties tab,
click on Main properties button
For Path, click on ... and browse to the folder where
your images are located.
Select an image
You should now see something similar to the image below.
Logos added to the composer
Now we’ll add some text in the rectangle below the logos:
From the main menu, select Layout > Add Label
Click somewhere within the lower right-hand box
A new text box will appear, reading QGIS.
From the right-hand panel, click on Item properties
In Main properties, change the text to refer to the
source of the data and copyright notices.
Copyright notices
Now we’ll add the map from the QGIS canvas.
From the main menu, select Layout > Add map
Drag a box in the main window
A map similar to the image below should now appear in the print composer.
To finalise the main map window, a few more changes will need to be made.
Adding map to the print composer
From the right panel, click on the Item properties tab
Under the Main properties, set the following values:
Tick the box for Lock layers for map item see
image below
Under Extents:
Tick the box for Controlled by atlas
Adding a preset for map view
To add a scalebar:
From the main menu, select Layout > Add Scalebar
Adding a scalebar to the map
A scalebar with default settings should appear on the map. See Figure [PrintComposerAddScalebar]. To change the scalebar:
Select the scalebar from the map
Click on Item Properties from the left-hand panel
Under Segments
Set left 0 and right 2
Set Size to 1000
Set Height to 2 mm
Next, we’ll add a legend:
From the main menu, select Layout > Add Legend
Click somewhere within the upper right-hand box
QGIS will automatically add a legend for all the layers loaded in the canvas. We can edit the legend’s settings and remove the legend entries that we do not want to show. Manual edits of the legend text can also be performed to make it more readable.
To be able to edit the legend items, you first need to untick the box for Auto update. Use icons under the Legend items to edit or change the order of the items.
The default legend
From the right-hand panel, click on the Item properties
Under Legend items:
Select London_boroughs and then click on the red
minus sign
Select MSOA_GL_ElecUsage and click on the edit
sign, a new window will appear.
Change the content of Item text to Electricity
Usage
Click on the first value 2576.2400 – 3000.0000
and select edit sign to change the range to
2000 – 3000
Repeat for the rest of the values to round them
and remove any unwanted zeros.
Editing legend values
Map with legend
To add an overview:
Add a new map (Map 1) to the place-holder between
the map legend and the logos
For Map 1, from the Item properties,
Under Main properties
Select Lock layers for map item
From the visibility preset, select overview map
Under Overviews tab
Click on the plus sign,
Overview 1 should be added to the overview list
Ensure Draw "Overview 1" overview is selected
For Map frame, select Map 0
Setting the preset view for the overview map
The map is now almost ready. You can export it as PNG or a PDF. Or you can keep reading for even more excitement!
The completed map
The map is now ready to be printed or outputted as an image. But, we are going to use this layout as a template and auto-generate energy consumption for each borough.
To set the coverage (this is already set, but make sure the option is enabled):
While Map 0 (the main map) item is selected, click on
Item properties from the right-hand panel
Select the option for Controlled by atlas generation
You can change the Margin around feature if you like, but we can keep at 10%. Further settings are also required:
Click on Atlas generation tab from the right-hand panel
Tick the box for Generate an atlas
Under Configuration, for Coverage layer select
London_boroughs
Under Output, for Output filename expression type:
'energy_'|| "NAME"
The expression for output will set the name of each output file. In this example we are going to have a name with ‘energy_‘ combined with the value of NAME column (what do you think this value is?)
To generate the maps:
In the print composer, from the main menu, select
Atlas > Export Atlas as Images...
Select a folder to output maps and press Choose
The process may take a while but you will eventually have 33 maps of the energy consumption for each borough. Note the overview for the maps and also the naming of the output files. See Figure beloq as an example of the maps created.
Final energy map for Kingston upon Thames generated by the Atlas
In this release, we have revamped the labelling engine and made it more flexible in-line with the rest of vector styling.
In this release, we have:
In previous versions of QGIS, users can select a field value or use an expression as labels for a vector layer. In QGIS 2.12, users can additionally define rules to label vector layers. Rule-based labelling works in the similar way as “Style”. A list of rules will be defined by users and they will be applied from top-to-bottom.
To achieve the same effect in the earlier versions of QGIS, users should add a complex expression.The example below shows the expression used in earlier versions of QGIS:
CASE WHEN length( "htmlname" ) > 13 AND strpos("htmlname",' ') > 6 THEN replace("htmlname",' ',' ') WHEN length( "htmlname" ) > 20 AND "htmlname" LIKE '%Golf Course' THEN regexp_replace("htmlname",'Golf Course',' Golf Course') WHEN length( "htmlname" ) > 20 AND "htmlname" LIKE '%Nature Reserve' THEN regexp_replace("htmlname",'Nature Reserve',' Nature Reserve') WHEN length( "htmlname" ) > 20 AND "htmlname" LIKE '%Church Of England%' THEN regexp_replace("htmlname",'Church Of England',' Church Of England ') WHEN length( "htmlname" ) > 13 AND "htmlname" LIKE '% Of The %' THEN regexp_replace("htmlname",'Of The','Of The ') WHEN length( "htmlname" ) > 13 AND "htmlname" LIKE '% of %' AND "fontcolour" <> 2 AND "fontcolour" <> 4 THEN regexp_replace("htmlname",' of ',' of ') WHEN "htmlname" LIKE '%/%' THEN regexp_replace("htmlname",'/','/ ') WHEN length( "htmlname" ) > 30 THEN replace("htmlname",' ',' ') ELSE "htmlname" END
As you can see, without rule-based labelling users have to define each case and the labelling text. Additionally, with the cumbersome task of defining each case, users are limited to using specific labelling formats (e.g. font color, font size, visibility range, etc). With the new labelling engine, users can define their labels similarly to styles - in fact they can copy the rules from styling tabs and use it within the new labelling section!
In order to facilitate addition of rule-based labelling, some internal changes were made to the QGIS labelling engine interface. The labelling is now driven by the new class QgsLabelingEngineV2
which may have several label providers associated with it.
The label providers are objects derived from QgsAbstractLabelProvider
and they are responsible
for:
providing “label features” that define properties of each label and geometry of the feature they represent; and
drawing of labels at the positions determined by the engine.
Currently there are label provider implementations for diagrams, simple labelling and rule-based labelling.
The existing labelling engine (QgsPalLabeling
class) is now built on top of the new labelling engine and works as a wrapper for it so that existing code that uses QgsPalLabeling
still works.
As of now, the API for the new labelling engine is not considered as complete and therefore not available in Python. The idea is to make it easier to use, more polished and better prepared for the future use cases. It will be likely finished during the 2.14 release cycle and some changes may need to be postponed to QGIS 3.0 where backwards incompatible API changes will be allowed (as of now, QGIS 3.0 is being intensively discussed on QGIS developer mailing list).
This rule-based labeling has been funded by Tuscany Region (Italy). Special thanks also to the QGIS developers for their help with bug fixing.
The ground works were done in Crayfish 2.0 for this feature. You can now generate animation from contours and vectors and export them as AVI. There are two methods of exporting to animation: basic and using QGIS print template (qpt).
With the basic option, you can define a title, a legend and a progress clock. Alternatively, for a smarter solution, you can set up a print composer with the benefit of all its rich features. The composer template (qpt) can be used as a frame layout for exporting your animation.
Below is an example of a multi-frame print composer template used to generate animation from a Crayfish layer.
Crayfish 2.1 now supports SWW file format generated by AnuGA</li>.
With this option, users can define a grid and Crayfish will interpolate values and displays results on the custom grid. Images below show vectors on the outputted mesh and user-defined grid.
With the ever-growing features in Crayfish, we have decided to dedicate a page on how to use Crayfish in QGIS. From the manual page, users can download a sample data and try the Crayfish features in QGIS.
We’d like to thank Maroondah City Council for sponsoring some of the great features in this release.
Nathan wrote a blog post about the feature and how to write a python with arguments. But in QGIS 2.8, the function editor does not correctly support functions without arguments.
In the example below, we are going to calculate proportion of area for each SAC (Special Areas of Conservation) in Great Britain to the total area of the layer.
Add GB_SAC_20130918_Dissolved.shp layer to QGIS, right-click on the layer and open attribute table. Make the layer editable and click on the Field calculator. We are now going to create a new column (propArea) and populate proportionate area of each SAC to the total area of the layer.
Under Function Editor tab, click on New file and type area for the name and save the file. For the content, copy and paste the following lines:
"""
A custom function to calculate total area of the GIS layer.
This function has no arguments.
"""
from qgis.core import *
from qgis.gui import *
from qgis.utils import iface
@qgsfunction(args='auto', group='Custom')
def total_area(x, feature, parent):
return sum( f.geometry().area() for f in iface.activeLayer().getFeatures() )
Click on Run Script to add total_area function to your Custom function list.
Now, click on Expression tab and type:
$area / total_area(0)
As you can see, we have passed 0 as an argument. If you click OK, your QGIS will freeze! As there are many features in the layer, the expression, calculates total area for each row.
Lets make the script a bit more elegant. Firstly, we need to add caching, so that area will be calculated only once and cached for the rest of operation. Secondly, we can make the script a bit more generic, so that we can use it to get the area of other loaded layers in QGIS:
from qgis.core import *
from qgis.gui import *
from qgis.utils import iface
cache = {}
def _layer_area(layer):
""" Internal method """
if layer.name() not in cache:
area = sum( f.geometry().area() for f in layer.getFeatures() )
cache[layer.name()] = area
return cache[layer.name()]
@qgsfunction(args='auto', group='Custom')
def total_area(layer_name, feature, parent):
for layer in iface.mapCanvas().layers():
if layer.name() == layer_name:
return _layer_area(layer)
return 0
Now, click on Expression tab and type:
$area / total_area('GB_SAC_20130918_Dissolved')
This time, it should be quicker!
In the new version of Crayfish a time slider allows users to quickly browse through time. A drop-down menu allows the selection of an exact time.
In previous versions of Crayfish it was only possible to load contours and vectors from the same dataset. For example, it was not possible to show velocity vectors on top of depth contours. With the new Crayfish you can "unlock" the legend and choose different vectors or contours to be displayed. The video below demonstrates this in action.
Some datasets contain a special time-step outside the outputted time range. For example, Maximums and Minimums are stored at time 99999 in TUFLOW modelling package. Within the layer tree, additional time-steps items will now be shown if they exist within the dataset.
We have added support for the XMDF file format. In addition, Hydro_AS 2D users should be able to open their files in the latest Crayfish.
We've refactored lots of code in the Crayfish library which makes it much easier to add support for further file formats and additional functionality. The Crayfish library now comes with a new Python module that allows easy manipulation with the mesh and results data - either in your custom scripts or within the QGIS Python console. For example, printing the coordinates of the nodes of a mesh together with their elevation takes just few lines of code:
import crayfish
m = crayfish.Mesh("/data/my_mesh.2dm")
o = m.dataset(0).output(0) # bed elevation data
for index, node in enumerate(m.nodes()):
print "Node XYZ: ", node.x, node.y, o.value(index)
If you use Profile tool plugin in QGIS, you can create a profile from the Crayfish layer and browse through the time. The profile gets updated as you change the output time.
If you have some feedback on our changes, suggestions for new functionality, or come across a bug, feel free to file a ticket on the issues page of the Crayfish github repository.
We’d like to thank Maroondah City Council for sponsoring some of the great features in this release.
First select the layers and groups, you’d like to manage visibility presets for. Now click at the top of the Layers panel and select Add preset … A new window will appear. Enter a name for the preset and click OK. Next time you click you should see the saved presets. The presets also save the visibility of categories in each layer.
In the video below, we create two presets. One for a specific site location and another zoomed to the outline of Great Britain which will be used as an overview in the print composer.
The print composer now allows multiple map windows, each locked to a different layer visibility preset. This means a single map composer can show map windows with different layer arrangements.
In the video below, one preset is used for the main map and the other for an overview:
We have also been busy beta-testing our new illuvis flood risk communication service. Crayfish 1.3 integrates the illuvis client, allowing users to interact easily with this flood risk mapping service.
Read on for a look at some of the new features in more detail...
You can now turn off values below or above certain thresholds. This feature is very useful when you work with rainfall-runoff models and want to turn off values below e.g. 2 cm.
You can now set your colour ramps once, save them and apply them to other layers. Export to Geo-TIFF ------------------
You can now easily export outputs from Crayfish to Geo-TIFF (.tif) files. Simply select the Quantity and Output time in the Crayfish viewer then right-click on the Crayfish layer in the QGIS Layers panel and select Export to grid...
For those users who have been using illuvis, you can now upload crayfish layers directly to the flood map publishing web service rather than first having to convert results to Geo-TIFF. If you would like to become an illuvis beta-tester, please contact us.
Support for BASEMENT -------------------- Users can now view outputs from the BASEMENT modelling package.Binary package for Ubuntu -------------------------
We love Linux and have finally managed to prepare crayfish binaries for Ubuntu 14.04. If you are using other distros, the compilation should be straight-forward using guide in the README file.
Problems -------- If you have some feedback or come across a bug, feel free to file a ticket on the issues page of the Crayfish github repository. Sponsors -------- We'd like to thank The Laboratory of Hydraulics, Hydrology and Glaciology (VAW) of ETH Zurich for sponsoring some of the great features in this release.The tool is OS Translator II - it makes use of the excellent GDAL library and is available now in the official QGIS Plugins repository.
This blog post talks about some simple benchmarks we've carried out.
If you are interested in using this tool and not familiar with Postgresql/PostGIS, you can sign up to one of our support packages and we will be able to set you up and running within a couple of hours!
National load times were as follows:
Installing PostgreSQL, PostGIS and QGIS took less than 10 minutes.
1 This is the most time-consuming test which filled the SSD on the first attempt. Importing to a tablespace on the main HDD completed after 20.3 hours but showed the import of tile 1592959-TR0585-5c3268.gz to have failed with this error. Until this issue is resolved the tile would need to be loaded and de-duplicated manually (e.g. using ogr2ogr to import and a SQL query to de-duplicate) to complete the dataset. De-duplication removes duplicate features caused by the chunking / supply process.
We were curious as to how OS Translator II load times compared with other open loading methods so we did some basic tests using the "SU" tile of MasterMap Topography and ITN datasets and compared it with the popular Loader scripts. The results looked like this:
Please note that OS Translator II had an unfair advantage in these tests as it automatically takes advantage of multiple-CPU cores whereas Loader presently does not.
We used the following hardware and software configuration:
2 Operating system and source gml.gz files located on the SSD and default PostgreSQL tablespace stored on secondary 2TB HDD.
The following changes were made to the default PostgreSQL configuration:
3 maintenance_work_mem was set to 1024MB for the national load of MasterMap Topography layer only.
Turning fsync off is dangerous and can lead to data loss in the event of an unexpected power outage. Always switch fsync back on after loading and never use this option on a database containing critical data.
Lutra’s very own QGIS core developer Martin Dobiaš has been working hard with others on bringing you new stuff. So, what exciting new features are provided by this jauntily-angled new release? There are plenty of eagerly-anticipated additions - both big new features, and under-the-hood improvements - as well as bugfixes.
Have you ever found yourself duplicating layers just so you could display the same data but with additional styles? Those days are now over.
You can now define multiple styles for a single layer, and easily switch between them in the layer view or layer properties dialog.
Just so our webmapping colleagues don’t feel left out this new behaviour is extended to the WMS server too. The available multiple styles are advertised in GetCapabilities and can be used in other requests, such as GetMap.
And of course it all works in the map composer too - which means more maps with the same layers displayed in different styles.
Work carried out in cooperation with Gis3W for Regione Toscana.
Snapping is made simpler with the addition of new modes for snapping, so you can now:
There’s also less confusion with snapping tolerances being to map units or layer units, so those anticipated 1 metre tolerances don’t become 1 degree tolerances without you knowing.
Snapping is not only more straightforward - it’s now much faster too thanks to using an index of geometries built when first needed. Because everybody loves faster, right?
For our developer colleagues there is an easy to use API which - amongst other things - adds fast point in polygon queries.
Funded by Ville de Vevey, SITNyon and QGIS Usergroup Switzerland.
More info here.
QGIS has had a simplify tool for a while, but now it’s been significantly improved with lots of tasty new ingredients.
Simplify tool tolerances used to be different for every feature - but not any more, and users can specify exact tolerances - which can be in map units or layer units.
Also new is on-the-fly reprojection support, tolerance settings are now persistent between sessions, dragging a rectangle instead of just clicking to simplify multiple features, support for multi-part features, and statistics about reduction of the number of vertices - because if you’re simplifying, it’s nice to know by how much! Oh yes, and faster too!
Work carried out in cooperation with Kartoza.
The display of rules in the legend for rule-based rendering has continued its evolution in features from QGIS 2.4:
QGIS 2.4: Good - a flat list of rules, but you can’t see the nesting.
QGIS 2.6: Better - shown as a pseudo-tree with checkboxes.
QGIS 2.8: Best - great for power-users who use complex styling rules to give their maps that extra zing.
Funded by SIGE.
QGIS is also the first Long-Term Release (LTR) version - starting with 2.8 every third release is a long-term-release - maintained until the next long-term-release occurs.
Even if you’re a new-feature-junkie, you’ll appreciate the commitment by the project to both innovation and stability, and this of course underlines the burgeoning success of QGIS in enterprise environments.
So, start your engines, and get ready to download!
While you’re still reeling from finding out about some of the new goodies that QGIS 2.8 brings (and check the official QGIS changelog for more!), I’ll remind you that all this cool stuff somehow needed to be paid for.
Thanks go to the organisations and individuals that sponsor or contribute to the QGIS project, and those that fund development of specific features. They make the software better for everyone, and you or your organisation could become one of them!
We at Lutra Consulting are Bronze level sponsors of the QGIS project, and use our development experience to contribute code and develop plugins for QGIS.
We tackle this in several ways, including direct approaches from clients to fund custom development, and crowdfunding campaigns. So, if you’re interested in some functionality that isn’t yet there - get in touch!
As we have seen earlier, a layer tree is a usual hierarchical data structure composed from nodes of two types - layers and groups. They do not provide any GUI functionality as they live in the QGIS core library. In order to visualize a layer tree, we will use Model/View approach from Qt framework. Readers not familiar with those concepts are recommended to read the Qt Model/View overview first.
There is QgsLayerTreeModel
class (derived from QAbstractItemModel
) which as you may have guessed provides
a model to access a layer tree. Instance of this class may be used in any QTreeView
class, however
it is recommended to use it together with QgsLayerTreeView
because of the extra convenience functionality
offerred by the custom view class.
Here is an example how to create another view of current project’s layer tree (try that in QGIS Python console):
from qgis.gui import *
root = QgsProject.instance().layerTreeRoot()
model = QgsLayerTreeModel(root)
view = QgsLayerTreeView()
view.setModel(model)
view.show()
Any changes that happen to the layer tree structure are automatically monitored by the model/view classes. After running the example above, try changing a layer’s name or add/remove/reorder some layers - all those actions will be immediately visible in all views.
As you can see, the layer tree view is just one way how to visualize the underlying layer tree - in the future it may be possible to have different ways to show the layer tree, for example using Qt’s QML framework (with all sorts of animated transitions known from mobile apps).
Plugin developers can access the layer tree view in the main window of QGIS through the following interface call:
view = iface.layerTreeView()
The model is meant to be flexible and it is possible to use it in various contexts. For example, by default the model also provides legend for the layers in the tree. This however may not be wanted in some cases. Similarly, sometimes the layer tree should be read-only while in other context it is desired that the user can reorder layers or change their names. These preferences can be passed to the model with flags:
model = QgsLayerTreeModel(root)
model.setFlag(QgsLayerTreeModel.AllowNodeReorder)
model.setFlag(QgsLayerTreeModel.AllowNodeChangeVisibility)
The setFlag()
method has optional second parameter “enabled” (True
by default). Flags can be also
set all at once with setFlags()
method which expects a combination of flags joined by binary OR operator.
There are also flags()
and testFlag()
methods to query the current flags.
The QgsLayerTreeModel
class also provides routines for conversion between QgsLayerTreeNode
instances
and corresponding QModelIndex
objects used by Qt Model/View framework - index2node()
and node2index()
may come handy especially when working with views.
There is also some functionality related to legend display - it has been greatly extended in QGIS 2.6 release and we will try to cover that in a future blog post.
As mentioned earlier, it is possible to use any QTreeView
instance in combination with QgsLayerTreeModel
to show the layer tree, but it is highly recommended to use QgsLayerTreeView
class (a subclass of QTreeView
)
because of the additional functionality it provides (and more may be added in the future):
Easier handling of selection. Normally one needs to work with selection through view’s selection model.
The QgsLayerTreeView
class adds higher level methods that operate with QgsLayerTreeNode
objects
like currentNode()
or with QgsMapLayer
objects
like currentLayer()
.
def onChange(layer):
QMessageBox.information(None, "Change", "Current Layer: "+str(layer))
# connect to the signal
view.currentLayerChanged.connect(onChange)
# change selection to the top-most layer (onChange will be also called)
view.setCurrentLayer( iface.mapCanvas().layers()[0] )
Display of context menu. It is possible to assign a menu provider to the view
(subclass of QgsLayerTreeViewMenuProvider
) - its createContextMenu()
implementation will return
a QMenu
object with custom actions whenever user right-clicks in the view. Additionally, there is a factory
class QgsLayerTreeViewDefaultActions
that can create commonly use actions for use in the menu,
such as “Add Group”, “Remove” or “Zoom to Layer”. The following example shows how to create a provider
with one action that shows the current layer’s extent:
class MyMenuProvider(QgsLayerTreeViewMenuProvider):
def __init__(self, view):
QgsLayerTreeViewMenuProvider.__init__(self)
self.view = view
def createContextMenu(self):
if not self.view.currentLayer():
return None
m = QMenu()
m.addAction("Show Extent", self.showExtent)
return m
def showExtent(self):
r = self.view.currentLayer().extent()
QMessageBox.information(None, "Extent", r.toString())
provider = MyMenuProvider(view)
view.setMenuProvider(provider)
The layer tree classes and map canvas class are separate components that are not dependent on each other.
This is a good thing and a great step forward, because until QGIS 2.2 there was big internal monolithic
QgsLegend
view class that handled everything and it was directly connected to map canvas and various other
components, making it impossible to reuse it elsewhere. With the new layer tree API this has been solved,
now it is possible use map canvas without an associated layer tree view or vice versa - to use a
layer tree view without map canvas. It is even possible to get creative and use one layer tree with several
map canvas instances at once.
Layer tree and map canvas can be connected via QgsLayerTreeMapCanvasBridge
class. It listens to signals
from the given layer tree hierarchy and updates canvas accordingly. Let’s see how we could create a new
map canvas that would show the same layers as the main canvas does:
canvas = QgsMapCanvas()
root = QgsProject.instance().layerTreeRoot()
bridge = QgsLayerTreeMapCanvasBridge(root, canvas)
canvas.zoomToFullExtent()
canvas.show()
That’s it! We have tied the new canvas with project’s layer tree. So any actions you do in the layer tree view (for example, add or remove layers, enable or disable layers) are automatically passed to the new canvas. And of course this does not work just with project’s layer tree - you could use any custom layer tree in your layer tree model/view or canvas.
The bridge class has advanced functionality worth mentioning. By default the ordering of layers in canvas is according to the order in the layer tree (with first layer in the tree being at the top), though there are API methods to override the default order.
For convenience, the bridge by default also configures some settings of the canvas (this can be disabled if necessary):
I hope you have enjoyed the three blog posts introducing QGIS layer tree API. They should cover everything you need to know in order to start using the new functionality. In a future post we will have a look at the new legend API that nicely complements the layer tree API - stay tuned!
In this tutorial you will develop your first QGIS plugin - a Map Tool for selecting the closest feature within multiple loaded vector layers. Knowledge of Python is recommended but not required.
Before we get started let’s look at where we’re going.
We will develop a plugin that implements a new QGIS Map Tool. The Identify Features and Pan Map tools are both examples of QGIS Map Tools. A Map Tool is a tool which performs an action when used with the map canvas.
We will create a new Select Nearest Feature Map Tool which will sit in the plugins toolbar.
Our Select Nearest Feature Map Tool will allow the user to select the feature nearest a mouse click on the canvas. For example, clicking here:
would select the following polygon:
Before getting started:
The QGIS Plugin Builder plugin was used to create a base plugin which we’ll modify to fit our requirements.
This base plugin can be found in the zip file mentioned above under code/01__Empty Plugin/NearestFeature
code/01__Empty Plugin contains a batch file install.bat that can be used to copy the plugin into your QGIS plugins folder, making it available to QGIS.
Let’s now load and run this simple base plugin in QGIS.
A new action should now be visible in the plugins toolbar which opens the following dialog:
When activated, our plugin currently shows a simple dialog (functionality provided by the Plugin Builder plugin. We’re going to adapt it to instead activate a Map Tool.
A basic Map Tool is included within the zip file mentioned above. It can be found in nearest_feature_map_tool.py in the Additional Files folder.
gray
) below link to relevant parts of the API docs. Those links will open in a dedicated browser tab.nearest_feature_tool.py defines a new NearestFeatureMapTool
class (line 28) which inherits (is based on) QgsMapTool
, the QGIS Map Tool class. Its __init__()
method expects to be passed a reference to a QgsMapCanvas
. This canvas reference is passed to the constructor of the underlying QgsMapTool
class on line 32 and then stored on line 33 for later use. The QGIS API documentation describes the functionality made available by QgsMapTool
.
On line 34 we define a simple, different-looking cursor (a QCursor
based on Qt.CrossCursor
) later used to indicate that the Map Tool is active.
Our class definition features a method called activate()
. Notice the API documentation for QgsMapTool
already defines a method with the same name. Any methods defined as virtual methods in the API documentation can be overwritten or redefined as they have been within this file. Here we have overwritten the default implementation of activate()
.
The activate()
method is called when the tool is activated. The new cursor based on Qt.CrossCursor
defined above is set with a call to QgsMapCanvas.setCursor()
.
For the moment, when activated, our Map Tool would simply change the cursor style - that’s all.
Great - next we’ll get our plugin to use the new Map Tool.
In this section we will modify the plugin to make use of our new Map Tool.
We need to first import the NearestFeatureMapTool
class before we can use it.
os.path
is imported:from nearest_feature_map_tool import NearestFeatureMapTool
Next we will create a new instance of the NearestFeatureMapTool
class and store a reference to it in self.nearestFeatureMapTool
.
initGui()
method just before the call to self.add_action()
taking care to ensure the indentation is correct:# Create a new NearestFeatureMapTool and keep reference
self.nearestFeatureMapTool = NearestFeatureMapTool(self.iface.mapCanvas())
Notice that a reference to the map canvas has been passed when creating the new NearestFeatureMapTool instance.
The run()
method is called when our plugin is called by the user. It’s currently used to show the dialog we saw previously. Let’s overwrite its current implementation with the following:
# Simply activate our tool
self.iface.mapCanvas().setMapTool(self.nearestFeatureMapTool)
The QGIS map canvas (QgsMapCanvas
class) provides the setMapTool()
method for setting map tools. This method takes a reference to the new map tool, in this case a reference to a NearestFeatureMapTool
.
To ensure that we leave things in a clean state when the plugin is unloaded (or reloaded) we should also ensure the Map Tool is unset when the plugin’s unload()
method is called.
unload()
method:# Unset the map tool in case it's set
self.iface.mapCanvas().unsetMapTool(self.nearestFeatureMapTool)
Now let’s see the new map tool in action.
When passing the mouse over the map canvas the cursor should now be shown as a simple cursor resembling a plus sign. Congratulations - the Map Tool is being activated.
When you use the Identify Features Map Tool you’ll notice that its button remains depressed when the tool is in use. The button for our map tool does not yet act in this way. Let’s fix that.
The action (QAction
) associated with our plugin is defined in the initGui()
method with a call to self.add_action()
.
self.add_action()
actually returns a reference to the new action that’s been added. We’ll make use of this behaviour to make the action / button associated with our Map Tool toggleable (checkable).
add_action()
as follows:action = self.add_action(
icon_path,
text=self.tr(u'Select nearest feature.'),
callback=self.run,
parent=self.iface.mainWindow())
action.setCheckable(True)
We now use the reference to the new action to make it checkable.
The QgsMapTool
class has a setAction()
method which can be used to associate a QAction
with the Map Tool. This allows the Map Tool to handle making the associated button look pressed.
initGui()
:self.nearestFeatureMapTool.setAction(action)
The button should now remain pressed, indicating that the tool is in use.
The Nearest Feature button should now appear unpressed.
The QgsMapTool
class has a number of methods for handling user events such as mouse clicks and movement. We will override the canvasReleaseEvent()
method to implement the search for the closest feature. canvasReleaseEvent()
is called whenever the user clicks on the map canvas and is passed a QMouseEvent
as an argument.
We will now write some functionality which:
def canvasReleaseEvent(self, mouseEvent):
"""
Each time the mouse is clicked on the map canvas, perform
the following tasks:
Loop through all visible vector layers and for each:
Ensure no features are selected
Determine the distance of the closes feature in the layer to the mouse click
Keep track of the layer id and id of the closest feature
Select the id of the closes feature
"""
layerData = []
for layer in self.canvas.layers():
if layer.type() != QgsMapLayer.VectorLayer:
# Ignore this layer as it's not a vector
continue
if layer.featureCount() == 0:
# There are no features - skip
continue
layer.removeSelection()
The layers()
method of QgsMapCanvas
(stored earlier in self.canvas
) returns a list of QgsMapLayer
. These are references to all visible layers and could represent vector layers, raster layers or even plugin layers.
We use the type()
and featureCount()
methods to skip non-vector layers and empty vector layers.
Finally we use the layer’s removeSelection()
method to clear any existing selection. layerData
is a list that we’ll use in a moment.
Our plugin now clears the selection in all visible vector layers.
We now need access to each feature and its geometry to determine its distance from the mouse click.
canvasReleaseEvent()
within the loop over layers:# Determine the location of the click in real-world coords
layerPoint = self.toLayerCoordinates( layer, mouseEvent.pos() )
shortestDistance = float("inf")
closestFeatureId = -1
# Loop through all features in the layer
for f in layer.getFeatures():
dist = f.geometry().distance( QgsGeometry.fromPoint( layerPoint) )
if dist < shortestDistance:
shortestDistance = dist
closestFeatureId = f.id()
info = (layer, closestFeatureId, shortestDistance)
layerData.append(info)
The mouse click event (a QMouseEvent
) is stored in mouseEvent
. Its pos()
method returns a QPoint
describing the position of the mouse click relative to the map canvas (x and y pixel coordinates). To calculate its distance to each feature we’ll need to first convert the mouse click position into real world (layer) coordinates. This can be done using a call to QgsMapTool.toLayerCoordinates()
which automatically deals with on-the-fly projection and returns a QPoint
in layer coordinates.
The features of a vector layer can be accessed using the layer’s getFeatures()
method which returns (by default) a list of all QgsFeature
in the layer that we can iterate over using a simple loop.
With access to features we can easily gain access to geometry using QgsFeature.geometry()
. The QgsGeometry
class has a number of spatial relationship methods including distance()
which returns the distance to a second QgsGeometry
passed as an argument.
In the code above we loop over all features, keeping track of the feature id of the closest feature using QgsFeature.id()
. The shortest distance and closest feature id are stored in shortestDistance
and closestFeature
. When we are finished iterating through all the features in this layer, we store a note of the layer, its closest feature id and associated distance into layerData
.
Note that we convert layerPoint
(a QgsPoint
) into a QgsGeometry
so we can use it directly in spatial relationship operations such as QgsGeometry.distance()
.
We’re almost done. At this point layerData
is a list of tuples, one for each vector layer containing:
Now we can simply sort layerData
by distance (its 3rd column) and make a selection based on the layer and feature in the first row of layerData
.
canvasReleaseEvent()
outside the outer for loop:if not len(layerData) > 0:
# Looks like no vector layers were found - do nothing
return
# Sort the layer information by shortest distance
layerData.sort( key=lambda element: element[2] )
# Select the closest feature
layerWithClosestFeature, closestFeatureId, shortestDistance = layerData[0]
layerWithClosestFeature.select( closestFeatureId )
The code above returns early if no workable vector layers were found. It sorts layerData
(the list of tuples) by the 3rd element (the distance).
The code then calls QgsVectorLayer.select()
to select the closest feature by its feature id.
The plugin should now be finished.
Within this tutorial we’ve worked briefly with the following parts of the QGIS API:
Hopefully this has been a useful tutorial. Please feel free to contact us with any specific questions.
Starting with an empty project, first we will get access to the layer tree and create some memory layers for testing:
root = QgsProject.instance().layerTreeRoot()
layer1 = QgsVectorLayer("Point", "Layer 1", "memory")
layer2 = QgsVectorLayer("Polygon", "Layer 2", "memory")
Now let’s add some layers to the project’s layer tree. There are two ways of doing that:
Explicit addition. This is done with the addLayer()
or insertLayer()
call of the QgsLayerTreeGroup
class.
The former appends to the group node, while the latter allows you to specify the index at which the layer
should be added.
# step 1: add the layer to the registry, False indicates not to add to the layer tree
QgsMapLayerRegistry.instance().addMapLayer(layer1, False)
# step 2: append layer to the root group node
node_layer1 = root.addLayer(layer1)
Implicit addition. The project’s layer tree is connected to the layer registry and listens
for the addition and removal of layers. When a layer is added to the registry, it will
be automatically added to the layer tree. It is therefore enough to simply add a layer to the map
layer registry (leaving the second argument of addMapLayer() with its default value of True
):
QgsMapLayerRegistry.instance().addMapLayer(layer1)
This behaviour is facilitated by the QgsLayerTreeRegistryBridge
class. By default it inserts layers
at the first position of the root node. The insertion point for new layers can be changed -
within the QGIS application the insertion point is updated whenever the current selection in the layer tree
view changes.
Groups can be added using the addGroup()
or insertGroup()
calls of the QgsLayerTreeGroup
class:
node_group1 = root.insertGroup(0, "Group 1")
# add another sub-group to group1
node_subgroup1 = node_group1.addGroup("Sub-group 1")
There are also the general addChildNode()
, insertChildNode()
and insertChildNodes()
calls
that allow the addition of existing nodes:
QgsMapLayerRegistry.instance().addMapLayer(layer2, False)
node_layer2 = QgsLayerTreeLayer(layer2)
root.insertChildNode(0, node_layer2)
node_group2 = QgsLayerTreeGroup("Group 2")
root.addChildNode(node_group2)
Nodes that are being added must not have any parent yet (i.e. being part of some layer tree). On the other hand, the nodes that get inserted may already have children, so it is possible to create a whole sub-tree and then add it in one operation to the project’s layer tree.
The removal of nodes from a layer tree is always done from the parent group node. For example, nodes displayed
as top-level items need to be removed from the root node. There are several ways of removing them.
The most general form is to use the removeChildren()
method that takes two arguments: the index of the first
child node to be removed and how many child nodes to remove. Removal of a group node will also remove all
of its children.
There are several convenience methods for removal:
root.removeChildNode(node_group2)
root.removeLayer(layer1)
There is one more way to remove layers from the project’s layer tree:
QgsMapLayerRegistry.instance().addMapLayer(layer1)
The project’s layer tree is notified when any map layers are being removed from the map layer registry
and the layer nodes representing affected layers will be automatically removed from the layer tree.
This is handled by the QgsLayerTreeRegistryBridge
class mentioned earlier.
When managing the layer tree, it is often necessary to move some nodes to a different position - within the same parent node or to a different parent node (group). Moving a node is done in three steps: 1. clone the existing node, 2. add the cloned node to the desired place in layer tree, 3. remove the original node. The following code assumes that the existing node we move is a child of the root node:
cloned_group1 = node_group1.clone()
root.insertChildNode(0, cloned_group1)
root.removeChildNode(node_group1)
There are a number of operations one can do with nodes:
Rename. Both group and layer nodes can be renamed. For layer nodes this will modify the name directly inside the map layers.
node_group1.setName("Group X")
node_layer2.setLayerName("Layer X")
Change visibility. This is actually a check state (checked or unchecked, for group nodes also partially checked) that is associated with the node and normally related to the visibility of layers and groups in the map canvas. In the GUI, the layer tree view is capable of showing a check box for changing the state.
print node_group1.isVisible()
node_group1.setVisible(Qt.Checked)
node_layer2.setVisible(Qt.Unchecked)
Change expanded state. The boolean expanded/collapsed state refers to how the node should be shown in layer tree view in the GUI - whether its children should be shown or hidden.
print node_group1.isExpanded()
node_group1.setExpanded(False)
Change custom properties. Each node may have some associated custom properties. The properties are key-value pairs, keys being strings, values being of variant type (QVariant). They can be used by other components of QGIS or plugins to store additional data. Custom properties are preserved when a layer tree is saved and loaded.
Use the customProperties()
call to get a list of keys of custom properties, then the customProperty()
method for getting the value of a particular key. To modify properties, there is
a setCustomProperty()
method which sets a key-value pair and a removeCustomProperty()
method to remove a pair.
node_group1.setCustomProperty("test_key", "test_value")
print node_group1.customProperties()
print node_group1.customProperty("test_key")
node_group1.removeCustomProperty("test_key")
print node_group1.customProperties()
There are various signals emitted by nodes which may be used by client code to follow changes to the layer tree. Signals from children are automatically propagated to their parent node, so it is enough to connect to the root node to listen for changes from any level of the tree.
The addition of new nodes always emits a pair of signals - before and after the actual addition. Signals pass information about which node is the parent node and the range of child indices:
willAddChildren(node, indexFrom, indexTo)
addedChildren(node, indexFrom, indexTo)
In order to access the newly added nodes, it is necessary to use the addedChildren
signal.
The following code sample illustrates how to connect to signals emitted from the layer tree. When the last line is executed, two lines from the newly defined methods should be printed to the console:
def onWillAddChildren(node, indexFrom, indexTo):
print "WILL ADD", node, indexFrom, indexTo
def onAddedChildren(node, indexFrom, indexTo):
print "ADDED", node, indexFrom, indexTo
root.willAddChildren.connect(onWillAddChildren)
root.addedChildren.connect(onAddedChildren)
g = root.addGroup("My Group")
Removal of nodes is handled in a very similar manner to the addition - there is also a pair of signals:
willRemoveChildren(node, indexFrom, indexTo)
removedChildren(node, indexFrom, indexTo)
This time in order to access nodes being removed it is necessary to connect to the willRemoveChildren
signal.
There are a few more signals that allow monitoring of internal changes to nodes:
visibilityChanged(node, state)
customPropertyChanged(node, key)
expandedChanged(node, expanded)
We have covered how to make changes to a layer tree structure and how to listen for changes possibly made by other pieces of code. In a future post we look at GUI components for displaying and modifying the layer tree and the connection between map canvas and layer tree.
Layers in the legend widget can be organised into groups. This grouping allows easier manipulation of layers. For example it is possible to toggle the visibility of all layers at once. In addition to layers, groups can also contain other groups, effectively creating a hierarchy of groups and layers. From now on, we will refer to this hierarchy as the layer tree.
The legend widget might look like this:
Until QGIS 2.4, there has been only limited support for interacting with the legend widget using the QGIS API.
There is a QgsLegendInterface
class (which can be obtained with iface.legendInterface()
) available for
plugins. The legend interface has emerged in an ad-hoc way, leading to various issues when used in plugins.
It is also worth noting that third-party applications based on QGIS have no access to the legend
interface.
The layer tree API has been introduced in QGIS 2.4 to overcome these existing problems and add even more flexibility to the way the layer tree can be queried or modified.
The layer tree is a classical tree structure built of nodes. There are currently two types of nodes: group nodes and layer nodes. Group nodes can contain other (child) nodes, while layer nodes are ‘leaves’ of the tree, without any child nodes. The layer tree for the legend widget shown in the picture above looks like this:
The green nodes are group nodes (QgsLayerTreeGroup
class) and the yellow nodes
are layer nodes (QgsLayerTreeLayer
class).
The legend widget also displays items using symbols, making it look like a real legend. The symbology is not part of the layer tree and will be discussed in an upcoming post.
To start working with the layer tree, we first need a reference to its root node. The project’s layer tree can be accessed easily:
root = QgsProject.instance().layerTreeRoot()
The root node is a group node - its children are shown as top-level items in the legend widget.
print root
print root.children()
This returns a list of the children of a node. The list includes only direct children - children of sub-groups need to be queried directly from those sub-groups.
Now let’s try to access the first child node in the tree and do a little bit of introspection:
child0 = root.children()[0]
print child0
print type(child0)
print isinstance(child0, QgsLayerTreeLayer)
print child0.parent()
With the children()
and parent()
methods it is possible to traverse the layer tree. A node is the root node
of a tree if it has no parent:
print root.parent()
The following example shows how to list top-level items of the layer tree. For group nodes it will print the group name, for layer nodes it will print the layer name and ID.
for child in root.children():
if isinstance(child, QgsLayerTreeGroup):
print "- group: " + child.name()
elif isinstance(child, QgsLayerTreeLayer):
print "- layer: " child.layerName() + " ID: " + child.layerId()
In order to traverse the full layer tree, it would be necessary to recursively call the same code for sub-groups.
There are some helper routines for common tasks like finding nodes representing layers in the tree. These take into account all descendants, not just top-level nodes.
ids = root.findLayerIds()
print ids
print root.findLayers()
print root.findLayer(ids[0])
It is assumed that a single layer is represented in a layer tree only once. There may however be temporary situations when a layer is represented by more than one node, for example when moving nodes (a new node is created before the old one is removed shortly after).
Similarly it is possible to search for group nodes by name:
print root.findGroup("POI")
Group names are not necessarily unique - if there are multiple groups with the same name, the first encountered during tree traversal will be returned.
In this blog post we have shown how to query the project’s layer tree. Upcoming blog entries will focus on modifying the layer tree and interacting with other parts of QGIS.
You can download the latest version of QGIS from here. On that page, you can find the appropriate QGIS installation package for your operating system.
If you are a MS Windows user, you have 2 options: the standalone installer or the OSGeo4W installer, each of which has its own strengths:
Once you finish installing QGIS, you can find its icon on your desktop and/or Start menu. Launch QGIS and wait for the application to start. If you're a MS Windows user, QGIS might take some time to start up for the first time but subsequent loads will be much faster.
QGIS features a number of tool-bars. You can move them around by clicking and dragging the vertical or horizontal dotted bars separating the tool-bars (for example, the bar to the left of the help tool's icon in the image above).
It is recommended to set the Coordinate Reference System (CRS) for your project before adding any data. CRS or SRS is a coordinate-based local, regional or global system used to locate geographical entities. Many CRSs are available and each is suited to a particular area of the globe. There is a comprehensive list of CRS codes available here. In this example, we will set the CRS to match the British National Grid coordinate reference system. The easiest way to search for a specific CRS is using its unique EPSG code. The EPSG code for British National Grid is 27700.
To set the CRS for your projects in QGIS, from the main menu, select Settings > Options. A new window will appear. Select CRS tab.
In the Search section, set Authority to "EPSG" and Search for to "ID". Type 27700 into the search box and click Find. Highlight the correct row in Coordinate Reference System section and click OK.
GIS data is usually in either raster or vector format. QGIS supports a large number of GIS data formats through the GDAL/OGR library and other plugins. In the example below we will download and add some OS OpenData™ raster and vector datasets into QGIS.
Ordnance Survey released a number of OS OpenData raster datasets to the public under a very permissive license. You can download the data from here.
For this particular example, follow this link and browse to OS Street View®. Select SX from the map. Move towards the bottom of the page and click next. Fill in the required information and click continue. You should receive an email with a link to download osstvw_sx.zip (note: it is a 383.9 MB file - you can order a DVD instead if you have a slow internet connection). Once the download has finished, unzip the file. You should now have a new folder called OS Street View SX which contains 2 subfolders and a readme file.
Browse to Street View SX > data > georeferencing files > tfw. Select all the TFW files and move them to the Street View SX > data > sx folder. The TFW files contain georeferencing information describing the location of each TIF file.
In QGIS, from the main menu, select Layer > Add Raster Layer... and browse to the Street View SX > data > sx folder. Select sx99nw.tif, sx99ne.tif, sx99sw.tif and sx99se.tif. Click Open. You should now be able to see the raster tiles in the QGIS canvas and the Layers panel at the left side of the screen.
Raster files do not always contain CRS information. We can easily organise the layers and assign the correct CRS (EPSG:27700) with the help of groups. Create a new group by right-clicking on the blank space (not on the sx99 layers) in the Layers panel and selecting Add group. Set the name of the group to OS Street View. Next, move the sx99 layers into the new group by selecting them all and dragging them into the OS Street View group. Once all the sx99 layers are inside the OS Street View group, right-click on the group and select Set group CRS. A new dialog, similar to that seen in the Setting the CRS chapter will appear. Assign the British National Grid CRS (EPSG:27700) and click OK.
Next, we'll bring some vector data into QGIS. Go to the OS OpenData Supply page and browse to OS VectorMap™ District (there are two OS VectorMap datasets on this page, for this example, ensure you select the vector version and not the raster version) and select SX from the map. Scroll to the bottom of the page and click next. Fill in the required information and click continue. Download vmdvec_sx.zip from the link you'll receive by email. Extract the contents of the ZIP file.
In QGIS, from the main menu, select Layer > Add Vector Layer... and browse to the OS VectorMap District (Vector) SX > data > SX. Select SX_Airport.shp, SX_RailwayTrack.shp and SX_Road.shp. Click Open. Click Open again.
To change the style of a vector layer, right-click on the layer in the Layers panel and select Properties. In the Style tab of the Layer Properties dialog, you can define exactly how the layer should look.
Internet based mapping can also be brought into QGIS, for example, a plugin exists that allows OpenStreetMap, Google, Bing and Yahoo maps to be added to QGIS.
Web map services (WMS) are another source of mapping data. In the next we'll add a WMS layer provided by British Geological Survey to our map. Please read the BGS WMS Terms of use. Another example of WMS is Ordnance Survey's OS OnDemand service. If you have OS OnDemand license, you can follow the instructions on Ordnance Survey's website (sorry, link no longer works) to add other useful WMS layers.
To add the BGS WMS, select Layer > Add WMS Layer... from the main menu. The Add Layer(s) from a Server dialog will appear. Click New.
Set the name to BGS and set the URL to the following:
http://maps.bgs.ac.uk/ArcGIS/services/BGS_Detailed_Geology/MapServer/WMS...
Click OK, and in the Add Layer(s) from a Server dialog, click Connect.
Select all the layers and click Add. Close the Add Layer(s) from a Server dialog. The BGS layer should become visible as you zoom-in to a scale of 1:50000 or closer. Alternatively, you can manually set the Scale in the status bar to 1:50000 and the BGS layer will appear.
QGIS is written in a manner that makes it possible for anyone it extend its functionality through the use of plugins. As a result, there are many plugins available to the user, making QGIS highly modular and flexible.
Core plugins are plugins that are shipped with QGIS and can be optionally enabled through the QGIS Plugin Manager. To access the QGIS Plugin Manager, from the main menu, Select Plugins > Manage Plugins...
Select the All tab and type OpenLayers Plugin into the Filter box. Select the plugin and click Install plugin. You should now be able to add OpenStreetMap, Google, Bing and Yahoo maps to your canvas using the Web > OpenLayers plugin menu.
For further help using QGIS, you can always check the manual, user or developer mailing lists or QGIS forum.
If you'd like to master QGIS as quickly as possible, why not attend one of our training courses.
To install OpenLayers plugin, from the main menu, click Plugins > Manage and Install Plugins.... A new window will appear.
You should be able to search and install the OpenLayers plugin within your list.
Although you can have different version of QGIS installed under Windows, it's recommended to uninstall old versions before attempting to install new versions.
On rare occasions, some anti-virus software has been known to remove the qgis.exe and python.exe files from the installation folder. If you're having problems running the QGIS shortcut, please ensure those 2 files exist in the installation folder.
If QGIS cannot find your Python folder, you may need to set the PYTHONPATH environment variable to your QGIS folder (\QGIS\apps\python).
To be able to access WMS, WFS and 3rd party plugins, you'll need to have internet access. In the event that you're behind a proxy server, you can enter the proxy server details in Settings > Options > Network:
This was not a big issue at the time as MapInfo had a really nice autotrace feature which made these tasks a doddle. However, when I moved to QGIS I could not find such a feature.
The image above shows AutoTrace in action. Please see the AutoTrace project page for full documentation and a list of co-funders.
The idea to develop a plugin to do MapInfo-style tracing was born at the QGIS developer conference (AKA Hackfest) in Lisbon in early 2011. It was there that Paolo Cavalini pointed me towards the traceDigitize plugin already developed by Cédric Möri. traceDigitize already supported tracing but required the user to move the cursor along the edge of the feature being traced. I still had my heart set on achieving that MapInfo-style tracing.
Rather than re-inventing the wheel, traceDigitize was modified to add MapInfo-style tracing as an optional component. This was my development focus for the HF (and the flight back). At this stage the basic functionality was in-place but the plugin had many bugs and was not reliable to use. Unfortunately the plugin was mothballed until recently.
It was at the first QGIS UK User Group that I met Matt Travis (GIS Officer at Dartmoor NPA) and we got talking about AutoTrace. It transpired that there were a number of local government organisations interested in getting this auto-tracing functionality into QGIS. We soon had commitment from a number of organisations (including and organised by Kevin Williams at Neath Port Talbot Council) to fund the work required to release a stable version of the tool.
There's talk of extending the functionality of AutoTrace to also implement some of he more complex tracing techniques seen in ArcGIS. If this is something you'd be interested in then please let us know.
Many thanks to all those who have supported us in this project.
The adoption of QGIS is accelerating globally in public, private and academic organisations. Developers from around the globe will meet in person to discuss, plan and implement various aspects of the QGIS project.
All university staff and students are welcome to join the Hackfest and meet the people behind QGIS! See the wiki page for more information.
Ongoing news and updates will be posted to this page during the hackfest.
Live streaming can be viewed here
Find us here
</p>
As part of their Ravensbourne Raised Defences project, Halcrow were required to update flood defence crest level and standard of protection data for defence assets along the rivers Ravensbourne and Quaggy, south-east London based on the results from extensive ISIS modelling.
The updates were originally carried out as a manual GIS task, requiring GIS technicians to edit NFCDD geometry and attributes manually. The editing was prone to human error and was a complex and time-consuming process.
Lutra Consulting were commissioned to develop NFCDD Updater - a bespoke extension (plugin) to the free and open-source GIS platform QGIS. NFCDD Updater automates the task of updating NFCDD while allowing the user to maintain a high level of control over the process.
NFCDD Updater performs the following tasks:
NFCDD Updater was designed to be simple to use and to allow user-interaction where required, for example, prompting the user to visually select a single defence line to update when more than one candidate defence line has been selected for updating automatically.
NFCDD Updater performed very well in terms of reducing manual GIS work required to complete the project and allowed approximately 30% of the project budget to be saved. With minor modifications, NFCDD updater can be updated to automate manual GIS tasks on other Halcrow projects.
For more information on NFCDD Updater or to find out more about the development of bespoke QGIS plugins, contact Peter Wells.