Interfacing with WhiteboxTools using Python scripting

• Using the whitebox_tools.py script
• Handling tool output
• Additional functions in whitebox_tools.py
• An example WhiteboxTools Python project

WhiteboxTools provides an elegant and powerful Python-based interface that allows for easy workflow automation of common geoprocessing tasks. This interface with the WhiteboxTools stand-alone executable is contained within the whitebox_tools.py script. Let's look at this Python programmer interface in this tutorial.

By combining the WhiteboxTools library with a high-level scripting language, such as Python, users are capable of creating powerful stand-alone geospatial applications and workflow automation scripts. In fact, WhiteboxTools functionality can be called from many different programming languages. However, given the prevalent use of the Python language in the geospatial fields, the library is distributed with several resources specifically aimed at Python scripting. This section focuses on how Python programming can be used to interact with the WhiteboxTools library.

Using the whitebox_tools.py script

Interacting with WhiteboxTools from Python scripts is easy. To begin, each script must start by importing the WhiteboxTools class, contained with the whitebox_tools.py script; a new WhiteboxTools object can then be created:

from WBT.whitebox_tools import WhiteboxTools

wbt = WhiteboxTools()

Depending on the relative location of the WhiteboxTools directory and the script file that you are importing to, the import statement may need to be altered slightly. In the above script, it is assumed that the folder containing the WhiteboxTools files (including the whitebox_tools Python script) is named WBT (Line 1) and that the calling script is located in the parent directory of WBT. See An Example WhiteboxTools Python Project for more details on project set-up. The use of wbt to designate the WhiteboxTools object variable in the above script (Line 3) is just the convention used in this manual and other project resources. In fact, any variable name can be used for this purpose.

The WhiteboxTools class expects to find the WhiteboxTools executable file (whitebox_tools.exe on Windows and whitebox_tools on other platforms) within the same directory (WBT) as the whitebox_tools.py script. If the binary file is located in a separate directory, you will need to set the executable directory as follows:

wbt.set_whitebox_dir('/local/path/to/whitebox/binary/')
# Or alternatively...
wbt.exe_path = '/local/path/to/whitebox/binary/'

Individual tools can be called using the convenience methods provided in the WhiteboxTools class:

# This line performs a 5 x 5 mean filter on 'inFile.tif':
wbt.mean_filter('/file/path/inFile.tif', '/file/path/outFile.tif', 5, 5)

Each tool has a cooresponding convenience method. The listing of tools in this manual includes information about each tool's Python convienience method, including default parameter values. Parameters with default values may be optionally left off of function calls. In addition to the convenience methods, tools can be called using the run_tool() method, specifying the tool name and a list of tool arguments. Each of the tool-specific convenience methods collect their parameters into a properly formated list and then ultimately call the run_tools() method. Notice that while internally whitebox_tools.exe uses CamelCase (e.g. MeanFilter) to denote tool names, the Python interface of whitebox_tools.py uses snake_case (e.g. mean_filter), according to Python style conventions. The only exceptions are tools with names that clash with Python keywords (e.g. And(), Not(), and Or()).

The return value can be used to check for errors during operation:

if wbt.ruggedness_index('/path/DEM.tif', '/path/ruggedness.tif') != 0:
   # Non-zero returns indicate an error.
   print('ERROR running ruggedness_index')

If, like me, your data files tend to be burried deeply in layers of sub-directories, specifying complete file names as input parameters can be tedius. In this case, the best option is setting the working directory before calling tools:

from whitebox_tools import WhiteboxTools

wbt = WhiteboxTools()
wbt.work_dir = "/path/to/data/" # Sets the Whitebox working directory

# Because the working directory has been set, file arguments can be
# specified simply using file names, without paths.
wbt.d_inf_flow_accumulation("DEM.tif", "output.tif", log=True)

An advanced text editor, such as VS Code or Atom, can provide hints and autocompletion for available tool convenience methods and their parameters, including default values (see below).

Sometimes it can be useful to print a complete list of available tools:

print(wbt.list_tools()) # List all tools in WhiteboxTools

The list_tools() method also takes an optional keywords list to search for tools:

# Lists tools with 'lidar' or 'LAS' in tool name or description.
print(wbt.list_tools(['lidar', 'LAS']))

To retrieve more detailed information for a specific tool, use the tool_help() method:

print(wbt.tool_help("elev_percentile"))

tool_help() prints tool details including a description, tool parameters (and their flags), and example usage at the command line prompt. The above statement prints this report:

 ElevPercentile
 Description:
 Calculates the elevation percentile raster from a DEM.
 Toolbox: Geomorphometric Analysis
 Parameters:

 Flag               Description
 -----------------  -----------
 -i, --input, --dem Input raster DEM file.
 -o, --output       Output raster file.
 --filterx          Size of the filter kernel in the x-direction.
 --filtery          Size of the filter kernel in the y-direction.
 --sig_digits       Number of significant digits.

 Example usage:
 >>./whitebox_tools -r=ElevPercentile -v --wd="/path/to/data/" --dem=DEM.tif
 >>-o=output.tif --filterx=25

hillshade(
dem,
output,
azimuth=315.0,
altitude=30.0,
zfactor=1.0,
callback=default_callback)

wbt.hillshade(
"DEM.tif",
"hillshade.tif",
altitude=20.0)

Handling tool output

Tools will frequently print text to the standard output during their execution, including warnings, progress updates and other notifications. Sometimes, when users run many tools in complex workflows and in batch mode, these output messages can be undesirable. Most tools will have their outputs suppressed by setting the verbose mode to False as follows:

wbt.set_verbose_mode(False)
# Or, alternatively...
wbt.verbose = False

Alternatively, it may be helpful to capture the text output of a tool for custom processing. This is achieved by specifying a custom callback function to the tool's convenience function:

# This callback function suppresses printing progress updates,
# which always use the '%' character. The callback function
# approach is flexible and allows for any level of complex
# interaction with tool outputs.
def my_callback(value):
   if not "%" in value:
       print(value)

wbt.slope('DEM.tif', 'slope_raster.tif', callback=my_callback)

Every convienience function takes an optional callback as the last parameter. The default callback simply prints tool outputs to the standard output without any additional processing. Callback functions can serve as a means of cancelling operations:

def my_callback(value):
   if user_selected_cancel_btn: # Assumes a 'Cancel' button on a GUI
       print('Cancelling operation...')
       wbt.cancel_op = True
   else:
       print(value)

wbt.breach_depressions('DEM.tif', 'DEM_breached.tif', callback=my_callback)

Additional functions in whitebox_tools.py

The whitebox_tools.py script provides several other functions for interacting with the WhiteboxTools library, including:

# Print the WhiteboxTools help...a listing of available commands
print(wbt.help())

# Print the WhiteboxTools license
print(wbt.license())

# Print the WhiteboxTools version
print("Version information: {}".format(wbt.version()))

# Get the toolbox associated with a tool
tb = wbt.toolbox('lidar_info')

# Retrieve a JSON object of a tool's parameters.
tp = tool_parameters('raster_histogram')

# Opens a browser and navigates to a tool's source code in the
# WhiteboxTools GitHub repository
wbt.view_code('watershed')

For a working example of how to call functions and run tools from Python, see the whitebox_example.py Python script, which is distributed with the WhiteboxTools library.

An example WhiteboxTools Python project

In this section, we will create a Python project that utilizes the WhiteboxTools library to interpolate a LiDAR point-cloud, to process the resulting digital elevation model (DEM) to make it suitable for hydrological applications, and to perform a simple flow-accumulation operation. I suggest using an advanced coding text editor, such as Visual Studio Code or Atom, for this tutorial, but Python code can be written using any basic text editor.

Begin by creating a dedicated project directory called FlowAccumExample and copy WhiteboxTools binary file (i.e. the compressed file downloaded from here) into this folder. Using the decompression software on your computer, decompress (i.e. an operation sometimes called unzipping) the file into the newly created FlowAccumExample directory. You will find the compressed file contains a folder with contents similar to the following:

The folder contains a number of files, including the WhiteboxTools executable file, the whitebox_tools.py python script, the WhiteboxTools Runner (wb_runner.py), and the user manual. It is likely that the folder has a name that reflects the operating system and architecture that the binary file was compiled for (e.g. WhiteboxTools_darwin_amd64). Rename this directory to WBT. Also note, depending on your decompression software, it may be the case that the contents of the WBT folder itself contains a sub-directory that actually holds these files. If this is the case, be sure to move the contents of the sub-directory into the WBT parent directory.

Using your text editor, create a new Python script file, called FlowAccumulation.py within the FlowAccumExample directory. We will begin by importing the WhiteboxTools class from the whitebox_tools.py script contained within the WBT sub-directory. Unfortunately, Python's module system is only able to import classes and function definitions declared in external Python scripts if these external files are contained somewhere on the Python path or in the directory containing the script file into which you are importing. This is important because based on the project structure that we have established, the whitebox_tools.py script is actually contained within a sub-directory of the FlowAccumExample directory and is therefore not directly accessible, unless you have previously installed the script on the Python path. Another, perhaps easier solution to this problem is to create a file named __init__.py (those are two leading and trailing underscore characters) within the FlowAccumExample directory. The presence of this empty file will make Python treat the WBT directory as containing packages, in this case, the whitebox_tools package. For more information, see the Python documentation on modules and packages.

At this stage, you should have a project directory structure like that of the following:

Many operating systems will disallow the execution of files that are downloaded directly from the Internet. As such, it is possible that you will need to explicitly give the whitebox_tools.exe permission to execute on your computer (Note: here we are referring to the compiled WhiteboxTools binary file and not the similarly named Python script whitebox_tools.py also contained in the folder). The procedure for doing this depends on your specific operating system. On MacOS, for example, this is usually achieved using the 'Security & Privacy' tab under 'System Preferences'. To test whether whitebox_tools.exe has permission to run on your system, double-click the file. If the file is configured to execute, a command terminal will automatically open and the WhiteboxTools help documentation and a listing of the available tools will be printed. If this does not occur, you likely need to give the file permission to execute.

Using your text editor, you may now add the following lines to the FlowAccumulation.py file:

from WBT.whitebox_tools import WhiteboxTools

wbt = WhiteboxTools()

In the import statement, WBT is a reference to the package folder containing the WhiteboxTools files; whitebox_tools is a reference to the whitebox_tools.py script contained with this package folder; and WhiteboxTools is a reference to the WhiteboxTools class contained within this script file. Please note that if you named your directory containing the WhiteboxTools files something other than WBT, you would need to alter the import statement accordingly.

Visit the Geomorphometry and Hydrogeomatics Research Group WhiteboxTools Downloads website and download the St. Elis Mountains and Gulf of Alaska sample data set (StElisAk.las). This file contains a LiDAR point cloud that has been previously filtered to remove points associated with non-ground returns, mainly trees. Create a sub-directory within the project folder called 'data' and copy StElisAk.las into the folder.

Now we can complete our flow accumulation analysis with the following code:

import os
from WBT.whitebox_tools import WhiteboxTools

wbt = WhiteboxTools()

# Set the working directory, i.e. the folder containing the data,
# to the 'data' sub-directory.
wbt.work_dir = os.path.dirname(os.path.abspath(__file__)) + "/data/"

# When you're running mulitple tools, the outputs can be a tad
# chatty. In this case, you may want to suppress the output by
# setting the verbose mode to False.
# wbt.verbose = False

# Interpolate the LiDAR data using an inverse-distance weighting
# (IDW) scheme.
print("Interpolating DEM...")
wbt.lidar_idw_interpolation(
i="StElisAk.las",
output="raw_dem.tif",
parameter="elevation",
returns="last",
resolution=1.0,
weight=1.0,
radius=2.5
)

# The resulting DEM will contain NoData gaps. We need to fill
# these in by interpolating across the gap.
print("Filling missing data...")
wbt.fill_missing_data(
i="raw_dem.tif",
output="dem_nodata_filled.tif",
filter=11
)

# This DEM will contain grid cells that have no lower neighbours.
# This condition is unsuited for flow-path modelling applications
# because these operations assume that each interior cell in the
# DEM has at least one downslope neighour. We'll use an operation
# called depression breaching to 'fix' the elevations within the
# DEM to enforce continuous flow.
print("Performing flow enforcement...")
wbt.breach_depressions(
dem="dem_nodata_filled.tif",
output="dem_hydro_enforced.tif"
)

# Lastly, perform the flow accumulation operation using the
# D-infinity flow algorithm.
print("Performing flow accumulation...")
wbt.d_inf_flow_accumulation(
dem="dem_hydro_enforced.tif",
output="flow_accum.tif",
log=True
)

print("Complete!")

To run the above script, open a terminal (command prompt), cd to the script containing folder, and run the following command:

>>python FlowAccumulation.py

If Python 3 is not your default Python version, substitute python3 for python in the above command line. The final D-infinity flow accumulation raster can be displayed in any GIS software of choice and should look similar to the following:

Dr. John Lindsay
Rm. 346 Hutt Building
Department Geography, Environment & Geomatics
University of Guelph
50 Stone Rd. East
Guelph, ON, Canada, N1G 2W1

Email: jlindsay@uoguelph.ca
Phone: 519-824-4120 ext. 56074
Find me: LinkedIn, ResearchGate.

No recent updates.