Mueez Khan

Generating Plotly charts for water data with Python and PolarsGenerating Plotly charts for water data with Python and Polars

"The dashboard was created to support water management decisions by individuals, communities, and utilities to promote better stewardship of water resources and greater resilience during droughts or water supply challenges." — Boerne Utilities

Why I used Rust for the dashboard's data processing

ETL Diagram

Interactive Flow Diagram

You may need to scroll or rotate your device to view this diagram better.

Water utility data

Groundwater data

Streamflow data

Precipitation data

Nightly Rust data processing scripts

Boerne Water Data Dashboard

Boerne Water Data Hub

100%

Press enter or space to select a node. You can then use the arrow keys to move the node around. Press delete to remove it and escape to cancel.

Press enter or space to select an edge. You can then press delete to remove it or escape to cancel.

Why I am using Python for this blog post

ETL process demonstration for total daily water demand

If you're on a mobile device then for the rest of this post (or at least for the tables and chart) you may want to rotate your phone to landscape mode .

Loading the water utility data

pip install polars plotly numpy

import polars as pl
df = pl.read_csv("utility-data.csv")
 
df.head()

Casting the Date column from str to date

df = df.with_columns(
    pl.col("Date").str.to_date("%m/%d/%Y")
)
 
df.head()

Representing the Total Water Demand column in Millions of Gallons per Day (MGD)

• Remove commas by replacing them with an empty string
• Cast the column from str to f64 (e.g. decimal numbers)
• Divide the values by 1000
• Round to 6 digits

df = df.with_columns(
    (pl.col("Total Water Demand").str.replace_all(",", "", literal=True).cast(pl.Float64) / 1000).round(6)
)
 
df.head()

Selecting relevant columns for the visualization

df = df.select(
    [
        pl.col("Date"),
        pl.col("Total Water Demand")
    ]
)
 
df.head()

Preparing dataframe for multiple chart visualizations

• Year (e.g. 2025) - for partitioning the dataframe by year
• Month and day (e.g. Jan-01) - for the x-axis in each visualization

df = df.with_columns(
    pl.col("Date").dt.year().alias("Year")
).with_columns(
    pl.col("Date").dt.strftime("%b-%d").alias("Month and day")
)
 
df.head()

Partitioning the dataframe by year

dfs = df.partition_by("Year")

Generating visualizations for each year as interactive scatter plots with Plotly express

import plotly.express as px
 
for curr_df in dfs:
    year = curr_df.item(0, "Year")
    px.scatter(
        curr_df,
        x="Month and day",
        y="Total Water Demand",
        title=f"Daily Total Water Demand (MGD) in {year}",
    ).show()
    print("")

Explore the live chart

Conclusion

Featured Publications

Technical writing on work experiences, projects, and tools you may find useful.

Mapping open-pit mines in the US

Python

Polars

Folium

Leaflet

Marimo

New "Typing with Typst" course

Typst

Gazetteers for New Mexico Water Data

React

Python

TypeScript

JavaScript

HTML/CSS

CKAN

MapLibre

Generating Plotly charts for water data

Python

Polars

Plotly

Data visualization

Gazetteers for Texas Water Data Hub

JavaScript

HTML/CSS

Python

Leaflet

Boerne Water Data Dashboard

Next.js

Rust

Data visualization

Data engineering

Polars

Plotly

commit-helper

Rust

Building a CKAN Release Timeline

Python

CKAN

TimelineJS v3

HTML

View more written content on my blog.