By Jackson Goode, Engineer at nLine
nLine built the Plotly.js Panel for Grafana to visualize, explore, and interact with the rich energy data we collect for our analysts and partners. We’ve open-sourced the code and contributed it as a Grafana community plugin for anyone on the platform to use. So far we’ve addressed dozens of community issues, features, and suggestions and are nearing 1 million downloads. This blog provides a look a how the project came to be, from tenable fork to a robust rewrite that fits seamlessly within the Grafana ecosystem.
At nLine, data collection and analysis are at the core of what we do. However, the quality of our data, from source to analysis, is best appreciated when effectively visualized. This motivation led us to create GridWatch, a robust, interactive, and multi-tenant dashboard experience built on Grafana, an excellent open-source tool that handles user management, data source connections, and variables for our dashboards. Grafana is a great example of a platform built in public and one that benefits from feedback and contributions from the open-source community.
Starting with a Fork
While Grafana provided a solid foundation, we still needed more control over how our data was displayed than provided by its standard suite of panels. Our team has long been comfortable with Plotly, a flexible visualization library written in JavaScript but accessible across multiple programming languages. Plotly is another pillar of the open-source graphing ecosystem. We were excited to find a Plotly panel in the Grafana community plugins, which allowed us to start integrating our data with basic plots on the web. However, we soon discovered limitations that prevented us from delivering the best experience to our partners in a live, interactive web app.
Like many open source projects, the maintenance and publication of ready-to-use packages is time-consuming, laborious, and often a labor of love. The Plotly plugin that we found had for a while gone untended to and severely needed upgrades to keep it compatible with the changes that Grafana was constantly making to its dashboard experience. We also found that the ease of actually interacting with the panel was a bit clunky and needed some cleanup and care.
Without the ability to contribute to the existing upstream project, we forked the existing Plotly panel project and began to integrate new features. We added time zone correction to ensure time series data corresponds to the Grafana dashboard's time zone, and improved image export functionality to allow dimensions and format to be set. We also enhanced integration with Grafana dashboard variables and functions, allowing for automatic insertion of titles, ranges, and other properties that adjust layout and formatting. Our modernization efforts brought the panel up to date with the most recent versions of Grafana.
We recently introduced the ability for the panel to dynamically adjust its time range based on the dashboards time select as well as dynamically matching the Grafana theme if a user switches from light to dark mode. We also ensured that font and sizing defaults are consistent with other built-in Grafana panels, making the integration of Plotly's wide range of visualization possibilities more aesthetically consistent with Grafana's native styles. These subtle details make the Plotly plugin much closer to a native panel within Grafana than other community built plugins.
Fitting into our Provisioned Suite
Our work on the Plotly panel was part of a larger initiative to create dashboards programmatically. By rendering Grafana through jsonnet, a templating language, we can describe the various aspects of our dashboards, panels, variables, time ranges, scripts, all entirely in code. We are able to separate unique types of the elements that compose each dashboard, such as JavaScript files for post-processing and SQL queries for data retrieval that are used in getting data to the Plotly panel and doing data wrangling to present it as a chart. This modularization has significantly improved our ability to track changes and reduce code repetition across the vast array of internal, partner, and public facing dashboards. As apart of our public data release of the GridWatch Accra Dataset, we have hosted public dashboards where you can view the plugin in action.
Since forking the Plotly plugin, our panel has effectively undergone a complete rewrite, both as a result of adding features we need as well as incorporating feedback from community members on GitHub. As we approach one million downloads, we're proud to have filled a niche within the data visualization community. Our enhancements have made it possible for others to enjoy the flexibility of using Plotly within a robust platform like Grafana. At nLine, we remain committed to independent auditing and analysis of improvements to energy infrastructure where data visualization serves as an essential piece. Given that much of our technology stack is built on FOSS, we are happy to give back to the ecosystem that has been crucial to our success.
You can check out the plugins source code here, install it on Grafana here and read more about our journey at nLine here.