When I joined the team at 71point4 almost a year ago, one of my first onboarding training sessions was entitled “Dashboarding in Kibana”. The session used our Employment Equity Data, based on EEA2 forms submitted by almost 30 000 employers in South Africa to the Employment Equity Commission in the Department of Labour. We download and process these forms each year when they are published by the Department. Kibana was positioned as the open-source BI tool of choice. I had never used it before, and I was excited to learn something new.

I have a macroeconomics background and spent a lot of time in SQL databases, where an aggregation is a thing you do once, and then build a visualisation from there. I quickly learnt that in Kibana, the aggregations happen inside the chart – every chart. One panel at a time.

Admittedly I struggled a bit.  The charts I thought would be simple, were simply not appearing on command.  Eventually I realised I wasn’t doing anything wrong: the tool couldn’t work with the data structured the way it was. Elasticsearch (which Kibana sits on top of) isn’t a SQL database, it’s a document store, where each record is self-contained and de-normalised at write time. So there’s nothing to join to, and every chart starts again from the raw documents. You can’t aggregate, then aggregate that result, the way you’d stack a query on a query. In SQL you reshape the data and then chart. In Kibana, the structure arrived with the data.

Meanwhile, I was quietly furious about the colours. I’d just spent two years co-leading a clothing manufacturer, where we’d agonise over the slightest difference between two Pantone shades. Now Kibana offered only a handful of uninspiring options. It felt like I was designing a load-shedding schedule where aesthetics must give way to function.

It turns out that I wasn’t the only one on the team with reservations about Kibana’s features and ease of use. My colleague and 71point4’s principal Data Scientist Hanjo did some digging on alternatives and came up with Superset. He quickly deployed it on one of our internal servers and connected it to a public dataset on mobile towers around the world, and told me to go play. We were all impressed, and fast forward a few months, we were deploying it in the transactions pipeline for a client.

What’s the difference? {ELK stack} versus {Superset + a database}

ELK stands for Elasticsearch, Logstash and Kibana, and unlike Superset, it is a whole stack: ingestion, storage and dashboards all from one vendor. It was born out of a search problem; before Elasticsearch, finding records in a database that contained a specific word was painfully slow.

How Elasticsearch works is that documents (or observations) are accepted in JSON format. On the way in, Elasticsearch breaks each document’s text down into terms and lists their corresponding document numbers. This is the so-called inverted index. Unlike a normal index which maps a document to the words it contains, an inverted index flips this and maps specific words to the documents where those words appear. Searching then never touches the documents at all: it looks up a search term and returns the documents linked to that word or phrase, and the amount of processing required is driven by the number of records returned by the search, rather than the total number of records available. A database without an index on that column has to scan through every single row to figure out the same thing.

This is where Elasticsearch is still particularly strong. It was built for searching and indexing unstructured text, making it extremely effective when the problem is finding words or phrases across large volumes of data. This remains relevant today, including in AI applications such as RAG systems and agents, where retrieving the right information from large collections of documents is the critical first step.

Elasticsearch found its market in log analytics and became the default way to sift through millions of server logs, and remains incredibly effective for this. Logstash sits in front of it, shaping and enriching the documents on the way in. Kibana started as a way to visualise what Elasticsearch had found and became the BI face that appears to have been slapped on later,  at least in my opinion.

Superset, on the other hand, was born out of a different need. Maxime Beauchemin built it at Airbnb in 2015 and donated it to Apache as an open-source alternative to Tableau and Looker, for analysts who already had a data warehouse and just needed a way to visualise their data. So it is deployed with no storage and no ingestion. It is a (good-looking) face in search of a body, and it speaks SQL to over forty different databases.

Two projects, two pipelines, which is better?

The first project is confidential so I can’t divulge too many details about it, so let’s call it the transactions pipeline. This is a daily feed of transactions data pulled from an API, turned into a dashboard for the client’s operational teams. The second project is the Employment Equity data, which as I mentioned we scraped from the EEA2 forms on the Department of Labour’s website.

The transactions pipeline was built using a Kedro framework. Nodes fetch from the API, clean, transform, aggregate and write the data to a ClickHouse database. The pipeline runs daily. The output is a handful of aggregated tables in ClickHouse, and Superset connects straight to those. In Superset, the aggregate tables remain flexible, and you don’t have to make changes to anything in the pipeline if you need a different view or structure.

The EE pipeline starts the same way: fetches, parses, cleans. Then it diverges. Instead of tables, the output is documents, posted to Elasticsearch, where each document represents a specific part of the EEA2 form the firms fill in. Some transformation does happen before the data is available to visualise in Kibana, but the difference is that if you want another view, or you change a definition, that change needs to be implemented from within the pipeline, rather than in Kibana. Wanting something different later doesn’t mean writing a different query, it means re-indexing.

And at the end of the day, the EE data is also sitting in ClickHouse, on the same server. Which is odd, because Kibana can’t read ClickHouse. After using Superset, I understood why.

Here’s what I learnt and some of my favourite features.

Joins, SQL lab, and virtual datasets

As I’ve already been harping on about, Superset connects directly to your SQL database of choice. This gives you great flexibility to transform data, create new tables, perform new aggregations and create visualisations from these aggregations. And the real kicker is that you can do this from inside the Superset UI, in the SQL lab where you can do anything you could do inside a ClickHouse client.

SQL Lab is a proper query editor: schema browser on the left, results below, query history saved so you can go back to what you ran last week. It means the exploratory work and the dashboard work happen in the same place, so there’s no jumping between a database client and the BI tool.

A virtual dataset is a saved query, with each query given a name. Charts get built on top of it exactly as if it were a real table in the database, but nothing was written and no pipeline run was needed. A join or a derived column that nobody anticipated is a simple ten-minute job an analyst could run, rather than a change request that could take a data scientist a few hours.

The upshot is that when we build a dashboard for a client in Superset, we’re not just handing over a dashboard. They get a broader analytical environment, saved queries, a place to poke at the underlying data and the means to answer the next question themselves.

Native tabs and filter scoping

A bigger limitation of Kibana, at least in the version we were working on, is that a Kibana dashboard can’t have tabs. If you want four views of the same data, you build four separate dashboards and fake a tab bar with a panel of links across the top. They look like tabs, but they aren’t. Clicking one is a full-page navigation to a different dashboard object, so every panel re-runs its queries from scratch, and nothing you were looking at a second ago is still in memory. Your filters don’t come along either, unless you remembered to pin them. And when you want to change something structural, you have to change it four times.

In Superset this isn’t a problem; tabs are a native layout element. One dashboard, with one set of filters, and the filter state follows you as you move between tabs. They also load lazily: the charts in a tab don’t run their queries until you actually open it. So a dashboard with four tabs and fifty charts can load as quickly as one with five, because on first load it only uses computing resources for the five charts you can see.

And speaking of filters, Superset has another feature I didn’t know I wanted until I had it. In Superset, when you add a filter to a dashboard, you can choose exactly which charts it applies to. You can apply it to everything, or pick selected panels, or tabs. Say for example, you’re filtering down to a single province, and you want every chart on the tab to respond, except the one showing the national total, which is there as a benchmark.  In Kibana, a filter hits the whole dashboard and that’s that. In Superset, you untick one box and the benchmark chart stays put, while everything around it moves.

Custom themes, CSS and colour palettes

In my opinion, many data scientists, analysts and consultants downplay the value and importance of the design aspect of a dashboard. You eat with your eyes, and if it looks like a load shedding schedule, but contains data you can live without, you live without.

Superset gives you two main options here, at different levels. Themes are application-wide and live in the settings but can also be edited from the front end: fonts, primary colours, borders, the general look and feel. You set it once and every dashboard on that instance inherits it, which is what you want when you’re handing a client something that should look like it is part of their business.

Cascading Style Sheets (CSS) is used to control the visual styling and layout of a web page or interface. In Superset, it is applied per dashboard through a text box. This lets you customise very specific elements. If you don’t like the default text wrapping in a table, or the colour of the text that pops up when you hover on a calendar chart, you can change it in the custom CSS in just a few lines of code.

Then the colours. Like Kibana, Superset ships with a handful of built-in colour palettes. But unlike Kibana, you’re not stuck with them. I defined our own palette in the Superset config file, with an id, a label and the list of hex codes in the order I wanted them used. This config file is essentially Superset’s settings that live in a Python file on the server called superset_config.py. Superset then reads that on startup, registers the palette, and from then on, it shows up in the colour scheme dropdown on every chart, alongside the defaults.

After two years of arguing about Pantones, being able to put the exact colours I wanted into the tool was very satisfying.

We had good reason to reach for Kibana in the first place and deploy it for the EE dashboard. The team members that were initially trained didn’t have any SQL knowledge, and Kibana is a great tool for non-technical people to explore big datasets through a ‘drag and drop’ interface. And while we knew we were making Kibana do things it wasn’t really designed to do, we loved the role-based access control (RBAC), which was consistent over the entire stack, because everything came from a single vendor. The EEA2 form is a document by nature: nested, uneven and different depending on which firm filled it in and how much of it they bothered to complete. You can load something like that into Elasticsearch more or less as it arrives, without first having to design a schema or decide exactly at what level the data should be stored. That makes it possible to get a dashboard up and running very quickly. But the lack of upfront structure comes at a cost. More of the work preparing and understanding the data gets pushed into the dashboards themselves.

The important difference is where those decisions are made. With Kibana, much of the data preparation happens when the data is ingested, and other decisions are then made within individual dashboard panels. With Superset, more of that work happens in the data pipeline itself, where the transformations are explicit, reproducible, and stored alongside the code.

Because dashboards are not the end of the analytical process – ultimately, both datasets ended up in ClickHouse anyway, because at some point we needed to ask questions that the Kibana dashboard couldn’t answer, and that meant going back to the underlying data.