DATA INTEGRATION
DATA ACTIVATION
EMBEDDED DATA CLOUD
Popular database connectors
Popular SaaS connectors
SOFTWARE COMPANIES
ACCOUNTING & CONSULTANCY
ENTERPRISE
TECH COMPANIES
In the world of data analytics, efficient dashboard management is crucial for delivering timely insights. Metabase, a popular open-source business intelligence tool, offers powerful capabilities for creating interactive dashboards. However, as analytics needs grow, maintaining and updating multiple SQL queries across dashboards can become a time-consuming task. This is especially true when it comes to managing field filters, which are essential for creating dynamic, user-friendly dashboards.
This blog post explores an innovative solution to automate the process of updating SQL field filters across multiple queries in a Metabase dashboard. It delves into the challenges of manual updates, introduces a Python script to automate the process, and demonstrates how to leverage the Peliqan platform to streamline the workflow.
Metabase excels at self-service analytics, providing a user-friendly interface for setting up dashboards and exploring data. However, as dashboards grow in complexity, a common challenge emerges: the need to update field filters across multiple SQL queries.
Metabase offers a great feature called ‘Field Filters‘ that allows users to create dynamic, interactive dashboards. These filters enable users to slice and dice data without needing to modify the underlying SQL. However, when you need to add a new filter to multiple queries on a dashboard, the process can be tedious and error-prone if done manually.
To address this challenge, a Python script has been developed that automates the process of updating SQL field filters across all queries on a Metabase dashboard. This approach significantly reduces the time and effort required to maintain and enhance dashboards.
Here’s a breakdown of the approach:
The Python script updates the SQL field filter in three crucial places:
Here’s a sample of the core function that handles these updates:
# Function to copy a variable parameter from template_query to new_query
def add_variable(template_query, new_query, variable):
# Extract 'rate_targetgroup' from existing query
template_tag = template_query['dataset_query']['native']['template-tags'].get(variable)
parameter = next(
(param for param in template_query['parameters'] if param['slug'] == variable), None
)
# Check if the parameter exists
if not template_tag or not parameter:
print("rate_targetgroup parameter not found in existing query.")
return
# 1. Add 'rate_targetgroup' to 'template-tags' in new query
new_query['dataset_query']['native']['template-tags'][variable] = template_tag
# 2. Add 'rate_targetgroup' to 'parameters' list in new query
new_query['parameters'].append(parameter)
# 3. Update the SQL query in the new query to include 'rate_targetgroup' in the WHERE clause
replace_input = "WHERE"
replace_output = f"WHERE {{{{{variable}}}}} AND "
new_query['dataset_query']['native']['query'] = new_query['dataset_query']['native']['query'].replace(
replace_input, replace_output
)
return new_query
To make this automation even more accessible and user-friendly, the Peliqan platform has been leveraged. Peliqan is a data platform that simplifies data operations by integrating various tools and workflows. Its Metabase connector makes it easy to interact with Metabase instances programmatically.
Using Peliqan’s Streamlit component, an interactive app was created that guides users through the process of updating field filters. The app allows users to:
The app then handles the rest, pushing updates to the selected queries automatically.
After running the script, there’s one final step that still requires manual intervention: assigning the right filter variables in the Metabase dashboard settings. While this step remains manual for now, it’s significantly faster than updating each query individually. There’s potential to automate this step in the future as well.
### Full Code Snippet
Below is the complete Python script used for automating Metabase field filter updates. The script utilizes two key libraries:
Both libraries are preloaded in the Peliqan data app environment.
## connections
dbconn = pq.dbconnect('dw_509')
metabase_api = pq.connect('Metabase')
st.title("Update METABASE queries with new variables")
st.text("Please provide the id and name of the query and variables we can consider as example")
template_query_id = st.text_input("Enter Template Query ID")
variable_name = st.text_input("Enter Variable Name")
#DASHBOARD SELECTOR :
st.header("Dashboard Card Selector")
#get dashboars from the metabase connector
dashboards = dbconn.fetch('dw_509', 'metabase', 'dashboards')
# Extract dashboard names and IDs for dropdown
dashboard_names = [dashboard["name"] for dashboard in dashboards]
dashboard_ids = {dashboard["name"]: dashboard["id"] for dashboard in dashboards}
# Dropdown to select a dashboard
selected_dashboard_name = st.selectbox("Select a Dashboard", options=dashboard_names)
# Get the selected dashboard ID
selected_dashboard_id = dashboard_ids[selected_dashboard_name]
## GET CARDS
dashboard = {
'id': selected_dashboard_id,
}
data = metabase_api.get('dashboard', dashboard)
cards = data['dashcards']
card_options = [{'name': card['card']['name'], 'id': card['card']['id']}
for card in cards
if card['card'].get('query_type') == "native"]
# Dictionary to store selected card IDs
selected_card_ids = []
if card_options:
st.write("Select the cards you want:")
# Iterate over card options and create checkboxes
for card in card_options:
is_checked = st.checkbox(f"{card['name']} (ID: {card['id']})", value=True)
if is_checked:
selected_card_ids.append(card['id'])
# Display selected card IDs
#st.write("Selected Card IDs:", selected_card_ids)
# Display selected card details
selected_cards_details = [card for card in card_options if card['id'] in selected_card_ids]
#st.write("Selected Card Details:", selected_cards_details)
else:
st.write("No cards available for the selected dashboard.")
exit()
## UPDATE CARDS
# Function to copy a variable parameter from template_query to new_query
def add_variable(template_query, new_query, variable):
# Extract 'rate_targetgroup' from existing query
template_tag = template_query['dataset_query']['native']['template-tags'].get(variable)
parameter = next(
(param for param in template_query['parameters'] if param['slug'] == variable), None
)
# Check if the parameter exists
if not template_tag or not parameter:
print("rate_targetgroup parameter not found in existing query.")
return
# 1. Add 'rate_targetgroup' to 'template-tags' in new query
new_query['dataset_query']['native']['template-tags'][variable] = template_tag
# 2. Add 'rate_targetgroup' to 'parameters' list in new query
new_query['parameters'].append(parameter)
# 3. Update the SQL query in the new query to include 'rate_targetgroup' in the WHERE clause
replace_input = "WHERE"
replace_output = f"WHERE {{{{{variable}}}}} AND "
new_query['dataset_query']['native']['query'] = new_query['dataset_query']['native']['query'].replace(
replace_input, replace_output
)
return new_query
if st.button("Upate queries"):
for card in selected_cards_details:
st.subheader(f"updating card {card['name']} with id {card['id']}")
if card['id'] == int(template_query_id):
st.write("we skip this one as its the template query")
continue
template_query = metabase_api.get('query', {'id': template_query_id,})
# New query data (new_query)
new_query = metabase_api.get('query', {'id': card['id'],})
# Use the function to update new_query with the 'rate_targetgroup' parameter
updated_new_query = add_variable(template_query, new_query,variable_name )
result = metabase_api.update('query', updated_new_query)
st.json(result, expanded=False)
Some key benefits of using Peliqan for this automation include:
Automating the process of updating field filters in Metabase dashboards can significantly streamline analytics workflows. By leveraging Python scripting and platforms like Peliqan, the Metabase experience becomes even more powerful and efficient.
Peliqan plays a crucial role in this automation process. As an all-in-one data platform designed for business teams, data teams, and developers, Peliqan simplifies data operations by integrating various tools and workflows. Its Metabase connector provides a seamless way to interact with Metabase instances programmatically, making it an ideal choice for implementing this automation solution.
By combining the power of Metabase with Peliqan’s automation capabilities, organizations can significantly reduce the time and effort required for dashboard maintenance, allowing data teams to focus more on analysis and deriving insights.
FAQ’s
To add a field filter to a Metabase dashboard:
A field filter in Metabase is a feature that allows users to dynamically filter data in dashboards and questions based on specific fields in your data. Field filters create interactive elements (like dropdown menus or date pickers) that users can manipulate to refine the data shown in visualizations without needing to modify the underlying queries.
To add filters to a Metabase dashboard:
Multiple filters can be added to a single dashboard to provide various ways for users to interact with the data.
To add a dropdown filter in Metabase:
The exact appearance of the filter (dropdown vs. other formats) may depend on the data type of the field being filtered and the specific settings chosen.
Piet-Michiel Rappelet is a founder of Peliqan. Before Peliqan, Piet-Michiel co-founded Blendr.io, a no-code iPaaS integration platform. Blendr.io was acquired by Qlik in 2020 and Piet-Michiel became Director of Product Management Foundational Services at Qlik. Piet-Michiel’s primary interest is in scaling SaaS software and rolling out customer-oriented service teams. Piet-Michiel holds a Masters degree in mathematics, he lives with his wife and two kids in Belgium.