Metabase field filter sql

Metabase Field Filter

Automate SQL Field Filter Updates in a Metabase Dashboards

In the world of data analytics, efficient dashboard management is crucial for delivering timely insights. Metabase, a popular open-source business intelligence tool, offers powerful capabilities for creating interactive dashboards. However, as analytics needs grow, maintaining and updating multiple SQL queries across dashboards can become a time-consuming task. This is especially true when it comes to managing field filters, which are essential for creating dynamic, user-friendly dashboards.

This blog post explores an innovative solution to automate the process of updating SQL field filters across multiple queries in a Metabase dashboard. It delves into the challenges of manual updates, introduces a Python script to automate the process, and demonstrates how to leverage the Peliqan platform to streamline the workflow.

The Challenge: Manual Field Filter Updates

Metabase excels at self-service analytics, providing a user-friendly interface for setting up dashboards and exploring data. However, as dashboards grow in complexity, a common challenge emerges: the need to update field filters across multiple SQL queries.

Example query, using field filtersMetabase offers a great feature called ‘Field Filters‘ that allows users to create dynamic, interactive dashboards. These filters enable users to slice and dice data without needing to modify the underlying SQL. However, when you need to add a new filter to multiple queries on a dashboard, the process can be tedious and error-prone if done manually.

Enter Automation: A Python Script Solution

To address this challenge, a Python script has been developed that automates the process of updating SQL field filters across all queries on a Metabase dashboard. This approach significantly reduces the time and effort required to maintain and enhance dashboards.

Here’s a breakdown of the approach:

#1 — Script to Update a Query (Card)

The Python script updates the SQL field filter in three crucial places:

  • Template Tags: Adds the new variable to the query’s template tags.
  • Parameters: Adds the new variable to the query parameters list.
  • SQL Query: Inserts the variable into the SQL query’s WHERE clause.

Here’s a sample of the core function that handles these updates:

				
					# Function to copy a variable parameter from template_query to new_query
def add_variable(template_query, new_query, variable):
    # Extract 'rate_targetgroup' from existing query
    template_tag = template_query['dataset_query']['native']['template-tags'].get(variable)
    parameter = next(
        (param for param in template_query['parameters'] if param['slug'] == variable), None
    )
    
    # Check if the parameter exists
    if not template_tag or not parameter:
        print("rate_targetgroup parameter not found in existing query.")
        return
    
    # 1. Add 'rate_targetgroup' to 'template-tags' in new query
    new_query['dataset_query']['native']['template-tags'][variable] = template_tag
    
    # 2. Add 'rate_targetgroup' to 'parameters' list in new query
    new_query['parameters'].append(parameter)
    
    # 3. Update the SQL query in the new query to include 'rate_targetgroup' in the WHERE clause
    replace_input = "WHERE"
    replace_output = f"WHERE {{{{{variable}}}}} AND "
    new_query['dataset_query']['native']['query'] = new_query['dataset_query']['native']['query'].replace(
        replace_input, replace_output
    )

    return new_query

				
			

#2 — Running the Script with Peliqan

To make this automation even more accessible and user-friendly, the Peliqan platform has been leveraged. Peliqan is a data platform that simplifies data operations by integrating various tools and workflows. Its Metabase connector makes it easy to interact with Metabase instances programmatically.

Using Peliqan’s Streamlit component, an interactive app was created that guides users through the process of updating field filters. The app allows users to:

  • Select the dashboard they want to update
  • Choose which queries (cards) on the dashboard to modify
  • Specify the new field filter to be added

The app then handles the rest, pushing updates to the selected queries automatically.

Peliqan data app to select the dashboard and queries, and run the script

#3 — Finalizing Dashboard Filters

After running the script, there’s one final step that still requires manual intervention: assigning the right filter variables in the Metabase dashboard settings. While this step remains manual for now, it’s significantly faster than updating each query individually. There’s potential to automate this step in the future as well.

metabase dashboard filters

### Full Code Snippet

Below is the complete Python script used for automating Metabase field filter updates. The script utilizes two key libraries:

  • pq: The Peliqan library, used to connect to the Peliqan datawarehouse and the Metabase connector.
  • st: The Streamlit library, used for creating the interactive user interface.

Both libraries are preloaded in the Peliqan data app environment.

				
					## connections

dbconn = pq.dbconnect('dw_509')
metabase_api = pq.connect('Metabase')


st.title("Update METABASE queries with new variables")
st.text("Please provide the id and name of the query and variables we can consider as example")
template_query_id = st.text_input("Enter Template Query ID")
variable_name = st.text_input("Enter Variable Name")



#DASHBOARD SELECTOR : 
st.header("Dashboard Card Selector")
#get dashboars from the metabase connector
dashboards = dbconn.fetch('dw_509', 'metabase', 'dashboards')
# Extract dashboard names and IDs for dropdown
dashboard_names = [dashboard["name"] for dashboard in dashboards]
dashboard_ids = {dashboard["name"]: dashboard["id"] for dashboard in dashboards}
# Dropdown to select a dashboard
selected_dashboard_name = st.selectbox("Select a Dashboard", options=dashboard_names)

# Get the selected dashboard ID
selected_dashboard_id = dashboard_ids[selected_dashboard_name]


## GET CARDS 
dashboard = {
    'id': selected_dashboard_id,
}
data = metabase_api.get('dashboard', dashboard)


cards = data['dashcards']


card_options = [{'name': card['card']['name'], 'id': card['card']['id']} 
                    for card in cards 
                    if card['card'].get('query_type') == "native"]

# Dictionary to store selected card IDs
selected_card_ids = []
if card_options:
    st.write("Select the cards you want:")
    
    # Iterate over card options and create checkboxes
    for card in card_options:
        is_checked = st.checkbox(f"{card['name']} (ID: {card['id']})", value=True)
        if is_checked:
            selected_card_ids.append(card['id'])

    # Display selected card IDs
    #st.write("Selected Card IDs:", selected_card_ids)

    # Display selected card details
    selected_cards_details = [card for card in card_options if card['id'] in selected_card_ids]
    #st.write("Selected Card Details:", selected_cards_details)
else:
    st.write("No cards available for the selected dashboard.")
    exit()



## UPDATE CARDS

# Function to copy a variable parameter from template_query to new_query
def add_variable(template_query, new_query, variable):
    # Extract 'rate_targetgroup' from existing query
    template_tag = template_query['dataset_query']['native']['template-tags'].get(variable)
    parameter = next(
        (param for param in template_query['parameters'] if param['slug'] == variable), None
    )
    
    # Check if the parameter exists
    if not template_tag or not parameter:
        print("rate_targetgroup parameter not found in existing query.")
        return
    
    # 1. Add 'rate_targetgroup' to 'template-tags' in new query
    new_query['dataset_query']['native']['template-tags'][variable] = template_tag
    
    # 2. Add 'rate_targetgroup' to 'parameters' list in new query
    new_query['parameters'].append(parameter)
    
    # 3. Update the SQL query in the new query to include 'rate_targetgroup' in the WHERE clause
    replace_input = "WHERE"
    replace_output = f"WHERE {{{{{variable}}}}} AND "
    new_query['dataset_query']['native']['query'] = new_query['dataset_query']['native']['query'].replace(
        replace_input, replace_output
    )


    return new_query

if st.button("Upate queries"): 
    for card in selected_cards_details:
        st.subheader(f"updating card {card['name']} with id {card['id']}")
        if card['id'] == int(template_query_id):
            st.write("we skip this one as its the template query")
            continue 

        
        template_query = metabase_api.get('query', {'id': template_query_id,})

        # New query data (new_query)
        new_query =  metabase_api.get('query', {'id': card['id'],})    
        # Use the function to update new_query with the 'rate_targetgroup' parameter
        updated_new_query = add_variable(template_query, new_query,variable_name )

        result = metabase_api.update('query', updated_new_query)
        st.json(result, expanded=False)
				
			

Benefits of Automation

Some key benefits of using Peliqan for this automation include:

  • Easy integration with existing Metabase setups
  • User-friendly interface for managing automation tasks
  • Scalability to handle large numbers of dashboards and queries
  • Built-in scheduling capabilities for regular filter updates
  • Collaboration features that allow teams to work together on dashboard management

Conclusion

Automating the process of updating field filters in Metabase dashboards can significantly streamline analytics workflows. By leveraging Python scripting and platforms like Peliqan, the Metabase experience becomes even more powerful and efficient.

Peliqan plays a crucial role in this automation process. As an all-in-one data platform designed for business teams, data teams, and developers, Peliqan simplifies data operations by integrating various tools and workflows. Its Metabase connector provides a seamless way to interact with Metabase instances programmatically, making it an ideal choice for implementing this automation solution.

By combining the power of Metabase with Peliqan’s automation capabilities, organizations can significantly reduce the time and effort required for dashboard maintenance, allowing data teams to focus more on analysis and deriving insights.

FAQ’s

1. How do I add a field filter in a Metabase dashboard?

To add a field filter to a Metabase dashboard:

  1. Open your dashboard in edit mode.
  2. Click on the “Add a Filter” button in the top-right corner.
  3. Choose the type of filter you want to add (e.g., Time, Location, ID, etc.).
  4. Select which cards (visualizations) on the dashboard should use this filter.
  5. Configure any additional settings for the filter.
  6. Click “Done” to save your changes.

2. What is a field filter?

A field filter in Metabase is a feature that allows users to dynamically filter data in dashboards and questions based on specific fields in your data. Field filters create interactive elements (like dropdown menus or date pickers) that users can manipulate to refine the data shown in visualizations without needing to modify the underlying queries.

3. How do I add filters in the dashboard?

To add filters to a Metabase dashboard:

  1. Edit the dashboard by clicking the pencil icon.
  2. Click “Add a Filter” in the top-right corner.
  3. Choose the filter type and configure its settings.
  4. Map the filter to specific cards on your dashboard.
  5. Save your changes.

Multiple filters can be added to a single dashboard to provide various ways for users to interact with the data.

4. How to add a dropdown filter in Metabase?

To add a dropdown filter in Metabase:

  1. While editing your dashboard, click “Add a Filter”.
  2. Choose “Category” or “Location” as the filter type (these typically create dropdown filters).
  3. Select the field you want to use for the filter.
  4. In the filter settings, you can choose to display the filter as a dropdown menu.
  5. Map the filter to the relevant cards on your dashboard.
  6. Save your changes.

The exact appearance of the filter (dropdown vs. other formats) may depend on the data type of the field being filtered and the specific settings chosen.

Piet-Michiel

Piet-Michiel

Piet-Michiel Rappelet is a founder of Peliqan. Before Peliqan, Piet-Michiel co-founded Blendr.io, a no-code iPaaS integration platform. Blendr.io was acquired by Qlik in 2020 and Piet-Michiel became Director of Product Management Foundational Services at Qlik. Piet-Michiel’s primary interest is in scaling SaaS software and rolling out customer-oriented service teams. Piet-Michiel holds a Masters degree in mathematics, he lives with his wife and two kids in Belgium.