Prepare for the PgBouncer and IPv4 deprecations on 26th January 2024

Learn more
Home

Self-Hosting Analytics

The Supabase Analytics server is a Logflare self-hostable instance that manages the ingestion and query pipelines for searching and aggregating structured analytics events.

When self-hosting the Analytics server, the full logging experience matching that of the Supabase Platform is available in the Studio instance, allowing for an integrated and enhanced development experience. However, it's important to note that certain differences may arise due to the platform's infrastructure.

Backends Supported

The Analytics server supports either Postgres or BigQuery as the backend. The supabase-cli experience uses the Postgres backend out-of-the-box. However, the Supabase Platform uses the BigQuery backend for storing all platform logs.

When using the BigQuery backend, a BigQuery dataset is created in the provided Google Cloud project, and tables are created for each source. Log events are streamed into each table, and all queries generated by Studio or by the Logs Explorer are executed against the BigQuery API. This backend requires internet access to work, and cannot be run fully locally.

When using the Postgres backend, tables are created for each source within the provided schema (for supabase-cli, this would be _analytics). Log events received by Logflare are inserted directly into the respective tables. All BigQuery-dialect SQL queries from Studio will be handled by a translation layer within the Analytics server. This translation layer translates the query to PostgreSQL dialect, and then executes it against the Postgres database.

The Postgres backend is not yet optimized for a high volume of inserts, or for heavy query usage. Today the translation layer only handles a limited subset of the BigQuery dialect. As such, the Log Explorer may produce errors for more advanced queries when using the Postgres Backend.

Getting Started

The Postgres backend is recommended when familiarizing and experimenting with self-hosting Supabase. For production, we recommend using the BigQuery backend. See production recommendations for more information.

To set up logging in self-hosted Supabase, see the docker-compose example. Two compose services are required: Logflare, and Vector. Logflare is the HTTP Analytics server, while Vector is the logging pipeline to route all compose services' syslog to the Logflare sever.

Regardless of the backend chosen, the following environment variables must be set for the supabase/logflare docker image:

  • LOGFLARE_SINGLE_TENANT_MODE=true: The feature flag for enabling single tenant mode for Logflare. Must be set to true
  • LOGFLARE_SUPABASE_MODE=true: The feature flag for seeding Supabase-related data. Must be set to true

For all other configuration environment variables, please refer to the Logflare self-hosting documentation.

Postgres Backend Setup

The example docker-compose uses the Postgres backend out of the box.


_10
# clone the supabase/supabase repo, and run the following
_10
cd docker
_10
docker compose -f docker-compose.yml up

Configuration and Requirements

  • supabase/logflare:1.4.0 or above
  • Relevant environment variables:
    • POSTGRES_BACKEND_URL : Required. The connection string to the Postgres database.
    • POSTGRES_BACKEND_SCHEMA : Optional. Allows customization of the schema used to scope all backend operations within the database.

BigQuery Backend Setup

The BigQuery backend is a more robust and scalable backend option that is battle-tested and production ready. Use this backend if you intend to have heavy logging usage and require advanced querying features such as the Logs Explorer.

Configuration and Requirements

The requirements are as follows after creating the project:

  • Google Cloud project with billing enabled
  • Project ID
  • Project number
  • A service account key.

Setting up BigQuery Service Account

The service account used must have sufficient permissions to insert into your Google Cloud BigQuery. Ensure that the service account has either:

  • BigQuery Admin role; or
  • The following permissions:
    • bigquery.datasets.create
    • bigquery.datasets.get
    • bigquery.datasets.getIamPolicy
    • bigquery.datasets.update
    • bigquery.jobs.create
    • bigquery.routines.create
    • bigquery.routines.update
    • bigquery.tables.create
    • bigquery.tables.delete
    • bigquery.tables.get
    • bigquery.tables.getData
    • bigquery.tables.update
    • bigquery.tables.updateData

You can create the service account via the web console or gcloud CLI, as per the Google Cloud documentation. In the web console, you can create the key by navigating to IAM > Service Accounts > Actions (dropdown) > Manage Keys

We recommend setting the BigQuery Admin role, as it simplifies permissions setup.

Download the Service Account Keys

After the service account is created, you will need to create a key for the service account. This key will sign the JWTs for API requests that the Analytics server makes with BigQuery. This can be done through the IAM section in Google Cloud console.

Docker Image Configuration

Using the example self-hosting stack based on docker-compose, you include the logging related services using the following command

  1. Update the .env.example file with the necessary environment variables.
  • GOOGLE_PROJECT_ID
  • GOOGLE_PROJECT_NUMBER
  1. Place your Service Account key in your present working directory with the filename gcloud.json.
  2. On docker-compose.yml, uncomment the block section below the commentary # Uncomment to use Big Query backend for analytics
  3. On docker-compose.yml, comment the block section below the commentary # Comment variables to use Big Query backend for analytics

Thereafter, you can start the example stack using the following command:


_10
# assuming you clone the supabase/supabase repo.
_10
cd docker
_10
docker compose -f docker-compose.yml

BigQuery Datset Storage Location

Currently, all BigQuery datasets stored and managed by Analytics, whether via CLI or self-hosted, will default to the US region.

Vector Usage

In the Docker Compose example, Vector is used for the logging pipieline, where log events are forwarded to the Analytics API for ingestion.

Please refer to the Vector configuration file when customizing your own setup.

You must ensure that the payloads matches the expected event schema structure. Without the correct structure, it would cause the Studio Logs UI features to break.

Differences from Platform

API logs rely on Kong instead of the Supabase Cloud API Gateway. Logs from Kong are not enriched with platform-only data.

Within the self-hosted setup, all logs are routed to Logflare via Vector. As Kong routes API requests to PostgREST, self-hosted or local deployments will result in Kong request logs instead. This would result in differences in the log event metadata between self-hosted API requests and Supabase Platform requests.

Production Recommendations

To self-host in a production setting, we recommend performing the following for a better experience.

Ensure that Logflare is behind a firewall and restrict all network access to it besides safe requests.

Self-hosted Logflare has UI authentication disabled and is intended for exposure to the internet. We recommend restricting access to the dashboard, accessible at the /dashboard path. If dashboard access is required for managing sources, we recommend having an authentication layer, such as a VPN.

Use a different Postgres Database to store Logflare data.

Logflare requires a Postgres database to function. However, if there is an issue with you self-hosted Postgres service, you would not be able to debug it as it would also bring Logflare down together.

The self-hosted example is only used as a minimal example on running the entire stack, however it is not recommended to use the same database server for both production and observability.

Use Big Query as the Logflare Backend

The current Postgres Ingestion backend isn't optimized for production usage. We recommend using Big Query for more heavy use cases.

We recommend using the BigQuery backend for production environments as it offers better scaling and querying/debugging experiences.

Client libraries

Integrations

List endpoints

get/api/endpoints

Responses

Endpoint List Response

{
  "schema": {
    "items": {
      "properties": {
        "cache_duration_seconds": {
          "type": "integer"
        },
        "enable_auth": {
          "type": "boolean"
        },
        "inserted_at": {
          "format": "date-time",
          "type": "string"
        },
        "max_limit": {
          "type": "integer"
        },
        "name": {
          "type": "string"
        },
        "proactive_requerying_seconds": {
          "type": "integer"
        },
        "query": {
          "type": "string"
        },
        "sandboxable": {
          "type": "boolean"
        },
        "source_mapping": {
          "type": "object"
        },
        "token": {
          "type": "string"
        },
        "updated_at": {
          "format": "date-time",
          "type": "string"
        }
      },
      "required": [
        "name",
        "query"
      ],
      "title": "Endpoint",
      "type": "object"
    },
    "type": "array"
  }
}

Create endpoint

post/api/endpoints

Responses

Created Response

{
  "schema": {
    "properties": {
      "cache_duration_seconds": {
        "type": "integer"
      },
      "enable_auth": {
        "type": "boolean"
      },
      "inserted_at": {
        "format": "date-time",
        "type": "string"
      },
      "max_limit": {
        "type": "integer"
      },
      "name": {
        "type": "string"
      },
      "proactive_requerying_seconds": {
        "type": "integer"
      },
      "query": {
        "type": "string"
      },
      "sandboxable": {
        "type": "boolean"
      },
      "source_mapping": {
        "type": "object"
      },
      "token": {
        "type": "string"
      },
      "updated_at": {
        "format": "date-time",
        "type": "string"
      }
    },
    "required": [
      "name",
      "query"
    ],
    "title": "Endpoint",
    "type": "object"
  }
}

Delete endpoint

delete/api/endpoints/{token}

Path Parameters
    token
    REQUIRED
    no type

    Endpoint Token

Responses

Accepted Response

Fetch endpoint

get/api/endpoints/{token}

Path Parameters
    token
    REQUIRED
    no type

    Endpoint Token

Responses

Endpoint Response

{
  "schema": {
    "properties": {
      "cache_duration_seconds": {
        "type": "integer"
      },
      "enable_auth": {
        "type": "boolean"
      },
      "inserted_at": {
        "format": "date-time",
        "type": "string"
      },
      "max_limit": {
        "type": "integer"
      },
      "name": {
        "type": "string"
      },
      "proactive_requerying_seconds": {
        "type": "integer"
      },
      "query": {
        "type": "string"
      },
      "sandboxable": {
        "type": "boolean"
      },
      "source_mapping": {
        "type": "object"
      },
      "token": {
        "type": "string"
      },
      "updated_at": {
        "format": "date-time",
        "type": "string"
      }
    },
    "required": [
      "name",
      "query"
    ],
    "title": "Endpoint",
    "type": "object"
  }
}

Update endpoint

patch/api/endpoints/{token}

Path Parameters
    token
    REQUIRED
    no type

    Endpoint Token

Responses

Created Response

{
  "schema": {
    "properties": {
      "cache_duration_seconds": {
        "type": "integer"
      },
      "enable_auth": {
        "type": "boolean"
      },
      "inserted_at": {
        "format": "date-time",
        "type": "string"
      },
      "max_limit": {
        "type": "integer"
      },
      "name": {
        "type": "string"
      },
      "proactive_requerying_seconds": {
        "type": "integer"
      },
      "query": {
        "type": "string"
      },
      "sandboxable": {
        "type": "boolean"
      },
      "source_mapping": {
        "type": "object"
      },
      "token": {
        "type": "string"
      },
      "updated_at": {
        "format": "date-time",
        "type": "string"
      }
    },
    "required": [
      "name",
      "query"
    ],
    "title": "Endpoint",
    "type": "object"
  }
}

List sources

get/api/sources

Responses

Source List Response

{
  "schema": {
    "items": {
      "properties": {
        "api_quota": {
          "type": "integer"
        },
        "bigquery_table_ttl": {
          "type": "integer"
        },
        "bq_table_id": {
          "type": "string"
        },
        "bq_table_schema": {
          "type": "object"
        },
        "custom_event_message_keys": {
          "type": "string"
        },
        "favorite": {
          "type": "boolean"
        },
        "has_rejected_events": {
          "type": "boolean"
        },
        "id": {},
        "inserted_at": {
          "format": "date-time",
          "type": "string"
        },
        "metrics": {
          "type": "object"
        },
        "name": {
          "type": "string"
        },
        "notifications": {
          "items": {
            "properties": {
              "other_email_notifications": {
                "type": "string"
              },
              "team_user_ids_for_email": {
                "allOf": {
                  "type": "string"
                },
                "type": "array"
              },
              "team_user_ids_for_schema_updates": {
                "allOf": {
                  "type": "string"
                },
                "type": "array"
              },
              "team_user_ids_for_sms": {
                "allOf": {
                  "type": "string"
                },
                "type": "array"
              },
              "user_email_notifications": {
                "type": "boolean"
              },
              "user_schema_update_notifications": {
                "type": "boolean"
              },
              "user_text_notifications": {
                "type": "boolean"
              }
            },
            "title": "Notification",
            "type": "object"
          },
          "type": "array"
        },
        "public_token": {
          "type": "string"
        },
        "slack_hook_url": {
          "type": "string"
        },
        "token": {
          "type": "string"
        },
        "updated_at": {
          "format": "date-time",
          "type": "string"
        },
        "webhook_notification_url": {
          "type": "string"
        }
      },
      "required": [
        "name"
      ],
      "title": "Source",
      "type": "object"
    },
    "type": "array"
  }
}

Create source

post/api/sources

Responses

Created Response

{
  "schema": {
    "properties": {
      "api_quota": {
        "type": "integer"
      },
      "bigquery_table_ttl": {
        "type": "integer"
      },
      "bq_table_id": {
        "type": "string"
      },
      "bq_table_schema": {
        "type": "object"
      },
      "custom_event_message_keys": {
        "type": "string"
      },
      "favorite": {
        "type": "boolean"
      },
      "has_rejected_events": {
        "type": "boolean"
      },
      "id": {},
      "inserted_at": {
        "format": "date-time",
        "type": "string"
      },
      "metrics": {
        "type": "object"
      },
      "name": {
        "type": "string"
      },
      "notifications": {
        "items": {
          "properties": {
            "other_email_notifications": {
              "type": "string"
            },
            "team_user_ids_for_email": {
              "allOf": {
                "type": "string"
              },
              "type": "array"
            },
            "team_user_ids_for_schema_updates": {
              "allOf": {
                "type": "string"
              },
              "type": "array"
            },
            "team_user_ids_for_sms": {
              "allOf": {
                "type": "string"
              },
              "type": "array"
            },
            "user_email_notifications": {
              "type": "boolean"
            },
            "user_schema_update_notifications": {
              "type": "boolean"
            },
            "user_text_notifications": {
              "type": "boolean"
            }
          },
          "title": "Notification",
          "type": "object"
        },
        "type": "array"
      },
      "public_token": {
        "type": "string"
      },
      "slack_hook_url": {
        "type": "string"
      },
      "token": {
        "type": "string"
      },
      "updated_at": {
        "format": "date-time",
        "type": "string"
      },
      "webhook_notification_url": {
        "type": "string"
      }
    },
    "required": [
      "name"
    ],
    "title": "Source",
    "type": "object"
  }
}

Delete source

delete/api/sources/{token}

Path Parameters
    token
    REQUIRED
    no type

    Source Token

Responses

Accepted Response

Fetch source

get/api/sources/{token}

Path Parameters
    token
    REQUIRED
    no type

    Source Token

Responses

Source Response

{
  "schema": {
    "properties": {
      "api_quota": {
        "type": "integer"
      },
      "bigquery_table_ttl": {
        "type": "integer"
      },
      "bq_table_id": {
        "type": "string"
      },
      "bq_table_schema": {
        "type": "object"
      },
      "custom_event_message_keys": {
        "type": "string"
      },
      "favorite": {
        "type": "boolean"
      },
      "has_rejected_events": {
        "type": "boolean"
      },
      "id": {},
      "inserted_at": {
        "format": "date-time",
        "type": "string"
      },
      "metrics": {
        "type": "object"
      },
      "name": {
        "type": "string"
      },
      "notifications": {
        "items": {
          "properties": {
            "other_email_notifications": {
              "type": "string"
            },
            "team_user_ids_for_email": {
              "allOf": {
                "type": "string"
              },
              "type": "array"
            },
            "team_user_ids_for_schema_updates": {
              "allOf": {
                "type": "string"
              },
              "type": "array"
            },
            "team_user_ids_for_sms": {
              "allOf": {
                "type": "string"
              },
              "type": "array"
            },
            "user_email_notifications": {
              "type": "boolean"
            },
            "user_schema_update_notifications": {
              "type": "boolean"
            },
            "user_text_notifications": {
              "type": "boolean"
            }
          },
          "title": "Notification",
          "type": "object"
        },
        "type": "array"
      },
      "public_token": {
        "type": "string"
      },
      "slack_hook_url": {
        "type": "string"
      },
      "token": {
        "type": "string"
      },
      "updated_at": {
        "format": "date-time",
        "type": "string"
      },
      "webhook_notification_url": {
        "type": "string"
      }
    },
    "required": [
      "name"
    ],
    "title": "Source",
    "type": "object"
  }
}

Update source

patch/api/sources/{token}

Path Parameters
    token
    REQUIRED
    no type

    Source Token

Responses

Created Response

{
  "schema": {
    "properties": {
      "api_quota": {
        "type": "integer"
      },
      "bigquery_table_ttl": {
        "type": "integer"
      },
      "bq_table_id": {
        "type": "string"
      },
      "bq_table_schema": {
        "type": "object"
      },
      "custom_event_message_keys": {
        "type": "string"
      },
      "favorite": {
        "type": "boolean"
      },
      "has_rejected_events": {
        "type": "boolean"
      },
      "id": {},
      "inserted_at": {
        "format": "date-time",
        "type": "string"
      },
      "metrics": {
        "type": "object"
      },
      "name": {
        "type": "string"
      },
      "notifications": {
        "items": {
          "properties": {
            "other_email_notifications": {
              "type": "string"
            },
            "team_user_ids_for_email": {
              "allOf": {
                "type": "string"
              },
              "type": "array"
            },
            "team_user_ids_for_schema_updates": {
              "allOf": {
                "type": "string"
              },
              "type": "array"
            },
            "team_user_ids_for_sms": {
              "allOf": {
                "type": "string"
              },
              "type": "array"
            },
            "user_email_notifications": {
              "type": "boolean"
            },
            "user_schema_update_notifications": {
              "type": "boolean"
            },
            "user_text_notifications": {
              "type": "boolean"
            }
          },
          "title": "Notification",
          "type": "object"
        },
        "type": "array"
      },
      "public_token": {
        "type": "string"
      },
      "slack_hook_url": {
        "type": "string"
      },
      "token": {
        "type": "string"
      },
      "updated_at": {
        "format": "date-time",
        "type": "string"
      },
      "webhook_notification_url": {
        "type": "string"
      }
    },
    "required": [
      "name"
    ],
    "title": "Source",
    "type": "object"
  }
}

List teams

get/api/teams

Responses

Team List Response

{
  "schema": {
    "items": {
      "properties": {
        "name": {
          "type": "string"
        },
        "team_users": {
          "items": {
            "properties": {
              "email": {
                "type": "string"
              },
              "name": {
                "type": "string"
              }
            },
            "required": [
              "email",
              "name"
            ],
            "title": "TeamUser",
            "type": "object"
          },
          "type": "array"
        },
        "token": {
          "type": "string"
        },
        "user": {
          "properties": {
            "api_key": {
              "type": "string"
            },
            "api_quota": {
              "type": "integer"
            },
            "bigquery_dataset_id": {
              "type": "string"
            },
            "bigquery_dataset_location": {
              "type": "string"
            },
            "bigquery_project_id": {
              "type": "string"
            },
            "company": {
              "type": "string"
            },
            "email": {
              "type": "string"
            },
            "email_me_product": {
              "type": "boolean"
            },
            "email_preferred": {
              "type": "string"
            },
            "image": {
              "type": "string"
            },
            "name": {
              "type": "string"
            },
            "phone": {
              "type": "string"
            },
            "provider": {
              "type": "string"
            },
            "token": {
              "type": "string"
            }
          },
          "required": [
            "email",
            "provider",
            "token",
            "provider_uid",
            "api_key"
          ],
          "title": "User",
          "type": "object"
        }
      },
      "required": [
        "name"
      ],
      "title": "Team",
      "type": "object"
    },
    "type": "array"
  }
}

Create Team

post/api/teams

Responses

Created Response

{
  "schema": {
    "properties": {
      "name": {
        "type": "string"
      },
      "team_users": {
        "items": {
          "properties": {
            "email": {
              "type": "string"
            },
            "name": {
              "type": "string"
            }
          },
          "required": [
            "email",
            "name"
          ],
          "title": "TeamUser",
          "type": "object"
        },
        "type": "array"
      },
      "token": {
        "type": "string"
      },
      "user": {
        "properties": {
          "api_key": {
            "type": "string"
          },
          "api_quota": {
            "type": "integer"
          },
          "bigquery_dataset_id": {
            "type": "string"
          },
          "bigquery_dataset_location": {
            "type": "string"
          },
          "bigquery_project_id": {
            "type": "string"
          },
          "company": {
            "type": "string"
          },
          "email": {
            "type": "string"
          },
          "email_me_product": {
            "type": "boolean"
          },
          "email_preferred": {
            "type": "string"
          },
          "image": {
            "type": "string"
          },
          "name": {
            "type": "string"
          },
          "phone": {
            "type": "string"
          },
          "provider": {
            "type": "string"
          },
          "token": {
            "type": "string"
          }
        },
        "required": [
          "email",
          "provider",
          "token",
          "provider_uid",
          "api_key"
        ],
        "title": "User",
        "type": "object"
      }
    },
    "required": [
      "name"
    ],
    "title": "Team",
    "type": "object"
  }
}

Delete Team

delete/api/teams/{token}

Path Parameters
    token
    REQUIRED
    no type

    Team Token

Responses

Accepted Response

Fetch team

get/api/teams/{token}

Path Parameters
    token
    REQUIRED
    no type

    Team Token

Responses

Team Response

{
  "schema": {
    "properties": {
      "name": {
        "type": "string"
      },
      "team_users": {
        "items": {
          "properties": {
            "email": {
              "type": "string"
            },
            "name": {
              "type": "string"
            }
          },
          "required": [
            "email",
            "name"
          ],
          "title": "TeamUser",
          "type": "object"
        },
        "type": "array"
      },
      "token": {
        "type": "string"
      },
      "user": {
        "properties": {
          "api_key": {
            "type": "string"
          },
          "api_quota": {
            "type": "integer"
          },
          "bigquery_dataset_id": {
            "type": "string"
          },
          "bigquery_dataset_location": {
            "type": "string"
          },
          "bigquery_project_id": {
            "type": "string"
          },
          "company": {
            "type": "string"
          },
          "email": {
            "type": "string"
          },
          "email_me_product": {
            "type": "boolean"
          },
          "email_preferred": {
            "type": "string"
          },
          "image": {
            "type": "string"
          },
          "name": {
            "type": "string"
          },
          "phone": {
            "type": "string"
          },
          "provider": {
            "type": "string"
          },
          "token": {
            "type": "string"
          }
        },
        "required": [
          "email",
          "provider",
          "token",
          "provider_uid",
          "api_key"
        ],
        "title": "User",
        "type": "object"
      }
    },
    "required": [
      "name"
    ],
    "title": "Team",
    "type": "object"
  }
}

Update team

patch/api/teams/{token}

Path Parameters
    token
    REQUIRED
    no type

    Team Token

Responses

Created Response

{
  "schema": {
    "properties": {
      "name": {
        "type": "string"
      },
      "team_users": {
        "items": {
          "properties": {
            "email": {
              "type": "string"
            },
            "name": {
              "type": "string"
            }
          },
          "required": [
            "email",
            "name"
          ],
          "title": "TeamUser",
          "type": "object"
        },
        "type": "array"
      },
      "token": {
        "type": "string"
      },
      "user": {
        "properties": {
          "api_key": {
            "type": "string"
          },
          "api_quota": {
            "type": "integer"
          },
          "bigquery_dataset_id": {
            "type": "string"
          },
          "bigquery_dataset_location": {
            "type": "string"
          },
          "bigquery_project_id": {
            "type": "string"
          },
          "company": {
            "type": "string"
          },
          "email": {
            "type": "string"
          },
          "email_me_product": {
            "type": "boolean"
          },
          "email_preferred": {
            "type": "string"
          },
          "image": {
            "type": "string"
          },
          "name": {
            "type": "string"
          },
          "phone": {
            "type": "string"
          },
          "provider": {
            "type": "string"
          },
          "token": {
            "type": "string"
          }
        },
        "required": [
          "email",
          "provider",
          "token",
          "provider_uid",
          "api_key"
        ],
        "title": "User",
        "type": "object"
      }
    },
    "required": [
      "name"
    ],
    "title": "Team",
    "type": "object"
  }
}