Databricks - data export
Automatically export message engagement events from OneSignal to your Databricks lakehouse for advanced analytics and reporting.
OneSignal Databricks integration overview
Overview
The OneSignal + Databricks integration enables automatic export of message event data from OneSignal to your Databricks lakehouse. This removes the need for manual data handling and provides near real-time access to user engagement data for custom analytics, dashboards, and cross-functional reporting.
You’ll gain insight into user interactions across email, push notifications, in-app messages, and SMS. Currently, the integration supports message send and engagement events, with more event types planned for future updates.
Requirements
OneSignal
Databricks
- Platform: AWS, Azure, or GCP
- Plan: Premium or higher
- Unity Catalog: Required (legacy deployments not supported)
- SQL Warehouse: Required for ingestion and querying
Setup
Collect SQL warehouse details
Log in to your Databricks workspace
Go to SQL Warehouses in your Databricks workspace.
Select your warehouse
Select your warehouse and open the Connection details tab.
Save the following details
- Server Hostname
- Port
- HTTP Path
Databricks SQL connection details for Fivetran setup
Create a service principal
Go to the Service Principals page
Go to Workspace Settings > Identity and Access > Service Principals.
Add a new service principal
Click Add Service Principal, then Add New.
Name the service principal
Name it (e.g., onesignal-sync).
Modal for adding a service principal, with the 'Add new' option highlighted
Generate a secret
Click into the created principal
Go to the Secrets tab
Generate a secret
Click Generate Secret and save it securely.
The secret is only visible once—store it safely.
Databricks 'Generate secret' modal showing OAuth secret and client ID for API authentication
Assign permissions
Navigate to your Catalog and open the Permissions tab
Click Grant
Assign the following permissions to the service principal
- USE CATALOG
- USE SCHEMA
- SELECT
- MODIFY
- CREATE SCHEMA
- CREATE TABLE
Privilege assignment screen for a Databricks principal with custom catalog permissions selected.
Connect OneSignal
Activate the integration
In OneSignal, navigate to Data > Integrations > Databricks.
Enter the details
- Server Hostname
- Port
- HTTP Path
- Catalog Name
- Schema Name
- Service Principal credentials (ID + Secret)
OneSignal Databricks Configuration form with fields for catalog, hostname, HTTP path, and OAuth credentials.
Configure the integration
- Sync Frequency: as often as every 15 minutes
- Dataset/Table Names: pre-set as
onesignal_events_<app-id>andmessage_events(editable) - Event Types: choose which to sync—select all or just what you need
Select events
Select the events you care to receive in your Databricks catalog.
OneSignal event export settings screen showing sync status, dataset configuration, and selected message event types.
Complete the setup
Click Save and wait for the success confirmation
Generate message events
Send messages via push, email, in-app, or SMS to trigger events that will sync based on your Integration Settings. You can update which events to sync within your OneSignal dashboard Data > Integrations > Databricks.
View data in Databricks
-
Open your Catalog in Databricks.
-
Once syncing completes, your configured schema will appear.
-
Access and query the
message_eventstable.Databricks Catalog view showing OneSignal message events table under a production schema.
-
Click into tables for sample data preview.
Sample data from the message_events_1 table with synced OneSignal event fields.
If you run into issues like missing schemas, permission errors, or malformed events, contact support@onesignal.com.
Events and properties
Message event kinds
Property: event_kind
Type: String
The kind of message and event (e.g. message.push.received, message.push.sent).
| Message Event (OneSignal) | event_kind | Description |
|---|---|---|
| Push Sent | message.push.sent | Push notification successfully sent. |
| Push Received | message.push.received | Delivered push (see Confirmed Delivery). |
| Push Clicked | message.push.clicked | User clicked the push. |
| Push Failed | message.push.failed | Delivery failure. See message reports. |
| Push Unsubscribed | message.push.unsubscribed | User unsubscribed from push. |
| In-App Impression | message.iam.displayed | In-App message shown. |
| In-App Clicked | message.iam.clicked | In-App message clicked. |
| In-App Page Viewed | message.iam.pagedisplayed | In-App page shown. |
| Email Sent | message.email.sent | Email delivered. |
| Email Received | message.email.received | Email accepted by recipient’s mail server. |
| Email Opened | message.email.opened | Email opened. See Email Reports. |
| Email Link Clicked | message.email.clicked | Link in email clicked. |
| Email Unsubscribed | message.email.unsubscribed | Recipient unsubscribed. |
| Email Marked Spam | message.email.resporedasspam | Marked as spam. See Email Deliverability. |
| Email Bounced | message.email.hardbounced | Bounce due to permanent delivery failure. |
| Email Failed | message.email.failed | Delivery failed. |
| Email Suppressed | message.email.supressed | Suppressed due to suppression list. |
| SMS Sent | message.sms.sent | SMS sent. |
| SMS Delivered | message.sms.delivered | SMS successfully delivered. |
| SMS Failed | message.sms.failed | SMS failed to deliver. |
| SMS Undelivered | message.sms.undelivered | SMS rejected or unreachable. |
Event data schema
For each message event generated by a user, the following metadata will be attached to the record.
| Column Name | Type | Description |
|---|---|---|
event_id | UUID | Unique identifier for the event |
event_timestamp | Timestamp | Time of event occurrence |
event_kind | String | The Event Kind |
subscription_device_type | String | Device type (e.g., iOS, Android, Web, Email, SMS) |
language | String | Subscription language code |
version | String | Integration version |
device_os | String | Device operating system version |
device_type | Number | Numeric device type |
token | String | Push token, phone number, or email |
subscription_id | UUID | Subscription ID |
subscribed | Boolean | Subscription status |
onesignal_id | UUID | OneSignal user ID |
last_active | String | Last active timestamp |
sdk | String | OneSignal SDK version |
external_id | String | External user ID that should match the integration user ID |
app_id | UUID | App ID from OneSignal |
template_id | UUID | Template ID (if applicable) |
message_id | UUID | Message batch/request ID |
message_name | String | Name of the message |
message_title | String | Message title (English only) |
message_contents | String | Truncated message body (English only) |
_created, _id, _index, _fivetran_synced | Internal use | Fivetran sync metadata |
Notes
- Syncs after saving/activating may take an additional 15-30 minutes to complete.
- Deactivating may still result in one final sync after deactivation.
- To ensure efficient data synchronization, our system automatically creates and manages staging datasets. These datasets, named with a pattern like
fivetran_{two random words}_staging, temporarily store data during processing before it’s integrated into your main schema. These staging datasets are essential for maintaining a streamlined workflow and should not be deleted, as they will be automatically recreated.
FAQ
Why do I see different message IDs with the same content?
This happens when the same message is sent more than once, likely via a transactional flow or message template reused across multiple sends.