Data sync
Troubleshooting
This section provides solutions to common issues that may arise when using Userpilot Data Sync.
Even with careful setup, you might occasionally encounter issues with Userpilot Data Sync. This section covers common problems and how to diagnose and resolve them.
Common Sync Issues and Resolutions
Issue: Sync Fails with Authentication Error
Issue: Sync Fails with Authentication Error
- Symptoms: Sync history shows failed attempts with error messages related to authentication, access denied, or invalid credentials.
- Possible Causes:
- Incorrect Access Key ID, Secret Access Key, Service Account JSON, or Connection String/SAS Token provided during configuration.
- Credentials have expired or been revoked (e.g., rotated AWS keys, expired SAS token).
- Insufficient permissions for the provided credentials (e.g., IAM user/role lacks
s3:PutObjectpermission, GCS service account lacksStorage Object Creatorrole). - Typo in bucket name, project ID, or container name.
- Resolution:
- Carefully verify all credentials and configuration parameters entered in the Userpilot Data Sync settings against your cloud provider console.
- Ensure the credentials are still active and have not expired.
- Check the permissions granted to the IAM user/role or service account in your cloud provider. Ensure they have the necessary write permissions to the specified bucket/container and path prefix.
- If using IP whitelisting, ensure Userpilot’s IPs (if applicable) are allowed.
- After correcting, try re-saving the configuration or manually triggering a sync if possible.
❗ Security Best Practice: Credential ManagementAlways use dedicated credentials (e.g., a specific IAM user or service account) for Userpilot Data Sync with the minimum necessary permissions (principle of least privilege). Regularly review and rotate these credentials as per your security policy.
Issue: Sync Completes, but No New Files Appear in Storage
Issue: Sync Completes, but No New Files Appear in Storage
- Symptoms: Userpilot indicates the sync was successful, but you don’t see new data files in your S3 bucket, or GCS bucket for the expected period. - Possible Causes: - Incorrect path prefix specified in Userpilot Data Sync settings, causing files to be written to an unexpected location within your bucket/container. - Delays in your cloud storage provider reflecting new objects (rare, but possible for very short periods). - No new event data was generated in Userpilot during the sync period (e.g., for a new setup with no user activity yet). - Filters on your storage browser or listing commands are hiding the files. - Resolution: 1. Double-check the Path Prefix in your Userpilot Data Sync configuration. Ensure it matches exactly where you expect the files. 2. Use your cloud provider’s console or CLI tools to list objects at the exact path prefix without any filters. 3. Verify that there was actual user activity in your application that would have generated Userpilot events during the sync window. 4. Check Userpilot’s sync logs (if available) for any specific file names or paths it attempted to write.
Issue: Data Discrepancies or Missing Data
Issue: Data Discrepancies or Missing Data
- Symptoms: You notice that the data in your storage seems incomplete compared to what you see in the Userpilot dashboard, or specific events/properties are missing. - Possible Causes: - Sync latency: Data for the most recent period might not have been included in the latest sync batch yet (especially for hourly syncs). - Schema changes: If Userpilot updated its event schema and your ingestion process wasn’t prepared, it might fail to parse new fields or events. - Errors during your ETL/ELT process: The issue might be in how you are loading or transforming the data after it lands in your storage. - Partial sync failure: A sync job might have partially completed before encountering an error. - Resolution: 1. Allow for some latency, especially if comparing to real-time Userpilot dashboards. Check the timestamp of the last successful sync. 2. Review the “Data Sync: Event Schema Reference” for any recent changes. Ensure your downstream processes can handle the current schema. 3. Thoroughly examine your own data ingestion and transformation logs for errors. 4. Check Userpilot’s sync history for any reported partial failures or errors. 5. Contact Userpilot support if you suspect an issue with the data export itself.
Issue: Slow Sync Performance
Issue: Slow Sync Performance
- Symptoms: Sync jobs are taking an unusually long time to complete.
- Possible Causes:
- Very large data volumes being exported.
- Throttling by your cloud storage provider if API limits are exceeded (less common for standard usage).
- Temporary network issues between Userpilot and the cloud storage provider.
- Resolution:
- If data volume is the cause, this might be normal. Consider if a less frequent sync (e.g., daily instead of hourly) is acceptable if performance is a concern.
- Monitor for persistent slowness. If it’s a consistent issue, contact Userpilot support to investigate.
Data Validation Tips
It’s good practice to periodically validate the synced data to ensure integrity and completeness.- File Counts: Check if the number of files being delivered per sync period is consistent (accounting for variations in event volume).
- Record Counts: After ingesting data into your warehouse, run basic counts. For example, count the number of events per day and compare this trend with general activity levels in the Userpilot dashboard. Expect some differences due to sync timing and processing, but look for major discrepancies.
- Schema Checks: Periodically verify that the data being received matches the expected schema. Tools in your data pipeline can often be configured to alert on schema drifts.
- Spot-Checking Key Metrics: Calculate a few key metrics from the raw data (e.g., number of specific key events, active user counts for a past period) and compare them directionally with Userpilot’s analytics. Remember that Data Sync provides raw data, and Userpilot’s internal analytics may have different calculation logic or aggregation windows.
ℹ️ Note: Comparing with Userpilot AnalyticsDirect 1:1 matching of numbers between your synced raw data and Userpilot’s aggregated dashboard analytics can be challenging due to differences in processing times, time zone handling (raw data is typically UTC), and specific metric definitions. Focus on directional consistency and trends rather than exact matches for high-level validation.