Monitoring
TaglessCRM can monitor exacly what CRM data was sent, what didn't send due to a format error, and what failed to send due to API error. The information is stored so you can easily retry (see next section) any failed events, sending only the failed events without sending the Successfully sent data again, allowing you to maintain your campaigns' data integrity (sending some data points more than once may distort your campaign data).
By default, Monitoring will be disabled so please make sure to enable it if you wish to use it.
The monitoring data will be stored in a new database in your GCP project, and
will be named tcrm_monitoring_dataset and the monitoring table
tcrm_monitoring_table by default. To change the default names, or to
enable/disable monitoring please refer to
section 3.3 in the installation guide.
TaglessCRM stores records for a few important events:
Data sent - describes exactly what CRM data was read from either BigQuery or Google Cloud Storage. This record is kept for data integrity purposes, so data that was once sent will not be sent again. Record schema:
Type TEXT DATETIME INT TEXT TEXT TEXT Column Name dag_name timestamp type_id location position info Description The name of the DAG this record refers to timestamp The type ID for "Data sent" record is -2 location is either BigQuery table URL or Google Cloud Storage bucket URL position within the source location number of rows read Data failed to send - describes the data that was meant to send with the error number and reason for failure.
Type TEXT DATETIME INT TEXT TEXT TEXT Column Name dag_name timestamp type_id location position info Description The name of the DAG this record refers to timestamp Error number location is either BigQuery table URL or Google Cloud Storage bucket URL position within the source location The data in JSON format Retry - describes the time a retry was ran and the DAG it was retrying.
Type TEXT DATETIME INT TEXT TEXT TEXT Column Name dag_name timestamp type_id location position info Description The name of the DAG this record refers to timestamp The type ID for "Retry" record is -4 location is either BigQuery table URL or Google Cloud Storage bucket URL Empty Empty
Monitoring Cleanup
If you have enabled monitoring, data will be saved to a BigQuery table for logging and retry purposes. You may wish to periodically delete data from this table to reduce costs and to comply with data privacy regulations.
You can setup monitoring cleaning to run automatically at any periods of time for each DAG separately or for all the DAGs together. You can also indicate the maximum time a record can live in the system before being deleted by the cleanup service.
To do this, you can run the tcrm_monitoring_cleanup DAG.