Migration from DT v1.0

Getting Files

The ingestion process for DTv2 is similar to DTv1 but there are some key things that you will need to change during your migration.

Processing Files

DTv2 processes hourly files (impression, click, and rich media) and daily files (activity and match tables). Hourly files (file name contains YYYYMMDDHH) are processed in UTC time. Daily files (file name contains YYYYMMDD) are processed in your local reporting time zone see file name format. The offset between your local time and UTC will differ depending on your location.

Files are also processed independently, and their processing time can vary. It's common for a later hour file to finish processing before an earlier file (like hour 6 before hour 5). Don't rely on file order for your ingestion process, otherwise your process may stall.

Events can appear in earlier or later processed files relative to their event time. Events are not always processed in the hour in which they occurred.

Column Order

Please do not rely on column order as a mechanism to consume your files. We want your processing to be resilient to change; if you request extra columns or if we make changes in the future then the order and number of columns in your reports can change. Read the header row of each file and map this to fields in your data warehouse before attempting to write data.

Duplicate files

Sometimes duplicate files are written for the same date and hour. If more than one file has the same date/hour stamp, use the one with the latest minutes / seconds, based on the filename timestamp. Duplicate files are created because back end processes determined there was an issue with the original file.

Fields

The field names and field name format all changed but there is a mapping from old to new (where available) at DCM Field migration including Match Tables. If you are not a DBM user, these will be empty and you can ignore the field.

You may also see DBM Fields in your file. If you are a DBM user these fields will populate only when the relevant permission is granted on the DBM advertiser level where advertisers are linked. There is a mapping from old to new (where available) at DBM Field migration. DBM entity read files will still be used for mapping purposes.

Bucket Names

The naming standards for DTv1 and DTv2 are different, specifically you can’t change the prefix on your existing bucket name to work out the DTv2 bucket name. The bucket name will be given to you by your support representative when your account is setup.

Generally DTv2 bucket names look like gs://dcdt_-dcm_account1234

You can see the contents of your bucket in a browser by looking at https://cloud.google.com/console#/storage/browser/dcdt_-dcm_account1234

File Name Format

Each file name will have a string of numbers, for example:
dcm_account1234_impression_2016022601_20160225_234912_218211994.csv.gz
2016022601 is in YYYYMMDDHH format. This is the UTC hour for events in that file (hours are numbered 0 to 23).
20160225_234912 is in YYYYMMDD_HHMMSS format. This is the time at which the report was generated.
218211994 is the file ID.

Activity Files

You get one Activity file per day and the filename looks like this:
dcm_account1234_activity_20160727_20160728_035750_268669761.csv.gz
This file contains data for 27 July 2016 and was generated at 3:57:50 on 28 July 2016.

Click Files

You get twenty four Click files per day and the filename looks like this:
dcm_account1234_click_2016072717_20160728_012331_268381796.csv.gz
Take note of the UTC Hour after the date string in the filename; hours are numbered from 0 to 23 so a 17 here indicates that the events in this file are for 17:00 to 17:59 (24 hour clock time) or 5:00pm to 6:00pm.
This file contains data for 5:00pm to 6:00pm, 27 July 2016 and was generated at 1:23:31 on 28 July 2016.

Impression Files

You get twenty four Impression files per day and the filename looks like this:
dcm_account7312_impression_2016072717_20160728_012355_268381795.csv.gz
Other than the filename this behaves the same as Click Files.

Match Table Files

Match Table files are generated once daily, and filenames look like this:
dcm_account1234_match_table_activity_cats_20160727_20160728_032226_268648829.csv.gz
This file contains data for 27 July 2016 and was generated at 3:22:26 on 28 July 2016.
Note: For some new match tables the data is static and no daily downloadable file is produced; you can get the data for these files from the reference pages e.g. Rich Media standard event types.

Data Transfer Fields

1.0 field name DT v2.0 field name
Time Deprecated
User-ID User ID
Advertiser-ID Advertiser ID
Buy-ID Deprecated
Order-ID Campaign ID
Ad-ID Ad ID
Creative-ID Rendering ID
Creative-Version Creative Version
Creative-Size-ID Deprecated (Retrieved from Match Table as Creative Pixel Size)
Site-ID Site ID (DCM)
Page-ID Placement ID
Keyword Deprecated
Country-ID Country Code
State/Province State/Region
Areacode Deprecated
Browser-ID Browser/Platform ID
Browser-Version Browser/Platform Version
OS-ID Operating System ID
DMA-ID Designated Market Area (DMA) ID
City-ID City ID
Zip-Code ZIP/Postal Code
Time-UTC-Sec Deprecated
Local-User-ID Deprecated
Activity-Type Deprecated (Retrieved from 'activity_cats' Match Table)
Activity-Sub-Type Deprecated (Retrieved from 'activity_cats' Match Table)
Other-Data Other Data
Ordinal ORD Value
Click-Time Interaction Time
Event-ID Conversion ID
SV1 Segment Value 1
PartnerID1 Partner 1 ID
PartnerID2 Partner 2 ID
Quantity Total Conversions
Revenue Total Revenue

Match table file names

1.0 file name 2.0 file name
activity_cat activity_cats
activity_type activity_types
ad ads
ad_page_assignments ad_placement_assignments
advertiser advertisers
browser browsers
campaign campaigns
cities cities
countries countries
creative creatives
creative_ad_assignments creative_ad_assignments
custom_creative_fields custom_creative_fields
dartsearch paid_search
designated_market_area designated_market_areas
keyval keyword_values
met_event rich_media_events
operating_system operating_systems
page placements
page_cost/page_flight_cost placement_cost
site sites
states states
user-defined_spot_vars custom_floodlight_variables

Match table fields

activity_cats

1.0 field name 2.0 field name Type
Spot-ID Floodlight Configuration integer
Activity-Type-ID Activity Group ID integer
Activity-Type Activity Type string
Activity-ID Activity ID integer
Activity Sub-Type Activity Sub-Type string
Report-Name Activity string
Tag-Method-ID Tag Counting Method ID integer

activity_types

1.0 field name 2.0 field name Type
Spot-ID Floodlight Configuration integer
Activity-Group-ID Activity Group ID integer
Activity-Type Activity Type string
Report-Name Activity Group string

ads

1.0 field name 2.0 field name Type
Advertiser-ID Advertiser ID integer
Buy-ID Campaign ID integer
Ad-ID Ad ID integer
Ad-Description Ad string
Click-URL Ad Click URL string
Ad-Type Ad Type string
Creative-Size-ID Creative Pixel Size string
Ad-Comments Ad Comments string

ad_placement_assignments

1.0 field name 2.0 field name Type
Ad-ID Ad ID integer
Page-ID Placement ID integer

advertisers

1.0 field name 2.0 field name Type
Spot-ID Floodlight Configuration integer
Advertiser-ID Advertiser ID integer
Advertiser Advertiser string
Advertiser-Group-ID Advertiser Group ID integer
Advertiser-Group Advertiser Group string

browsers

1.0 field name 2.0 field name Type
Browser-ID Browser/Platform ID integer
Browser Browswer/Platform string

campaigns

1.0 field name 2.0 field name Type
Advertiser-ID Advertiser ID integer

Order-ID (formerly Buy-ID )

Campaign ID integer
Order (formerly Buy) Campaign string
Start-Date Campaign Start Date date
End-Date Campaign End Date date
Billing-Invoice-Code Billing Invoice Code string

cities

1.0 field name 2.0 field name Type
City-ID City ID integer
City City string
Creative-Library-Enabled Deprecated (now always 1) integer

countries

1.0 field name 2.0 field name Type
Country-ID Deprecated integer
Country Deprecated string

creatives

1.0 field name 2.0 field name Type
Advertiser-ID Advertiser ID integer
Rendering-ID Rendering ID integer
UI-Creative-ID deprecated integer
Creative Creative string
Last-Modified-Date Creative Last Modified Date date
Creative-Type Creative Type string
Creative-Sub-Type Deprecated string
Creative-Size-ID Creative Pixel Size string
Image-URL Creative Image URL string
Creative-Version Creative Version integer

creative_ad_assignments

1.0 field name 2.0 field name Type
Ad-ID Ad ID integer
Creative-ID Creative ID integer
Start-Date Creative Start Date date
End-Date Creative End Date date
Rotation-Type Creative Rotation Type string
Creative-Group-1 Creative Groups 1 string
Creative-Group-2 Creative Groups 2 string
Click-URL Ad Click URL string

custom_creative_fields

1.0 field name 2.0 field name Type
Advertiser-ID Advertiser ID integer
Creative-ID Creative ID integer
Custom-Field-Number Creative Custom Field Number integer
Custom-Field-Name Creative Custom Field Name string
Custom-Field-Value Creative Custom Field Value string

custom_rich_media

1.0 field name 2.0 field name Type
Advertiser-ID Advertiser ID integer
Met-Event-ID Rich Media Event ID integer
Met-Event-Name Rich Media Event string
Met-Event-Type-ID Rich Media Event Type ID integer
Met-Event-Type-Name Rich Media Event Type string

designated_market_areas

1.0 field name 2.0 field name Type
DMA-ID Designated Market Area (DMA) ID integer
DMA Designated Market Area string

keyword_values

1.0 field name 2.0 field name Type
Ad-ID Ad ID integer
Keyword-Expression Keyword string
1.0 field name 2.0 field name Type
Ad-ID Ad ID integer
Advertiser-ID Advertiser ID integer
Buy-ID Campaign ID integer
DS-Ad-ID Paid Search Ad ID integer
Extended-Keyword-ID Paid Search Keyword ID integer
Keyword-ID Paid Search Legacy Keyword ID integer
DS-Campaign-Name Paid Search Campaign string
KW-Ad-Group-Name Paid Search Ad Group string
Bid-Strategy-Name Paid Search Bid Strategy string
Landing-Page-URL Paid Search Landing Page URL string
Keyword-Name Paid Search Keyword string
Match-Type Paid Search Match Type string

operating_systems

1.0 field name 2.0 field name Type
OS-ID Operating System ID integer
OS Operating System string

placements

1.0 field name 2.0 field name Type
Buy-ID Campaign ID integer
Site-ID Site ID (DCM) integer
Page-ID Placement ID integer
Page Site keyname string
Site-Placement Placement string
Content-Category Content Category string
Strategy Placement Strategy string
Start-Date Placement Start Date date
End-Date Placement End Date date
Group-Type Placement Group Type string
Group-Parent-ID Package/Roadblock ID integer
Price-Type Placement Cost Structure string
Cap-Cost Placement Cap Cost Option float
CPA-Activity-ID Activity ID integer

placement_cost

1.0 field name 2.0 field name Type
Page-ID Placement ID integer
Effective-Date Placement Start Date date
End-Date Placement End Date date
Units Placement Total Booked Units integer
Rate Placement Rate float
Comments Placement Comments string

sites

1.0 field name 2.0 field name Type
Site-ID Site ID (DCM) integer
Site Site (DCM) string
Directory-Site-ID Site ID (Site Directory) integer
Directory-Site Site (Site Directory) string

states

1.0 field name 2.0 field name Type
State/Province State/Region string
State/Province Name State/Region Full Name string

custom_floodlight_variables

1.0 field name 2.0 field name Type
Spot-ID Floodlight Configuration ID integer
Variable-ID Floodlight Variable ID string
Variable-Name Floodlight Variable Name string