This tutorial shows you how to create the Node.js backend
component of the Transport Tracker, using a vehicle location
simulator. The simulated locations are useful during development and testing
of your application, so that you can run the
Transport Tracker without having to move Android devices around
to represent your vehicles.
The backend receives updates of vehicle locations from
the Firebase Realtime Database, and sets up the panel configurations for use by the
Transport Tracker map.
The tutorial includes a link to the code repository, setup instructions, and
a detailed walkthrough of the main parts of the code.
Skip this step if you have already set up a
Firebase Realtime Database for your Transport Tracker.
The Transport Tracker uses a Firebase Realtime Database to
communicate location updates between the various components of the server
and front end applications. When the vehicle locator, or the
simulator, stores the location updates in the Firebase Realtime Database, Firebase
sends automatic updates to the backend, which in turn updates the
front end display.
Set up your Firebase Realtime Database project:
Go to the Firebase
console and click Add project to create a project for
your Transport Tracker.
Enter a project name.
Click Create Project.
Get a Google Maps API key
Skip this step if you have already obtained a Maps API key
for your Transport Tracker.
Click the button below, which guides you through the process of registering a
project in the Google Cloud Platform Console, activates the required Google Maps Platform
and related services automatically, and generates a generic, unrestricted API
key.
During development and testing, you can use a single, unrestricted Google
API key for all the Google Maps Platform in your
Transport Tracker. When you are ready to move your system into
production, you must create separate API keys for each type of API, so that
you can secure the keys by HTTP referrer or IP address. For help, see the
pre-launch
checklist.
Go to Google Cloud Platform
console, sign in using your Google Account (Gmail or Google Apps)
and create a new project.
Enter a name for your project, and take note of your allocated project ID.
Activate Google Cloud Shell from the Cloud platform console, by
clicking the button at top right of the Cloud Shell toolbar.
The Cloud Shell window opens at the bottom of the Cloud console. This
is where you can enter commands to interact with the shell. Refer to the
Cloud Shell
documentation for details.
In Google Cloud Shell, or locally if you're not using Google Cloud,
create a transport-tracker directory for your application,
and copy across the backend directory from the cloned GitHub repo:
mkdir transport-tracker
cd transport-tracker
cp <my-cloned-repo>/backend/ .
cd backend
Edit the file serviceAccountKey.json, and paste in your
Firebase adminsdk service account credentials from the file
you downloaded earlier. Hint: If you're using the Google
Cloud Shell, you can use Cloud's
code
editor.
Edit the tracker_configuration.jsonfile and add the
following values:
databaseURL - the address of your
Firebase Realtime Database. You can find the URL in the Admin SDK
configuration snippet on the Firebase Service Accounts tab.
simulation - a configuration setting that determines
whether to use the vehicle simulator or real location data from the
vehicle locator. While developing and testing, set this
value to true.
Run npm install to install your dependencies. This may
take a few minutes.
Run the application:
npm run main
Open your Firebase Realtime Database to see the results:
You should see data in your Firebase Realtime Database, something like this:
Understand the code
This guide focuses on the Google Maps Platform used in the
Transport Tracker. Where relevant, the guide mentions other
concepts and related code. You can get the full code from the GitHub
repository linked at the top of the page.
It's helpful if you have a basic understanding of
Node.js and
npm.
Data flow
Below is a conceptual data flow diagram for the
Transport Tracker, focusing on the backend modules.
Node.js modules
These are the modules that make up the backend. Refer to the
diagram to see how they fit together:
Module
bus_simulator.js
Defines the BusSimulator class, which simulates the
positions of buses. It uses the time signal from HeartBeat
to look up bus locations in the generated paths.json store.
gtfs.js
Defines the GTFS class, which loads the GTFS
(General Transit Feed Specification) data
into an in-memory SQLite database, and provides functionality to query
the GTFS timetable.
heart_beat.js
Defines the HeartBeat class, which publishes time
updates, either real or simulated, to the Firebase Realtime Database.
main.js
The main logic for the Node.js application.
panel_changer.js
Defines the PanelChanger class, which publishes
the relevant part of the panels_config.json file. The front
end uses this file to determine the geographical area it needs to
display, along with the appropriate hotel markers for each panel.
road_snapper.js
Defines the RoadSnapper class, which uses the
Roads API
to snap GPS signals to actual roads.
time_table.js
Defines the TimeTable class, containing the core business
logic of the Transport Tracker. In summary:
Query the GTFS database in response to time change notifications
from the HeartBeat instance, via the time published in
the Firebase Realtime Database.
Gather predicted travel times from the
Directions API
and cache the results.
Group the transformed data in accordance with the panel
configuration for the map.
Generate fake vehicle paths for use in the simulator
While you're developing and testing, it's useful to have a simulated feed of
vehicle locations, instead of needing live data from the
vehicle locator. As a once-off step, you can prepare and store a
set of vehicle paths for use by the Transport Tracker's
simulation module. The GitHub repo includes a pre-generated set of paths in
paths.json. Note: This is a very large file.
The GitHub repo also includes the generate_paths.js module, which
you can use to generate the paths yourself:
generate_paths() calls the
Directions API to
plot a route from one location to another. It uses the
Node.js Client for Google Maps Services to
interact with the Directions API, and
bluebird as a JavaScript
promise library, along with
asyncawait, to handle
asynchronous communications.
Start the application from main.js
The starting point for the Transport Tracker backend
is main.js.
The following excerpt from the main.js file sets up your
credentials for the Firebase Admin SDK, which you obtained from Firebase
earlier in this tutorial. For details, see the
Firebase
documentation.
The core logic of main.js determines whether to use the vehicle
simulator or the real location data from the vehicle locator,
based on the value of the simulation setting in the
configuration file:
const gtfs = new GTFS();
new HeartBeat(timeRef, trackerConfig.simulation);
new TimeTable(timeRef, panelsRef, gtfs, panelConfig, googleMapsClient);
new PanelChanger(mapRef, panelConfig);
new PromoChanger(promoRef);
if (trackerConfig.simulation) {
const {BusSimulator} = require('./bus_simulator.js');
const generatedPaths = require('./paths.json');
new BusSimulator(timeRef, gtfs, busLocationsRef, generatedPaths);
} else {
// Exercise for the reader: integrate real bus location data
}
Load and parse the GTFS timetable data
The timetable information is available as a set of CSV files in
GTFS (General Transit Feed Specification) format.
(The sample
data in the GitHub repository is from the Google I/O Bus Tracker.)
The
gtfs.js
module defines the GTFS class, which loads the timetable data
into an in-memory SQLite database. The class offers a number of methods giving
access to the data, so that you can figure out where a specific bus should be
at a specific time. Examples of the methods are
getTripsForCalendarDate() and getStopInfoForTrip().
Simulate the location of moving vehicles
While you're developing and testing, it's useful to have a simulated feed of
vehicle locations, instead of needing live data from the
vehicle locator. Based on the value of the simulation
setting in the application configuration file, the core logic in
main.js (described above) determines
whether to use a vehicle simulator (bus_simulator.js) or the real
vehicle location data.
The bus_simulator.js module creates fake vehicle locations and
pushes them to the Firebase Realtime Database, based on:
The routes in the GTFS timetables.
A set of generated paths in generate_paths.js, created by
generate_paths.js (see above).
A series of latitude/longitude coordinates often doesn't give an accurate
indication of a route that follows a road. This is true whether the
coordinates originate as GPS signals from the vehicle locator or
as estimated coordinates from the simulator.
You can call the
Roads API to snap the
geographical coordinates to the actual road traveled.
road_snapper.js uses the
Node.js Client for Google Maps Services to
interact with the Roads API, and
bluebird as a JavaScript
promise library, along with
asyncawait, to handle
asynchronous communications.
< > Show/Hide
road_snapper.js
const _async = require('asyncawait/async');
const _await = require('asyncawait/await');
const TRIP_HISTORY_LENGTH = 20;
exports.RoadSnapper = class {
constructor(
timeRef,
rawBusLocationsRef,
snappedBusLocationsRef,
googleMapsClient
) {
this.snappedBusLocationsRef = snappedBusLocationsRef;
this.googleMapsClient = googleMapsClient;
this.history = {};
this.snapped = {};
this.val = null;
this.time = null;
timeRef.on('value', snapshot => {
this.time = snapshot.val();
});
rawBusLocationsRef.on('value', snapshot => {
const val = snapshot.val();
this.gatherHistory(val);
});
// Snap to roads every ten seconds
this.timeTimerId = setInterval(() => {
_async(() => {
this.snapToRoads();
})().catch(err => {
console.error(err);
});
}, 10000);
}
gatherHistory(val) {
this.val = val;
if (val && this.time) {
const tripnames = new Set(Object.keys(val));
Object.keys(this.history).forEach(historicTripname => {
if (!tripnames.has(historicTripname)) {
delete this.history[historicTripname];
}
});
tripnames.forEach(tripname => {
if (!this.history[tripname]) {
this.history[tripname] = [];
}
const point = {
lat: val[tripname].lat,
lng: val[tripname].lng,
moment: this.time.moment
};
this.history[tripname].push(point);
if (this.history[tripname].length > TRIP_HISTORY_LENGTH) {
this.history[tripname] = this.history[tripname].slice(
-TRIP_HISTORY_LENGTH
);
}
});
}
}
snapToRoads() {
// Work around concurrency issues by taking a snapshot of val
const valSnapshot = this.val;
if (valSnapshot && this.time) {
const tripnames = new Set(Object.keys(valSnapshot));
Object.keys(this.snapped).forEach(snappedTripname => {
if (!tripnames.has(snappedTripname)) {
delete this.snapped[snappedTripname];
}
});
tripnames.forEach(tripname => {
if (this.history.hasOwnProperty(tripname)) {
const path = this.history[tripname].map(point => {
return [point.lat, point.lng];
});
const result = _await(
this.googleMapsClient.snapToRoads({path}).asPromise()
);
if (result.json.snappedPoints) {
this.snapped[tripname] =
result.json.snappedPoints[
result.json.snappedPoints.length - 1
].location;
} else {
console.error(result);
this.snapped[tripname] = {};
}
} else {
this.snapped[tripname] = {};
}
});
let snappedBusPositions = {};
tripnames.forEach(tripname => {
snappedBusPositions[tripname] = {
lat: this.snapped.hasOwnProperty(tripname) &&
this.snapped[tripname].latitude
? this.snapped[tripname].latitude
: valSnapshot[tripname].lat,
lng: this.snapped.hasOwnProperty(tripname) &&
this.snapped[tripname].longitude
? this.snapped[tripname].longitude
: valSnapshot[tripname].lng,
route_color: valSnapshot[tripname].route_color,
route_id: valSnapshot[tripname].route_id,
route_name: valSnapshot[tripname].route_name
};
});
this.snappedBusLocationsRef.set(snappedBusPositions);
} else {
// If valSnapshot is empty, we have no buses on the road.
this.snappedBusLocationsRef.set({});
}
}
};
Generate a stream of time updates
At the heart of the application is the HeartBeat class, which
sends a stream of updates to the Firebase Realtime Database using either
simulated time or real time, depending on whether you're using the
vehicle simulator or live updates from the vehicle locator.
< > Show/Hide
heart_beat.js
const moment = require('moment-timezone');
const DATE_FORMAT = 'YYYYMMDD HH:mm:ss';
const SIMULATION_START = '2017-05-17 06:00';
const SIMULATION_END = '2017-05-19 18:00';
// HeartBeat generates a stream of updates to `timeRef`, with either
// simulated time updates, or real time updates, depending on the
// truthyness of `simulatedTime`
exports.HeartBeat = class {
constructor(timeRef, simulatedTime) {
this.simulationTime = moment.utc(SIMULATION_START, DATE_FORMAT);
this.endOfSimulation = moment.utc(SIMULATION_END, DATE_FORMAT);
this.timeRef = timeRef;
this.simulated = simulatedTime;
// Update the time once a second
this.timeTimerId = setInterval(() => {
this.timeAdvance();
}, 1000);
}
timeAdvance() {
if (this.simulated) {
this.timeRef.set({
display: this.simulationTime.format('h:mm A, MMM Do'),
moment: this.simulationTime.valueOf()
});
this.simulationTime = this.simulationTime.add(30, 'seconds');
if (this.simulationTime.diff(this.endOfSimulation, 'minutes') > 0) {
// Reset simulation to start once we run out of bus trips.
this.simulationTime = moment.utc(SIMULATION_START, DATE_FORMAT);
}
} else {
const now = moment();
this.timeRef.set({
display: now.tz('America/Los_Angeles').format('h:mm A, MMM Do'),
moment: now.valueOf()
});
}
}
};
Publish the timetable
The TimeTable class subscribes to the time updates in the
Firebase Realtime Database, generated by HeartBeat. At each time
update, TimeTable does the following:
Query the GTFS database to find the next scheduled trips.
Call the
Directions API
to get predicted travel time for each route.
Get the grouping of routes per UI panel from
panels_config.json.
Construct the UI panels for the map.
< > Show/Hide
time_table.js
const moment = require('moment-timezone');
const _async = require('asyncawait/async');
const _await = require('asyncawait/await');
const DATE_FORMAT = 'YYYYMMDD HH:mm:ss';
// TimeTable listens for updates on `timeRef`, and then publishes updated
// time table information on `panelsRef`, using `gtfs` as a source of
// next trips, `panelConfig` for the grouping of routes to panels, and
// `googleMapsClient` to access Directions API for Predicted Travel Times.
exports.TimeTable = class {
constructor(timeRef, panelsRef, gtfs, panelConfig, googleMapsClient) {
this.timeRef = timeRef;
this.panelsRef = panelsRef;
this.gtfs = gtfs;
this.panelConfig = panelConfig;
this.googleMapsClient = googleMapsClient;
// Cache of Predicted Travel Times
this.pttForTrip = {};
// When we last issued a Predicted Travel Time request for a route.
this.pttLookupTime = {};
this.timeRef.on(
'value',
snapshot => {
_async(() => {
const now = moment.utc(snapshot.val().moment);
this.publishTimeTables(now);
})().catch(err => {
console.error(err);
});
},
errorObject => {
console.error('The read failed: ' + errorObject.code);
}
);
}
publishTimeTables(now) {
const panels = _await(
this.panelConfig.map(panel => {
return {
left: _await(
panel.routesGroups[0].map(route_id => {
return this.tripsLookup(route_id, now);
})
),
right: _await(
panel.routesGroups[1].map(route_id => {
return this.tripsLookup(route_id, now);
})
)
};
})
);
this.panelsRef.set(panels);
}
tripLookup(trip) {
const stop_info = _await(this.gtfs.getStopInfoForTrip(trip.trip_id));
return {trip, stop_info};
}
haveDirectionsResponseCachedForTrip(trip) {
return this.directionsResponseForTrip(trip) !== undefined;
}
directionsResponseForTrip(trip) {
return this.pttForTrip[trip.trip_id];
}
tripsLookup(route_id, now) {
function round_moment(m) {
if (m.second() > 30) {
return m.add(1, 'minute').startOf('minute');
}
return m.startOf('minute');
}
const date = now.tz('America/Los_Angeles').format('YYYYMMDD');
const time = now.tz('America/Los_Angeles').format('HH:mm:ss');
const route = _await(this.gtfs.getRouteById(route_id));
const nextTrips = _await(
this.gtfs.getNextThreeTripsForRoute(route_id, date, time)
);
nextTrips.forEach(trip => {
this.cacheDirectionsResponseForTrip(trip);
});
const returnValue = {route, next_in_label: '', next_in: ''};
if (nextTrips.length >= 1) {
const next_trip = _await(this.tripLookup(nextTrips[0]));
returnValue.next_trip = next_trip;
if (
this.haveDirectionsResponseCachedForTrip(returnValue.next_trip.trip)
) {
const ptt = this.directionsResponseForTrip(returnValue.next_trip.trip);
const time = moment.tz(
`${next_trip.stop_info[0].date} ${next_trip.stop_info[0].departure_time}`,
DATE_FORMAT,
'America/Los_Angeles'
);
let index = 1;
ptt.routes[0].legs.forEach(leg => {
const delta = leg.duration.value;
time.add(delta, 'seconds');
const time_display = round_moment(time).format('HH:mm:ss');
next_trip.stop_info[index].departure_time = time_display;
next_trip.stop_info[index].arrival_time = time_display;
// Assume we stop at each way point for three minutes
time.add(3, 'minutes');
index++;
});
}
const next_trip_time_str = `${next_trip.stop_info[0].date} ${next_trip.stop_info[0].departure_time}`;
const next_trip_time = moment.tz(
next_trip_time_str,
DATE_FORMAT,
'America/Los_Angeles'
);
const next_trip_delta = next_trip_time.diff(now, 'minutes');
if (next_trip_delta <= 120) {
returnValue['leaving_in_label'] = 'Leaving in';
returnValue['leaving_in'] = `${next_trip_delta} mins`;
if (nextTrips.length >= 2) {
let trip_after = _await(this.tripLookup(nextTrips[1]));
// In the mornings we have a bunch of overlapping trips on inbound routes
if (
trip_after.stop_info[0].date === next_trip.stop_info[0].date &&
trip_after.stop_info[0].departure_time ===
next_trip.stop_info[0].departure_time
) {
trip_after = _await(this.tripLookup(nextTrips[2]));
}
const trip_after_time = moment.tz(
`${trip_after.stop_info[0].date} ${trip_after.stop_info[0].departure_time}`,
DATE_FORMAT,
'America/Los_Angeles'
);
returnValue['next_in_label'] = 'Next in';
const trip_after_delta = trip_after_time.diff(now, 'minutes');
if (trip_after_delta <= 120) {
returnValue['next_in'] = `${trip_after_delta} min`;
} else {
returnValue[
'next_in'
] = `${trip_after_time.diff(now, 'hours')} hrs`;
}
}
} else {
returnValue['leaving_in_label'] = next_trip_time.format('MMM Do');
returnValue['leaving_in'] = next_trip_time.format('h A');
}
}
return returnValue;
}
requestDirectionsForTrip(trip) {
const trip_info = this.tripLookup(trip);
const stops = [];
trip_info.stop_info.forEach(stop => {
stops.push({lat: stop.lat, lng: stop.lng});
});
const request = {origin: stops[0], destination: stops[stops.length - 1]};
if (stops.length > 2) {
request['waypoints'] = stops.slice(1, -1);
}
return request;
}
cacheDirectionsResponseForTrip(trip) {
if (
this.pttLookupTime[trip.trip_id] === undefined ||
moment().diff(this.pttLookupTime[trip.trip_id], 'minutes') > 20 ||
(this.pttForTrip[trip.trip_id] === undefined &&
moment().diff(this.pttLookupTime[trip.trip_id], 'minutes') > 3)
) {
this.pttLookupTime[trip.trip_id] = moment();
const request = this.requestDirectionsForTrip(trip);
if (
this.pttLookupFailure == undefined ||
moment().diff(this.pttLookupFailure, 'minutes') > 20
) {
const initiatedAt = moment();
this.googleMapsClient
.directions(request)
.asPromise()
.then(response => {
this.pttForTrip[trip.trip_id] = response.json;
})
.catch(err => {
this.pttLookupFailure = moment();
console.error(
`Google Maps Directions API request failed, initiated at ${initiatedAt.format('hh:mm a')}: ${err}`
);
});
} else {
console.log(
`Not looking up ${trip.trip_id}, rate limiting due to API error, at ${moment().format('hh:mm a')}`
);
}
}
}
};
Swap the UI panels at regular intervals
The Transport Tracker map displays one of three
geographical areas at any one time, along with the routes and markers
applicable to that area. The markers indicate hotels and bus stops.
The panels_config.json file defines the three panels. Each panel
defines the geographical bounds of the area that appears in that panel, and
the routes and markers that are relevant for that geographical area.
< > Show/Hide
panels_config.json (just one panel is included here)
The PanelChanger class rotates the panels, based on the
configuration in panels_config.json.
< > Show/Hide
panel_changer.js
// PanelChanger changes the focus of the front end display
// by publishing the three panels defined in `panelConfig`,
// one after the other every ten seconds. This changes the
// bounds of the displayed map, along with which Hotels and
// bus stops are featured.
exports.PanelChanger = class {
constructor(mapRef, panelConfig) {
this.mapRef = mapRef;
this.panelConfig = panelConfig;
this.panelIndex = 0;
// Change the panel once every ten seconds
this.timeTimerId = setInterval(() => {
this.panelAdvance();
}, 20000);
}
panelAdvance() {
this.mapRef.set(this.panelConfig[this.panelIndex]);
this.panelIndex = (this.panelIndex + 1) % this.panelConfig.length;
}
};
Other notable files and packages
As well as the Node.js modules, the backend includes these
files:
File
generate_paths.js
Constructs a prediction of where the vehicles will be at given
times.
The timetable information for the tracked vehicles. The sample data
is based on the timetable for the Google I/O Bus Tracker. This
tiemtable is stored in a set of CSV files that follow GTFS,
the General Transit Feed Specification.
package.json
The information required by npm to correctly download the development
and runtime dependencies, and to run the resulting application.
panels_config.json
The configuration for the panel layout in the
Transport Tracker map. Defines the three panels
of the display, each with a group of routes.
paths.json
Predicted vehicle locations and times. The
generate_paths.js script creates this file.
Note: Don't open this file in the editor, as it's very
large.
serviceAccountKey.json
The credentials for your Firebase service account.
tracker_configuration.json
Various configuration values, including the Google Maps API
key and configuration values for the Firebase Realtime Database.
Below is a list of the packages installed in the backend, with
a summary of their functionality.