7.5.142-stable Switch to dev
In this page
  1. Setup
  2. Project structure
  3. Data Model
  4. Import the data
    1. Authorization
    2. Contracts
    3. Stations
  5. Scheduled import
  6. Explorer
    1. Home
    2. Graph
  7. API
  8. UI (Frontend)

Bike Rental Service Digital Twin

Before reading this, make sure you have at least read trough the concepts.

To build our first digital twin with greycat we will be using JCDecaux’s Open Data.

Their data contains the geographic location of renting bikes world wide as well as their availability frequently updated.

This is the perfect use case for leveraging the full potential of GreyCat.

We will periodically fetch the data from their API, store it in our graph database, do some simple analytics, without ever leaving our GreyCat ecosystem and relying on third party libraries.

Project Repository link

Setup

Start by installing greycat as explained here

Create your app directory we will name ours biking, and create a project.gcl file at it’s root

Project structure

Let’s start by creating a simple file structure to host all our logic in an organized way.

├── project.gcl
└── backend
    ├── importer.gcl
    ├── model.gcl
    └── api.gcl

You can of course use your own preferred structure

We put all our server logic in a backend directory to be able to separate it from our frontend code

  • project: is mandatory and make this an actual GreyCat project, it also conventionally hosts @include and @library pragmas
  • api: will contain all our endpoints logic to fetch data from our graph
  • importer: will contain our data fetching logic from JCDecaux
  • model: will be used to store our graphs entry points as well as the data structures (types)

We setup our project.gcl with the following content:

@library("std", "7.5.142-stable");    // explicitly set the version of GreyCat
@library("explorer", "7.5.8-stable"); // add the Explorer to the project

@include("backend"); // tell GreyCat to load modules from that directory

/// The project entrypoint
fn main() {
    println("Hello, world!");
}

Data Model

Our graph entrypoints will be declared at the top of our src/model.gcl module:

/// All stations indexed by coordinates
var stations_by_geo: nodeGeo<node<Station>>;

/// All contracts indexed by contract name
var contracts_by_name: nodeIndex<String, node<Contract>>;

For a more in depth explanation of GreyCat’s nodes refer to the concepts/nodes documentation

We separate the Station into two distinct types, to avoid storing redundant data, Station has static data and StationData has dynamic data, so we can store them in a nodeTime for efficient and fast sampling.

Still in src/model.gcl, we add the following:

type Station {
    number: int;
    position: geo;
    /// Static information about the Station
    ///
    /// We want that data to be lazily loaded, therefore we put it in a node
    detail: node<StationDetail>;
    /// Dynamic data of a station indexed by the last update timestamp
    records: nodeTime<StationRecord>;
    /// The stations Contract, this is a graph edge or an SQL relationship
    contract: node<Contract>;
    profile: node<StationProfile>;
}

type StationDetail {
    name: String;
    address: String;
    banking: bool;
    bonus: bool;
}

type StationRecord {
    status: StationStatus;
    bike_stands: int;
    available_bike_stands: int;
    available_bikes: int;
}

enum StationStatus {
    OPEN,
    CLOSED,
}

type StationProfile {
    /// `24 * 7` slots, one for each hour of the week, will help to visualize the station activity in a typical week
    hourlyProfile: GaussianProfile;
    hourlyQuantizer: MultiQuantizer;

    fn updateProfile(t: time, data: StationRecord) {
        if (data.bike_stands > 0) {
            // Convert the time to the correct slot
            var date = t.toDate(null);
            var slot = this.hourlyProfile.quantizer.quantize(Array<int> { t.dayOfWeek(null), date.hour });
            this.hourlyProfile.add(
                this.hourlyQuantizer.slot_vector(slot),
                data.available_bikes as float / data.bike_stands as float
            );
        }
    }

    fn profileToTable(): Table {
        var table = Table {};
        var endSlot = 24 * 7;
        var startSlot = 0;

        while (startSlot < endSlot) {
            var row = startSlot / 24;
            var col = startSlot % 24;
            var val = this.hourlyProfile.avg(Array<int> { row, col });

            table.set_cell(row, col, val);
            startSlot++;
        }
        return table;
    }
}

type Contract {
    name: String;
    commercial_name: String?;
    country_code: String?;
    cities: Array<String>?;

    /// list of nodes to all stations of the contract indexed by the number property
    ///
    /// This translates to a "many relationship" in SQL
    stations: nodeIndex<int, node<Station>>;
}

Import the data

We are going to use 2 endpoints from JCDecaux Dev API: v1/contracts and v1/stations.

In order to import the JSON response we will model the types of those two endpoints in our src/importer.gcl module:

@volatile
private type JsonContract {
    /// identifier of the contract
    name: String;
    /// commercial name of the contract, the one users usually know
    commercial_name: String?;
    /// code (ISO 3166) of the country
    country_code: String?;
    /// cities that are concerned by this contract
    cities: Array<String>?;
}

@volatile
private type JsonStation {
    /// number of the station. This is NOT an id, thus it is unique only inside a contract.
    number: int;
    /// name of the contract of the station
    contract_name: String;
    /// name of the station
    name: String;
    /// address of the station. As it is raw data, sometimes it will be more of a comment than an address.
    address: String;
    /// position of the station in WGS84 format
    position: geo;
    /// indicates whether this station has a payment terminal
    banking: bool;
    /// indicates whether this is a bonus station
    bonus: bool;
    /// total number of bike stands in the station
    bike_stands: int;
    /// currently available number of bike stands in the station
    available_bike_stands: int;
    /// currently available number of bikes in the station
    available_bikes: int;
    /// indicates whether this station is `"CLOSED"` or `"OPEN"`
    status: StationStatus;
    /// timestamp indicating the last update time in milliseconds
    last_update: int;
}

We define those two types as @volatile which prevents us from storing instances of those types in the graph. This is because volatile types in GreyCat can change their shape during the database lifecycle.

We also modify the visibility of those type with the private keyword, to minimize discovery of those type throughout the application as those type should really only be used in the importer module.

For a more in depth explanation of GreyCat’s type updates refer to the concepts/updates documentation

Authorization

In order to issue HTTP requests to JCDecaux’s API, we need to create an api key.

Creating an api key is free for signed-in users at JCDecaux Dev.

Once we have our key, we create a new file at the root of the project named .env with the following content:

JCDECAUX_API_KEY=paste-your-api-key-here

Save the file and then open src/importer.gcl, we will add a helper function to get the api key:

/// Tries to load the API key from env var `"JCDECAUX_API_KEY"`
///
/// If the key is not defined, it will throw an error.
fn api_key(): String {
    var key = System::getEnv("JCDECAUX_API_KEY");
    if (key == null) {
        throw "no JCDECAUX_API_KEY found in environment";
    }
    return key;
}

We save the file, and then move to the next step which is actually writing the HTTP logic for each endpoints.

Contracts

Make sure you have an .env file with a valid JCDECAUX_API_KEY for this step, see Authorization

The importer logic to fetch data from the contract API is rather straightforward, and it will turn our app into an actual digital twin.

In src/importer.gcl module, we add a new function, this function is heavily commented to explain each steps:

/// Imports contracts from the JCDecaux API.
///
/// Every non-existent contract will be initialized and set in `model::contracts_by_name`
fn import_contracts() {
    // Read the api key from env, or early exits with an exception
    var api_key = api_key();

    // Will return an array of objects with all the contracts
    //
    // GreyCat's `Http` type is generic over the expected response type
    // if the deserialization fails this call will throw an exception
    var contracts = Http<Array<JsonContract>> {}.get(
        "https://api.jcdecaux.com/vls/v1/contracts?apiKey=${api_key}",
        Map<String, String> { "accept": "application/json" }
    );

    // We then iterate the items, the first argument `_` is the index in the array
    // we do not need it, so we ignore it with this notation
    for (_, item in contracts) {
        // The name works as the identifier of the contract as explained in the JCDecaux API documentation
        // We check if the contract does not exists in the graph
        if (contracts_by_name.get(item.name) == null) {
            // Create the contract typed object using the data from the response
            var contract = Contract {
                name: item.name,
                cities: item.cities,
                commercial_name: item.commercial_name,
                country_code: item.country_code,
                // we initialize an index of stations
                stations: nodeIndex<int, node<Station>> {},
            };

            // We create the contract node
            // This will make the contract persistent in the graph
            // For more information about nodes see: https://doc.greycat.io/concepts/nodes/index.html
            var n_contract = node<Contract> { contract };

            // We add the contract to the index declared in `src/model.gcl`
            contracts_by_name.set(item.name, n_contract);
        }
    }
}

Stations

Make sure you have an .env file with a valid JCDECAUX_API_KEY for this step, see Authorization

The importer logic to fetch data from the stations API is similar to the contract, but because this API contains dynamic data it will be a bit more involved, so let’s open src/importer.gcl and add a new function:


/// Imports stations from the JCDecaux API.
///
/// Stores the stations in their respective `Contract` if it exists, otheriwe ignore the station.
fn import_stations() {
    // Read the api key from env, or early exits with an exception
    var api_key = api_key();

    // Will return an array of objects with all the stations
    //
    // GreyCat's `Http` type is generic over the expected response type
    // if the deserialization fails this call will throw an exception
    var data = Http<Array<JsonStation>> {}.get(
        "https://api.jcdecaux.com/vls/v1/stations?apiKey=${api_key}",
        Map<String, String> { "accept": "application/json" }
    );

    for (_, item in data) {
        // Get the station number, which is unique by contract
        var number = item.number;

        // Get the contract name to get our locally stored contract indexed by contract name
        var contract = contracts_by_name.get(item.contract_name);
        if (contract == null) {
            // If the contract does not exist we skip the station
            // This could happen if since the last time we fetched the contracts a new contract with stations has been added
            // We could refetch the contracts but for the sake of simplicity we will just skip the station
            continue;
        }

        var n_station = contract->stations.get(number);
        if (n_station == null) {
            var quantizer = MultiQuantizer {
                quantizers: Array<Quantizer> {
                    LinearQuantizer<int> { min: 0, max: 6, bins: 7 },
                    LinearQuantizer<int> { min: 0, max: 23, bins: 24 }
                }
            };
            var stationProfile = StationProfile {
                hourlyProfile: GaussianProfile {
                    quantizer: quantizer,
                    precision: FloatPrecision::p10
                },
                hourlyQuantizer: clone(quantizer),
            };

            var station = Station {
                number: number,
                position: item.position,
                detail: node<StationDetail> {
                    StationDetail {
                        name: item.name,
                        address: item.address,
                        banking: item.banking,
                        bonus: item.bonus,
                    }
                },
                records: nodeTime<StationRecord> {},
                contract: contract,
                profile: node<StationProfile> { stationProfile }
            };

            n_station = node<Station> { station };
            stations_by_geo.set(item.position, n_station);

            // We add the station to the contract
            contract->stations.set(number, n_station);
        }

        // Convert the update time in millisecond into GreyCat's `time` representation
        var timestamp = time::new(item.last_update, DurationUnit::milliseconds);

        // We check if we alraedy have the data for this timestamp.
        if (n_station->records.getAt(timestamp) == null) {
            var record = StationRecord {
                status: item.status,
                available_bikes: item.available_bikes,
                available_bike_stands: item.available_bike_stands,
                bike_stands: item.bike_stands,
            };
            n_station->records.setAt(timestamp, record);
            n_station->profile->updateProfile(timestamp, record);
        }
    }
}

We could now call the 2 functions from a terminal, but we will see how to automate those call in the next section:

  • greycat run importer::import_contracts
  • greycat run importer::import_stations

The naming scheme is always module::function, module being the name of our file

Scheduled import

Calling import function periodically is so common that GreyCat provides a scheduler that serves exactly that purpose. It is akin to cron on UNIX systems, but accessible directly from GCL.

Let’s write a new function in project.gcl:

fn schedule_importers() {
    // import contracts every day at midnight
    Scheduler::add(importer::import_contracts, DailyPeriodicity {}, null);
    // import stations every 5 minutes
    Scheduler::add(importer::import_stations, FixedPeriodicity { every: 5min }, null);
}

We update our main() function in project.gcl with the following:

fn main() {
    import_contracts(); // manually import contracts
    import_stations(); // manually import stations
    schedule_importers(); // schedule periodic imports
}

Running greycat install & greycat serve from the terminal will start a greycat instance which will periodically fetch data, as long as the instance is alive

Explorer

We have data, let’s leverage the integrated Explorer app to visualize it.

In you terminal run greycat serve --user=1 to start the server, if you have the previous instance still running make sure to kill it and start a new one with the –user=1 argument this will make it so all request to the server are treated as coming from user with the specified id which in our case is the root user of the app

For a more in depth explanation look here

Home

If you navigate to /explorer/ you should see this:

Explorer Home

This page contains all the information about your currently running greycat server

Graph

If you navigate to the /explorer/graph/ you will see your graph. As you can see the model module contains the two defined graph entrypoints we created.

You can now click on the nodes and explore the graph

Explorer Graph

If you want a more in depth explanation of all the explorer features, they can be found here

API

The main purpose of the @expose decorator is to make your function accessible from the outside, to test this let’s add a new endpoint to fetch a geojson from all our stations.

In backend/api.gcl:

/// Used by the frontend to display station
@volatile
type StationItemView {
    ref: node<Station>;
    coords: geo;
    detail: StationDetail;
    record: StationRecord?;
    last_update: time;
}

/// Used by the frontend to display a station
@volatile
type StationView {
    stations: Array<StationItemView>;
    minTime: time?;
    maxTime: time?;
}

/// `from`: the top right corner of the map
/// `to`: the bottom left corner of the map
@expose
fn getStations(from: geo, to: geo, t: time?): StationView {
    var result = Array<StationItemView> {};

    var min = time::max;
    var max = time::min;
    // GreyCat's geo index supports range queries
    for (coords, station in stations_by_geo[from..to]) {
        var firstTime = station->records.firstTime();
        var lastTime = station->records.lastTime();

        // If time point not specified pick the last available one, if that one is also empty time series has no values, skip.
        if (t == null && lastTime != null) {
            t = lastTime;
        }
        if (t == null) {
            continue;
        }

        // Compute min max time points to limit the time slider in the ui to existing values.
        if (firstTime != null && min > firstTime) {
            min = firstTime;
        }
        if (lastTime != null && max < lastTime) {
            max = lastTime;
        }

        // Will return the last record of the station at the specified time or the closest one before
        var stationData = station->records.resolveAt(t);

        result.add(
            StationItemView {
                ref: station,
                coords: coords,
                detail: station->detail.resolve(),
                record: stationData,
                last_update: t
            }
        );
    }

    return StationView {
        stations: result,
        minTime: min,
        maxTime: max
    };
}

Make sure your server is running if not run greycat serve --user=1

Using cURL we can query our exposed endpoint:

curl \
  -X POST \
  -H 'accept: application/json' \
  -d '[{"lat":49.548572,"lng":5.9963165},{"lat":49.651373,"lng":6.2036834},null]' \
  http://localhost:8080/api::getStations

UI (Frontend)

Let’s build an ui interface to consume our data and visualize it in a more user friendly way

We will be building our Frontend using our javascript SDK that also comes with a bootstrap to get you started

GreyCat comes with a type generator for different languages, we will be using typescript for the output

Running greycat codegen will generate you a ts file containing all types and exposed functions defined in greycat ready to be used

For example, our previous cURL command translates to:

const data = await gc.api.getStations(
  gc.geo.fromLatLng({ lat: 49.548572, lng: 5.9963165 }),
  gc.geo.fromLatLng({ lat: 49.651373, lng: 6.2036834 }),
  null,
);

With one simple command you will have type safe & efficient data fetching (also available for python and java)

We will not be going more into detail for building an front end application since it goes outside the scope of this doc, don’t hesitate to take a look inside the repository linked a the top of this document for more details concerning the UI.