Bike Rental Service Digital Twin

Before reading this, make sure you have at least read trough the concepts.

To build our first digital twin with greycat we will be using JCDecaux’s Open Data.

Their data contains the geographic location of renting bikes world wide as well as their availability frequently updated.

This is the perfect use case for leveraging the full potential of GreyCat.

We will periodically fetch the data from their API, store it in our graph database, do some simple analytics, without ever leaving our GreyCat ecosystem and relying on third party libraries.

Project Repository link

Setup

Start by installing greycat as explained here

Create your app directory we will name ours biking, and create a project.gcl file at it’s root

Project structure

Let’s start by creating a simple file structure to host all our logic in an organized way. You can of course use your own preferred structure

project.gcl // mandatory, program entry point
importer.gcl
model.gcl
api.gcl

We put all our server logic in a backend directory to be able to separate it from our frontend code

• api will contain all our endpoints logic to fetch data from our graph
• importer will contain our data fetching logic from JCDecaux
• model will be used to store our graphs entry points as well as the data structures(types)

Data Model

Our Graphs entrypoint, will be declared at the top of our biking.gcl file inside model directory, this will contain all our applications data structures For a more in depth explanation Nodes

// file: model.gcl

// will contain all stations indexed by coordinates
// We will leverage this to build a geojson for display purposes
var stations_by_geo: nodeGeo<node<Station>>;

// will contain all contracts indexed by contract name
var contracts_by_name: nodeIndex<String, node<Contract>>;

The types defined on JCDecaux are not up to date, reference the types defined in the demo repository

We separated the Station into two types, to avoid storing redundant data, Station has static data and StationData has dynamic data, so we can store them in a nodeTime for efficient and fast sampling.

// file: model.gcl

type Station {
    //id
    number: int;
    position: geo;

    // will contain all static information about the Station
    // you will want to put information that is not important inside a node to avoid loading it into memory on every Station resolve
    detail: node<StationDetail>;

    // will contain all dynamic data of a station indexed by the last update timestamp
    records: nodeTime<StationRecord>;

    // The stations Contract,  Belongs To relationship in sql
    contract: node<Contract>;

    // Will be used to store various statistics about the Station, this object could have been stored as a plain object since it will be unique by station,
    // but every time we load a Station we will also load the whole profile object into memory it could potentially
    // contain a lot of data, storing it as a node will only load the reference every time we resolve the Station
    profile: node<StationProfile>;

}

type StationDetail {
    name: String;
    address: String;
    banking: bool;
    bonus: bool;
}

type StationRecord {
    status: StationStatus;
    bikeStands: int;
    availableBikeStands: int;
    availableBikes: int;
}

enum StationStatus {
    open("OPEN");
    closed("CLOSED");
}

type StationProfile {
    // 24 * 7 slots, one for each hour of the week, will help to visualize the station activity in a typical week
    hourlyProfile: GaussianProfile;
    hourlyQuantizer: MultiQuantizer;

    fn updateProfile(t: time, data: StationData) {
         if (data.bike_stands > 0) {
            //convert the time to the correct slot
            var date = t.toDate(null);
            var slot = this.hourlyProfile.quantizer.quantize(Array<int> {t.dayOfWeek(null), date.hour});
            this.hourlyProfile.add(this.hourlyQuantizer.slot_vector(slot), data.available_bikes as float / data.bike_stands as float);
        }
    }

    fn profileToTable(): Table {
        var table = Table {};

        var endSlot = 24 * 7;
        var startSlot = 0;

        while (startSlot < endSlot) {
            var row = startSlot / 24;
            var col = startSlot % 24;

            var val = this.hourlyProfile.avg(Array<int> {row, col});

            table.set_cell(row, col, val);
            startSlot++;
        }
        return table;
    }
}

type Contract {
    name: String;
    commercialName: String?;
    countryCode: String?;
    cities: Array<String>?;

    // will contain a list of refs to all stations of the contract indexed by the number property, Has Many relationship in sql
    stations: nodeIndex<int, node<Station>>;
}

Importer

Second step will be to write the importer logic to periodically fetch data from their api, this will turn our app into an actual digital twin.

Since greycat currently doesn’t support custom env variables we will be adding our api key to a txt file, here called api_key.txt and reading it from there

// file: importer.gcl

var jcdeaux_key: node<String?>;

fn getKey(): String {
    if (*jcdeaux_key == null) {
        var tr = TextReader{path:"api_key.txt"};
        jcdeaux_key.set(tr.readLine());
    }

    return *jcdeaux_key!!;
}

// file: importer.gcl

fn importContracts(){
    var apiKey = jcdeaux_key.resolve() ?? getKey();

    var headers = Array<HttpHeader> {HttpHeader { name: "accept", value: "application/json" }};
    var data = Http::get( "https://api.jcdecaux.com/vls/v1/contracts?apiKey=${apiKey}"  , headers);

    // Since in greycat all json objects are converted to a Map we can explicitly specify it in the for loop
    for (_, item: Map in data?) {
        // The name works as the identifier of the contract as explained in the JCDecaux API documentation
        var name = item.get("name") as String;
        // We check if the contract does not exists in the graph
        if (contracts.get(name) == null) {
            //Create the contract typed object
            var contract = Contract {
                name: name,
                cities: item.get("cities") as Array<String>,
                commercialName: item.get("commercial_name") as String,
                countryCode: item.get("country_code") as String,
                stations: nodeIndex<int, node<Station>>{},
            };

            // We create the contract node
            // This will make the contract persistent in the graph, for more information about **nodes** check the documentation
            var contractRef = node<Contract>{contract};

            // We add the contract to the index declared at top the biking module
            contracts_by_name.set(name, contractRef);
        }
    }
}

// file: importer.gcl

fn importStations() {
    var apiKey = jcdeaux_key.resolve() ?? getKey();

    var headers = Array<HttpHeader> {HttpHeader { name: "accept", value: "application/json" }};
    var data = Http::get("https://api.jcdecaux.com/vls/v1/stations?apiKey=${apiKey}", headers);

    for (_, item: Map in data?) {
        // Get the station number, which is unique by contract
        var number = item.get("number") as int;

        //Get the contract name to get our locally stored contract indexed by contract name
        var contractName = item.get("contract_name") as String;
        var contract = contracts.get(contractName);

        if (contract == null) {
            // If the contract does not exist we skip the station, this could happen if since the last time we fetched
            // the contracts a new contract with stations has been added,
            // We could refetch the contracts but for the sake of simplicity we will skip the station
            continue;
        }

        // Convert the position to a greycat geo object
        // here again we tell the lsp it's a Map since all json objects are converted to a Map
        var coordsMap = item.get("position") as Map;
        var coords = geo {coordsMap.get("lat") as float, coordsMap.get("lng") as float};

        // We see if our contract contains the station
        var stationRef = contract->stations.get(number);

        // If not create it
        if (stationRef == null) {
            var quantizer = MultiQuantizer<int> {
                quantizers: Array<LinearQuantizer<int>> {
                    LinearQuantizer<int> {min: 0, max: 6, bins: 7 },
                    LinearQuantizer<int> {min: 0, max: 23, bins: 24 }
                }
            };
            var stationProfile = StationProfile {
                hourlyProfile: GaussianProfile { quantizer: quantizer, precision: FloatPrecision::p10 },
                hourlyQuantizer: quantizer
            };

            var detail = StationDetail {
                name: item.get("name") as String,
                address: item.get("address") as String,
                banking: item.get("banking") as bool,
                bonus: item.get("bonus") as bool,
            };

            var station = Station {
                number: number,
                position: coords,
                detail: node<StationDetail>{detail},
                records: nodeTime<StationRecords>{},
                contract: contract,
                profile: node<StationProfile>{stationProfile}
            };

            // Make the object persistent in the graph
            stationRef = node<Station>{station};

            stations_by_geo.set(coords, stationRef);

            // We add the station to the contract
            contract->stations.set(number, stationRef);
        }

        // timestamp of the last update
        var lastUpdate = item.get("last_update") as int;
        // convert the timestamp to a greycat time, by also specifying the time unit, in this case it's in milliseconds
        var t = time::new(lastUpdate, DurationUnit::milliseconds);

        // We check if we already have the data for this timestamp.
        if (stationRef->data.getAt(t) == null) {
            var record = StationRecords {
                //a mapping function to convert a string into an enum
                status: stationStatusImportMapper(item.get("status") as String),
                available_bikes: item.get("available_bikes") as int,
                available_bike_stands: item.get("available_bike_stands") as int,
                bike_stands: item.get("bike_stands") as int,
            };
            stationRef->records.setAt(t, record);
            stationRef->profile->updateProfile(t, record);
        }
    }
}

You can now call from the terminal your functions greycat run importer::importContracts && greycat run importer::importStations

The naming scheme is always module::function, module being the name of our file

Which will run the tasks as specified and populate your graph with data

Periodic data fetching

Let’s define our function which will handle the periodic data fetching setup

// file: project.gcl

fn startImportingScheduler() {
    var periodicStationTask = PeriodicTask {
        arguments: null,
        every: 10_min,
        user_id: 1,
        function: importer::importStations,
        start: time::now()
    };

    var periodicContractTask = PeriodicTask {
        arguments: null,
        every: 1_day,
        user_id: 1,
        function: importer::importContracts,
        start: time::now()
    };

    PeriodicTask::set(Array<PeriodicTask> {periodicContractTask, periodicStationTask});
}

We need to add it to our projects main function inside project.gcl.

// file: project.gcl

// Libraries
@library("std", "7.0.1692-testing");
@library("explorer", "7.0.4-testing");

//Your apps entry point
fn main() {
    startImportingScheduler();
}

Running greycat install & bin/greycat serve from the terminal will start a greycat instance which will periodically fetch data, as long as the instance is alive

Explorer

We have data, let’s leverage the integrated explorer app to visualize it.

In you terminal run greycat serve --user=1 to start the server, if you have the previous instance still running make sure to kill it and start a new one with the –user=1 argument this will make it so all request to the server are treated as coming from user with the specified id which in our case is the root user of the app

For a more in depth explanation look here

If you navigate to localhost:8080/explorer you should see this

This page will contain all the information about your currently running greycat server

If you navigate to the Graph page you will see your graph. As you can see the biking module contains the two defined graph entrypoint we did before You can now click on the nodes and traverse the graph

If you want a more in depth explanation of all the explorer features, they can be found here

API

The main purpose of the @expose decorator is to make your function accessible from the outside, to test this let’s add a new endpoint to fetch a geojson from all our stations

// directory: api.gcl

// Object to be used by the ui
@volatile
type StationView {
    ref: node<Station>;
    coords: geo;
    detail: StationDetail;
    record: StationRecord?;
    last_update: time;
}

// From: the top right corner of the map
// To: the bottom left corner of the map
@expose
fn getStations(from: geo, to: geo, t: time?): Array<StationView> {
    var result: Array<StationView> = [];

    // greycat's geo index supports range queries
    for (coords, station in stations_by_geo[from..to]) {
        if (t == null) {
            t = station->records.lastTime()!!;
        }

        //will return the last record of the station at the specified time or the closest one before
        var stationRecord = station->records.resolveAt(t);

        result.add(StationView {
                ref: station,
                coords: coords,
                detail: station->detail.resolve(),
                record: stationRecord,
                last_update: t
            });
    }

    return result;
}

Make sure your server is running if not run greycat serve --user=1 Using a simple curl request should return you now a json as response containing the specified structure curl -X POST http://localhost:8080/stations::getStations

UI (Frontend)

Let’s build an ui interface to consume our data and visualize it in a more user friendly way

We will be building our Frontend using our javascript SDK that also comes with a bootstrap to get you started

GreyCat comes with a type generator for different languages, we will be using typescript for the output

Running greycat codegen will generate you a ts file containing all types and exposed functions defined in greycat ready to be used

For example, this will return you a typed js object, using our own rpc binary protocol.

const data = await api.getStations();

With one simple command you will have type safe & efficient data fetching,(also available for python and java)

We will not be going more into detail for building an front end application since it goes outside the scope of this doc, don’t hesitate to take a look inside the repository linked a the top of this document for more details concerning the UI.