Metrics

Available since GreyCat 8.4

A running GreyCat server can record what it does as metrics: how long each task took and what it cost, how the host and its workers are doing, and every HTTP request it answered. They let you find slow endpoints, size --cache and --workers, spot a saturated worker pool, or keep an access log, without any external agent.

There are three metrics, each turned on separately:

Metric	Record	Written	File
task	TaskMetric	once per task, when it ends	files/root/streams/task_metrics.csv
host	HostMetric	every --metrics_host_step (60s)	files/root/streams/host_metrics.csv
http	HttpMetric	once per HTTP request	files/root/streams/http_metrics.csv

Metrics are not log records. They have no level, they are never printed, and --log does not turn them on.

Enabling metrics

Metrics are off by default: nothing is recorded and nothing is built. --metrics (or GREYCAT_METRICS) takes a comma-separated list of the metrics to record:

greycat serve --metrics=task,host,http

or in the .env file of the project:

GREYCAT_METRICS=task,host,http

The option applies to greycat serve and greycat run. An unknown name is refused at startup.

Configuration

Option	Environment variable	Default	Description
--metrics	GREYCAT_METRICS	none	The metrics to record: task, host, http
--metrics_host_step	GREYCAT_METRICS_HOST_STEP	60s	Period of the HostMetric record
--metrics_http_fields	GREYCAT_METRICS_HTTP_FIELDS	none	Personal data an HttpMetric records: ip, forwarded_for, user_agent
--metrics_task_flush_size	GREYCAT_METRICS_TASK_FLUSH_SIZE	64KB	Buffered bytes that trigger a write of task_metrics.csv
--metrics_task_flush_step	GREYCAT_METRICS_TASK_FLUSH_STEP	1s	Longest time a TaskMetric stays buffered
--metrics_host_flush_size	GREYCAT_METRICS_HOST_FLUSH_SIZE	64KB	Buffered bytes that trigger a write of host_metrics.csv
--metrics_host_flush_step	GREYCAT_METRICS_HOST_FLUSH_STEP	1s	Longest time a HostMetric stays buffered
--metrics_http_flush_size	GREYCAT_METRICS_HTTP_FLUSH_SIZE	64KB	Buffered bytes that trigger a write of http_metrics.csv
--metrics_http_flush_step	GREYCAT_METRICS_HTTP_FLUSH_STEP	1s	Longest time an HttpMetric stays buffered

A flush size of 0 writes each record at once. See Storage and buffering for what the two flush settings mean for durability.

Task metrics

With task enabled, every task writes one TaskMetric once it ended and its transaction committed. That covers every RPC call (each one runs as a task since 8.4), calls with a task header, scheduled tasks, greycat run, and each await job.

Identifying a task

Field	Description
time	When the record was written, right after the task ended
src	The function the task ran, the key for per-endpoint figures
user_id	The caller
task_id	The task id, shared by a task and its await jobs
job_id	null for the task itself, the index of the job in its await for an await job
world_id	The world the task ran in, 0 for main (see Many Worlds)
status	How it ended: ended, error (a --request_ttl expiry included) or cancelled

Durations are in microseconds and sizes in bytes. A task’s figures cover all of its runs (a task suspended in an await runs again after it), but not its jobs, which write their own record. Add up the records sharing a task_id to get the total work of an await tree.

Timings

Field	Description
wait	Time queued before a worker first picked the task up
exec	From then to the end, so exec = run + suspended
run	Time actually spent on a worker
suspended	Time parked in await
commit	Time spent committing, part of run

The latency seen by a caller is wait + exec.

Scheduling

Field	Description
task_class	The worker class it ran in: small, medium or large
queued	Times it waited in a queue instead of going straight to an idle worker
borrowed	Runs taken by a worker of a heavier class, because its own class had none idle
awaits	await suspensions
jobs	Jobs spawned by those awaits
fusion_conflicts	Jobs whose transaction could not be merged
catches	Exceptions that reached a catch

Memory and store

Field	Description
cache_blocks, memory, cache_budget	The object cache of the worker when the task ended, against its budget
cache_misses	Blocks the object cache had to load
cache_evictions	Blocks evicted to stay within budget
dirty_blocks	Distinct blocks in the committed transaction, the size of the commit
dirty_evictions	Modified blocks written out before the commit to stay within budget
cache_bytes, cache_hits	Blocks served from the binary cache of the zones instead of disk
read_bytes, read_hits, read_wasted	Reads from the store zones
write_bytes, write_hits	Writes to the store zones (write_hits counts batched writes, not blocks)
args_bytes, result_bytes	Size of the arguments and of the result

Reading the signals

• Saturated workers. A high wait with queued above 0 means the pool of the task’s class is busy. With borrowed above 0, a heavier class picked the task up.
• Not enough memory. A dirty_evictions or cache_evictions above 0 means the cache budget of the worker is too small for the task. Compare memory with cache_budget to see how close a task came to it, then raise --cache or the ratio of its class (--cache_ratio_small, --cache_ratio_large).
• Read amplification. read_wasted / read_bytes is the share of what was read from disk and not used.
• Exceptions on a hot path. Each throw that reaches a catch builds an Error and unwinds. A high catches on a busy endpoint costs time.

Cost

The counters are only updated off the hot paths (a store read or write, a cache miss, a caught exception, a suspension). Turning task on costs no measurable time, but it costs volume: a record is about 550 bytes, which adds up to megabytes per second at a high RPC rate.

Host metrics

With host enabled, the server writes one HostMetric every --metrics_host_step (60s by default). It describes the process and the machine. Counters cover the period since the previous record, the other fields are values at the time of the record.

Field	Description
time, period	When the sample was taken, and the time since the previous one
cores, load	Cores of the host and its one-minute load average
cpu_user, cpu_system	CPU time used by the process during period
os_memory_total, os_memory_used	Memory of the host
process_resident, process_virtual, process_shared	Memory of the process
malloc_total, memory_drift	Memory allocated by the runtime, and what it does not account for
io_read, io_write	Storage bytes the OS accounted to the process
store_read, store_write	Bytes the workers moved from and to the store zones
disk_free, disk_meta	Free space where gcdata/ lives, and the size of the metadata database
tasks_live, http_connections, sse_subscribers	Task slots in use, open connections, open event streams
http_bytes_in, http_bytes_out	Bytes read from and written to HTTP clients
files_served, files_pushed	Files sent in full (downloads, webroot assets) and received in full (uploads)
small, medium, large	One HostMetricClass per worker class, see below
zones	A HostMetricZones summary of the store, see below

Divide cpu_user + cpu_system by period and cores to get the share of the machine the process used.

Each HostMetricClass describes one worker class:

Field	Description
workers	Workers of the class
busy	Share of the time of the class spent running tasks, 0 to 1
queued, idle	Tasks waiting and idle workers at the time of the record
ended, errors, cancelled, timeouts	Tasks that ended during period, by outcome
memory, cache_budget	Object cache memory of the class, against its budget

HostMetricZones summarizes the store zones: count and used zones, their total size, committed_blocks, reserved_blocks and written_blocks (written_blocks / committed_blocks is the fragmentation), the binary cache bin_cache, the most fragmented zone (worst_zone, worst_ratio), and whether a defrag is selected.

A record is about 300 bytes, so about 450 KB a day at the default step. Taking a sample costs well under a millisecond.

HTTP metrics

With http enabled, the server writes one HttpMetric per HTTP request once its response ended. It is the access log of the server: file downloads and uploads, webroot assets, RPC calls, task calls, event streams and refused requests alike.

Field	Description
time	When the response ended
user_id	The authenticated user, 0 when anonymous
method, path	The request line. path has no query string, which could carry a token or arguments
kind	file, upload, rpc, task (a call with the task header), sse or other
status	The HTTP status code, 0 when the connection ended before a response
ttfb	From the first byte of the request to the first byte of the response
duration	From the first byte of the request to the last byte of the response
request_bytes, response_bytes	Bytes on the wire, headers included
task_id	The task that ran an rpc or task call, to join with its TaskMetric
aborted	The exchange was cut short: the client left, an upload stopped, or --request_ttl ran out
ip, forwarded_for, user_agent	The client address, the X-Forwarded-For header and the User-Agent header

duration - ttfb is the transfer time, which the bandwidth of the client governs on a large download. An sse event stream writes one record for its whole lifetime.

Personal data

ip, forwarded_for and user_agent are personal data, so they are null unless named in --metrics_http_fields:

GREYCAT_METRICS=http
GREYCAT_METRICS_HTTP_FIELDS=ip,user_agent

Behind a reverse proxy, ip is the address of the proxy, and forwarded_for holds the address the proxy reports (it is never checked). Note that path is always recorded, and a path under /files/<user>/ names a user.

Cost

A record is about 100 bytes plus its path. Building it costs about 3 to 5% of the requests per second on the smallest file downloads (the worst case), and nothing measurable on RPC calls. With --metrics_http_flush_size=0, the smallest downloads lose about 10%.

Reading metrics

Each record is one CSV line, without a header. A field of a nested type, such as HostMetric.small, is spread over one column per field, in the order the fields are declared in runtime.gcl.

From GCL

The metric files are streams owned by root, so a function running as root reads them back as typed records with Stream<T>::get(name).reader(from):

/// Calls, total latency and memory-starved calls per function, from the task metrics.
@expose
@permission("admin")
fn task_summary(): Map<String, Array<int>> {
  var by_fn = Map<String, Array<int>> {};
  var reader = Stream<TaskMetric>::get("task_metrics").reader(0);
  while (reader.can_read()) {
    var m = reader.read();
    if (m.job_id == null) {
      var key = "${m.src}";
      var agg = by_fn.get(key) ?? [0, 0, 0];
      agg[0] = agg[0] + 1;
      agg[1] = agg[1] + (m.wait + m.exec).to(DurationUnit::microseconds);
      if (m.dirty_evictions > 0 || m.cache_evictions > 0) {
        agg[2] = agg[2] + 1;
      }
      by_fn.set(key, agg);
    }
  }
  return by_fn;
}

/// Requests per status, method and path, from the HTTP metrics.
@expose
@permission("admin")
fn http_summary(): Map<String, int> {
  var res = Map<String, int> {};
  var reader = Stream<HttpMetric>::get("http_metrics").reader(0);
  while (reader.can_read()) {
    var r = reader.read();
    var method = "-";
    var m = r.method;
    if (m != null) {
      method = type::enum_name(m);
    }
    var key = "${r.status} ${method} ${r.path}";
    res.set(key, (res.get(key) ?? 0) + 1);
  }
  return res;
}

/// The latest host sample, or null when none was written yet.
@expose
@permission("admin")
fn last_host_metric(): HostMetric? {
  var last: HostMetric?;
  var reader = Stream<HostMetric>::get("host_metrics").reader(0);
  while (reader.can_read()) {
    last = reader.read();
  }
  return last;
}

A metric stream only exists while its metric is enabled: with host off, Stream<HostMetric>::get("host_metrics") throws.

These examples read the whole file on each call. For a dashboard that runs often, keep the offset reached by the last read in the graph and pass it to reader(from), together with the figures aggregated so far.

Over HTTP

The metric files are served by the /files API. Root, an admin, or an identity granted read access to the folder of root (greycat user grant root r <reader>) can download them:

curl -H "Authorization: $TOKEN" http://localhost:8080/files/root/streams/task_metrics.csv

A collector follows a file by polling it with a Range header, starting at offset 0 and adding the size of the complete lines it received:

curl -H "Authorization: $TOKEN" -H "Range: bytes=$OFFSET-" \
  http://localhost:8080/files/root/streams/http_metrics.csv

• 206 Partial Content carries the bytes written since OFFSET. A last line without its \n is still being written, and the next poll fetches it again.
• 416 Range Not Satisfiable means nothing is new. Content-Range: bytes */<size> gives the size of the file.
• A size below OFFSET means the file lost bytes after a machine crash. Resume from the new size.

Storage and buffering

Each metric is written to a Stream, the append-only record file introduced in 8.4, owned by root. The runtime log uses the same mechanism (files/root/streams/log.ndjson, read with Stream<Log>::get("log")).

Records are buffered in memory and reach their file in batches:

• once the flush size of the metric is pending (64KB by default),
• at the latest the flush step after the first pending record (1s by default),
• when the file is read through /files (a GET or HEAD), so an HTTP reader always gets every record written so far,
• when a GCL reader is opened on the stream,
• when the process stops cleanly.

A killed or crashed process loses the records still in the buffer. Lower the flush step, or set the flush size to 0 to write each record at once, when losing the last second of metrics matters more than the extra writes.

The server never rotates these files. On a long-running server, plan the disk space they take, keeping in mind that http and task grow with the traffic.