Observability

GoFr by default manages observability in different ways once the server starts:

Logs

Logs offer real-time information, providing valuable insights and immediate visibility into the ongoing state and activities of the system. It helps in identifying errors, debugging and troubleshooting, monitor performance, analyzing application usage, communications etc.

GoFr logger allows customizing log level which provides flexibility to adjust logs based on specific needs.

Logs are generated only for events equal to or above the specified log level, by default GoFr logs at INFO level. Log Level can be changed by setting the environment variable LOG_LEVEL value to WARN,DEBUG,ERROR,NOTICE or FATAL.

When GoFr server runs, it prints log for reading configs, database connection, requests, database queries, missing configs etc. They contain information such as request's correlation ID, status codes, request time etc.

Logs are well-structured, they are of type JSON when exported to a file, such that they can be pushed to logging systems such as Loki, Elasticsearch etc.

Metrics

Metrics enable performance monitoring by providing insights into response times, latency, throughput, resource utilization, tracking CPU, memory, and disk I/O consumption across services, facilitating capacity planning and scalability efforts.

Metrics play a pivotal role in fault detection and troubleshooting, offering visibility into system behavior.

They are instrumental in measuring and meeting service-level agreements (SLAs) to ensure expected performance and reliability.

GoFr publishes metrics to port: 2121 on /metrics endpoint in Prometheus format.

Name	Type	Description
app_go_numGC	gauge	Number of completed Garbage Collector cycles
app_go_routines	gauge	Number of Go routines running
app_go_sys	gauge	Number of total bytes of memory
app_sys_memory_alloc	gauge	Number of bytes allocated for heap objects
app_sys_total_alloc	gauge	Number of cumulative bytes allocated for heap objects
app_info	gauge	Number of instances running with info of app and framework
app_http_response	histogram	Response time of HTTP requests in seconds
app_http_service_response	histogram	Response time of HTTP service requests in seconds
app_sql_open_connections	gauge	Number of open SQL connections
app_sql_inUse_connections	gauge	Number of inUse SQL connections
app_sql_stats	histogram	Response time of SQL queries in milliseconds
app_redis_stats	histogram	Response time of Redis commands in milliseconds
app_pubsub_publish_total_count	counter	Number of total publish operations
app_pubsub_publish_success_count	counter	Number of successful publish operations
app_pubsub_subscribe_total_count	counter	Number of total subscribe operations
app_pubsub_subscribe_success_count	counter	Number of successful subscribe operations

For example: When running application locally, you can access /metrics endpoint on port 2121 from: http://localhost:2121/metrics

GoFr also supports creating custom metrics.

Tracing

Tracing is a powerful tool for gaining insights into your application's behavior, identifying bottlenecks, and improving system performance. A trace is a tree of spans. It is a collective of observable signals showing the path of work through a system. A trace on its own is distinguishable by a TraceID.

In complex distributed systems, understanding how requests flow through the system is crucial for troubleshooting performance issues and identifying bottlenecks. Traditional logging approaches often fall short, providing limited visibility into the intricate interactions between components.

Automated Tracing in GoFr

GoFr automatically exports traces for all requests and responses. GoFr uses OpenTelemetry , a popular tracing framework, to automatically add traces to all requests and responses.

Automatic Correlation ID Propagation:

When a request enters your GoFr application, GoFr automatically generates a correlation-ID X-Correlation-ID and adds it to the response headers. This correlation ID is then propagated to all downstream requests. This means that you can track a request as it travels through your distributed system by simply looking at the correlation ID in the request headers.

Configuration & Usage:

GoFr has support for following trace-exporters:

1. Zipkin:

To see the traces install zipkin image using the following Docker command:

  docker run --name gofr-zipkin -p 2005:9411 -d openzipkin/zipkin:latest

Add Tracer configs in .env file, your .env will be updated to

APP_NAME=test-service
HTTP_PORT=9000

REDIS_HOST=localhost
REDIS_PORT=6379

DB_HOST=localhost
DB_USER=root
DB_PASSWORD=root123
DB_NAME=test_db
DB_PORT=3306

# tracing configs
TRACE_EXPORTER=zipkin
TRACER_URL=http://localhost:2005/api/v2/spans
TRACER_RATIO=0.1

LOG_LEVEL=DEBUG

NOTE: If the value of TRACER_PORT is not provided, GoFr uses port 9411 by default.

Open zipkin and search by TraceID (correlationID) to see the trace.

2. Jaeger:

To see the traces install jaeger image using the following Docker command:

docker run -d --name jaeger \
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 16686:16686 \
  -p 14317:4317 \
  -p 14318:4318 \
  jaegertracing/all-in-one:1.41

Add Jaeger Tracer configs in .env file, your .env will be updated to

# ... no change in other env variables

# tracing configs
TRACE_EXPORTER=jaeger
TRACER_URL=localhost:14317
TRACER_RATIO=0.1

Open jaeger and search by TraceID (correlationID) to see the trace.

3. OpenTelemetry Protocol:

The OpenTelemetry Protocol (OTLP) underlying gRPC is one of general-purpose telemetry data delivery protocol designed in the scope of the OpenTelemetry project.

Add OTLP configs in .env file, your .env will be updated to

# ... no change in other env variables

# tracing configs 
TRACE_EXPORTER=otlp
TRACER_URL=localhost:4317
TRACER_RATIO=0.1

4. GoFr Tracer:

GoFr tracer is GoFr's own custom trace exporter as well as collector. You can search a trace by its TraceID (correlationID) in GoFr's own tracer service available anywhere, anytime.

Add GoFr Tracer configs in .env file, your .env will be updated to

# ... no change in other env variables

# tracing configs
TRACE_EXPORTER=gofr
TRACER_RATIO=0.1

NOTE: TRACER_RATIO refers to the proportion of traces that are exported through sampling. It ranges between 0 and 1. By default, this ratio is set to 1, meaning all traces are exported.

Open gofr-tracer and search by TraceID (correlationID) to see the trace.