Monitoring

testgrinder service will be discontinued on January 31, 2025

testgrinder SMPC was discontinued as of February 1, 2024

tg-app

Access tg-app instance

To access a tg-app instance, use ssh and the key that you used when launching it. You first connect to the instance as user ubuntu. Then you need to switch to user testgrinder as all testgrinder services are run under that user. testgrinder code is installed in the tg-app directory under testgrinder user home.

Substitute my-tg-app with public DNS name of the tg-app server

$ ssh ubuntu@my-tg-app
$ sudo su - testgrinder
$ cd ~/tg-app

Monit

To get a quick view into health of tg-app, use monit:

$ sudo monit summary

You should see output similar to this:

Description of services monitored by Monit:

  • <hostname> - the first line in Monit Summary is for the system overall, the service name is the tg-app hostname, it will generate an alert if:

    • an excessive number of active processes are waiting in the queue

    • memory usage is too high

    • CPU usage is too high

  • tg-app-unicorn-worker-N - unicorn (application server) worker processes

  • tg-app-unicorn - main unicorn (application server) process

  • tg-app-super - tg-app background process responsible for handling periodic tasks

  • tg-app-dj-worker-N - tg-app Delayed Job worker processes responsible for handling tg-app asynchronous tasks

  • nginx - http server

  • rootfs - root file system; an alert will be generated disk space is low

For any of the processes above, Monit will cycle it if its memroy or CPU usage is too high.

Monit can send emails when it generates an alert or takes a remedial action. To receive these emails, make sure SMTP server is configured, and Monit settings are provided in the .env configuration file as described in Finalize Configuration File .env.

To get detailed Monit report, run:

$ sudo monit status

Find out more about Monit at https://mmonit.com/monit/

systemd services

These systemd services are at the core of testgrinder:

systemd ServiceUnit FileDescription

nginx

/lib/systemd/system/nginx.service

tg-app-provision

/etc/systemd/system/tg-app-provision.service

tg-app provisioning service

tg-app-unicorn

/etc/systemd/system/tg-app-unicorn.service

tg-app application server

tg-app-dj@n

/etc/systemd/system/tg-app-dj@.service

tg-app Delayed Job workers

tg-app-super

/etc/systemd/system/tg-app-super.service

tg-app periodic tasks handler

To see status of a service, use the sudo systemctl status <service> command, for example:

$ sudo systemctl status tg-app-provision

tg-app-unicorn, tg-app-dj@, and tg-app-super services depend on the provisioning service tg-app-provision to complete successfully. They will not start if the provisioning tasks, such as extracting configuration files from User Data, have failed.

Use the sudo journalctl -u <service> command to get log for a service, for example:

$ sudo journalctl -u tg-app-provision

You may also use the script/status script to get the status of testgrinder services:

$ ssh ubuntu@my-tg-app
$ sudo su - testgrinder
$ cd tg-app
$ sudo script/status

The script will show the output of the monit summary command then after a key press it will output the status of all testgrinder services.

If you setup Papertrail integration as described in Configuration Files, you can see aggregated logs on the Papertrail website.

Starting, stopping, and restarting testgrinder services

You may use the start and stop scripts to start and stop testgrinder services:

$ ssh ubuntu@my-tg-app
$ sudo su - testgrinder
$ cd tg-app
$ sudo script/stop
$ sudo script/start
$ sudo script/restart

tg-bot

Access tg-bot instances

You may login to running tg-bot instances if you configured the ec2_instance_key_name setting as desribed in Database Stored Settings.

Monit

Monitoring tg-bot instances is similar to tg-app. You can use Monit commands to get a view into the health of the instance. tg-app will convey Monit settings to launched tg-bots. So if you have configured getting Monit emails from tg-app, they will also be configured on tg-bots.

systemd Services

The following systemd services are in use on tg-bots:

systemd serviceUnit FileDescription

tg-bot-provision

/etc/systemd/system/tg-bot-provision.service

tg-bot provisioning service

tg-bot

/etc/systemd/system/tg-bot.service

tg-bot main process

If you enabled Papertrail integration for tg-app, it will also be enabled for tg-bot instances.

Last updated