Monitoring
Last updated
Last updated
To access a tg-app instance, use ssh and the key that you used when launching it. You first connect to the instance as user ubuntu. Then you need to switch to user testgrinder as all testgrinder services are run under that user. testgrinder code is installed in the tg-app directory under testgrinder user home.
Substitute my-tg-app with public DNS name of the tg-app server
To get a quick view into health of tg-app, use monit:
You should see output similar to this:
Description of services monitored by Monit:
<hostname> - the first line in Monit Summary is for the system overall, the service name is the tg-app hostname, it will generate an alert if:
an excessive number of active processes are waiting in the queue
memory usage is too high
CPU usage is too high
tg-app-unicorn-worker-N - unicorn (application server) worker processes
tg-app-unicorn - main unicorn (application server) process
tg-app-super - tg-app background process responsible for handling periodic tasks
tg-app-dj-worker-N - tg-app Delayed Job worker processes responsible for handling tg-app asynchronous tasks
nginx - http server
rootfs - root file system; an alert will be generated disk space is low
For any of the processes above, Monit will cycle it if its memroy or CPU usage is too high.
Monit can send emails when it generates an alert or takes a remedial action. To receive these emails, make sure SMTP server is configured, and Monit settings are provided in the .env configuration file as described in Finalize Configuration File .env.
To get detailed Monit report, run:
Find out more about Monit at https://mmonit.com/monit/
These systemd services are at the core of testgrinder:
nginx
/lib/systemd/system/nginx.service
tg-app-provision
/etc/systemd/system/tg-app-provision.service
tg-app provisioning service
tg-app-unicorn
/etc/systemd/system/tg-app-unicorn.service
tg-app application server
tg-app-dj@n
/etc/systemd/system/tg-app-dj@.service
tg-app Delayed Job workers
tg-app-super
/etc/systemd/system/tg-app-super.service
tg-app periodic tasks handler
To see status of a service, use the sudo systemctl status <service> command, for example:
tg-app-unicorn, tg-app-dj@, and tg-app-super services depend on the provisioning service tg-app-provision to complete successfully. They will not start if the provisioning tasks, such as extracting configuration files from User Data, have failed.
Use the sudo journalctl -u <service> command to get log for a service, for example:
You may also use the script/status script to get the status of testgrinder services:
The script will show the output of the monit summary command then after a key press it will output the status of all testgrinder services.
If you setup Papertrail integration as described in Configuration Files, you can see aggregated logs on the Papertrail website.
You may use the start and stop scripts to start and stop testgrinder services:
You may login to running tg-bot instances if you configured the ec2_instance_key_name setting as desribed in Database Stored Settings.
Monitoring tg-bot instances is similar to tg-app. You can use Monit commands to get a view into the health of the instance. tg-app will convey Monit settings to launched tg-bots. So if you have configured getting Monit emails from tg-app, they will also be configured on tg-bots.
The following systemd services are in use on tg-bots:
tg-bot-provision
/etc/systemd/system/tg-bot-provision.service
tg-bot provisioning service
tg-bot
/etc/systemd/system/tg-bot.service
tg-bot main process
If you enabled Papertrail integration for tg-app, it will also be enabled for tg-bot instances.
http server
