Generated 2026-08-20 14:50 UTC
| Status | Source | Check | Summary & detail | Runbook |
|---|---|---|---|---|
| FAIL | darvaza | darvaza_source_pod_count | darvaza-source pod count mismatch: active(log)=0 running(k8s)=8 | This check compares the number of darvaza-source pods that are 'active' in logs to the number of pods k8s reports running.
Sometimes the clients hang and the logs stop. This can be fixed by doing a
|
| FAIL | darvaza | darvaza_prod_upload_requests | 0 'New upload metadata request' in 12h (min 100) | This check counts the number of 'New upload metadata request' entries in the darvaza-prod logs over the last 24 hours. These are generated when darvaza is told by ET about a new upload. If there are fewer than 100 entries, it indicates we are not receiving data. Problems with this depend on ET talking to darvaza directly. It could be network or problems with darvaza itself. |
| FAIL | darvaza | darvaza_prod_confirming_upload | 0 'Confirming upload' in 24h; latest 23.3h ago 0 entries in 24h, need >= 100; latest entry 23.3h old, max 6h | This check counts the number of 'Confirming upload' entries in the darvaza-prod logs over the last 24 hours, and checks that the most recent entry is within 6 hours of now (UTC).
These are generated when darvaza's pulsar notification processors are confirming that an upload has been completed.
If there are fewer than 100 entries, it indicates we are not receiving data. If the darvaza_prod_upload_requests check failed, that's the reason this one failed.
It could also be that we aren't receiving notifications from pulsar. Check that pulsar has new notifications ( |
| FAIL | darvaza | darvaza_source_success | 2 'Success=true' in 24h (min 100) | This check counts the number of 'Success=true' entries in the darvaza-source logs over the last 24 hours. These are generated when darvaza-source successfully processes a notification from pulsar and that file was decrypted, archived, and sent to dovetail. Check the logs in loki using the {'namespace="dovetail", cluster="rc-one", service_name="darvaza-source"} |= "Success=true"'} query to see if there are any errors. (Jig Failures dashboard https://it-grafana.cloud.vtti.vt.edu/goto/dfpnt2u1bi39cd?orgId=1) |
| FAIL | dovetail | dovetail_deployments_ready | 22/25 jig deployments healthy daisy-face-obscure-deepface: 58/60 ready transcode-video: deployment not found video-face-deid: deployment not found | If any jig deployment has 0 replicas or ready != desired, the jig is not running properly. Check the logs of the deployment's pods to see why they are failing.
Go to https://it-grafana.cloud.vtti.vt.edu click explore and search |
| FAIL | dovetail | dovetail_backlog_draining | 3 jig(s) backed up with no processing daisy-lane-construction: backlog 1, 0 msg/min processed rlog-parser: backlog 1, 0 msg/min processed transcode-video: backlog 1, 0 msg/min processed | If any jig has a backlog > 0 and a processing rate of 0, it is stuck. Check the logs of the deployment's pods to see why they are failing.
Go to https://it-grafana.cloud.vtti.vt.edu click explore and search |
| FAIL | dovetail | dovetail_failure_rates | 3 service(s) over failure-rate threshold daisy-face-obscure-deepface: 5.2% failures (48/924, max 5%) daisy-face-swap: 97.9% failures (41706/42597, max 5%) gps-mapmatch-valhalla: 36.2% failures (25/69, max 5%) | If any jig has a failure rate above the threshold, check the logs of the deployment's pods to see why they are failing. darvaza-source has high error rates because a no-op is reported as an error.
Go to https://it-grafana.cloud.vtti.vt.edu click explore and search `{namespace="dovetail", cluster="rc-one", service_name=" |
| FAIL | db2_database | db2_file_info_recent | FILE_INFO new files: ivs2=35,453/10d jdsy=13,235 cond=864/5d cttc=0/5d ivsm=0/15d hhyd=0 (last 1d) cttc: 0 new rows in 5d ivsm: 0 new rows in 15d hhyd: 0 new rows in 1d | |
| FAIL | db2_database | db2_file_summary_info_recent | FILE_SUMMARY_INFO new trips: cond=2/5d cttc=0/5d jdsy=0 ivsm=0/15d ivs2=0/10d hhyd=0 (last 1d) cttc: 0 new rows in 5d jdsy: 0 new rows in 1d ivsm: 0 new rows in 15d ivs2: 0 new rows in 10d hhyd: 0 new rows in 1d | |
| FAIL | db2_database | db2_load_queue_recent | LOAD_QUEUE completed loads: ivs2=24,215/10d cond=831/5d cttc=0/5d jdsy=0 ivsm=0/15d hhyd=0 (last 1d) cttc: 0 new rows in 5d jdsy: 0 new rows in 1d ivsm: 0 new rows in 15d hhyd: 0 new rows in 1d | |
| FAIL | infrastructure | s3_dovetail_performance | dovetail (jdsp): write 72 MB/s, read 113 MB/s (min 80 MB/s) | This check uses mcli to upload and download a 1 GiB object to the S3 endpoint. It measures the throughput and if either write or read is below 800 MB/s, it returns FAIL. If the mcli commands fail, it returns ERROR. Neal or Kenny need to be contacted when this happpens. It usually indicates a larger network problem or sometimes a ceph problem. |
| FAIL | infrastructure | s3_ceph_darvaza_performance | ceph-darvaza (darvaza): write 67 MB/s, read 114 MB/s (min 80 MB/s) | This check uses mcli to upload and download a 1 GiB object to the S3 endpoint. It measures the throughput and if either write or read is below 800 MB/s, it returns FAIL. If the mcli commands fail, it returns ERROR. Neal or Kenny need to be contacted when this happpens. It usually indicates a larger network problem or sometimes a ceph problem. |
| ERROR | dovetail | dovetail_oomkilled_pods | dovetail_oomkilled_pods timed out Command '['kubectl', '--context', 'rc-one', '-n', 'dovetail', 'get', 'pods', '-o', 'json']' timed out after 30 seconds | |
| OK | darvaza | pulsar_backlog_24h_avg | 24h avg backlog 0 (threshold 10000, topic persistent://notifications/buckets/darvaza-partition-0) | |
| OK | darvaza | pulsar_darvaza_subscription | backlog=0 consumers=2 ('darvaza' on notifications/buckets/darvaza-partition-0) | |
| OK | darvaza | darvaza_source_headobject_404 | 0 HeadObject 404 errors in 24h (want 0) | |
| OK | dovetail_feeder | dovetail_feeder_subscription_backlog | 0 jig subscription(s) over backlog 20 | |
| OK | dovetail_feeder | dovetail_feeder_sink_backlog | 0 jig sink(s) over backlog 20 | |
| OK | dovetail_feeder | dovetail_feeder_sink_consumers | 25/25 jig sinks have 1 consumer | |
| OK | dovetail_feeder | dovetail_feeder_processes | feeder container up, all 25 jig processes running | |
| OK | infrastructure | pulsar_cluster_disk_space | all 9 volumes >= 25% free in pulsar-dovetail (min 76.6%) | |
| OK | infrastructure | aisb_infra_disk_space | all 4 volumes >= 25% free in dovetail @ aisb-infra (min 27.1%) | |
| OK | infrastructure | dovetail_postgres_connection | connected to dovetail postgres (dovetail_feeder@dovetail-postgres-tailscale/dovetail_feeder) | |
| OK | infrastructure | reconciliation_postgres_connection | connected to reconciliation DB (app/app) | |
| OK | infrastructure | isilon4_fs_performance | isilon-4 (/vtti/vt_archive): write 257 MB/s, read 520 MB/s | |
| OK | infrastructure | isilon3_fs_performance | isilon-3 (/vtti/analysis01/public_data): write 136 MB/s, read 511 MB/s | |
| OK | infrastructure | qumulo5_fs_performance | qumulo-5 (/vtti/scratch/agent007): write 171 MB/s, read 519 MB/s |