|
voteers-Kong Ingress HTTP Errors¶
4 Troubleshooting Commands
Last updated 37 weeks ago 
Contributed by 
Private Configuration
Join Discussion
Troubleshooting Commands¶
Check If Kong Ingress HTTP Error Rate Violates HTTP Error Threshold¶
What does it do?
This command activates the Google Cloud authentication service account, makes an API request to the Google Cloud Monitoring API using a specific query, and then processes and displays the results in a human-readable format. It's used for monitoring HTTP error rates for a specific service running on Google Cloud.
Command
response=$(curl -s -d "query=rate(kong_http_requests_total{service='trouble-town.voteers.voteservice.80',code=~'5.*'}[1m]) > 0.5" -H "Authorization: Bearer $(gcloud auth print-access-token)" 'https://monitoring.googleapis.com/v1/projects/runwhen-nonprod-sandbox/location/global/prometheus/api/v1/query') && echo "$response" | jq -e '.data.result | length > 0' && echo "$response" | jq -r '.data.result[] | "Route:" + .metric.route + " Service:" + .metric.service + " Kong Instance:" + .metric.instance + " HTTP Error Count:" + .value[1]' || echo "No HTTP Error threshold violations found for trouble-town.voteers.voteservice.80."
Copy to clipboard
Copied to clipboard
Learn more
This multi-line content is auto-generated and used for educational purposes. Copying and pasting the multi-line text might not function as expected.
# Authenticate as a service account using the key file stored in the environment variable GOOGLE_APPLICATION_CREDENTIALS
gcloud auth activate-service-account --key-file=$GOOGLE_APPLICATION_CREDENTIALS
# Perform a POST request to the Google Cloud Monitoring API to get error rate data from Prometheus
# Store the response in the 'response' variable
response=$(curl -s -d "query=rate(kong_http_requests_total{service='${INGRESS_SERVICE}',code=~'${HTTP_ERROR_CODES}'}[${TIME_SLICE}]) > ${HTTP_ERROR_RATE_THRESHOLD}" -H "Authorization: Bearer $(gcloud auth print-access-token)" 'https://monitoring.googleapis.com/v1/projects/runwhen-nonprod-sandbox/location/global/prometheus/api/v1/query')
# Check if there are any results in the response using jq and output the length
echo "$response" | jq -e '.data.result | length > 0'
# Output route, service, kong instance, and HTTP error count for each result in the response
echo "$response" | jq -r '.data.result[] | "Route:" + .metric.route + " Service:" + .metric.service + " Kong Instance:" + .metric.instance + " HTTP Error Count:" + .value[1]'
# If there are no HTTP error threshold violations found for the specified ingress service, output this message
# Otherwise, output the route, service, kong instance, and HTTP error count for each violation
echo "No HTTP Error threshold violations found for ${INGRESS_SERVICE}."
In this multi-line command, we first authenticate the service account with gcloud. Then we make a curl request to the Google Cloud Monitoring API to query Prometheus for error rate data. Finally, we use jq to parse and display the results based on certain conditions, outputting relevant information if HTTP error threshold violations are found or a message indicating none were found.
Check If Kong Ingress HTTP Request Latency Violates Threshold¶
What does it do?
This command activates a service account in Google Cloud and then uses curl to query the Prometheus API for monitoring data, checking if the HTTP request latency exceeds a specified threshold for a particular service. If violations are found, it will display the service name and request latency; if not, it will indicate that no violations were found.
Command
response=$(curl -s -d "query=histogram_quantile(0.99, sum(rate(kong_request_latency_ms_bucket{service='trouble-town.voteers.voteservice.80'}[1m])) by (le)) > 100" -H "Authorization: Bearer $(gcloud auth print-access-token)" 'https://monitoring.googleapis.com/v1/projects/runwhen-nonprod-sandbox/location/global/prometheus/api/v1/query') && echo "$response" | jq -e '.data.result | length > 0' && echo "$response" | jq -r '.data.result[] | "Service: trouble-town.voteers.voteservice.80" + " HTTP Request Latency(ms):" + .value[1]' || echo "No HTTP request latency threshold violations found for trouble-town.voteers.voteservice.80."Learn more
This multi-line content is auto-generated and used for educational purposes. Copying and pasting the multi-line text might not function as expected.
# Activate the service account using the key file specified in the GOOGLE_APPLICATION_CREDENTIALS environment variable
gcloud auth activate-service-account --key-file=$GOOGLE_APPLICATION_CREDENTIALS
# Make a request to the Google Cloud Monitoring API to query for HTTP request latency data
# The response is stored in the 'response' variable
response=$(curl -s -d "query=histogram_quantile(0.99, sum(rate(kong_request_latency_ms_bucket{service='${INGRESS_SERVICE}'}[${TIME_SLICE}])) by (le)) > ${REQUEST_LATENCY_THRESHOLD}" -H "Authorization: Bearer $(gcloud auth print-access-token)" 'https://monitoring.googleapis.com/v1/projects/runwhen-nonprod-sandbox/location/global/prometheus/api/v1/query')
# Check if the query returned any results
# If the result contains at least one item, print the latency data for each item
# If the result is empty, print a message indicating that no threshold violations were found
echo "$response" | jq -e '.data.result | length > 0' && echo "$response" | jq -r '.data.result[] | "Service: ${INGRESS_SERVICE}" + " HTTP Request Latency(ms):" + .value[1]' || echo "No HTTP request latency threshold violations found for ${INGRESS_SERVICE}."
Check If Kong Ingress Controller Reports Upstream Errors¶
What does it do?
This command uses the gcloud tool to activate a service account and then sends a query to the Google Cloud Monitoring API to check the health status of an upstream target, displaying whether healthchecks are enabled or disabled.
Command
response=$(curl -s -d "query=kong_upstream_target_health{upstream='voteservice.trouble-town.80',state='healthchecks_off'} > 0" -H "Authorization: Bearer $(gcloud auth print-access-token)" 'https://monitoring.googleapis.com/v1/projects/runwhen-nonprod-sandbox/location/global/prometheus/api/v1/query') && echo "$response" | jq -e '.data.result | length > 0' && echo "$response" | jq -r '.data.result[] | "Service: voteservice.trouble-town.80" + " Healthchecks Disabled!' || echo "voteservice.trouble-town.80 has healthchecks enabled."Learn more
This multi-line content is auto-generated and used for educational purposes. Copying and pasting the multi-line text might not function as expected.
# Step 1: Authenticate with Google Cloud using a service account key
gcloud auth activate-service-account --key-file=$GOOGLE_APPLICATION_CREDENTIALS
# Step 2: Make a request to the Google Cloud Monitoring API to check the health of a specific upstream target
response=$(curl -s -d "query=kong_upstream_target_health{upstream='${INGRESS_UPSTREAM}',state='healthchecks_off'} > 0" -H "Authorization: Bearer $(gcloud auth print-access-token)" 'https://monitoring.googleapis.com/v1/projects/runwhen-nonprod-sandbox/location/global/prometheus/api/v1/query')
# Step 3: Check if the response contains any healthcheck results
echo "$response" | jq -e '.data.result | length > 0'
# Step 4: Output the status of the healthchecks for the specified upstream target
echo "$response" | jq -r '.data.result[] | "Service: ${INGRESS_UPSTREAM}" + " Healthchecks Disabled!' || echo "${INGRESS_UPSTREAM} has healthchecks enabled."
Check If Kong Ingress Controller Reports Upstream Errors¶
What does it do?
This command uses the gcloud tool to activate a service account and then sends a query to the Google Cloud Monitoring API to check the health status of an upstream target, displaying whether healthchecks are enabled or disabled.
Command
response=$(curl -s -d "query=kong_upstream_target_health{upstream='voteservice.trouble-town.80',state=~'dns_error|unhealthy'} > 0" -H "Authorization: Bearer $(gcloud auth print-access-token)" 'https://monitoring.googleapis.com/v1/projects/runwhen-nonprod-sandbox/location/global/prometheus/api/v1/query') && echo "$response" | jq -e '.data.result | length > 0' && echo "$response" | jq -r '.data.result[] | "Issue detected with Service: voteservice.trouble-town.80" + " Healthcheck subsystem-state: " + .metric.subsystem + "-" + .metric.state + " Target: " + .metric.target' || echo "voteservice.trouble-town.80 is reported as healthy from the Kong ingress controller."Learn more
This multi-line content is auto-generated and used for educational purposes. Copying and pasting the multi-line text might not function as expected.
# Step 1: Authenticate with Google Cloud using a service account key
gcloud auth activate-service-account --key-file=$GOOGLE_APPLICATION_CREDENTIALS
# Step 2: Make a request to the Google Cloud Monitoring API to check the health of a specific upstream target
response=$(curl -s -d "query=kong_upstream_target_health{upstream='${INGRESS_UPSTREAM}',state='healthchecks_off'} > 0" -H "Authorization: Bearer $(gcloud auth print-access-token)" 'https://monitoring.googleapis.com/v1/projects/runwhen-nonprod-sandbox/location/global/prometheus/api/v1/query')
# Step 3: Check if the response contains any healthcheck results
echo "$response" | jq -e '.data.result | length > 0'
# Step 4: Output the status of the healthchecks for the specified upstream target
echo "$response" | jq -r '.data.result[] | "Service: ${INGRESS_UPSTREAM}" + " Healthchecks Disabled!' || echo "${INGRESS_UPSTREAM} has healthchecks enabled."