Skip to content
Icon

recipes Image Check

Profile Avatar

Icon 1 4 Troubleshooting Commands

Icon 1 Last updated 13 weeks ago

Icon 1 Contributed by jon-funk

Icon Public Source Code
Icon Private Configuration
Icon Join Discussion

Troubleshooting Commands

Check Image Rollover Times for Namespace recipes

What does it do?

This command uses kubectl to retrieve information about running pods within a specific context and namespace, filtering for those in the "Running" state. It then formats the output in JSON using jq to display each pod's container image and last start time.

Command 
kubectl get pods --context=gke_runwhen-nonprod-sandbox_us-central1_sandbox-cluster-1-cluster -n recipes --field-selector=status.phase==Running -o json | jq -r '[.items[] | "Images: " + (.spec.containers[].image|tostring) + ", Last Started Times:" + (.status.containerStatuses[].state.running.startedAt|tostring)]'
IconCopy to clipboard Copied to clipboard

Learn more

This multi-line content is auto-generated and used for educational purposes. Copying and pasting the multi-line text might not function as expected.

# This command retrieves information about running pods in a specific context and namespace, then formats the output using jq to display container images and their last start times.

# Set the context and namespace variables for the kubectl command
CONTEXT=my-context
NAMESPACE=my-namespace

# Get the pods in the specified context and namespace that are in the Running state and output the results as JSON
kubectl get pods --context=${CONTEXT} -n ${NAMESPACE} --field-selector=status.phase==Running -o json | \

# Use jq to format the JSON output into a readable format including container images and their last start times
jq -r '[.items[] | "Images: " + (.spec.containers[].image|tostring) + ", Last Started Times:" + \
(.status.containerStatuses[].state.running.startedAt|tostring)]'
Helpful Links

List Images and Tags for Every Container in Running Pods for Namespace recipes

What does it do?

This command uses kubectl to retrieve information about running pods in a specific context and namespace, then formats the output using JSON and jq to display the pod name, status, and container details.

Command 
kubectl get pods --context=gke_runwhen-nonprod-sandbox_us-central1_sandbox-cluster-1-cluster -n recipes --field-selector=status.phase==Running -o=json | jq -r '.items[] | "---", "pod_name: " + .metadata.name, "Status: " + .status.phase, "containers:", (.spec.containers[] | "- container_name: " + .name, "  image_path: " + (.image | split(":")[0]), "  image_tag: " + (.image | split(":")[1])), "---"'
IconCopy to clipboard Copied to clipboard

Learn more

This multi-line content is auto-generated and used for educational purposes. Copying and pasting the multi-line text might not function as expected.

# Set the kubectl context to the value of the environment variable ${CONTEXT}
# Set the namespace for the kubectl command using the value of the environment variable ${NAMESPACE}
kubectl get pods --context=${CONTEXT} -n ${NAMESPACE} \
  --field-selector=status.phase==Running -o=json | \
  # Use jq to parse the JSON output and format it in a human-readable way
  jq -r '.items[] | 
    "---", 
    "pod_name: " + .metadata.name, 
    "Status: " + .status.phase, 
    "containers:", 
    (.spec.containers[] | 
      "- container_name: " + .name, 
     " \ image_path: " + (.image | split(":")[0]), 
     " \ image_tag: " + (.image | split(":")[1])
    ), 
   "---"'

In this multi-line command, we've added comments to explain each part of the command for those who may be new or less experienced with using kubectl and jq. This should make it easier for them to understand what the command is doing.
Helpful Links

List Images and Tags for Every Container in Failed Pods for Namespace recipes

What does it do?

This command retrieves information about specific pods in a particular namespace that have failed, and then formats and outputs the information in JSON format using the jq tool.

Command 
kubectl get pods --context=gke_runwhen-nonprod-sandbox_us-central1_sandbox-cluster-1-cluster -n recipes --field-selector=status.phase==Failed -o=json | jq -r '.items[] | "---", "pod_name: " + .metadata.name, "Status: " + .status.phase, "containers:", (.spec.containers[] | "- container_name: " + .name, "  image_path: " + \(.image | split(":")[0]), "  image_tag: " + (.image | split(":")[1])), "---"'
IconCopy to clipboard Copied to clipboard

Learn more

This multi-line content is auto-generated and used for educational purposes. Copying and pasting the multi-line text might not function as expected.

# Get pods from a specific context and namespace that have failed
kubectl get pods --context=${CONTEXT} -n ${NAMESPACE} --field-selector=status.phase==Failed -o=json \
  # Use jq to parse the JSON output and format it for easy reading
  | jq -r '.items[] | "---", "pod_name: " + .metadata.name, "Status: " + .status.phase, "containers:", (.spec.containers[] | 
  "- container_name: " + .name, 
  " \ image_path: " + \(.image | split(":")[0]), 
  " \ image_tag: " + (.image | split(":")[1])), "---"'

This multi-line command breaks down each step of the original command with helpful comments for better understanding. It also maintains the same functionality while making it easier to read and comprehend for new or less experienced devops engineers.
Helpful Links

List ImagePullBackOff Events and Test Path and Tags for Namespace recipes

What does it do?

This command is a shell script that uses kubectl to gather information about recent events in a specific namespace and context. It then evaluates whether there have been any "BackOff" events related to pulling container images, and if so, it deploys a new pod using the Skopeo tool to inspect the problematic container image and its available tags. If the image exists, it prints a message, otherwise it lists available tags and finally deletes the Skopeo pod.

Command 
NAMESPACE=recipes; POD_NAME="skopeo-pod"; CONTEXT="gke_runwhen-nonprod-sandbox_us-central1_sandbox-cluster-1-cluster"; events=$(kubectl get events -n $NAMESPACE --context=$CONTEXT -o json | jq --arg timestamp "$(date -u -v -5M +"%Y-%m-%dT%H:%M:%SZ" 2>/dev/null || date -u -d "-5 minutes" +"%Y-%m-%dT%H:%M:%SZ")" '.items[] | select(.lastTimestamp > $timestamp)'); if [[ ! -z "${events-unset}" ]]; then image_pull_backoff_events=$(echo "$events" | jq -s '[.[] | select(.reason == "BackOff") | .message] | .[]'); else echo "No events found in the last 5 minutes"; exit; fi; if [[ $image_pull_backoff_events =~ "Back-off pulling image" ]]; then echo "Running Skopeo Pod"; kubectl run $POD_NAME --restart=Never -n $NAMESPACE --context=$CONTEXT --image=quay.io/containers/skopeo:latest --command -- sleep infinity && echo "Waiting for the $POD_NAME to be running..." && kubectl wait --for=condition=Ready pod/$POD_NAME -n $NAMESPACE --context=$CONTEXT; else echo "No image pull backoff events found"; exit; fi; while IFS= read -r event; do echo "Found BackOff with message: $event"; echo "Checking if we can reach the image with skopeo and what tags exist"; container_image_path_tag=$(echo "$event" | cut -d' ' -f4 | tr -d '"' | tr -d '\'); container_image_path="${container_image_path_tag%:*}"; container_image_tag="${container_image_path_tag#*:}"; if [ -z "$container_image_path" ] || [ -z "$container_image_tag" ]; then continue; fi; skopeo_output=$(kubectl exec $POD_NAME -n $NAMESPACE --context=$CONTEXT -- skopeo inspect docker://$container_image_path:$container_image_tag); skopeo_exit_code=$?; if [ $skopeo_exit_code -eq 0 ]; then echo "Container image '$container_image_path:$container_image_tag' exists."; else echo "Container image '$container_image_path:$container_image_tag' does not exist."; echo "Available tags for '$container_image_path':"; available_tags=$(kubectl exec $POD_NAME -n $NAMESPACE --context=$CONTEXT -- skopeo list-tags docker://$container_image_path ); echo "$available_tags"; fi; done <<<"$image_pull_backoff_events" && echo "Deleting Skopeo pod" && kubectl delete pod $POD_NAME -n $NAMESPACE --context=$CONTEXT && echo "Done"
IconCopy to clipboard Copied to clipboard

Learn more

This multi-line content is auto-generated and used for educational purposes. Copying and pasting the multi-line text might not function as expected.

# Set the NAMESPACE variable to the environment variable of the same name
NAMESPACE=${NAMESPACE}

# Set the POD_NAME variable to "skopeo-pod"
POD_NAME="skopeo-pod"

# Set the CONTEXT variable to the environment variable of the same name
CONTEXT="${CONTEXT}"

# Get the events from the Kubernetes cluster and filter them based on timestamp
events=$(kubectl get events -n $NAMESPACE --context=$CONTEXT -o json | jq --arg timestamp "$(date -u -v -5M +"%Y-%m-%dT%H:%M:%SZ" 2>/dev/null || date -u -d "-5 minutes" +"%Y-%m-%dT%H:%M:%SZ")" '.items[] | select(.lastTimestamp > $timestamp)')

# If there are no events available, exit with a message
if [[ ! -z "${events-unset}" ]]; then
  image_pull_backoff_events=$(echo "$events" | jq -s '[.[] | select(.reason == "BackOff") | .message] | .[]')
else
  echo "No events found in the last 5 minutes"
  exit
fi

# Check if the image pull backoff event contains the specific message
if [[ $image_pull_backoff_events =~ "Back-off pulling image" ]]; then
  # Run Skopeo Pod with the specified configuration
  echo "Running Skopeo Pod"
  kubectl run $POD_NAME --restart=Never -n $NAMESPACE --context=$CONTEXT --image=quay.io/containers/skopeo:latest --command -- sleep infinity && echo "Waiting for the $POD_NAME to be running..." && kubectl wait --for=condition=Ready pod/$POD_NAME -n $NAMESPACE --context=$CONTEXT
else
  echo "No image pull backoff events found"
  exit
fi

# Process each image pull backoff event by checking for its availability and retrieving available tags using Skopeo
while IFS= read -r event; do
  echo "Found BackOff with message: $event"
  echo "Checking if we can reach the image with skopeo and what tags exist"
  container_image_path_tag=$(echo "$event" | cut -d' ' -f4 | tr -d '"' | tr -d '\')
  container_image_path="${container_image_path_tag%:*}"
  container_image_tag="${container_image_path_tag#*:}"
  if [ -z "$container_image_path" ] || [ -z "$container_image_tag" ]; then
    continue
  fi
  skopeo_output=$(kubectl exec $POD_NAME -n $NAMESPACE --context=$CONTEXT -- skopeo inspect docker://$container_image_path:$container_image_tag)
  skopeo_exit_code=$?
  if [ $skopeo_exit_code -eq 0 ]; then
    echo "Container image '$container_image_path:$container_image_tag' exists."
  else
    echo "Container image '$container_image_path:$container_image_tag' does not exist."
    echo "Available tags for '$container_image_path':"
    available_tags=$(kubectl exec $POD_NAME -n $NAMESPACE --context=$CONTEXT -- skopeo list-tags docker://$container_image_path )
    echo "$available_tags"
  fi
done <<<"$image_pull_backoff_events"

# Delete the Skopeo pod after processing all events
echo "Deleting Skopeo pod"
kubectl delete pod $POD_NAME -n $NAMESPACE --context=$CONTEXT
echo "Done"
Helpful Links