|
vault2 StatefulSet Health¶
6 Troubleshooting Commands
Last updated 13 weeks ago 
Contributed by 
Private Configuration
Join Discussion
Troubleshooting Commands¶
Troubleshoot StatefulSet Warning Events for vault2¶
What does it do?
This command retrieves Kubernetes events from a specific context and namespace, filtering for warnings related to a particular StatefulSet name within the last hour, and formats the output in JSON using the jq tool.
Command
kubectl get events --context gke_runwhen-nonprod-sandbox_us-central1_sandbox-cluster-1-cluster -n vault -o json | jq '(now - (60*60)) as $time_limit | [ .items[] | select(.type == "Warning" and (.involvedObject.kind == "StatefulSet" or .involvedObject.kind == "Pod") and (.involvedObject.name | tostring | contains("vault2")) and (.lastTimestamp | fromdateiso8601) >= $time_limit) | {kind: .involvedObject.kind, name: .involvedObject.name, reason: .reason, message: .message, firstTimestamp: .firstTimestamp, lastTimestamp: .lastTimestamp} ] | group_by([.kind, .name]) | map({kind: .[0].kind, name: .[0].name, count: length, reasons: map(.reason) | unique, messages: map(.message) | unique, firstTimestamp: map(.firstTimestamp | fromdateiso8601) | sort | .[0] | todateiso8601, lastTimestamp: map(.lastTimestamp | fromdateiso8601) | sort | reverse | .[0] | todateiso8601})'
Copy to clipboard
Copied to clipboard
Learn more
This multi-line content is auto-generated and used for educational purposes. Copying and pasting the multi-line text might not function as expected.
# Set the context and namespace for the Kubernetes cluster
CONTEXT=mycontext
NAMESPACE=mynamespace
# Create a time limit to filter out events older than 1 hour
time_limit=$(($($(date +%s) - 3600)))
# Use kubectl to get events in JSON format from the specified context and namespace, then extract only certain event details using jq
kubectl get events --context $CONTEXT -n $NAMESPACE -o json | jq \
'(now - (60*60)) as $time_limit |
[ .items[] |
select(
.type == "Warning" and
(.involvedObject.kind == "StatefulSet" or .involvedObject.kind == "Pod") and
(.involvedObject.name | tostring | contains("statefulset_name_here")) and
(.lastTimestamp | fromdateiso8601) >= $time_limit
)
| {kind: .involvedObject.kind, name: .involvedObject.name, reason: .reason, message: .message, firstTimestamp: .firstTimestamp, lastTimestamp: .lastTimestamp} ] |
group_by([.kind, .name]) |
map({
kind: .[0].kind,
name: .[0].name,
count: length,
reasons: map(.reason) | unique,
messages: map(.message) | unique,
firstTimestamp: map(.firstTimestamp | fromdateiso8601) | sort | .[0] | todateiso8601,
lastTimestamp: map(.lastTimestamp | fromdateiso8601) | sort | reverse | .[0] | todateiso8601
})'
Helpful Links
Check StatefulSet Event Anomalies for vault2¶
What does it do?
This command uses kubectl to get events from a specific context and namespace, then filters and formats the data using jq to provide information about events that occurred in the last hour for a specific StatefulSet.
Command
kubectl get events --context gke_runwhen-nonprod-sandbox_us-central1_sandbox-cluster-1-cluster -n vault -o json | jq '(now - (60*60)) as $time_limit | [ .items[] | select(.type != "Warning" and (.involvedObject.kind == "StatefulSet" or .involvedObject.kind == "Pod") and (.involvedObject.name | tostring | contains("vault2"))) | {kind: .involvedObject.kind, count: .count, name: .involvedObject.name, reason: .reason, message: .message, firstTimestamp: .firstTimestamp, lastTimestamp: .lastTimestamp, duration: (if (((.lastTimestamp | fromdateiso8601) - (.firstTimestamp | fromdateiso8601)) == 0) then 1 else (((.lastTimestamp | fromdateiso8601) - (.firstTimestamp | fromdateiso8601))/60) end) } ] | group_by([.kind, .name]) | map({kind: .[0].kind, name: .[0].name, count: (map(.count) | add), reasons: map(.reason) | unique, messages: map(.message) | unique, average_events_per_minute: (if .[0].duration == 1 then 1 else ((map(.count) | add)/.[0].duration ) end),firstTimestamp: map(.firstTimestamp | fromdateiso8601) | sort | .[0] | todateiso8601, lastTimestamp: map(.lastTimestamp | fromdateiso8601) | sort | reverse | .[0] | todateiso8601})'Learn more
This multi-line content is auto-generated and used for educational purposes. Copying and pasting the multi-line text might not function as expected.
# set the context and namespace to use
kubectl get events --context ${CONTEXT} -n ${NAMESPACE} -o json | jq '
# calculate the time limit one hour ago
(now - (60*60)) as $time_limit |
# filter the events based on specific criteria
[ .items[] |
select(
.type != "Warning" and
(.involvedObject.kind == "StatefulSet" or .involvedObject.kind == "Pod") and
(.involvedObject.name | tostring | contains("${STATEFULSET_NAME}"))
) |
# transform the filtered events into a new format
{
kind: .involvedObject.kind,
count: .count,
name: .involvedObject.name,
reason: .reason,
message: .message,
firstTimestamp: .firstTimestamp,
lastTimestamp: .lastTimestamp,
duration: (
if (((.lastTimestamp | fromdateiso8601) - (.firstTimestamp | fromdateiso8601)) == 0)
then 1
else (((.lastTimestamp | fromdateiso8601) - (.firstTimestamp | fromdateiso8601))/60)
end
)
}
] |
# group the transformed events by kind and name
group_by([.kind, .name]) |
# map the grouped events to a summary format
map({
kind: .[0].kind,
name: .[0].name,
count: (map(.count) | add),
reasons: map(.reason) | unique,
messages: map(.message) | unique,
average_events_per_minute: (
if .[0].duration == 1
then 1
else ((map(.count) | add)/.[0].duration )
end
),
firstTimestamp: map(.firstTimestamp | fromdateiso8601) | sort | .[0] | todateiso8601,
lastTimestamp: map(.lastTimestamp | fromdateiso8601) | sort | reverse | .[0] | todateiso8601
})'
Fetch StatefulSet Logs for vault2¶
What does it do?
This command prints the last 100 lines of logs for a specific StatefulSet in a Kubernetes cluster. The context flag specifies which Kubernetes cluster to use and the namespace flag specifies the namespace within the cluster.
Command
kubectl logs --tail=100 statefulset/vault2 --context gke_runwhen-nonprod-sandbox_us-central1_sandbox-cluster-1-cluster -n vaultLearn more
This multi-line content is auto-generated and used for educational purposes. Copying and pasting the multi-line text might not function as expected.
# The following command is used to retrieve the logs for a specific statefulset in a Kubernetes cluster.
# Use kubectl to print the logs of a specific statefulset with the given STATEFULSET_NAME
# --tail=100 flag indicates that last 100 lines of logs will be shown
# --context flag specifies the context to use, which allows us to interact with different clusters
# Specify the namespace using -n flag followed by the NAMESPACE variable
kubectl logs --tail=100 statefulset/${STATEFULSET_NAME} --context ${CONTEXT} -n ${NAMESPACE}
Get Related StatefulSet vault2 Events¶
What does it do?
This command retrieves events related to warning types within a specific namespace and context, then uses the grep tool to search for the specified statefulset name, and will return true if no matching event is found.
Command
kubectl get events --field-selector type=Warning --context gke_runwhen-nonprod-sandbox_us-central1_sandbox-cluster-1-cluster -n vault | grep -i "vault2" || trueLearn more
This multi-line content is auto-generated and used for educational purposes. Copying and pasting the multi-line text might not function as expected.
# This command retrieves events for a specific namespace and context and then filters the results to find any warnings related to a particular statefulset.
# Set the context and namespace variables for the kubectl command
CONTEXT="your_context_here"
NAMESPACE="your_namespace_here"
# Use kubectl to get events with warnings based on the specified context and namespace
kubectl get events --field-selector type=Warning --context ${CONTEXT} -n ${NAMESPACE} \
# Then use grep to filter the events for any mentions of the specified statefulset name and ignore case sensitivity
| grep -i "${STATEFULSET_NAME}" \
# Finally, include the '|| true' statement to ensure that the overall exit status of the command is successful even if there are no matching events
|| true
Helpful Links
Fetch Manifest Details for StatefulSet vault2¶
What does it do?
This command retrieves the configuration of a stateful set in Kubernetes, using specific labels and context, and outputs the details in YAML format within a specific namespace.
Command
kubectl get statefulset --context=gke_runwhen-nonprod-sandbox_us-central1_sandbox-cluster-1-cluster -n vault -o yamlLearn more
This multi-line content is auto-generated and used for educational purposes. Copying and pasting the multi-line text might not function as expected.
# This command retrieves the YAML definition of a stateful set in a Kubernetes cluster
# Replace ${LABELS} with the labels of the stateful set you want to retrieve
# Replace ${CONTEXT} with the context of your Kubernetes cluster
# Replace ${NAMESPACE} with the namespace where the stateful set is located
kubectl get statefulset ${LABELS} \ # Using the kubectl get command to retrieve the stateful set
--context=${CONTEXT} \ # Specifying the context of the Kubernetes cluster
-n ${NAMESPACE} \ # Specifying the namespace where the stateful set is located
-o yaml # Outputting the result in YAML format
List StatefulSets with Unhealthy Replica Counts In Namespace vault¶
What does it do?
This command uses kubectl to get information about stateful sets in a specific namespace and context, then uses jq to filter and format the output to show StatefulSet names, their desired and available replicas, and whether there are any discrepancies between the two.
Command
kubectl get statefulset -n vault -o json --context gke_runwhen-nonprod-sandbox_us-central1_sandbox-cluster-1-cluster | jq -r '.items[] | select(.status.availableReplicas < .status.replicas) | "---\nStatefulSet Name: " + (.metadata.name|tostring) + "\nDesired Replicas: " + (.status.replicas|tostring) + "\nAvailable Replicas: " + (.status.availableReplicas|tostring)'Learn more
This multi-line content is auto-generated and used for educational purposes. Copying and pasting the multi-line text might not function as expected.
# Set the namespace and context for the kubectl command
NAMESPACE="your-namespace"
CONTEXT="your-context"
# Use kubectl to get the statefulset in the specified namespace as JSON, pipe the output to jq for parsing
kubectl get statefulset -n $NAMESPACE -o json --context $CONTEXT | \
# Use jq to filter the items where available replicas are less than desired replicas
jq -r '.items[] | select(.status.availableReplicas < .status.replicas) |
# Print out the StatefulSet name, desired replicas, and available replicas with a separator
"---\nStatefulSet Name: " + (.metadata.name|tostring) + "\nDesired Replicas: " + (.status.replicas|tostring) + "\nAvailable Replicas: " + (.status.availableReplicas|tostring)'
In this multi-line command, we first set the NAMESPACE and CONTEXT variables for the kubectl command. Then, we use kubectl to get the statefulset in the specified namespace as JSON and pipe the output to jq for parsing. In the jq command, we filter the items where available replicas are less than desired replicas and then print out the StatefulSet name, desired replicas, and available replicas with a separator. This can be useful for identifying any StatefulSets that may not have the correct number of replicas running.