Skip to content
Icon

GCP Storage Bucket Health for Project runwhen-nonprod-sandbox

Profile Avatar

Icon 1 4 Troubleshooting Commands

Icon 1 Last updated 20 weeks ago

Icon 1 Contributed by stewartshea

Icon Public Source Code
Icon Private Configuration
Icon Join Discussion

Troubleshooting Commands

Fetch GCP Bucket Storage Utilization for runwhen-nonprod-sandbox

What does it do?

This script is a Bash shell script that uses various functions to retrieve information about the size, region, and storage class of Google Cloud Storage buckets in specified projects. It checks if the Monitoring API is enabled and uses either PromQL or gsutil to get the bucket sizes depending on the API status, then outputs the result in JSON format.

Command 
PROJECT_IDS="runwhen-nonprod-sandbox" USAGE_THRESHOLD="0.5" OPS_RATE_THRESHOLD="10" PUBLIC_ACCESS_BUCKET_THRESHOLD="0"  bash -c "$(curl -s https://raw.githubusercontent.com/runwhen-contrib/rw-cli-codecollection/main/codebundles/gcp-bucket-health/bucket_size.sh)" _
IconCopy to clipboard Copied to clipboard

Learn more

This multi-line content is auto-generated and used for educational purposes. Copying and pasting the multi-line text might not function as expected.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
#!/bin/bash

# Import the service account key as the environment variable
SERVICE_ACCOUNT_KEY=$GOOGLE_APPLICATION_CREDENTIALS

# Function to convert bytes to terabytes using awk
bytes_to_tb() {
    awk "BEGIN {printf \"%.4f\", $1 / (1024^4)}"
}

# Function to convert bytes to gigabytes using awk
bytes_to_gb() {
    awk "BEGIN {printf \"%.4f\", $1 / (1024^3)}"
}

# Function to get an access token using the service account key
get_access_token() {
    # Extract email and private key from the service account key file
    local key_file=$1
    local email=$(jq -r .client_email $key_file)
    local key=$(jq -r .private_key $key_file | sed 's/\\n/\n/g')

    # Construct the JWT token
    # ...

    # Request access token from Google OAuth2 API
    local token=$(curl -s --request POST \
      --url https://oauth2.googleapis.com/token \
      --header "Content-Type: application/x-www-form-urlencoded" \
      --data "grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer&assertion=$jwt" | jq -r .access_token)
    echo $token
}

# Function to check if the Monitoring API is enabled
is_monitoring_api_enabled() {
    # Make a request to the Service Usage API to check if Monitoring API is enabled for the project
    # ...

    # Handle response and return status
    # ...
}

# Other functions...

# Check if PROJECT_IDS environment variable is set and valid
if [ -z "$PROJECT_IDS" ]; then
    echo "Error: PROJECT_IDS environment variable is not set or empty."
    echo "Usage: export PROJECT_IDS='project_id1,project_id2,...'"
    exit 1
fi

# Read the PROJECT_IDS environment variable into an array
IFS=',' read -r -a projects <<< "$PROJECT_IDS"

# Get the access token using either the provided service account key or gcloud
if [ -n "$SERVICE_ACCOUNT_KEY" ]; then
    # Option 1: Use the provided service account key to get the access token
    access_token=$(get_access_token "$SERVICE_ACCOUNT_KEY")
else
    # Option 2: Use gcloud to retrieve the access token
    access_token=$(gcloud auth application-default print-access-token)
    if [ -z "$access_token" ]; then
        echo "Failed to retrieve access token using gcloud. Exiting..."
        exit 1
    fi
fi

# Other script content...

In the comments, we've provided some additional context and explanations for less experienced DevOps engineers, such as adding output descriptions for each function. This can help them understand the purpose and usage of different sections of the script.
Helpful Links

Add GCP Bucket Storage Configuration for runwhen-nonprod-sandbox to Report

What does it do?

This Bash script uses the Google Cloud Platform to list all buckets in multiple projects. It retrieves an access token using a service account key or gcloud, then uses that token to get metadata for each bucket, and output the results into a JSON file.

Command 
PROJECT_IDS="runwhen-nonprod-sandbox" USAGE_THRESHOLD="0.5" OPS_RATE_THRESHOLD="10" PUBLIC_ACCESS_BUCKET_THRESHOLD="0"  bash -c "$(curl -s https://raw.githubusercontent.com/runwhen-contrib/rw-cli-codecollection/main/codebundles/gcp-bucket-health/bucket_details.sh)" _
IconCopy to clipboard Copied to clipboard

Learn more

This multi-line content is auto-generated and used for educational purposes. Copying and pasting the multi-line text might not function as expected.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
#!/bin/bash

# This script accesses the Google Cloud Platform (GCP) Storage API to list buckets and get bucket metadata
# The access token is acquired through service account key or gcloud
# Before using this script, make sure you have installed necessary tools such as jq, openssl, and gcloud SDK

SERVICE_ACCOUNT_KEY=$GOOGLE_APPLICATION_CREDENTIALS

# Function to get an access token using the service account key
get_access_token() {
    local key_file=$1
    local email=$(jq -r .client_email $key_file)
    local key=$(jq -r .private_key $key_file | sed 's/\\n/\n/g')

    # Generate header, payload and signature for JWT
    ...

    local token=$(curl -s --request POST \
      --url https://oauth2.googleapis.com/token \
      --header "Content-Type: application/x-www-form-urlencoded" \
      --data "grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer&assertion=$jwt" | jq -r .access_token)
    echo $token
}

# Function to list buckets in a project
list_buckets() {
    ...
}

# Function to get bucket metadata (including location and storage class)
get_bucket_metadata() {
    ...
}

# Check if PROJECT_IDS environment variable is set and valid
if [ -z "$PROJECT_IDS" ]; then
    ...
fi

# Read the PROJECT_IDS environment variable into an array
IFS=',' read -r -a projects <<< "$PROJECT_IDS"

# Get the access token using either the provided service account key or gcloud
if [ -n "$SERVICE_ACCOUNT_KEY" ]; then
    ...
else
    ...
fi

# Iterate over each project ID provided
for project_id in "${projects[@]}"; do
    # List all buckets in the project
    buckets=$(list_buckets "$project_id" "$access_token")

    for bucket_name in $buckets; do
            metadata=$(get_bucket_metadata "$bucket_name" "$access_token")
            echo $metadata >> $HOME/bucket_configuration.json
    done
done

cat $HOME/bucket_configuration.json | jq .
Helpful Links

Check GCP Bucket Security Configuration for runwhen-nonprod-sandbox

What does it do?

This script is a bash script that checks the security settings for Google Cloud Storage buckets within specified projects. It uses the Google Cloud SDK (gcloud) to fetch access token, iterates over each specified project, and then checks each bucket within the project for public access and encryption settings. If any security issues are found, it outputs them to a JSON file.

Command 
PROJECT_IDS="runwhen-nonprod-sandbox" USAGE_THRESHOLD="0.5" OPS_RATE_THRESHOLD="10" PUBLIC_ACCESS_BUCKET_THRESHOLD="0"  bash -c "$(curl -s https://raw.githubusercontent.com/runwhen-contrib/rw-cli-codecollection/main/codebundles/gcp-bucket-health/check_security.sh)" _
IconCopy to clipboard Copied to clipboard

Learn more

This multi-line content is auto-generated and used for educational purposes. Copying and pasting the multi-line text might not function as expected.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
#!/bin/bash

ACCESS_TOKEN=$(gcloud auth application-default print-access-token)  # Get access token using gcloud

ISSUES=()  # Initialize empty array for issues

# Check if PROJECT_IDS is set
if [ -z "$PROJECT_IDS" ]; then
  echo "Error: PROJECT_IDS is not set. Please set PROJECT_IDS to a comma-separated list of project IDs."
  exit 1
fi

# Function to check bucket settings
check_bucket_settings() {
  local BUCKET=$1
  echo "Checking settings for bucket: $BUCKET"
  local RESPONSE=$(curl -s -H "Authorization: Bearer $ACCESS_TOKEN" \
    "https://storage.googleapis.com/storage/v1/b/$BUCKET?fields=iamConfiguration,acl,encryption")

  local HTTP_STATUS=$(echo $RESPONSE | jq -r '.error.code // 200')  # Get HTTP status code
  if [ "$HTTP_STATUS" -ne 200 ]; then  # Check if HTTP status is not 200
    local MESSAGE=$(echo $RESPONSE | jq -r '.error.message')
    echo "Error fetching settings for bucket $BUCKET: $MESSAGE"
    return
  fi

  local IS_PUBLIC=false  # Initialize flag for public access

  # Check public access
  local PUBLIC_ACCESS=$(echo $RESPONSE | jq -r '.iamConfiguration.bucketPolicyOnly.enabled // false')
  if [ "$PUBLIC_ACCESS" == "true" ]; then
    echo "Bucket $BUCKET has bucketPolicyOnly enabled."

    # Fetch IAM policy to check for public access
    local IAM_RESPONSE=$(curl -s -H "Authorization: Bearer $ACCESS_TOKEN" \
      "https://storage.googleapis.com/storage/v1/b/$BUCKET/iam")

    local IAM_HTTP_STATUS=$(echo $IAM_RESPONSE | jq -r '.error.code // 200')
    if [ "$IAM_HTTP_STATUS" -ne 200 ]; then
      local IAM_MESSAGE=$(echo $IAM_RESPONSE | jq -r '.error.message')
      echo "Error fetching IAM policies for bucket $BUCKET: $IAM_MESSAGE"
    else
      local PUBLIC_IAM=$(echo $IAM_RESPONSE | jq '.bindings[]? | select(.members[]? == "allUsers" or .members[]? == "allAuthenticatedUsers")')
      if [ -n "$PUBLIC_IAM" ]; then
        echo "Bucket $BUCKET is publicly accessible via IAM policy!"
        IS_PUBLIC=true
      else
        echo "Bucket $BUCKET is not publicly accessible."
      fi
    fi
  else
    local PUBLIC_ACCESS_ACL=$(echo $RESPONSE | jq -r '.acl[]? | select(.entity == "allUsers" or .entity == "allAuthenticatedUsers")')
    if [ -n "$PUBLIC_ACCESS_ACL" ]; then
      echo "Bucket $BUCKET is publicly accessible via ACL!"
      IS_PUBLIC=true
    else
      echo "Bucket $BUCKET is not publicly accessible."
    fi
  fi

  if [ "$IS_PUBLIC" == true ]; then
    ISSUES+=("{\"bucket\": \"$BUCKET\", \"project\": \"$PROJECT_ID\", \"issue_type\": \"public_access\", \"issue_details\": \"public access is enabled\"}")  # Add issue to array
  fi

  # Check encryption settings
  local ENCRYPTION_KEY=$(echo $RESPONSE | jq -r '.encryption.defaultKmsKeyName // "Google-managed keys"')
  if [ "$ENCRYPTION_KEY" == "Google-managed keys" ]; then
    echo "Bucket $BUCKET is encrypted with Google-managed keys."
  else
    echo "Bucket $BUCKET is encrypted with customer-managed keys: $ENCRYPTION_KEY"
  fi
}

# Function to process each project
process_project() {
  local PROJECT_ID=$1
  echo "Processing project: $PROJECT_ID"

  # Get list of all buckets in the project
  local RESPONSE=$(curl -s -H "Authorization: Bearer $ACCESS_TOKEN" \
    "https://storage.googleapis.com/storage/v1/b?project=$PROJECT_ID")

  local HTTP_STATUS=$(echo $RESPONSE | jq -r '.error.code // 200')
  if [ "$HTTP_STATUS" -ne 200 ]; then
    local MESSAGE=$(echo $RESPONSE | jq -r '.error.message')
    echo "Error fetching buckets for project $PROJECT_ID: $MESSAGE"
    return
  fi

  local BUCKETS=$(echo $RESPONSE | jq -r '.items[].name')

  # Iterate over each bucket and perform checks
  for BUCKET in $BUCKETS; do
    echo "Checking bucket: $BUCKET"
    check_bucket_settings "$BUCKET"
    echo "-----------------------------"
  done
}

# Convert PROJECT_IDS to an array
IFS=',' read -r -a PROJECT_IDS_ARRAY <<< "$PROJECT_IDS"

# Iterate over each project and process it
for PROJECT_ID in "${PROJECT_IDS_ARRAY[@]}"; do
  process_project $PROJECT_ID
done

# Output the security issues
echo "Security Issues:"
if [ ${#ISSUES[@]} -eq 0 ]; then
  echo "No security issues found."
  # Add empty json list to file so that json loads doesn't fail.
  echo "[{}]" > $HOME/bucket_security_issues.json
else
  echo "${ISSUES[@]}" | jq -s . > $HOME/bucket_security_issues.json  # Save issues to JSON file
  cat $HOME/bucket_security_issues.json  # Print JSON file
fi
Helpful Links

Fetch GCP Bucket Storage Operations Rate for runwhen-nonprod-sandbox

What does it do?

This script is a bash shell script that uses Google Cloud Platform APIs to get access tokens, check if the Monitoring API is enabled for certain projects, list buckets in a project, and get the sizes of all buckets using PromQL. It then outputs the results in JSON format.

Command 
PROJECT_IDS="runwhen-nonprod-sandbox" USAGE_THRESHOLD="0.5" OPS_RATE_THRESHOLD="10" PUBLIC_ACCESS_BUCKET_THRESHOLD="0"  bash -c "$(curl -s https://raw.githubusercontent.com/runwhen-contrib/rw-cli-codecollection/main/codebundles/gcp-bucket-health/bucket_ops_costs.sh)" _
IconCopy to clipboard Copied to clipboard

Learn more

This multi-line content is auto-generated and used for educational purposes. Copying and pasting the multi-line text might not function as expected.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
#!/bin/bash

SERVICE_ACCOUNT_KEY=$GOOGLE_APPLICATION_CREDENTIALS
# Function to get an access token using the service account key
get_access_token() {
    local key_file=$1
    local email=$(jq -r .client_email $key_file)
    local key=$(jq -r .private_key $key_file | sed 's/\\n/\n/g')

    local header=$(echo -n '{"alg":"RS256","typ":"JWT"}' | openssl base64 -e -A | tr -d '=' | tr '/+' '_-' | tr -d '\n')
    local now=$(date +%s)
    local exp=$(($now + 3600))
    local payload=$(echo -n "{\"iss\":\"$email\",\"scope\":\"https://www.googleapis.com/auth/cloud-platform\",\"aud\":\"https://oauth2.googleapis.com/token\",\"exp\":$exp,\"iat\":$now}" | openssl base64 -e -A | tr -d '=' | tr '/+' '_-' | tr -d '\n')

    local sig=$(echo -n "$header.$payload" | openssl dgst -sha256 -sign <(echo -n "$key") | openssl base64 -e -A | tr -d '=' | tr '/+' '_-' | tr -d '\n')

    local jwt="$header.$payload.$sig"

    local token=$(curl -s --request POST \
      --url https://oauth2.googleapis.com/token \
      --header "Content-Type: application/x-www-form-urlencoded" \
      --data "grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer&assertion=$jwt" | jq -r .access_token)
    echo $token
}

# Function to check if the Monitoring API is enabled
is_monitoring_api_enabled() {
    local project_id=$1
    local token=$2
    local response=$(curl -s -w "\nHTTP_STATUS:%{http_code}" --header "Authorization: Bearer $token" \
        "https://serviceusage.googleapis.com/v1/projects/$project_id/services/monitoring.googleapis.com")

    local http_status=$(echo "$response" | sed -n 's/.*HTTP_STATUS:\([0-9]*\)$/\1/p')
    local response_body=$(echo "$response" | sed -n '1,/^HTTP_STATUS:/p' | sed '$d')

    if [[ "$http_status" -ne 200 ]]; then
        echo "Error checking Monitoring API status for project $project_id:"
        echo "HTTP Status: $http_status"
        echo "Response: $response_body"
        return 1
    fi

    local state=$(echo "$response_body" | jq -r '.state')
    if [[ "$state" == "ENABLED" ]]; then
        return 0
    else
        echo "Monitoring API is not enabled for project $project_id."
        echo "State: $state"
        return 1
    fi
}

# Function to list buckets in a project
list_buckets() {
    local project_id=$1
    local token=$2
    local response=$(curl -s --header "Authorization: Bearer $token" \
        "https://storage.googleapis.com/storage/v1/b?project=$project_id")

    echo $response | jq -r '.items[].name'
}

# Function to get the sizes of all buckets using PromQL
get_bucket_read_ops() {
    local project_id=$1
    local token=$2

    local response=$(curl -s --header "Authorization: Bearer $token" \
        --header "Content-Type: application/x-www-form-urlencoded" \
        --data 'query=sum by (bucket_name)(rate(storage_googleapis_com:api_request_count{monitored_resource="gcs_bucket",method=~"Read.*|List.*|Get.*"}[30m]))' \
        "https://monitoring.googleapis.com/v1/projects/$project_id/location/global/prometheus/api/v1/query")

    echo $response | jq -r '.data.result[] | {bucket_name: .metric.bucket_name, ops: .value[1]}'
}

# Function to get the sizes of all buckets using PromQL
get_bucket_write_ops() {
    local project_id=$1
    local token=$2

    local response=$(curl -s --header "Authorization: Bearer $token" \
        --header "Content-Type: application/x-www-form-urlencoded" \
        --data 'query=sum by (bucket_name)(rate(storage_googleapis_com:api_request_count{monitored_resource="gcs_bucket",method=~"Write.*"}[30m]))' \
        "https://monitoring.googleapis.com/v1/projects/$project_id/location/global/prometheus/api/v1/query")

    echo $response | jq -r '.data.result[] | {bucket_name: .metric.bucket_name, ops: .value[1]}'
}
# Function to get bucket metadata (including location and storage class)
get_bucket_metadata() {
    local bucket_name=$1
    local token=$2
    local response=$(curl -s --header "Authorization: Bearer $token" \
        "https://storage.googleapis.com/storage/v1/b/$bucket_name")

    echo $response
}

# Check if PROJECT_IDS environment variable is set and valid
if [ -z "$PROJECT_IDS" ]; then
    echo "Error: PROJECT_IDS environment variable is not set or empty."
    echo "Usage: export PROJECT_IDS='project_id1,project_id2,...'"
    exit 1
fi

# Read the PROJECT_IDS environment variable into an array
IFS=',' read -r -a projects <<< "$PROJECT_IDS"

# Get the access token using either the provided service account key or gcloud
if [ -n "$SERVICE_ACCOUNT_KEY" ]; then
    echo "Using the SERVICE_ACCOUNT_KEY to get the access token."
    access_token=$(get_access_token "$SERVICE_ACCOUNT_KEY")
else
    echo "SERVICE_ACCOUNT_KEY is not set. Attempting to set access token using gcloud."
    access_token=$(gcloud auth application-default print-access-token)
    if [ -z "$access_token" ]; then
        echo "Failed to retrieve access token using gcloud. Exiting..."
        exit 1
    fi
fi

read_ops=()
write_ops=()

# Iterate over each project ID provided
for project_id in "${projects[@]}"; do
    echo "Processing project: $project_id"

    # List all buckets in the project
    buckets=$(list_buckets "$project_id" "$access_token")

        # Iterate over each bucket and match its size
    if is_monitoring_api_enabled "$project_id" "$access_token"; then
        echo "Monitoring API is enabled for project: $project_id"

        # Get the read/write operations of all buckets
        all_bucket_read_ops=$(get_bucket_read_ops "$project_id" "$access_token")
        all_bucket_write_ops=$(get_bucket_write_ops "$project_id" "$access_token")

        # Iterate over each bucket and match its size
        for bucket_name in $buckets; do
            echo "Processing bucket: $bucket_name"

            # Initialize operations to zero
            read_ops=0
            write_ops=0

            # Get the read/write operations for the current bucket
            read_ops=$(echo "$all_bucket_read_ops" | jq -r --arg bucket_name "$bucket_name" '. | select(.bucket_name == $bucket_name) | .ops // 0 | tonumber | round')
            write_ops=$(echo "$all_bucket_write_ops" | jq -r --arg bucket_name "$bucket_name" '. | select(.bucket_name == $bucket_name) | .ops // 0 | tonumber | round')

            # Calculate the total operations and cost using jq for arithmetic
            total_ops=$(echo "$write_ops $read_ops" | jq -n '[inputs] | add')

            # Print results
            echo "Read Rate: $read_ops ops/s, Write Rate: $write_ops ops/s, Total rate: $total_ops ops/s"

            # Get the region
            region=$(echo "$metadata" | jq -r '.location')

            # Add bucket operations to the list
            bucket_ops+=("{\"project\": \"$project_id\", \"bucket\": \"$bucket_name\", \"write_ops\": \"$write_ops\", \"read_ops\": \"$read_ops\", \"total_ops\": \"$total_ops\", \"region\": \"$region\"}")
        done
    else
        echo "Monitoring API is not enabled for project: $project_id"
    fi

done

# Output the result in JSON format
echo "["$(IFS=,; echo "${bucket_ops[*]}")"]" > $HOME/bucket_ops_report.json
cat $HOME/bucket_ops_report.json | jq 'sort_by(.total_ops) | reverse'
Helpful Links